Last week I posted an article on the future of Java build tools, which received lots of positive feedback both here on this blog and on JavaLobby. That post focused more on the differences between various build tools and advantages of one over another. This week I decided to follow up with an article, which instead will talk about something that all kinds of software have in common: the risk of feature creep (in the context of build tools).
Feature creep in action
Build scripts are very fragile pieces of software. They are treated differently than other programs that we write. The companies coding rules rarely apply to code written in XML, and refactoring or testing build scripts is something that’s hardly ever done. Because of that, it’s often easier to break the build by changing the build logic, than it is by changing application logic (the other one happens more often, but that’s because the application code is changed way more frequently).
The other problem, which is sometimes mentioned as a main headache with build scripts, is their complexity. Scripts can get thousands of LOC long, split (chaotically) over a dozen or more files. Can you imagine the pleasure of debugging those, when something starts failing?
In my opinion fighting complexity of build scripts can be done two-fold:
- by using a better build tool
- by reducing the scope of the script
Compared to regular application development, picking a build tool (Ant, Maven, Gradle, etc.) is more or less like choosing a programming language: you consider IDE support, friendliness of the syntax, sometimes performance characteristics (rarely ever done for build tools, though). There’s also your team’s qualifications, and more often than not, your companies standards (so the choice can be completely out of your hands).
On the other hand, there’s reducing scope, or managing feature creep if you will. The more your build script does, the longer and more complex it becomes. The more features you add over the time, the less maintainable it comes to be. This leads to conclusion that the smaller the scope, and the rarer the changes to it are, the better.
Build scripts can range from the most ascetic, which only build and package the application, to the most byzantine, which will download the applications sources, compile, package, test, generate reports, generate and upload documentation, install all prerequisites on the build and deployment machines and finally deploy and run the built application. As the CI servers grow in popularity, it is possible to do more and more with appropriate plugins instead of home-made build scripts, but while they help reducing the scope a little, there’s still a lot of stuff that they can not do.
It seems reasonable to ask: what should a build script actually do then? Should it stick to the very basics? Does it make sense to have A-Z build scripts which can do everything with a single command? I wish I could give you an answer to those questions right here and right now, but I can’t. The feature list of a build script is in my opinion heavily dependent on the application itself, and the technologies in use.
One thing that could possibly help fighting the feature creep, is defining what the build script should do in advance. What reports are relevant to this project? Should the script be able to deploy the application, and if yes, which deployment scenarios should it support? Should it just generate, or also upload the build artifacts, like .jar files or documentation somewhere? What kind of notifications should be implemented, and so on. With this, a build script once created, should be left relatively unchanged. However, this seems rather restrictive, and definitely not “agile” at all.
The more agile approach could be by mimicking what has been done in the Grails framework, which has a very friendly integrated build system (based on Gant), which can be easily extended. As Grails is very keen on “convention-over-configuration”, the commands reside in appropriately named files, thus helping you find the code you’re interested in easier. The separation is cleaner, and adding your own command is unobtrusive and straightforward. The problem here is mostly time, because rolling a build system like that on your own is definitely harder than using something “out of the box”.
So the best practices I can name are: do as little as possible, but as much as necessary and keep your build code in bite-size, easily identifiable pieces - not one 10k LOC build file, but maybe dozens of small files named by convention. Not much, huh? That’s why I’m counting on you, to comment on this post, and share your opinions.
Can you share any good/bad practices for build scripts? Do CI servers make your life easier, and if yes, how so, and which one do you use? How often does your build script change? Do you have any particular way of organizing your build scripts?
Waiting for you feedback,
Adam
PS. Thanks to Pete Johnson for inspiring me to write this:)
Tags: ant, build tools, grails, maven
June 9th, 2009 at 3:37 am #Philippe Paravicini
I expect the build to build and package artifacts for production/staging environments. I typically de-compose my projects into smaller projects/modules so that I don’t have to use the whole engine if I am only interested in the carburetor. Hence, I find the multi-project features very important.
I think that build conventions are very important, because it reduces the learning curve when having to learn a new project. Nevertheless, conventions can only go so far, and it is useful to be able to do small customizations to a build without launching on a search of the proper convention for specialized useages, or having to develop a more formal plug-in in a compiled language.
I don’t like to store the jars in the source control system (svn/cvs), because it can cause the source control repository to become quite large, more difficult to backup, and hence increases the chance of data loss of the much more valuable source code in the project. I thus like the ability to access public lib repositories, and create private lib repositories when necessary.
I also like the ability to create a runnable jar file that contains both a project, and all its supporting runtime libraries.
We happen to use and like maven-1.1, because of its emphasis on convention, easy accessibility to ant tasks, dependency management, multi-project build support, and scripting via jelly in its maven.xml file. It’s not perfect, and the documentation is not great, but it works well for our purposes.
I have been reluctant to migrate to maven-2 because a lot of the feedback that I have read seems to indicate that it ads about as much overhead as it saves time, and maven-1.1 has been working a-ok for us. I am also leery of migrating to other build systems, because I feel that to properly evaluate a build system one needs to use it on a real life project with a multi-person team, and this represents quite an investment in time. For better or for worse, maven-1.x and maven-2 are at least used widely use.
So far, I have only seriously looked at Gradle as an alternative, and I like the fact that in many ways it is very similar to maven-1.1 in its ability to operate by convention, yet has a lightweight scripting language at its disposal, accessibility to ant tasks, and most of the core tools that maven-1.1 seems to have.
March 20th, 2010 at 4:21 am #playfish
im continually bouncing all over the internet almost all of the afternoon which means that I have a tendency to browse a whole lot, which unfortunately isn’t normally a beneficial matter as many of the websites I look at are made up of worthless trash copied from some other websites a trillion times, nevertheless I’ll hand it to ya this site is honestly not bad at all and also delivers some unique information, therefore cheers for helping to stop the phenomena of just replicating other peoples’ blogs, in case you ever want to play a few hands of zynga poker together just let me know - you have my email :)
June 16th, 2010 at 5:34 pm #Free Artikle
gives employ a good ınternet site decent Gives gives thanks for the working hard to help me personally
August 26th, 2010 at 5:22 pm #****************MATRIX*************************
Fajny artykuł, oby takich więcej Adasiu!
August 27th, 2010 at 9:22 pm #baby jumperoo
Thank you for your wonderful publish. Actually informative and I enjoyed reading it along with your other articles. Thank you for sharing and continue the excellent work.
May 18th, 2253 at 2:20 am #Adam Leggett
As part of our development of Kundo we’ve been grappling with similar issues such as - what should the scope of our build tool be? How much abstraction should it provide?
For example, in terms of scope we have decided to not try and create a ‘project site’ in the style of Maven. We feel that if a build generates raw data about tests, static analysis etc developers can use other tools, such as Hudson and Sonar to produce the reports.
Abstraction is a tricky one. In complex builds you can often find a proliferation of ‘edge cases’ such as ad-hoc scripting tasks and unusual packaging scenarios; situations when conventions do not apply. We decided that we wanted a build tool that could in these circumstances simply leverage the power of Ant, without being bound to a ‘build lifecycle’. But the normal modus operandi would be declarative and convention based (in Kundo this is realised using a build ‘recipe’).
IMO deployment to a managed environment is different problem domain. I’m not sure its a place where Java based systems are a best fit. The type of functionality you need is not tied up in JAR files - it’s in *nix commands. Something like Fabric (http://www.nongnu.org/fab/), which is python based, is an interesting emerging candidate in the deployment automation space.