One more commit

Last week I posted an article on the future of Java build tools, which received lots of positive feedback both here on this blog and on JavaLobby. That post focused more on the differences between various build tools and advantages of one over another. This week I decided to follow up with an article, which instead will talk about something that all kinds of software have in common: the risk of feature creep (in the context of build tools).

Feature creep in action

Build scripts are very fragile pieces of software. They are treated differently than other programs that we write. The companies coding rules rarely apply to code written in XML, and refactoring or testing build scripts is something that’s hardly ever done. Because of that, it’s often easier to break the build by changing the build logic, than it is by changing application logic (the other one happens more often, but that’s because the application code is changed way more frequently).

The other problem, which is sometimes mentioned as a main headache with build scripts, is their complexity. Scripts can get thousands of LOC long, split (chaotically) over a dozen or more files. Can you imagine the pleasure of debugging those, when something starts failing?

In my opinion fighting complexity of build scripts can be done two-fold:

• by using a better build tool
• by reducing the scope of the script

Compared to regular application development, picking a build tool (Ant, Maven, Gradle, etc.) is more or less like choosing a programming language: you consider IDE support, friendliness of the syntax, sometimes performance characteristics (rarely ever done for build tools, though). There’s also your team’s qualifications, and more often than not, your companies standards (so the choice can be completely out of your hands).

On the other hand, there’s reducing scope, or managing feature creep if you will. The more your build script does, the longer and more complex it becomes. The more features you add over the time, the less maintainable it comes to be. This leads to conclusion that the smaller the scope, and the rarer the changes to it are, the better.

Build scripts can range from the most ascetic, which only build and package the application, to the most byzantine, which will download the applications sources, compile, package, test, generate reports, generate and upload documentation, install all prerequisites on the build and deployment machines and finally deploy and run the built application. As the CI servers grow in popularity, it is possible to do more and more with appropriate plugins instead of home-made build scripts, but while they help reducing the scope a little, there’s still a lot of stuff that they can not do.

It seems reasonable to ask: what should a build script actually do then? Should it stick to the very basics? Does it make sense to have A-Z build scripts which can do everything with a single command? I wish I could give you an answer to those questions right here and right now, but I can’t. The feature list of a build script is in my opinion heavily dependent on the application itself, and the technologies in use.

One thing that could possibly help fighting the feature creep, is defining what the build script should do in advance. What reports are relevant to this project? Should the script be able to deploy the application, and if yes, which deployment scenarios should it support? Should it just generate, or also upload the build artifacts, like .jar files or documentation somewhere? What kind of notifications should be implemented, and so on. With this, a build script once created, should be left relatively unchanged. However, this seems rather restrictive, and definitely not “agile” at all.

The more agile approach could be by mimicking what has been done in the Grails framework, which has a very friendly integrated build system (based on Gant), which can be easily extended. As Grails is very keen on “convention-over-configuration”, the commands reside in appropriately named files, thus helping you find the code you’re interested in easier. The separation is cleaner, and adding your own command is unobtrusive and straightforward. The problem here is mostly time, because rolling a build system like that on your own is definitely harder than using something “out of the box”.

So the best practices I can name are: do as little as possible, but as much as necessary and keep your build code in bite-size, easily identifiable pieces - not one 10k LOC build file, but maybe dozens of small files named by convention. Not much, huh? That’s why I’m counting on you, to comment on this post, and share your opinions.

Can you share any good/bad practices for build scripts? Do CI servers make your life easier, and if yes, how so, and which one do you use? How often does your build script change? Do you have any particular way of organizing your build scripts?

Waiting for you feedback,

Adam

PS. Thanks to Pete Johnson for inspiring me to write this:)

UPDATE: A follow up to this post has been posted here.

With a short post on dzone Adam Bien revived an old flame-war: should we use Ant or Maven as our build tool. The post itself seems a little biased on the maven side,  but not much. The only really questionable sentence is:

The real strength of Maven is its dependency management. (…) Ant doesn’t have such capabilities out-of-the-box (except via Ivy) and it will take a considerable amount of work to emulate Maven’s dependency management with Ant.

Although it’s certainly true that Ant does not have dependency management built in, it’s fairly easy to incorporate Ivy into your build.xml, so the “considerable amount of work” part is in my opinion rather misleading. UPDATE: There are of course other tools, which can be integrated with Ant for dependency management: Maven Ant tasks and Savant (thanks to Brennan Spies and Brian Pontarelli for pointing that out).

However, given a choice “Ant or Maven”, I would most likely choose the former. I can see why so many people today use Maven as their primary build tool. It is pretty cool to have this drop-in build script, that with only a few lines of configuration will give you so many possibilities: dependency management, built in tasks for compiling and packaging applications, integration with Jetty, neat project web site, integration with cobertura, pmd or findbugs. In a situation, where one starts a new project, which is fairly typical (say a web application using Spring & Hibernate) this seems like a very good choice. I will say even more: for a project like that, this is probably a good choice.

No one really wants to write build scripts. This is the kind of code that does not have any real business value. Unfortunately that last statement holds only if the build script actually works, otherwise it has business value: a negative one. I feel that a build system like Maven is really great until you start encountering problems with it. Maybe your project just does not fit so very well in the convention. Maybe plugins you want to use conflict with each other. There’s lots of points of failure in a build system like Maven - and oftentimes it’s very hard to track the source of them and fix them.

The choice of Ant over Maven is not a very clear one. It truly is a chore to be forced to write all this build code by hand. But when the things go south, the build script is just another piece of code you’ve written. If you did it right, you should be able to find your way around that code, and fix it. No magic here.

OK, I’ve written enough to spill some oil to the flame-war fire. Now to the point of this article: are Maven and Ant our only choices? Is there really nothing better than that?

I claim there is. Over the last couple of years we’ve seen many such projects spring up to life. Those tools no longer use XML to define build logic, but real programming languages like Groovy, Ruby or Python(? - I thought I’ve seen an announcement somewhere for build tool in Python, but can’t find it now). They often allow dependency management (through Ivy or Maven) or Build By Convention. Some of those tools are:

• GRADLE
• Gant
• Kundo
• Raven
• Buildr
• UPDATE: Schmant (thanks to Erik Tjernlund)
• UPDATE2: Lancet (thanks to Vassil Dichev)

My favorite build tool was Gant, up until the first release of Gradle was announced a couple of months ago. Gant is basically a Groovy way of invoking Ant tasks, which by itself gives you a lot compared to a good old build.xml. By using a real programming language instead of XML, some tasks which could become tedious in XML, are very easy: extracting common code into methods, loops, conditional logic (yes, I know about ant-contrib and use it whenever I have to use Ant, but trust me, it’s not the same). The fact is, that XML is not a programming language and it should not be used as one. Build scripts in languages like Groovy or Ruby are more concise and easier to read. There’s less clutter, boilerplate code, and your build script can be structured better.

In my opinion, the future of build automation in Java environments is Gradle, or at the very least some other tool, which will exhibit similar properties. Gradle is the kind of drop-in build script that I like: for simple, standard projects it allows you to use itself with just a few lines of configuration. It even allows transitive dependency management without using Maven or Ivy repositories. You know, how when you first learn about this “dependency management” thing, you wrote down your dependencies in ivy.xml or pom.xml and it took forever while “Maven downloaded the Internet”? How eventually you had to install a private repository just to store those dependencies which could not be found in the Maven repositories? How you had to fix pom.xml files, because some of the artifacts didn’t declare all their dependencies? Well, now you can have DM without all those problems. Just store the required libs in the SVN repository, and switch to “real” DM when needed.

On the other hand, for bigger projects, among many other features Gradle has first-class support for multi-project builds (for example, you can specify project dependencies, and when you build one of the sub-projects, all it’s dependencies will be built too). There’s plenty of other features and cool “magic tricks” you can do with Gradle. Check out the User’s Guide on the project web site. It’s about 100 pages long, and provides plenty of information. I mean it: you owe it to yourself to give up an hour of your time and read that manual to see how much better your life would be if you used Gradle :o)

That being said, I have to admit: Gradle might not be suitable for “production use” yet. The project is still fresh out of the oven - it’s less than a year old. It’s impressive how much Hans Dockter and other commiters managed to achieve in that time, though. The documentation is IMHO very good - short enough and informative. If you are looking for a replacement for Ant - Gradle is the thing for you. If you’re looking to replace Maven, you might want to wait a couple of months, since Gradle does not have support for many of the features Maven users rely on, like the project web site or generating project files for main IDEs. There’s also currently no IDE support for Gradle.

To summ up: while there might not be a “one size fits all” solution to build automation, new tools being developed seem to come close. By allowing both build by convention, and combining it with do-it-yourself-if-you-like approach Gradle has a shot at attracting both Ant and Maven fans. In the future I expect build scripts to be written in programming languages like Groovy or Ruby, and Gradle is likely to gain a significant market share.

Not everybody knows that you can specify for some ant targets to run only when some set of files was updated. It is particullary useful when working with GWT or Ivy.

In my GWT projects I usually have tasks like run-shell (which runs a hosted-mode browser) and compile-gwt (which runs the GWT compiler). The run-shell task depends on the compile-gwt task, but this would cause my GWT code to be recompiled every time I run the hosted browser. You may ask - why would you even rerun GWT shell, if you didn’t modify any of the GWT source files? The answer is quite simple - sometimes you don’t make any changes on the client, but only on the server-side and wan’t to see how these changes affect the client. Or maybe you just don’t want to recompile your GWT code every time you need to create the .war file and deploy your application or run ivy:retrieve everytime you run your build script. Anyway, I find this trick pretty usefull.

So how does the magic hapen? The whole trick is possible thanks to Ant’s uptodate core task. Below is the snippet from the example build.xml file:

<uptodate property="gwtBuild.notRequired"
    targetfile=".gwtBuild">
        <srcfiles dir="src"/>
</uptodate>

Uptodate task sets Ant property (here called “gwtBuild.notRequired”) when targetfile is newer (more up-to-date) than any of the files in the srcfiles fileset. Here I’m comparing it to the all directory contents, but srcfiles is a normal ant fileset so you can always use include and exclude tasks within. You can also use mapper subtask instead of targetfile for more complicated use cases.

So now that we know how the magic is done, it’s time to put it to a good use:

<target name="compile-gwt" unless="gwtBuild.notRequired">
    <!-- invoke the GWT compiler here -->
    <touch file=".gwtBuild"/>
</target>

The touch target will create .gwtBuild file if it doesn’t exist or update it’s last modified time otherwise. It should be invoked after the compilation succeeds, bacause if it wasn’t, in case of compilation failure the target would not be executed again unless any of the source files was modified. The “compile-gwt” target will execute only if the gwtBuild.notRequired property is not set thanks to the “unless” attribute.

Now for the finishing touch we need to delete the flag file in our clean task:

<target name="clean">
    <!-- Delete the flag file for compile-gwt -->
    <delete file=".gwtBuild"/>
    <!-- Delete other files and directories -->
</target>

To illustrate usage of this technique I’m attaching to this post a simple GWT project with an Ant script. The project was tested on Windows with GWT 1.5M1 and GWT 1.4.61. To run it simply change the gwt.home property in build.properties file to point to your GWT installation directory.

Click here to download sample project

Hope you’ll find this tip usefull:)

Psyho