2012.08.03

Node.js, Ant, Grunt and other build tools

I started using node.js to write build scripts since last year and even wrote a post about it before. The main reason why I decided to write my build scripts in plain JS is because I want them to be flexible and easy to edit. (Use the language you and your team are familiar with).

On this post I will try to cover some issues and the main reason why I’m not using a tool like grunt, buildr.npm, smoosh and gear on my projects. I’ll focus on Grunt since it is the most popular node.js build tool out there but the issues are present on other build tools as well (written in JS or not).

A couple days ago I sent a few tweets about it:

TL;DR; Build scripts should be real scripts and not configuration files.

Which “mistakes”?

Configuration over scripting

Descriptive languages aren’t a good way to solve arbitrary needs, configuration options never fit all scenarios. It doesn’t matter if you use XML/JSON/JS, it still won’t cover them all. If every edge-case requires a new setting you will soon end up with a highly complex system that is hard to understand and maintain.

Ant and Maven failed in recognizing this, they thought people would just write new JARs that executed custom tasks and that every kind of task could be easily represented with strings, but it isn’t a trivial process, specially when the tool is so focused around configuration. Ant-Contrib tried to solve some of the problems imposed by XML by adding loops and conditionals but still feels weirder than just writing a simple JavaScript script. Functions, Array#map, Array#filter and if/else to the rescue.

It’s JavaScript running on a very fast engine! Functions calls are inexpensive and the language is very flexible. Why use a single config file when you can call multiple methods? The grunt directives looks total non-sense to me, why not a method call instead of a magic string? I understand why bundling a template language into all strings (since JS doesn’t do automatic var interpolation) but I would still favor a function call that accepts the replacements/config that way the tasks don’t need to share state and aren’t that “magical”.

Also when you have a huge amount of configuration options you need good documentation and a lot of tests to cover all cases. – Complexity lies on the exceptions – Sometimes I spend more time reading documentation than it would take me to code the same feature from scratch for my own use case. On many times code that is easy to tweak is better than code that accepts multiple options to fit all scenarios (specially if you know how to code).

A good thing on Grunt is that you can register custom tasks from inside the “gruntfile” that way you can avoid the configuration step altogether, even tho most tasks seems to favor the configuration approach and are built as plugins.

Plugins / proprietary tasks

That for me is a huge mistake. I said it before and I will say it again STOP WRITING PLUGINS!. Even if all the tasks are indeed related to a common goal (helping the dev/deploy/test automation) you are still tying your solution to a single tool/platform. I’m not saying you shouldn’t reuse other modules, NPM is great at handling conflicting dependencies and complex dependency graphs, so abuse it. There is no reason why a helper function (concat, lint, minify, etc) wouldn’t work on multiple build tools, so why lock it to a single task runner? Why bundle it with the task runner / arguments parser? Why wrap every single node.js tool into a plugin for your build system? Why put all the “helpers” inside the same namespace when you have a real module system?

You are shooting yourself in the foot, if the project stop being maintained in the future you will have a tight coupled plugin that probably won’t fit the new build environment without a lot of refactoring. Requirements change all the time and tools evolve, build small blocks that can work together instead of a single monolithic system. Write methods that receive an input and produces an output. Don’t share state across the methods, don’t share config between methods. You should treat your build scripts as you treat your own application code, build it to scale and in a way that is easy to test.

Some tasks are very useful on other contexts besides build scripts, there is no reason why they should be tied to a build tool.

I added some feedback on this grunt issue explaining how to avoid tight coupling and why it’s a bad thing.

Global runner

Installing the build tool globally is a mistake. What if changes on the tool aren’t backwards compatible and you don’t remember which version you were using before? What if different projects requires different versions of the build tool? What if a future dev doesn’t have admin rights so he can’t install it globally? What if a future dev is on Windows and don’t know that he needs to set the environment PATH variable to be able to locate global commands installed with NPM?

Running node build mytask should work everywhere without admin rights, there is no advantage in running it as build mytask (“node” is a 4 letter word, easy to type and remember).

Since the “gruntfile” is already a JS script there is no point in wrapping the whole script inside a module.exports = function(grunt){ ... }; and executing it from an external script. You could simply require grunt as a common node.js library and call methods directly on it, the script would be more portable (could be called from other tools).

It was also a mistake to name the “gruntfile” with the same name as the grunt command since on windows the system will try to open the js file instead of running the command (unless you configure it to not do it).

Reinventing the “same” wheel

OK, I really like to reinvent the wheel, but in some cases that isn’t the smartest decision one can make. Specially when your wheel doesn’t work as good as an existing one and it consumes more time than you are willing to spend.

Yesterday I did a research and almost all the tools listed on NPM with the “build” tag have different implementations of the same tasks.

It will take an absurd amount of time before all the relevant tasks present on battle tested tools like Ant, Maven, Gradle are re-implemented on node.js specially if everyone starts coding the exact same tasks for every single node-driven build tool and just committing the same mistakes instead of experimenting and arguing about what could be improved.

Does it really make sense to every single build tool to re-implement arguments parsing since libs like nopt, commander.js and node-optimist already exists? Does it really make sense to have a rigid structure for you build tasks since you can just reuse one of these option parsers, write a few functions and glue together some other independent modules? If a task “depends” on another one you can simply call the other method before executing it or use something like Async.js to run the tasks in parallel or in sequence. (each “task” can be a separate function that receives a callback)

Learn from other peoples mistakes, don’t commit them all over again using a different technology.

Addendum

My intention with this post was to make people question their ideas/workflows so they can improve the tools and/or come up with “better” solutions. Depending on your needs the tools discussed above might be more than enough.

Try to keep your build files as clean as possible so other devs can understand how it works and can make changes to the file, you certainly don’t want to create a knowledge silo around the build script, problems can rise on different environments and requirements usually change.

I got really curious if it was possible to distribute Ant as a standalone package and execute it without needing to install so I ended up coding an adapter for node.js (even tho I said I wouldn’t do it). That way I can avoid rewriting some of the tasks all over again and can use it as an “utility belt” for now. More info about it on the project repository. The idea was inspired by Gradle since it leverages Ant tasks internally.

Automate all the things!

Further reading


Comments

Great points. I was going to reply here, but my response was so long that I decided to post it over at my blog.

Awesome post thanks! That's exactly what I think and why I finally removed grunt from my project! :)

As a counter point against the troublesome nature of Ant Xml control structures, , , etc... it should be noted that Ant allows you to code in JavaScript with embedded script blocks so you can access all the project tasks, functions, ant properties via it's Java bridge. I think your Ant connector to Node.js is a good idea as well.

I can't believe you were so brave to say something against Maven :))) hahahha Great post... Thanks

Hey Miller, found this post while looking for totally unrelated node build stuff. I wrote flour out of the same frustration.

It simply piggybacks on Cakefiles exposing useful build methods, so every task is a function - no configuration objects required. I plan to rewrite it soon to use it's own CLI so that plain JS can also be used. I've been using it on my own projects for almost a year and it's been a breeze.

Andrew Pennebaker

Yeah, the state of Node.js built tools is not good. NPM supports individual one-off shell commands, but not dependent tasks. Grunt's the opposite, way too complex. Every time I take a peek at the introductory Gruntfile.js, my eyes glaze over. I wish there were a direct Node.js equivalent to Ruby's rake, that one seems best.

@Andrew: you might want to have a look at Jake: http://howtonode.org/intro-to-jake

I see you mentioned gear, I created gear after exactly this frustration. I wanted a very light build system that can be easily extended with JS functions that process a stream of data in paralllel.

"before executing it or use something like Async.js to run the tasks in parallel or in sequence. (each “task” can be a separate function that receives a callback)"

gear uses async.js and does exactly that! It is really easy to extend the build system with your own functions that receive callbacks [1].

[1] - http://gearjs.org/#custom

have you heard of Gradle ?, (seems like you have not ) :)

[…] may first read this argument here, and Ben Alman’s answer to it here, as some good points are made in both […]

I'm a little late to this party (over a year!) but I recently started a project aimed at solving (most) of the issues you outlined in this post. It's built more as a task runner than a build system, but it can certainly be used as a build tool.

Check it out?! :) https://github.com/jaylach/whimper

I personally use a standard structure involving an "/etc/" folder in all of my projects where you can find some [git-branch-name].deploy scripts, beginning like "#!/usr/bin/env node" (or whatever language you like). Even if i copy/paste a lot of common tasks, it provides me flexibility when needed.

I guess every programmer can't skip some sort of initiation rite: implement your own web framework, build system, task runner or whatever else. Good for you! Just don't start a new religion :)

I'm looking for good arguments to use Grunt instead ANT Excellent post thank you!

Leave a Comment

Please post only comments that will add value to the content of this page. Please read the about page to understand the objective of this blog and the way I think about it. Thanks.

Comments are parsed as Markdown and you can use basic HTML tags (a, blockquote, cite, code, del, em, strong) but some code may be striped if not escaped (specially PHP and HTML tags that aren't on the list). Line and paragraph breaks automatic. Code blocks should be indented with 4 spaces.