2012.01.03

Node.js as a build script

There are a lot of build tools that covers specific use cases and/or try to cover as many scenarios as possible, amongst the most famous ones are make, rake, Ant and maven. I’m going to talk about why I’ve been favoring plain Node.js scripts as my “build tool” and how to do some simple things. This work flow may not be the best one for you and your team, understand the reasoning behind it and pick the tools based on your needs and preferences.

Why?

Most of my projects I just need some basic features like file concatenation, running an optimizer/compressor, some simple string replacements or running some other 3rd party command-line tool, sometimes I also need to do some things that aren’t that common (code generation, documentation, etc..) and also depending on the project my structure may not fit existing tools.

I’ve been using Ant for the past few years on almost all my projects, the good thing about Ant is that it’s very mature, can run on multiple platforms, well documented, pre-installed on Mac OS X, popular (specially for Java and AS3 developers), and almost anything you want to do you just need to search for the “term” plus “ant task” (eg. “concat files ant task”, “delete files ant task”) and the first results probably describe how to do what you need. The problem with Ant is that XML is a markup language and when you try to do things that requires a programming language (logic, loops, etc..) things get nasty really quickly so you need to extend Ant by using the script task or creating an external command line tool/app or an Ant plugin (Java), not really convenient or straightforward (too much overhead, need to learn new APIs and/or languages…)

Make and bash scripts are way more concise than Ant, but they also impose a barrier to people that doesn’t know how to edit them. Another problem with bash is that it isn’t very portable which for me is a big issue, I work with multiple devs and who knows which OS they might be using or how the build process will be triggered? It’s very important for users to be able build the project without too much trouble and also that multiple developers on the team understands how the build process work and how to edit the files, it will avoid “knowledge silos” and “code ownership”. If your team is proficient in JavaScript I suggest that the build script should be written in JavaScript, if the team is more comfortable with Python or Ruby (or anything else) than write it on these languages instead…

A few months ago I finally decided to try “vanilla” Node.js scripts and I’m very pleased with it, I see it as an excuse to learn Node.js API and it’s libraries. It’s also very fast to code since I’m very familiar with JavaScript syntax, I don’t need to spend time searching how to do things I simply code it, also since I’m not constrained by an existing tool structure I can go “nuts” and code whatever I want and the way I want it, the complexity of my scripts are only limited by my needs and deadlines.

I tend to think that tools that try to fit all scenarios end up not fitting anything properly, so I prefer code that is easy to edit/tweak than code that can be overly customized. Having the build script written on the same language as the project is a very good thing since contributors will feel more comfortable at updating it and improving it, low entry barrier is usually a good motivation for contributing.

How?

I’m not really using any specific build tool, just putting together a few npm modules as needed, kinda following the Node.js philosophy of keeping packages small and loosely coupled, some of the modules I’ve used so far and that helped a lot in the process:

  • shelljs
    • ShellJS is a portable (Windows included) implementation of Unix shell commands on top of the Node.js API. Also contains a basic task runner.
  • commander
    • very useful tool to generate self-documenting command-line interfaces, using it on most of my Node.js command-line tools.
  • node-optimist
    • option parsing similar to commander.
  • wrench-js
    • recursive file operations (delete, create, copy directories)
  • node-glob
    • returns a list of filepaths based on a string pattern.
  • handlebars / mustache
    • for templating in case you need to do some code generation and/or process some text files.
  • UglifyJS
    • JavaScript minification.
  • async
    • Asynchronous utilities and control flow.
  • node-ant
    • Apache Ant adapter to run Ant tasks from inside your node.js app. (experimental)
  • prompt
    • command-line prompt for node.js

I’m also calling external tools when needed using the child_process.exec command, so you can still call some JAR files or other command-line tools if needed…

I’m not using any build tool because I want flexibility and also because it’s very easy to put those things together, there are a plenty of Node.js modules available through npm that does all sorts of things and it’s a very good excuse to learn how to use them and to discover new libraries.

Examples

Most of the examples are very trivial, use it as a reference not as the only way of doing it.

Concat files

You just need to read the content of a file list and merge them:

var _fs = require('fs');

function concat(opts) {
    var fileList = opts.src;
    var distPath = opts.dest;
    var out = fileList.map(function(filePath){
            return _fs.readFileSync(filePath).toString();
        });
    _fs.writeFileSync(distPath, out.join('\n'));
    console.log(' '+ distPath +' built.');
}

concat({
    src : [
        'foo/bar.js',
        'foo/lorem.js',
        'foo/maecennas.js'
    ],
    dest : 'dist/myAwesomeScript.js'
});

Or you could use shelljs to simplify the process.

// shelljs is great! use it!
var shell = require('shelljs');

cat([
  'foo/bar.js',
  'foo/lorem.js',
  'foo/maecennas.js'
]).to('dist/myAwesomeScript.js');

It’s important to notice that I’m not concatenating files that much since I’m using r.js (RequireJS optimizer) on almost all my projects, r.js can be also used to merge CSS files (inline @imported stylesheets).

Minify JS file with UglifyJS

function uglify(srcPath, distPath) {
    var
      uglyfyJS = require('uglify-js'),
      jsp = uglyfyJS.parser,
      pro = uglyfyJS.uglify,
      ast = jsp.parse( _fs.readFileSync(srcPath).toString() );

    ast = pro.ast_mangle(ast);
    ast = pro.ast_squeeze(ast);

    _fs.writeFileSync(distPath, pro.gen_code(ast));
    console.log(' '+ distPath +' built.');
}

uglify('dist/awsum.js', 'dist/awsum.min.js');

Minify JS with google closure compiler (run external JAR)

You can run shell commands with child_process.exec. Very useful for reusing existing command-line tools and code that isn’t Node.js specific.

var exec = require('child_process').exec;

function compile(srcPath, distPath) {
  // exec is asynchronous
  exec('java -jar build/compiler.jar --js '+ srcPath +' --js_output_file '+ distPath,
    function (error, stdout, stderr) {
      if (error) {
        console.error(stderr);
        process.exit(1);
      } else {
        console.log(stdout);
        console.log(' '+ distPath + ' built.');
      }
    });
}

compile('dist/myAwesomeScript.js', 'dist/myAwesomeScript.min.js');

Use commander to parse CLI arguments

Commander.js is very useful for parsing command-line arguments, specially since it is self-documenting and extremely easy to use.

var DIST_PATH = 'dist/awsum.js';

var _cli = require('commander'),
    _fs = require('fs');

// global options
_cli
    .version('0.0.1')
    .option('--silent', 'suppress log messages.');

// commands
_cli
    .command('deploy')
    .description('Optimize site for deploy.')
    .action(deploy);

_cli
    .command('purge')
    .description('Delete old files from dist folder.')
    .action(purgeDeploy);

// parse commands
_cli.parse(process.argv);

function deploy() {
    purgeDeploy();
    build();
}

function purgeDeploy(){
    _fs.unlinkSync(DIST_PATH);
    if (! _cli.silent) {
        console.log(' Deleted deploy files!');
    }
}

function build(){
    // concat files here or do anything that generates the dist files
    if (! _cli.silent) {
        console.log(' Built!');
    }
}

node build -h will show the available commands and options. Running node build deploy will execute the “deploy” command, node build deploy --silent will execute the “deploy” command but suppress log messages.

Code generation / String replacements

Template engines aren’t useful only for HTML documents, they can be used for simple string replacements and also for code generation. I’ve been favoring Handlebars.js because of the “helpers” and nested paths support. I’ve been using it to add some basic info like version number, build date and even to generate some source files.

On AMD-Utils I’m using it to generate the package files and also to update the test runners (so I make sure I have unit tests to all modules).

If you are distributing the library through npm and/or have a packages.json file it is a good idea to reuse the data on the build as well so it doesn’t get out of sync. Let’s say we have a license at the top of the file like:

/**@license
 * My Awesome Lib <http://example.com>
 * Version: 0.1.0 (Tue, 03 Jan 2012 03:48:58 GMT)
 * License: MIT
 */

It could be described as:

/**@license
 * {{name}} <{{homepage}}>
 * Version: {{version}} ({{build_date}})
 * License: {{license}}
 */

And we could replace the data using handlebars:

var DIST_PATH = 'dist/awsum.js';

var _handlebars = require('Handlebars'),
    _fs = require('fs');

var distContent = _fs.readFileSync(DIST_PATH).toString();
var template = _handlebars.compile(distContent);

//reuse package.json data + add extra stuff
var data = require('package.json');
data.build_date = (new Date()).toUTCString();
data.license = data.licenses.map(function(license){
  return license.type;
}).join(', ');

_fs.writeFileSync(DIST_PATH, template(data));

Lint JS files and prompt user to confirm before continuing added 2012/01/03

Joss Crowcroft asked for it in the comments and since it isn’t hard to implement (thanks to Commander.js .choose() method) I decided to do it.

var _cli = require('commander'),
    _jshint = require('jshint'),
    _fs = require('fs');


function lint(path, callback) {
    var buf = _fs.readFileSync(path, 'utf-8');
    // remove Byte Order Mark
    buf = buf.replace(/^\uFEFF/, '');

    _jshint.JSHINT(buf);

    var nErrors = _jshint.JSHINT.errors.length;

    if (nErrors) {
        console.log(' Found %j lint errors on %s, do you want to continue?', nErrors, path);
        _cli.choose(['no', 'yes'], function(i){
            if (i) {
                process.stdin.destroy();
                if(callback) callback();
            } else {
                process.exit(0);
            }
        });
    } else if (callback) {
        callback();
    }
}


// run
lint('dist/awsum.js', function(){
    console.log(' Built!');
});

Protip (added 2012/08/03)

Keep the build dependencies on the package.json file (so it’s easier to update them) and commit them to your VCS so the next dev can simply clone the repository and start working. You never know when a dependency might become unavailable and/or if future updates to the libraries will break your build. I recommend using static version numbers and/or npm shrinkwrap (so every single npm install gets the same dependencies) or if you are confident the project maintainer follows semver properly use Tilde Version Ranges so you get the bug fixes updates without breaking existing code (eg. ~1.0.3). - Imagine yourself in 2 years not able to run the build script because some nested dependency is not compatible anymore - If something doesn’t work the first thing to do is to force an install of the minimum version listed on the package.json (problem might be caused by an incompatible dependency). Example package.json.

More

There are a few Node.js general purpose build tools like shelljs make-tool (which I’ve been using a lot), node-jake and cake (coffeescript), and also some tools that assumes your project is following a specific structure and contains options for lint/concatenation/minification like smoosh, buildr.npm and grunt. I tend to prefer tools that doesn’t impose a specific structure since we never know if the constraints might become an issue in the future or when the project requirements will change (it usually does), but these tools may be a good option depending on the project.

If you need to do something that isn’t listed here search npm, there is a big chance that someone already coded a package that does what you need, if not, do it yourself and share with the open source community.

I’ve also created a gist with a RequireJS optimization example + copying/filtering files, not as easy to follow as the previous examples but might be helpful to someone trying to make something a little bit more complex. I also have a complex build script (and better organized) on MOUT and a simple one that uses most of the stuff described on this post on Crossroads.js.

That’s it for now.

Edit 2012/01/03: Added JSHint example with confirm prompt and link to caolan/async.
Edit 2012/05/08: Added link to gist containing a full build script.
Edit 2012/08/03: Added link to node-ant, updated the concat code snippet and added protip section.
Edit 2012/11/01: Added links to amd-utils and Crossroads.js build scripts. Added ShellJS to the list of resources.
Edit 2013/10/16: Added some shelljs examples and info about npm shrinkwrap.


Comments

Fab write-up, especially happy to find out about wrench-js which looks like it'll save me tonnes of hassle.

Another super useful tool for these scripts, checkout the rimraf npm module (for doing "rm -rf" operations) at [https://github.com/isaacs/rimraf](https://github.com/isaacs/rimraf). This is pimp for deleting existing production directories, and cleaning up the built filesystem after running e.g. r.js (which leaves loads of unnecessary dependencies in place).

You might want to include a section about incorporating the r.js build tool as part of your build process - I didn't like having to run r.js THEN our build script separately, so I made it a part of the build script using child_process.exec, like this: (hope the code comes out correctly, feel free to reformat)

exec('r.js -o ./path/to/app.build.js', function(err) {
    if ( err ) {
        console.log('error in r.js optimizer!');
        throw err;
    }
    console.log('r.js optimizer successfully completed...');
   // Rest of build script
});

Oh, and another thing - the requireJS optimizer adds a lot of bloat to the main output file in two ways: it concatenates all of the licenses from 3rd-party scripts (so you can end up with 30 lines of comments) - so I added a post-optimizer operation to replace those with a single license block linking to a text file with all the license info (which I keep by hand for now, but could also generate)

Secondly, when you use requireJS to include HTML-like templates and they're inlined into your main javascript app file, all tabs and newlines are preserved as escaped characters, which adds significant weight if you tab everything religiously (I'm guilty)

So I use a simple loop over the built production directory and do a simple regex replace to strip tab and newline characters out, as well as HTML comments. Saved a bunch of kb for us.

One other avenue I'd be interested to explore is running JSHint as part of the build process - so in the first step, it validates all the files, and if errors are found, the script throws a confirmation like "321 errors in JS files. continue? y/n" into the command line. Any take on that?

Other than that, really handy article thanks :o)

@josscrowcroft that's exactly the kind of situation where flexibility is crucial, if we were using a tool that only had configuration options I doubt it would cover all the scenarios, and if it did it would be hard to configure it. Commander.js has the .prompt() and .choose() methods which are very handy, just added a new example with the JSHint+confirm. thx!

Nice! Cheers for adding that, I wasn't aware of commander's prompt and choose methods. Super useful.

Nice article Miller! I've recently started doing this myself, and am a huge fan. I just released a reusable command line tool for optimizing LESS into CSS for an entire project. I use it inside of a custom build.js that also runs r.js. I'm very happy with the solution and am looking into adding more steps like linting and tests (w busterjs).

Thanks for pointing out commander, I'll definitely be using it. FYI node-glob doesn't support windows in case any of your team uses it. I've been using node-dive and node-diveSync then doing the filtering myself, and it works well.

Thanks for sharing!

[...] for testing and CINode.js + Express (speaking of Node, Miller Medeiros has an excellent write-up on how to use it as a build script)MongoDB as a noSQL data-storeI know that some developers may ask [...]

Hey, just wanted to say thanks for the shout-out on wrench-js. It's much appreciated, great post!

I think node-css-compressor is also need when u need compress some css!

funny enough I just found out a post that says exactly the same thing but in a different context: Micro-Build Systems and the Death of a Prominent DSL

It's strange how we get into the same conclusions than other people even working on completely different stuff...

[...] your websites or apps. You can have a check how Miller Medeiros is using node.js as a build system http://blog.millermedeiros.com/node-js-as-a-build-script/ You can download ANT from here: http://ant.apache.org/ or via [...]

Great write-up... I've been doing something similar.. http://frugalcoder.us/post/2012/06/21/vs-node-builder-min-merge-js-and-css.aspx ...combined with a package.json, it's amazing what you can do with "npm install" then "node build.node.js" ... I've put a build directory under my project, with build related stuff.

Not sure if jshint has changed since the post but JSHINT.data() is a required call to get access to the error objects.

Corrected code from your post.

var nErrors = _jshint.JSHINT.data().errors.length;

[...] started using node.js to write build scripts since last year and even wrote a post about it before. The main reason why I decided to write my build scripts in plain JS is because I want them to be [...]

[...] JS jako deploy nástroj Node.js as a build script | Blog | Miller Medeiros A blog about design, code and some other stuff Intro to Jake - JavaScript build tool for Node.js - How To Node - NodeJS Learn the zen of coding [...]

[...] nitty gritty, we are going to need 2 functions one that concats and one that minifies scripts. Here one article that really helped me get jump [...]

[...] Node.js as a Build Script was the article I used to base my scripts off. I am using a slightly modified version, but the majority of my build script contains what in the article. [...]

Hello,

I'm the author of Node FileUtils, a huge library to work with files and directories asynchronously. I found this page searchig on google and I've seen that you mentioned Wrench-js. I didn't know about this library and after seeing the code I can't understand how this library has 152 stars and 29 forks, really, I can't understand.

First, it promotes the use of synchronous I/O access. Node.js is asynchronous in its nature, the synchronous functions exists for vague developpers who don't care about performance. Node.js should remove them or put a giant warning. If you're using Node.js do it asynchronous, do it the right way. Yes, shelljs it's very nice but useless. If you plan to write synchronous code with node forget it.

Second and most important, the asynchronous functions are bugged, bad design, bad error management. Take or example readdirRecursive, the callback it's executed a lot of times, when a directory is found and when an error is produced, but because is asynchronous, you can obtain a lot of parallel errors, bad bad bad. All you put in the callback is executed more than one time. This is a very bad approach.

Third, it lacks many features.

[...] Node.js as a Build Script Tags: best practices, build, continuous integration, nodejs, opensource, tools, tutorial Comments (0) [...]

In the section you need to require('fs') . It is probably a copypaste error. :)

// settings var FILE_ENCODING = 'utf-8', EOL = '\n'; // setup var _fs = require('fs'); ...

Keep up the good work! I;m also in the process of migrating all my build "scripts" to node.js (without any build "frameworks").

Inspired by this post I created a node module that versions your static assets and updates references to them in files.

It uses md5 hashing for the version numbering so that if the file contents haven't changed then it won't blow the browser cache.

All other versioning modules I found required adding markup to the files linking to the static assets. So I wrote a module where all you have to do is list the locations of files and the module takes care of the rest.

Check out the github repo.

[...] Node.js as a build script [...]

Paragraph writing is also a excitement, if you be familiar with after that you can write if not it is complex to write.

my web blog ... [free psn code](http://Www.Youtube.com/watch?v=Zbktsk84t_w "free psn code")

Hi, I log on to your new stuff regularly. Your writing style is awesome, keep it up!

Feel free to surf to my site - Professor Joseph Plazo ([Christina](http://writerstogether.com/members/miriat67/activity/24191/ "Christina"))

I love your blog.. very nice colors & theme. Did you design this website yourself or did you hire someone to do it for you? Plz answer back as I'm looking to construct my own blog and would like to know where u got this from. thank you

My page :: Joseph Plazo Ph.D ([Santiago](http://ssjason.com/activity/p/5808/ "Santiago"))

Hi, i think that i saw you visited my website so i came to “return the favor”.I'm trying to find things to enhance my web site!I suppose its ok to use a few of your ideas!!

My site: [comptoir quartz ou granit ville de Quebec](http://www.scribd.com/doc/231357856/Le-Granit-Vous-Connaissez "comptoir quartz ou granit ville de Quebec")

Nice blog here! Also your website loads up very fast!

What web host are you using? Can I get your affiliate link to your host? I wish my website loaded up as quickly as yours lol

my homepage ... [Clash of Clans attack strategy](http://Youtube.com/watch?v=w2qNWWet2pc "Clash of Clans attack strategy")

Quality articles or reviews is the key to invite the people to go to see the website, that's what this site is providing.

Here is my page: [quartz comptoir Quebec](http://www.scribd.com/doc/231357854/Le-Granit-en-Quelques-Mot2 "quartz comptoir Quebec")

Howdy! I'm at work surfing around your blog from my new iphone 4! Just wanted to say I love reading your blog and look forward to all your posts!

Carry on the great work!

Here is my blog ... [Trikes for sale](http://www.leboda.net/wiki/index.php?title=Utilisateur:ChristieCarvoss "Trikes for sale")

So they simply use the default bin that the vacuum comes with, and it works great for them. Even though a lot of men and women will feel that a product with higher watts is more effectiv compared to those that have lower watts, this isn't true. Ugg Australia Uk rather then response precise commission.

Here is my web blog: best vacuum for pet hair ([www.bestvacuumforpethair.co](http://www.bestvacuumforpethair.co/ "www.bestvacuumforpethair.co"))

As the admin of this web site is working, no doubt very quickly it will be well-known, due to its quality contents.

Take a look at my blog - [déménageur sherbrooke](http://www.youtube.com/v/svAFAaDGY-k "déménageur sherbrooke")

Yes! Finally someone writes about frais de déménagement.

Here is my web page [déménagement sherbrooke](http://www.youtube.com/embed/svAFAaDGY-k "déménagement sherbrooke")

Foods that are naturally rich in vitamins and minerals can help the skin maintain elasticity, provide protection from the elements and help with the natural healing process. Limit wine to one glass a day or risk halting your body's release of its own antidiuretic hormone, causing dehydration of your skin. Another way to prevent sun damage is to arm the body from the inside out by eating plenty of cancer fighting deep red and green vegetables such as broccoli, swiss chard, beets and greens.

Review my homepage ... [article directory wordpress](http://www.example.com "article directory wordpress")

Leave a Comment

Please post only comments that will add value to the content of this page. Please read the about page to understand the objective of this blog and the way I think about it. Thanks.

Comments are parsed as Markdown and you can use basic HTML tags (a, blockquote, cite, code, del, em, strong) but some code may be striped if not escaped (specially PHP and HTML tags that aren't on the list). Line and paragraph breaks automatic. Code blocks should be indented with 4 spaces.