A blog about design, code and some other stuff

There are a lot of build tools that covers specific use cases and/or try to cover as many scenarios as possible, amongst the most famous ones are make, rake, Ant and maven. I’m going to talk about why I’ve been favoring plain Node.js scripts as my “build tool” and how to do some simple things. This work flow may not be the best one for you and your team, understand the reasoning behind it and pick the tools based on your needs and preferences.

Why?

Most of my projects I just need some basic features like file concatenation, running an optimizer/compressor, some simple string replacements or running some other 3rd party command-line tool, sometimes I also need to do some things that aren’t that common (code generation, documentation, etc..) and also depending on the project my structure may not fit existing tools.

I’ve been using Ant for the past few years on almost all my projects, the good thing about Ant is that it’s very mature, can run on multiple platforms, well documented, pre-installed on Mac OS X, popular (specially for Java and AS3 developers), and almost anything you want to do you just need to search for the “term” plus “ant task” (eg. “concat files ant task”, “delete files ant task”) and the first results probably describe how to do what you need. The problem with Ant is that XML is a markup language and when you try to do things that requires a programming language (logic, loops, etc..) things get nasty really quickly so you need to extend Ant by using the script task or creating an external command line tool/app or an Ant plugin (Java), not really convenient or straightforward (too much overhead, need to learn new APIs and/or languages…)

Make and bash scripts are way more concise than Ant, but they also impose a barrier to people that doesn’t know how to edit them. Another problem with bash is that it isn’t very portable which for me is a big issue, I work with multiple devs and who knows which OS they might be using or how the build process will be triggered? It’s very important for users to be able build the project without too much trouble and also that multiple developers on the team understands how the build process work and how to edit the files, it will avoid “knowledge silos” and “code ownership”. If your team is proficient in JavaScript I suggest that the build script should be written in JavaScript, if the team is more comfortable with Python or Ruby (or anything else) than write it on these languages instead…

A few months ago I finally decided to try “vanilla” Node.js scripts and I’m very pleased with it, I see it as an excuse to learn Node.js API and it’s libraries. It’s also very fast to code since I’m very familiar with JavaScript syntax, I don’t need to spend time searching how to do things I simply code it, also since I’m not constrained by an existing tool structure I can go “nuts” and code whatever I want and the way I want it, the complexity of my scripts are only limited by my needs and deadlines.

I tend to think that tools that try to fit all scenarios end up not fitting anything properly, so I prefer code that is easy to edit/tweak than code that can be overly customized. Having the build script written on the same language as the project is a very good thing since contributors will feel more comfortable at updating it and improving it, low entry barrier is usually a good motivation for contributing.

How?

I’m not really using any specific build tool, just putting together a few npm modules as needed, kinda following the Node.js philosophy of keeping packages small and loosely coupled, some of the modules I’ve used so far and that helped a lot in the process:

• commander
  • very useful tool to generate self-documenting command-line interfaces,
    using it on most of my Node.js command-line tools.
• wrench-js
  • recursive file operations (delete, create, copy directories)
• node-glob
  • returns a list of filepaths based on a string pattern.
• handlebars
  • for templating in case you need to do some code generation and/or process
    some text files.
• UglifyJS
  • JavaScript minification.
• async
  • Asynchronous utilities and control flow.

I’m also calling external tools when needed using the child_process.exec command, so you can still call some JAR files or other command-line tools if needed…

I’m not using any build tool because I want flexibility and also because it’s very easy to put those things together, there are a plenty of Node.js modules available through npm that does all sorts of things and it’s a very good excuse to learn how to use them and to discover new libraries.

Examples

Most of the examples are very trivial, use it as a reference not as the only way of doing it.

Concat files

You just need to read the content of a file list and merge them:

// settings
var FILE_ENCODING = 'utf-8',
    EOL = '\n',
    DIST_FILE_PATH = 'dist/myAwesomeScript.js';

// setup
var _fs = require('fs');

function concat(fileList, distPath) {
    var out = fileList.map(function(filePath){
            return _fs.readFileSync(filePath, FILE_ENCODING);
        });
    _fs.writeFileSync(distPath, out.join(EOL), FILE_ENCODING);
    console.log(' '+ distPath +' built.');
}

concat([
    'foo/bar.js',
    'foo/lorem.js',
    'foo/maecennas.js'
], DIST_FILE_PATH);

It’s important to notice that I’m not concatenating files that much since I’m using r.js (RequireJS optimizer) on almost all my projects, r.js can be also used to merge CSS files (inline @imported stylesheets).

Minify JS file with UglifyJS

// settings
var FILE_ENCODING = 'utf-8';

function uglify(srcPath, distPath) {
    var
      uglyfyJS = require('uglify-js'),
      jsp = uglyfyJS.parser,
      pro = uglyfyJS.uglify,
      ast = jsp.parse( _fs.readFileSync(srcPath, FILE_ENCODING) );

    ast = pro.ast_mangle(ast);
    ast = pro.ast_squeeze(ast);

    _fs.writeFileSync(distPath, pro.gen_code(ast), FILE_ENCODING);
    console.log(' '+ distPath +' built.');
}

uglify('dist/awsum.js', 'dist/awsum.min.js');

Minify JS with google closure compiler (run external JAR)

You can run shell commands with child_process.exec. Very useful for reusing existing command-line tools and code that isn’t Node.js specific.

// --- SETTINGS ---
var COMPILER_JAR = 'build/compiler.jar';

// --- SETUP ---
var _exec = require('child_process').exec;

function compile(srcPath, distPath) {
    // exec is asynchronous
    _exec(
      'java -jar '+ COMPILER_JAR +' --js '+ srcPath +' --js_output_file '+ distPath,
      function (error, stdout, stderr) {
        if (error) {
          console.log(stderr);
        } else {
            console.log(' '+ distPath + ' built.');
        }
      }
    );
}

compile('dist/myAwesomeScript.js', 'dist/myAwesomeScript.min.js');

Use commander to parse CLI arguments

Commander.js is very useful for parsing command-line arguments, specially since it is self-documenting and extremely easy to use.

// --- SETTINGS ---
var DIST_PATH = 'dist/awsum.js';

// --- SETUP ---
var _cli = require('commander'),
    _fs = require('fs');

// global options
_cli
    .version('0.0.1')
    .option('--silent', 'suppress log messages.');

// commands
_cli
    .command('deploy')
    .description('Optimize site for deploy.')
    .action(deploy);

_cli
    .command('purge')
    .description('Delete old files from dist folder.')
    .action(purgeDeploy);

// parse commands
_cli.parse(process.argv);

function deploy() {
    purgeDeploy();
    build();
}

function purgeDeploy(){
    _fs.unlinkSync(DIST_PATH);
    if (! _cli.silent) {
        console.log(' Deleted deploy files!');
    }
}

function build(){
    // concat files here or do anything that generates the dist files
    if (! _cli.silent) {
        console.log(' Built!');
    }
}

node build -h will show the available commands and options. Running node build deploy will execute the “deploy” command, node build deploy --silent will execute the “deploy” command but suppress log messages.

Code generation / String replacements

Template engines aren’t useful only for HTML documents, they can be used for simple string replacements and also for code generation. I’ve been favoring Handlebars.js because of the “helpers” and nested paths support. I’ve been using it to add some basic info like version number, build date and even to generate some source files.

On AMD-Utils I’m using it to generate the package files and also to update the test runners (so I make sure I have unit tests to all modules).

If you are distributing the library through npm and/or have a packages.json file it is a good idea to reuse the data on the build as well so it doesn’t get out of sync. Let’s say we have a license at the top of the file like:

/**@license
 * My Awesome Lib <http://example.com>
 * Version: 0.1.0 (Tue, 03 Jan 2012 03:48:58 GMT)
 * License: MIT
 */

It could be described as:

/**@license
 * {{name}} <{{homepage}}>
 * Version: {{version}} ({{build_date}})
 * License: {{#license licenses}}
 */

And we could replace the data using handlebars:

var FILE_ENCODING = 'utf-8',
    DIST_PATH = 'dist/awsum.js';

var _handlebars = require('Handlebars'),
    _fs = require('fs');

// will generate a CSV if package.json contains multiple licenses
_handlebars.registerHelper('license', function(items){
    items = items.map(function(val){
        return val.type;
    });
    return items.join(', ');
});

var distContent = _fs.readFileSync(DIST_PATH, FILE_ENCODING);
var template = _handlebars.compile(distContent);

//reuse package.json data and add build date
var data = JSON.parse( _fs.readFileSync('package.json', FILE_ENCODING) );
data.build_date = (new Date()).toUTCString();

_fs.writeFileSync(DIST_PATH, template(data), FILE_ENCODING);

Lint JS files and prompt user to confirm before continuing added 2012/01/03

Joss Crowcroft asked for it in the comments and since it isn’t hard to implement (thanks to Commander.js .choose() method) I decided to do it.

var _cli = require('commander'),
    _jshint = require('jshint'),
    _fs = require('fs');

function lint(path, callback) {
    var buf = _fs.readFileSync(path, 'utf-8');
    // remove Byte Order Mark
    buf = buf.replace(/^\uFEFF/, '');

    _jshint.JSHINT(buf);

    var nErrors = _jshint.JSHINT.errors.length;

    if (nErrors) {
        console.log(' Found %j lint errors on %s, do you want to continue?', nErrors, path);
        _cli.choose(['no', 'yes'], function(i){
            if (i) {
                process.stdin.destroy();
                if(callback) callback();
            } else {
                process.exit(0);
            }
        });
    } else if (callback) {
        callback();
    }
}

// run
lint('dist/awsum.js', function(){
    console.log(' Built!');
});

More

There are a few Node.js general purpose build tools like node-jake and cake (coffeescript), and also some tools that assumes your project is following a specific structure and contains options for lint/concatenation/minification like smoosh, buildr.npm and grunt. I tend to prefer tools that doesn’t impose a specific structure since we never know if the constraints might become an issue or when the project requirements may change, but these tools may be a good option depending on the project.

If you need to do something that isn’t listed here search npm, there is a big chance that someone already coded a package that does what you need, if not, do it yourself and share with the open source community.

That’s it for now.

Edit 2012/01/03: Added JSHint example with confirm prompt and link to caolan/async.