2013.03.05

Mout & Modularity

Discussions about modularity are recurrent. Some people say that each function should be a separate module/file/package; others say that methods should be contained by a package and grouped by similarity/concerns; and there is still a 3rd group that thinks that a single namespace is the way to go. I will try to explain the design decisions that influenced the creation and current structure of moutjs and why single function packages are not always the best solution.

How it started

It all started back in 2009, when I decided to extract and port some common methods that I used in different projects into reusable packages. I started by creating a stringUtils and arrayUtils namespaces and adding new methods as needed. The helpers grew at each project and after a few months I had more than 20 functions inside the stringUtils namespace (some of them pretty large since they handled HTML entities conversion), which means I had to load the 20+ functions even if I only needed a single method - keep in mind that I do mostly front-end development and at the time I was coding mobile sites very often, file size was a big concern.

// used a long (global) namespace to avoid name collisions
// loaded all methods even if not used by the app
MM.arrayUtils.forEach(arr, doSomething);
var slug = MM.stringUtils.hyphenate(myStr);

Going modular

At the end of 2010 I was working on bigger JavaScript projects (+10K LOC) and started to use RequireJS to manage my dependencies, and by doing so I was motivated to split my code into smaller files, that way I could keep the code organized and load only what was actually being used by the app. That enhanced my JavaScript development experience a lot, I was able to write apps with more than 50K lines of code with ease, code reuse increased exponentially at each project, no more conflicts when working with other devs and specially no more reliance on globals - I do a lot of projects that need to run inside bigger sites/apps, relying on globals is a path that usually leads to pain.

// require a single method (recommended)
var forEach = require('mout/array/forEach');
forEach(arr, doSomething);
// or the whole package if you want to (I rarely do this)
var stringUtils = require('mout/string');
var slug = stringUtils.hyphenate(myStr);

Why a single package?

The state of front-end Package Managers

One of the main reasons to keep mout as single package is that package management for the front-end is still in flux. Some people been using npm while others use bower/volo/jamjs, so it’s hard to get into a common ground. Adopting one package manager over another might cause some potential users to not use the lib, since that might not be their preferred approach or just not how they are handling dependencies. - node.js people, mind yourself that not everyone is writing node.js apps, so npm and cjs modules are not the silver bullets that you guys like to pretend they are.

Distributing a single package is easier for consumption in the browser, since you can just download a zip/tarball into the right folder and call it a day. In fact I’ve been using some very basic shell scripts to download/update my dependencies over the past couple years and it’s working pretty well for packages that have a shallow dependency tree.

Git submodules are a pain to maintain

The package manager isn’t the only reason, there is also the fact that maintaining multiple git repositories can be a huge PITA. Everyone knows that git submodules doesn’t work that well and there are some things that are just simpler when you deal with a single repository (mout contains ~200 methods). Things like scaffolding, automation and running tests is easier since you have less moving parts - single build script, single test runner, single CI server, etc.

The mout build script is responsible for keeping all packages up-to-date before running the tests, if a module doesn’t contain tests it creates a failing spec automatically to warn you. It also contains some scaffolding commands to create new modules based on templates and a command to help deploy the project. That is the kind of automation that is easier to implement if the project follows a sane structure.

PS: I know npm doesn’t require all packages to be separate git repositories (you don’t even need a public repository), but if they aren’t separate repositories then it probably shouldn’t be a separate package as well.

Consistency

The main reason why code reuse is such a hard thing to achieve is because most libraries/frameworks aren’t designed to work together and makes a lot of assumptions on how they will/should be used. Since everything is coded together there is a reduced chance of naming conflicts and you also increase the consistency/cohesion between all functions (including naming conventions). It motivates code reuse between internal modules as well.

Some bugs and enhancements affects multiple modules at once, having a decentralized issue tracker can lead to many duplicates and also makes it harder for users/maintainers to follow the progress. As an example we decided to borrow a feature from Lo-Dash on mout v0.4, the ability to use a shorthand notation on most array/object/collection methods, so you can do find(users, {name:'john'}) instead of find(users, function(u){return u.name === 'john'}). This change affected 18 modules and required 2 new methods (1 private). This is the kind of change that should be done across the board, all at once. Releasing 20 packages !== fun. If each package was maintained by different users it would surely not work since we could never get into an agreement if this feature should really be implemented or even coordinate a release date…

If each method accepts a different set of arguments (or different order) and/or are named following different conventions you will probably make more mistakes. Common conventions means less time spent reading the docs or fixing silly bugs.

Less bureaucracy

Since everything is part of the same package, it’s easier to handle the dependency on the “project context” and still require individual methods as needed. If every single function was a separate package you would need to list it twice, once on the package.json and another every time you require the module.

“Discoverability”

Since everything is part of the same package you increase the chance of the user to discover that feature. There are many functions that aren’t popular and the user would never find out they existed, or how useful they are. Sometimes you don’t know you need it until you see it. How can you find something if you don’t even know what you are looking for?

It means less time spent hunting for packages. You can trust that all functions have somewhat the same quality level and are meant to work together.

Project goals

I created mout with the intention of using it as a replacement to the JavaScript Standard Library, where you don’t need to worry about browser quirks and can use all methods in all environments (IE 7+) with a good amount of confidence.

Since it’s meant to be a “standard lib” you wouldn’t expect it to be scattered across multiple places.

Cross dependencies

There are a lot of cross dependencies between modules. Handling it between multiple separate projects would be a tricky thing, specially for browsers.


Dependency graph generated with node-madge (excluded packages to make it easier to identify real dependencies).

“Identity”

Multiple packages doesn’t hold any “identity” and probably won’t create a strong sense of “community”.

Why AMD as the authoring format?

The authoring module format is just a detail. It can be easily translated into any other module format if needed.

Since I do mostly front-end development it makes more sense to write the modules in a format that works in the browser without needing a build. It also makes the whole deployment process easier. Since most node.js modules are installed through npm, we can convert it into a node.js compatible format during npm publish and make sure that the users will get the files they need without any extra overhead (no performance or readability drawbacks). Since the browser package managers usually use the git repository as the source (don’t have something similar to npm publish) it’s better to leave the files on github always ready for browser consumption.

There are no silver bullets

mout have very specific goals and needs, that’s why it’s coded in this specific way. The reasoning described above surely doesn’t apply to all projects, in fact I believe it applies to the minority.

I truly believe that single function modules will become more popular in the following years, but it surely isn’t how I write most of my application code. Not all code need to be reused in different contexts, and modularity is not always an advantage (sometimes it can be an extra layer of bureaucracy). Consider the options and pick the one that fits better your needs.

My rule of thumb nowadays is that if something doesn’t help me it will probably slow me down. I won’t follow some specific convention/pattern/dogma just because other people say it’s the right thing unless I think it does apply to my own scenario. Working on a fast paced environment with many changes during the development process made me become more pragmatic about how I approach problems, same rule might not apply to yourself.

“It just works” is fantastic, when it really does just work. But it’s brutal when it doesn’t actually work. Then I prefer transparency. – @JohnDCook

Give it a try

mout contains many useful features, consistent API, good documentation and is being actively maintained. We also have an extensive set of unit tests to ensure correctness. Expect even more features in the coming months and a fast release cycle. We’ve been working hard to make it as awesome/reliable as possible. Check it out at http://moutjs.com and star/fork the project on github. We are also on the #moutjs channel on irc.freenode.net.

Further reading


Comments

Leave a Comment

Please post only comments that will add value to the content of this page. Please read the about page to understand the objective of this blog and the way I think about it. Thanks.

Comments are parsed as Markdown and you can use basic HTML tags (a, blockquote, cite, code, del, em, strong) but some code may be striped if not escaped (specially PHP and HTML tags that aren't on the list). Line and paragraph breaks automatic. Code blocks should be indented with 4 spaces.