Managing Client/Server Web Apps with Lerna

This post describes the approach I use for managing client/server web apps using Lerna. The code snippets are simplified versions of the code used to power this website whose source is available on Github

If you are building a Javascript based client/server or fullstack web app you may have considered how to manage the two parts of the application in the same repository. It’s more convenient to keep the client and server together but, traditionally, this means one package.json, particularly if using the same package manager for client and server. And this is where dependencies, scripts, config etc for the two applications can get muddled up. There have been NPM and Yarn issues discussing grouping dependencies into more specific sets that have never materialized. But since I’ve started using Lerna to organize my Javascript projects I’ve realized it provides a better solution.

Lerna was born out of Babel.JS, where it was used to manage a large number of plugins within the same monorepo. Lerna’s website’s introduction explains its purpose nicely:

Splitting up large codebases into separate independently versioned packages is extremely useful for code sharing. However, making changes across many repositories is messy and difficult to track, and testing across repositories gets complicated really fast.

But Lerna isn’t necessarily just for large monorepos with a large number of packages. Whether you’re managing 24 or just 2 packages, Lerna can become an indispensable part of your Javascript toolbox. Pertinently, it can be useful for simpler scenarios, such as cleanly bisecting a client and server in the same repository. Lerna also ships with utilities for versioning and publishing, which aren’t always needed in such leaf projects at the end of the dependency tree.

Let’s now look at how Lerna can solve the above problem.

Given that you have installed and initialized Lerna in your project, we can begin by looking at the basic directory structure. By convention, Lerna packages are kept within a root packages directory, like so:

myapp
├── packages/myapp-client/
├── packages/myapp-server/
├── lerna.json
└── package.json

For simplicity, you could also forgo the packages directory and locate the myapp-client and myapp-server directories directly in the root of the repo (with a modification to lerna.js). However, this won’t scale well if more packages are added in the future.

The Client

Think of the client as a typical modern frontend. One that has its own build process with Wepback (or Browserify) and manages its dependencies with NPM or Yarn. Essentially one that is sophisticated enough to warrant its own package.

Detailing the client application’s configuration in any depth is out of scope here, but a few things are relevant, beginning with the basic directory structure of the client:

myapp-client
├── node_modules/
├── config/
│   └── webpack.conf
├── dist/
│   └── main.js
├── src/
│   └── index.js
└── package.json

The package.json in this package is now only concerned with the client application; it’s dependencies, scripts, Babel config, etc. This app is built using Webpack and the important thing to note about the Webpack config is that its output directory is the dist directory (ignored from source control). Via Lerna’s linking model, the Webpack build will be picked up and served by the server.

The server

An Express Node.JS server runs in the myapp-server package:

myapp-server
├── node_modules/
├── server.js
└── package.json

As with the client, the server’s package.json is also now only concerned with the server application, however in order to serve the client, it is a dependency:

"dependencies": {
    "express": "^4.16.2",
    "myapp-client": "1.0.0",
    "otherstuff" : "1.2.3"
}

Once this dependency is added to the package.json. Lerna’s bootstrap or link commands can be used in the root of the repository to wire the packages together using symlinks.

Next, in order to serve the static files generated by Webpack, we can use Express to mount the client’s dist directory on the /static/ path. Since Lerna has linked it, it can be referenced at node_modules/myapp-client/dist

const { join } = require('path');
const express = require('express');
const staticDir = path.join(__dirname, 'node_modules/myapp-client/dist');

const app = express();
app.set('view engine', 'html');
app.use('/static', express.static(staticDir));
app.use((req, res, next) => {
    res.render('index');
});
app.listen(8080);

and the view file can now now reference your static assets generated by the Webpack build on the /static/ path:

<body>
    ...
    <script src="/static/main.js"></script>
</body>

UPDATE: I should not that this setup is useful for development environments. In a production deployment though, ideally you would have a reverse proxy server (such as Nginx) serving the statics before the requests reach Node.

To take this a step further, when rendering the Express view, you may use a hash of the Javascript file to fingerprint its path for aggressive browser caching:

const { join } = require('path');
const express = require('express');
const lodash = require('lodash-express');
const md5File = require('md5-file');

const staticDir = path.join(__dirname, 'node_modules/myapp-client/dist');

const app = express();

lodash(app, 'html');
app.set('view engine', 'html');

// since this happens on app startup (not every request), its ok/simpler to use sync
const fingerprints = {
    js: md5File.sync(join(staticDir, 'main.js')),
};
app.use('/static', express.static(staticDir));
app.use((req, res, next) => {
    res.render('index', {
        bodyInject: `<script src="/static/main.js?${fingerprints.js}"></script>`,
    });
});
app.listen(8080);

This example uses simple Lodash templating to inject the script tag into the index.html view:

<body>
    ...
    <%= bodyInject %>
</body>

More things

Dev server

So far, this gives us essentially the production setup. For development though you’ll probably want things like hot-reloading to work. For this, Webpack Dev Server can be configured with a few tricks whereby requests are proxied to the server except for the assets generated by Webpack. Alongside the HotModuleReplacementPlugin this all seems to work nicely. Although, I won’t lie that the first time I set this all up, it took a bit of tweaking and Googling to get right.

The devServer block of my Webpack config looks like this:

devServer: {
    publicPath: "http://localhost:3000/static/",
    port: 3000,
    hot: true,
    proxy: {
        '/': {
            target: 'http://localhost:8080',
            bypass: function (req) {
                if (req.url.includes('/static/') || req.url.includes('hot-update')) {
                    return req.url;
                }
            }
        }
    },
}

Server side rendering

Rendering on the server is also an option. How this is achieved will vary on your approach to server-side rendering. This is my approach. Also, I’ve only implemented this approach with project using Preact so far, but the gist should be relevant to other libraries where universal rendering is possible.

Personally I like to keep my server-side tool-chain simple and, for development, I don’t do any transplilation. However, since I use ES Modules and JSX on the client and (currently) Common.JS on the server, its necessary to transpile the client code before it can run on the server. Since the Babel config is used by Wepback, and therefore already exists in the client’s package.json, Babel can just be used on the command line to transpile the client side code:

> cd myapp-client
> ../node_modules/.bin/babel src --out-dir lib

You can also add this to a script in myapp-client/package.json to make it a regular part of the build.

The transpiled code is output to the myapp-client/lib and it can now be required in server.js from the linked myapp-client package. Here’s a rough idea of the fundamentals of the server-side rendering code using Preact:

const { join } = require('path');
const express = require('express');
const lodash = require('lodash-express');
const md5File = require('md5-file');

// ssr dependencies
const { h } = require('preact');
const preactRenderToString = require('preact-render-to-string');

// its important to note that when requiring the client view, I have
// separated the code that renders it to the DOM on the client (`index.js`)
// from the root Preact component, which is defined in `view.js`
const clientView = require('versatile-client/lib/view').default;

const staticDir = path.join(__dirname, 'node_modules/myapp-client/dist');

const app = express();

lodash(app, 'html');
app.set('view engine', 'html');

// since this happens on app startup, its ok/simpler to use sync
const fingerprints = {
    js: md5File.sync(join(staticDir, 'main.js')),
};
app.use('/static', express.static(staticDir));
app.use((req, res, next) => {
    const { url } = req;
    const initialState = {};
    const app = preactRenderToString(h(clientView, { url, initialState }));
    res.render('index', {
        bodyInject: `${app}\n<script src="/static/main.js?${fingerprints.js}"></script>`,
    });
});
app.listen(8080);

More packages

Of course, this scales to more than one web app and multiple server packages may share code, which in turn, lives in its own Lerna package(s).

This post very much scratches the surface of what Lerna is capable of. If you are not familiar with Lerna, I recommend you have a read through the REAMDE; it does a lot and can be very useful even if you don’t require its full feature set.

Please leave any comments/suggestions below.