aeroheim-logoblog
projects
about

back to blog

Behind Aeroheim.moe: Backend

AUG 10, 2017second part of a technical overview
    metadev

Behind Aeroheim.moe: Backend


In the previous post I covered the frontend behind aeroheim.moe. For this post, I’ll provide a technical overview of the architecture and design considerations behind the backend of aeroheim.moe.

The main requirements for the backend:

  • RESTful API for content
  • DB with a flexible schema
  • support for server-side rendering (when I actually get off my lazy ass and learn how this works aaaaaaaaa)

The goal of the backend is to allow for arbitrary content (e.g markdown, media, etc.) to be served by an easy to consume RESTful API. In addition, the database that stores data relevant to this content should be flexible enough to support large changes in the future. Server-side rendering would also be nice to allow for users with slow connections to instantly view the site while the app downloads.


Full Stack Javascript - Node.js

As a bit of background, I’ve had prior experience writing simple backends in Python (using Flask) for a few projects back in college. While I initially considered using Flask for my backend again this time due to my familiarity with it, I learned that Node.js, a Javascript backend framework, is apparently all the rage nowadays. After some more research, I decided to cave in and switch to Node.js (like the conformist loser that I am) for the following reasons:

  • intuitive asynchronous I/O handling with callbacks
  • full stack Javascript which also means much easier server-side rendering
  • npm - Node’s package manager that is fairly good

Node.js is also pretty fast. It uses Google’s V8 engine (also used in Chrome), a highly performant Javascript engine that JIT compiles JS into machine code, to execute your server-side Javascript. On top of that, its asynchronous I/O calls libuv (a cross-platform I/O library written in C) under the hood, so you know it’s gonna be fast.

You can execute Javascript code in files using Node like this:

> node your-app.js

Here’s an example of the main file that Node executes on the server for aeroheim.moe:

// server.js
const frontend = require('./frontend');
const backend = require('./backend');

const PORT = process.env.PORT || 8080;
const PROD = process.env.NODE_ENV === "production";

if (PROD)
{
    // Run only the backend on production. It will serve the frontend on request.
    backend(PORT);
}
else
{
    // Run the backend server and the frontend using webpack-dev-server. While the frontend server
    // is still configured to proxy all requests back to the backend, using webpack-dev-server allows 
    // for Hot Module Replacement to be utilized for the frontend.
    backend(PORT - 1);
    frontend(PORT);
}

Node has it’s own module implementation independent of native Javascript (ES6) modules. Node modules can be imported for use using the require keyword, and every Javascript file can define what it exports by setting the module.exports object (which Node defines for you) like below:

// frontend.js
const webpack = require('webpack');
const webpackConfig = require('../../webpack.dev');
const WebpackDevServer = require ('webpack-dev-server');

// Node defines the 'module' object for every Javascript file.
// this function that takes in a port# and starts the frontend will be imported by others.
module.exports = (PORT) => 
{
    const frontend = new WebpackDevServer(webpack(webpackConfig), 
    {
        hot: true,
        historyApiFallback: true,
        proxy: 
        {
            '*' : `http://localhost:${PORT - 1}`
        },
    });

    frontend.listen(PORT, 'localhost');
};

While not shown in any of the examples above, Node’s main features are all provided by its modules (like fs or path) that you can import when needed.

NPM (Node Package Manager) is also a great part of Node that makes package management very easy. Packages can be grabbed using the npm install command:

npm install <pkg> --save
common options: [-P|--save-prod|-D|--save-dev|-O|--save-optional] [-E|--save-exact] [-B|--save-bundle] [--no-save] [--dry-run]

When you pass in a --save flag, the installed package gets marked as a dependency of your node app’s package.json. Every node app utilizes a package.json file that manages the dependencies, scripts, and metadata for it. Here’s a snippet of the the package.json for aeroheim.moe as an example:

{
  "name": "aeroheim.moe",
  "version": "1.0.0",
  "description": "hi",
  "main": "app.js",
  "homepage": "https://github.com/aeroheim/aeroheim.moe#readme",
  "dependencies": {
    "axios": "^0.16.2",
    "babel-core": "^6.25.0",
    "babel-loader": "^7.1.1",
    // etc...
  }
}

Anyways!!~ this section is starting to turn into a full-blown tutorial for Node.js, so I’ll stop now. But it’s always great to learn more about the new hotness every now and then! (´・ω・`)

Modular REST API with Express

Node’s basic modules provides everything you need to setup a web server and receive/send HTTP requests/responses. As usual though after some research, I found a library called Express that builds improves this functionality. Express is an excellent lightweight and minimalist server framework built on top of Node.js that simplifies the nuances of web servers with features such as automatic streaming for files, preset http headers, and more.

Using Express, I can setup the server for aeroheim.moe like this:

// backend.js
const express = require('express');

// initialize express server
const app = express();

// some additional routes...
// define routes and their responses using express' get() method
app.get('/favicon.ico', (req, res) => res.sendFile(path.join(__dirname, '..', 'favicon.ico')));
app.get('/index.css', (req, res) => res.sendFile(path.join(__dirname, '..', 'index.css')));
app.get('*', (req, res) => res.sendFile(path.join(__dirname, '..', 'index.html')));

// start listening to requests!
app.listen(PORT);

Another great part about Express is its functional approach towards handling requests. Express has the concept of middleware, which are basically functions that you can chain together to handle a request and send a response. Each middleware has the ability to read the current request, modify the response to be sent, and also send the response immediately or pass it on to the next middleware in the chain.

Using this approach towards handling requests, I was able to modularize the different parts of aeroheim.moe’s RESTful API into separate groups of middleware. Here’s an example of how the routes for the blog API are defined:

// blog.js
const express = require('express');

// create a router object which you can apply middleware functions to
const router = express.Router();

// apply the following middleware functions to the router
router.get('/api/blog', (req, res) =>
{
    // handle request, send response...
});
router.get('/api/blog/:id', (req, res) =>
{
    // handle request, send response...
});

// now export the router for use!
module.exports = 
{
    router: router,
}

In the file where my express server is defined, I can then import this router with its applied middleware to handle the blog API for my server:

// backend.js
// initialize express server
const express = require('express');
const blog = require('./api/blog');

const app = express();

// add the blog api to the server
app.use('/', blog.router);

// define routes using express' get() method and their responses
app.get('/favicon.ico', (req, res) => res.sendFile(path.join(__dirname, '..', 'favicon.ico')));
app.get('/index.css', (req, res) => res.sendFile(path.join(__dirname, '..', 'index.css')));
app.get('*', (req, res) => res.sendFile(path.join(__dirname, '..', 'index.html')));

// start listening to requests, this time with a blog api!
app.listen(PORT);

Express’ support for modular routers allows me to write the different parts of my API independently, and then piece them together when necessary. This is super neat as routes can be easily decoupled from the server and tested independently. Hurray for less spaghetti code and better testability!

Lightweight & Flexible Static Content Using NoSQL

Even though I’m only serving static content with this site, I wanted to have a database so that I could store metadata about my content and do queries on them as well. In addition, the database that I would be using should be have a flexible schema to support arbitrary types of content that I choose to add in the future.

In the past I’ve worked with relational databases using PostgreSQL to store data for simple backends. I would design a schema with the appropriate tables, add data by hand manually (i’m not a DBA or anything so pls don’t hurt me), use an ORM to map database objects to objects in the backend, and then persist the data.

This a rather tedious workflow just to manage simple metadata for static content however, so I did some research to see if there were any alternatives. I learned about NoSQL databases and how they differed from standard relational databases, and decided that going NoSQL would be a better fit for my needs because of the following properties:

  • schema-less approach
  • good support for json blobs

While NoSQL has basically become a meme for how fast and “web scale” it is (see MongoDB), neither of those properties are realistic concerns for my app. Instead, I was attracted by how a schema-less approach would allow me to easily add new content types and change existing ones, and also by the support for json blobs which would allow me to skip having to write ORM code and instead just work with JSON directly.

For my Node backend, I ended up using NeDB, a Javascript NoSQL DB that provides a subset of MongoDB’s API. Using MongoDB’s API is somewhat nice since it isn’t too hard to use and it gives me the option to easily transition to MongoDB in the future when necessary (basically never).

Content metadata for aeroheim.moe is stored inside version-controlled .json files, which act as the persistence layer for my db:

// blog.json
[
    {
        "_id": "checkpoint",
        "title": "Checkpoint",
        "description": "the first checkpoint of this blog",
        "date": "2017-6-16",
        "tags": ["meta"]
    },
    // more content...
]

During the startup process of my backend, an in-memory NeDB instance is constructed that reads these .json files and stores their data:

// backend.js
const Datastore = require('nedb');
const blog = require('./api/blog');

// initialize db
const db = {};
db.blog = new Datastore();
initializeStoreFromDisk(db.blog, blog.storePath);

function initializeStoreFromDisk(store, path)
{
    // parse the JSON from the file and dump it inside the db!
    store.insert(JSON.parse(fs.readFileSync(path, 'utf8')), (err) => logError(err, `Failed to populate db store: ${err}`));
}

Here’s an example of how I use NeDB to perform queries for my blog API:

// blog.js

router.get('/api/blog', (req, res) =>
{
    // query all blog posts
    req.app.locals.db.blog.find({}, (err, posts) =>
    {
        // send metadata for all blog posts
    });
});
router.get('/api/blog/:id', (req, res) =>
{
    // query the blog post whose '_id' matches :id
    req.app.locals.db.blog.findOne({'_id': req.params.id}, (err, post) =>
    {
        // locate the blog post on disk and send it to the client
    });
});

As you can see, the whole process is very simple and lightweight. To add new content to aeroheim.moe, I simply check in both the content and my updates to the appropriate .json file into my source control. Since this site won’t ever be storing a massive amount of content, I can afford to host my entire database in-memory without having to worry about how to persist its state.


Conclusion

This concludes the technical overview for the backend behind aeroheim.moe.

Just like I mentioned in the previous technical overview post, a lot of the tech mentioned here may very well be outdated in just a couple of years. This post is primarly here to serve as a reference to how this site was built at this point in time.

Wew!~ That was a wild ride. Thanks for reading and have a nice day! ‘w’