December 18, 2011

How to deploy a nodejs application with monit, nginx and bouncy

Disclaimer: I'm not a sysadmin, the information on this blog post may be wrong, or not.

Deploying nodejs

A deployment usually is:

  • A mechanism to upload new versions of your app
  • Setting up a technology stack
  • Monitoring status of the machines and processes

To simplify this guide I will assume that:

  • You are setting up a Debian machine
  • Your app is at /var/www/your_app and that it has /pids and /logs folders.
  • You have a deploy user (uid:1000, gid:1000)
  • You already installed node and npm
  • You use git as a version control system

A mechanism to upload new versions of your app

First we have to clone our repository in our server:

cd /var/www/
git clone your_repository_url.git your_app

Then we will create a script in our server /etc/init.d/your_app that will allow us to stop/start/restart our app:

#!/bin/bash
DIR=/var/www/your_app
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
NODE_PATH=/usr/local/lib/node_modules
NODE=/usr/local/bin/node

test -x $NODE || exit 0

function start_app {
  NODE_ENV=production nohup "$NODE" "$DIR/app.js" 1>>"$DIR/logs/your_app.log" 2>&1 &
  echo $! > "$DIR/pids/your_app.pid"
}

function stop_app {
  kill `cat $DIR/pids/your_app.pid`
}

case $1 in
   start)
      start_app ;;
    stop)
      stop_app ;;
    restart)
      stop_app
      start_app
      ;;
    *)
      echo "usage: your_app {start|stop}" ;;
esac
exit 0

The last part is to write a simple bash script for our development enviroment that will allows us to connect to our machine with ssh and pull changes. This is what I personally use:

#!/bin/bash

SERVER=( 'your_hostname.your_domain.com' )
DEPLOY_PATH=/var/www/your_app
REPO=user@your_repository_url.git
USER=deploy

ENVIRONMENT=${1:-"production"}
REF=${2:-"master"}

trap 'test -n "$SUCCESS" || echo "  error: aborted"' EXIT
echo "* Deploying $ENVIRONMENT/$REF"

ssh $USER@$SERVER "cd $DEPLOY_PATH && \
                   git reset --hard && \
                   git checkout $REF && \
                   git pull && \
                   npm install && \
                   /etc/init.d/your_app stop"

SUCCESS=true

To deploy we just need to give execution permissions to that file (chmod +x deploy) and we are ready to go!

git commit -am 'some changes'
git push
./deploy

Setting up a technology stack

My favorite technology stack so far is:

  • bouncy in front of everything. It listens the port 80 and redirects http requests to either nginx or node. It supports websockets!
  • nginx serves the assets listening the port 3000. Nginx is super fast serving static files, its stable and easy to configure.
  • nodejs will run our app listening the port 3001.

Install and set up bouncy

Since its a nodejs module its as easy as:

npm install -g bouncy

Then we just provide a routes.json in our application folder /var/www/your_app/routes.json:

{
  "assets.your_domain.com" : 3000,
  "" : 3001
}

Then we need a script on /etc/init.d/bouncy to start/stop our bouncy process.

#!/bin/bash

DIR=/var/www/your_app
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
NODE_PATH=/usr/local/lib/node_modules
BOUNCY=/usr/local/bin/bouncy

test -x $BOUNCY || exit 0

case $1 in
   start)
      nohup "$BOUNCY" "$DIR/routes.json" 80 1>>"$DIR/logs/bouncy.log" 2>&1 &
      echo $! > "$DIR/pids/bouncy.pid";
      ;;
    stop)
      kill `cat $DIR/pids/bouncy.pid` ;;
    *)
      echo "usage: /etc/init.d/bouncy {start|stop}" ;;
esac
exit 0

Install and set up nginx

To install nginx I usually follow the excellent linode's guide.

apt-get update
apt-get upgrade --show-upgraded
apt-get install libpcre3-dev build-essential libssl-dev

cd /opt/
wget http://nginx.org/download/nginx-1.0.4.tar.gz
tar -zxvf nginx-1.0.4.tar.gz
cd /opt/nginx-1.0.4/

./configure --prefix=/opt/nginx --user=nginx --group=nginx --with-http_ssl_module --with-ipv6
make
make install

adduser --system --no-create-home --disabled-login --disabled-password --group nginx

Then we will configure nginx to:

  • Listen the port 3000
  • Serve assets from /var/www/your_app/public folder
  • Compress some our assets with gzip
  • Serve favicons with 7 day expire headers
  • Serve the other assets with a 3 hours expire headers
  • Proxy the request to the node process if the asset is not found under the /public folder.

Edit /opt/nginx/conf/nginx.conf:

pid /opt/nginx/logs/nginx.pid;

user nginx nginx;
worker_processes  2;

error_log  /var/log/nginx/error.log notice;

events {
  worker_connections 1024;
}

http {
    include            mime.types;
    default_type       application/octet-stream;
    access_log         /var/log/nginx/access.log;
    sendfile           on;
    keepalive_timeout  65;
    server_tokens      off;

    gzip on;
    gzip_comp_level 2;
    gzip_proxied any;
    gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
    gzip_buffers 16 8k;
    gzip_vary on;

    upstream node {
      server 127.0.0.1:3001;
    }

    server {
        listen       3000;
        server_name  your_server_name subdomain.your_domain.com;
        root /var/www/your_app/public;

        location ~* ^.+\.ico$ {
          access_log        off;
          expires           7d;
        }

        location ~* ^.+\.(jpg|jpeg|gif|png|css|js|mp3)$ {
          access_log        off;
          expires           3h;
        }

        try_files $uri @node;

        location @node {
          proxy_set_header  X-Real-IP        $remote_addr;
          proxy_set_header  X-Forwarded-For  $proxy_add_x_forwarded_for;
          proxy_set_header  Host             $http_host;
          proxy_redirect    off;
          proxy_pass        http://node;
        }
    }
}

To create the script to start/stop the process we will use this:

wget -O init-deb.sh http://library.linode.com/assets/658-init-deb.sh
mv init-deb.sh /etc/init.d/nginx
chmod +x /etc/init.d/nginx
/usr/sbin/update-rc.d -f nginx defaults

Monitor status of the machines and processes

Let's install our favorite daemon-barking-dog.

(sudo) apt-get install monit

We want monit to:

  • Check our process every 30 seconds
  • Send us emails when something wrong happens
  • Provide us with a nice http interface. This will allow us to restart processes even if you are on a machine without ssh access

Edit /etc/monit/monitrc/:

set daemon  30

set logfile /var/log/monit.log

set mailserver localhost
set mail-format { from: monit@your_domain.com }

set alert user@your_domain.com
set httpd port 2812 and
     allow user:password

include /etc/monit/conf.d/*

Monit your server status /etc/monit/conf.d/system:

check system your_server_name
  if loadavg (1min) > 4 then alert
  if loadavg (5min) > 2 then alert
  if memory usage > 75% then alert
  if cpu usage (user) > 70% then alert
  if cpu usage (system) > 30% then alert
  if cpu usage (wait) > 20% then alert

Monit your filesystem status /etc/monit/conf.d/filesystem:

check filesystem datafs with path /dev/xvda
  start program  = "/bin/mount /data"
  stop program  = "/bin/umount /data"
  if failed permission 660 then unmonitor
  if failed uid root then unmonitor
  if failed gid disk then unmonitor
  if space usage > 80% for 5 times within 15 cycles then alert
  if space usage > 99% then stop
  if inode usage > 99% then stop
  group server

Monit sshd /etc/monit/conf.d/sshd:

check process sshd with pidfile /var/run/sshd.pid
  start program "/etc/init.d/ssh start"
  stop program "/etc/init.d/ssh stop"
  if failed port 22 protocol ssh then restart
  if 5 restarts within 5 cycles then timeout

Monit nginx /etc/monit/conf.d/nginx:

check process nginx with pidfile /opt/nginx/logs/nginx.pid
  start program = "/etc/init.d/nginx start"
  stop program  = "/etc/init.d/nginx stop"
  if 5 restarts with 5 cycles then timeout

Monit bouncy /etc/monit/conf.d/bouncy:

check process bouncy with pidfile /var/www/your_app/pids/bouncy.pid
  start program = "/etc/init.d/bouncy start"
  stop program  = "/etc/init.d/bouncy stop"

Monit your nodejs app /etc/monit/conf.d/app:

check process your_app with pidfile /var/www/your_app/pids/your_app.pid
  start program = "/etc/init.d/your_app start" as uid 1000 with gid 1000
  stop program  = "/etc/init.d/your_app stop" as uid 1000 with gid 1000

You can now start and stop monit by typing /etc/init.d/monit start and /etc/init.d/monit stop. Awesome! :)

Bonus: Open your browser at http://your_hostname.your_domain.com:2812 and introduce the user/password you specified on the /etc/monit/monitrc file. Lovely, isn't it?