Elixir/Phoenix deployments using Distillery

Yuva  - November 26, 2016  |   ,

Of late, we have been porting one of our internal apps from Rails to Phoenix. We are using Capistrano for deploying Rails apps. We have Jenkins CI which listens to features merged into master, and uses Capistrano to deploy to production.

For Elixir/Phoenix, we are looking for something similar. Merge features into master, let Jenkins CI run, and package Phoenix app which can be run on production. In Elixir world, there are bunch of package managers

You can read more about Exrm and Distillery here

Since Distillery is the latest one and also fits our use case nicely, let’s dig through that tool more. Documentation of Distillery is quite nice. In this blog post, we are going to explore:

Before getting started, you need these:

Since Elixir has a compilation step where it compiles Elixir code to BEAM, we need to set up a work flow for compiling and deploying our Elixir application. A typical work flow would look like:

Interesting thing to note here is, CI server needs source code of all dependencies of the Elixir app, but production server uses only the compiled BEAM code just to make it clear. So there is no bloat on production servers, it’s easy to spin up new servers, unpack the package, and start the app.

Let’s explore Distillery, and how it helps in deploying Phoenix apps. The steps to create and run the distillery based app are taken from Distillery documentation.

Create Phonenix app and initialize Distillery

Let’s create a simple Phoenix app which we will use throughout our discussion.

> mix phoenix.new --no-ecto --no-brunch phoenix_app
* creating phoenix_app/config/config.exs
* creating phoenix_app/config/prod.secret.exs
* creating phoenix_app/config/test.exs
* ...

Fetch and install dependencies? [Yn] Y
* running mix deps.get

We are all set! Run your Phoenix application:

> cd phoenix_app
> mix phoenix.server

If it’s all good, you should be able to see the Phoenix app at localhost:4000 Now, open mix.exs and append distillery

  defp deps do
    [{:phoenix, "~> 1.2.1"},
     {:phoenix_pubsub, "~> 1.0"},
     {:phoenix_html, "~> 2.6"},
     {:phoenix_live_reload, "~> 1.0", only: :dev},
     {:gettext, "~> 0.11"},
     {:cowboy, "~> 1.0"},
     {:distillery, "~> 0.10.1"}]
  end

Now, fetch dependencies using mix, and initialize repository with Distillery

> mix deps.get
> mix release.init

This should create a rel folder with config.exs file in it. Please take a moment to go through this file which is well commented to understand what it does.

Configuring CI and generating release pack

CI does all the heavy lifting in order to cut a release. You can run these commands locally and replicate the same on CI server. Make sure CI server has these packages installed:

Cutting a release is very easy, just run this command:

> MIX_ENV=prod mix release --env=prod
==> gettext
Compiling 1 file (.erl)
...
==> Assembling release..
==> Building release phoenix_app:0.0.1 using environment prod
==> Including ERTS 8.1 from /usr/local/Cellar/erlang/19.1/lib/erlang/erts-8.1
==> Packaging release..
==> Release successfully built!
    You can run it in one of the following ways:
      Interactive: rel/phoenix_app/bin/phoenix_app console
      Foreground: rel/phoenix_app/bin/phoenix_app foreground
      Daemon: rel/phoenix_app/bin/phoenix_app start

If you look at the output of this command, you’ll notice that mix is packaging everything along with the app. Interpreting mix release output:

Optionally you can set up Travis/Jenkins to observe features merged into git, automatically pulling latest source code, and packaging app

Now, take a look at rel/phoenix_app folder:

rel
└── phoenix_app
    ├── bin
    │   ├── nodetool
    │   ├── phoenix_app
    │   ├── release_utils.escript
    │   └── start_clean.boot
    ├── erts-8.1
    │   ├── bin
    │   │   ├── beam
    │   │   ├── beam.smp
    ....
    │   ├── include
    │   │   ├── driver_int.h
    │   │   ├── erl_nif.h
    │   ├── lib
    │   │   ├── internal
    │   │   ├── liberts.a
    │   │   └── liberts_r.a
    ├── lib
    │   ├── compiler-7.0.2
    │   ├── cowboy-1.0.4
    │   ├── crypto-3.7.1
    ...
    │   ├── phoenix-1.2.1
    │   ├── phoenix_app-0.0.1
    │   ├── ranch-1.2.1
    └── releases
        ├── 0.0.1
        │   ├── commands
        │   ├── hooks
        │   ├── phoenix_app.boot
        │   ├── phoenix_app.rel
        │   ├── phoenix_app.script
        │   ├── phoenix_app.sh
        │   ├── phoenix_app.tar.gz
        │   ├── start_clean.boot
        │   ├── sys.config
        │   └── vm.args
        ├── RELEASES
        └── start_erl.data

Going through the top level folders:

Making successive deployments

Erlang has a rich heritage, and Erlang programs are designed to run for years without bringing servers down, which guarantees nearly 100% up time. Rolling out bug fixes, new features, improvements are done using hot updates to servers. Erlang provides ways to patch existing running code on production servers so that there is no need to stop and start the app. Let’s look at ways to deploy phoenix_app

Hot upgrades

distillery provides support to create releases which can be applied as hot upgrades. The process of generating the tar file is same, but the command is different. For the sake of brevity, change version number in mix.exs from 0.0.1 to 0.0.2 before proceeding

> MIX_ENV=prod mix release --env=prod --upgrade
==> Assembling release..
==> Building release phoenix_app:0.0.2 using environment prod
==> Including ERTS 8.1 from /usr/local/Cellar/erlang/19.1/lib/erlang/erts-8.1
==> Generated .appup for phoenix_app 0.0.1 -> 0.0.2
==> Relup successfully created
==> Packaging release..
==> Release successfully built!
    You can run it in one of the following ways:
      Interactive: rel/phoenix_app/bin/phoenix_app console
      Foreground: rel/phoenix_app/bin/phoenix_app foreground
      Daemon: rel/phoenix_app/bin/phoenix_app start

The command to create a new release is same as that of first run, but there is a new argument, i.e --upgrade. Also, output of command is also slightly different. Generated .appup for phoenix_app 0.0.1 -> 0.0.2, where .appup means hot upgrade. There will be new folder 0.0.2 under releases. There will be another file called relup which contains instructions about how to upgrade. It looks like this:

{"0.0.2",
 [{"0.0.1",[],
   [{load_object_code,{phoenix_app,"0.0.2",
                                   ['Elixir.PhoenixApp.Endpoint',
                                    'Elixir.PhoenixApp.Gettext']}},
    point_of_no_return,
    {load,{'Elixir.PhoenixApp.Endpoint',brutal_purge,brutal_purge}},
    {load,{'Elixir.PhoenixApp.Gettext',brutal_purge,brutal_purge}}]}],
 [{"0.0.1",[],
   [{load_object_code,{phoenix_app,"0.0.1",
                                   ['Elixir.PhoenixApp.Endpoint',
                                    'Elixir.PhoenixApp.Gettext']}},
    point_of_no_return,
    {load,{'Elixir.PhoenixApp.Endpoint',brutal_purge,brutal_purge}},
    {load,{'Elixir.PhoenixApp.Gettext',brutal_purge,brutal_purge}}]}]}.

This file contains instructions about how to switch between 0.0.1 and 0.0.2. If the upgrade needs to be rolled back, this file helps in downgrading from 0.0.2 to 0.0.1.

Note about versions

Mix knows only semantic versioning. If one has to use SHA-IDs as the version, mix will throw errors. Say, you change version from 0.0.2 to 48dcbccd, and try to generate new package, mix throws this error:

> MIX_ENV=prod mix release --env=prod --upgrade
Compiling 11 files (.ex)
** (Mix) Expected :version to be a SemVer version, got: "48dcbccd"

If you are coming from Rails world, and used to continuous deployments using Capistrano, editing version every time for deployments is a pain. Let’s see how we can get Capistrano kind of continuous deployments.

Generating versions on the fly

One way to generate versions on the fly is to read version from environment variable. Also, since mix enforces semantic versioning, generating incremental versions makes sense. Capistrano generates folders with YYYYMMDDHHMMSS format, i.e. year, month, date. Change mix.exs file to have version like this:

def project do
  [app: :phoenix_app,
   version: (if Mix.env == :prod, do: System.get_env("APP_VERSION"), else: "0.0.1"),
   elixir: "~> 1.2",
   elixirc_paths: elixirc_paths(Mix.env),
   compilers: [:phoenix, :gettext] ++ Mix.compilers,
   build_embedded: Mix.env == :prod,
   start_permanent: Mix.env == :prod,
   deps: deps()]
end

Now when mix is building package for production, the app read the version from the environment variable, otherwise it’s hard coded to 0.0.1. Since version can be dynamic now, we can let CI specify what version needs to be generated. Following how Capistrano generates versions, we can do

export APP_VERSION=`date +%Y.%m.%d%H%M`

npm install                     # for brunch
mix deps.get
MIX_ENV=prod mix release --env=prod --upgrade

So, this code generates the app version with year as major version, month as minor version, and as patch version. Note that in order for `--upgrade` to work on CI, you should have previous versions in releases folder, otherwise upgrade will fail because there is no reference release. If you are using Travis/Circle-CI, make sure that releases folder is cached.

Recovering from failures, doing proper patch updates

Say code which is pushed in is buggy, and CI has made a release. It can happen that hot upgrade itself fails. In such cases, CI will have a stale releases like this:

rel/
  phoenix_app/
    releases/
      0.0.1             # deploy succeeded
      2016.09.250826    # deploy succeeded
      2016.09.250829    # deploy failed
      2016.09.251007    # new package based on failed deploy package

When CI runs again, it will generate a new package assuming previous package is a good one. Hot upgrades will start failing from now on!

It’s very important to keep a note of which deploy succeeded, or which one failed. There are two ways this can be done:

If you follow 2nd approach, you can read current running version from the file start_erl.data. It contains runtime version and app version. CI can read that version from production server, and create a hot upgrade from that version. In addition to --upgrade option, Distillery supports --upfrom option also. This takes in app version from which upgrade package has to be generated. Typically CI code looks like this:

export PREV_APP_VERSION=`ssh mix@10.0.0.5 cat /srv/app/releases/start_erl.data | cut -d' ' -f2`
export APP_VERSION=`date +%Y.%m.%d%H%M`

touch mix.exs                   # so that new app_version is picked
npm install                     # for brunch
mix deps.get
MIX_ENV=prod mix release --env=prod --upgrade --upfrom=$PREV_APP_VERSION

Now hot upgrades always work!

Immutable infrastructure

Immutable infra means - services in production have to be treated as immutable entities and upgrades should not change anything in production, but instead should be redeployed each time. This means no hot-upgrades are allowed in production. If you are running stateless services like web servers written in Phoenix, sticking with immutable infrastructure is recommended. Just create a package, spin up a new node behind load balancer, run the app, and nuke old nodes.

Compiling assets using plugins

NOTE: We haven’t initialized our app to use brunch, so the following section won’t work. Try creating a new app without --no-brunch option, and follow this section.

Mostly people use Phoenix for APIs, and use Reactjs or other JS framework for front-end. If you are reading through Distillery docs, it suggests to push assets compilation to shell script itself. Let’s take a small detour and see how we can do that using plugins. A Distillery plugin can be used to hook into the package generation process. It has 5 hooks:

Names are pretty self-explanatory. If you want to know more about these, you can take a look at them here. You can hook into before_assembly block, and compile assets.

defmodule PhoenixApp.PhoenixDigestTask do
  use Mix.Releases.Plugin

  def before_assembly(%Release{} = _release) do
    info "before assembly!"
    case System.cmd("npm", ["run", "deploy"]) do
      {output, 0} ->
        info output
        Mix.Task.run("phoenix.digest")
        nil
      {output, error_code} ->
        {:error, output, error_code}
    end
  end

  def after_assembly(%Release{} = _release) do
    info "after assembly!"
    nil
  end

  def before_package(%Release{} = _release) do
    info "before package!"
    nil
  end

  def after_package(%Release{} = _release) do
    info "after package!"
    nil
  end

  def after_cleanup(%Release{} = _release) do
    info "after cleanup!"
    nil
  end
end


environment :prod do
  set plugins: [PhoenixApp.PhoenixDigestTask]
  set include_erts: true
  set include_src: false
end

So, more Elixir code and less shell scripting. When you run release command again, output looks like this:

==> Assembling release..
==> Building release phoenix_app:2016.10.251007 using environment prod
==> before assembly!                 ## <- Plugin print here
==>
> brunch build --production

25 Oct 10:07:47 - info: compiled 3 files into 2 files, copied 3 in 3.1 sec

Check your digested files at "priv/static"
==> Including ERTS 8.0.2 from /usr/lib/erlang/erts-8.0.2
==> Generated .appup for phoenix_app 2016.09.261241 -> 2016.10.251007
==> Relup successfully created
==> after assembly!                  ## <- Plugin print here
==> Packaging release..
==> before package!                  ## <- Plugin print here
==> after package!                   ## <- Plugin print here
==> Release successfully built!
    You can run it in one of the following ways:
      Interactive: rel/phoenix_app/bin/phoenix_app console
      Foreground: rel/phoenix_app/bin/phoenix_app foreground
      Daemon: rel/phoenix_app/bin/phoenix_app start

You can see logs from the plugin throughout the deploy process. Plugins is an interesting concept, please take a look at that.

You can find whole source code here

Thanks for reading! If you would like to get updates about subsequent blog posts from Codemancers, do follows us on twitter: [@codemancershq][twitter]. Feel free to get in touch if you’d like Codemancers to help you build your Elixir/Phoenix app.