APIs supporting snake_case and camelCase with Rails

Yuva  - August 25, 2016  |   ,

Recently, for one of our clients, we encountered one interesting problem. Their stack is Rails + React, where Rails is for providing APIs, and React is for consuming those APIs. Frontend team prefers API payloads to be in camelCase. There are existing npm modules which can automatically convert snakecase to camelCase, but (un)fortunately front-end team is not using those.

We use AMS (active model serializers) to generate json responses. We are dealing with Rails 3, and app cannot be upgraded :). Latest version of AMS has out of box support for transformation. Say if we have to expose attributes of model Post, we have this code:

# ignore attributes (or schema), they are for demo'ing the usecase
class Post < ActiveModel::Serializer
  attributes :id, :title, :authorName, :totalComments
end

Using camelCase in ruby code looks ugly, but legacy code responds with camelCase for apis. One (obvious) solution is to request frontend team to use npm packages to convert snakecase to camelCase for GET requests, and camelCase to snakecase for all POST requests. Making that change would involve touching lots of components, which we didn’t want to do right away. We looked for alternatives and figured out this approach.

Let me show you how the tests for the Post object described above look, so that you get undestand what the problem is:

describe PostsController do
  let(:post) { FactoryGirl.create(:post, title: "hello world", author_name: 'Jon') }

  describe 'GET show' do
    it "returns all details of a post" do
      get :show, id: post.id
      expect(response).to be_success

      expected_json = {
        "id" => post.id,
        "title" => "hello world",
        "authorName" => 'Jon',       # <-------- camelCase here!
        "totalComments" => 0         # <-------- camelCase here!
      }

      post_json = JSON.parse(response.body)
      expect(expected_json).to eq post_json
    end
  end

  describe 'PUT update' do
    it "updates post fields successfully" do
      #                                  params has camelCase here!
      put :update, id: post.id, title: "new title", authorName: "Ben"
      expect(response).to be_success
    end
  end
end

If you look at specs, generally we don’t use camelCase in ruby to write specs. The code tries to map authorName to author_name, lots of copying going around. Let’s go step-by-step in improving this situation:

Step 1: Make snakecase/camelCase configurable via urls

We modified all APIs to support a param called snakecase. If this query param is set, APIs are served in snakecase, otherwise they are served in camelCase. So, modified specs look like this:

describe PostsController do
  let(:post) { FactoryGirl.create(:post, title: "hello world", author_name: 'Jon') }

  describe 'GET show' do
    it "returns all details of a post" do
      get :show, id: post.id, snakecase: 1           # see, snakecase=1 here
      expect(response).to be_success

      expected_json = {
        "id" => post.id,
        "title" => "hello world",
        "author_name" => 'Jon',     # note we have snakecase here!
        "total_comments" => 0       # note we have snakecase here!
      }

      post_json = JSON.parse(response.body)
      expect(expected_json).to eq post_json
    end
  end

  describe 'PUT update' do
    it "updates post fields successfully" do
      #                          note: authorName is author_name, we have snakecase!
      put :update, id: post.id, title: "new title", author_name: "Ben",  snakecase: 1
      expect(response).to be_success
    end
  end
end

And AMS also looks sane, ie it looks like this:

# ignore attributes (or schema), they are for demo'ing the usecase
# note: we have snakecase here for author_name, and total_comments!
class Post < ActiveModel::Serializer
  attributes :id, :title, :author_name, :total_comments
end

Bit of faith restored for ruby developers. APIs typically look like this now:

But frontend still expects payload to be in camelCase. It’s simple, avoid the query param snakecase=1 and we are all good.

Step 2: Implement snakecase/camelCase intelligence in controller params

Some problems can be solved with another level of indirection. We are going to play with controller level params, and provide one nice wrapper around it. We are not going to use params directly in our controllers, we are going to use a wrapper called api_params. Before, we jump into code, we have to support usecases like these for frontend:

Note those camelCases in urls, and POST/PUT body. Now, onto code:

class ApisController < ApplicationController
  # yet to implement snakecase_params method.
  def api_params
    params[:snakecase].present? ? params : snakecase_params
  end
end

class PostsController < ApisController
  def index
    search_params = api_params.slice(:author_name, :page, :per_page)
    @posts = SearchPostsService.new(search_params).fetch
    render json: @posts, serializer: PostSerializer
  end

  def update
    @post = Post.find(api_params[:id])
    update_params = api_params.slice(:title, :author_name)
    UpdatePostService.new(@post, update_params).save
  end
end

Ignoring specifics of code, instead of using params in controller, we are using api_params. What this does is:

snakecase_params method is interesting, and simple. All it has to do is to perform a deep key transformation. Basecamp has already written some code to deep transform hashes. Code can be found in deep hash transform repo. We are going to re-use that code. Code looks like this:

module ApiConventionsHelper
  extend ActiveSupport::Concern

  class HashTransformer
    # Returns a new hash with all keys converted by the block operation.
    #  hash = { person: { name: 'Rob', age: '28' } }
    #  hash.deep_transform_keys { |key| key.to_s.upcase }
    #  # => {"PERSON"=>{"NAME"=>"Rob", "AGE"=>"28"}}
    def deep_transform_keys(hash, &block)
      result = {}
      hash.each do |k, v|
        result[yield(k)] = value.is_a?(Hash) ? deep_transform_keys(v, &block) : v
      end
      result
    end

    def snakecase_keys(hash)
      deep_transform_keys(hash) { |k| k.to_s.underscore.to_sym }
    end
  end

  def snakecase_params
    HashTransformer.new.snakecase_keys(params)
  end
end

Now, we can inject this concern into our ApisController, and boom, we have snakecase_params helper. Now, we have backend developers happy with using snakecase, and frontend developers are happy working with camelCase. What else is left? Yes, automatically transforming payload for GET requests.

Step 3: Teaching AMS to be aware of snakecase/camelCase

Remember, our AMS for Post is still using snakecase. Now based on url params, we have to transform attributes. Let’s take a look at AMS for Post again:

class Post < ActiveModel::Serializer
  attributes :id, :title, :author_name, :total_comments
end

AMS makes it easy to play with attributes, again with another level of indirection.

First, we will use serializer options at controller level to tell AMS to transform payload to camelCase or snakecase.

class ApisController < ApplicationController
  def default_serializer_options
    { root: false, snakecase: params[:snakecase].present? }
  end

  def api_params
    params[:snakecase].present? ? params : snakecase_params
  end
end

All AMSes will now pickup default_serializer_options from controller. In these default options, we are appending snakecase option into AMS world. Now, how do we tell AMS to transform payload to camelCase or snakecase? Its simple: Use a base serializer, and derive all serializers from it.

class BaseSerializer < ActiveModel::Serializer
  # override attributes method.
  def attributes
    @options[:snakecase].present? ? super : camelize(super)
  end
end

We have to implement camelize now. We will extend ApiConventionsHelper and implementcamelize

module ApiConventionsHelper
  extend ActiveSupport::Concern

  class HashTransformer
    def deep_transform_keys(hash, &block)
      result = {}
      hash.each do |k, v|
        result[yield(k)] = value.is_a?(Hash) ? deep_transform_keys(v, &block) : v
      end
      result
    end

    def snakecase_keys(hash)
      deep_transform_keys(hash) { |k| k.to_s.underscore.to_sym }
    end

    def camelize_keys(hash)
      deep_transform_keys(hash) { |k| k.to_s.camelize(:lower) }
    end
  end

  def snakecase_params
    HashTransformer.new.snakecase_keys(params)
  end

  def camelize(hash)
    HashTransformer.new.camelize_keys(hash)
  end
end

# use ApiConventionsHelper and derive PostSerializer from this class
class BaseSerializer < ActiveModel::Serializer
  include ApiConventionsHelper

  def attributes
    @options[:snakecase].present? ? super : camelize(super)
  end
end

class Post < BaseSerializer
  attributes :id, :title, :author_name, :total_comments
end

Finally

With bit of conventions in place, and playing around controllers, and AMS, we are able to keep both backend developers and frontend developers happy. Interesting point to note here is, there is hardly any meta programming.

Thanks for reading! If you would like to get updates about subsequent blog posts Codemancers, do follows us on twitter: @codemancershq.

Working with Rails, Webpack and Heroku

Kashyap  - April 8, 2016  |  

Deployment of a Rails app that uses Webpack to Heroku is tricky because it requires both Node.js and Ruby. After various unsuccessful attempts, I had a solution that worked without customizing Heroku buildpacks. If you know how Heroku buildpacks work, skip to the summary for the TL;DR version.

A little bit of background first. The application I’m working on uses React for writing interactive components on the front-end. Rails handles routing and database-related functions. The React components are written in ES6 syntax that gets transpiled to the more widely supported ES5 API via the Babel transpiler. ESLint is used for code linting, and Redux for handling complex state transitions of React components. All the modular JavaScript components are bundled into a single file using Webpack.

It’s worth pointing out that there’s already a well supported React gem for use with Rails react-rails, but we chose to use npm instead. We wanted to leverage the latest version(s) of React, Redux, ESLint, and we felt that using the npm-based package management workflow works best in this case.

This is the application directory structure we decided to go with:

app/
lib/
config/
... (usual rails directories)
webpack/
  |-- package.json
  |-- components
  |-- reducers

Initially the package.json required by npm was placed inside the webpack/ directory, but this made deployment to Heroku a bit complicated. The Webpack asset compilation step bundles all the JavaScript files into a single file called react_bundle.js and places it under the app/assets/javascripts directory. This file is in turn loaded into Rails’ asset pipeline. So the prerequisite for the asset pipeline to work properly is that a react_bundle.js file must be present in app/assets/javascripts by the time assets:precompile step is run during deployment.

Heroku buildpacks

Heroku uses setup scripts they call buildpacks for figuring out the type of application, and to run the necessary build steps. For example, Heroku runs build steps for a Rack-based application if it finds a Gemfile in the root directory. If it finds a Railties gem in the Gemfile specs, it will trigger the Rails related build steps. At a simplified level, these are the steps that are run for a Rack application:

  1. Check for Gemfile and Gemfile.lock in the root directory. If it’s present, assume this is a Ruby application and start Ruby related build steps.
  2. Check to see if specs in Gemfile.lock has Rack gem as a dependency.
  3. Install necessary Ruby version
  4. Install bundler and install all dependencies
  5. Run a web process based on a Procfile or the default web process

In the same way, Heroku figures out the build steps for a Node.js application based on the presence of a package.json file in the root directory. So what if we need both npm related steps and Ruby related steps? We need to keep both the Gemfile and package.json in the root directory. That’s step 1.

Multiple Heroku Buildpacks

Heroku supports having multiple buildpacks for a single application. If we use both Node.js and Ruby buildpacks, we need a way to tell Heroku to run the npm-related steps first, and only then to run the Ruby related steps. This is to ensure that the react_bundle.js file is present under app/assets/javascripts when Rails’ asset pipeline commands are run. The Heroku command line provides a way configure the build packs and specify an order to them:

heroku buildpacks:clear
heroku buildpacks:set heroku/nodejs
heroku buildpacks:add heroku/ruby --index 2

The --index argument means the heroku/ruby buildpack is inserted after the heroku/nodejs buildpack. This completes step 2.

Post build hook on Heroku

Once the buildpacks are setup, we need a way to run a Webpack command right after the Node.js setup is completed, and just before the Ruby/Rails related setup starts. Heroku’s Node.js buildpack provides a way for users to run a custom script right after the normal Node.js build steps are completed. For this, a command must be specified in package.json under the scripts section with a heroku-postbuild key. Here is the relevant section from package.json:

"scripts": {
  "webpack:deploy": "webpack --config=webpack/webpack.config.js -p",
  "heroku-postbuild": "npm run webpack:deploy"
},

This completes step 3. With these three steps, I was able to setup a workable Webpack + Rails scaffold on Heroku.

Summary

  1. Put the package.json in the root folder of the Rails application.
  2. Setup multiple buildpacks on Heroku via Heroku’s command line utility, with heroku/nodejs in the first position and heroku/ruby second.
  3. Add the webpack bundling command in package.json’s scripts section with heroku-postbuild as the key.

New in ActiveRecord: #in_batches

Kashyap  - December 23, 2015  |  

Rails recently introduced a new method #in_batches similar to the existing #find_in_batches method, the difference being what class the yielded object belongs to. In the case of the latter the yielded collection is an Array whereas in the new method, the yielded collection is an ActiveRecord::Relation. The following post is a usecase that we found for this new method.

In one of our client projects, there is code that looks like:

def fetch_payments_for_merchants(merchants)
  PaymentFetcher.where(merchant_id: merchants.map(&:id))
end

Mapping over a collection of ActiveRecord objects results in an N+1 query syndrome, which means that for each iteration of the #map method, we make an SQL call. A detailed description about N+1 queries and how to avoid them is provided in this Rails guides article. To summarize that section, we should prefer using the #pluck that works on any ActiveRecord collection object to using #map.

I was looking to refactor the codebase to use the #pluck method, but it was not possible here as the argument is not always an ActiveRecord collection object. This method is also used for report generation method, which in turn uses the #find_in_batches method to fetch a batch of merchant objects and operate upon them, inorder to reduce memory usage. The #find_in_batches method takes a block and yields an Array containing the batch of merchants. Something like this:

def generate_csv
  Merchant.find_in_batches do |merchants|
    payments = fetch_payments_for(merchants)

    # do some report generation
  end
end

# elsewhere in the controller

def show
  @merchants = Merchant.where(id: params[:id])
  @payments  = PaymentFetcher.fetch_payments_for(@merchants)
end

So using #pluck would fail with a NoMethodError when the app has to generate a report—at least, as of Rails 4.1.x. To make method work in both cases, the original author used the .map method, since it works on both Arrays and ActiveRecord collection objects. But this penalizes the case where the user loads the page in the browser because there is a possibilty of the show method resulting in an N+1 query. And this is precisely where the new #in_batches method helps. So the generate_csv action can be rewritten as:

def generate_csv
  Merchant.in_batches do |merchants|
    payments = fetch_payments_for(merchants)

    # do some report generation
  end
end

private

def fetch_payments_for(merchants)
  PaymentFetcher.where(merchant_id: merchants.pluck(:id))
end

Now for the bad news: the #in_batches method is only available in Rails master (that would mean version 5, as of today). As per Rails’ maintenance policy, new features are not backported to older version. This leaves us with some options:

Polymorphic style conditional

In the fetch_payments_for method, we check the class of the merchants object, and switch between #map and #pluck accordingly.

def fetch_payments_for(merchants)
  merchant_ids =
    if merchants.is_a?(Array)
      merchants.map(&:id)
    else
      merchants.pluck(:id)
    end
  PaymentFetcher.where(merchant_id: merchant_ids)
end

This works, but pretty ugly.

Monkeypatching Array

The other, equally ugly/hacky, way would be to monkeypatch the Array class and define a #pluck method:

def fetch_payments_for(merchants)
  PaymentFetcher.where(merchant_id: merchants.pluck(:id))
end


class Array
  # The argument is named columns because this method is expected to be
  # used only on an Array of ActiveRecord items.
  def pluck(*columns)
    map do |item|
      columns.map do |column|
        item.send(column)
      end
    end
  end
end

IMO, this implementation is incredibly hacky because of how the definition of a method in a general purpose type (Array) is now coupled with ActiveRecord object structure. Although Duck typing suggests that this is the way, I tend to think twice before polluting a language-level module. That said, as of Rails 5.x, #pluck is defined for Enumerable module via ActiveSupport core extensions, and so, technically, we could use this.

Monkeypatching ActiveRecord::Batches

The other option would be to backport the #in_batches method for the Rails version we use. This would be a bit more complicated than monkey-patching Array class, but it’s restricted only to the ActiveRecord::Batches module. This seemed like a nicer change, so we went with this for our case.

Cake walk: Using bower with rails

Yuva  - December 10, 2014  |   , , ,

Traditionally, in order to use any javascript library (js lib), Rails users will do either of these two things:

First option is problematic because users have to keep track of changes to js lib, and update them whenever required. Second option delegates this responsibility to gem author, and users will just bump up gem version and most of the times assume that latest gem will have latest js lib. Both these approaches have problems because everytime js lib author improves lib either users have to copy sources or gem authors have to make a new release.

Of late, creating js libs and distributing them through bower has gained lot of traction. There are different ways to use bower with Rails. A popular way is to use bower rails gem. This blog will not use this gem, but explores sprockets inbuilt support for bower.

Sprockets - Bower support

Sprockets has support for bower. It doesn’t do package management on its own, but it can understand bower package structure, and pick up js, and css files from bower package. Lets go through this simple example:

Setting up bower json file

Packages installed from bower need to be specified in a bower.json file. Run bower init command at the root of rails app to generate this file. This file will be checked into version control, so that other devs can also know about dependencies.

> bower init
? name: rails app
? version: 0.0.0
? description:
? main file:
? what types of modules does this package expose?:
? keywords:
? authors: Yuva <yuva@codemancers.com>
? license: MIT
? homepage: rails-app.dev
? set currently installed components as dependencies?: No
? add commonly ignored files to ignore list?: No
? would you like to mark this package as private which prevents
  it from being accidentally published to the registry?: Yes

{
  name: 'rails app',
  version: '0.0.0',
  homepage: 'rails-app.dev',
  authors: [
    'Yuva <yuva@codemancers.com>'
  ],
  license: 'MIT',
  private: true
}

? Looks good?: Yes

One of the important things to note here is to mark this package as private so that its not published by mistake. The generated bower.json file can be further edited, and un-necessary fields like homepage, author can be removed.

{
  "name": "rails app",
  "version": "0.0.0",
  "private": true
}

Note: This is a one time process.

Setting up .bowerrc file

Since rails automatically picks up assets from fixed locations, bower can be instructed to install packages to one of the pre defined locations. Create a .bowerrc file like this:

{
  "directory": "vendor/assets/javascripts"
}

Since bower brings in third party js libs, its recommended to put them under vendor folder. Note: This is a one time process

Installing Faker js lib and using it

Use bower install to install above said lib. Since .bowerrc defaults the directory to vendor/assets/javascript, Faker will be installed under this directory. Use --save option with bower to update bower.json.

> bower install Faker --save

Thats it! Faker lib is installed. Add an entry in application.js and use the lib.

//= require Faker

Note: Make sure that its just Faker and not Faker.js. More details on why no extension will be explained later in the blog post.

What just happened? How did it work?

The vendor/assets/javascript folder has a folder called Faker, and that folder does not have any file called Faker.js. Inspecting any page from the rails app, script tab looks like this:

Firefox inspect script tab

Looking at source code of Faker, there is a file called faker.js, and is under build/build folder. How did rails app know the location of this file, even though application.js doesnot have explicit path? This is where sprockets support for bower kicks in:

Now, bower.json for Faker has explicit path for faker.js, which will be returned by sprockets

{
  "name": "faker",
  "main": "./build/build/faker.js",
  "version": "2.0.0",
  # ...
}
Bonus: Digging into sprockets source code

First, sprockets will populate asset search paths. When sprockets sees require Faker in application.js, it checks for extension, and since there is no extension, ie .js, asset search paths will be populated with 3 paths:

Gist of the source:

  def search_paths
    paths = [pathname.to_s]

    # optimization: bower.json can only be nested one level deep
    if !path_without_extension.to_s.index('/')
      paths << path_without_extension.join(".bower.json").to_s
      paths << path_without_extension.join("bower.json").to_s
      # DEPRECATED bower configuration file
      paths << path_without_extension.join("component.json").to_s
    end
  end

Source code is here: populating asset paths

Second, while resolving require Faker, if bower.json file is found, sprockets will parse the json file, and fetches main entry. Gist of the source:

  def resolve(logical_path, options = {})
    args = attributes_for(logical_path).search_paths + [options]
    pathname = Pathname.new(path)
    if %w( .bower.json bower.json component.json ).include?(pathname.basename.to_s)
      bower = json_decode(pathname.read)
      yield pathname.dirname.join(bower['main'])
    end
  end

Source code is here: resolving bower json

Form object validations in Rails 4

Yuva  - March 22, 2014  |   ,

Of late, at Codemancers, we are using form objects to decouple forms in views. This also helps in cleaning up how the data filled by end user is consumed and persisted in the backend. So, far the results have been good.

What are form objects

This blog post assumes that you are already familiar with form objects. Railscasts has a nice screencast about form objects. Do check it out if you haven’t already.

Use case

Let’s say that there is an organization and it has several employees. We’re tasked to build a Rails app that provides an interface where an admin can select one or more employees and send them emails. A typical interface implementation might look like this:

Employee email form

After selecting employees, filling-in the subject and body, and upon clicking “Send”, the backend should send emails to the selected employees. This is done by passing the array of the ids of employees, the subject and body to the backend. The POST parameters for that request look like this:

{
  "utf8"=>"✓",
  "email_form"=>{"employee_ids"=>[""], "subject"=>"", "body"=>""},
  "commit"=>"Send emails to employees"
}

Mass mailer form

We will create a EmployeesMassMailerForm form to encapsulate the validations and performing the actual action of sending email. This form should accept the params sent by the form, perform validations like checking whether all the employee ids belong to organization etc., and then send the emails.

class Organization < ActiveRecord::Base
  def get_employees(ids)
    employees.where(id: ids)
  end
end

class EmployeeMassMailerForm
  include ActiveModel::Model

  attr_accessor :organization, :employee_ids, :subject, :body

  validates :organization, :employee_ids, :subject, :body, presence: true
  validate  :employee_ids_should_belong_to_organization

  def perform
    return false unless valid?

    @employees = organization.get_employees(xemployee_ids)
    @employees.each { |e| schedule_email_for(e) }
    true
  end

  private

  def employee_ids_should_belong_to_organization
    if organization.get_employees(employee_ids).length != employee_ids.length
      errors.add(:employee_ids, :invalid)
    end
  end

  def schedule_email_for(e)
    Mailer.send_email(e, subject, body)
  end
end

With Rails 4, ActiveModel ships with Model module which helps in assigning attributes, just like how you can do with ActiveRecord class, along with helpers for validations. It is no longer necessary to use other libraries for form objects. Just include ActiveModel in a PORO class and you are good to go.

Testing using rspec and shoulda

All the form objects can be broken down into 2 main sections:

  1. Validations
  2. Performing actions
Testing validations

Adding validations on forms and models is pretty straight forward. Except for database-related validations like uniqueness, all the ActiveRecord validations can be used on form objects. These validations also make it easy to display validation errors in the view.

At Codemancers, we mostly use rspec and shoulda for testing. Validations on forms can be tested like this:

describe EmployeeMassMailerForm do
  describe 'Validations' do
    it { should validate_presence_of(:organization) }
    it { should validate_presence_of(:employee_ids) }
    it { should validate_presence_of(:subject)      }
    it { should validate_presence_of(:body)         }

    context 'when employee ids belong to organization' do
      it 'validates form successfully' do
        employee_ids = [1, 2]
        organization = mock_model(Organization, get_employees: employees_ids)

        form = described_class.new(organization: organization, subject: 'Test',
                                   employee_ids: employee_ids, body: 'Test')
        expect(form).to be_valid
      end
    end

    context 'when one or more employee ids donot belong organization' do
      it 'fails to validate the form' do
        organization = mock_model(Organization, get_employees: [])

        form = described_class.new(organization: organization, subject: 'Test',
                                   employee_ids: [1, 2, 3], body: 'Test')
        expect(form).to be_invalid
      end
    end
  end
end

You can notice here that while validating employee ids, we use stubs and mock models so that tests never hit database. Testing a form that has validations is a bit hard, because one has to heavily stub and mock models until form becomes valid. But testing an invalid form is easy and sometimes easy to maintain. Notice that we do not care what get_employees returns and that we hard coded it with an empty array whose length is 0. Always try to put as many validations as possible on form object, so that very less exceptions are raised while performing actions.

Testing actions performed by form

Once all the validations pass, the form object will go ahead and perform the action it is supposed to do. It can be anything from sending emails to persisting objects to database. Lets see how we can test the action perform from above form.

describe EmployeeMassMailerForm do
  describe '#perform' do
    let(:organization) do
      employees = [stub(email: 'a@b.com'), stub(email: 'b@c.com')]
      mock_model(Organization, get_employees: employees)
    end

    let(:form) do
      described_class.new(organization: organization, subject: 'Test',
                          employee_ids: [1, 2], body: 'Test')
    end

    before(:each) do
      described_class.any_instance.should_receive(:valid?).and_return(true)
      InvitesMailer.deliveries.clear
    end

    it 'sends emails to all employees' do
      form.perform
      expect(InvitesMailer.deliveries.length).to eq 2
    end

    it 'returns true' do
      expect(form.perform).to be_true
    end
  end
end

The trick here is to hard-code valid? to be true in before block. Since we have already tested validations, we can hard code the return value of valid? to be true. This saves a bunch of db calls and mocks.

I hope you enjoyed this article and if you want to keep updated with latest stuff we are building or blogging about, follow us on twitter @codemancershq.

Random Ruby tips from trenches #1

Hemant  - January 7, 2014  |   , ,

  1. Rubocop and Flycheck :

    Flycheck is a emacs mode which helps us with IDE like warnings in Emacs. I am already using enh-ruby-mode which helps with some of the syntax errors and stuff, but what is nice about flycheck is it integrates with rubocop and shows rubocop errors in-place in the editor.

    A picture is worth thousand words so:

    rubocop with flycheck

  2. pry ––gem:

    pry --gem opens a pry session with ./lib added to $LOAD_PATH and ‘require’s the gem. A good shortcut while working on gems and you want a quick console with the gem loaded.

  3. ruby –S :

    This can be used for running binaries which are present in local directory. For example, if you are working on bundler and want to run bundle command with local version of bundler rather than one installed globally you can use:

       ruby -S ./bin/bundle -I ./lib/ install
    

    The advantages are:

    • ruby -S allows you to ignore #!/usr/bin/env ruby line and even if current version of ruby is X using ruby -S you can run the command with another version of Ruby. Especially useful for running scripts with JRuby, Rbx etc.

    • Based on RUBYPATH environment variable running a script via ruby -S (or jruby -S) allows PATH environment to be modified. For example running jruby -S gem does not run gem which is in current path, but it runs gem command provided by JRuby because JRuby defines different RUBYPATH.

  4. Faster rbenv load :

    If you are using rbenv and there is a lag while opening a new shell, consider updating the rbenv initializing line in your shell profile to:

     eval "$(rbenv init - --no-rehash)"
    

    The --no-rehash flag tells rbenv to skip automatic rehash when opening a new shell and speeds up the process. This also speeds up VIM if you are using vim-rails or vim-ruby.

Setting up Emacs as Ruby development environment on OSX

Hemant  - October 17, 2013  |   , , ,

Learning and setting up Emacs can be intimidating and while some folks find much use in using a pre-configured setup like - Emacs Prelude, I personally think it is better that someone new to Emacs installs each and every module himself and learns from it. I am assuming OSX as development environment here but the instructions should work well enough on Linux as well.

  1. Install Emacs : Install Emacs using Homebrew. I personally do not recommend using Emacs for OSX distribution. You can find out why here.

     brew install emacs --use-git-head --cocoa --srgb
    

    There is a bug in Emacs HEAD that prevents it to be working with Cask and thus try not to use Emacs HEAD.

  2. Install a Package Manager : I use pallet for managing various packages. Think of pallet as Bundler/Maven for Emacs. Follow instructions available in section For newly installed Emacs. Once you have pallet installed each package you install via package-install command will automatically update the Cask and you can checkin the file in version control system and carry it around.

    Sometimes you have to install a package directly from git. I personally use Git Subtrees for adding packages from git. Feel free to use submodules or directly copying files as well.

  3. Structuring your Customization : Over last couple of years I personally have been using following style of structuring my emacs setup.

     .emacs.d/
       init.el -> The real .emacs file.
       custom/
         01ruby.el
         02org.el
         03auto-complete.el
         04web-mode.el
    
  4. The .emacs file : Contents of init.el looks something like this:

     (require 'cask "~/.cask/cask.el")
     (cask-initialize)
     (require 'pallet)
     (add-to-list 'load-path "~/.emacs.d/custom")
     (add-to-list 'load-path "~/.emacs.d/other_paths)
     ..
     ..
     (load "00common-setup.el")
     (load "01ruby.el")
     (load "02org.el")
    
  5. Global Config : I use a lot of global configuration stuff which isn’t mode specific, some important ones are:

     ;; Navigate between windows using Alt-1, Alt-2, Shift-left, shift-up, shift-right
     (windmove-default-keybindings)
    
     ;; Enable copy and pasting from clipboard
     (setq x-select-enable-clipboard t)
    

    You can see complete file on, 00common-setup.el. If you are completely new to Emacs you should use C-h t (That is press Control-h and then t) to launch inbuilt Emacs tutorial from within Emacs.

Packages You need

For complete list of package see my Caskfile. Also some packages I use straight from git and hence may not be listed in Caskfile. With that out of the way, I will try to cover most important packages I love and use everyday:

  1. auto-complete : You can install it via package-install and below is my configuration of auto-complete mode:

     (require 'auto-complete-config)
     (add-to-list 'ac-dictionary-directories
         "~/.emacs.d/.cask/24.3.50.1/elpa/auto-complete-20130724.1750/dict")
     (ac-config-default)
     (setq ac-ignore-case nil)
     (add-to-list 'ac-modes 'enh-ruby-mode)
     (add-to-list 'ac-modes 'web-mode)
    

    It gives me access to all powerful auto-completion system for Emacs. Below is a screenshot.

    [Emacs auto-complete](/assets/images/emacs/auto-complete-7a77150cd11c64f1b5f22de4b0a52d72de61d7358109e0654dcf074cce7971cf.png

  2. ag : You should use silver searcher and corresponding Emacs mode ag. They beat likes of ack or grep hands down.

  3. enh-ruby-mode : Personally I am big fan of Enhanced Ruby Mode. It gives you better syntax highligting, better indentation schemes and code intelligence without running flymake kind of thing.

    ![enhanced ruby mode](/assets/images/emacs/enh-ruby-mode-adedbb79b2f586b70a48d9d3cebe318592a5db4e6867523124409c524dc2c4ff.png

    You can see rest of my enh-ruby-mode specific configuration in file 01ruby.el. You will notice that I have included a mini rspec-mode there , which allows me to run specs of a complete file or just spec under the cursor.

  4. smartparens : smartparens mode is too much of a moving target and hence I always use it from git. My smartparens configuration is:

     (require 'smartparens-config)
     (require 'smartparens-ruby)
     (smartparens-global-mode)
     (show-smartparens-global-mode t)
     (sp-with-modes '(rhtml-mode)
       (sp-local-pair "<" ">")
       (sp-local-pair ""))
    

    SmartParens mode is many things. It automatically inserts closing parethesis, tags, end’s depending on major-mode. Highlights them, allows you to move between them. Allows you to wrap existing texts. However you may not even have to go through their documentation, it mostly just works.

  5. Projectile Mode : Projectile is one among many ways of handling projects in Emacs. My projectile configuration looks like:

     (require 'grizzl)
     (projectile-global-mode)
     (setq projectile-enable-caching t)
     (setq projectile-completion-system 'grizzl)
     ;; Press Command-p for fuzzy find in project
     (global-set-key (kbd "s-p") 'projectile-find-file)
     ;; Press Command-b for fuzzy switch buffer
     (global-set-key (kbd "s-b") 'projectile-switch-to-buffer)
    
     [![It allows me to do:](/assets/images/emacs/emacs4-2be08a609329db017ab61dafd234d821fe408e5fffa0f0002957e8ca52fc0a65.gif](/assets/images/emacs/emacs4-2be08a609329db017ab61dafd234d821fe408e5fffa0f0002957e8ca52fc0a65.gif)
    
  6. Rainbow Mode : Add color to your css/scss. Check screenshot below:

    Add color

  7. Robe Mode : Robe mode is what makes Emacs full featured IDE even comparable to likes of RubyMine etc.

Robe mode:

Robe can also integrate with `auto-complete` mode to provide more intelligent auto-completion.
I personally do not use that feature because it makes things slower.
  1. Highlight indentation for Emacs : Highlight Indentation mode allows you to see indentation guides. My setup for the mode is:

     (require 'highlight-indentation)
     (add-hook 'enh-ruby-mode-hook
               (lambda () (highlight-indentation-current-column-mode)))
    
     (add-hook 'coffee-mode-hook
               (lambda () (highlight-indentation-current-column-mode)))
    
  2. Yasnippet : Yasnippet is Emacs snippet library. Out of box it won’t work with enh-ruby-mode because it searches for ruby-mode. You can make it work by copying ruby-mode snippets to enh-ruby-mode folder.

  3. Flyspell : If you are like me, you make lot of typos in code comments, strings etc. Using Flyspell mode you need not worry about it - interfering with programming language keywords etc. My setup for Flyspell is :

    (require 'flyspell)
    (setq flyspell-issue-message-flg nil)
    (add-hook 'enh-ruby-mode-hook
              (lambda () (flyspell-prog-mode)))
    
    (add-hook 'web-mode-hook
              (lambda () (flyspell-prog-mode)))
    ;; flyspell mode breaks auto-complete mode without this.
    (ac-flyspell-workaround)
    

    If you are on OSX you may have to run brew install ispell to make it work.

  4. Rinari : With Projectile mode, I am finding less and less use of this mode. But this is your canonical rails mode for Emacs. Get it via package-install or Rinari Github repo.

  5. dash-at-point : If you are on OSX you should totally use Dash. Using dash-at-point you can view documentation of methods, classes etc right from Emacs.

  6. multiple-cursors : Multiple Cursors is Emacs answer to multi-line editing feature of Sublime etc. Anything I can add here is probably better covered in this Emacs Rocks screencast.

  7. textmate.el : textmate.el allows me to quickly move between methods in a file, independent of programming language mode (and without ctags). It has many other utilities.

  8. WebMode : For a long time Emacs users were mostly stuck with half-assed multi-language mode when editing html templates. WebMode is like breath of fresh air for those who mix - ruby/js/css/coffee in their HTML!

Apart from aforementioned modes, I still use too many other modes - sometimes even unknowingly! Some modes that deserve honourable mention but I don’t use them is - Magit, Expand Region. If you do lot of Javascript then have a look at - Swank-js. If you are coming from Vim and looking for decent nerdtree replacement, you can try Dirtree or ECB or Project explorer.

Learning Emacs in a way is - as deep as you are willing go. For resources I recommend:

I hope you enjoyed this article and if you want to keep updated with latest stuff we are building or blogging about, follow us on twitter @codemancershq.

How rails reloads your source code in development mode?

Yuva  - October 3, 2013  |   ,

We all know Rails has this feature, which reloads source code in development mode everytime a request hits server. Starting from version 3.2.0, it introduced faster dev mode where it reloads application code when the code changes, and not on every request.

This blog will talk about important parts of Rails source code which helps in achieving faster dev mode.

Lets go through them one by one.

ActiveSupport::FileUpdateChecker

This class helps in checking whether application code is updated or not. It exposes 2 methods: updated? and execute. The former tells whether files are updated or not, while latter executes a block given by updating timestamp of latest changed file. The code can be found here. How file checker checks whether a file is updated or not depends on the modified time of the file. There is a small function max_time which returns timestamp for recently modified path (most recent one)

class ActiveSupport::FileUpdateChecker
  # NOTE: Removed some code to reflect the logic
  def updated?
    current_updated_at = updated_at(current_watched)
    if @last_update_at < current_updated_at
      @updated_at = current_updated_at
      true
    else
      false
    end
  end

  # Executes the given block and updates the timestamp.
  def execute
    @last_update_at = updated_at(@last_watched)
    @block.call
  end

  def updated_at(paths)
    @updated_at || max_mtime(paths) || Time.at(0)
  end

  # This method returns the maximum mtime of the files in +paths+
  def max_mtime(paths)
    paths.map {|path| File.mtime(path)}.max
  end
end

ActiveSupport::Dependencies

This module consists of the core mechanism to load classes, by following Rails naming conventions. It uses const_missing to catch missing classes, and then searches in autoload_paths to load those missing classes. The code can be found here.

module Dependencies
  def const_missing(const_name)
    from_mod = anonymous? ? ::Object : self
    Dependencies.load_missing_constant(from_mod, const_name)
  end

  def load_missing_constant(from_mod, const_name)
    qualified_name = qualified_name_for(from_mod, const_name)
    path_suffix = qualified_name.underscore

    file_path = search_for_file(path_suffix)

    if file_path
      expanded = File.expand_path(file_path)
      require_or_load(expanded)
    end

    raise NameError, "uninitialized constant #{qualified_name}"
  end
end

ActionDispatch::Reloader

This is a middleware which provides hooks that can be run while code reloading. It has 2 callback hooks, :prepare and :cleanup. Rails code will make use of these hooks to install code which determine whether to reload code or not. :prepare callbacks will run before request is processed, and :cleanup callbacks will run after request is processed. You can see call(env) of reloader here

class ActionDispatch::Reloader
  def call(env)
    @validated = @condition.call
    prepare!

    response = @app.call(env)
    response[2] = ::Rack::BodyProxy.new(response[2]) { cleanup! }

    response
  rescue Exception
    cleanup!
    raise
  end
end

reload_classes_only_on_change

This configuration option is defined in railties. By default, it is set to true, so Rails reloads classes only if code changes. Set it to false, and Rails will reload on each request. Digging into the place where this boolean is defined, we find that there is an initializer set_clear_dependencies_hook. This initializer is defined here.

initializer :set_clear_dependencies_hook, group: :all do
  callback = lambda do
    ActiveSupport::DescendantsTracker.clear
    ActiveSupport::Dependencies.clear
  end
 if config.reload_classes_only_on_change
    reloader = config.file_watcher.new(*watchable_args, &callback)
    self.reloaders << reloader
   ActionDispatch::Reloader.to_prepare(prepend: true) do
      reloader.execute
    end
  else
    ActionDispatch::Reloader.to_cleanup(&callback)
  end
end

The above code installs a file watcher if config var is true. watchable_args consists of autoload_paths along with other files like schema.rb. So, file_watcher is configured to watch these paths. If config var is false, it just installs callback as :cleanup hook, which means all the code will be unloaded after each request is processed.

How do these components fall in place?

By joining all the dots, the sequence is:

Bonus

If you want to know how routes reloading happens, check this file

Hope you have enjoyed this article, and follow us on twitter @codemancershq for all the awesome blogs, or you can use rss feeds.

Isolate Namespace in Rails Engines - A hack in itself

Yuva  - September 22, 2013  |   , , ,

isolate_namespace is that one feature which Rails boasts about to use while creating gems, but doesn’t work when one wants to extend models, controllers and views which are provided by that gem. We have used it while developing one of our gems and found that its really hard to extend models, controllers and views. For instance, if you copy views from gem to main application in order to customize them, and try to use routes defined in the main application, you will be slapped with an error saying those routes are not defined! It takes some time to understand how it works under the hood, and all it boils down to is: isolate_namespace is nothing but a bunch of hacks. This blog post will do a code walk-through and will try to explain how it works:

The isolate_namespace method

This method can be found in railties-engine file. Its reads as:

def isolate_namespace(mod)
  engine_name(generate_railtie_name(mod))

  self.routes.default_scope =
    { module: ActiveSupport::Inflector.underscore(mod.name) }
  self.isolated = true

  unless mod.respond_to?(:railtie_namespace)
    name, railtie = engine_name, self

    mod.singleton_class.instance_eval do
      define_method(:railtie_namespace) { railtie }

      unless mod.respond_to?(:table_name_prefix)
        define_method(:table_name_prefix) { "#{name}_" }
      end

      # removed code for :use_relative_model_naming?
      # removed code for :railtie_helpers_paths

      unless mod.respond_to?(:railtie_routes_url_helpers)
        define_method(:railtie_routes_url_helpers) {
          railtie.routes.url_helpers
        }
      end
    end
  end
end

mod is module, which is has to be isolated. In case of blorgh gem, its Blorgh itself. Hence forth, we will use blorgh gem as an example given in rails guides

  1. routes.default_scope: It defines default scope for routes. This scope will be used while generating routes for Rails engine. It says module to be used is blorgh, and all the controllers will be searched under gem. This can be easily understood. Just put a binding.pry inside routes.rb of gem, and you can see this:

    [4] pry(#<Mapper>)> self.instance_variable_get(:@scope)
    => {:path_names=>{:new=>"new", :edit=>"edit"},
    :module=>"blorgh",
    :constraints=>{},
    :defaults=>{},
    :options=>{}}

    It says default module that should be used is blorgh. All the controllers will be prepended by blorgh/

  2. If module doesn’t respond to railtie_namespace {generally modules dont!}, it goes ahead and adds bunch of methods to module {i.e Blorgh for example}, and not to engine. Thats why its a hack! There is nothing done on engine! Everything is added to Blorgh module. So, what it adds exactly?

  3. table_name_prefix: This can be easily guessed. It will be used by active record. Now searching through activerecord source, we find this:

    def full_table_name_prefix #:nodoc:
      (parents.detect{ |p| p.respond_to?(:table_name_prefix) } || self).
        table_name_prefix
    end

    It looks tricky, but this is how it works. It searches through all the parents of the AR class, and checks whether any of them responds to table_name_prefix and returns it. {Default value is empty string}. Well, parents are not the parents of the class, but the hierarchy of the modules. activesupport defines this method:

    # Returns all the parents of this module according to its name,
    # ordered from nested outwards. The receiver is not contained
    # within the result.
    #
    #   module M
    #     module N
    #     end
    #   end
    #   X = M::N
    #
    #   M.parents    # => [Object]
    #   M::N.parents # => [M, Object]
    #   X.parents    # => [M, Object]
    def parents
      parents = []
      if parent_name
        parts = parent_name.split('::')
        until parts.empty?
          parents << ActiveSupport::Inflector.constantize(parts * '::')
          parts.pop
        end
      end
      parents << Object unless parents.include? Object
      parents
    end

    Now, here comes the funny part: Create a folder called blorgh in your main application under app/models folder, and create a model called Test

    module Blorgh
      class Test < ActiveRecord::Base
      end
    end

    Now, fire up console, and execute the following:

    [1] pry(main)> Blorgh::Test.table_name
    => "blorgh_tests"

    See, it automatically prepends tests table name with blorgh_. One will be wondering how did that happen? Well it happened because of the module called Blorgh. So anything one puts under Blorgh module will get special treatment. Do it with any other namespace, {i.e module}, it will just return tests. The only way you can get rid of this behavior is to specify the table name explicity on model using self.table_name = "tests". If someone in some gem magically says to isolate a namespace which you are using in your application, hell breaks loose! You will be wondering why your application code is behaving strangely.

    You can find other hacks by searching through the Rails code. We will cover another hack here:

  4. railtie_routes_url_helpers: This method is used to define route helpers which can be accessible to generate paths. Digging through the code, you can find it in actionpack.

    module AbstractController
      module Railties
        module RoutesHelpers
          def self.with(routes)
            Module.new do
              define_method(:inherited) do |klass|
                super(klass)
                namespace = klass.parents.detect { |m|
                  m.respond_to?(:railtie_routes_url_helpers)
                }
    
                if namespace
                  klass.send(:include,
                      namespace.railtie_routes_url_helpers)
                else
                  klass.send(:include, routes.url_helpers)
                end
              end
            end
          end
        end
      end
    end

    This inherited method will be called in the context of your controllers. Again if your controller is under Blorgh module, it magically includes only the routes defined by gem, otherwise it includes the application route helpers. Thats why even though you copy views from gem to your main app, they still cannot access helpers defined by main application. Generally we all know how to fix this: Call the url helpers by prepending with either main_app or gem mount point, i.e blorgh here. This way all the route helpers will be available in all the views.

    = link_to "Surveys", blorgh.question_groups_path
    = link_to "Login", main_app.new_user_session_path
  5. Other issues include extending models and controllers. Rails guides gives two options here. One to use class_eval, and other to use concerns introduced in Rails 4. Both are kind of hacky. Hope there is a better solution.

Setting up Source Maps for CoffeeScript in Rails

Amitava  - September 5, 2013  |   ,

[IMPORTANT] Only works in development environment.

Source maps are a language agnostic way to map minified, combined or pre-processed code to the original source file. This becomes highly useful as it helps to debug or inspect compressed client-side code in browser (depends on browser support).

When using CoffeeScript we often find it difficult to locate the exact location of an error, since the line numbers in generated javascript code does not match with the original coffee file.

Using source maps we can instruct the browser to load the original coffee file and show us where the error has occurred. Also it is possible to debug a coffee file in a browser by setting breakpoints.

Let’s find out how to add source maps support for CoffeeScript. Requires CoffeeScript 1.6.1 or above.

![Enable source maps in Google Chrome](/assets/images/enable-chrome-source-maps-c0df1227641a3f3167da100c92096c1ee6062f159d8f25532ea4aa1c38101b71.png

Next time reloading the browser will generate source maps inside public/assets/source-maps directory and Chorme DevTools will display any js error right inside the coffee file instead of generated js code.

Source maps greatly simplifies the process of debugging CoffeeScript right inside the browser. However as mentioned at the beginning of this post, it only works in development at the moment. Adding support for production is quite challenging since it would require to map a single combined and minified file to the original source files. uglifier already seems to have support with this pull request and sprockets is also getting ready to add it to the asset pipeline. Hopefully Rails will pretty soon have support for source maps in production environment.

What about Firefox?

At the time of this writing Firefox 24 also has source maps support, however it didn’t work as expected as it did seemlessly in Chrome. Here is a good example of how to enable source maps it http://www.codeproject.com/Articles/649271/How-to-Enable-Source-Maps-in-Firefox.

Redefine rake routes to add your own custom tag in Rails

Deepak  - July 12, 2013  |   , ,

WTFs per minute

The only valid measurement of code quality is: WTFs/minute

Now Rails has this nifty rake task called rake notes. You can always see notes for a custom tag by:

~> rake notes:custom ANNOTATION=WTF

But rake notes by default only shows notes for OPTIMIZE, FIXME and TODO
The Rakefile for changing those defaults is

# for re-defining the Rake task
# otherwise the previous Rake task is still called
task(:notes).clear

desc "Enumerate all annotations 
      (use notes:optimize, :fixme, :todo, :wtf for focus)"
task :notes do
  SourceAnnotationExtractor.enumerate "OPTIMIZE|FIXME|TODO|WTF", 
                                      :tag => true
end

namespace :notes do
  task(:wtf).clear
  task :wtf do
    SourceAnnotationExtractor.enumerate "WTF", :tag => true
  end
end

Here we have redefined rake notes. The only gotcha is to remember
to clear the old rake task by calling task(:notes).clear

For other customizations eg. search the spec folder also
you will have to monkey-patch the SourceAnnotationExtractor class

PS: also check this Stackoverflow thread for some fun picks

All you need to know about writing a mountable Rails Gem with rspec

Yuva  - May 30, 2013  |   , , ,

This article briefly talks about how to write a mountable rails gem and what are the practices that I followed

First things first

Rails generates nice boilerplate for your gem which is sometimes hard to get it right in the first go. Use this simple command and you are good to go

$ rails plugin new <gem-name> --mountable --skip-test-unit --dummy-path=spec/dummy
  1. The --mountable option helps in creating a mountable gem.
  2. The --skip-test-unit skips generation of test templates in test folder
  3. The --dummy-path specifies where the dummy test application {which we will be using for testing} should be generated. It is usually created in test folder but you want that to be created in spec folder so that you can use rspec

Importance of dummy app

Some of the gems that you develop which depend on rails need to run specs in rails environment. Since gem doesn’t have a rails env, it uses dummy app’s rails env to run such tests. This is useful when you write specs for controllers, helpers etc.

When you set up rspec by adding rspec-rails to your Gemfile, you need to add these lines at the top of spec_helper.rb so that rspec loads rails env of dummy app.

ENV["RAILS_ENV"] ||= 'test'
require File.expand_path("../dummy/config/environment", __FILE__)

General flow of development

  1. Never checkin Gemfile.lock in your git repository {or whatever SCM that you are using}
  2. First, add all the dependent gems that your gem needs in Gemfile and develop your gem.
  3. Once you have finalized your dependent gems, move those gem declarations to gemspec file. Add all dependent gems with add_dependency method and add all development dependent gems with add_development_dependency method

    Gem::Specification.new do |s|
      # your information
    
      s.add_dependency "strong_parameters", "~> 0.2.1"
    
      s.add_development_dependency "factory_girl_rails"
    end
  4. Its important to note that you should require your dependent gems explicitly in the root file of your gem. Say if your gem is named my_cool_gem, then you should have my_cool_gem.rb created inside lib folder. If your gem is dependent on strong_parameters, then you need to add these lines:

    require "my_cool_gem/engine"
    require "strong_parameters"
    
    module MyCoolGem
    end

Profile Ruby 2.0 apps using DTrace - part 2

Hemant  - May 27, 2013  |   , , ,

This is Part - 2 in series of articles we are writing on Dtrace. If you haven’t already - do read Part 1.

Running Dtrace scripts

Before we delve deeper into guts of Dtrace, one of the questions which I often get asked is, How can I profile one off programs (i.e not a application running in background) with Dtrace.

There are basically couple of ways you can fire Dtrace.

~> sudo dtrace -qs <dtrace_script>.d
~> sudo dtrace -s count.d -c '/Users/gnufied/.rbenv/versions/20/bin/ruby leaky_app.rb'

What is a predicate?

In last part of tutorial we described what a probe description looks like. For a refresh:

probe description
/predicate/ 
{
    action;
}

Predicate is a way to filter the number of probes fired based on certain critirea. For example, in last tutorial we wrote a Dtrace script to keep track of all system calls being made by all currently running applications.

Lets say, now we want to restrict those probes only to system calls that are being made by Ruby applications. We can do this by introducing a predicate in our Dtrace script.

syscall:::entry
/execname=="ruby"/
{
	printf("%s(%d) called %s\n", execname, pid, probefunc);
}

Or you can use pid predicate:

/*
 * Print system calls made by pid 61465
 *
 */
syscall:::entry
/pid == 61465/
{
	printf("%s(%d) called %s\n", execname, pid, probefunc);
}

Now when I was learning Dtrace, I was mightly confused from where these things called execname, pid, probefunc are coming from? Turns out, Dtrace automatically provides built-in variable definitions for inspecting processes. They make Dtrace scripts quite powerful.

Built-in Variables

Here is a list of important built-in variables available in your DTrace scripts:

Now, I believe previous scripts would have started making sense. Above list does not include all built-in variables though.

Lets do an exercise and try to print number of bytes being read (from network or file system) by process with pid 61465. The script will look like:

syscall::read:entry
/pid == 61465/
{
    printf("%s read %d\n", execname, arg2);
}

If you crack open man 2 read you will find out that read system call’s third argument contains number of bytes being read.

Having just ability to print numbers as probes fire is useful but you can easily imagine it would be nice, if we can somehow aggregate the data. For example, how much of your program’s memory is taken by string allocation or how much time is being spent in total in the function foo. Ability to aggregate such data would be really useful.

Aggregates and Collections

Agrregates, Collections and variables are most important tools DTrace has to offer for data gathering, analyzing and printing. We will start with different types of variables available to us.

Variable types

D scripting language has 3 types of variables which you can use to keep track of things.

Block local

You can define a block local variable with :

syscall::read:entry
{
  this->currentTime = timestamp;
}

Scope of a block local variable is limited to the block in which it is defined.

Global scalar variables

You can define a global variable like:

BEGIN {
  startTime = timestamp;
}

Variables defined like this are global and they are available in all the probe descriptions. Also, if you haven’t guessed it - BEGIN and END[2] are two special probes that fire at the beginning of a DTrace script and at the end of it. They can be very useful for initializing things or printing summary results at the end of execution.

Thread local variables

If you are trying to profile a multi-threaded application you will want to keep DTrace variables separate for different threads. You can do that with thread local variables:

syscall::read:entry
{
  self->currentTime = timestamp;
}

Variables declared like this won’t be shared between threads and hence can be used reliably.

Associative Arrays

If you are coming from Ruby, think of associative array as a Hash where that keys can be a single value or a tuple of values. In either case once declared type of key shouldn’t change. For example, following script will calcuate time spent by each process in a system call.

syscall:::entry
{
  callTime[execname, probefunc] = timestamp;
}

syscall:::return {
    this->timeElapsed = timestamp - callTime[execname,probefun];
}

You can learn more about these different kind of variables at[1].

Aggregates

Armed with knowledge of variables and collections, we can finally dip our toes in aggregates. Aggregates is a way of gathering data from individual probes without you the programmer doing actual work of summing things up,calculating the median, frequency of occurence etc.

For example, lets write a DTrace script that prints number of top 20 objects Rails creates when it boots up. You are obviously going to need Ruby 2.0 for following script to work.

/* rails_boot.d
 * object-create is the name of the probe that fires 
 * when any new ruby object is created. 
 * Corresponding function called is - rb_obj_alloc
 */
ruby*:::object-create
{
   /* If you look up code of rb_obj_alloc, first argument to 
    * function contains class which is being instantiated
    */
   @objects[copyinstr(arg0)] = count();
}

END {
  /* Truncate the aggregate to top 20 results */
  trunc(@objects,20);
  /* Print the aggregate */
  printa(@objects);
}

The output should look something like:

  Mime::Type                                                       22
  Psych::Nodes::Scalar                                             27
  Bundler::LazySpecification                                       39
  Set                                                              54
  Sass::Script::Functions::Signature                               55
  Rails::Paths::Path                                               71
  Class                                                            73
  Pathname                                                         74
  Bundler::DepProxy                                                77
  Rails::Initializable::Collection                                 93
  Regexp                                                           95
  Rails::Initializable::Initializer                               143
  Range                                                           332
  Gem::Specification                                              410
  Hash                                                            579
  Gem::Dependency                                                1660
  Gem::Version                                                   2416
  Gem::Requirement                                               3061
  String                                                         4723
  Array                                                          4948

Keep in mind, running above script isn’t as simple as running sudo dtrace -qs rails_boot.d -c 'rails s' . Depending on whether you are using RVM or rbenv all those shell scripts shims create a problem when running dtrace scripts. What I ended up doing is, created a file called load_env.rb that had just one line require_relative config/environment and then ran the Dtrace script with:

sudo dtrace -qs rails_boot.d -c "/Users/gnufied/.rbenv/versions/20/bin/ruby load_env.rb" 

Now there are some new keywords in above script and we will to get them in a bit, but first let us talk about aggregates. DTrace allows you to store result of aggregate functions in objects called aggregates, they are defined like:

@name[ keys ] = aggfunc ( args );

Aggregate object name starts with @ and behave very similar to associative arrays and hence can use tuples as keys. Using aggregate functions frees you up from keeping counter yourself. In above script we are using count function which keeps track of number of times that particular probe fired. There are other aggergate functions available as well:

You can find out more about aggregates from [3].

That about sums up content for part 2 of DTrace tutorial. In the next part we are going to cover in built functions, printing and profiling of real world Ruby applications. You should also follow us on twitter @codemancershq if you don’t want to miss out Part 3.

References

Rails Autoload Path

Yuva  - May 1, 2013  |  

Typically autoload paths configured in application.rb will be like this:

config.autoload_paths += %W(lib)

Recently we faced a problem where we were adding another root level folder app to autoload_paths, and using one of utility classes inside that folder. Everthing worked fine in development, but not in production. We started seeing strange errors while running migrations:

cannot load such file -- app/subdir1/utility_class

Inspecting $LOAD_PATH revealed that there is no absolute path for app folder. Appending autoload_path with absolute path solved this issue.

config.autoload_paths += [ Rails.root.join("app") ]

Profile Ruby 2.0 apps using DTrace - Part 1

Hemant  - April 16, 2013  |   , , ,

This is a series of blog posts we are going to do in tutorial format, this is Part 1 of our tutorial on Dtrace.

DTrace is awesome. If your operating system supports it, using DTrace can give incredible insight into processes that are running on your system. If you are still not convinced about its usefullness you should checkout - How Dtrace helped twitter identify bottlenecks, Profiling Rails startup with Dtrace.

So what exactly is DTrace? According to Wikipedia - “DTrace is a dynamic tracing framework that allows users to tune and troubleshoot applications and the OS itself.” It is available on Solaris, Mac OSX, FreeBSD and recent versions of Oracle Linux. If you are running any of the above operating systems, there is nothing to install - just hop in.

Dtrace allows you to instrument a live process by defining trace points called probes, which can be turned on or off dynamically. The probes fire when a particular condition is met and it requires no change in application code, no special flags and there is very little penalty of running a production app with probes turned on.

What is a probe?

At its core a probe is like a light bulb that switches on when a condition is met. The condition could be entering a C function defined within a module, exiting a C function, entering a system call provided by Kernel etc. A probe is made of 5 parts,

All the processes currently running on operating system (including the kernel itself) have some probes defined. As an example, lets try and list all the probes defined by Ruby 2.0

~> sudo dtrace -l | grep ruby

Ruby 2.0 probes

Running Dtrace

Dtrace probes can be set and run via the command line itself or can be run via Dtrace scripts written in a programming language called D.

Some simple command line examples:

~> sudo dtrace -f write

Above probe will start firing each time any application makes write system call.

Dtrace scripting capabilities

The real power of Dtrace lies in ability to script - tracing of running applications, gathering results and then printing or plotting them.

A DTrace script is essentially made of 3 parts:

probe description
/predicate/ 
{
    action;
}

D language has some similarity with C but at its core it is a scripting language, probably closer to awk etc.

For example, lets write a D script for printing all system calls being made by currently running applications.

~> cat > dtrace_scripts/syscall.d
syscall:::entry 
{
  printf("%s(%d) called %s\n", execname, pid, probefunc)
}

Above script when ran via sudo dtrace -qs syscall.d will start printing name,pid of processes that are making system calls and what system calls they are making. The script does not have predicate, but only probe description.

Probe description

A probe description can be written as:

provider:module:function:probename

Few simple examples:

# A probe that will fire each time a system call called write is entered
syscall::write:entry
# This probe will fire each time someone creates a new ruby array
ruby*::ary_new::

As you can see, you can skip parts of the probe and can use wildcards such as * or ?.

That about sums up content for part 1 of DTrace tutorial. In the next part we are going to cover predicates, aggregates and we will do profiling of real Ruby applications.I will recommend you to play with various probes and stay tuned for Part 2 which is coming up very soon. You should also follow us on twitter @codemancershq if you don’t want to miss out Part 2.

Using Backbone JS in multipage application

Amitava  - April 6, 2013  |   ,

Recently we worked on a cloud resouces management application written on rails for one of our clients. Think about this app as GUI for Amazon AWS. All the resources will be displayed in tabular form and ajax requests for management like creation, updation and deletion of resources. Soon application became more complex because of lots of server side javascript code directly interacting with DOM. As a result testing became more complex, DOM changes used to break javascript code, and tests used to break often. This encouraged us to find a better alternative where we could use javascript in more modular fashion and also test it. Hence we decided to use backbone.js

Since we were already using coffeescript we added the rails-backbone gem as it has generators for creating scaffolds in coffeescript. Initially we were using the generated code but as we became more confident we stopped using the generator. Also we didn’t use the router as this was not a single page app we were building. We were using backbonejs only for pages where we needed to implement client-side filtering of tabular data. This worked really well than we expected. Not only the filtering code was easier to maintain, it became quite easy to test as well. Did we mention we used jasmine for testing all our javascript code :). Later we started to use backbone for forms and other pages as well.

It was all working fine in terms of performance, but then we found a few challenges as the code-base was growing and becoming complex. Most views and models were using similar code and logic so we decided to abstract common behaviour and logic in base classes and subclass our views and models to share the same code. For the entire application we used the following flow -

  1. When the page is loaded it initializes the code to create a ListView.
  2. ListView then fetches the resources through collection and renders in a tabular format.
  3. Each row in the table itself is rendered with a ModelView.
  4. Create and update forms are managed using a FormView.

To begin with we created a ListView class with common features to fetch and dispaly data in a table.

class App.Views.ListView extends Backbone.View
  initialize: (options) ->
    # Here we bind collection events
    @collection.listenTo 'add', 'onAdded'
    @collection.listenTo 'remove', 'onRemoved'

  addAll: () =>
    # Iterate the collection and call `@addOne`

  addOne: (model) =>
    # Render each model inside the table

  renderEmpty: =>
    # Render message if the collection is empty

  fetchCollection: (options)->
    # This is where the data is fetched through a collection

  render: (options) =>
    # Call `@addAll` to start the rendering process
    this

Then we built the ModelView to extract all common features to display a single row and handle events.

class App.Views.ModelView extends Backbone.View
  events: ->
    "click  .edit" : "onEdit"
    "click  .destroy" : "onDestroy"

  initialize: ->
    # Here we bind or initialize model events
    @model.listenTo 'change', 'onChanged'

  onEdit: (e)=>
    # Handle displaying the edit form

  onDestroy: (e)=>
    # Handle deletion of a single model

And finally there is the FormView for which is responsible for rendering the forms for creating and updating resources.

class CloudGui.Views.ModalFormView extends Backbone.View
  # Since we were using twitter-bootstrap modal, we set the dom attributes here.
  attributes:
    class: "modal hide fade"
    tabindex: "-1"
    "data-backdrop": "static"
    "data-keyboard": "false"

  events: ->
    "submit       form" : "onFormSubmit"
    "ajax:success form" : "onFormSuccess"
    "ajax:error   form" : "onFormError"

  onFormSubmit: =>
    # It does client-side validation if required and then submits the
    # form using ajax.

  onFormSuccess: (event, data)=>
    # Here we add the newly created resource to the collection.
    # For update, it updates the model attributes.
    # In both the cases the `ModelView` automatically renders the changes.

  onFormError: (event, xhr, status, error) =>
    # Handles server-side errors

We did not use JSON format for submitting the data since we had to support file uploads. And also the format of input attributes were different than the models. So we decided to use regular ajax form submission where the data is sent using form-url-encoded (or multipart/form-data for file uploads) content-type and the server returning JSON. The rest is then handled by onFormSuccess or onFormError.

One important lesson we learnt is to be careful with handling events in Backbone. It can create memory leaks if the event binding is not done in a correct manner. We were using on to listen to collection or model events but then we found that it does not release the event handlers when the remove function is called on a view object. We used listenTo instead of on and all the events were unbound when the view was removed.

Using backbone.js outside single page application is easy and worked pretty well for us. It enabled us to have proper frontend architecture (rather than jquery callback spaghetti), it made unit testing frontend features easier without resorting to a integration test framework (like Cucumber), we totally recommend this approach if you are consuming bunch of JSON and don’t want to go whole hog single page.

More to follow on other interesting aspects of backbone pagination and real time event handling. Watch out.

Poor man's rspec mode for emacs

Hemant  - March 27, 2013  |   , ,

For whatever reason rspec-mode didn’t work for me. I couldn’t either get it to use current version of Ruby or it used to crap out on Bundler detection.

Annoyed, I decided to write a very simple rspec-mode that works almost always. More importantly it runs rspec from root of current project directory using your current shell and hence you don’t need to configure RVM or rbenv from Emacs. Whatever version of Ruby your shell has it will run using that. As a bonus it runs rspec in a comint buffer that means if you are using pry or a debugger you can interact with it from Emacs itself.

(require 'compile)
;; Find root directory by searching for Gemfile
(defun* get-closest-gemfile-root (&optional (file "Gemfile"))
  (let ((root (expand-file-name "/")))
    (loop 
     for d = default-directory then (expand-file-name ".." d)
     if (file-exists-p (expand-file-name file d))
     return d
     if (equal d root)
     return nil)))

(defun rspec-compile-file ()
  (interactive)
  (compile (format "cd %s;bundle exec rspec %s"
                   (get-closest-gemfile-root)
                   (file-relative-name (buffer-file-name) (get-closest-gemfile-root))
                   ) t))

(defun rspec-compile-on-line ()
  (interactive)
  (compile (format "cd %s;bundle exec rspec %s -l %s"
                   (get-closest-gemfile-root)
                   (file-relative-name (buffer-file-name) (get-closest-gemfile-root))
                   (line-number-at-pos)
                   ) t))

(add-hook 'enh-ruby-mode-hook
          (lambda ()
            (local-set-key (kbd "C-c l") 'rspec-compile-on-line)
            (local-set-key (kbd "C-c k") 'rspec-compile-file)
            ))

An astute reader will notice that I am not using default ruby-mode. You can change the hook to ruby-mode-hook if you are using default ruby-mode. So last few lines will become:

(add-hook 'ruby-mode-hook
          (lambda ()
            (local-set-key (kbd "C-c l") 'rspec-compile-on-line)
            (local-set-key (kbd "C-c k") 'rspec-compile-file)
            ))

Bundler: Cleanup old unused gems

Yuva  - March 27, 2013  |   , ,

Recently I resumed an old project {after 2 months}. My peers are still working on it and they have upgraded lots of gems. After running bundle install, many gems got upgraded and old version were still hanging around in vendor folder. This little handy command helped in removing them:

bundle clean -V

NOTE: Btw, I always install gems in ./vendor/ruby folder, and don’t maintain any rvm gemsets for projects.

The kitchen sink libraries

Hemant  - May 8, 2012  |   , ,

There was a time when Ruby libraries where like small Unix utilities, they did one thing and they did it well. If you go read code of some of the older plugins, it will be quite simple and straightforward affair.

Today however trend is, to write kitchen sink libraries that do a lot. You want file upload, they work at Rack level and support not only Rails but all the rack complaint frameworks under the sun, not only relational databases but Mongo as well, not just local file storage but S3/Cloudfiles as well, not just file uploads but some kind of caching via Rack::Cache as well.

Now at its core, handling file uploads is very simple and straightforward affair. Uploading to S3 or Cloudfiles is even simpler. I daresay, getting started with the kitchen sink library will be easy and simple as well. But if one thing Law of leaky Abstractions has taught us that is - eventually you will have look under the hood of these kitchen sink libraries and suddently MysqlAdapter which subclasses AbstractPersistenceAdapter won’t appear very attractive, because your application doesn’t gives two hoots about Mongo support. Jumping through hooves to grab that uploaded file via Rack will appear waste of time as well because you are probably never going to need this working on framework other than Rails. And while support for Rack::Cache was nice, you can’t use it in production anyways.

I don’t know about you, but I prefer gems and plugins that I include in my code easily grokkable. Below is a time bomb taken from a popular file upload gem and good luck - if you will have to grok this code and fix something yourself.

autoload_files_in_dir = proc do |path, namespace|
  # Define the module
  eval("module #{namespace}; end")
  # Autoload modules/classes in that module
  Dir.glob("#{path}/*.rb").each do |file|
    file = File.expand_path(file)
    sub_const_name = camelize[ File.basename(file, '.rb') ]
    eval("#{namespace}.autoload('#{sub_const_name}', '#{file}')")
  end
  # Recurse on subdirectories
  Dir.glob("#{path}/*/").each do |dir|
    sub_namespace = camelize[ File.basename(dir) ]
    autoload_files_in_dir[dir, "#{namespace}::#{sub_namespace}"]
  end
end