DIY Ruby CPU profiling - Part IV

Emil Soman  - July 18, 2015  |   , ,

In Part I we saw how CPU profilers work, in Part II we learned how to measure execution time of programs using CPU time and Wall time and in Part III we learned how to use Ruby’s TracePoint API to trace method calls and returns in Ruby VM. We also started building our DiyProf gem. It’s recommended to read those first before you continue. In this part we’re going to create a simple call graph visualization using the data we collected using the TracePoint API.

Part IV. Call Graphs

In Part III, we used Ruby’s TracePoint API to print method call and return traces. It’d be more useful if we can parse these traces and create a call graph out of it. A call graph is basically a visual representation of the method calls, in the form of a tree. It looks like this :

Call tree

This graph tells us that outer_method calls 3 methods from inside it: inner_method{1,2,3} and inner_method_3 calls another method named inner_most_method.

Graphviz and the DOT language

Graphviz is a graph visualization tool which can take a graph description in text format and generate diagrams as PDF and other formats. For describing graphs in text format, Graphviz uses a language called DOT, which is a simple declarative way of describing nodes and connections between them.

Before proceeding, make sure you’ve installed graphviz and the dot command is available in your $PATH. Follow their official documentation for installation steps.

Now let’s take an example.

digraph graph_name {
  a -> b -> c;
  b -> d;
}

Save the above lines as sample.dot and run the following command:

dot -Tpdf sample.dot -o graph.pdf

The above command will generate a directed graph that looks like this:

Directed graph example

There’s another way to represent graphs, by separating nodes and connections like so:

digraph graph_name {
  <nodes>
  <connections>
}

So the following DOT notation represents the same graph:

digraph graph_name {
  a;
  b;
  c;
  d;
  a -> b;
  b -> c;
  b -> d;
}

It’s also possible to add additional labels to nodes and connections.

digraph graph_name {
  a [label="a \n foo: bar"];
  b [label="b \n foo: baz"]
  a -> b [label="xyz"];
}

Above DOT code represents the following graph:

Graph with labels

Call Graph with Graphviz

We’re going to create a reporter class for our DiyProf gem which will keep collecting all method calls and create a DOT file that represents the call graph.

We’ll add a record method under the DotReporter class which pushes method infos into a stack when calls happen and pop them out of the stack when the methods return. When the method info is popped from the stack, the time it spent in the stack is recorded(which is the execution time) and it adds the method’s call info(method name and time) to a call tree. See the code below:

module DiyProf
  # Struct that will hold a method name and a timestamp
  CallInfo = Struct.new(:name, :time)

  class DotReporter
    def initialize
      # A stack for pushing/popping methods when methods get called/returned
      @call_stack = []
      # Nodes for all methods
      @methods = {}
      # Connections between the nodes
      @calls = {}
    end

    def record(event, method_name, time)
      case event
      when :call
        @call_stack << CallInfo.new(method_name, time)
      when :return
        # Return cannot be the first event in the call stack
        return if @call_stack.empty?

        method = @call_stack.pop
        # Set execution time of method in call info
        method.time = time - method.time
        add_method_to_call_tree(method)
      end
    end
  end
end

add_method_to_call_tree adds nodes and connections for the added method. Here’s the code that does that:

# Struct that holds the time spent in a method
# and the total number of times a method was called
MethodInfo = Struct.new(:count, :total_time, :self_time)

def add_method_to_call_tree(method)
  # Add method as a node to the call graph
  @methods[method.name] ||= MethodInfo.new(0, 0, 0)
  # Update total time(spent inside the method and methods called inside this method)
  @methods[method.name].total_time += method.time
  # Update self time(spent inside the method and not methods called inside this method)
  # This will be subtracted when children are added to the graph
  @methods[method.name].self_time += method.time
  # Update total no of times the method was called
  @methods[method.name].count += 1

  # If the method has a parent in the call stack
  # Add a connection from the parent node to this method
  if parent = @call_stack.last
    @calls[parent.name] ||= {}
    @calls[parent.name][method.name] ||= 0
    @calls[parent.name][method.name] += 1

    # Take away self time of parent
    @methods[parent.name] ||= MethodInfo.new(0, 0, 0)
    @methods[parent.name].self_time -= method.time
  end
end

Now that we have the nodes and connections in @methods and @calls, it’ll be trivial to generate a DOT file that looks like this:

digraph G {
  <graph_nodes>
  <graph_links>
}

Here’s the code that will walk through @methods and @calls and prepare such a DOT file:

def dot_notation
  dot = %Q(
    digraph G {
      #{graph_nodes}
      #{graph_links}
    }
  )
end

def graph_nodes
  nodes = ""
  @methods.each do |name, method_info|
    nodes << "#{name} [label=\"#{name}\\ncalls: #{method_info.count}\\ntotal time: #{method_info.total_time}\\nself time: #{method_info.self_time}\"];\n"
  end
  nodes
end

def graph_links
  links = ""
  @calls.each do |parent, children|
    children.each do |child, count|
      links << "#{parent} -> #{child} [label=\" #{count}\"];\n"
    end
  end
  links
end

Check out the lib/diy_prof/dot_reporter.rb file on the github repo to see how all of this will fit together. We also need some changes in the Tracer class we wrote in Part III to make use of a custom reporter class:

module DiyProf
  class Tracer
    # We'll need to initialize the Tracer with DotReporter.new
    def initialize(reporter)
      @reporter = reporter
      @tracepoints = [:call, :return].collect do |event|
        TracePoint.new(event) do |trace|
          # Use the reporter instead of just printing the traces
          reporter.record(event, trace.method_id, cpu_time)
        end
      end
    end

    ...

    # New method added to collect the report after profiling
    def result
      @reporter.result
    end
  end
end

Now that our tracer is able to generate a DOT formatted report, we can use it on our sample program we wrote in the previous parts:

def main
  3.times do
    find_many_square_roots
    find_many_squares
  end
end

def find_many_square_roots
  5000.times{|i| Math.sqrt(i)}
end

def find_many_squares
  5000.times{|i| i**2 }
end

DiyProf.start_profiling
main
result = DiyProf.stop_profiling

require 'tempfile'
f = Tempfile.open('dot')
f.write result
f.close
system("dot -Tpdf #{f.path} -o call_graph.pdf")

We should get a call graph that looks like this:

Demo

Congratulations if you got this far! Check out the github repo which has the complete code if you got stuck somewhere.

Bonus!

Wouldn’t it be cool if we could vary the size of the nodes based on self or total time of the method? The DOT syntax accepts a bunch of attributes for nodes and edges. Looks like we can use size and fontsize attributes to vary the size of the nodes and the text inside the nodes. I’ll leave it to you to think of how to use these to make the most expensive methods stand out in the call graph.

Recap

We’ve learned about Graphviz and the DOT language and how to use them to create call graph diagrams out of the data collected by Ruby Tracepoint API. In Part V, we’ll start building a sampling profiler for Ruby, in particular we’ll learn some basics about interrupts and signal handling in C. Thanks for reading! If you would like to get updates about subsequent blog posts in this DIY CPU profiler series, do follows us on twitter @codemancershq.

DIY Ruby CPU profiling - Part III

Emil Soman  - April 15, 2015  |   , ,

In Part I we saw how instrumentation profilers work, and in Part II we learned how to measure execution time of programs using CPU time and Wall time. It’s recommended to read those first before you continue. In this part we’ll apply what we have learned so far and build a very basic instrumentation CPU profiler.

Part III. DIY Instrumentation CPU Profiler

Instrumentation in Ruby

In Part I, we learned that instrumentation profiling tools make use of hooks which can either be installed by the profiling tools or provided by the language itself to do their job. Luckily, Ruby already has these hooks baked into it and gives us the nifty TracePoint class to make use of these hooks.

TracePoint API

Ruby VM emits a bunch of events at various points while executing your code. Ruby allows you to enable callbacks for these events using the TracePoint class. The API documentation of TracePoint has a list of all the events you could listen to, but we’re interested in only two of those for now:

  1. :call , which is triggered when a ruby method gets called
  2. :return, which is triggered when a ruby method returns.

We can use the TracePoint.new method to create a tracepoint object and pass it an event type to listen to and a block of code which should get executed when the event gets triggered. Here’s the code to create a tracepoint for :call and :return events:

trace = TracePoint.new(:call, :return) do |trace|
  puts "Method: #{trace.method_id}, event: #{trace.event}"
end

The argument trace which is passed into the callback code block gives you access to a bunch of tracepoint attributes which can tell you more about the event that got triggered. In our example, we’re going to need the method_id attribute which gives us the method name and the event attribute which gives us the name of the event which gets triggered. The callback code block above will print the method name and the type of the triggered event (call/return), every time a Ruby method gets called or returned.

After you create the tracepoint, you need to enable it:

trace.enable

Once the tracepoint is enabled, we can create a method and call it, and see if the callback that we registered gets executed:

def hello
end

hello

#=> Method: hello, event: call
#=> Method: hello, event: return

Great, our callback got executed twice as expected.

diy_prof gem

It’s finally time to start putting together all that we’ve learned and build a RubyGem. We’re going to call this gem diy_prof. The source for this gem is on GitHub and you can find it here.

Now let’s build the gem from scratch using bundler:

bundle gem diy_prof

This creates a skeleton project which we’ll build upon. Next, we’ll create a TimeHelpers module which will contain the methods cpu_time and wall_time which we wrote in Part II:

# lib/diy_prof/time_helpers.rb

module DiyProf::TimeHelpers
  # These methods make use of `clock_gettime` method introduced in Ruby 2.1
  # to measure CPU time and Wall clock time.

  def cpu_time
    Process.clock_gettime(Process::CLOCK_PROCESS_CPUTIME_ID, :microsecond)
  end

  def wall_time
    Process.clock_gettime(Process::CLOCK_MONOTONIC, :microsecond)
  end
end

We also need to require this file in our gem’s main file lib/diy_prof.rb:

# lib/diy_prof.rb

require 'diy_prof/version'
require 'diy_prof/time_helpers'
# Rest of the original file ..

Next, we’ll use the TracePoint API in our gem to print the time (using our TimeHelpers) when a :call or :return event occurs.

# lib/diy_prof/tracer.rb


module DiyProf
  class Tracer
    include TimeHelpers

    def initialize(clock_type: :cpu)
      @tracepoint = TracePoint.new(:call, :return) do |trace|
        time = clock_type == :wall ? wall_time : cpu_time
        printf("%-20s:%-20s%-20s\n", time, trace.event, trace.method_id)
      end
    end
    def enable
      @tracepoint.enable
    end

    def disable
      @tracepoint.disable
    end
  end
end

Also, we’ll require this file from our gem’s main file too:

# lib/diy_prof.rb

require 'diy_prof/version'
require 'diy_prof/time_helpers'
require 'diy_prof/tracer'
# Rest of the original file ..

Now let’s write an example script to test what we’ve built so far. We’ll use the same example we saw in Part I:

# examples/demo.rb

$:<< File.join(File.dirname(__FILE__), "../lib")
require 'diy_prof'

### Begin sample program ###
def main
  3.times do
    find_many_square_roots
    find_many_squares
  end
end

def find_many_square_roots
  5000.times{|i| Math.sqrt(i)}
end

def find_many_squares
  5000.times{|i| i**2 }
end
### End sample program ###

tracer = DiyProf::Tracer.new(clock_type: :cpu)
tracer.enable
main
tracer.disable

If we run the above Ruby program, we should get the following output:

70508               :return              enable
70587               :call                main
70604               :call                find_many_square_roots
73225               :return              find_many_square_roots
73256               :call                find_many_squares
74527               :return              find_many_squares
74562               :call                find_many_square_roots
77042               :return              find_many_square_roots
77064               :call                find_many_squares
77653               :return              find_many_squares
77690               :call                find_many_square_roots
79681               :return              find_many_square_roots
79698               :call                find_many_squares
80288               :return              find_many_squares
80306               :return              main
80326               :call                disable

The first column of the output is the CPU time of the triggered events, the second column is the name of the event that got triggered(call/return) and the third column is the name of the method which got called or returned. This is similar to the output we saw in Part I where we learned how instrumentation profilers work. As you can see, we can easily recreate a call stack using this output because we know which method called which other methods. We also have the time of all call and return events, so we can also tell how much time was spent in each method. It wouldn’t be too difficult to write a script that parses this output and prints a list of methods ordered by their execution them. But that’s too boring, we can build something way more interesting than that. Watch out for the next part to learn what else we can do with this data.

Recap

We’ve learned about Ruby’s TracePoint API and how to use it to listen to method calls and returns. We’ve also taken the code we’ve written so far and put everything together to build a gem which allows you to print the execution trace of your Ruby program. In Part 4 we’ll learn how to make sense of the data that our gem collects and build some cool call graphs out of it. Thanks for reading! If you would like to get updates about subsequent blog posts in this DIY CPU profiler series, do follows us on twitter @codemancershq.

DIY Ruby CPU profiling - Part II

Emil Soman  - March 12, 2015  |   , ,

In Part I we learned what CPU profiling means and also about the two modes of CPU profiling. In this part we’re going to explore CPU time and Wall time, the units used to measure execution cost. We’ll also write some code to get these measurements as the first step towards building our CPU profiler.

Part II. CPU time and Wall time

Wall time

Wall time is just the regular real world time that elapses between a method call and return. So if you were to measure the “Wall clock time” taken for a method to run, it would be theoretically possible to do so with a stopwatch. Just start the stopwatch when the method starts, and stop when the method returns. This is also called real time.

One important point about wall time is that it’s unpredictable and you may get different results every time you try to measure the same piece of code. This is because wall time is affected by the number of processes running in the background. When the CPU has to work on a bunch of processes at the same time, the operating system does a scheduling of the processes which are running at the time and tries to give a fair share of CPU to each of them. This means the total time spent by the CPU is divided into many slices and our method gets only some of these slices and not all of them. So while the wall clock ticks away, our process may be sitting idle and giving way for other processes which are running in parallel. This means the time spent on other processes will add to our Wall time too!

CPU time

CPU time is the time for which the CPU is dedicated to run the method. CPU time is measured in terms of CPU cycles(or ticks) which are used to execute the method. We can convert this into time if we know the frequency of the CPU in units of cycles per second aka Hertz. So if the CPU took x ticks to execute a method, and the frequency of the CPU is y hertz, the time taken by the CPU to execute the method = x/y seconds. Sometimes the OS does this conversion for us so we don’t have to do this calculation by ourselves.

CPU time will not be equal to Wall time. The difference depends on the type of instructions in our method. We can categorize the instructions into broadly 2 types: CPU bound and I/O bound. When I/O instructions are being executed, the CPU becomes idle and can move on to process other CPU bound instructions. So if our method has a time consuming I/O instruction, the CPU stops spending time on our method and moves on to something else until the I/O operation is completed. During this time the Wall time keeps ticking but the CPU time stops and lags behind Wall time.

Let’s say a very slow running method took 5 minutes on your clock to finish running. If you were to ask how much time was spent on the method, your wall clock would say “It took 5 minutes to run this method”, but the CPU would say “I spent 3 minutes of my time on this method”. So who are you going to listen to? Which time more accurately measures the cost of executing the method?

The answer is, it depends™. It depends on the kind of method you want to measure. If the method spends most of its time doing I/O operations, or it doesn’t deal with CPU bound instructions directly, the cost of execution depicted by CPU time is going to be grossly inaccurate. For these types of methods, it makes more sense to use Wall time as the measurement. For all other cases, it’s safe to stick with CPU time.

Measuring CPU time and Wall time

Since we’re going to write a CPU profiler, we’ll need a way to measure CPU time and wall time. Let’s take a look at the code in Ruby’s Benchmark module which already measures CPU time and Wall time.

def measure(label = "") # :yield:
  t0, r0 = Process.times, Process.clock_gettime(BENCHMARK_CLOCK)
  yield
  t1, r1 = Process.times, Process.clock_gettime(BENCHMARK_CLOCK)
  Benchmark::Tms.new(t1.utime  - t0.utime,
                     t1.stime  - t0.stime,
                     t1.cutime - t0.cutime,
                     t1.cstime - t0.cstime,
                     r1 - r0,
                     label)
end

So Ruby uses 2 methods from the Process class to measure time :

  1. times to measure CPU time
  2. clock_gettime to measure real time aka Wall time

But unfortunately the resolution of time returned by the times method is 1 second which means if we use times to measure CPU time in our profiler, we’ll only be able to profile methods that take a few seconds to complete. clock_gettime looks interesting, though.

clock_gettime

Process::clock_gettime is a method added in Ruby 2.1 and it uses POSIX clock_gettime() function and falls back to OS specific emulations to get the value of time in case clock_gettime is not available in the OS or when the type of clock we are measuring with clock_gettime is not implemented in the OS. This function accepts a clock_id and the time resolution as arguments. There are a bunch of clock_ids you could pass in to pick the kind of clock to use, but the ones we’re interested in are:

  1. CLOCK_MONOTONIC: This clock measures the elapsed wall clock time since an arbitrary point in the past and is not affected by changes in the system clock. Perfect for measuring Wall time.
  2. CLOCK_PROCESS_CPUTIME_ID: This clock measures per-process CPU time, ie the time consumed by all threads in the process. We can use this to measure CPU time.

Let’s make use of this and write some code:

module DiyProf
  # These methods make use of `clock_gettime` method introduced in Ruby 2.1
  # to measure CPU time and Wall clock time.

  def self.cpu_time
    Process.clock_gettime(Process::CLOCK_PROCESS_CPUTIME_ID, :microsecond)
  end

  def self.wall_time
    Process.clock_gettime(Process::CLOCK_MONOTONIC, :microsecond)
  end
end

We could use these methods to benchmark code:

puts "****CPU Bound****"
c1, w1 = DiyProf::cpu_time, DiyProf::wall_time
10000.times do |i|
  Math.sqrt(i)
end
c2, w2 = DiyProf::cpu_time, DiyProf::wall_time
puts "CPU time\t=\t#{c2-c1}\nWall time\t=\t#{w2-w1}"

puts "\n****IO Bound****"
require 'tempfile'

c1, w1 = DiyProf::cpu_time, DiyProf::wall_time
1000.times do |i|
  Tempfile.create('file') do |f|
    f.puts(i)
  end
end
c2, w2 = DiyProf::cpu_time, DiyProf::wall_time
puts "CPU time\t=\t#{c2-c1}\nWall time\t=\t#{w2-w1}"

Running this code would give an output similar to this:

****CPU Bound****
CPU time	=	5038
Wall time	=	5142

****IO Bound****
CPU time	=	337898
Wall time	=	475864

This clearly shows that on a single CPU core, CPU time and Wall time are nearly equal when running purely CPU bound instructions whereas CPU time is always less that Wall time when running I/O bound instructions.

Recap

We’ve learned what CPU time and Wall time mean, their differences, and when to use which. We also wrote some Ruby code to measure CPU time and Wall time which would help us measure time in the CPU profiler we’re building. In part 3 we’ll take a look at Ruby’s tracepoint API and use it to build an instrumentation profiler. Thanks for reading! If you would like to get updates about subsequent blog posts in this DIY CPU profiler series, do follows us on twitter @codemancershq.

DIY Ruby CPU profiling - Part I

Emil Soman  - March 6, 2015  |   , ,

At Codemancers, we’re building Rbkit, a fresh code profiler for the Ruby language with tonnes of cool features. I’m currently working on implementing a CPU profiler inside rbkit gem which would help rbkit UI to reconstruct the call graph of the profiled ruby process and draw useful visualizations on the screen. I learned a bunch of new things along the way and I’d love to share them with you in this series of blog posts.

We’re going to start from the fundamentals and step by step, we’re going to write a rudimentary CPU profiler for Ruby ourselves! By the time we finish we would learn :

Part I. An Introduction to CPU Profiling

By doing a CPU profile of your program, you can find out how expensive your program is with respect to CPU usage. In order to profile your program, you’ll need to use a profiling tool and follow the following steps :

  1. Start CPU profiling
  2. Execute the code you want to profile
  3. Stop CPU profiling and get profiling result
  4. Analyze result

By analyzing the profiling result, you can find the bottlenecks which slow down your whole program.

Profiling modes

CPU profiling is done in broadly 2 ways:

1. Instrumentation

In this mode, the profiling tool makes use of some hooks, either provided by the interpreter or inserted into the program, to understand the call graph and measure the execution time of each method in the call graph.

For example, consider the following piece of Ruby code :

def main
  3.times do
    find_many_square_roots
    find_many_squares
  end
end

def find_many_square_roots
  5000.times{|i| Math.sqrt(i)}
end

def find_many_squares
  5000.times{|i| i**2 }
end

main

I’ve inserted some comments to help understand how the hooks would get executed if the Ruby interpreter gave us method call and return hooks:

def main
  # method call hook gets executed
  3.times do
    find_many_square_roots
    find_many_squares
  end
  # method end hook gets executed
end

def find_many_square_roots
  # method call hook gets executed
  5000.times{|i| Math.sqrt(i)}
  # method end hook gets executed
end

def find_many_squares
  # method call hook gets executed
  5000.times{|i| i**2 }
  # method end hook gets executed
end

main

Now if we could print the current time and the name of the current method inside these hooks, we’d get an output which looks somewhat like this :

sec:00 usec:201007	called  	main
sec:00 usec:201108	called  	find_many_square_roots
sec:00 usec:692123	returned	find_many_square_roots
sec:00 usec:692178	called  	find_many_squares
sec:00 usec:846540	returned	find_many_squares
sec:00 usec:846594	called  	find_many_square_roots
sec:01 usec:336166	returned	find_many_square_roots
sec:01 usec:336215	called  	find_many_squares
sec:01 usec:484880	returned	find_many_squares
sec:01 usec:484945	called  	find_many_square_roots
sec:01 usec:959254	returned	find_many_square_roots
sec:01 usec:959315	called  	find_many_squares
sec:02 usec:106474	returned	find_many_squares
sec:02 usec:106526	returned	main

As you can see, this output can tell us how much time was spent inside each method. It also tells us how many times each method was called. This is roughly how instrumentation profiling works.

Pros :

Cons :

2. Sampling

In this mode of profiling, the profiler interrupts the program execution once in every x unit of time, and takes a peek into the call stack and records what it sees(called a “sample”). Once the program finishes running, the profiler collects all the samples and finds out the number of times each method appears across all the samples.

Hard to visualize? Let’s look at the same example code and see how different the output would be, if we used a sampling profiler.

Output from a sampling profiler would look like this :

Call stack at 0.5sec: main/find_many_square_roots
Call stack at 1.0sec: main/find_many_square_roots
Call stack at 1.5sec: main/find_many_square_roots
Call stack at 2.0sec: main/find_many_squares

In this example, the process was interrupted every 0.5 second and the call stack was recorded. Thus we got 4 samples over the lifetime of the program and out of those 4 samples, find_many_square_roots is present in 3 and find_many_squares is present in only one sample. From this sampling, we say that find_many_square_roots took 75% of the CPU where as find_many_squares took only 25%. That’s roughly how sampling profilers work.

Pros :

Cons :

Recap

We just looked into what CPU profiling means and the 2 common strategies of CPU profiling. In part 2, we’ll explore the 2 units of measuring CPU usage - CPU time and Wall time. We’ll also get our hands dirty and write some code to get these measurements. Thanks for reading!

An Introduction to JSON Schema

Kashyap  - April 5, 2014  |   , ,

JSON, or JavaScript Object Notation has become the most widely used serialization and transport mechanism for information across various web-services. From it’s initial conception, the format garnered swift and wide appreciation for being really simple and non-verbose.

Lets say you want to consume the following JSON object via an API:

{
  id: 3232,
  name: "Kashyap",
  email: "kashyap@example.com"
  contact: {
    id: 123,
    address1: "Shire",
    zipcode: LSQ424
  }
}

Now, let’s assume that you want to ensure that before consuming this data, email and contact.zipcode must be present in the JSON. If that data is not present, you shouldn’t be using it. The typical way is to check for presence of those fields but this whack-a-mole quickly gets tiresome.

Similarly, lets say you are an API provider and you want to let your API users know the basic structure to which data is going to conform to, so that your API users can automatically test validity of data.

If you ever had to deal with above two problems, you should be using JSON schemas.

What’s a Schema?

A schema is defined in Wikipedia as a way to define the structure, content, and to some extent, the semantics of XML documents; which probably is the simplest way one could explain it. For every element — or node — in a document, a rule is given to which it needs to conform. Having constraints defined at this level will make it unnecessary to handle the edge cases in the application logic. This is a pretty powerful tool. This was missing from the original JSON specification but efforts were made to design one later on.

Why do we need a Schema?

If you’re familiar with HTML, the doctype declaration on the first line is a schema declaration. (Specific to HTML 4 and below.)

HTML 4 Transitional DOCTYPE declaration:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">

This line declares that the rest of the document conforms to the directives specified at the url http://www.w3.org/TR/html4/loose.dtd. That means, if you declare the document as strict, then the usage of any new elements like <sp></sp> will cause the page to display nothing. In other words, if you make a typo or forget to close a tag somewhere, then the page will not get rendered and your users will end up with a blank page.

At first glance, this looks like a pain — and it is, actually. That’s part of the reason why this was abandoned altogether in the newer version of HTML. However, HTML is not really a good use case for a schema. Having a well-defined schema upfront helps in validating user input at the language/protocol level than at the application’s implementation level. Let’s see how defining a schema makes it easy to handle user input errors.

JSON Schema

The JSON Schema specification is divided into three parts:

  1. JSON Schema Core: The JSON Schema Core specification is where the terminology for a schema is defined. Technically, this is simply the JSON spec with the only addition being definition of a new media type of application/schema+json. Oh! a more important contribution of this document is the $schema keyword which is used to identify the version of the schema and the location of a resource that defines a schema. This is analogous to the DOCTYPE declaration in the HTML 4.01 and other older HTML versions.

    The versions of the schema separate changes in the keywords and the general structure of a schema document. The resource of a schema is usually a webpage which provides a JSON object that defines a specification. Confused? Go open up the url http://www.w3.org/TR/html4/loose.dtd which I’m linking to here in a browser and go through the contents. This is the specification of HTML 4.01 Loose API. Tags like ENTITY, ELEMENT, ATTLIST are used to define the accepted elements, entities and attributes for a valid HTML document.

    Similarly, the JSON Schema Core resource URL (downloads the schema document) defines a superset of constraints.

  2. JSON Schema Validation: The JSON Schema Validation specification is the document that defines the valid ways to define validation constraints. This document also defines a set of keywords that can be used to specify validations for a JSON API. For example, keywords like multipleOf, maxLength, minLength etc. are defined in this specification. In the examples that follow, we will be using some of these keywords.

  3. JSON Hyper-Schema: This is another extension of the JSON Schema spec, where-in, the hyperlink and hypermedia-related keywords are defined. For example, consider the case of a globally available avatar (or, Gravatar). Every Gravatar is composed of three different components:

    1. A Picture ID,
    2. A Link to the picture,
    3. Details of the User (name and email ID).

    When we query the API provided by Gravatar, we get a reponse typically having this data encoded as JSON. This JSON response will not download the entire image but will have a link to the image. Let’s look at a JSON representation of a fake profile I’ve setup on Gravatar:

    {
      "entry":[{
        "id":"61443191",
        "hash":"756b5a91c931f6177e2ca3f3687298db",
        "requestHash":"756b5a91c931f6177e2ca3f3687298db",
        "profileUrl":"http:\/\/gravatar.com\/jsonguerilla",
        "preferredUsername":"jsonguerilla",
        "thumbnailUrl":"http:\/\/1.gravatar.com\/avatar\/756b5a91c931f6177e2ca3f3687298db",
        "photos":[{
          "value":"http:\/\/1.gravatar.com\/avatar\/756b5a91c931f6177e2ca3f3687298db",
          "type":"thumbnail"
        }],
        "name":{
          "givenName":"JSON",
          "familyName":"Schema",
          "formatted":"JSON Schema Blogpost"
        },
        "displayName":"jsonguerilla",
        "urls":[]
      }]
    }

    In this JSON response, the images are represented by hyperlinks but they are encoded as strings. Although this example is for a JSON object returned from a server, this is how traditional APIs handle input as well. This is due to the fact that JSON natively does not provide a way to handle hyperlinks; they are only Strings.

    JSON hyperschema attempts to specify a way to have a more semantic way of representing hyperlinks and images. It does this by defining keywords (as JSON properties) such as links, rel, href. Note that this specification does not try to re-define these words in general (as they are defined in HTTP protocol already) but it tries to normalize the way those keywords are used in JSON.

Drafts

The schema is still under development and the progress can be tracked by comparing the versions known as “drafts”. Currently, the schema is in the 4th version. The validation keywords can be dropped or added between versions. This article — and many more over the interwebs — refer to the 4th version of the draft.

Usage

Let’s build a basic JSON API that accepts the following data with some constraints:

  1. A post ID. This is a number and is a required parameter.
  2. Some free-form text with an attribute of body. This is a required parameter.
  3. A list of tags with an attribute of ‘tags’. Our paranoid API cannot accept more than 6 tags though. This is a required parameter.
  4. An optional list of hyperlinks with an attribute of ‘references’

Let’s face it, almost every app you might’ve ever written must’ve had some or the other constraints. We end up repeating the same verification logic everytime. Let’s see how we can simplify that.

We will be using Sinatra for building the API. This is the basic structure of our app.rb:

require 'sinatra'
require 'sinatra/json'
require 'json-schema'

post '/' do
end

The Gemfile:

gem 'sinatra'
gem 'sinatra-contrib'
gem 'json-schema'

We will be using the JSON-Schema gem for the app. Let’s look at the schema that we will define in a schema.json file:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "required": [ "id", "body", "tags" ],
  "properties": {
    "id": {
      "type": "integer"
    },

    "body": {
      "type": "string"
    },

    "tags": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "maxItems": 6
    },

    "references": {
      "type": "array",
      "items": {
        "type": "string",
        "format": "uri"
      }
    }
  }
}
  1. The properties attribute holds the main chunk of the schema definition. This is the attribute under which each of the individual API attribute is explained in the form of a schema of it’s own.
  2. The required attribute takes in a list of strings that mention which of the API parameters are required. If any of these parameters is missing from the JSON input to our app, an error will be logged and the input won’t get validated.
  3. The type keyword specifies the schema type for that particular block. So, at the first level, we say it’s an object (analogous to a Ruby Hash). For the body, tags and references, the types are string, array and array respectively.
  4. In case an API parameter can accept an array, the items inside that array can be explained by a schema definition of their own. This is done by using an items attribute and defining how each of the item in the array should be validated.
  5. The format attribute is a built-in format for validation in the JSON Schema specification. This alleviates the pain of adding regex for validating common items like uri, ip4, ip6, email, date-time and hostname. That’s right, no more copy-pasting URI validation regexes from StackOverflow.
  6. The $schema attribute is a non-mandatory attribute that specifies the type of the schema being used. For our example, we will be using the draft#4 of the JSON Schema spec.

To use this schema in our app, we will create a helper method that uses validates the input with the schema we just defined. The json-schema gem provides three methods for validation — a validate method that returns either true or false, a validate! that raises an exception when validation of an attribute fails and a fully_validate method that builds up an array of errors similar to what Rails’ ActiveRecord#save method provides.

We will be using the JSON::Validator.fully_validate method in our app and return a nicely formatted JSON response to the user if the validation fails.

helpers do
  def validate(json_string_or_hash)
    JSON::Validator.fully_validate('schema.json', json_string_or_hash)
  end
end

Now, we can use this helper inside routes to check the validity of the input JSON like so:

post '/' do
  input = JSON.load(request.body.read)
  errors = validate(input)

  if errors.empty?
    json({ message: "The blog post has been saved!" })
  else
    status 400
    json({ errors: a })
  end
end

If the input is valid, the errors object will be empty. Otherwise, it will hold a list of errors. This object will be returned as a JSON response with the appropriate HTTP status code. For example, if we run this app and send in a request with a missing id parameter, the response will be something similar to the following:

[
  "The property '#/' did not contain a required property of 'id' in
  schema schema.json#"
]

Let’s say if we send in a request with id having a string parameter. The errors object will hold the following:

[
  "The property '#/id' of type String did not match the following type:
  integer in schema schema.json#"
]

Last example. Let’s try sending a references parameter with a malformed URI. We will send the following request:

{
  "id": 1,
  "body": "Hello, Universe",
  "tags": ["start", "first"],
  "references": [ "data:image/svg+xml;base64 C==" ]
}

(This input is in the file not_working_wrong_uri.txt)

curl \
  -d @not_working_wrong_uri.txt
  -H 'Content-Type: application/json' \
  http://localhost:4567

The output of this would be:

[
  "The property '#/references/0' must be a valid URI in schema
  schema.json#"
]

Thus, with a really simple validation library and a standard that library implementers in different languages use, we can achieve input validation with a really simple setup. One really great advantage of following a schema standard is that we can be sure about the basic implementation no matter what the language which might implment the schema. For example, we can use the same schema.json description with a JavaScript library for validating the user input — for example, in the front-end of the API we’ve just built.

Summary

The full app, some sample input files are present in this repo. The json-schema gem is not yet official and might have some unfinished components — For example, the format validations of hostname and email for a string type have not been implemented yet — and the JSON Schema specification itself is under constant revisions. But that doesn’t mean it’s not ready for usage. Few of our developers use the gem in one of our projects and are pretty happy with it. Try out the gem and go through the specfication to gain an idea of why this would be beneficial yourself.

More Reading

  1. Understanding JSON Schema
  2. JSON Schema Documentation
  3. This excellent article by David Walsh
  4. JSON Schema Example: This example uses more keywords that weren’t discussed in this post. For example, title and description.

Random Ruby tips from trenches #1

Hemant  - January 7, 2014  |   , ,

  1. Rubocop and Flycheck :

    Flycheck is a emacs mode which helps us with IDE like warnings in Emacs. I am already using enh-ruby-mode which helps with some of the syntax errors and stuff, but what is nice about flycheck is it integrates with rubocop and shows rubocop errors in-place in the editor.

    A picture is worth thousand words so:

    rubocop with flycheck

  2. pry ––gem:

    pry --gem opens a pry session with ./lib added to $LOAD_PATH and ‘require’s the gem. A good shortcut while working on gems and you want a quick console with the gem loaded.

  3. ruby –S :

    This can be used for running binaries which are present in local directory. For example, if you are working on bundler and want to run bundle command with local version of bundler rather than one installed globally you can use:

       ruby -S ./bin/bundle -I ./lib/ install
    

    The advantages are:

    • ruby -S allows you to ignore #!/usr/bin/env ruby line and even if current version of ruby is X using ruby -S you can run the command with another version of Ruby. Especially useful for running scripts with JRuby, Rbx etc.

    • Based on RUBYPATH environment variable running a script via ruby -S (or jruby -S) allows PATH environment to be modified. For example running jruby -S gem does not run gem which is in current path, but it runs gem command provided by JRuby because JRuby defines different RUBYPATH.

  4. Faster rbenv load :

    If you are using rbenv and there is a lag while opening a new shell, consider updating the rbenv initializing line in your shell profile to:

     eval "$(rbenv init - --no-rehash)"
    

    The --no-rehash flag tells rbenv to skip automatic rehash when opening a new shell and speeds up the process. This also speeds up VIM if you are using vim-rails or vim-ruby.

Setting up Emacs as Ruby development environment on OSX

Hemant  - October 17, 2013  |   , , ,

Learning and setting up Emacs can be intimidating and while some folks find much use in using a pre-configured setup like - Emacs Prelude, I personally think it is better that someone new to Emacs installs each and every module himself and learns from it. I am assuming OSX as development environment here but the instructions should work well enough on Linux as well.

  1. Install Emacs : Install Emacs using Homebrew. I personally do not recommend using Emacs for OSX distribution. You can find out why here.

     brew install emacs --use-git-head --cocoa --srgb
    

    There is a bug in Emacs HEAD that prevents it to be working with Cask and thus try not to use Emacs HEAD.

  2. Install a Package Manager : I use pallet for managing various packages. Think of pallet as Bundler/Maven for Emacs. Follow instructions available in section For newly installed Emacs. Once you have pallet installed each package you install via package-install command will automatically update the Cask and you can checkin the file in version control system and carry it around.

    Sometimes you have to install a package directly from git. I personally use Git Subtrees for adding packages from git. Feel free to use submodules or directly copying files as well.

  3. Structuring your Customization : Over last couple of years I personally have been using following style of structuring my emacs setup.

     .emacs.d/
       init.el -> The real .emacs file.
       custom/
         01ruby.el
         02org.el
         03auto-complete.el
         04web-mode.el
    
  4. The .emacs file : Contents of init.el looks something like this:

     (require 'cask "~/.cask/cask.el")
     (cask-initialize)
     (require 'pallet)
     (add-to-list 'load-path "~/.emacs.d/custom")
     (add-to-list 'load-path "~/.emacs.d/other_paths)
     ..
     ..
     (load "00common-setup.el")
     (load "01ruby.el")
     (load "02org.el")
    
  5. Global Config : I use a lot of global configuration stuff which isn’t mode specific, some important ones are:

     ;; Navigate between windows using Alt-1, Alt-2, Shift-left, shift-up, shift-right
     (windmove-default-keybindings)
    
     ;; Enable copy and pasting from clipboard
     (setq x-select-enable-clipboard t)
    

    You can see complete file on, 00common-setup.el. If you are completely new to Emacs you should use C-h t (That is press Control-h and then t) to launch inbuilt Emacs tutorial from within Emacs.

Packages You need

For complete list of package see my Caskfile. Also some packages I use straight from git and hence may not be listed in Caskfile. With that out of the way, I will try to cover most important packages I love and use everyday:

  1. auto-complete : You can install it via package-install and below is my configuration of auto-complete mode:

     (require 'auto-complete-config)
     (add-to-list 'ac-dictionary-directories
         "~/.emacs.d/.cask/24.3.50.1/elpa/auto-complete-20130724.1750/dict")
     (ac-config-default)
     (setq ac-ignore-case nil)
     (add-to-list 'ac-modes 'enh-ruby-mode)
     (add-to-list 'ac-modes 'web-mode)
    

    It gives me access to all powerful auto-completion system for Emacs. Below is a screenshot.

    [Emacs auto-complete](/assets/images/emacs/auto-complete-7a77150cd11c64f1b5f22de4b0a52d72de61d7358109e0654dcf074cce7971cf.png

  2. ag : You should use silver searcher and corresponding Emacs mode ag. They beat likes of ack or grep hands down.

  3. enh-ruby-mode : Personally I am big fan of Enhanced Ruby Mode. It gives you better syntax highligting, better indentation schemes and code intelligence without running flymake kind of thing.

    ![enhanced ruby mode](/assets/images/emacs/enh-ruby-mode-adedbb79b2f586b70a48d9d3cebe318592a5db4e6867523124409c524dc2c4ff.png

    You can see rest of my enh-ruby-mode specific configuration in file 01ruby.el. You will notice that I have included a mini rspec-mode there , which allows me to run specs of a complete file or just spec under the cursor.

  4. smartparens : smartparens mode is too much of a moving target and hence I always use it from git. My smartparens configuration is:

     (require 'smartparens-config)
     (require 'smartparens-ruby)
     (smartparens-global-mode)
     (show-smartparens-global-mode t)
     (sp-with-modes '(rhtml-mode)
       (sp-local-pair "<" ">")
       (sp-local-pair ""))
    

    SmartParens mode is many things. It automatically inserts closing parethesis, tags, end’s depending on major-mode. Highlights them, allows you to move between them. Allows you to wrap existing texts. However you may not even have to go through their documentation, it mostly just works.

  5. Projectile Mode : Projectile is one among many ways of handling projects in Emacs. My projectile configuration looks like:

     (require 'grizzl)
     (projectile-global-mode)
     (setq projectile-enable-caching t)
     (setq projectile-completion-system 'grizzl)
     ;; Press Command-p for fuzzy find in project
     (global-set-key (kbd "s-p") 'projectile-find-file)
     ;; Press Command-b for fuzzy switch buffer
     (global-set-key (kbd "s-b") 'projectile-switch-to-buffer)
    
     [![It allows me to do:](/assets/images/emacs/emacs4-2be08a609329db017ab61dafd234d821fe408e5fffa0f0002957e8ca52fc0a65.gif](/assets/images/emacs/emacs4-2be08a609329db017ab61dafd234d821fe408e5fffa0f0002957e8ca52fc0a65.gif)
    
  6. Rainbow Mode : Add color to your css/scss. Check screenshot below:

    Add color

  7. Robe Mode : Robe mode is what makes Emacs full featured IDE even comparable to likes of RubyMine etc.

Robe mode:

Robe can also integrate with `auto-complete` mode to provide more intelligent auto-completion.
I personally do not use that feature because it makes things slower.
  1. Highlight indentation for Emacs : Highlight Indentation mode allows you to see indentation guides. My setup for the mode is:

     (require 'highlight-indentation)
     (add-hook 'enh-ruby-mode-hook
               (lambda () (highlight-indentation-current-column-mode)))
    
     (add-hook 'coffee-mode-hook
               (lambda () (highlight-indentation-current-column-mode)))
    
  2. Yasnippet : Yasnippet is Emacs snippet library. Out of box it won’t work with enh-ruby-mode because it searches for ruby-mode. You can make it work by copying ruby-mode snippets to enh-ruby-mode folder.

  3. Flyspell : If you are like me, you make lot of typos in code comments, strings etc. Using Flyspell mode you need not worry about it - interfering with programming language keywords etc. My setup for Flyspell is :

    (require 'flyspell)
    (setq flyspell-issue-message-flg nil)
    (add-hook 'enh-ruby-mode-hook
              (lambda () (flyspell-prog-mode)))
    
    (add-hook 'web-mode-hook
              (lambda () (flyspell-prog-mode)))
    ;; flyspell mode breaks auto-complete mode without this.
    (ac-flyspell-workaround)
    

    If you are on OSX you may have to run brew install ispell to make it work.

  4. Rinari : With Projectile mode, I am finding less and less use of this mode. But this is your canonical rails mode for Emacs. Get it via package-install or Rinari Github repo.

  5. dash-at-point : If you are on OSX you should totally use Dash. Using dash-at-point you can view documentation of methods, classes etc right from Emacs.

  6. multiple-cursors : Multiple Cursors is Emacs answer to multi-line editing feature of Sublime etc. Anything I can add here is probably better covered in this Emacs Rocks screencast.

  7. textmate.el : textmate.el allows me to quickly move between methods in a file, independent of programming language mode (and without ctags). It has many other utilities.

  8. WebMode : For a long time Emacs users were mostly stuck with half-assed multi-language mode when editing html templates. WebMode is like breath of fresh air for those who mix - ruby/js/css/coffee in their HTML!

Apart from aforementioned modes, I still use too many other modes - sometimes even unknowingly! Some modes that deserve honourable mention but I don’t use them is - Magit, Expand Region. If you do lot of Javascript then have a look at - Swank-js. If you are coming from Vim and looking for decent nerdtree replacement, you can try Dirtree or ECB or Project explorer.

Learning Emacs in a way is - as deep as you are willing go. For resources I recommend:

I hope you enjoyed this article and if you want to keep updated with latest stuff we are building or blogging about, follow us on twitter @codemancershq.

Introducing Invoker - Manage processes in development envionment

Hemant  - June 8, 2013  |   , ,

If you are working on a large application with several dependencies such as - more than one application server, background job, it can get tedious to start and stop and check their logs in bunch of different places.

Invoker is a small utility we wrote to manage all such processes from one place. Invoker owes much to Foreman for inspiration, however unlike Foreman it has been exclusively written to be used in development environment. Some major differences are:

To get started we need to install invoker gem which will give us command line utility called invoker, we can do that via:

gem install invoker

Currently it only works with Ruby 1.9.3 and 2.0.

After that we need to start by creating a ini file which will define processes we want to manage using invoker. For example:

[rails]
directory = /home/gnufied/god_particle
command = zsh -c 'bundle exec rails s -p 5000'

[dj]
directory = /home/gnufied/god_particle
command = zsh -c 'bundle exec ruby script/delayed_job'

[events]
directory = /home/gnufied/god_particle
command = zsh -c 'bundle exec ruby script/event_server'

After that we can start process manager via:

~> invoker start invoker.ini

Above command will start all your processes in one terminal with their stdout/stderr merged and labelled.

Now additionally we can control individual process by,

# Will try to stop running delayed job by sending SIGINT to the process
~> invoker remove dj

# If Process can't be killed by SIGINT send a custom signal
~> invoker remove dj -s 9

# add and start running
~> invoker add dj

One can also enable OSX notifications for crashed processes by installing terminal-notification gem. It is not a dependency, but can be useful if something crashed and you weren’t paying attention.

Using with rbenv or rvm

The way rbenv and rvm work sometimes creates problems when you are trying to use a process supervisor like invoker. There are couple of things to keep in mind, If you are running invoker with Ruby version x, but your application requires Ruby version Y:

I hope you find this gem useful and do not forget to follow us on twitter @codemancershq if you want to keep up with latest utility we are making to make our lives wee bit easier.

Source Code

Profile Ruby 2.0 apps using DTrace - part 2

Hemant  - May 27, 2013  |   , , ,

This is Part - 2 in series of articles we are writing on Dtrace. If you haven’t already - do read Part 1.

Running Dtrace scripts

Before we delve deeper into guts of Dtrace, one of the questions which I often get asked is, How can I profile one off programs (i.e not a application running in background) with Dtrace.

There are basically couple of ways you can fire Dtrace.

~> sudo dtrace -qs <dtrace_script>.d
~> sudo dtrace -s count.d -c '/Users/gnufied/.rbenv/versions/20/bin/ruby leaky_app.rb'

What is a predicate?

In last part of tutorial we described what a probe description looks like. For a refresh:

probe description
/predicate/ 
{
    action;
}

Predicate is a way to filter the number of probes fired based on certain critirea. For example, in last tutorial we wrote a Dtrace script to keep track of all system calls being made by all currently running applications.

Lets say, now we want to restrict those probes only to system calls that are being made by Ruby applications. We can do this by introducing a predicate in our Dtrace script.

syscall:::entry
/execname=="ruby"/
{
	printf("%s(%d) called %s\n", execname, pid, probefunc);
}

Or you can use pid predicate:

/*
 * Print system calls made by pid 61465
 *
 */
syscall:::entry
/pid == 61465/
{
	printf("%s(%d) called %s\n", execname, pid, probefunc);
}

Now when I was learning Dtrace, I was mightly confused from where these things called execname, pid, probefunc are coming from? Turns out, Dtrace automatically provides built-in variable definitions for inspecting processes. They make Dtrace scripts quite powerful.

Built-in Variables

Here is a list of important built-in variables available in your DTrace scripts:

Now, I believe previous scripts would have started making sense. Above list does not include all built-in variables though.

Lets do an exercise and try to print number of bytes being read (from network or file system) by process with pid 61465. The script will look like:

syscall::read:entry
/pid == 61465/
{
    printf("%s read %d\n", execname, arg2);
}

If you crack open man 2 read you will find out that read system call’s third argument contains number of bytes being read.

Having just ability to print numbers as probes fire is useful but you can easily imagine it would be nice, if we can somehow aggregate the data. For example, how much of your program’s memory is taken by string allocation or how much time is being spent in total in the function foo. Ability to aggregate such data would be really useful.

Aggregates and Collections

Agrregates, Collections and variables are most important tools DTrace has to offer for data gathering, analyzing and printing. We will start with different types of variables available to us.

Variable types

D scripting language has 3 types of variables which you can use to keep track of things.

Block local

You can define a block local variable with :

syscall::read:entry
{
  this->currentTime = timestamp;
}

Scope of a block local variable is limited to the block in which it is defined.

Global scalar variables

You can define a global variable like:

BEGIN {
  startTime = timestamp;
}

Variables defined like this are global and they are available in all the probe descriptions. Also, if you haven’t guessed it - BEGIN and END[2] are two special probes that fire at the beginning of a DTrace script and at the end of it. They can be very useful for initializing things or printing summary results at the end of execution.

Thread local variables

If you are trying to profile a multi-threaded application you will want to keep DTrace variables separate for different threads. You can do that with thread local variables:

syscall::read:entry
{
  self->currentTime = timestamp;
}

Variables declared like this won’t be shared between threads and hence can be used reliably.

Associative Arrays

If you are coming from Ruby, think of associative array as a Hash where that keys can be a single value or a tuple of values. In either case once declared type of key shouldn’t change. For example, following script will calcuate time spent by each process in a system call.

syscall:::entry
{
  callTime[execname, probefunc] = timestamp;
}

syscall:::return {
    this->timeElapsed = timestamp - callTime[execname,probefun];
}

You can learn more about these different kind of variables at[1].

Aggregates

Armed with knowledge of variables and collections, we can finally dip our toes in aggregates. Aggregates is a way of gathering data from individual probes without you the programmer doing actual work of summing things up,calculating the median, frequency of occurence etc.

For example, lets write a DTrace script that prints number of top 20 objects Rails creates when it boots up. You are obviously going to need Ruby 2.0 for following script to work.

/* rails_boot.d
 * object-create is the name of the probe that fires 
 * when any new ruby object is created. 
 * Corresponding function called is - rb_obj_alloc
 */
ruby*:::object-create
{
   /* If you look up code of rb_obj_alloc, first argument to 
    * function contains class which is being instantiated
    */
   @objects[copyinstr(arg0)] = count();
}

END {
  /* Truncate the aggregate to top 20 results */
  trunc(@objects,20);
  /* Print the aggregate */
  printa(@objects);
}

The output should look something like:

  Mime::Type                                                       22
  Psych::Nodes::Scalar                                             27
  Bundler::LazySpecification                                       39
  Set                                                              54
  Sass::Script::Functions::Signature                               55
  Rails::Paths::Path                                               71
  Class                                                            73
  Pathname                                                         74
  Bundler::DepProxy                                                77
  Rails::Initializable::Collection                                 93
  Regexp                                                           95
  Rails::Initializable::Initializer                               143
  Range                                                           332
  Gem::Specification                                              410
  Hash                                                            579
  Gem::Dependency                                                1660
  Gem::Version                                                   2416
  Gem::Requirement                                               3061
  String                                                         4723
  Array                                                          4948

Keep in mind, running above script isn’t as simple as running sudo dtrace -qs rails_boot.d -c 'rails s' . Depending on whether you are using RVM or rbenv all those shell scripts shims create a problem when running dtrace scripts. What I ended up doing is, created a file called load_env.rb that had just one line require_relative config/environment and then ran the Dtrace script with:

sudo dtrace -qs rails_boot.d -c "/Users/gnufied/.rbenv/versions/20/bin/ruby load_env.rb" 

Now there are some new keywords in above script and we will to get them in a bit, but first let us talk about aggregates. DTrace allows you to store result of aggregate functions in objects called aggregates, they are defined like:

@name[ keys ] = aggfunc ( args );

Aggregate object name starts with @ and behave very similar to associative arrays and hence can use tuples as keys. Using aggregate functions frees you up from keeping counter yourself. In above script we are using count function which keeps track of number of times that particular probe fired. There are other aggergate functions available as well:

You can find out more about aggregates from [3].

That about sums up content for part 2 of DTrace tutorial. In the next part we are going to cover in built functions, printing and profiling of real world Ruby applications. You should also follow us on twitter @codemancershq if you don’t want to miss out Part 3.

References

Profile Ruby 2.0 apps using DTrace - Part 1

Hemant  - April 16, 2013  |   , , ,

This is a series of blog posts we are going to do in tutorial format, this is Part 1 of our tutorial on Dtrace.

DTrace is awesome. If your operating system supports it, using DTrace can give incredible insight into processes that are running on your system. If you are still not convinced about its usefullness you should checkout - How Dtrace helped twitter identify bottlenecks, Profiling Rails startup with Dtrace.

So what exactly is DTrace? According to Wikipedia - “DTrace is a dynamic tracing framework that allows users to tune and troubleshoot applications and the OS itself.” It is available on Solaris, Mac OSX, FreeBSD and recent versions of Oracle Linux. If you are running any of the above operating systems, there is nothing to install - just hop in.

Dtrace allows you to instrument a live process by defining trace points called probes, which can be turned on or off dynamically. The probes fire when a particular condition is met and it requires no change in application code, no special flags and there is very little penalty of running a production app with probes turned on.

What is a probe?

At its core a probe is like a light bulb that switches on when a condition is met. The condition could be entering a C function defined within a module, exiting a C function, entering a system call provided by Kernel etc. A probe is made of 5 parts,

All the processes currently running on operating system (including the kernel itself) have some probes defined. As an example, lets try and list all the probes defined by Ruby 2.0

~> sudo dtrace -l | grep ruby

Ruby 2.0 probes

Running Dtrace

Dtrace probes can be set and run via the command line itself or can be run via Dtrace scripts written in a programming language called D.

Some simple command line examples:

~> sudo dtrace -f write

Above probe will start firing each time any application makes write system call.

Dtrace scripting capabilities

The real power of Dtrace lies in ability to script - tracing of running applications, gathering results and then printing or plotting them.

A DTrace script is essentially made of 3 parts:

probe description
/predicate/ 
{
    action;
}

D language has some similarity with C but at its core it is a scripting language, probably closer to awk etc.

For example, lets write a D script for printing all system calls being made by currently running applications.

~> cat > dtrace_scripts/syscall.d
syscall:::entry 
{
  printf("%s(%d) called %s\n", execname, pid, probefunc)
}

Above script when ran via sudo dtrace -qs syscall.d will start printing name,pid of processes that are making system calls and what system calls they are making. The script does not have predicate, but only probe description.

Probe description

A probe description can be written as:

provider:module:function:probename

Few simple examples:

# A probe that will fire each time a system call called write is entered
syscall::write:entry
# This probe will fire each time someone creates a new ruby array
ruby*::ary_new::

As you can see, you can skip parts of the probe and can use wildcards such as * or ?.

That about sums up content for part 1 of DTrace tutorial. In the next part we are going to cover predicates, aggregates and we will do profiling of real Ruby applications.I will recommend you to play with various probes and stay tuned for Part 2 which is coming up very soon. You should also follow us on twitter @codemancershq if you don’t want to miss out Part 2.

Bundler: Cleanup old unused gems

Yuva  - March 27, 2013  |   , ,

Recently I resumed an old project {after 2 months}. My peers are still working on it and they have upgraded lots of gems. After running bundle install, many gems got upgraded and old version were still hanging around in vendor folder. This little handy command helped in removing them:

bundle clean -V

NOTE: Btw, I always install gems in ./vendor/ruby folder, and don’t maintain any rvm gemsets for projects.

Upgrading to Mountain Lion and your development environment

Hemant  - July 27, 2012  |   , , ,

Prem of thoughtbot already had some brief instructions about upgrading to mountain lion and then thanks to Kenny for adding much to those instructions.

What follows is my own upgrade path:

  1. Take backup via Time Machine or superduper!.
  2. Upgrade using installer from App store and reboot.
  3. Install XCode 4.4 from App store. Looks like, if you don’t care much about developing iOS or OSX applications, you can skip this step. But I didn’t.
  4. Install command line tools of XCode as instructed in Prem’s blog post.
  5. Change permissions of Homebrew directory.

     sudo chown -R `whoami` /usr/local
     brew update
    
  6. If you are like me, you still have few applications running on 1.8.7. Ruby versions before 1.9.3 can’t be easily compiled using llvm-gcc compiler. Install gcc-4.2 using homebrew:

     brew tap homebrew/dupes
     brew install apple-gcc42
    
  7. Configure apple-gcc42 as default compiler. Update your ~/.zshrc or ~/.bashrc and add following lines:

     export CC=/usr/local/bin/gcc-4.2
    

    Above line should take care of compiling ruby itself and many native gems which use proper compiler detection. Unfortunately this won’t work for few odd gems and hence, you also need to do:

     sudo ln -s /usr/local/bin/gcc-4.2 /usr/bin/gcc-4.2
    
  8. Install XQuartz. This step may require you to logout and login again.

  9. Even if you already had ruby 1.8.7/1.9.3 compiled, I will urge you to recompile new one - just to ensure everything works perfectly.

    For compiling Ruby 1.8.7/REE:

     export CPPFLAGS="-I/opt/X11/include"
     export CFLAGS="-I/opt/X11/include"
    

    For compiling ruby:

     rvm reinstall ree
     rvm reinstall 1.9.3
    

    You should be able to compile ruby 1.9.3 without requiring any of above flags.

  10. Some recommend reinstalling key packages that you have installed using homebrew before ML upgrade.I did not bother to reinstall all the packages, but I did reinstall imagemagick.

     brew unlink imagemagick
     brew install imagemagick
    
  11. At this point you should have a working ruby and rails setup. I faced a minor problem where I was running mysql using packages downloaded from Mysql website and they were installed in /usr/local and after I changed permissions to restore Homebrew setup, my Mysql installation broke. To restore the sanity:

    sudo chown -R _mysql:_mysql /usr/local/mysql-5.xx
    
  12. In some of Rails applications, Nokogiri was throwing warning about being compiled against different version of libxml2. To get rid of those, I simply nuked Rails.root/vendor/ruby and reinstalled the gems my app was using via bundle install.

  13. Mark has suggested these instructions for getting pow running with Safari 6.

  14. I found that automatic reconnecting to wi-fi after coming out of sleep was broken. To fix this, I removed all the remembered locations and then in Configure Network Prefrences -> TCP/IP changed Configure IPV6 to Link-local only.

  15. I also noticed that external monitor when plugged in, won’t be detected at all.To fix this I had to reset my PRAM. You can follow this article for more information on resetting PRAM.

The kitchen sink libraries

Hemant  - May 8, 2012  |   , ,

There was a time when Ruby libraries where like small Unix utilities, they did one thing and they did it well. If you go read code of some of the older plugins, it will be quite simple and straightforward affair.

Today however trend is, to write kitchen sink libraries that do a lot. You want file upload, they work at Rack level and support not only Rails but all the rack complaint frameworks under the sun, not only relational databases but Mongo as well, not just local file storage but S3/Cloudfiles as well, not just file uploads but some kind of caching via Rack::Cache as well.

Now at its core, handling file uploads is very simple and straightforward affair. Uploading to S3 or Cloudfiles is even simpler. I daresay, getting started with the kitchen sink library will be easy and simple as well. But if one thing Law of leaky Abstractions has taught us that is - eventually you will have look under the hood of these kitchen sink libraries and suddently MysqlAdapter which subclasses AbstractPersistenceAdapter won’t appear very attractive, because your application doesn’t gives two hoots about Mongo support. Jumping through hooves to grab that uploaded file via Rack will appear waste of time as well because you are probably never going to need this working on framework other than Rails. And while support for Rack::Cache was nice, you can’t use it in production anyways.

I don’t know about you, but I prefer gems and plugins that I include in my code easily grokkable. Below is a time bomb taken from a popular file upload gem and good luck - if you will have to grok this code and fix something yourself.

autoload_files_in_dir = proc do |path, namespace|
  # Define the module
  eval("module #{namespace}; end")
  # Autoload modules/classes in that module
  Dir.glob("#{path}/*.rb").each do |file|
    file = File.expand_path(file)
    sub_const_name = camelize[ File.basename(file, '.rb') ]
    eval("#{namespace}.autoload('#{sub_const_name}', '#{file}')")
  end
  # Recurse on subdirectories
  Dir.glob("#{path}/*/").each do |dir|
    sub_namespace = camelize[ File.basename(dir) ]
    autoload_files_in_dir[dir, "#{namespace}::#{sub_namespace}"]
  end
end