Milovan Zogovic

Benchmarking gem load times

Tue, 28 May 2013 10:04:09 +0200

There is an awesome gist written by Pan Thomakos, and later refined by Andrew Kane here: https://gist.github.com/ankane/5022636

Here is simple fish shell function that can be used to run this script in any project you need:

function benchmark_bundler
  set -x BUNDLE_GROUPS $argv
  curl -fsSL https://gist.github.com/raw/5022636/benchmark.rb | ruby
end

Just put it in your ~/.config/fish/config.fish file and off you go.

Monitoring memory usage on long-running scripts

Wed, 27 Mar 2013 11:36:45 +0100

There is a simple way to monitor memory usage on long-running scripts. You can just create new thread which will output current memory usage to console in short time intervals:

Thread.new do
  while true do
    memory = `ps -o rss -p #{Process::pid}`.chomp.split("\n").last.strip.to_i
    puts "Memory: #{memory/1024} MB"
    sleep 0.25
  end
end

Super fast way of launching postgres console for rails apps

Wed, 13 Mar 2013 10:01:01 +0100

Database console can be run with bundle exec rails dbconsole command. However, this operation performs relatively slow (due to bundler and deps).

We can make it much faster with simple ruby script:

#!/usr/bin/env ruby

require 'yaml'
require 'fileutils'

database_config = YAML.load_file(Dir.pwd + '/config/database.yml')['development'];
database_name = database_config['database']

exec('psql', "-d#{database_name}")

Just save the script somewhere in your $PATH and give it u+x permission.

I like to place scripts like these in my ~/dev/bin folder (included in $PATH). I gave this one db name, so the only thing i need to do to fire up postgres console is to type db. Simple and extremely fast.

Optimizing postgresql query for DISTINCT values

Fri, 08 Mar 2013 14:10:00 +0100

If you want to get DISTINCT values from large tables (and by large I mean hundreds of thousands of rows), it is painfully slow.

Simple query like this: SELECT DISTINCT(day) FROM reports can take a lot of time to be processed.

Here is how you can improve its speed dramatically using postgres recursive functionality:

WITH RECURSIVE t(n) AS (
    SELECT MIN(day) FROM reports
  UNION
    SELECT (SELECT day FROM reports WHERE day &gt; n ORDER BY day LIMIT 1)
    FROM t WHERE n IS NOT NULL
)
SELECT n FROM t;

I’ve tested it against 50 million rows with 365 distinct values for day, and it executed in 8ms, opposed to 15.7 seconds for SELECT DISTINCT query.

There are only two prerequisites:

you need index column you want to get distincts from.
you have postgres 8.4+

For more info, check out the postgresql docs on the topic: http://www.postgresql.org/docs/9.1/static/queries-with.html

Rails consecutive migrations acting weird

Sat, 29 Dec 2012 12:54:55 +0100

If you’re using models in your database migrations, its good practice to define dummy active record clases for them:

class SomeMigration < ActiveRecord::Migration
  class User < ActiveRecord::Base
    reset_column_information
  end
  # ...
end

If you have many migrations to run and model definition changes over time (which is likely to happen), you need to call reset_column_information in order to “reload” the model on beginning of every migration.

Showing git branch in fish shell prompt

Fri, 14 Dec 2012 15:00:00 +0100

Put following code at the end of ~/.config/fish/config.fish. It will also highlight in red if branch is dirty.

set fish_git_dirty_color red
set fish_git_not_dirty_color green

function parse_git_branch
  set -l branch (git branch 2> /dev/null | grep -e '\* ' | sed 's/^..\(.*\)/\1/')
  set -l git_diff (git diff)

  if test -n "$git_diff"
    echo (set_color $fish_git_dirty_color)$branch(set_color normal)
  else
    echo (set_color $fish_git_not_dirty_color)$branch(set_color normal)
  end
end

function fish_prompt
  if test -d .git
    printf '%s@%s %s%s%s:%s> ' (whoami) (hostname|cut -d . -f 1) (set_color $fish_color_cwd) (prompt_pwd) (set_color normal) (parse_git_branch)
  else
    printf '%s@%s %s%s%s> ' (whoami) (hostname|cut -d . -f 1) (set_color $fish_color_cwd) (prompt_pwd) (set_color normal)
  end
end

"42 is a nice number that you can take home and introduce to your family"

Tue, 30 Oct 2012 10:02:28 +0100

“42 is a nice number that you can take home and introduce to your family”

- Douglas Adams

Attachinary v1.2.0 released

Sat, 27 Oct 2012 16:35:00 +0200

Changelog

Drag and drop support (on supported browsers)
Selecting multiple files (on supported browsers)
Upload progress indicator (prepended on submit button)
Rake task for fetching assets (rake attachinary:fetch_fileupload)
Ability to assing image urls (e.g. user.avatar_url = 'http://..')
Ability to assign IO objects (e.g. user.avatar = File.open(...))
No-JS support

How to validate Array fields in Mongoid

Fri, 05 Oct 2012 13:09:00 +0200

Define custom ArrayValidator in app/validators/array_validator.rb:

class ArrayValidator < ActiveModel::EachValidator
  def validate_each(record, attribute, values)
    [values].flatten.each do |value|
      options.each do |key, args|
        validator_options = { attributes: attribute }
        validator_options.merge!(args) if args.is_a?(Hash)

        next if value.nil? && validator_options[:allow_nil]
        next if value.blank? && validator_options[:allow_blank]

        validator_class_name = "#{key.to_s.camelize}Validator"
        validator_class = begin
          validator_class_name.constantize
        rescue NameError
          "ActiveModel::Validations::#{validator_class_name}".constantize
        end

        validator = validator_class.new(validator_options)
        validator.validate_each(record, attribute, value)
      end
    end
  end
end

You can use it like this in your models:

class User
  include Mongoid::Document
  field :tags, Array

  validates :tags, array: { presence: true, inclusion: { in: %w{ ruby rails } }
end

It will validate each element from the array against every validator specified within array hash.

I've just released v1 of attachinary gem for cloudinary with monoid support and much cleaner integration.

Mon, 10 Sep 2012 13:58:10 +0200

I've just released v1 of attachinary gem for cloudinary with monoid support and much cleaner integration.

Integrating Prefixr with Rails asset pipeline

Thu, 30 Aug 2012 20:04:00 +0200

Prefixr is cool online service created by Jeffrey Way for adding vendor prefixes to your css. It has nice and simple API to let you integrate it into your workflow. Here is how you can integrate it with Rails asset pipeline.

Just create prefixr.rb file inside config/initializers folder with following content:

require 'uri'
require 'net/http'

class PrefixrPostprocessor < ::Tilt::Template
  def prepare
  end

  def evaluate(context, locals, &block)
    uri = URI.parse "http://prefixr.com/api/index.php"
    Net::HTTP.post_form(uri, { css: data }).body
  end
end

Rails.application.assets.register_postprocessor 'text/css', PrefixrPostprocessor

It defines and registers postprocessor for text/css files. Now, every css file served through rails will be automatically prefixr’d! This will work even if you use sass or less!

The downside of this approach is that it will slow down the development because it will need to make HTTP request to prefixr.com API every time the css file is requested by browser. I wish that there is ruby version of prefixr service as a gem that I could just use directly inside PrefixrPostprocessor. That would be just awesome! Who knows.. maybe I just make one!

Cachex: tag based fragment caching on steriods

Wed, 15 Aug 2012 16:07:14 +0200

I’ve just released another gem. Its called Cachex.

This gem is inspired by excellent Cashier gem. I wanted to take it a step further and try to automate tag dependency generation.

I had this scenario in mind:

Blog has many posts
Each post has an author (user)
Each post has many comments
Each comment has its own author (user)

Each model has its own partial. Inside _post partial we have _user partial that renders post’s author. _post partial also include many _comment partials. Each _comment partial have author’s name displayed in it.

We’re also using fragment caching in order to cache entire post for faster serving. The burning question is: what happens if user decides to change its name (or perhaps, more commonly, an avatar)?. We have to invalidate every cached partial that displayed that user. How do we do that?

If we’re using auto-expiring key strategy the only way to accomplish this would be to touch everything that has to do with that user. That just doesn’t make any sense.

If we’re using tag-based strategy, then we would need to manually tag every fragment with all users that have to do something with it (even users from its comments). Such approach is going to be very hard when we get to deeply nested partials. Top most partial will have to be aware of all dependencies down to the deepest one.

And finally, my solution: to automatically extract all dependencies from all sub-partials and to store them to help cache invalidation.

Testing API from Rails Console

Fri, 15 Jun 2012 09:30:51 +0200

Recently I created small API for one of my apps that is to be accessed by an iPhone application. I followed Railscasts guides on how to create, version, and secure it with access token. Even though you can test the api with CURL command, it tend to get a bit more complex when you need to specify token header every single time.

I wanted to be able to test the API in more simplistic way. I wanted it to be something like this:

api User.first, :get, '/api/projects'

Here is the simple helper method that can make this process much easier:

def api(user, method, url, params=nil)
  headers = {
    'Accept' => 'application/roommates.v1',
    'HTTP_AUTHORIZATION' => ActionController::HttpAuthentication::Token.encode_credentials(user.api_token)
  }
  app.send(:"#{method}", url, params, headers)
  ap JSON.parse(app.response.body)
end

Method simply sets proper headers, issues request, pretty-prints returned json (with awesome_print gem), and returns it. You can customize any way you want.. this is just an idea of how it might look.

All you have to do is put this method in .irbrc file in your project root and it will be automatically loaded each time you start your rails console.

Redis Marshaling

Sun, 13 May 2012 09:58:32 +0200

If you have time consuming method in your app that you call frequently, chances are that you’d like to cache it. Redis is great key-value store that can be used for this purpose. However, redis only store string values. In order to store other kinds of objects, you’ll have to use marshaling.

There is a built-in ruby class called Marshal that does the job well. The only thing you have to do is to call Marshal.dump method and pass it an object. The exact opposite is the Marshal.load method.

I’ve created helper method for easier integration with Redis. Here is how it looks:

module RedisHelper

  # Returns the cached value for given key.
  # If it doesn't exist, it evaluates the block and caches what it returns.
  # 
  # @param [String] key the cache key
  # @param [Hash] options
  # @option options [Integer, nil] :ttl time to live in seconds
  # @option options [Boolean, false] :marshal whether to perform marshaling
  # @yield [] to be evaluated if the key does not exist
  def self.fetch(key, options={}, &block)
    raise ArgumentError.new("Must Supply Block") unless block_given?
    return yield unless Rails.application.config.action_controller.perform_caching

    val = nil

    if REDIS.exists(key)
      val = REDIS.get(key)
      begin
        val = Marshal.load(val) if options[:marshal]
      rescue TypeError
        val = nil
      end
    end

    if !val
      val = yield

      if options[:marshal]
        REDIS.set key, Marshal.dump(val)
      else
        REDIS.set key, val
      end

      REDIS.expire key, options[:ttl] if options[:ttl]
    end

    val
  end

end

There is not much to describe here since its already documented using yard. Instead, I’ll show how I used in in one of my apps:

module Geocoder

  def self.geocode(location)
    key = "geolocations/#{location}"
    RedisHelper.fetch key, ttl: 24.hours, marshal: true do
      Geokit::Geocoders::GoogleGeocoder.geocode(location)
    end
  end

end

I’m using geokit gem for discovering geocoordinate of an address. It uses Google geocoding API which is limited to 2500 queries per day. This piece of code will cache the result of GoogleGeocoder.geocode method so all successive requests will be returned from cache.

Simple HTML5 details polyfill

Wed, 25 Apr 2012 17:16:03 +0200

Although there are many HTML5 <details> polyfills already available, I didn’t like the complexity behind them. Here is how you can create your own from scratch in just few lines of css and js.

The idea was to put as much “logic” as possible in the stylesheet itself. It should hide/show the content based on the presence of open attribute. Here is the sass file:

.no-details details {
  > * {
    position: absolute;
    visibility: hidden;
  }

  > summary, &[open] > * {
    position: static;
    visibility: visible;
  }


  > summary {
    display: block;
    &:before {
      content: "►";
      padding-right: 5px;
      font-size: 11px;
    }
  }

  &[open] > summary:before {
    content:"▼"
  }
}

All children of details tag (except summary) will be by default hidden. If details element has open attribute set, its content will be shown. We could use css :not selector to simplify the css a whole lot, but it is only supported by IE9+, and I’d like polyfill to target IE8+.

And here is the coffeescript:

jQuery ->
  return if Modernizr.details

  $(document).on 'click', 'summary', (event) ->
    $summary = $(this)
    $details = $summary.parent()

    if $details.attr('open')
      $details.removeAttr('open')
    else
      $details.attr('open', 'open')

It simply toggles the open attribute when user clicks on summary. Its as simple as that.

Responsive Images with Cloudinary on Rails

Wed, 18 Apr 2012 16:14:40 +0200

Responsive Images is a hot topic lately. There are few solutions to the problem: introduction of new html element (which will take years to be ready for use), new image format (which may take even more time), and using other technologies to solve the problem. Here, I’ll focus on the last one.

The goal is clear:

Detect user’s device screen resolution
Serve appropriate images for best browsing experience

In order to handle cases where you cannot detect user’s device capabilities (e.g. no javascript support), we need to provide reasonable defaults (e.g. medium sized images).

Here is how you can accomplish this in few simple steps by using Cloudinary service for hosting your assets.

The concept is simple:

When user enters our website, we’ll use javascript to detect his device resolution.
We’ll store that resolution in a cookie.
Backend will utilize information inside cookie to serve appropriate image format.

First two steps are quite easy to do. All you have to do is inject following javascript

document.cookie='resolution='+screen.width+'x'+screen.height)+'; path=/';

That would store user’s device screen resolution in a cookie. On the next page load, that cookie will be available to your backend.

Now, lets override cloudinary cl_image_tag to allow :responsive option. You can do so by including this in application_helper file:

def cl_image_tag(source, options = {})
  if (sizes = options.delete(:responsive)) && (resolution = cookies[:resolution])
    res = case resolution.split('x').first.to_i
          when 0..480 then :phones
          when 481..767 then :tablets
          else :desktops
          end

    size = sizes[res]
    options[:size] = size if size
  end

  super(source, options)
end

Method simply checks whether :responsive option and :resolution cookie are present. If so, it replaces the :size parameter with appropriate value. The usage would be like this:

<%= cl_image_tag "sample.jpg", size: '200x200', crop: 'fit', 
    responsive: { phones: '50x50', tablets: '100x100' } %>

That would serve 50x50px image for phones, 100x100px version for tablets and fallback to 200x200px version for all others. Sweet!

Running time in Rails logs

Sun, 15 Apr 2012 11:48:00 +0200

Since Rails 3.2, you can add custom tags to your logs. You can define tags by setting config.log_tags in your config file. It accepts array of symbols that matches methods of request object. Additionally, you can also pass lambdas to form custom tags. Here is how it might look like:

config.log_tags = [:uuid, :remote_ip, lambda { |req| Time.now }]

If you want to lear more about logging, I highly recommend The Logger (revised) screencast by Ryan Bates.

One gotcha here: lambdas are evaluated only once at the very beginning of request delegation (in Rails::Rack::Logger middleware. So, our lambda from above will return the same time for every log entry within same request.

I must admit I’m obsessed with performance lately :) Thats why I wanted to customize my logs to include “running time” next to each log entry. I wanted it to look something like this:

Started GET "/users/sign_in" for 127.0.0.1 at 2012-04-15 11:37:21 +0200
[time:0.008] Processing by Devise::SessionsController#new as HTML
[time:0.019]   Rendered devise/_links.erb (0.5ms)
[time:0.019]   Rendered devise/sessions/new.html.erb within layouts/application (7.5ms)
[time:0.022]   Rendered layouts/_navbar.html.erb (0.4ms)
[time:0.022]   Rendered layouts/_skeleton.html.erb (2.3ms)
[time:0.022]   Rendered layouts/_public.html.erb (2.8ms)
[time:0.024] Completed 200 OK in 16ms (Views: 13.5ms | ActiveRecord: 0.0ms)

Let’s get started. First, we need to define custom log tag in our development.rb config file:

config.log_tags = [ lambda { |req| "time:#{Time.now.to_f}" } ]

This will prefix each log entry with tag like this [time:1334480797.098824]. We can later use regexp to replace this with running time.

Next, lets incude some regexp magic in TaggedLogging.tags_text method. Since this is something we only need in development, we can define this at the very bottom of development.rb file.

class ActiveSupport::TaggedLogging
  def tags_text_with_timestamp
    text = tags_text_without_timestamp
    text && text.gsub(/(?<=time\:)([\d\.]+)/) do |m|
      "%#.3f" % (Time.now.to_f - m.to_f).round(3)
    end
  end
  alias_method_chain :tags_text, :timestamp
end

Note: this code will work only in Rails 3.2.x. I’ve noticed that TaggedLogging class is currently being refactored by rails team, but it won’t be too hard to make this work in future versions of rails.

Highlighting navigation links with javascript

Sat, 14 Apr 2012 11:12:00 +0200

Recently I blogged about Tagging the HTML tag. The general idea is to apply some data- attributes to html tag in order to more easily target the page from css and js files. I also blogged about Javascript Processors, technique to keep your “DOM enhancers” organized.

Using the two, here is how you can very easily implement processor that would highlight active link in your navigation. Lets say that we have following nav element:

<nav class="main">
  <a class="home">Home</a>
  <a class="projects">Projects</a>
  <a class="account">My Account</a>
</nav>

Here is how our NavHighlighter may look like:

@NavHighlighter =
  mapping:
    "nav.main .home":      [ /^\/welcome/ ]
    "nav.main .projects":  [ /^\/projects/ ]
    "nav.main .account":   [ /^\/registrations\/(edit|update)/, /^\/profile/ ]

  process: ->
    for elem, paths of @mapping
      $(elem).addClass("active") if App.onPath.apply(App, paths)

We’re using mapping hash to define highlight conditions for each link. Hash key is just css selector, so you can use anything you want. Hash value is an array representing list of paths for which the given link is active.

Can’t we do it on server? Sure. There are two reasons I’m doing it on the client: cleaner and faster erb template, and fragment caching.

What about users with js disabled? Well, they could probably live without highlighted menu items. If you really want to do this on server and still use caching, read more on Cache Personalization.

Uploading directly to Cloudinary, bypassing your Rails app

Sat, 07 Apr 2012 11:03:00 +0200

IMPORTANT UPDATE: cl_form_tag and valid_cloudinary_response? are now part of cloudinary 1.0.6+ gem, so you can use them “out of the box”.

For those of you who haven’t heard about Cloudinary, check it out! Cloudinary provides everything you (as developer) would ever need for managing images. It basically works like this: you upload an image, then you request transformation of that image from the server. The transformed image is generated on the fly (unless cached) and served to you. There are plenty of transformations like: resizing, cropping, setting quality, changing file format… etc. And, if your designer ever wanted to scale thumbnails a bit, you wouldn’t need to reprocess thousands of images again!

Cloudinary has a ruby gem that can be used in Rails app and it even supports CarrierWave if you use that to manage your uploads. However all these approaches require your server to handle file upload process. Here is how it looks:

User uploads a photo to your server
Rails application receives the file
Rails application sends the file to Cloudinary

Rails is the middleman and it needs to do some relatively heavy processing in order to decode the data received from the user, and prepare it for upload to the Cloudinary. From scaling aspect, this is something you’d probably like to avoid.

It turns out that it is really easy to let your users upload files directly to Cloudinary. After reading some documentation and browsing through source code, I came up with cl_form_tag helper method:

def cl_form_tag(callback_url, options={}, &block)
  form_options = options.delete(:form) || {}
  form_options[:method] = :post
  form_options[:multipart] = true

  options[:timestamp] = Time.now.to_i
  options[:callback] = callback_url
  if options[:transformation]
    options[:transformation] = Cloudinary::Utils.
      generate_transformation_string(options[:transformation])
  end

  options[:signature] = Cloudinary::Utils.
    api_sign_request(options, Cloudinary.config.api_secret)

  options[:api_key] = Cloudinary.config.api_key

  url = "https://api.cloudinary.com/v1_1/"
  url<< "#{Cloudinary.config.cloud_name}/image/upload"

  form_tag(url, form_options) do
    content = []

    options.each do |name, value|
      content<< hidden_field_tag(name, value)
    end

    content<< capture(&block)

    content.join("\n").html_safe
  end
end

Its just a wrapper around form_tag that adds few hidden fields and calculate request signature.

The usage is really simple:

<%= cl_form_tag CALLBACK_URL_HERE,
  form: { class: 'cloudinary' },
  format: 'jpg',
  transformation: { size: '300x300', crop: 'fit' } do %>

  <%= file_field_tag :file %>
  <%= submit_tag "Upload" %>
<% end %>

When user picks file and clicks Upload button, file will start uploading to Cloudinary (completely bypassing your server). Once the upload process is complete, you’ll be redirected back to specified callback url.

cl_form_tag is calculating signature and including it into hidden field. It calculates signature based on your Cloudinary.config.api_scret option. The same thing happens on Cloudinary server when it responds. Thats why you always need to validate response signature to make sure that the Cloudinary itself made that request to callback url. Here is simple method that can do this check:

def valid_cloudinary_response?
  received_signature = request.query_parameters[:signature]
  calculated_signature = Cloudinary::Utils.api_sign_request \
    request.query_parameters.except(:signature),
    Cloudinary.config.api_secret
  received_signature == calculated_signature
end

That’s it! Pretty simple but powerful. There is bunch more that Cloudinary can do, so check the documentation for more details.

Milovan Zogovic

Benchmarking gem load times

Monitoring memory usage on long-running scripts

More readable specs with dynamic date methods

Super fast way of launching postgres console for rails apps

Optimizing postgresql query for DISTINCT values

Rails consecutive migrations acting weird

Showing git branch in fish shell prompt

"42 is a nice number that you can take home and introduce to your family"

Attachinary v1.2.0 released

Changelog

How to validate Array fields in Mongoid

I've just released v1 of attachinary gem for cloudinary with monoid support and much cleaner integration.

Integrating Prefixr with Rails asset pipeline

Cachex: tag based fragment caching on steriods

Testing API from Rails Console

Redis Marshaling

Simple HTML5 details polyfill

Responsive Images with Cloudinary on Rails

Running time in Rails logs

Highlighting navigation links with javascript

Uploading directly to Cloudinary, bypassing your Rails app