Milovan Zogovic

Redis Marshaling

If you have time consuming method in your app that you call frequently, chances are that you’d like to cache it. Redis is great key-value store that can be used for this purpose. However, redis only store string values. In order to store other kinds of objects, you’ll have to use marshaling.

There is a built-in ruby class called Marshal that does the job well. The only thing you have to do is to call Marshal.dump method and pass it an object. The exact opposite is the Marshal.load method.

I’ve created helper method for easier integration with Redis. Here is how it looks:

module RedisHelper

  # Returns the cached value for given key.
  # If it doesn't exist, it evaluates the block and caches what it returns.
  # 
  # @param [String] key the cache key
  # @param [Hash] options
  # @option options [Integer, nil] :ttl time to live in seconds
  # @option options [Boolean, false] :marshal whether to perform marshaling
  # @yield [] to be evaluated if the key does not exist
  def self.fetch(key, options={}, &block)
    raise ArgumentError.new("Must Supply Block") unless block_given?
    return yield unless Rails.application.config.action_controller.perform_caching

    val = nil

    if REDIS.exists(key)
      val = REDIS.get(key)
      begin
        val = Marshal.load(val) if options[:marshal]
      rescue TypeError
        val = nil
      end
    end

    if !val
      val = yield

      if options[:marshal]
        REDIS.set key, Marshal.dump(val)
      else
        REDIS.set key, val
      end

      REDIS.expire key, options[:ttl] if options[:ttl]
    end

    val
  end

end

There is not much to describe here since its already documented using yard. Instead, I’ll show how I used in in one of my apps:

module Geocoder

  def self.geocode(location)
    key = "geolocations/#{location}"
    RedisHelper.fetch key, ttl: 24.hours, marshal: true do
      Geokit::Geocoders::GoogleGeocoder.geocode(location)
    end
  end

end

I’m using geokit gem for discovering geocoordinate of an address. It uses Google geocoding API which is limited to 2500 queries per day. This piece of code will cache the result of GoogleGeocoder.geocode method so all successive requests will be returned from cache.

Simple HTML5 details polyfill

Although there are many HTML5 <details> polyfills already available, I didn’t like the complexity behind them. Here is how you can create your own from scratch in just few lines of css and js.

The idea was to put as much “logic” as possible in the stylesheet itself. It should hide/show the content based on the presence of open attribute. Here is the sass file:

.no-details details {
  > * {
    position: absolute;
    visibility: hidden;
  }

  > summary, &[open] > * {
    position: static;
    visibility: visible;
  }


  > summary {
    display: block;
    &:before {
      content: "►";
      padding-right: 5px;
      font-size: 11px;
    }
  }

  &[open] > summary:before {
    content:"▼"
  }
}

All children of details tag (except summary) will be by default hidden. If details element has open attribute set, its content will be shown. We could use css :not selector to simplify the css a whole lot, but it is only supported by IE9+, and I’d like polyfill to target IE8+.

And here is the coffeescript:

jQuery ->
  return if Modernizr.details

  $(document).on 'click', 'summary', (event) ->
    $summary = $(this)
    $details = $summary.parent()

    if $details.attr('open')
      $details.removeAttr('open')
    else
      $details.attr('open', 'open')

It simply toggles the open attribute when user clicks on summary. Its as simple as that.

Responsive Images with Cloudinary on Rails

Responsive Images is a hot topic lately. There are few solutions to the problem: introduction of new html element (which will take years to be ready for use), new image format (which may take even more time), and using other technologies to solve the problem. Here, I’ll focus on the last one.

The goal is clear:

1. Detect user’s device screen resolution
2. Serve appropriate images for best browsing experience

In order to handle cases where you cannot detect user’s device capabilities (e.g. no javascript support), we need to provide reasonable defaults (e.g. medium sized images).

Here is how you can accomplish this in few simple steps by using Cloudinary service for hosting your assets.

The concept is simple:

1. When user enters our website, we’ll use javascript to detect his device resolution.
2. We’ll store that resolution in a cookie.
3. Backend will utilize information inside cookie to serve appropriate image format.

First two steps are quite easy to do. All you have to do is inject following javascript

document.cookie='resolution='+screen.width+'x'+screen.height)+'; path=/';

That would store user’s device screen resolution in a cookie. On the next page load, that cookie will be available to your backend.

Now, lets override cloudinary cl_image_tag to allow :responsive option. You can do so by including this in application_helper file:

def cl_image_tag(source, options = {})
  if (sizes = options.delete(:responsive)) && (resolution = cookies[:resolution])
    res = case resolution.split('x').first.to_i
          when 0..480 then :phones
          when 481..767 then :tablets
          else :desktops
          end

    size = sizes[res]
    options[:size] = size if size
  end

  super(source, options)
end

Method simply checks whether :responsive option and :resolution cookie are present. If so, it replaces the :size parameter with appropriate value. The usage would be like this:

<%= cl_image_tag "sample.jpg", size: '200x200', crop: 'fit', 
    responsive: { phones: '50x50', tablets: '100x100' } %>

That would serve 50x50px image for phones, 100x100px version for tablets and fallback to 200x200px version for all others. Sweet!

Running time in Rails logs

Since Rails 3.2, you can add custom tags to your logs. You can define tags by setting config.log_tags in your config file. It accepts array of symbols that matches methods of request object. Additionally, you can also pass lambdas to form custom tags. Here is how it might look like:

config.log_tags = [:uuid, :remote_ip, lambda { |req| Time.now }]

If you want to lear more about logging, I highly recommend The Logger (revised) screencast by Ryan Bates.

One gotcha here: lambdas are evaluated only once at the very beginning of request delegation (in Rails::Rack::Logger middleware. So, our lambda from above will return the same time for every log entry within same request.

I must admit I’m obsessed with performance lately :) Thats why I wanted to customize my logs to include “running time” next to each log entry. I wanted it to look something like this:

Started GET "/users/sign_in" for 127.0.0.1 at 2012-04-15 11:37:21 +0200
[time:0.008] Processing by Devise::SessionsController#new as HTML
[time:0.019]   Rendered devise/_links.erb (0.5ms)
[time:0.019]   Rendered devise/sessions/new.html.erb within layouts/application (7.5ms)
[time:0.022]   Rendered layouts/_navbar.html.erb (0.4ms)
[time:0.022]   Rendered layouts/_skeleton.html.erb (2.3ms)
[time:0.022]   Rendered layouts/_public.html.erb (2.8ms)
[time:0.024] Completed 200 OK in 16ms (Views: 13.5ms | ActiveRecord: 0.0ms)

Let’s get started. First, we need to define custom log tag in our development.rb config file:

config.log_tags = [ lambda { |req| "time:#{Time.now.to_f}" } ]

This will prefix each log entry with tag like this [time:1334480797.098824]. We can later use regexp to replace this with running time.

Next, lets incude some regexp magic in TaggedLogging.tags_text method. Since this is something we only need in development, we can define this at the very bottom of development.rb file.

class ActiveSupport::TaggedLogging
  def tags_text_with_timestamp
    text = tags_text_without_timestamp
    text && text.gsub(/(?<=time\:)([\d\.]+)/) do |m|
      "%#.3f" % (Time.now.to_f - m.to_f).round(3)
    end
  end
  alias_method_chain :tags_text, :timestamp
end

Note: this code will work only in Rails 3.2.x. I’ve noticed that TaggedLogging class is currently being refactored by rails team, but it won’t be too hard to make this work in future versions of rails.

Highlighting navigation links with javascript

Recently I blogged about Tagging the HTML tag. The general idea is to apply some data- attributes to html tag in order to more easily target the page from css and js files. I also blogged about Javascript Processors, technique to keep your “DOM enhancers” organized.

Using the two, here is how you can very easily implement processor that would highlight active link in your navigation. Lets say that we have following nav element:

<nav class="main">
  <a class="home">Home</a>
  <a class="projects">Projects</a>
  <a class="account">My Account</a>
</nav>

Here is how our NavHighlighter may look like:

@NavHighlighter =
  mapping:
    "nav.main .home":      [ /^\/welcome/ ]
    "nav.main .projects":  [ /^\/projects/ ]
    "nav.main .account":   [ /^\/registrations\/(edit|update)/, /^\/profile/ ]

  process: ->
    for elem, paths of @mapping
      $(elem).addClass("active") if App.onPath.apply(App, paths)

We’re using mapping hash to define highlight conditions for each link. Hash key is just css selector, so you can use anything you want. Hash value is an array representing list of paths for which the given link is active.

Can’t we do it on server? Sure. There are two reasons I’m doing it on the client: cleaner and faster erb template, and fragment caching.

What about users with js disabled? Well, they could probably live without highlighted menu items. If you really want to do this on server and still use caching, read more on Cache Personalization.

Uploading directly to Cloudinary, bypassing your Rails app

IMPORTANT UPDATE: cl_form_tag and valid_cloudinary_response? are now part of cloudinary 1.0.6+ gem, so you can use them “out of the box”.

For those of you who haven’t heard about Cloudinary, check it out! Cloudinary provides everything you (as developer) would ever need for managing images. It basically works like this: you upload an image, then you request transformation of that image from the server. The transformed image is generated on the fly (unless cached) and served to you. There are plenty of transformations like: resizing, cropping, setting quality, changing file format… etc. And, if your designer ever wanted to scale thumbnails a bit, you wouldn’t need to reprocess thousands of images again!

Cloudinary has a ruby gem that can be used in Rails app and it even supports CarrierWave if you use that to manage your uploads. However all these approaches require your server to handle file upload process. Here is how it looks:

1. User uploads a photo to your server
2. Rails application receives the file
3. Rails application sends the file to Cloudinary

Rails is the middleman and it needs to do some relatively heavy processing in order to decode the data received from the user, and prepare it for upload to the Cloudinary. From scaling aspect, this is something you’d probably like to avoid.

It turns out that it is really easy to let your users upload files directly to Cloudinary. After reading some documentation and browsing through source code, I came up with cl_form_tag helper method:

def cl_form_tag(callback_url, options={}, &block)
  form_options = options.delete(:form) || {}
  form_options[:method] = :post
  form_options[:multipart] = true

  options[:timestamp] = Time.now.to_i
  options[:callback] = callback_url
  if options[:transformation]
    options[:transformation] = Cloudinary::Utils.
      generate_transformation_string(options[:transformation])
  end

  options[:signature] = Cloudinary::Utils.
    api_sign_request(options, Cloudinary.config.api_secret)

  options[:api_key] = Cloudinary.config.api_key

  url = "https://api.cloudinary.com/v1_1/"
  url<< "#{Cloudinary.config.cloud_name}/image/upload"

  form_tag(url, form_options) do
    content = []

    options.each do |name, value|
      content<< hidden_field_tag(name, value)
    end

    content<< capture(&block)

    content.join("\n").html_safe
  end
end

Its just a wrapper around form_tag that adds few hidden fields and calculate request signature.

The usage is really simple:

<%= cl_form_tag CALLBACK_URL_HERE,
  form: { class: 'cloudinary' },
  format: 'jpg',
  transformation: { size: '300x300', crop: 'fit' } do %>

  <%= file_field_tag :file %>
  <%= submit_tag "Upload" %>
<% end %>

When user picks file and clicks Upload button, file will start uploading to Cloudinary (completely bypassing your server). Once the upload process is complete, you’ll be redirected back to specified callback url.

cl_form_tag is calculating signature and including it into hidden field. It calculates signature based on your Cloudinary.config.api_scret option. The same thing happens on Cloudinary server when it responds. Thats why you always need to validate response signature to make sure that the Cloudinary itself made that request to callback url. Here is simple method that can do this check:

def valid_cloudinary_response?
  received_signature = request.query_parameters[:signature]
  calculated_signature = Cloudinary::Utils.api_sign_request \
    request.query_parameters.except(:signature),
    Cloudinary.config.api_secret
  received_signature == calculated_signature
end

That’s it! Pretty simple but powerful. There is bunch more that Cloudinary can do, so check the documentation for more details.

Cache personalization in Rails

Server side caching is a great way to dramatically speed up the serving of your pages. The easiest way to implement caching in your rails app is to use key-based fragment caching.

For those unfamiliar with this concept, take a look at How key-based cache expiration works on 37signals blog.

The all-new-basecamp (kinda Apple’ish, but I like to call it that:) utilizes this technique heavily. As a result they have page lads faster than the blink of the eye, and who wouldn’t like that? Since pages are served much faster, web server can serve much more visitors (lower hosting price). On the other hand, you’ll need huge amount of RAM, which will probably compensate for the former benefit.

Snappy page loads remove the need for custom ajax, which simplifies the app’s internals greatly, which in turn makes it easier to test.

You start developing your app with this approach, and everything works great and fast until you read the following on your feature list:

Display unread messages in bold

Although your first idea may be to wrap the cached fragment with a div tag and conditionally apply unread attribute on the wrapper. This could work in most simple cases, but it is wrong:

• n+1 queries are likely to happen - unless you use association preloading which i really hate since it hurts my OOP fealings (more on that in future post:)
• Russian doll caching wont work
• wrapping things with divs can’t be a good thing, right? :)

I refer to this problem as cache personalization, and there are 2 ways it can be handled.

Client-side cache personalization

This is usually the best option (if it is possible for given scenario). Lets consider this request:

Replace username with you for current user

This can be solved fairly easily. Instead of simply outputing the user’s name, we can output this:

<span class="username" data-id="4">zogash</span>

Later we can create simple javascript that could take all usernames, and if the data-id matches the id of the current user (which we can store as meta tag or something), then replace it with the “you”. We can even apply some classes to it and style it in different way. So, after being processed with some js, the previous snippet should look like:

<span class="username current" data-id="4">you</span>

However, when dealing with more complex requirements, which, for example, involve querying the database, then we must fallback to server-side cache personalization.

Server-side cache personalization

If you can’t make it work on client side, you can always resort to your all-mighty-and-powerful server. Even though following may smell like a wrong thing to do, please stick with me and see all the benefits of this approach.

The general idea is to have some kind of response filter. A thing that would take the generated HTML, manipulate it in a way (like we do with jQuery), and return the outcome to the user. We’re doing this all the time with jQuery, why can’t we do it on the server too?.

Ok, so first, how do we get the generated HTML? The simplest solution is to hook after_filter to your ApplicationController. It would look something like this:

class ApplicationController < ActionController::Base
  after_filter ResponseFilter, 
    if: lambda { |c| c.response.content_type == 'text/html' }
  # ...
end

Idealy you should create a Rack app for this purpose, but for simplicity sake, lets stick with after_filter.

Now, we need to define the ResponseFilter class. If you refer to the docs it needs to implement the filter class method which retrieves the controller object as only argument. You can access the controller.response.body and manipulate it in any way you want.

I won’t go into details here, but the previous example could be done on the server the same way we did it on client. The only difference is that we wouldn’t use jQuery to manipulate the DOM, but nokogiri to manipulate the html document.

One benefit of this approach is that we can analyze the entire page, and take all we need from database table in one query. For example, if there are 10 articles on the page, we can grab their ids from the DOM, fire single query to readals table to find out which ones are unread, then apply “unread” class on them.

I tend to organize these response filters as separate files in /app/filters directory. I have one special filter that acts as a collection and applies chain of filters on response. This proved to be very fast and efficient.

Special Case: Lazy Cache Evaluation

Lets consider the following cached html fragment:

<% cache ['v1', post] do %>
    <article class="post">
        <%= render post.author %>
        <%= post.body %>
    </article>
<% end %>

The author’s partial is also cached. It contains its avatar and name. Now, what happens if user changes its avatar? If we don’t include the author in the cache key, his old avatar will still be displayed in all posts. If we, however include the author in the cache key for the post, we’ll need to make 2 queries when calculating the cache key: one for the post, and one for the author. If we have multiple users displayed in an article (e.g. reviewer, publisher.. etc) this would tend to get messy. And if we start using russian doll caching concept, it would almost be impossible to track all this.

The concept i came up with can be called “lazy cache evaluation”. Instead rendering the author’s partial and including it inside the post’s cached fragment, we could use this instead:

<% cache ['v1', post] do %>
    <article class="post">
        <article class="user" data-id="<%= post.author_id %>" data-lazy />
        <%= post.body %>
    </article>
<% end %>

Now, we could create new response filter that would get all articles with data-lazy attribute and replace them with rendered partials. Once again, since user partials are also cached, render method would simply read those from cache. The benefit from this approach is that:

• no need to specify user as part of cache key
• smaller caches
• all users mentioned on entire page loaded with single query
• partial is rendered only once per user (and applied in several places)

Tagging the HTML tag

Before rails asset pipeline was introduced, I was loading bunch of separate css and js files on every page. Depending on my needs, I would load different set of assets. For example, when I am on the articles/index path, I loaded articles.css and articles.js files (among default ones).

Asset pipeline changed things (for better, of course). Now, all assets are bundled into single application.css and application.js files. This approach was good for a lot of reasons including less requests made to the server (load once, use everywhere) and higher compression rate (when using gzip) all resulting in faster page loads and better ux.

I like to keep things organized. When styling specific page, I want to be very explicit on what I’m styling. Likewise, when writing a script, I want to be very specific when that script should run. Thats why I decided to tag every page in a way that I can reference that info within my css and js files.

In past I tried to do this with html classes, but html5 data- attributes now seem like a way to go.

The idea is to have this:

<html data-controller="articles" data-action="index">
    ...
</html>

This is very easy to achieve by creating custom rails helper method:

module ApplicationHelper
  def html_tag(attributes = {})
    attributes[:data] ||= {}
    attributes[:data][:controller] = controller_name
    attributes[:data][:action] = action_name
    tag(:html, attributes, true)
  end
end

And in the application.html.erb layout file, we simply include

<!DOCTYPE html>
<%= html_tag lang: 'en' %>
  <head>...</head>
  <body>...</body>
</html>

Although a bit controversial, I prefer to generate only <html> opening tag with html_tag method in order to avoid wrapping entire layout file in a block.

Now, if we want to set #content’s background color to yellow only for the articles/index page, we could do this:

html[data-controller=articles][data-action=index] {
  #content { background-color: yellow; }
}

Likewise, we could do this in our coffeescript:

jQuery ($) ->
  return unless $('html').data('controller') == 'articles'
  alert "Welcome to Articles controller"

We can easily add javascript page detection helper method to our App module to simplify this process. It can look something like this:

@App =
  data: (name) -> $('html').data(name)
  controller: -> @data('controller')
  action: -> @data('action')
  path: ->
    p = []
    p.push @controller()
    p.push @action()
    "/" + p.join("/")
  onPath: ->
    for arg in arguments
      return true if @path() == arg
    false

We can rewrite the previous script:

jQuery ($) ->
  return unless App.onPath '/articles/index'
  alert "Welcome to Articles controller, index action"

We can store whatever we like in these attributes. I store vars like module name, current environment, current user’s id and even the public api keys (e.g. for google maps).

Deferring Google API

I just love jQuery Deferreds. There are so many things that can be greatly simplified with this concept.

Here is how we can write simple wrapper around google api to make it less painful to load and work with.

@GoogleAPI =
  dfd: null,

  load: ->
    script = document.createElement("script")
    script.src = "https://www.google.com/jsapi"
    script.src+= "?key=API-KEY"
    script.src+= "&callback=GoogleAPI.loading_complete"
    script.type = "text/javascript"
    document.getElementsByTagName("head")[0].appendChild(script)

  loading_complete: ->
    GoogleAPI.dfd.resolve()

  loaded: ->
    if !GoogleAPI.dfd
      GoogleAPI.dfd = $.Deferred()
      GoogleAPI.load()
    GoogleAPI.dfd.promise()

First time GoogleAPI.loaded method is called, it will return the promise object and start loading jsapi script by appending it to document head. Once the script is loaded, the promise will be resolved and the callback will fire.

The beauty of this is that you can use these promises anywhere you want regardless of whether jsapi is loaded, is loading, or is yet to load. As soon as its available, all callbacks will fire.

Another cool thing is that Google jsapi will never load if its not required by your code.

The usage is really simple. It goes something like this:

$.when(GoogleAPI.loaded()).then -> console.log("jsapi loaded!")

It really reads nice: when GoogleAPI is loaded, then do the dance :)

Whats even more cool is that we can stack such modules one on top of the another. This is how GoogleMaps module would look like:

@GoogleMaps =
  dfd: null,

  load: ->
    google.load 'maps', '3', 
      callback: GoogleMaps.loading_complete, 
      other_params: "sensor=false"

  loading_complete: ->
    GoogleMaps.dfd.resolve()

  loaded: ->
    if !GoogleMaps.dfd
      GoogleMaps.dfd = $.Deferred()
      $.when(GoogleAPI.loaded()).then -> GoogleMaps.load()
    GoogleMaps.dfd.promise()

This one will load GoogleAPI first, then maps package. Once both load, you’re good to go. Once again, usage is real candy:

$.when(GoogleMaps.loaded()).then -> console.log("maps loaded!")

We can now even create GoogleCharts module with similar implementation. The awesome thing is that it won’t reload GoogleAPI if GoogleMaps already loaded it.

It all just works, and it works nicely!

JavaScript Processors

Often times we use jQuery plugins and other scripts to enhance the user experience. These plugins are usually initialized when the page loads and they process html and add some fancy new behavior. But when we start adding content via ajax, things start to get messy.

Some of the plugins can be developed to use live event handlers and that they can handle the problem elegantly (with more or less performance impact), but most of them cannot.

General idea is to have these blocks of code declared once and re-applied whenever needed. To do so, we need all of them to share same interface. Lets call these blocks processors and created a module for each of them. Here are few written in CoffeeScript:

@TimeagoProcessor =
  process: ->
    $('time.ago').timeago()


@SimpleTooltipsProcessor =
  process: ->
    $('[data-tooltip]').qtip
      position:
        at: 'bottom center'
        my: 'top left'

Next, we need to store these in a collection, so that we can run them all once the page loads, and after every ajax call. For that purpose, lets create new module called App.

@App =
  processorData: []

  registerProcessor: (processor, options) ->
    @processorData.push { processor: processor, options: options }

  bindEventHandlers: ->
    $('body').ajaxComplete => @runProcessors(ajax: true)

  runProcessors: (options) ->
    for data in @processorData
      if !options || options.ajax && data.options.ajax
        data.processor.process()

  init: ->
    @runProcessors()
    @bindEventHandlers()

The App enables us to register new processors through registerProcessor method. We can also specify other custom options for each processor. Currently, only ajax option is implemented to denote whether processor should run after each ajax call, but other options can be added easily.

Finally, lets hook it up:

jQuery ($) ->
  App.registerProcessor TimeagoProcessor,        ajax: true
  App.registerProcessor SimpleTooltipsProcessor, ajax: false
  App.init()

That’s it! We no longer have to worry about initializing various plugins and other blocks of code after every ajax call.