clog

A theme should allow the look and feel of an application to change. A theme has a one-to-one relationship with the application, meaning that an application should only be running one theme at a time. A theme per user would be a ‘skin’ and an application that runs multiple sites with a different “theme” for each one is a whole new can of worms. I’ve had a need to do the latter recently, but I’ll go over my (site_support) solution to that in the next post. Meanwhile, here’s my solution for adding themes to your application.

When to use it: A single application connects to a single database and has multiple theme options, but only one is running at a time.

Start by installing my hacked version of theme_support

script/plugin install http://www.chrispcritter.com/theme_support_mod/ 

ActionMailer currently doesn’t work with it, I’ll have to tackle a fix for it soon. I’ll post my findings.

At this point you should at least be able to get your poject running in mongrel.

To create the them, you’ll need to create a “themes” folder in your project and start your first theme. The theme must have the following structure:

  $app_root
    themes/
      [theme_name]
        cache/
        images/
        stylesheets/
        javascripts/
        views/           
          layouts/      

You may notice that this is a little different than the structure described in the theme_support documentation. I could not get layouts to work outside of the view path (so I just moved it into views). I really think this is better anyway - since it matches the standard view structure. The layout I describe also has a “cache” folder. The theme_support plugin supposedly supports cached files in a public/themes structure but I had some difficulty getting this to work and really wanted to keep all theme related content in it’s own folder. I ended up setting page caching to go into the themes/[theme_name] folder with some mod_rewrite magic. It may be better to add a “public” folder to each theme for this at some point.

Add a line similar to the following to config/environment.rb (where is your default theme):

THEME = ENV["THEME"] ? ENV["THEME"] : ''

And add the below line to pull out your site specific configuration options.

CONFIG = YAML.load(File.read("#{RAILS_ROOT}/config/#{THEME}.yml"))

If you’re using apache, your rewrite rules will look like the following:

RewriteRule ^([^.]+)$ themes//$1.html [QSA]
RewriteRule ^images/(.*)$ themes/THEME>/images/$1 [QSA]

You will need to modify the caching paths in environment.rb so they point to the right place. Add the following lines:

ActionController::Base.fragment_cache_store = :file_store, "#{RAILS_ROOT}/themes/#{THEME}/cache"
ActionController::Base.page_cache_directory = "#{RAILS_ROOT}/themes/#{THEME}"

Add a link in public that points to your themes directory “public/themes -> ../themes/” That will preserve the routes.

It’s not the most elegant solution but it works. It bypasses some routes functionality and the sym-link in particular is ugly. I’ll post more mods to theme_support as/if I make them along with simpler install instructions, but this was really just a stepping stone to my site_support setup - so it may be awhile before I get back to it.

Caching a highly dynamic application can be hard. Page caching doesn’t work well because some elements of the page need to change where others don’t. Action caching helps in the very few instances where you want all of your before filters to run before loading the cached view. In this instance the before filters are typically redundant and may be doing some heavy lifting that can really slow things down. Fragment caching will cache a certain section of the view or layout but seems somewhat useless at first glance because all of the heavy lifting was already done in the controller.

How do you get around this problem? There are really only three options:

1) Put the necessary code into a model and call it in the view inside a fragment cache. This essentially by-passes the controller and completely breaks MVC rules.

2) Put the necessary code into a helper and call it in the view inside a fragment cache. This works in situations where a helper is necessary, but usually this isn’t what we’re looking for. Calling heavy lifting that’s in helpers from a view is not quite as ugly as calling from a model, but is generally bad design. Helpers should only manipulate existing data, not pull new data.

3) Pass a Proc object from the controller to the view, and call it from inside the fragment cache.

The third option is best. It doesn’t break MVC because the code that gets executed is only executed inside the controller argument. It’s execution is simply delayed until it gets called from within the view. It does dirty your views somewhat and probably won’t work for liquid templates, but it (mostly) follows the rules and gets the job done in a way that is more elegant than the first two options.

This is how you do it.

In this instance, I want to retrieve data that is used for the ‘category_links’ action and view (normally called as a partial). Notice that the Event model contains the actual code to retrieve the data. Recent trends in how to separate model and controller functionality have convinced me this is the way to go. Also note that the controller still lies between the view in model in this setup. See: skinny-controller-fat-model

def category_links
  # Set up a Ruby Proc method to pass (usually to a view).  This 
  # method is defined here but cannot be called until @get_event_categories.call 
  # is used.  This allows the logic execution to be deferred until view 
  # time, but it is defined here (where it should be).  This doesn't 
  # break MVC rules while maintaining a clean syntax.  See Ruby Proc Class    
  @get_event_categories = lambda do
    categories = Event.categories.collect {|c| [c.category, c.event_count, {
      :controller => 'events',
      :action => 'list',
      'event[category]' => c.category,
      'event_date[start_date]' => date_range_start,
      'event_date[end_date]'   => date_range_end}]}

    sub_categories = Event.sub_categories.collect {|c| [c.sub_category, c.event_count, {
        :controller => 'events',
        :action => 'list',
        'event[sub_category]' => c.sub_category,
        'event_date[start_date]' => date_range_start,
        'event_date[end_date]'   => date_range_end}]}  

    return categories, sub_categories
  end
end

@get_event_categories is defined as an object proc variable that contains the code needed to retrieve the categories and sub_categories variables. None of the code was actually executed. Ruby has just stored the code and the environment needed to run it in @get_event_categories.

For completeness sake, here is what gets called above via the Event API:

def categories
  conditions = Event.theme_filters

  find_hash = {}
  find_hash[:select] = 'events.id,category,count(events.id) AS event_count'
  find_hash[:conditions] = conditions if conditions && conditions != ''
  find_hash[:group] = 'category'
  find(:all, find_hash)      
end

def sub_categories
  conditions = Event.theme_filters
  conditions += ' AND ' if conditions && conditions != ''

  find(:all,
    :select => 'id,sub_category, count(events.id) AS event_count',
    :conditions => "#{conditions} sub_category != ''",
    :group  => 'sub_category')      
end

Here is the view code for category_links:

<% categories, sub_categories = @get_event_categories.call%>
<% categories.each do |category, event_count, cat_params| %>
  <%= link_to_remote category, {
    :url => cat_params,
      :loading => "Element.show('status')"}, {
    :href => url_for(cat_params)} %>
  (<%= event_count %>)
    

<% end %>

<% sub_categories.each do |sub_category, event_count, sub_cat_params| %>
  <%= link_to_remote sub_category, {
    :url => sub_cat_params, 
      :loading => "Element.show('status')"}, {
    :href => url_for(sub_cat_params)}%>
  (<%= event_count %>)
    

<% end %>

The @get_event_categories proc method is called first and the result is copied into categories, and sub_catagories variables. These are then used to generate the actual links.

No caching you say? Since these are almost always called as partials, I don’t cache here. It is assumed that a direct call wants a non-cached version.

My event layout contains the following rail section:

<% cache(:action_suffix => "rail_links")  do %>    
  

    

        <%= render :partial => 'price_links', :layout => false %>
    
    

      Browse by date
    
            
    

      <%= render :partial => 'date_links', :layout => false %>
    
    

      Browse by category
    
  
    
                   
        <%= render :partial => 'category_links', :layout => false %>
    
    

      Browse by location
    
  
    
                      
      <%= render :partial => 'location_links', :layout => false %>
    
    

      Browse by venue
    
          
    
          
      <%= render :partial => 'venue_links', :layout => false %>
    
  

<% end %>

Notice that the entire rail gets cached, and the category links (generated with our proc call) are among them. All of the other links have similar proc calls. I have a before_filter in my controller that retrieves the proc objects before going to the view (for the appropriate actions). If the cache is hit, then the proc methods are never called and no time is wasted on the expensive data pulls and calculations. You may notice that the category link calls are fairly expensive (they group and count the results, which adds a lot of time to the query). The other links are just as expensive. Doing this literally saves seconds on the page load speed.

So there it is. Try it out and marvel at the newfound speed of your application!

Perhaps I tried this before and it didn’t work for me, but through some trial and error I figured out how to do nested includes in Rails. This means you can join a join of a joined table without ever having to mess with the join syntax. Of course you also get all of the other cool benefits of using Rails relationships.

My previous article on the subject touched on using single nested includes, but didn’t touch on multiple nesting. Here’s how you do it.

Take the following classes and relationships:

class A < Base
  has_many :B
  #relationship to C via the join table (B)
  has_many :C,  :through => :B
end

class B < Base
  belongs_to :A
  has_many :C
  has_many :D
end

class C < Base
  belongs_to :B
  belongs_to :D,
    :include => :A
end

class D < Base
  has_many :C
  has_many :B
    :through => :C,
    :include => :D
end

Say you want to search on A, and include B, C, D. All you need to do is:

A.find(:all, :include => {:B => {:C => 'D'}})

You simply keep embedding Hashes for each relationship. You can apparently do this indefinitely. So also joining an ‘E’ model that belonged to D would require:

A.find(:all, :include => {:B => {:C => {:D => 'E'}}})

Just another one of the really nifty and less advertised features of Rails.

My Rails 1.2 upgrade could have gone a lot better. At first everything seemed fine. Then I tried setting up a new test environment (which involves using ar_fixtures). The fixtures were created fine, but when I tried to test with them, I got the following.

TypeError: BigDecimal can't be coerced into BigDecimal

So after perusing I noticed that the Rails 1.2+ar_fixtures is outputting an empty BigDecimal object (I think it used to just output 0.0). Well.. that seems legit.. but it’s not working. After doing some extensive googling I don’t seem to uncover many other people having the problem, but I do find this bug reporting the same bug. Low and behold this is a Ruby core library! The ticket has been open for two months, so I think surely it doesn’t still exist… I try it out with irb and get the following:

irb -r 'yaml' -r 'bigdecimal'
irb(main):001:0> YAML::load( BigDecimal.new('1.1').to_yaml )
TypeError: BigDecimal can't be coerced into BigDecimal
        from /usr/local/lib/ruby/1.8/irb.rb:298:in `inspect'
        from /usr/local/lib/ruby/1.8/irb.rb:298:in `output_value'
        from /usr/local/lib/ruby/1.8/irb.rb:151:in `eval_input'
        from /usr/local/lib/ruby/1.8/irb.rb:259:in `signal_status'
        from /usr/local/lib/ruby/1.8/irb.rb:147:in `eval_input'
        from /usr/local/lib/ruby/1.8/irb/ruby-lex.rb:244:in `each_top_level_statement'
        from /usr/local/lib/ruby/1.8/irb/ruby-lex.rb:230:in `loop'
        from /usr/local/lib/ruby/1.8/irb/ruby-lex.rb:230:in `each_top_level_statement'
        from /usr/local/lib/ruby/1.8/irb/ruby-lex.rb:229:in `catch'
        from /usr/local/lib/ruby/1.8/irb/ruby-lex.rb:229:in `each_top_level_statement'
        from /usr/local/lib/ruby/1.8/irb.rb:146:in `eval_input'
        from /usr/local/lib/ruby/1.8/irb.rb:70:in `start'
        from /usr/local/lib/ruby/1.8/irb.rb:69:in `catch'
        from /usr/local/lib/ruby/1.8/irb.rb:69:in `start'
        from /usr/local/bin/irb:13
Maybe IRB bug!!

After going through the source I end up in C libraries and patching is certainly an extreme measure here. So it seems I have to live with the bug. On a hunch I search/replace all of the empty BigDecimal objects to 0.0 and it finally works. But… what’s up with this? This looks like a rather serious bug. I would think Matz himself would be in there fixing it after two months! Maybe I’ll take a shot at fixing it myself…

So… if you get bit by the “BigDecimal can’t be coerced into BigDecimal” bug you can get around it by converting it to string:

irb(main):002:0> YAML::load( BigDecimal.new('1.1').to_s.to_yaml )
=> "0.11E1"

In my case I have to search and replace these fixtures post export, with:

contents = ''
if Event.to_fixture  && 
   File.open('test/fixtures/events.yml', 'r') {|fin| contents << fin.read;} &&
   File.open('test/fixtures/events.yml', 'w') {|fout| fout << contents.gsub('!ruby/object:BigDecimal {}', '0.0'); fout.flush}  

  puts 'Event fixtures created'
else
  puts 'Failed to create Event fixtures'
end

You can probably figure out most workarounds from that.

I posted one way to get rid of the TextMate beachball earlier. It just keeps coming back though. My current project is heavy on themes, meaning I have a lot of cache and images in a theme directory that also contains views and stylesheets that I need to edit. This really began to slow things down for me, and there’s no way to remove subfolders from the project. I finally figured out a way to put the beachball to rest for good though (hopefully).

First follow my previous advice and create a brand new TextMate project and add all folders that contain files that you will edit (and no others). Don’t add the log or tmp directories. Be thorough though - don’t leave out folders that MIGHT contain editable files in the future either.

TextMate will automatically scan all of these folders for filesystem changes, but there is a way to tell it NOT to scan certain subfolders. To do this you select the top-level folder that contains subfolders that don’t need editing (usually image and large static content areas). Right click and select the “Show information” option.

The “Folder pattern” textbox contains a regular expression that filters out unwanted folders. All you need to do is add the name of the folder(s) you want left out. Just add it to the existing list of names that are | separated. Make sure you have a single | on both ends of the folder’s name. You don’t have to put the entire path, just the name.

For example, I don’t want to check image subfolders, so I change this: !./(.[^/]|CVS|_dar to this: !./(.[^/]|images|CVS|_dar

Now my massive images folder is hidden. After doing this for all applicable folders, TextMate is faster than ever! Plus I still get the benefits of the live filesystem check that other methods don’t have.

There’s always a need for importing, exporting, and synchronizing data between various applications, especially in corporate environments.

As data has grown by orders of magnitude in my databases, the amount of resources required to run these processes has shot up dramatically. Many databases are now 10 times larger and exports that once took 2 or 3 minutes now take a good hour or more.

Think about that for a second and you’ll notice something strange. Shouldn’t 10X more data only take 20-30 minutes to export rather than an hour.

I’ve recently had to go back and make changes to some (of my own) legacy code doing such an export and was so aggravated by the excessive run times, I decided to do something about it.

After looking into it, I figured out that the problem I was running into is the excessive use of garbage collection. By pushing gigabytes of data into memory, not only was the OS working overtime to reallocate all of that, the garbage collector was having to constantly run in an effort to keep memory use down. When I tried running the process on my macbook that “only” has 2 gig of ram, the process ate up all 2 gig and virtually starved itself to death. I finally gave up and killed it.

The solution may seem obvious, but I really haven’t come across it before:

Page all data that gets pulled.

I always thought of data paging as something done to improve the user experience and not a method of improving performance when you need to work on all of the data (quite the opposite really). While it does slow things down for a small number of results, it really makes a HUGE difference as your application grows.

As a Rubyist, I now rely on my relatively new friend paginating_find to meet my paging needs. I set up an enumerated version of my export process to pull 1000 record pages at a time. Experiment with results-per-page until you get your desired performance. A lower number of results_per_page will tax your database, while a higher number will tax your web server and increase your memory requirements. I found that at 1000 records the database would carry about 15% of the load, and about 100MB of memory would be used.

What was the performance difference? The process has gone from 60+ minutes per run to just over 7 minutes. Wow!

Joins in SQL queries are notorious for breaking code. If you have to modify how your tables are associated in any way, you’ll most likely have to revisit any joins you’ve written to keep your code from breaking. In a large system this can be A LOT of joins. Additionally, writing queries with multiple joins can be very cryptic and confusing. It’s easy to make mistakes and difficult to see the relationships by looking at the queries.

Rails 1.1 introduced some awesome features that virtually eliminate joins. The has_one, has_many, belongs_to, and has_and_belongs_to_many methods provide a bulk of this.

Unfortunately for me (in a way), I learned Rails in the pre-1.0 era and then migrated my company at 1.0. I was stuck there for some time and got behind on the 1.1 improvements. While catching up, I ran across some things that aren’t so well documented and very handy.

One such technique I came across should be documented. So here it is (mostly for my own good).

In this instance I need a true join table rather than has_and_belongs_to_many. Say you have a relationship structure defined below:

class A < Base
  has_many :B
  #relationship to D via the join table (B)
  has_many :D,  :through => :B
end

class B < Base
  belongs_to :A
  belongs_to :D
  has_many :C
end

class C < Base
  belongs_to :B
end

class D < Base
  belongs_to :B
end

You are searching on class A and want to join all of the matching rows from both B and C. This is a nested has_many relationship (meaning nested joins). How do you do this? It’s not so obvious, but it’s beautiful once you learn. All you need to do to stack up has_many relationships is to stack up (embed) Hashes that get sent to the include argument of the find call. Rails will recursively extract each of these tables and create the nested relationships for you!

This is it… no more ugly nested joins!

A.find(:all, :include => {:B => :C})

There may yet be instances where you need joins, but the occurrences are fewer and fewer these days. The hardest thing now is going to be how to remember SQL. It’s actually easier to do test queries in the console now!