Presented by: Daniel Lucraft & Matt Wynne
De-normalisation is just one of many strategies to cope with the scaling requirements of an application. It is the process of devising a method to group relevant data together in an optimal way for the system to return what is required quickly.
During the talk Daniel and Matt explained how Songkick has devised a series of classes to implement a de-normalisation strategy using a key value store in a document based database (MongoDB).
They use the Model View Presenter (MVP Presenter first) design pattern to serve up content for the views on the website. Presenters are first class domain objects tailored for displaying a single page or module of information. So when looking at information on the site you are actually seeing content returned from a document store through the Presenter.
This however presents it’s own bunch of problems. The first and probably most obvious is consistency. Unlike caching, content is not expired and then re-generated on demand, content will be regenerated as and when it is created or updated.
Presenter, Soli and Silovator
A presenter is basically a model which has methods to get only the data required for the view element and store these in the specific document. A typical presenter looks like this:
class EventListingPresenter
def initialize(event)
@event = event
end
def title; @event.artists.map(&:name).to_sentance; end
def image_count; @event.images.count; end
def attendance_count; @event.attending_users.count; end
#silo creates document on save using silo methods
silo_method :title
silo_method :image_count
silo_method :attendance_count
ends
The silo_method works in a similar way to memoization. Data will attempt to return from the document store and failover to using the actual database through ActiveRecord.
Cache control
Cache expiration is controlled through asynchronous observers called Silovators. These silovators listen for events like create, update and delete on objects and run processes to generate the presenter data for any given presenter. For example:
class EventListingSilovator < AsyncObserver
listen :create, Attendance do
Silo.generate(presenter_for(attendance.event))
end
listen :update, Venue do
venue.events.each do |event|
Silo.generate(presenter_for(event.venue))
end
end
end
The silovator processes could be running on any number of machines, which creates another problem with possible race conditions. This is solved using http://github.com/songkick/mega_mutex which will ensure when a job is picked up, no other client will run that job at the same time. Megamutex uses memcache-client so your environment must be setup to use it.
Songkick have multiple background tasks which consume events and perform actions, these include ImageResizer, EmailNotifier and Silovater which all take advantage of mega_mutex to guarantee consistency.
