Monthly Archives: July 2018

37 signals was not lying, you win by being remote

Your ads will be inserted here by

Easy Plugin for AdSense.

Please go to the plugin admin page to
Paste your ad code OR
Suppress this ad slot.

37 signals was not lying, you win by being remote

There is such moment in developer’s life when you start looking for a new job, sooner or later. You can observe that even in Poland, there are plenty of Ruby on Rails job offers, often in very perspective companies. I probably could find interesting job in Poznań, where I live, but there were some presumptions which pushed me to apply to Arkency.

Due to my life situation becoming a bit complicated and me feeling totally whacked out, I needed to find workplace where I can feel really comfortable. I wanted it to be a place where I could develop my skills and face new challenges.

Few months ago I have found “webworktravel” facebook page and started thinking that it would be great to have a possibility to work this way. I developed my first commercial project as a remote team member, so I was close to this nominally. Unfortunatelly I was studying simultaenously and it was really hard for me to achieve this. Then I run for three and a half year of “classic” office work. During this, I had some bad experiences in cooperating with some remote workers and I tried to figure why some things gone wrong and whether it is possible to make things right in the future. I started reading articles about remote work and trying to figure out how to handle it properly. Then Arkency released preview of their Async Remote book. I bought it without a falter and read within single breath. Suddenly, 37signals released “Remote”. I was joking that maybe it would be as good as the Arkency one. I started to believe again that remote work could be possible if company has well organised processes to handle this way of working.

When I was looking for a job I received some propositions, mainly with relocation to some other cities in Poland or to Berlin, Germany. Some of the companies looked promising to me, some of them even offered remote work. I was able to relocate, but working remotely was an interesting alternative for me. Even so I felt that I could become some kind of a dropout because of not being in the office and having less contact with the rest of the team. Working remote, but still from 9am till 5pm makes not really big sense for me. In this case, the only value is no need of relocation. Then I realised that there’s a team that works remotely and asynchronously. Asynchrony is the key factor which makes remote working really attractive. It’s really great feature when you can work in your comfort zone, you can go to the doc or plan your working hours to spend more time with your family, especially when your spouse doesn’t have such flexible work.

I have known some of Arkency guys and I have known their skills, care about proper design and engagement in Ruby community. Their articles are often published in Ruby Weekly. That really impressed me, but thing which tipped the scale was their approach to remote work, described in the Async Remote book.

During first day of my new job I was coworking, pair programming and when I was travelling back home from Wrocław, I started writing this blog post. Cool, huh? At last my new company is developer oriented.

Stop including Enumerable, return Enumerator instead

Your ads will be inserted here by

Easy Plugin for AdSense.

Please go to the plugin admin page to
Paste your ad code OR
Suppress this ad slot.

Stop including Enumerable, return Enumerator instead

Many times I have seen people including Enumerable module into their classes. But I cannot stop thinking that in many cases having methods such as each_with_index or take_while or minmax and many others that are available in Enumerable are not core responsibility of the class that is including them itself.

In such case I prefer to go Java-way and provide external Enumerator for those who need to call one of the many useful Enumerable methods on the collection. I think that we need to ask ourselves a question: Is that class a collection?. If it really is then it absolutely makes sense to include Enumerable. If however it is not a collection, but rather a class which happens contain something else, or providing a collection, well then maybe external Enumerator is your solution.

Standard library

If you call the most famous Array#each method without a block, you will see that you get an enumerator in the response.

 e = [1,2,3].each # => #<Enumerator: [1, 2, 3]:each>  

You can manually fetch new elemens:

e.next # => 1   e.next # => 2   e.next #=> 3   e.next # StopIteration: iteration reached an end 

Or use one of the Enumerable method that Enumerator gladly provides for you

 e = [1,2,3].each  e.partition{|x| x % 2 == 0} # => [[2], [1, 3]]  

Create Enumerator

There are 3 ways to create your own Enumerator:

  • Kernel#to_enum
  • Kernel#enum_for
  • Enumerator.new

But if you look into MRI implemention you will notice that both #to_enum and #enum_for are implemented in the same way:

rb_define_method(rb_mKernel, "to_enum", obj_to_enum, -1); rb_define_method(rb_mKernel, "enum_for", obj_to_enum, -1);  rb_define_method(rb_cLazy, "to_enum", lazy_to_enum, -1); rb_define_method(rb_cLazy, "enum_for", lazy_to_enum, -1); 

You can check it out here:

And if you look into rubyspec you will also notice that they are supposed to have identicial behavior, so I guess currently there is really no difference between them

Therfore whenever you see an example using one of them, you can just substitue it with the other.

#to_enum & #enum_for

What can #to_enum & #enum_for do for you? Well, they can create the Enumerator based on any method which yields arguments. Usually the convention is to create the Enumerator based on method #each (no surprise here).

a = [1,2,3] enumerator = a.to_enum(:each) 

We will see it in action later in the post.

Enumerator.new

This way (contrary to the previous) has a nice documentation in Ruby doc which I am just gonna paste here:

Iteration is defined by the given block, in which a “yielder” object, given as block parameter, can be used to yield a value:

Your ads will be inserted here by

Easy Plugin for AdSense.

Please go to the plugin admin page to
Paste your ad code OR
Suppress this ad slot.

fib = Enumerator.new do |y|   a = b = 1   loop do     y << a     a, b = b, a + b   end end  fib.take(10) # => [1, 1, 2, 3, 5, 8, 13, 21, 34, 55] 

The optional parameter can be used to specify how to calculate the size in a lazy fashion. It can either be a value or a callable object.

Here is my example:

polish_postal_codes = Enumerator.new(100_000) do |y|   100_000.times do |number|     code    = sprintf("%05d", number)     code[1] = code[1] + "-"     y.yield(code)   end end  polish_postal_codes.size    # => 100000                              # returned without computing                             # all elements  polish_postal_codes.take(3) # => ["00-000", "00-001", "00-002"] 

Why?

Of course returning Enumerator makes most sense when returning collection (such as Array) would be inconvinient or impossible due to performance reasons, like IO#each_byte or IO#each_char.

What do you need to remember?

Not much actually. Whenever your method yields values, just use #to_enum (or #enum_for as you already know there are identical) to create Enumerator based on the method itself, if block code is not provided. Sounds complicated? It is not. Have a look at the example.

require 'digest/md5'  class UsersWithGravatar   def each     return enum_for(:each) unless block_given? # Sparkling magic!      User.find_each do |user|       hash  = Digest::MD5.hexdigest(user.email)       image = "http://www.gravatar.com/avatar/#{hash}"       yield user unless Net::HTTP.get_response(URI.parse(image)).body == missing_avatar     end   end     private    def missing_avatar     @missing_avatar ||= begin       image_url = "http://www.gravatar.com/avatar/fake"       Net::HTTP.get_response(URI.parse(image_src)).body     end   end end 

We are working in super startup having milions of users. And thousands of them can have gravatar. We would prefer not to return them all in an array right? No problem. Thanks to our magic oneliner return enum_for(:each) unless block_given? we can share the collection without computing all the data.

This might be really usefull, especially when the caller does not need to have it all:

class PutUsersWithAvatarsOnFrontPage   def users     @users ||= UsersWithGravatar.new.each.take(20)   end end 

Or when the caller wants to be a bit #lazy :

UsersWithGravatar.   new.   each.   lazy.   select{|user| FacebookFriends.new(user).has_more_than?(10) }.   and_what_not # ... 

Did i just say lazy? I think I should stop here now, because that is a completely different story.

TLDR

To be consistent with Ruby Standard Library behavior, please return Enumerator for your yielding methods when block is not provided. Use this code

return enum_for(:your_method_name_which_is_usually_each) unless block_given?` 

to just do that.

Your class does not always need to be Enumerable. It is ok if it just returns Enumerator.

Would you like to continue learning more?

If you enjoyed the article, subscribe to our newsletter so that you are always the first one to get the knowledge that you might find useful in your everyday Rails programmer job.

Content is mostly focused on (but not limited to) Ruby, Rails, Web-development and refactoring Rails applications.

Also, make sure to check out our latest book Domain-Driven Rails. Especially if you work with big, complex Rails apps.

Rails and SOA: Do I really need to have this big app?

Developing complex web applications (not particularly Rails apps) is a tricky task. You write your models, controllers, services and optimize application architecture iteratively. But even with the great architecture when your application starts to be huge, bad things happen. Every little feature you’ll add will cost you precious time – tests must pass on your CI and workers needs to be reloaded every time you deploy. In addition, you begin to realize that mental overhead with keeping modules separated is huge – you have to remember a lot of things to keep it in right fashion. Ultimately, working on a project is hard, TDD technique is inefficient to use and you’re not happy at all with your code.

Have something gone wrong? No. You just didn’t notice that you need more applications than just one.

What is Service Oriented Architecture?

Complex applications tend to create whole ecosystems inside of them – we can think of such applications as of galaxies. Inside these galaxies we have stars – our services and adapters. They should be separated, but they’re still in the one galaxy which is our big application. What glues these stars together is their purpose – they’re created by us to solve certain problems.

Service Oriented Architecture takes different approach. We can put the same stars into different galaxies and add an explicit communication between them. This way we create a solid boundaries between our services and make our solution simpler to maintain – working on a small Rails application is easy, right?

Why it’s good?

The most attractive thing about this kind of architecture is an ease of working with smaller applications – you have a small subset of tests and external libraries. Your mini-application has a narrow scope, business domain is singular (as opposed to your previous approach, where you had payments and business logic inside a single app – maybe separated, but still inside one app) and you don’t need to have such sophisticated architecture inside – when you choose Rails to implement it, you can even be fine with your ActiveRecord and tight coupling with a framework.

What is more, when it comes to replication of your app, you have much more control about which part and how much you want to replicate it. You can distribute your API within some kind of CDN or cloud, keeping your data processing app centralized. Small application is not as heavy as your previous monolithic application – you can spawn more workers with the same resources you had before.

As a developer, you would certainly appreciate that with this approach you’re absolutely technology agnostic. With this approach you can create your API app in Rails with a traditional RDBMS like PostgreSQL and payments processing application in Haskell with MongoDB as a persistence layer. It’s your choice and service oriented architecture allows you to have such flexibility.

When you change something in one part of your system, you don’t have to reload all subcomponents – deploys are separated from each other and you have zero downtime with API when you only update a data processing app. That makes your system more reliable and clients happier. When something goes wrong and one subsystem hangs, you can defer message passing to it and go on – before you had single point of failure, now your system is much more durable.

You can define or choose protocols you choose, not just HTTP – you can tailor your message passing to suit your needs. You can provide reliable or unreliable communication, use different data formats – it’s your choice. When it comes to optimalisation, it’s a huge improvement compared to monolithic Rails app, which is adjusted to work with a HTTP protocol and simple, stateless request-response cycle. In chillout, our application which gathers metrics about creation of a certain models within your Rails app, we use ZMQ sockets for internal communication thorough our system and only use HTTP to get requests from our clients. That allowed us to be flexible about reliability of our transmission. We certainly want to be sure when someone pays us, but we don’t need to be exactly sure that 1 of 100 gathered metric won’t be delivered.

When it comes to coupling, there is less possibilities to couple independent components of your system – in fact, you can’t be more explicit about saying “Hey, now we’re dealing with a completely different domain”!

Of course, there is no perfect solution – and SOA have its flaws.

Nothing is perfect (SOA too)

Unfortunately, with this approach you have to provide code for internal communication between subsystems. Often it would imply that your total codebase (sum of codebase of all subcomponents of your system) will be bigger. To overcome this issue I recommend to put this communication code as an external library, shared between your components.

Another issue is that every application which wants to communicate with a certain subsystem needs to know a message format – thus, knows something about a domain of the receiving app. This can be fixed, but with a cost – you can provide a “mediator” app, which is reponsible for translating a message from a sender domain to (possibly multiple) recievers domain format. It’s nothing new, though – you made it before when you introduced an adapter to your application. This issue induced a nice discussion inside Arkency team, and it’s the only solution we’ve found so far. It’s good, but not as good – we have to provide more code to do so. I would recommend creating a simple adapter first – when you feel it’s not enough, you can easily extract it as a new application.

If you’re ready to pay this price, SOA is a viable architecture to work with your system. And the best part is that…

It’s not all or nothing

Very good thing about SOA is that it’s not all-or-nothing – we can iteratively transform the code from a certain point to the stand-alone mini-application. In fact, when we transform our business logic into services it’s quite simple. Let’s define steps of this extraction, and I’ll provide a simple example.

  1. (If you have not done it before) Create a service object from a given business process inside your app. My previous post can be helpful with this.
  2. Choose a service OR adapter which you want to be a separate app. Copy the code (with dependencies) to a separate directory. Copy external dependencies (gems) used and include it into your brand-new app.
  3. Create the code which processes requests. We can use Rails for it. In my example, I’ll use webmachine for simplicity.
  4. If an action needs to send back some kind of message, introduce the code which creates it.
  5. Inside your old application, change your controller action’s code to the new code which makes communication with your new application.
  6. Remove external dependencies which were exclusive to your service from your complex application. Remove copied code from complex application’s codebase. Move unit service tests (if any) to your new application.

Step one:

Let’s introduce our example. We have a simple service object which processes payment creation requests – it communicates with an external service to delegate the payment processing and handles response from this service.

It’s important to see it’s a boundary context of our application – it’s not tightly related with what our application does and it’s business rules – it’s only providing an additional (but needed) feature. These kind services are the most viable choice for extraction, because we can easily build an abstraction around it (more about this later).

The callback object here is usually a controller in a Rails application. If you’re unfamiliar with this kind of handling outcoming messages from service within Rails controller, here’s an example how the code may look inside the controller:

def action   service = PaymentCreationService.new(PaymentsDB.new, PaymentsProviderAdapter.new, PaymentsMailer.new, self)   service.call(params[:payment]) end  def payment_successful(uuid)   # code processing successful payment request end  def payment_data_invalid(reason)   # code processing invalid data within request end  def payment_unknown_error(payment_request)   # code processing unknown error end 

Here’s how our code might look like:

class PaymentCreationService   def initialize(payments_db, payments_provider_adapter, payments_mailer, callback)     @payments_db = payments_db     @payments_provider_adapter = payments_provided_adapter     @payments_mailer = payments_mailer     @callback = callback   end    def call(payment_request)     payment = payments_provider_adapter.request(payment_request)      if payment.accepted?       payments_db.store(payment.uuid, payment_request)       payments_mailer.send_confirmation_of(payment_request)       callback.payment_successful(payment.uuid)     end      if payment.data_invalid?       callback.payment_data_invalid(payment.reason)     end      if payment.unknown_error?       callback.payment_unknown_error(payment)     end   end    private   attr_reader :payments_db,                :payments_provider_adapter,               :payments_mailer,                :callback end 

We put this file (with it’s dependencies, but without a callback object – it’s not a dependency!) to the separate directory.

Step two:

Here we create the code which processes our requests. It’s a lie it’s only for a request processing – in our example it’s also setting up a HTTP server – but it’s all about a protocol. We use HTTP, so we need a HTTP server for this. We can use many technologies – like raw sockets, ZMQ and such.

You can really skip this code if you don’t want to learn about webmachine internals. In short it creates HTTP server which processes /payments POST calls and binds methods read and render appropiate JSON to it. Since webmachine is really small compared to Rails, we have to create resources (we can think about it as controllers) by ourselves.

require 'webmachine' require 'multi_json'  require 'payment_creation_service' require 'payments_db' require 'payments_provider_adapter' require 'payments_mailer'  class ResourceCreator   def ecall(route, request, response)     resource = route.resource.new(request, response)     service = PaymentsCreationService.new(PaymentsDB.new,                                            PaymentsProviderAdapter.new,                                            PaymentsMailer.new,                                           resource)     resource.payments_creation_service = service     resource   end end  class PaymentResource < Webmachine::Resource   attr_accessor :payments_creation_service,                 :rendered_data    def allowed_methods     %w(POST)   end    def content_types_accepted     [['application/vnd.your-app.v1+json', :accept_resource]]   end    def content_types_provided     [['application/vnd.your-app.v1+json', :render_resource]]   end    def accept_resource     body = MultiJson.load(request.body.to_s)      payments_creation_service.call(body)   end    def payment_unknown_error(payment)     self.rendered_data = MultiJson.dump(message: "UNKNOWN_ERROR",                                         inspect: payment.inspect)   end    def payment_data_invalid(reason)     self.rendered_data = MultiJson.dump(message: "DATA_INVALID",                                         reason: reason)   end    def payment_successful(id)     self.rendered_data = MultiJson.dump(message: "SUCCESS",                                         uuid: id)   end    def render_resource     self.rendered_data   end end  @webmachine = Webmachine::Application.new do |app|                 app.routes do                   add ['payments'], PaymentResource                 end                  app.configure do |config|                   config.adapter = :Rack                   config.ip      = '127.0.0.1'                   config.port    = 5555                   config.adapter = :Webrick                 end                  app.dispatcher.resource_creator = ResourceCreator.new               end                  @webmachine.run 

Step three:

We did it already. The #render_resource, #payment_data_invalid, #payment_unknown_error and #payment_successful methods is the response creation code.

All it does is providing an interface for a service and creating a JSON response based on callback service calls. When it gets bigger, I recommend putting response creation code into a separate object.

Step four:

Now we have to change our old controllers code. It can now look like this, using Faraday library:

def action   response = Faraday.post("http://127.0.0.1:5555/payments", params[:payment_request])   # response handling code end 

This ends our tricky parts of extraction. Step five is dependent on your application and it’s fairly easy. Now we have the tiny bit of our complex application as a separate application instead. We can run our new application and check out if everything works fine.

More abstraction

Remember that I mentioned this is a great candidate for extraction due it’s a boundary context? You always have this kind of context in your application – for example, controllers are managing boundary context of Rails applications, like logging or rendering responses.

Coming back to our cosmic metaphor – after our step we have a brand-new created galaxy which contains exactly one star. It’s not quite efficient to leave a one-star galaxy – we just added some code and separated one particular action away from our old, big galaxy. But added code/profit ratio is poor for now. Your next step should be finding stars which share similar dependencies and nature of actions, like PaymentNotificationService and transfer this kind of stars to your new galaxy.

How SOA works in Arkency?

We have used an service-oriented approach in our product called Chillout. It’s a relatively simple app which is gathering metrics about model creations within clients’ Rails applications. We are sending a mail report each day/week/month, containing changes and charts which shows trends of model creations.

During development, 6 applications were created:

  • api – responsible for receiving data from clients. It takes HTTP requests, and communicates with brain part using ZMQ sockets.
  • brain – responsible for aggregating data and making it persistent.
  • reporter – responsible for creating reports from a given interval (day, month, year) from the data aggregated by brain.
  • mailer – responsible for generating and sending mails from reports generated by reporter.
  • dashboard – provides a front-end for our clients – they can manage their payments and add new projects to track within this application.
  • artisan – it’s responsibility is to generate a chart from a given interval, using aggregated data.

These application have narrow scopes and totally separate domains. With this kind of logical separation, we can distribute our system whatever we want. In future we can, for example, provide more brains without a hassle.

Further read

I would greatly recommend a video by Fred George about micro-services. It’s a great example how SOA can improve thinking and development of your systems.

Also, Clean Architecture is a great start to organize monolithic app to create a SOA from it in the future. You can read more about it here.

Conclusion

Service oriented architecture can be a great alternative for maintaining a big, complex singular application. When we were developing chillout we had lots of fun with this kind of architecture – features adding were simple, tests were quick and thinking about an application was simpler.

What do you think about this kind of architecture? Have you tried it before? I really looking forward for your opinions.

CoffeeScript acceptance tests

CoffeeScript acceptance tests

You’ve already learned how to implement simple model and view tests, so as I promised now I’ll show you how you can introduce acceptance tests on client-side. We’re not too far from this goal – you know how to write test cases, make assertions and “click through DOM”. Let’s apply some abstraction then – to look at our app like the end user.

Preparation

First let’s think what our application really is – it could be single page app that control whole DOM or just a widget (subtree of DOM). The real question is where would you put the border between widget and rest of HTML. Here are some aspects of widget that should help you finding the border – from most to least important:

  1. clear responsibilities – you should be able to write scenario of it’s usage in end-user language
  2. inside one container – it should have zero or minimized number of external elements it uses
  3. independent start-up – it can render itself and load all initial data
  4. own data source – it has an access to storage – through AJAX calls, LocalStorage etc.

If you go with the approach presented in Testing client-side views in Rails app you should be able to extract widget you’ve found – for a moment you can just assume it’s a huge view with a state and access to external services (called “big ball of mud”). You know how to unit test the view, even if the unit is so big. The job is to handle external services interactions with mocks and write tests as scenarios using higher-level language.

External services

Your application may use backend via AJAX, WebSockets or external library with backend – like Facebook’s JS SDK. You may try to use real data sources, but it will be hard – if you use konacha gem you won’t have easy access to your backend, it won’t be easy to clear state on backend or in external service. So it will be easier to just mock them – it violates end-to-end testing principles, but I didn’t find better way yet.

Please remember that there might be external services with easy access from DOM – like LocalStorage, Web Audio API – some of them could be used directly with no need to mock them, but you still might need to mock other – i. e. if you don’t want to hear the sound when testing application which uses Web Audio API.

Test scenarios

You probably know what Cucumber is, but just to remind you – testing framework that uses Gherkin DSL to describe context and expectations. You write your test suite almost in natural language and it translates instructions to real actions and assertions. I’m not a big fan of this approach, however it really influenced me on how should acceptance scenario look like. Such test should focus on end-user perspective – how one interacts with GUI and sees results.

Let’s have the first attempt on writing test for sample TODO application using such perspective.

describe "TODO app", ->   beforeEach ->     @app = new TodoApplication()     @app.start()    it "user adds two items and finishes of them", ->     itemsList = $("[data-container-name='items_list']")     form = $("form[name='new_item']")     form.find("input[name='label']").val("Buy milk")     expect(itemsList).to.contain("Buy milk")     form.submit()     form.find("input[name='label']").val("Sell milk")     form.submit()     expect(itemsList).to.contain("Sell milk")     itemsList.find(":checkbox:first").check()     expect(itemsList).not.to.contain("Buy milk") 

Yeah, you’re right – it’s not end user perspective. It’s jQuery perspective. Let’s fix this with our own capybara-like wrapper for DOM, based on jQuery:

describe "TODO app", ->   beforeEach ->     @app = new TodoApplication()     @app.start()    it "user adds two items and finishes of them", ->     fillIn("Task", with: "Buy milk")     submitForm()     expect(itemsList()).to.contain("Buy milk")     fillIn("Task", with: "Sell milk")     submitForm()     expect(itemsList()).to.contain("Sell milk")     checkCheckbox("Buy milk")     expect(itemsList()).not.to.contain("Buy milk") 

It’s still focused on GUI details, but for small scenarios it may be good enough. For longer scenarios you can extract chunks of interaction, like logging in, adding product to cart etc.

What’s next

In this blog series I didn’t cover few interesting, but quite heavy, topics: object orientation (like bbq), implementing own, meaningful assertions in chai.js and technics to work with 3rd party services. If you’re interested please leave your email in the form below.

3 ways to do eager loading (preloading) in Rails 3 & 4 & 5

3 ways to do eager loading (preloading) in Rails&nbsp;3&nbsp;&&nbsp;4&nbsp;&&nbsp;5

You are probably already familiar with the method #includes for eager loading data from database if you are using Rails and ActiveRecord. But do you know why you someties get few small and nice SQL queries and sometimes one giant query with every table and column renamed? And do you know about #preload and #eager_load which can help you achieve the same goal? Are you aware of what changed in Rails 4 in that matter? If not, sit down and listen. This lesson won’t take long and will help you clarify some aspects of eager loading that you might not be yet familiar with.

Let’s start with our Active Record class and associations definitions that we are going to use throughout the whole post:

class User < ActiveRecord::Base   has_many :addresses end  class Address < ActiveRecord::Base   belongs_to :user end 

And here is the seed data that will help us check the results of our queries:

rob = User.create!(name: "Robert Pankowecki", email: "robert@example.org") bob = User.create!(name: "Bob Doe", email: "bob@example.org")  rob.addresses.create!(country: "Poland", city: "Wrocław", postal_code: "55-555", street: "Rynek") rob.addresses.create!(country: "France", city: "Paris", postal_code: "75008", street: "8 rue Chambiges") bob.addresses.create!(country: "Germany", city: "Berlin", postal_code: "10551", street: "Tiergarten") 

Rails 3

Typically, when you want to use the eager loading feature you would use the #includes method, which Rails encouraged you to use since Rails2 or maybe even Rails1 ;). And that works like a charm doing 2 queries:

User.includes(:addresses) #  SELECT "users".* FROM "users" #  SELECT "addresses".* FROM "addresses" WHERE "addresses"."user_id" IN (1, 2) 

So what are those two other methods for? First let’s see them in action.

User.preload(:addresses) #  SELECT "users".* FROM "users" #  SELECT "addresses".* FROM "addresses" WHERE "addresses"."user_id" IN (1, 2) 

Apparently #preload behave just like #includes. Or is it the other way around? Keep reading to find out.

And as for the #eager_load:

User.eager_load(:addresses) #  SELECT #  "users"."id" AS t0_r0, "users"."name" AS t0_r1, "users"."email" AS t0_r2, "users"."created_at" AS t0_r3, "users"."updated_at" AS t0_r4, #  "addresses"."id" AS t1_r0, "addresses"."user_id" AS t1_r1, "addresses"."country" AS t1_r2, "addresses"."street" AS t1_r3, "addresses"."postal_code" AS t1_r4, "addresses"."city" AS t1_r5, "addresses"."created_at" AS t1_r6, "addresses"."updated_at" AS t1_r7 #  FROM "users" #  LEFT OUTER JOIN "addresses" ON "addresses"."user_id" = "users"."id" 

It is a completely different story, isn’t it? The whole mystery is that Rails has 2 ways of preloading data. One is using separate db queries to obtain the additional data. And one is using one query (with left join) to get them all.

If you use #preload, it means you always want separate queries. If you use #eager_load you are doing one query. So what is #includes for? It decides for you which way it is going to be. You let Rails handle that decision. What is the decision based on, you might ask. It is based on query conditions. Let’s see an example where #includes delegates to #eager_load so that there is one big query only.

User.includes(:addresses).where("addresses.country = ?", "Poland") User.eager_load(:addresses).where("addresses.country = ?", "Poland")  # SELECT # "users"."id" AS t0_r0, "users"."name" AS t0_r1, "users"."email" AS t0_r2, "users"."created_at" AS t0_r3, "users"."updated_at" AS t0_r4, # "addresses"."id" AS t1_r0, "addresses"."user_id" AS t1_r1, "addresses"."country" AS t1_r2, "addresses"."street" AS t1_r3, "addresses"."postal_code" AS t1_r4, "addresses"."city" AS t1_r5, "addresses"."created_at" AS t1_r6, "addresses"."updated_at" AS t1_r7 # FROM "users" # LEFT OUTER JOIN "addresses" # ON "addresses"."user_id" = "users"."id" # WHERE (addresses.country = 'Poland') 

In the last example Rails detected that the condition in where clause is using columns from preloaded (included) table names. So #includes delegates the job to #eager_load. You can always achieve the same result by using the #eager_load method directly.

What happens if you instead try to use #preload explicitly?

User.preload(:addresses).where("addresses.country = ?", "Poland") #  SELECT "users".* FROM "users" WHERE (addresses.country = 'Poland') # #  SQLite3::SQLException: no such column: addresses.country 

We get an exception because we haven’t joined users table with addresses table in any way.

Is this intention revealing?

If you look at our example again

User.includes(:addresses).where("addresses.country = ?", "Poland") 

you might wonder, what is the original intention of this code. What did the author mean by that? What are we trying to achieve here with our simple Rails code:

  • Give me users with polish addresses and preload only polish addresses
  • Give me users with polish addresses and preload all of their addresses
  • Give me all users and their polish addresses.

Do you know which goal we achieved? The first one. Let’s see if we can achieve the second and the third ones.

Is #preload any good?

Our current goal: Give me users with polish addresses but preload all of their addresses. I need to know all addreeses of people whose at least one address is in Poland.

We know that we need only users with polish addresses. That itself is easy: User.joins(:addresses).where("addresses.country = ?", "Poland") and we know that we want to eager load the addresses so we also need includes(:addresses) part right?

r = User.joins(:addresses).where("addresses.country = ?", "Poland").includes(:addresses)  r[0] #=> #<User id: 1, name: "Robert Pankowecki", email: "robert@example.org", created_at: "2013-12-08 11:26:24", updated_at: "2013-12-08 11:26:24">  r[0].addresses # [ #   #<Address id: 1, user_id: 1, country: "Poland", street: "Rynek", postal_code: "55-555", city: "Wrocław", created_at: "2013-12-08 11:26:50", updated_at: "2013-12-08 11:26:50"> # ] 

Well, that didn’t work exactly like we wanted. We are missing the user’s second address that expected to have this time. Rails still detected that we are using included table in where statement and used #eager_load implementation under the hood. The only difference compared to previous example is that is that Rails used INNER JOIN instead of LEFT JOIN, but for that query it doesn’t even make any difference.

SELECT "users"."id" AS t0_r0, "users"."name" AS t0_r1, "users"."email" AS t0_r2, "users"."created_at" AS t0_r3, "users"."updated_at" AS t0_r4, "addresses"."id" AS t1_r0, "addresses"."user_id" AS t1_r1, "addresses"."country" AS t1_r2, "addresses"."street" AS t1_r3, "addresses"."postal_code" AS t1_r4, "addresses"."city" AS t1_r5, "addresses"."created_at" AS t1_r6, "addresses"."updated_at" AS t1_r7 FROM "users" INNER JOIN "addresses" ON "addresses"."user_id" = "users"."id" WHERE (addresses.country = 'Poland') 

This is that kind of situation where you can outsmart Rails and be explicit about what you want to achieve by directly calling #preload instead of #includes.

r = User.joins(:addresses).where("addresses.country = ?", "Poland").preload(:addresses) # SELECT "users".* FROM "users" # INNER JOIN "addresses" ON "addresses"."user_id" = "users"."id" # WHERE (addresses.country = 'Poland')  # SELECT "addresses".* FROM "addresses" WHERE "addresses"."user_id" IN (1)  r[0] # [#<User id: 1, name: "Robert Pankowecki", email: "robert@example.org", created_at: "2013-12-08 11:26:24", updated_at: "2013-12-08 11:26:24">]  r[0].addresses # [ #  <Address id: 1, user_id: 1, country: "Poland", street: "Rynek", postal_code: "55-555", city: "Wrocław", created_at: "2013-12-08 11:26:50", updated_at: "2013-12-08 11:26:50">, #  <Address id: 3, user_id: 1, country: "France", street: "8 rue Chambiges", postal_code: "75008", city: "Paris", created_at: "2013-12-08 11:36:30", updated_at: "2013-12-08 11:36:30">] # ] 

This is exactly what we wanted to achieve. Thanks to using #preload we are no longer mixing which users we want to fetch with what data we would like to preload for them. And the queries are plain and simple again.

Preloading subset of association

The goal of the next exercise is: Give me all users and their polish addresses.

To be honest, I never like preloading only a subset of association because some parts of your application probably assume that it is fully loaded. It might only make sense if you are getting the data to display it.

I prefer to add the condition to the association itself:

class User < ActiveRecord::Base   has_many :addresses   has_many :polish_addresses, conditions: {country: "Poland"}, class_name: "Address" end 

And just preload it explicitely using one way:

r = User.preload(:polish_addresses)  # SELECT "users".* FROM "users" # SELECT "addresses".* FROM "addresses" WHERE "addresses"."country" = 'Poland' AND "addresses"."user_id" IN (1, 2)  r  # [ #   <User id: 1, name: "Robert Pankowecki", email: "robert@example.org", created_at: "2013-12-08 11:26:24", updated_at: "2013-12-08 11:26:24"> #   <User id: 2, name: "Bob Doe", email: "bob@example.org", created_at: "2013-12-08 11:26:25", updated_at: "2013-12-08 11:26:25"> # ]  r[0].polish_addresses  # [ #   #<Address id: 1, user_id: 1, country: "Poland", street: "Rynek", postal_code: "55-555", city: "Wrocław", created_at: "2013-12-08 11:26:50", updated_at: "2013-12-08 11:26:50"> # ]  r[1].polish_addresses # [] 

or another:

r = User.eager_load(:polish_addresses)  # SELECT "users"."id" AS t0_r0, "users"."name" AS t0_r1, "users"."email" AS t0_r2, "users"."created_at" AS t0_r3, "users"."updated_at" AS t0_r4, #        "addresses"."id" AS t1_r0, "addresses"."user_id" AS t1_r1, "addresses"."country" AS t1_r2, "addresses"."street" AS t1_r3, "addresses"."postal_code" AS t1_r4, "addresses"."city" AS t1_r5, "addresses"."created_at" AS t1_r6, "addresses"."updated_at" AS t1_r7 # FROM "users" # LEFT OUTER JOIN "addresses" # ON "addresses"."user_id" = "users"."id" AND "addresses"."country" = 'Poland'  r # [ #   #<User id: 1, name: "Robert Pankowecki", email: "robert@example.org", created_at: "2013-12-08 11:26:24", updated_at: "2013-12-08 11:26:24">, #   #<User id: 2, name: "Bob Doe", email: "bob@example.org", created_at: "2013-12-08 11:26:25", updated_at: "2013-12-08 11:26:25"> # ]  r[0].polish_addresses # [ #   #<Address id: 1, user_id: 1, country: "Poland", street: "Rynek", postal_code: "55-555", city: "Wrocław", created_at: "2013-12-08 11:26:50", updated_at: "2013-12-08 11:26:50"> # ]  r[1].polish_addresses # [] 

What should we do when we only know at runtime about the association conditions that we would like to apply? I honestly don’t know. Please tell me in the comments if you found it out.

The ultimate question

You might ask: What is this stuff so hard? I am not sure but I think most ORMs are build to help you construct single query and load data from one table. With eager loading the situation gest more complicated and we want load multiple data from multiple tables with multiple conditions. In Rails we are using chainable API to build 2 or more queries (in case of using #preload).

What kind of API would I love? I am thinking about something like:

User.joins(:addresses).where("addresses.country = ?", "Poland").preload do |users|   users.preload(:addresses).where("addresses.country = ?", "Germany")   users.preload(:lists) do |lists|     lists.preload(:tasks).where("tasks.state = ?", "unfinished")   end end 

I hope you get the idea 🙂 But this is just a dream. Let’s get back to reality…

Receive free Ruby and Rails tips like this one regularly.
Over 3200 professional Rails devs already subscribed.

Rails 4 & 5 changes

… and talk about what changed in Rails 4.

class User < ActiveRecord::Base   has_many :addresses   has_many :polish_addresses, -> {where(country: "Poland")}, class_name: "Address" end 

Rails now encourages you to use the new lambda syntax for defining association conditions. This is very good because I have seen many times errors in that area where the condition were interpreted only once when the class was loaded.

It is the same way you are encouraged to use lambda syntax or method syntax to express scope conditions.

# Bad, Time.now would be always the time when the class was loaded # You might not even spot the bug in development because classes are # automatically reloaded for you after changes. scope :from_the_past, where("happens_at <= ?", Time.now)  # OK scope :from_the_past, -> { where("happens_at <= ?", Time.now) }  # OK def self.from_the_past   where("happens_at <= ?", Time.now) end 

In our case the condition where(country: "Poland") is always the same, no matter wheter interpreted dynamically or once at the beginning. But it is good that rails is trying to make the syntax coherent in both cases (association and scope conditions) and protect us from the such kind of bugs.

Now that we have the syntax changes in place, we can check for any differences in the behavior.

User.includes(:addresses) #  SELECT "users".* FROM "users" #  SELECT "addresses".* FROM "addresses" WHERE "addresses"."user_id" IN (1, 2)  User.preload(:addresses) #  SELECT "users".* FROM "users" #  SELECT "addresses".* FROM "addresses" WHERE "addresses"."user_id" IN (1, 2)  User.eager_load(:addresses) #  SELECT "users"."id" AS t0_r0, "users"."name" AS t0_r1, "users"."email" AS t0_r2, "users"."created_at" AS t0_r3, "users"."updated_at" AS t0_r4, #         "addresses"."id" AS t1_r0, "addresses"."user_id" AS t1_r1, "addresses"."country" AS t1_r2, "addresses"."street" AS t1_r3, "addresses"."postal_code" AS t1_r4, "addresses"."city" AS t1_r5, "addresses"."created_at" AS t1_r6, "addresses"."updated_at" AS t1_r7 #  FROM "users" #  LEFT OUTER JOIN "addresses" #  ON "addresses"."user_id" = "users"."id" 

Well, this looks pretty much the same. No surprise here. Let’s add the condition that caused us so much trouble before:

User.includes(:addresses).where("addresses.country = ?", "Poland")  #DEPRECATION WARNING: It looks like you are eager loading table(s) # (one of: users, addresses) that are referenced in a string SQL # snippet. For example: # #    Post.includes(:comments).where("comments.title = 'foo'") # # Currently, Active Record recognizes the table in the string, and knows # to JOIN the comments table to the query, rather than loading comments # in a separate query. However, doing this without writing a full-blown # SQL parser is inherently flawed. Since we don't want to write an SQL # parser, we are removing this functionality. From now on, you must explicitly # tell Active Record when you are referencing a table from a string: # #   Post.includes(:comments).where("comments.title = 'foo'").references(:comments) # # If you don't rely on implicit join references you can disable the # feature entirely by setting `config.active_record.disable_implicit_join_references = true`. (  # SELECT "users"."id" AS t0_r0, "users"."name" AS t0_r1, "users"."email" AS t0_r2, "users"."created_at" AS t0_r3, "users"."updated_at" AS t0_r4, #        "addresses"."id" AS t1_r0, "addresses"."user_id" AS t1_r1, "addresses"."country" AS t1_r2, "addresses"."street" AS t1_r3, "addresses"."postal_code" AS t1_r4, "addresses"."city" AS t1_r5, "addresses"."created_at" AS t1_r6, "addresses"."updated_at" AS t1_r7 # FROM "users" # LEFT OUTER JOIN "addresses" ON "addresses"."user_id" = "users"."id" # WHERE (addresses.country = 'Poland') 

Wow, now that is quite a verbose deprection 🙂 I recommend that you read it all because it explains the situation quite accuratelly.

In other words, because Rails does not want to be super smart anymore and spy on our where conditions to detect which algorithm to use, it expects our help. We must tell it that there is condition for one of the tables. Like that:

User.includes(:addresses).where("addresses.country = ?", "Poland").references(:addresses) 

I was wondering what would happen if we try to preload more tables but reference only one of them:

User.includes(:addresses, :places).where("addresses.country = ?", "Poland").references(:addresses)  #  SELECT "users"."id" AS t0_r0, "users"."name" AS t0_r1, "users"."email" AS t0_r2, "users"."created_at" AS t0_r3, "users"."updated_at" AS t0_r4, #         "addresses"."id" AS t1_r0, "addresses"."user_id" AS t1_r1, "addresses"."country" AS t1_r2, "addresses"."street" AS t1_r3, "addresses"."postal_code" AS t1_r4, "addresses"."city" AS t1_r5, "addresses"."created_at" AS t1_r6, "addresses"."updated_at" AS t1_r7, #         "places"."id" AS t2_r0, "places"."user_id" AS t2_r1, "places"."name" AS t2_r2, "places"."created_at" AS t2_r3, "places"."updated_at" AS t2_r4 #  FROM "users" #  LEFT OUTER JOIN "addresses" ON "addresses"."user_id" = "users"."id" #  LEFT OUTER JOIN "places" ON "places"."user_id" = "users"."id" #  WHERE (addresses.country = 'Poland') 

I imagined that addresses would be loaded using the #eager_load algorithm (by doing LEFT JOIN) and places would be loaded using the #preload algorithm (by doing separate query to get them) but as you can see that’s not the case. Maybe they will change the behavior in the future.

Rails 4 does not warn you to use the #references method if you explicitely use #eager_load to get the data and the executed query is identical:

User.eager_load(:addresses).where("addresses.country = ?", "Poland") 

In other words, these two are the same:

User.includes(:addresses).where("addresses.country = ?", "Poland").references(:addresses) User.eager_load(:addresses).where("addresses.country = ?", "Poland") 

And if you try to use #preload, you still get the same exception:

User.preload(:addresses).where("addresses.country = ?", "Poland") #  SELECT "users".* FROM "users" WHERE (addresses.country = 'Poland') # #  SQLite3::SQLException: no such column: addresses.country: SELECT "users".* FROM "users"  WHERE (addresses.country = 'Poland') 

If you try to use the other queries that I showed you, they still work the same way in Rails 4:

# Give me users with polish addresses and preload all of their addresses User.joins(:addresses).where("addresses.country = ?", "Poland").preload(:addresses)  #Give me all users and their polish addresses. User.preload(:polish_addresses) 

Finally in Rails 4 there is at least some documentation for the methods, which Rails 3 has been missing for years:

Testing

Check out how you can test eager-loading the associations and make sure the code actually makes as many queries as you expected.

Summary

There are 3 ways to do eager loading in Rails:

  • #includes
  • #preload
  • #eager_load

#includes delegates the job to #preload or #eager_load depending on the presence or absence of condition related to one of the preloaded table.

#preload is using separate DB queries to get the data.

#eager_load is using one big query with LEFT JOIN for each eager loaded table.

In Rails 4 & 5 you should use #references combined with #includes if you have the additional condition for one of the eager loaded table.

Would you like to continue learning more?

If you enjoyed the article, subscribe to our newsletter so that you are always the first one to get the knowledge that you might find useful in your everyday Rails programmer job.

Content is mostly focused on (but not limited to) Ruby, Rails, Web-development and refactoring Rails applications.

Also, make sure to check out our latest book Domain-Driven Rails. Especially if you work with big, complex Rails apps.

Chronos vs Kairos: Find out how you think about time when working on a project

Chronos vs Kairos: Find out how you think about time when working on a project

When working on any project (personal or professional) we are always confrontend with tremendous amount of tasks that bring us closer to the goal or let us finish the project (many, many times there is no such a thing as the end of a project, just years of long, ongoing, constant improvement of the process and code). There are two ways you can look at it…

  • I have so much to do and so little time and the time is just passing by…

    This is the classic way of perceiving time. Time in general. Chronological time as indicated by the dates on a calendar or the ticking of a clock. The Greeks called it Chronos (or Kronos).

  • I have 2 hours of my time, what is the best way I can use it?

    For a special and unique time in your life, especially regarding important events such as wedding there was a separate word: Kairos. What is special depends on who is articulating the word. And that was the word used when you were encouraged to make the best use of the time, of your time. While Chronos is quantitative, Kairos has a qualitative nature.

From now on, I encourage you to look at your project in the second way always. The amount of tasks is probably never going to drop down. Everything that we do needs improvment. There are always thousands of ideas on how to make the software better, which technical debt to pay, what new features should be added, what parts of the design improved, etc. The backlog is probably full no matter how hard you try. And that is ok. When you work, time will just pass by (Chronos), but what happens to you and your project when working, can be Kairos. Therefore:

That is our recipe to make those moments of time uniqe and worthwhile to you as a developer. To be sure that we actually did someting in that time, instead of just letting it pass.

Read more about improving your current project in our book: Async Remote ebook

Links

Developers oriented project management: What our new book is all about?

At first, we thought that our new book is all about remote work. But then after writing few chapters we saw that it is not. That those techniques can be applied to any team. So what is it all about?

Working in a nicely organized project.

And happiness. Exactly this kind of happiness related to working in a nicely organized project. Because even if we develop applications in our favorite programming language (Ruby anyone?) we can still be in a lot of stress due to the way the project is managed, right?

Are developers feeling unhappy in your current project? Is it draining energy from them? Maybe the whole team is trapped in micro-management and does not know a way of escape from it. Perhaps peoples skills stuck on certain level and everybody is working on the same, fenced parts of the code over and over, leading to an isolation. Possible you are all working from the office, when sometimes you just want need to be alone to focus fully. And even when you deliver new features you still feel like nothing is actually finished-finished. But does it have to be like that? Is your project easy to manage for the managers, to work with for the programmers and to follow by the customer?

What if programmers working on a project could be more happy? Feel trusted and enjoy tremendous freedom. What if they could experience progress in their skills, learn and master new technologies without having to do it only in their free time? Could the team be like a small community, supportive and interested in each other problems? What can help making such team?

What if you could improve many areas of your current project management that has a direct effect on you and your team happiness. Like immedietaly knowing what you should be working on when you start your day. Delivering something valuable even twice a day and being proud of that. Being able to organize your time according to your liking and priorities? Take your time in the middle of the day to create a meal compatible with your diet, exercise, walk or take a lovely nap to refresh your mind. What if tasks in unfamiliar technology are learning opportunities and small challenges and instead of being scary monsters? Can you imagine your team collaborating together on the whole code?

Our ebook is for you if you value Independence, Clarity, Focus, Freedom and Collaboration. It contains a list of guidelines that we established at Arkency throughout years of working remotely. You can apply them to your project slowly and every one of them will help you improve some of the aspects. Together they make a tremendous difference and let you enjoy lot of the benefits that a programmer job can offer. They create a programmer friendly environment in which you can feel comfortable and productive. After all, most of your team consist of programmers, so the project should be optimized for their efficiency and happiness. But it also creates a nice set of rules that make the customers and product owners communication with developers easier.

As programmers we are privileged. In our lives we can enjoy many things that people doing other kinds of job cannot. But we need a work-flow that will let us actually seize the opportunities such as eg. the ability to work remotely.

As team leaders and company owners we always want to hire and work with the best people. But they also have very high expectations. Is your company ready to provide them a nicely working experience to benefit most from their talent and experience? We are also always overloaded with tons of other duties and activities so we would like the dev team to take care of itself as much as possible. It would be best if they could self-organize the work, find priorities and assign tasks. But do we let them? Do we know how? Don’t worry, we will show you how.

When we started writing the book, we promised that we will help your team transition into remote working and we will still do that. Because the crucial ingredient for successful remote working is project organization and excellent communication. And we intend to write about that. But you can still benefit greatly by following the advices from the book even if remote working is not your highest priority right now.

Developers oriented project management

You can find out more in our book: Async Remote ebook or if you are not sure yet, by subscribing to the newsletter with tips and excerpts from the book similar in form to what you can read in our previous blog posts:

Developers oriented project management: Take the first task

We recently talked how small and unassigned tasks can help you manage the risk in the project. In short: they help to guarantee that despite the obstacles people keep working on the most important things.

But to take most from such approach we need one more rule. Let’s talk a little about it.

When project managers assign tasks to developers they will usually try to do it based on who they think are most qualified for them. Unfortunately when developers are in charge of assigning stories the effect is not much better. They will usually try to take tasks, that they feel most comfortable with. Frontend developers will take frontend tasks. Developers with more backend background will take backend tasks. The long term the effect is that people specialize in particular areas of code and Collective Ownership declines. And this is exactly what we would like to avoid.

This happens because sometimes the developers have different goals to your goals. They want to make their job easy or fun, where the goal of the project owners is to have the most important task get done. And the most important task is not always the most compelling, neither related to the most fancy, recent technology that the developers would like to learn and use

The solution is to have very strict and simple rule for the developers to follow. And the rule says Take the first task. Where First means unstarted tasks with highest business value. When I join the project as a developer for the first time, or on Monday after weekend, or after a two-week-long vacation, I am interested in one thing only: What can I do for you?. The answer should be immediately visible for me. I do not care if you use Pivotal, Redmine or Trello. I just want to look at the top of the list and finish your most important task today. Get it done, deliver on production and forget about it.

Get more

You can find out more in our book: Async Remote or if you are not sure yet, by subscribing to the newsletter with tips and excerpts from the book similar in form to what you just read.

In next episode

The next blog post will be about Project Managers. Whether you actually need them and what are the alternatives to hiring them. Subscribe to be notified when it is out.

How about you?

Is it immediately clear to you in your current project what you should be working on as a developer? How long does it take to figure it out? Do you need to talk to someone to get that information? Leave your feedback in comments or on twitter.

In this series

Developers oriented project management: Leave tasks unassigned

As with software, improving your company is an ongoing process consisting of multiple small steps. If you want your company to be more remote-friendly it is also not all-or-nothing decision. You can gradually start using techniques that will bring you closer to your goal. This series of blogposts is not about why you should go remote, but rather how. Also, we believe that those techniques will help you even if you are not planning to work remotely.

The first part was about having very small stories and how it helps everyone track progress and provides closure. Today we are going to talk about leaving tasks unassigned instead of delegating them to particular developers at the beginning of an iteration.

You do not know how long given task is going to take

Let’s start with an assumption. You want your developers to be doing the most important task for the project that can be currently done. Usually this is one of the most important criteria taken into consideration when assigning a task. So let’s say you have three developers and ten tasks to be done during the iteration. Here is how you decided to split the work.

Developers oriented project management: Leave tasks unassigned

Looks very promising. But here comes the reality. The most important task (no. 1) takes longer than anticipated. And suprisingly one of the tasks (no. 3) turned out to be easier and was finished earlier.

Developers oriented project management: Leave tasks unassigned

What effect does it have for your project? Multiple times during this iteration people are working on tasks which are not most important. The first developer is working on task 6, when task 4 and 5 are not even started. Task 7 won’t be finished by the end of the sprint even if it could be done instead of task 8. And so on…

You can simply avoid these kind of problems by leaving tasks unassigned and prioritized. Let your developer start the most important task when they are free working on anything else.

You might not know or be sure about the availability of your developers

If one of your developers gets sick or needs a little break your most important tasks might be damaged. If you look at the original estimation image the third developer is crucial to the success of this iteration. If (s)he leaves the team due to health or personal reasons for some time there will be some damage. The effects are similar to the ones mentioned in the previous paragraph. People are not working on the most important tasks, the ones bringing the most business value.

If you want most possible freedom for your developers and coworkers they cannot be tight to the project they are currently working on. If your prioritiy is for them to balance their personal and work life you cannot relay that they will be available for given number of hours for the work on the project. Not to mention the fact that the performance of developers vary greatly on daily basis depending on their mood, health, and many external factors. There is only one way to manage it. Surrender ;). Accept the fact that our power over reality is limited and adapt often and quickly. Leave unassigned tasks to be done and let people do them. If someone is not working today, someone else can simply do the task instead.

It all comes down to managing risk

Small, unassigned tasks are your tools for managing risk on daily basis. The problems are constantly attacking our projects from all sides. People effectiveness vary, illness happen, personal life interferes, tasks take longer, external events occur. To finish things as planned is almost impossible. So we need guidance that will help us deliver as much business value much as possible and being constantly focused on most important aspects of our project. We need flexibility in managing the project. The more the better. But good things also happen. Sometimes people suprise us with their solutions and performance. Opportunities arrive and we would like to take them. So we need elasticity to manage all that complexity that a software project is. And small, unassigned tasks bring you that.

Also it lowers the amount of micrmanagement and attention that the project needs. Instead of daily assigning tasks to people you just need to set their prorities which you do for everything anyway.

Find out more

There is more beyond what we wrote in this article. Find out in our Async Remote ebook or by subscribing to the newsletter with tips and excerpts from the book similar in form to what you just read. The newsletter will also containt exclusive content from the book that is not published on the blog.

In next episode

The next blog post will be about Collective Ownership and what it has in common with the techniques that we already described. Subscribe to make sure you do not miss it.

What is your opinion

Does your company practice these techniques? How is that working for you? Share your opinion in comments or on twitter.

In this series

Services – what are they and why we need them?

Model-View-Controller is a design pattern which absolutely dominated web frameworks. On the first look it provides a great and logical separation between our application components. When we apply some basic principles (like ‘fat models, slim controllers’) to our application, we can live happily very long with this basic fragmentation.

However, when our application grows, our skinny controllers become not so skinny over time. We can’t test in isolation, because we’re highly coupled with the framework. To fix this problem, we can use service objects as a new layer in our design.

Entry point

I bet many readers had some experience with languages like C++ or Java. This languages have a lot in common, yet are completely different. But one thing is similar in them – they have well defined entry point in every application. In C++ it’s a main() function. The example main function in C++ application looks like this:

#include <iostream> // Many includes...  int main(int argc, char *argv[]) {   // Fetch your data.   // Ex. Input data = Input.readFromUser(argc, argv);    Application app = Application(data);   app.start();    // Cleanup logic...   return 0; } 

If you run your application (let it be ./foo), main function is called and all arguments after it (./foo a b c) are passed in argv as strings. Simple.

When C++ application grows, nobody sane puts logic within main. This function only initializes long-living objects and runs a method like start in above example.

But why we should be concerned about C++ when we’re Rails developers?

Controller actions are entry points

As title states, Rails has multiple entry points. Every controller action in Rails is the entry point! Additionaly, it handles a lot of responsibilities (parsing user input, routing logic [like redirects], logging, rendering… ouch!).

We can think about actions as separate application within our framework – each one with its private main. As I stated before, nobody sane puts logic in main. And how it applies to our controller, which in addition to it’s responsibilities takes part in computing response for a client?

Introducing service objects

That’s where service objects comes to play. Service objects encapsulates single process of our business. They take all collaborators (database, logging, external adapters like Facebook, user parameters) and performs a given process. Services belongs to our domain – They shouldn’t know they’re within Rails or webapp!

We get a lot of benefits when we introduce services, including:

  • Ability to test controllers – controller becomes a really thin wrapper which provides collaborators to services – thus we can only check if certain methods within controller are called when certain action occurs,

  • Ability to test business process in isolation – when we separate process from it’s environment, we can easily stub all collaborators and only check if certain steps are performed within our service.

  • Lesser coupling between our application and a framework – in an ideal world, with service objects we can achieve an absolutely technology-independent domain world with very small Rails part which only supplies entry points, routing and all ‘middleware’. In this case we can even copy our application code without Rails and put it into, for example, desktop application.

  • They make controllers slim – even in bigger applications actions using service objects usually don’t take more than 10 LoC.

  • It’s a solid border between domain and the framework – without services our framework works directly on domain objects to produce desired result to clients. When we introduce this new layer we obtain a very solid border between Rails and domain – controllers see only services and should only interact with domain using them.

Example

Let’s see a basic example of refactoring controller without service to one which uses it. Imagine we’re working on app where users can order trips to interesting places. Every user can book a trip, but of course number of tickets is limited and some travel agencies have it’s special conditions.

Consider this action, which can be part of our system:

 class TripReservationsController < ApplicationController   def create     reservation = TripReservation.new(params[:trip_reservation])     trip = Trip.find_by_id(reservation.trip_id)     agency = trip.agency      payment_adapter = PaymentAdapter.new(buyer: current_user)      unless current_user.can_book_from?(agency)       redirect_to trip_reservations_page, notice: TripReservationNotice.new(:agency_rejection)     end      unless trip.has_free_tickets?       redirect_to trip_reservations_page, notice: TripReservationNotice.new(:tickets_sold)     end      begin       receipt = payment_adapter.pay(trip.price)         reservation.receipt_id = receipt.uuid        unless reservation.save         logger.info "Failed to save reservation: #{reservation.errors.inspect}"         redirect_to trip_reservations_page, notice: TripReservationNotice.new(:save_failed)       end        redirect_to trip_reservations_page(reservation), notice: :reservation_booked     rescue PaymentError       logger.info "User #{current_user.name} failed to pay for a trip #{trip.name}: #{$!.message}"       redirect_to trip_reservations_page, notice: TripReservationNotice.new(:payment_failed, reason: $!.message)     end   end end 

Although we packed our logic into models (like agency, trip), we still have a lot of corner cases – and our have explicit knowledge about them. This action is big – we can split it to separate methods, but still we share too much domain knowledge with this controller. We can fix it by introducing a new service:

 class TripReservationService   class TripPaymentError < StandardError; end   class ReservationError < StandardError; end   class NoTicketError < StandardError; end   class AgencyRejectionError < StandardError; end    attr_reader :payment_adapter, :logger    def initialize(payment_adapter, logger)     @payment_adapter = payment_adapter     @logger = logger   end    def process(user, trip, agency, reservation)     raise AgencyRejectionError.new unless user.can_book_from?(agency)     raise NoTicketError.new unless trip.has_free_tickets?      begin       receipt = payment_adapter.pay(trip.price)         reservation.receipt_id = receipt.uuid        unless reservation.save         logger.info "Failed to save reservation: #{reservation.errors.inspect}"         raise ReservationError.new       end     rescue PaymentError       logger.info "User #{user.name} failed to pay for a trip #{trip.name}: #{$!.message}"       raise TripPaymentError.new $!.message     end   end end 

As you can see, there is a pure business process extracted from a controller – without routing logic.

Our controller now looks like this:

 class TripReservationsController < ApplicationController   def create     user = current_user     trip = Trip.find_by_id(reservation.trip_id)     agency = trip.agency     reservation = TripReservation.new(params[:trip_reservation])      begin       trip_reservation_service.process(user, trip, agency, reservation)     rescue TripReservationService::TripPaymentError       redirect_to trip_reservations_page, notice: TripReservationNotice.new(:payment_failed, reason: $!.message)     rescue TripReservationService::ReservationError       redirect_to trip_reservations_page, notice: TripReservationNotice.new(:save_failed)     rescue TripReservationService::NoTicketError       redirect_to trip_reservations_page, notice: TripReservationNotice.new(:tickets_sold)     rescue TripReservationService::AgencyRejectionError       redirect_to trip_reservations_page, notice: TripReservationNotice.new(:agency_rejection)     end      redirect_to trip_reservations_page(reservation), notice: :reservation_booked   end    private   def trip_reservation_service     TripReservationService.new(PaymentAdapter(buyer: current_user), logger)   end end 

It’s much more concise. Also, all the knowledge about process are gone from it – now it’s only aware which situations can occur, but not when it may occur.

A word about testing

You can easily test your service using a simple unit testing, mocking your PaymentAdapter and Logger. Also, when testing controller you can stub trip_reservation_service method to easily test it. That’s a huge improvement – in a previous version you would’ve been used a tool like Capybara or Selenium – both are very slow and makes tests very implicit – it’s a 1:1 user experience after all!

Conclusion

Services in Rails can greatly improve our overall design as our application grow. We used this pattern combined with service-based architecture and repository objects in Chillout.io to improve maintainability even more. Our payment controllers heavy uses services to handle each situation – like payment renewal, initial payments etc. Results are excellent and we can be (and we are!) proud of Chillout’s codebase. Also, we use Dependor and AOP to simplify and decouple our services even more. But that’s a topic for another post.

What are your patterns to increase maintainability of your Rails applications? Do you stick with your framework, or try to escape from it? I wait for your comments!