After Fun With Rakefiles, here’s another ‘behaviour’ I found in a Rakefile.
Some example code is available on GitHub: https://github.com/s2k/rake-loop.
The Setting
The project team worked on a larger number of modules, each in its own sub directory and a global rake task spec:all to run, well, all the specs for all the modules. Since they could run independently, we used the parallel gem to run RSpec in several processes, so our continuous integration system (or local machine) could use the available CPUs (as opposed to running in Ruby threads, which doesn’t utilise all processors [well, at the time of this writing]).
Also, while most of the RSpec code used the (now) old-style should-notation, some new specs were written using the new expect notation. For more on that topic see e.g. RSpec’s New Expectation Syntax by Myron Marston and the GitHub repo for RSpec expectations.
Trying To Fix A Hack
When working with the Rakefile and RSpec files I noticed something odd: The RSpec files using expect weren’t executed. The reason was some code in the Rakefile which (essentially) did this:
run(rspec_filename) if File.read(rspec_filename).contains? 'should'
That explains why the newer RSpec code was not executed: It didn’t contain any shoulds. But what was going on?
Right after removing that code and re-running bundle exec rake spec:all, I found out that the command a) didn’t finish in any reasonable amount of time and b) over time there were more and more Ruby processes showing up in the system monitor. What was happening?
Why excluding RSpec files not containing ‘should’ helped
As far as the symptom was concerned, excluding RSpec files that didn’t contain the word ‘should’ helped avoiding the behaviour I observed. In any case, I wanted to also execute the other RSpec files, but I would not add to the hack by executing file which contained either ‘should’ or ‘expect’…
A Rake Feature
When rake is called from the command line in a directory that does not contain a Rakefile, then rake looks for a Rakefile in the parent directory. That’s a feature, since it allows you to have one description of a task in some directory, and use that task in all sub directories. (In our case however, something was going horribly wrong.)
Where It went wrong
I then figured out that the modules where the RSpec files didn’t contain the word ‘should’ also didn’t contain a Rakefile. So, when invoking rake in that case, it would move upwards the folder structure finally get to the project global Rakefile and call the task defined there.
Alas, the ‘global’ Rakefile not only contained a task definition for spec:all, but also a task called spec—which only invoked spec:all in a new shell:
namespace :spec do
desc 'runs all rspec suites'
task :all => :clean do
# do stuff, in particular:
sh "cd #{spec_dir}/../; bundle exec rake spec > #{report_folder}/#{module_name}.tmp 2>&1"
end
desc 'delete old report files'
task :clean do
FileUtils.rm_rf report_folder
end
end
task :spec do
sh 'bundle exec rake spec:all'
end
Notice that the global Rakefile expects a Rakefile containing a definition for the task ‘spec’ to be defined somewhere (see line 5 in the code snippet above), probably in the sub directory where rake is called.
Alas, if there is no Rakefile, rake will step up the directory structure, eventually find the global Rakefile call the task ‘spec’… and restart running ‘spec:all’ again, only to run into the exact same situation again and therefore start a loop.
Lessons Learned
- Using the parallel gem made a bad situation worse, but it also helped finding the real problem by maximising the effect.
- It’s probably a good idea to call rake tasks from within a rake task using the methods Rake itself provides (invoke and execute). There’s a discussion on stack overflow named How to run Rake tasks from within Rake tasks?
- If something goes wrong, fix the underlying issue, rather than working around the problem. It’ll save later generations a lot of work.
- Generally speaking, it’s a good idea not to let specialised rake tasks (like running RSpec in some sub directory) end up invoking a more global rake task. That’s especially true if that more global task is going to invoke the more specialised one in turn.
Ph values 