More Fun With Rakefiles

After Fun With Rakefiles, here’s another ‘behaviour’ I found in a Rakefile.
Some example code is available on GitHub: https://github.com/s2k/rake-loop.

The Setting

The project team worked on a larger number of modules, each in its own sub directory and a global rake task spec:all to run, well, all the specs for all the modules. Since they could run independently, we used the parallel gem to run RSpec in several processes, so our continuous integration system (or local machine) could use the available CPUs (as opposed to running in Ruby threads, which doesn’t utilise all processors [well, at the time of this writing]).

Also, while most of the RSpec code used the (now) old-style should-notation, some new specs were written using the new expect notation. For more on that topic see e.g. RSpec’s New Expectation Syntax by Myron Marston and the GitHub repo for RSpec expectations.

Trying To Fix A Hack

When working with the Rakefile and RSpec files I noticed something odd: The RSpec files using expect weren’t executed. The reason was some code in the Rakefile which (essentially) did this:

 run(rspec_filename) if File.read(rspec_filename).contains? 'should'

That explains why the newer RSpec code was not executed: It didn’t contain any shoulds. But what was going on?

Right after removing that code and re-running bundle exec rake spec:all, I found out that the command a) didn’t finish in any reasonable amount of time and b) over time there were more and more Ruby processes showing up in the system monitor. What was happening?

Why excluding RSpec files not containing ‘should’ helped

As far as the symptom was concerned, excluding RSpec files that didn’t contain the word ‘should’ helped avoiding the behaviour I observed. In any case, I wanted to also execute the other RSpec files, but I would not add to the hack by executing file which contained either ‘should’ or ‘expect’…

A Rake Feature

When rake is called from the command line in a directory that does not contain a Rakefile, then rake looks for a Rakefile in the parent directory. That’s a feature, since it allows you to have one description of a task in some directory, and use that task in all sub directories. (In our case however, something was going horribly wrong.)

Where It went wrong

I then figured out that the modules where the RSpec files didn’t contain the word ‘should’ also didn’t contain a Rakefile. So, when invoking rake in that case, it would move upwards the folder structure finally get to the project global Rakefile and call the task defined there.

Alas, the ‘global’ Rakefile not only contained a task definition for spec:all, but also a task called spec—which only invoked spec:all in a new shell:

namespace :spec do
  desc 'runs all rspec suites'
  task :all => :clean do
    # do stuff, in particular:
    sh "cd #{spec_dir}/../; bundle exec rake spec > #{report_folder}/#{module_name}.tmp 2>&1"
  end

 desc 'delete old report files'
 task :clean do
   FileUtils.rm_rf report_folder
 end
end

task :spec do
  sh 'bundle exec rake spec:all'
end

Notice that the global Rakefile expects a Rakefile containing a definition for the task ‘spec’ to be defined somewhere (see line 5 in the code snippet above), probably in the sub directory where rake is called.

Alas, if there is no Rakefile, rake will step up the directory structure, eventually find the global Rakefile call the task ‘spec’… and restart running ‘spec:all’ again, only to run into the exact same situation again and therefore start a loop.

Lessons Learned

  • Using the parallel gem made a bad situation worse, but it also helped finding the real problem by maximising the effect.
  • It’s probably a good idea to call rake tasks from within a rake task using the methods Rake itself provides (invoke and execute). There’s a discussion on stack overflow named How to run Rake tasks from within Rake tasks?
  • If something goes wrong, fix the underlying issue, rather than working around the problem. It’ll save later generations a lot of work.
  •  Generally speaking, it’s a good idea not to let specialised rake tasks (like running RSpec in some sub directory) end up invoking a more global rake task. That’s especially true if that more global task is going to invoke the more specialised one in turn.

Ruby Details: clone and dup

Another post that’s mostly a reminder to self. ;-)

Over at Know Ruby: clone and dup Aaron Lasseigne explained the differences between dup and clone.

However there’s another difference, and that’s the way the methods reproduce the methods an object can respond to. As is so often the case, a short session in a Ruby REPL (like irb or pry) helps to see things in action:

[1] pry(main)> o = Object.new
=> #<Object:0x00000102d298a0>
[2] pry(main)> def o.meth
[2] pry(main)* puts "meth called on object #{self.inspect}"
[2] pry(main)* end
=> :meth
[3] pry(main)> o.meth
meth called on object #<Object:0x00000102d298a0>
=> nil
[4] pry(main)> o_dupped = o.dup
=> #<Object:0x00000102101eb0>
[5] pry(main)> o_dupped.meth
NoMethodError: undefined method `meth' for #
from (pry):7:in `__pry__'
[6] pry(main)> o_cloned = o.clone
=> #<Object:0x00000102138168>
[7] pry(main)> o_cloned.meth
meth called on object #<Object:0x00000102138168>
=> nil

Note that both, clone and dup, return a new object, but only the one returned by clone also has the method defined for object o.
In other words: dup does not replicate an objects singleton methods.

Fun With Rakefiles

I recently ran into an ‘interesting’ issue in a set of  Rakefiles.

Here’s a part of the global Rakefile, which imports the *.rake files from a tasks folder:

$LOAD_PATH.unshift File.join(File.dirname(__FILE__), 'lib')

Dir.glob('tasks/*.rake').each { |r| import r }

Some of these Rakefiles require other Ruby files (which is stored in the lib folder). Since executing one task involves loading all that code, there’s a chance that an error in one supporting Ruby file may cause an error in an entirely unrelated task.

Let’s look at an example, the code is available over at GitHub: https://github.com/s2k/fun_with_rakefiles

For simplicity there are just two rake files within tasks and a ruby file in lib. What we like to run is this:

rake make_it_so

However what we get is an error message pointing to some rather unrelated part of the code:

$rake make_it_so
rake aborted!
Errno::ENOENT: No such file or directory @ rb_sysopen - partlist.txt
…dev/fun_with_rakefiles/lib/thing.rb:7:in `readlines'
…dev/fun_with_rakefiles/lib/thing.rb:7:in `<class:Thing>'
…dev/fun_with_rakefiles/lib/thing.rb:1:in `<top (required)>'
…dev/fun_with_rakefiles/tasks/unrelated_thing_tasks.rake:1:in `require'
…dev/fun_with_rakefiles/tasks/unrelated_thing_tasks.rake:1:in `<top (required)>'
…/.rvm/gems/ruby-2.1.1/bin/ruby_executable_hooks:15:in `eval'
…/.rvm/gems/ruby-2.1.1/bin/ruby_executable_hooks:15:in `'
(See full trace by running task with --trace)

Hmmm… When I first saw the error message I really wondered why suddenly there’s a file partlist.txt missing and then I asked myself where that Thing class was used in the code.

In this example it’s relatively easy to see what’s going on: The ‘central’ Rakefile gathers all the other .rake files that may be necessary (which, in turn, require all the files they need). However, if there are a few (sub) rake files around things can get a bit more confusing.

In my case, it was simple enough to repair the issue, since the missing file could be added to source control and everything worked again. But still there are a few things to think about:

  1. Should the be code in a Ruby class definition that’s executed while defining the class? Especially: Should it be there, in case you can avoid it?
    In my opinion it’s a good idea to only have code like that if it’s really necessary. In the example given, one could load the file when an instance of the class is created. Doing so will reduce the consequences of the missing file drastically: Only the rake tasks actually using that class would fail, not every rake task.
  2. Could one avoid the failure and still be able to run the desired task?
    It turns out that you can:

    $rake -f tasks/what_we_want_to_do.rake  make_it_so
    Processing something…

    In some cases you may also need to supply this command line parameter, in order to allow rake to find other Ruby files:

    -I, --libdir LIBDIR      Include LIBDIR in the search path for required modules.
  3. Can we check that nothing’s broken?
    In fact, we can:

    rake -T

    If that actually lists the available rake tasks and doesn’t exit with an error code, it looks OK, at least it didn’t break everything. However, if a file is missing form the version control system, it’s a works-on-my-machine situation.

In a project with a large number of rake files and supporting library code, it may be worth while to have rake -T as part of a smoke test on the continuous integration system.

 

 

Mac OS X Mavericks & the git command line

Note: This one may not affect you at all, if you own a MacBook with an English keyboard layout (and corresponding native language settings).

I however, bought a MacBook Pro with a German keyboard layout, and when using, for example ‘git pull’ on the command line I got this message:

$git pull
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LANG = (unset)
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

I added this to my .bash_profile:

export LC_CTYPE=en_US.UTF-8
export LC_ALL=en_US.UTF-8

Problem gone — thanks to the folks at spindicator.com who solved this problem already.

Note, that you may want to put in other values, depending on your language preferences.

Mac OS X, Parallels & the Back Slash

Yet another short note to my future self:

Running a Mac with a German keyboard layout (likely having the key sequence “7 8 9 0 ß ` <Backspace>” in the top row) and using a Parallels Desktop to run Windows and needing to enter a back slash (\) now & then?

Try the right-hand-side CRTL-ALT + ß (the key with the German ‘sz’ as the primary character).

The Work of Being Lucky

Lisa Crispin‘s article about her team not being special (We’re Not ‘Special’) reminded me of something I heard a few times in the past: Me being lucky in being were I am.

And it’s true: I was lucky a few times: In 2009 (if I remember correctly) I went to a presentation at Lehmann’s  (@Lehmanns in case you prefer Twitter) where, apart after a great presentation about Scrum and during the discussion and socializing part someone from the Xing team told me about the Agile Testing Days later that same year. Sure enough, I went there — and met a whole lot of exciting (and excited) people there. The connections to colleagues from all over the world also was worth going there.
So yes, I was lucky meeting the person who would give me the information I needed.
However, to me at least, being at the presentation was not lucky or coincidental at all: It took me a 200km drive by car to get there and another 200km drive to get home again.
So was it sheer luck? I don’t think so. In fact I truly believe that in order to be lucky, one needs to go to the places where it happens.

There have been other times I was lucky: Being (kind of) dragged into a dinner of a conference (the same one as above, BTW), was another time I was there, ready to be lucky: Someone asked me whether I’d like to join dinner. Of course I did and had a great time. A great time until someone asked me about the presentation I would give, and someone else then asked me whether I knew that this was the speakers dinner. Oopsie, I didn’t know! I found that was embarrassing, even though everyone else thought it was amusing. Now, a while later, it’s certainly a funny story to tell.

I think that to be lucky, you need to go out and be there, be present in a context where you’d like to be lucky: This might be a local user group gathering, some conference … or twitter (for this also see my post over at Zen & the Art of Automated Testing). Did I mention that I was lucky to find a cool new project via Twitter? Well, I did. :-)

In other words: It’s work and you (well, I at least) will have to pay a price to be lucky and I totally find it worth the hassle.

Now go out there and get lucky. And if you’re lucky in getting lucky you may become happy as well. Good Luck!

That said, I think Lisa’s team might in fact be special, but not in the way Lisa is frustrated about: To me it seems special because of the hard work, experiments and continuous improvements it went though. In this (may be special) sense they’re special and lucky.