Bundler

Dealing with dependencies -- multiple versions of Ruby and multiple versions of Gems -- is a significant issue in Ruby. A project may need a Ruby version that differs from your default Ruby. Even if it requires the same version of Ruby, it may need a different version of a RubyGem.

This problem is not unique to Ruby; dependency issues arise in all languages. The techniques used to deal with the dilemma differ with each language. In Ruby, most developers use a Ruby version manager such as RVM or rbenv to manage multiple Ruby versions. You can also use your version manager to manage Gem dependencies, but the favored approach is to use a dependency manager.

The most widely used dependency manager in the Ruby community, by far, is the Bundler Gem. This Gem lets you configure which Ruby and which Gems each of your projects need.

Installing Bundler

Bundler is a Gem, so you must use the gem command to install it. If you use a Ruby version manager, you must install the Gem in each version of Ruby for which you wish to use Bundler. After switching to the appropriate Ruby you wish to use, use this command to install the Bundler Gem:

$ gem install bundler

That's all there is to it.

Gemfile and Gemfile.lock

Bundler relies on a text file named Gemfile to tell it which version of Ruby and its Gems it should use. This file is a simple Ruby program that uses a Domain Specific Language (DSL) to provide details about the Ruby and Gem versions. It's the configuration or instruction file for Bundler.

After you create Gemfile, the bundle install command scans it, downloads and installs all the dependencies listed, and produces a Gemfile.lock file. Gemfile.lock shows all the dependencies for your program; this includes the Gems listed in Gemfile, as well as the Gems they depend on (the dependencies), which may not be explicitly listed in the Gemfile. It's very common for RubyGems you install for use in your project to rely on many other gems, creating a large dependency tree.

Lets examine a simple example. Suppose you are writing a program that requires Ruby 2.3.1 and the sinatra, erubis, and rack Gems. Our Gemfile incorporates these dependencies, and looks like this:

source 'https://rubygems.org'

ruby '2.3.1'
gem 'sinatra'
gem 'erubis'
gem 'rack'
gem 'rake'

Furthermore, our ruby installation looks like Figure 5:

Figure 5

$ tree /usr/local/rvm # the following is partial output
/usr/local/rvm # RVM path directory
└── gems
    ├── ruby-2.2.2
    └── ruby-2.3.1
        ├── bin
        │   ├── bundle
        │   └── rubocop
        └── gems
            ├── erubis-2.7.0
            ├── rack-1.6.4
            ├── rack-protection-1.5.3
            ├── rake-10.4.2
            ├── rake-11.3.0
            ├── sinatra-1.4.6
            ├── sinatra-1.4.7
            └── tilt-2.0.5

We now run:

$ bundle install

to install the specified Gems (if needed) and create a Gemfile.lock, which looks like this:

GEM
  remote: https://RubyGems.org/
  specs:
    erubis (2.7.0)
    rack (1.6.4)
    rack-protection (1.5.3)
      rack
    rake (10.4.2)
    sinatra (1.4.7)
      rack (~> 1.5)
      rack-protection (~> 1.4)
      tilt (>= 1.3, < 3)
    tilt (2.0.5)

PLATFORMS
  ruby

DEPENDENCIES
  erubis
  rack
  rake
  sinatra

RUBY VERSION
   ruby 2.3.1p112

BUNDLED WITH
   1.13.6

The specs section under the GEM heading provides a list of the Gems (and their versions) that your app will load. Beneath each listed Gem is a list of the Gem's dependencies; that is, the Gems and versions it needs to work. Here, we can see that:

• We have sinatra version 1.4.7. Note that we chose 1.4.7 over 1.4.6. Bundler won't always choose the latest version like this. It will choose a version that works in conjunction with the other dependencies.
• sinatra requires rack (version >= 1.5.0 and < 2.0.0); we have version 1.6.4.
• sinatra requires rack-protection (version >= 1.4.0 and < 2.0.0); we have version 1.5.3.
• sinatra requires tilt (versions >= 1.3.0 and < 3.0.0); we have version 2.0.5

We didn't have to provide any information about rack-protection and tilt in our Gemfile; Bundler found this information on its own by examining the Gemfiles for those Gems -- that is, not our application's Gemfile, but the Gemfile that came with the Gems specified in our Gemfile. It then added the information to our Gemfile.lock.

If you're interested in even more details about how the Gemfile works, see the Bundler documentation for more information on how to construct a Gemfile.

Running Apps With Bundler

Once Bundler creates your Gemfile.lock, add:

require 'bundler/setup'

to the beginning of your app, before any other Gems. (This is unneeded if your app is a Rails app).

bundler/setup first removes all Gem directories from Ruby's $LOAD_PATH global array. Ruby uses $LOAD_PATH to list the directories that it searches when it needs to locate a required file. When bundler/setup removes those directories from $LOAD_PATH, Ruby can no longer find Gems.

To fix this, bundler/setup reads Gemfile.lock; for each Gem listed, it adds the directory that contains that Gem back to $LOAD_PATH. When finished, require only finds the proper versions of each Gem. This ensures that the specific Gem and version your app depends on is loaded, and not a conflicting version of that Gem.

Now, all you need to do is run your app and the correct Gem will be loaded when you require files.

require 'rubygems'

Where Are My Rubies, Gems and Apps Now?

Bundler does not interfere with your Rubies nor their Gems. They remain where they were before you installed Bundler, and will continue to use the same setup in the future. This means that you can still use gem env, rvm info, and rbenv version and other informational commands to find information you may need.

However, Bundler provides a feature called binstubs; if you use this feature, you may have to add some directories to your PATH. The gem env and rvm info commands will reflect this.

bundle exec

As we saw earlier, an app that relies on Bundler should require the bundler/setup package before it loads any Gems. This package ensures that the app loads the desired Gems.

Unfortunately, you will surely encounter situations where you can't just add require 'bundler/setup' to the code, or the program itself may run code that has conflicting needs. When this happens, you need the often mysterious bundle exec command.

You can use bundle exec to run most any command in an environment that conforms to the Gemfile.lock versioning info. In fact, we can use this feature to see how bundle exec modifies your environment:

# This command compares the output of 'bundle exec env' with the output of 'env'

$ diff <(bundle exec env) <(env)
< PATH=/usr/local/rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/bin:/usr/local/rbenv/versions/2.3.1/bin:/usr/local/Cellar/rbenv/1.0.0/libexec:/usr/local/rbenv/shims:...
---
> PATH=/usr/local/rbenv/shims:...
< RBENV_HOOK_PATH=/usr/local/rbenv/rbenv.d:/usr/local/Cellar/rbenv/1.0.0/rbenv.d:/usr/local/etc/rbenv.d:/etc/rbenv.d:/usr/lib/rbenv/hooks:/usr/local/rbenv/plugins/rbenv-default-gems/etc/rbenv.d
< RBENV_DIR=/Users/wolfy/my_app
< RUBYLIB=/usr/local/rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/bundler-1.13.6/lib:/usr/local/Cellar/rbenv/1.0.0/rbenv.d/exec/gem-rehash
< RBENV_VERSION=2.3.1
< BUNDLE_BIN_PATH=/usr/local/rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/bundler-1.13.6/exe/bundle
< BUNDLE_GEMFILE=/Users/wolfy/my_app/Gemfile
< RUBYOPT=-rbundler/setup

(The above shows partial output only.) As you can see, the two commands produce different results; this output shows the modifications and additions that bundle exec makes to your environment.

Of special importance is the RUBYOPT value: this tells Ruby to require 'bundler/setup' (recall that 'bundle' and 'bundler' are aliases) before it starts running your code. This lets Bundler gain control in time to configure things so that the app loads the proper Gems, and only those Gems.

When Should You Use bundle exec?

We use it above with env primarily to demonstrate what bundle exec does. Using bundle exec with a non-Ruby command is rare, though. You usually use bundle exec with commands written in Ruby and installed as Gems, e.g., Rake, Pry, and Rackup. But, exactly when would one need to use it?

We use it to resolve dependency conflicts when issuing shell commands. From time to time, you may encounter an error message that looks like this:

Gem::LoadError: You have already activated rake 11.3.0, but your Gemfile requires rake 10.4.2. Prepending `bundle exec` to your command may solve this.

This error usually appears when you use a Gem command whose version differs from the Gem version in your Gemfile. For example, let's say your default version of rake is version 11.3.0, but you're in a directory where the Gemfile wants version 10.4.2. This is the situation in Figure 5 and our Gemfile.

When you run rake from the command line, your system will find and execute rake version 11.3.0; your shell doesn't know about Gemfiles, so it just invokes the version of rake it finds in the PATH.

However, rake sometimes runs code that Bundler manages but isn't part of rake, and that's where things get ugly. When that code runs, it checks your Gemfile.lock, and sees that it needs rake 10.4.2, so it tries to load and run it. Unfortunately, rake is already running, but it is version 11.3.0. Since you can't run two versions of rake in the same process, the require fails with a LoadError.

Fortunately, the solution is easy: the error message tells you what to do. All you have to do is run the command with bundle exec:

$ bundle exec rake

This changes the environment so that rake 10.4.2 runs instead of your system default, 11.3.0; now, when rake runs the external code, Bundler sees that you are already running 10.4.2, so everything is okay, and execution continues.

Discrepancies with rake and other executables are the main reason to use bundle exec; it's easy to find advice that says "always use bundle exec rake", and this is good advice. However, this problem can happen with other commands as well. Any Gem command that requires other Gems may load a Gem that conflicts with your app's requirements. bundle exec is the easiest way to fix this issue.

binstubs

Earlier, we mentioned the binstubs feature. binstubs is an alternative to using bundle exec. It sets up a directory of short Ruby scripts (wrappers) with the same names as executables installed by your Gems. By default, binstubs names this directory as bin, but you should override that if your app also needs a bin directory of its own.

The scripts in the binstubs-provided directory effectively replace bundle exec; if you include the directory in your PATH, you can avoid using bundle exec.

When Things Go Wrong

In all likelihood, you will use Bundler to solve problems; it won't usually create problems. However, that doesn't mean that things won't go wrong.

We've already talked about one major problem that you will surely encounter:

Gem::LoadError: You have already activated ...

This error merely tells you that you need to use bundle exec to run the command.

Another common issue occurs when you try to run your app, and you get something like this:

in `require': cannot load such file -- colorize (LoadError)

This message means that bundler/setup can't find the named Gem (colorize here). However, you've confirmed that the Gem is installed, has the proper permissions, and you're using the proper version of Ruby and the gem command. The problem here is that the Gemfile.lock file doesn't list the colorize Gem; bundler/setup insists that your Gemfile.lock contains all needed Gems. To add this Gem to yours, add it to your Gemfile, then run bundle install again to generate a new Gemfile.lock file.

Another potential issue is that you may use the wrong version of the bundler command. Remember that Bundler is a RubyGem, and every Ruby version on your system has its own Gems; this includes the bundle command. If you use bundle from version 2.2.2 of Ruby when you mean to use Ruby 2.3.1, you may end up with unexpected results. For instance, if your Gemfile lists a specific version of a Gem that only runs under Ruby 2.3.0 or higher, the bundle command will fail to find a Gem that meets that requirement. Make sure you use the correct version of bundle.

Here are some more things to try if problems continue:

• Remove your Gemfile.lock and run bundle install again. This creates a new Gemfile.lock file.

• Remove the .bundle directory and its contents from your project directory and run bundle install again.

• If you're using the binstubs feature, remove the directory used by binstubs and run bundle install --binstubs again. Don't do this if you aren't using binstubs.

• Remove and reinstall Bundler:

  $ gem uninstall bundler
  $ gem install bundler
  

• If gem list shows that either rubygems-bundler or open_gem are installed, uninstall them. These old Gems are incompatible with Bundler. Repeat the above items if you remove either Gem.

• Issue this command in the command line from your app's top-level directory:

  $ rm Gemfile.lock ; DEBUG_RESOLVER=1 bundle install
  

  This command removes the Gemfile.lock file, then runs bundle install while producing debug information. You can use the debug information to see how Bundler resolves each Gem. This can be valuable when you aren't sure if your app is loading the correct Gems. Note that you must include the rm Gemfile.lock part; this mode only produces useful output when Gemfile.lock doesn't exist. For additional information on how Bundler's "resolver" works, see How Does Bundler Bundle.

More Info About Bundler

For a little bit of history about Ruby, RubyGems, and Bundler, check out this informative article.

Summary

Bundler lets you describe exactly which Ruby and Gems you want to use with your Ruby apps. Specifically, it lets you install multiple versions of each Gem under a specific version of Ruby and then use the proper version in your app.

Bundler is a RubyGem, so you must install it like a normal Gem: gem install bundler.

To use Bundler, you provide a file named Gemfile that describes the Ruby and Gem versions you want for your app. You use a DSL described on the Bundler website to provide this information.

Bundler uses the Gemfile to generate a Gemfile.lock file via the bundle install command. Gemfile.lock describes the actual versions of each Gem that your app needs, including any Gems that the Gems listed in Gemfile depend on. The bundler/setup package tells your Ruby program to use Gemfile.lock to determine which Gem versions it should load.

The bundle exec command ensures that executable programs installed by Gems don't interfere with your app's requirements. For instance, if your app needs a specific version of rake but the default version of rake differs, bundle exec ensures that you can still run the specific rake version compatible with your app.

In the next chapter, we'll take a look at Rake, Ruby's answer to the long time Unix development tool, Make. Rake lets you automate a lot of tasks common to many Ruby development projects.