Last updated at Wed, 07 Feb 2024 19:20:17 GMT

Introduction

As Metasploit adopts community best practices related to testing and code structure, we've started breaking up our new (and part of our old) work into separate projects that can developed and tested independently.  These smaller projects take the form of Ruby gems and Rails::Engines.  As we've made more and more gems, we've gotten a repeatable process for making new gems, but we thought our process might we useful for other developers in the community that are looking for a newer guide on how to setup ruby gems from scratch and get the most out of all the free tools available to open source projects.

Skeleton

Github

Your gem will be open source, so you're going to want to host your source control server on one of the free source control services.  We use github.com and I'd recommend it because other services are likely to have Github integration before support for the other source control service like Gitorious or Bitbucket.

Account Setup

If you don't have a Github account, set one up by signing up at https://github.com.

Repository Creation

Once logged into your Github account, when you go to https://github.com, you'll see two columns, the left hand column is an activity stream while the right column lists repository. Click the big green, " New Repository" button to add a repository.

Repository Naming

What your name your repository is up to you, but you should keep rubygems naming conventions in mind:

Other Options

You can leave the rest of the options with their default values and click "Create repository".  After the repository is created, Github will display options for upload your code, but we haven't created any code yet, so let's do that.

RVM Installation

When developing Ruby projects on OSX (or Linux) we recommend using rvm (Ruby Version Manager), it's what we use.  Go to https://rvm.io and follow the installation instructions.

Local Repository Creation

The point of RVM is you can have a different version of ruby and gems that you depend on for each project you work on.  We'll use the ability to setup the tools to create your gem's repository.  In the below steps, replace <GEM_NAME> with the name of the repository you created on Github and <GITHUB_USERNAME> with your username on Github.

  1. Create a gemset for your gem: rvm use --create ruby-2.1@<GEM_NAME>

  2. Create the parent directory for your local git repository

    1. cd ~
    2. mkdir git
    3. mkdir <GITHUB_USERNAME>
    4. cd <GITHUB_USERNAME>

  3. Create the gem skeleton using bundle: bundle gem <GEM_NAME>

  4. cd  <GEM_NAME>

  5. bundle gem won't create a commit in the git repository, so do that now: git commit -m "bundle gem <GEM_NAME>"

Now, look back at Github in your browser and you'll see the instructions for pushing your local git repository to the remote repository on Github.  I've copied the instructions here for easier use:

git remote add origin git@github.com:/.git git push -u origin master

Once the push has finished, refresh your browser and you'll see the gem skeleton in your Github repository.

Tool Integration

With the push to Github, you've now successfully started an open source gem project, but instead of stopping there, let's go on to add a common set of tools to take advantage of the project being open source.

Local Tools

We'll start with setting up some local tools that will help with development on our own machine.

RSpec

Metasploit uses [RSpec](http://rspec.info/) for a unit test framework.  RSpec can set itself up on installation, but must also be declared in the gemspec file.

  1. Add the rspec as a development dependency to <GEM_NAME.gemspec>:
Gem::Specification.new do |spec|
    # ...
    spec.add_development_dependency 'rspec', '~> 3.1'
    # ...
end
  1. Install the new gems: bundle install
  2. Use rspec to set itself up: rspec --init

rspec --init will choose some good defaults and include comments on other options you can enable in spec/spec_helper.rb, but for Metasploit we want to make sure we're using the rspec 3 features as well as possible, so we turn on some additional options and our spec/spec_helper.rb looks like this:

$LOAD_PATH.unshift File.expand_path('../../lib', __FILE__)

require 'GEM_NAME'

RSpec.configure do |config|
  config.expose_dsl_globally = false

  # These two settings work together to allow you to limit a spec run
  # to individual examples or groups you care about by tagging them with
  # `:focus` metadata. When nothing is tagged with `:focus`, all examples
  # get run.
  config.filter_run :focus
  config.run_all_when_everything_filtered = true

  # allow more verbose output when running an individual spec file.
  if config.files_to_run.one?
    # RSpec filters the backtrace by default so as not to be so noisy.
    # This causes the full backtrace to be printed when running a single
    # spec file (e.g. to troubleshoot a particular spec failure).
    config.full_backtrace = true
  end

  # Print the 10 slowest examples and example groups at the
  # end of the spec run, to help surface which specs are running
  # particularly slow.
  config.profile_examples = 10

  # Run specs in random order to surface order dependencies. If you find an
  # order dependency and want to debug it, you can fix the order by providing
  # the seed, which is printed after each run.
  #     --seed 1234
  config.order = :random

  # Seed global randomization in this process using the `--seed` CLI option.
  # Setting this allows you to use `--seed` to deterministically reproduce
  # test failures related to randomization by passing the same `--seed` value
  # as the one that triggered the failure.
  Kernel.srand config.seed

  config.expect_with :rspec do |expectations|
    # Enable only the newer, non-monkey-patching expect syntax.
    expectations.syntax = :expect
  end

  # rspec-mocks config goes here. You can use an alternate test double
  # library (such as bogus or mocha) by changing the `mock_with` option here.
  config.mock_with :rspec do |mocks|
    # Enable only the newer, non-monkey-patching expect syntax.
    # For more details, see:
    #   - http://teaisaweso.me/blog/2013/05/27/rspecs-new-message-expectation-syntax/
    mocks.syntax = :expect

    mocks.patch_marshal_to_support_partial_doubles = false

    # Prevents you from mocking or stubbing a method that does not exist on
    # a real object.
    mocks.verify_partial_doubles = true
  end
end

To finish the RSpec setup, we want it runnable using rake.  We can do this by adding the following to Rakefile:

require 'rspec/core/rake_task'

RSpec::Core::RakeTask.new(:spec)

task :default => :spec

You can test that your specs by calling rspec, rake spec, or just rake now, but no tests are defined, so rspec isn't testing anything for us.  To get some benefit from rspec we'll need to add spec file.  Usually, we'd test the gem versioning at this point.

  1. mkdir spec/<GEM_NAME>
  2. edit spec/<GEM_NAME>/version_spec.rb
  3. Add the following code to spec/<GEM_NAME>/version_spec.rb<:
require 'spec_helper'

RSpec.describe GEM_NAME do
  context 'CONSTANTS' do
    context 'VERSION' do
      subject(:version) {
        described_class::VERSION
      }

      it { is_expected.to be_a String }
      it { is_expected.to match_regex(/\d+.\d+.\d+(-[a-zA-Z0-9]+)*/) }
    end
  end
end
  1. Run your new spec: rake spec

SimpleCov

We can use RSpec to test our code, but how do we know our tests are testing all the gem code?  We need to use a tool that measure test coverage, SimpleCov.

  1. Add simplecov as a development dependency to <GEM_NAME>.gemspec:
Gem::Specification.new do |spec|
    # ...
    spec.add_development_dependency 'simplecov'
    # ...
end
  1. Install simplecov: bundle install
  2. Require simplecov before any other file in spec/spec_helper.rb
# require before anything else so coverage is shown for all project files
require 'simplecov'
  1. Metasploit uses a SimpleCov configuration file, .simplecov, that works both with rake spec and Rubymine, our Ruby IDE of choice:
# RM_INFO is set when using Rubymine.  In Rubymine, starting SimpleCov is
# controlled by running with coverage, so don't explicitly start coverage (and
# therefore generate a report) when in Rubymine.  This _will_ generate a report
# whenever `rake spec` is run.
unless ENV['RM_INFO']
  SimpleCov.start
end
 
SimpleCov.configure do
  # ignore this file
  add_filter '.simplecov'
 
  # Rake tasks aren't tested with rspec
  add_filter 'Rakefile'
  add_filter 'lib/tasks'
 
  #
  # Changed Files in Git Group
  # @see http://fredwu.me/post/35625566267/simplecov-test-coverage-for-changed-files-only
  #
 
  untracked = `git ls-files --exclude-standard --others`
  unstaged = `git diff --name-only`
  staged = `git diff --name-only --cached`
  all = untracked + unstaged + staged
  changed_filenames = all.split("\n")
 
  add_group 'Changed' do |source_file|
    changed_filenames.detect { |changed_filename|
      source_file.filename.end_with?(changed_filename)
    }
  end
 
  add_group 'Libraries', 'lib'
 
  #
  # Specs are reported on to ensure that all examples are being run and all
  # lets, befores, afters, etc are being used.
  #
 
  add_group 'Specs', 'spec'
end
  1. Now, when you run rake spec, you'll generate a coverage report.
  2. Open the coverage report open coverage/index.html

YARD

By default, Ruby comes with a documentation format called RDoc, but for Metasploit, we use YARD because it allows for more structured documentation with the addition of @tags that would be familiar to anyone that's used Doxygen or Javadoc and the ability to write extensions as Metasploit has done with yard-metasploit-erd, which allows use to include ERDs (Entity-Relationship Diagram in our documentation to make working with the database easier.

At Rapid7, we've just released metasploit-yard, so we can share a common rake yard task.  The READMEexplains how to set it up:

  1. Add metasploit-yard as a development dependency to <GEM_NAME>.gemspec:
Gem::Specification.new do |spec|
    # ...
    spec.add_development_dependency 'metasploit-yard', '~> 1.0'
    # ...
end
  1. Install metasploit-yard: bundle install
  2. Add the following to your Rakefile to load yard.rake from metasploit-yard:
# Use find_all_by_name instead of find_by_name as find_all_by_name will return pre-release versions
gem_specification = Gem::Specification.find_all_by_name('metasploit-yard').first

Dir[File.join(gem_specification.gem_dir, 'lib', 'tasks', '**', '*.rake')].each do |rake|
  load rake
end
  1. Run rake yard to see what you need to document.

Remote Tools

With RSpec, SimpleCov, and YARD setup, we can run tests and generate documentation locally, but doing everything locally is not enough, we need a second system, that isn't our develop machine to run these steps too to ensure that they aren't passing just because of some fluke of our local machine.

Travis CI

To test remotely, Metasploit uses Travis CI.  Travis CI is free for open source software and integrates well with github, as Travis CI is able to automatically test that Pull Requests pass our specs.  This is a big help on metasploit-framework where we're up to 3785 pull requests and 9200 builds.  Without Travis CI we'd either have had to run those all by hand, go through an error-prone and time-consuming manual process, or roll a Jenkins build pipeline to handle all of that, which would be a significant maintenance overhead.

On Travis CI, we can build against multiple implementations and versions of Ruby.  Metasploit tries to build all versions of MRI Ruby and the latest stable versions of JRuby and Rubinius (rbx in the .travis.yml).  We need to setup a Travis-CI account and then push a .travis.yml to Github:

  1. Go to https://travis-ci.org/
  2. Click "Sign in with Github" in the top-right corner
  3. On the left sidebar, click the " " button to add a new repository for Travis CI to build.
  4. Select a repository to build by changing the slider on the right hand side of the table row from "Off" to "On"
  5. Add a .travis.ymlto the root of your local git repository:
env:
  matrix:
    - RAKE_TASK=spec
language: ruby
matrix:
  # documentation coverage is not influenced by ruby implementation, so only run once
  include:
    - rvm: 2.1
      env: RAKE_TASK=yard
rvm:
  - '1.9.3'
  - '2.0'
  - '2.1'
  - 'ruby-head'
  - 'jruby-19mode'
  - 'rbx-2.2'
script: "bundle exec rake $RAKE_TASK"
  1. Add a Travis CI badge to your README.md so users can quickly see if your project's build is failing when visiting the repository on Github: <span>[![Build Status](</span>[https://travis-ci.org/](https://travis-ci.org/)<span><GITHUB_USER>/<GEM_NAME>.svg?branch=master)](</span>[https://travis-ci.org/](https://travis-ci.org/)<span><GITHUB_USER>/<GEM_NAME>)</span>. When done correctly it will look like this:
  2. Commit your change: git commit -am "Travis CI"
  3. Push your changes: git push
  4. Watch the build on Travis CI

CodeClimate

CodeClimate is a automated code quality analyzer. Metasploit uses it for both our open source and private repositories; however, CodeClimate can't cope with the sure volume of code in metasploit-framework: metasploit-framework itself is a DoS attack on static analysis tools.  Like Travis CI, CodeClimate has a badge, which shows the Code Quality using a 4.0 GPA scale. CodeClimate also monitors code coverage, so we'll be adding a coverage report on top of SimpleCov.

  1. Go to https://codeclimate.com and sign up for an account
  2. On your dashboard click the "Add Open Source Repo" button
  3. Enter your repo name: <GITHUB_USER>/&ltGEM_NAME>
  4. Click "Import Repo from Github"
  5. On the repository page, click the "Setting" link.
  6. Click the "Test Coverage" button in the left sidebar.
  7. Copy the CODECLIMATE_REPO_TOKEN
  8. You'll want to encrypt the CODECLIMATE_REPO_TOKEN on travis-ci to prevent forks from affecting the original repo's coverage, so install the travis gem: gem install travis
  9. Encrypt the token: travis encrypt CODECLIMATE_REPO_TOKEN=<YOUR_TOKEN_HERE>
  10. Put the encrypted value (remember to substitute your value and not to use the example value below) in your .travis.yml:
.travis.yml:
env:
  global:
    secure: "G0LDGrupZ+RAFzoPwd6bjfrWfwoU/V9RTswQXIUNmi640rW/CP86a8F9hQcAXdUwy7Ag1cwmlEE
  1. Add the CodeClimate badge to the README: <span>[![Code Climate](</span>[https://codeclimate.com/github/](https://codeclimate.com/github/)<span><GITHUB_USER>/<GEM_NAME>.png)](</span>[https://codeclimate.com/github/](https://codeclimate.com/github/)<span><GITHUB_USER>/<GEM_NAME>)</span>
  2. Add the codeclimate-test-reporteras a development dependency:
Gem::Specification.new do |spec|
  # ...
  spec.add_development_dependency 'codeclimate-test-reporter'
  # ...
end
  1. bundle install
  2. Add following after require 'simplecov'in spec/spec_helper.rb
# ...

require 'codeclimate-test-reporter'

if ENV['TRAVIS'] == 'true'
  formatters = []

  # don't use `CodeClimate::TestReporter.start` as it will overwrite some .simplecov settings
  if CodeClimate::TestReporter.run?
    formatters << CodeClimate::TestReporter::Formatter
  end

  SimpleCov.formatter = SimpleCov::Formatter::MultiFormatter[
    *formatters
  ]
end

# ...
  1. git commit -am "CodeClimate"
  2. git push

Coveralls

Although CodeClimate has a coverage reporter, it only reports if there is coverage issues, to view the actual coverage reports online, as outputed locally by simplecov, Metasploit uses Coveralls.

  1. Go to https://coveralls.io
  2. Click "Sign up free with Github" in the top-right corner
  3. Click the "Add Repo" button on the Your Repositories screen
  4. Change the slider from "Off" to "On" next to your repository
  5. Add the Coveralls badge to your README: <span>[![Coverage Status](</span>[https://img.shields.io/coveralls/](https://img.shields.io/coveralls/)<span><GITHUB_USER>/<GEM_NAME>.svg)](</span>[https://coveralls.io/r/](https://coveralls.io/r/)<span><GITHUB_USER>/<GEM_NAME>)</span>, which will look like this:
  6. Add coveralls as a development dependency:
Gem::Specification.new do |spec|
  # ...
  spec.add_development_dependency 'coveralls'
  # ...
end
  1. bundle install
  2. Change the SimpleCov.formatters in spec/spec_helper.rb to include Coveralls::SimpleCov::Formatter:
Coveralls::SimpleCov::Formatter:
# ...

require 'codeclimate-test-reporter'
require 'coveralls'

if ENV['TRAVIS'] == 'true'
  formatters = []

  # don't use `CodeClimate::TestReporter.start` as it will overwrite some .simplecov settings
  if CodeClimate::TestReporter.run?
    formatters << CodeClimate::TestReporter::Formatter
  end

  formatters << Coveralls::SimpleCov::Formatter

  SimpleCov.formatter = SimpleCov::Formatter::MultiFormatter[
    *formatters
  ]
end

# ...

  1. Add a .coveralls.yml file

    service_name: travis-ci

  2. git commit -am "Coveralls"

  3. git push

Gemnasium

Metasploit's various gems depend on many other gems from the community.  We need to keep track of new versions of those gems in case there are vulnerabilities both so we can update our usage of those gems and so we can make Metasploit Framework modules to exploit those vulnerability.  To monitor our gems, we use Gemnasium.  Gemnasium has sent us alerts about Rails vulnerabilities that have led to Rapid7's rapid vulnerability mitigation response with new updates across our Portfolio multiple times in the past.

  1. Go to https://gemnasium.com
  2. Click "Sign up"
  3. Click "Sign up with Github"
  4. Click "Authorize application" on Github
  5. Click " Add New Project"
  6. Click "Hosted on Github"
  7. Click "Allow Access" in the "Public Project" column so Gemnasium can automatically configure the Github hooks
  8. Click "Authorize application" on Github to authorize the expanded permissions for Gemnasium
  9. Click your username to expand the list of your repositories
  10. Check the repositories to monitor
  11. Click Submit
  12. Add the Gemnasium badge to the README: <span>[![Dependency Status](</span>[https://gemnasium.com/](https://gemnasium.com/)<span><GITHUB_USER>/<GEM_NAME>.svg)](</span>[https://gemnasium.com/](https://gemnasium.com/)<span><GITHUB_USER>/<GEM_NAME>)</span>
  13. git commit -am "Gemnasium"
  14. git push

Rubygems

Although you haven't released your gem yet, once you do, you'll want users that visit your github repository to be able to find the latest released gem, which you can do with a Rubygems version badge

  1. Add to your README: <span>[![Gem Version](</span>[https://badge.fury.io/rb/](https://badge.fury.io/rb/)<span><GEM_NAME>.svg)](</span>[http://badge.fury.io/rb/](http://badge.fury.io/rb/)<span><GEM_NAME>)</span>
  2. git commit -am "Gemnasium"
  3. git push

Inch CI

Inch CI checks our documentation as a second review of the checking Metasploit already does with metasploit-yard.

  1. Go to http://inch-ci.org
  2. Enter the URL to your github repository: [https://github.com/](https://github.com/)<span><GITHUB_USER>/<GEM_NAME></span>
  3. In the top-right corner click the "doc" badge
  4. Select "SVG" for the image format
  5. Copy the "Markdown" badge
  6. Add it to your README
  7. git commit -am "Inch CI"
  8. git push

Pull Review

PullReview is another static analysis tool like CodeClimate.  Metasploit uses both because PullReview has a more detailed reviews and better handling of branches than CodeClimate, but CodeClimate is cheaper for our team for private repositories.  Just like CodeClimate, PullReview can't handle analysing metasploit-framework, but instead of lagging behind and reporting odd or out-of-date analysis, PullReview just won't return an analysis. Metasploit has had discussion with PullReview and they were nice enough to send us an offline analysis.

  1. Go to https://www.pullreview.com
  2. Enter your github username and sign up
  3. Click "Authorize application" on Github
  4. Click "Authorize access to my repositories on Github"
  5. Click "Authorize application" on Github to authorize PullReview to post reviews and setup hooks
  6. Click the " Review" button next to repository you want reviewed
  7. Add the badge to your README: <span>[![PullReview stats](</span>[https://www.pullreview.com/github/](https://www.pullreview.com/github/)<span><GITHUB_USER>/<GEM_NAME>/badges/master.svg?)](</span>[https://www.pullreview.com/github/](https://www.pullreview.com/github/)<span><GITHUB_USER>/<GEM_NAME>/reviews/master)</span>, which should look like this: , which is a bit cooler than the CodeClimate badge.

Conclusion

So, that's everything.  Don't be intimidated by the length of this article: it takes about a day or less for me to set this all up on new projects.  If there are open source tools you think Metasploit should be using (such a static analysis tool that can handle metasploit framework and our odd Metasploit module format) let us know in the comments below.