Documentation

Learn how to install UndercoverCI for your repos, report coverage data and use the local CLI.

How it works

UndercoverCI is a robot who protects your Ruby codebase from untested code changes. It works with every Ruby project that can report test coverage with SimpleCov. Chances are, you're already doing that while running tests on a CI server.

A passed coverage check.

The GitHub App reacts to commit and pull request events by queueing a new coverage check. Once your tests finish and upload the coverage report file, UndercoverCI parses the commit diff and correlates the structure of your changes (classes, modules, methods and blocks) against the uploaded test coverage report. The result is a passed or failed commit check reported to GitHub.

Passed

When the diff lines have full test coverage, you'll see a passed check with an additional breakdown of all changed code locations. Use it to find additional test coverage improvement opportunities in the surrounding lines and methods - reviewing your pull request is a good time to do that.

Failed

When untested lines or branches are found, a failed check will highlight them and ask you to add tests. Once tests are added, the comments will go away and the check status will turn to passed ✅.

Every check from UndercoverCI, no matter whether passed or failed, includes some general stats: a list of changed methods, classes and blocks, their filenames, line coverage and branch coverage values.

Getting started

Follow these 3 steps to get started with UndercoverCI and receive automated test coverage comments.

1. Install the app on your GitHub account

Sign up from the home page or just follow the GitHub sign up link to sign up. You'll be prompted to install the app on your GitHub account or organization and give permissions to selected repositories.

Once installed, UndercoverCI will send checks for every commit and pull request. There is just one final step to receive meaningful coverage results in them.

2. Report test coverage data to UndercoverCI

UndercoverCI analyses each commit against a matching test coverage file. Make sure your Ruby project reports test coverage and uploads the report file to UndercoverCI by following the instructions below:

If your CI system isn't listed below, take a look at the general uploader documentation for more details.

  1. Add simplecov and simplecov-lcov gems to report test coverage. Configure them in spec_helper.rb or test_helper.rb by appending this snippet to the beginning of the file:
    
                  require 'simplecov'
                  require 'simplecov-lcov'
                  SimpleCov::Formatter::LcovFormatter.config.report_with_single_file = true
                  SimpleCov.formatter = SimpleCov::Formatter::LcovFormatter
                  SimpleCov.start do
                    add_filter(/^\/spec\//) # For RSpec, use `test` for MiniTest
                    enable_coverage(:branch)
                  end
                
  2. Update your CI configuration (.circleci/config.yml in case of CircleCI) to use our uploader script after running tests. Replace things in bold with your values. Here's an example:
    
                  ruby -e "$(curl -s https://undercover-ci.com/uploader.rb)" -- \
                    --repo $your-username/$repository-name \
                    --commit $CIRCLE_SHA1 \
                    --lcov coverage/lcov/$repository-name.lcov
                
  3. Store your changes, push a commit and... profit! Soon you'll receive your first GitHub check looking for untested code. 🔍✨

* Undercover requires a single coverage report file. Check out how to merge coverage reports if you use parallel builds to run specs in CI.

  1. Add simplecov and simplecov-lcov gems to report test coverage. Configure them in spec_helper.rb or test_helper.rb by appending this snippet to the beginning of the file:
    
                  require 'simplecov'
                  require 'simplecov-lcov'
                  SimpleCov::Formatter::LcovFormatter.config.report_with_single_file = true
                  SimpleCov.formatter = SimpleCov::Formatter::LcovFormatter
                  SimpleCov.start do
                    add_filter(/^\/spec\//) # For RSpec, use `test` for MiniTest
                    enable_coverage(:branch)
                  end
                
  2. Update your CI configuration (e.g. .github/workflows/ruby.yml) to use the uploader script after running tests. Replace things in bold with your values. Here's a full workflow example:
    
                  name: Rails Unit Tests
                  on: [push, pull_request]
                  jobs:
                    build:
                      runs-on: ubuntu-latest
                      steps:
                      - uses: actions/[email protected]
                      - name: Set up Ruby 2.7
                        uses: actions/[email protected]
                        with:
                          ruby-version: 2.7
                      - name: Build and test with Rake
                        env:
                          RAILS_ENV: test
                        run: |
                          sudo apt-get -yqq install libsqlite3-dev
                          gem install bundler
                          bundle install --jobs 4 --retry 3
                          bundle exec rails webpacker:install
                          bundle exec rails db:create
                          bundle exec rails db:migrate
                          bundle exec rails test
                          ruby -e "$(curl -s https://undercover-ci.com/uploader.rb)" -- \
                            --repo $your-username/$repository-name \
                            --commit $GITHUB_SHA \
                            --lcov coverage/lcov/$repository-name.lcov
                
  3. Store your changes, push a commit and... profit! Soon you'll receive your first GitHub check looking for untested code. 🔍✨

* Undercover requires a single coverage report file. Check out how to merge coverage reports if you use parallel builds to run specs in CI.

3. Verify your setup

You have installed the UndercoverCI GitHub app and configured your CI to publish test coverage reports for analysis. In order to verify that your setup is working, create a branch and push a small code change to your repository – adding a dummy method should suffice to trigger a sample coverage warning:

A GitHub code annotation triggered by untested diff lines.

If you made it here, it means your code review setup with UndercoverCI is ready!

Branch coverage

UndercoverCI and undercover CLI (starting with version >= 0.4.0) both accept coverage reports with branch coverage data.

The configuration example already shows how to enable branch coverage in SimpleCov, which is all you need to receive more fine-grained coverage checks.

If you decide to enable branch coverage reporting, every changed line of code with at least one uncovered branch will be flagged and result in a GitHub annotation for the entire method. This is a more rigorous approach compared to just validating line-by-line test coverage.

Bonus: shorten the feedback loop with the local CLI

UndercoverCI coverage comments are generated by the undercover ruby gem that you can use locally too. While GitHub Check comments provide coding standards and consistency, you can shorten your feedback loop and check for missing coverage with every local commit thanks to the undercover CLI command. You can install it from RubyGems with:


          gem install undercover
        
Then, check for missing coverage inside uncommited changes by running specs and invoking undercover. Use the --compare flag to compare against a specific commit or branch, which is similar to what UndercoverCI performs when analysing your pushed pull requests:

          undercover --compare base-branch
          undercover --compare HEAD~1
        

Visit the undercover GitHub page to see more examples including integrations with Overcommit and Pronto hooks.


Configuration

UndercoverCI uploader

The uploader script runs in your CI build environment and uploads coverage reports to UndercoverCI. This way the coverage files are processed against a git diff and generate a commit check. A sample call for the grodowski/undercover repository built on CircleCI could look like this:


          ruby -e "$(curl -s https://undercover-ci.com/uploader.rb)" -- \
                    --repo grodowski/undercover \
                    --commit $CIRCLE_SHA1 \
                    --lcov coverage/lcov/undercover.lcov
        
The uploader.rb script is securely downloaded and evaluated with the required command-line options:

--repo
The $org/$repo formatted name matching exactly how your repository appears on GitHub.
--commit
Current build commit SHA to identify on GitHub. This value will be provided by the CI build environment, e.g. $CIRCLE_SHA1 for CircleCI, $GITHUB_SHA for GitHub Actions or $TRAVIS_COMMIT for TravisCI.
--lcov
A relative path to the coverage report, will look like coverage/lcov/$reponame.lcov unless SimpleCov's config defaults were changed.

Parallel tests

If your CI build runs tests in parallel, there's an extra merge step to be performed before uploading coverage results, because the uploader script only accepts a single file at the time of writing this document. While partial coverage results might be supported in a future release, this Ruby snippet should get you started with merging multiple coverage reports:


          #!/usr/bin/env ruby
          # frozen_string_literal: true

          require 'simplecov'
          require 'simplecov-lcov'

          puts('Merging coverage results from parallel CircleCI tests containers into a single LCOV report...')

          results = []
          Dir['/home/circleci/rspec/*.resultset.json'].each do |path|
            resultset = JSON.parse(File.read(path))
            resultset.each do |_, data|
              cmd = File.basename(path)
              results << SimpleCov::Result.from_hash(cmd => data)
            end
            print '.'
          end
          puts

          merged_result = SimpleCov::ResultMerger.merge_results(*results)

          SimpleCov::Formatter::LcovFormatter.config.report_with_single_file = true
          SimpleCov::Formatter::LcovFormatter.config.single_report_path = ARGV[0] || 'coverage.lcov'
          SimpleCov::Formatter::LcovFormatter.new.format(merged_result)

          puts("Done! LCOV saved to #{SimpleCov::Formatter::LcovFormatter.config.single_report_path}")
        


Development

The UndercoverCI Github App and the underlying undercover gem are developed and available on GitHub under a standard MIT license. Your contributions are welcome!