Documentation

How it works

UndercoverCI is a robot who protects your Ruby codebase from untested code changes. It works with every Ruby project that can report test coverage with SimpleCov. Chances are, you're already doing that while running tests on a CI server.

The GitHub App reacts to commit and pull request events by queueing a new coverage check. Once your tests finish and upload the coverage report file, UndercoverCI parses the commit diff and correlates the structure of your changes (classes, modules, methods and blocks) against the uploaded test coverage report. The result is a passed or failed commit check reported to GitHub.

Passed

When the diff lines have full test coverage, you'll see a passed check with an additional breakdown of all changed code locations. Use it to find additional test coverage improvement opportunities in the surrounding lines and methods - reviewing your pull request is a good time to do that.

Failed

When untested lines or branches are found, a failed check will highlight them and ask you to add tests. Once tests are added, the comments will go away and the check status will turn to passed ✅.

---

Getting started

Follow these 3 steps to get started with UndercoverCI and receive automated test coverage comments.

1. Install the app on your GitHub account

Sign up from the home page or just follow the GitHub sign up link to sign up. You'll be prompted to install the app on your GitHub account or organization and give permissions to selected repositories.

Once installed, UndercoverCI will send checks for every commit and pull request. There is just one final step to receive meaningful coverage results in them.

2. Report test coverage data to UndercoverCI

UndercoverCI analyses each commit against a matching test coverage file. Make sure your Ruby project reports test coverage and uploads the report file to UndercoverCI by following the instructions below:

If your CI system isn't listed below, take a look at the general uploader documentation for more details.

3. Verify your setup

You have installed the UndercoverCI GitHub app and configured your CI to publish test coverage reports for analysis. In order to verify that your setup is working, create a branch and push a small code change to your repository – adding a dummy method should suffice to trigger a sample coverage warning:

If you made it here, it means your code review setup with UndercoverCI is ready!

Branch coverage

UndercoverCI and undercover CLI (starting with version >= 0.4.0) both accept coverage reports with branch coverage data.

The configuration example already shows how to enable branch coverage in SimpleCov, which is all you need to receive more fine-grained coverage checks.

If you decide to enable branch coverage reporting, every changed line of code with at least one uncovered branch will be flagged and result in a GitHub annotation for the entire method. This is a more rigorous approach compared to just validating line-by-line test coverage.

Bonus: shorten the feedback loop with the local CLI

UndercoverCI coverage comments are generated by the undercover ruby gem that you can use locally too. While GitHub Check comments provide coding standards and consistency, you can shorten your feedback loop and check for missing coverage with every local commit thanks to the undercover CLI command. You can install it from RubyGems with:

          gem install undercover
        

Then, check for missing coverage inside uncommited changes by running specs and invoking undercover. Use the --compare flag to compare against a specific commit or branch, which is similar to what UndercoverCI performs when analysing your pushed pull requests:

          undercover --compare base-branch
          undercover --compare HEAD~1
        

Visit the undercover GitHub page to see more examples including integrations with Overcommit and Pronto hooks.

---

Configuration

UndercoverCI uploader

The uploader script runs in your CI build environment and uploads coverage reports to UndercoverCI. This way the coverage files are processed against a git diff and generate a commit check. A sample call for the grodowski/undercover repository built on CircleCI could look like this:

          ruby -e "$(curl -s https://undercover-ci.com/uploader.rb)" -- \
                    --repo grodowski/undercover \
                    --commit $CIRCLE_SHA1 \
                    --lcov coverage/lcov/undercover.lcov
        

The uploader.rb script is securely downloaded and evaluated with the required command-line options:

--repo

The $org/$repo formatted name matching exactly how your repository appears on GitHub.

--commit

Current build commit SHA to identify on GitHub. This value will be provided by the CI build environment, e.g. $CIRCLE_SHA1 for CircleCI, ${{ github.event.pull_request.head.sha || github.sha }} for GitHub Actions or $TRAVIS_COMMIT for TravisCI. Consult your CI service documentation to get the right head SHA value for your build environment.

--lcov

A relative path to the coverage report, will look like coverage/lcov/$reponame.lcov unless SimpleCov's config defaults were changed.

Parallel tests

If your CI build runs tests in parallel, there's an extra merge step to be performed before uploading coverage results, because the uploader script only accepts a single file at the time of writing this document. While partial coverage results might be supported in a future release, this Ruby snippet should get you started with merging multiple coverage reports:

          #!/usr/bin/env ruby
          # frozen_string_literal: true

          require 'simplecov'
          require 'simplecov-lcov'

          puts('Merging coverage results from parallel CircleCI tests containers into a single LCOV report...')

          SimpleCov.collate(Dir['/home/circleci/rspec/*.resultset.json']) do
            enable_coverage(:branch)
          end

          report_path = ARGV[0] || 'coverage.lcov'
          SimpleCov.formatter = SimpleCov::Formatter::LcovFormatter
          SimpleCov::Formatter::LcovFormatter.config.report_with_single_file = true
          SimpleCov::Formatter::LcovFormatter.config.single_report_path = report_path

          merged_result = SimpleCov.result
          merged_result.format!

          if File.size(report_path).zero?
            puts('Written report has 0 bytes')
            exit 1
          end
          puts("Done! LCOV saved to #{SimpleCov::Formatter::LcovFormatter.config.single_report_path}")
        

Ignoring/skipping coverage

Pull requests in projects with low or nonexistent test coverage are likely to generate large numbers of check warnings. While the default workflow would be to address them before the PR approval, your strategy might be different.

In order to acknowledge an untested change and remove the UndercoverCI warning with the intention to improve later (or never), you can wrap the code block with the :nocov: syntax, e.g.

          # :nocov:
          def skip_this_method
            never_reached
          end
          # :nocov:
        

Read more about the :nocov: syntax in SimpleCov's readme.

API V1

To access Undercover HTTP API, configure your per-user auth token using the API section of the Settings page.

GET /v1/checks/:commit_sha

Returns the coverage check information for the given commit SHA. Example:

          curl -H "Authorization: Bearer $UNDERCOVER_API_TOKEN" \
          https://undercover-ci.com/v1/checks/aa412c105eff7a7af918fc422bf5085101295798
        

Response format:

          {
            "id": 327889,
            "head_sha": "aa412c105eff7a7af918fc422bf5085101295798",
            "base_sha": "master",
            "state": "complete",
            "state_log": [
              {
                "to": "awaiting_coverage",
                "ts": "2023-11-25T15:27:14Z",
                "from": "created"
              },
              {
                "to": "in_progress",
                "ts": "2023-11-25T15:28:48Z",
                "from": "awaiting_coverage"
              },
              {
                "to": "complete",
                "ts": "2023-11-25T15:37:22Z",
                "from": "in_progress"
              }
            ],
            "repo_full_name": "twitchy-tortoise/undercover-test"
          }
        

GET /v1/checks/:commit_sha/coverage

Returns the coverage report for the given commit SHA. The coverage report will be in the LCOV format. The `curl` example below includes the -L flag to ensure curl follows the redirect to the storage provider.

          curl -LH "Authorization: Bearer $UNDERCOVER_API_TOKEN" \
          -o check_coverage.lcov \
          https://undercover-ci.com/v1/checks/aa412c105eff7a7af918fc422bf5085101295798/coverage
        

---

Development

The UndercoverCI Github App and the underlying undercover gem are developed and available on GitHub under a standard MIT license. Your contributions are welcome!

• grodowski/undercover-ci
• grodowski/undercover

---