We recently migrated from Circle CI to Github Actions to take advantage of the ability to have all CI runs start immediately no matter how many branches we pushed up at once (on Circle we paid for a limited number of runners, whereas on Github Actions you pay for the minutes used).
One thing that we missed however was the short summary of failures that Circle would give us if any spec failed. Generally this is the only CI output that we are interested in beyond how long it took to run.
On Github Actions however we could only click on the RSpec step and had to scroll through the full output to get to the failures at the bottom. Searching online didn’t turn up others experiencing the same problem which was confusing at first but ultimately lead to the answer: the reason others are not complaining is perhaps in part because they’re using the default RSpec settings, which include the ‘progress’ format for the output.
I like the ‘documentation’ output which is verbose but it allows me to scan spec output locally to evaluate it as a test plan for the code under test. Here’s the output for a small module which aims to strip out PII from comments:
Sanitize
.phone_numbers_in
replaces likely phone number segments with asterisks
.mask(text)
applies all sanitization filters to the supplied text
returns a blank string when given nil
.email_addresses_in
replaces emails with asterisks
Usefulness of the format aside, on CI this format is wasteful and produces 1000+ lines per spec run that the Actions UI forces us to (slowly) download and scroll through.
The simple answer: switch back to --format=progress or even
--format=failures for the most terse output.
This tip is not that interesting on its own, but it serves as a good example of how you can get used to things being a certain way and stop to see how this is holding you back. In the end it was the lack of search results for the problem that triggered the thought to step back and check my assumptions–I had assumed that others must also be experiencing the same problem, but in all likelihood this is not the case. Approaching the problem from this perspective (that we’re the ones doing something unusual, not Github Actions) allowed the answer to appear almost immediately.
If you are looking to print your own summary of spec failures then I would
recommend causing the rspec step to always pass (return 0), then have a
subsequent step load and print the RSpec metadata for failing specs only.