Support for Github Actions job summaries

[Tolsto]

Github Actions has a new features that allows to add arbitrary markdown code for job summaries. It would be great if Knapsack Pro would support that out of the box when running on Github.

https://github.blog/2022-05-09-supercharging-github-actions-with-job-summaries/

This is a library for dotnet that has already support for summaries: https://github.com/Tyrrrz/GitHubActionsTestLogger

[ArturT]

We can use rspec_junit_formatter gem to generate an XML report with tests results. We would need something to convert the XML report to a nice markdown summary.

[singhprd]

Hey - I've done exactly what you're looking for with this code:

This assumes rspec running with --json output (rather than junit, although you could do something similar with junit), written to the spec/reports/rspec*.json directory. It works well for us in conjunction with the github artifacts uploader/downloader collecting the json formatted spec result from each node:

      - name: Upload json rspec reports
        uses: actions/upload-artifact@v3
        if: always()
        continue-on-error: true
        with:
          name: rspec-json-reports
          retention-days: 10
          path: |
            /your/app/spec/reports/*.json

when rspec is run like:

bundle exec rspec --format json --out spec/reports/rspec_<RSPEC_NODE_NUMBER>.json

# frozen_string_literal: true

require_relative "rspec_report_merger"

require "terminal-table"
require "json"

class GithubActionsStepSummaryCreator
  def initialize(overall_summary = RSpecReportMerger.new.overall_summary,
per_dir_summary = RSpecReportMerger.new.per_dir_summary)
    @overall_summary = overall_summary
    @per_dir_summary = per_dir_summary
    @spec_reports = Dir["spec/reports/rspec*.json"]
    raise "Error: No spec reports discovered!" if @spec_reports.empty?
  end

  def headings
    ["directory"] + @overall_summary.keys
  end

  def main_app_suite_data
    [["**All tests**"] + @overall_summary.values.map{ |v| v.round(2) }]
  end

  def per_dir_rows
    @per_dir_summary.map do |dir_spec|
      [
        dir_spec.first,
        dir_spec.last["duration"],
        dir_spec.last["example_count"],
        dir_spec.last["failure_count"],
        dir_spec.last["pending_count"],
      ]
    end
  end

  def add_step_summary
    rows = main_app_suite_data + per_dir_rows

    rows.sort_by!{ |data| -data[1] } # sort by duration

    table = Terminal::Table.new(
      headings: headings,
      rows: rows,
      style: { border: :markdown }
    )

    puts table

    message = "### Spec report: :rocket: \n \n #{table}"
    File.write(ENV["GITHUB_STEP_SUMMARY"] || "/tmp/summary", message, mode: "a+")
  end

  def run
    add_step_summary
  end
end

GithubActionsStepSummaryCreator.new.run

# frozen_string_literal: true

class RSpecReportMerger
  def initialize(options = {})
    @spec_reports = Dir["spec/reports/rspec*.json"]
    @options = options
    raise "Error: No spec reports discovered!" if @spec_reports.empty?
  end

  def read_reports(key)
    @spec_reports.flat_map do |file|
      JSON.parse(File.read(file)).fetch(key)
    end
  end

  def per_dir_summary
    # examples grouped by path
    grouped = read_reports("examples").group_by do |example|
      example["file_path"].split("/").first(3).last(2)
    end

    result = {}
    grouped.each_key do |key|
      duration = grouped[key].map{ |ex| ex["run_time"] }.sum
      status_counts = grouped[key].map{ |ex| ex["status"] }.tally

      result[key.join("/")] = {
        "example_count" => grouped[key].count,
        "failure_count" => status_counts["failed"] || 0,
        "pending_count" => status_counts["pending"] || 0,
        "duration" => (duration/60).round(2)
      }
    end

    result
  end

  def overall_summary
    result = {
      "duration" => 0,
      "example_count" => 0,
      "failure_count" => 0,
      "pending_count" => 0,
    }

    summaries = read_reports("summary")

    summaries.each do |summary|
      result["duration"] += summary["duration"]
      result["example_count"] += summary["example_count"]
      result["failure_count"] += summary["failure_count"]
      result["pending_count"] += summary["pending_count"]
    end

    result
  end
end

The code is quick and dirty and could definitely be improved, but it works. Here's what it looks like:

Note if you're missing rspec data and are running in queue mode, you might need to follow these instructions:
https://docs.knapsackpro.com/ruby/rspec/#queue-mode

[ArturT]

@singhprd Thank you for sharing this example! 🚀

[ArturT]

Ideas

Here are ideas what to do before we close this issue. Info for Knapsack team:

• Maybe we could add an example of using the above script and GitHub Actions to our Rails example project. Or we could create an article on our blog.

• In the case of Queue Mode, we could generate JSON reports this way.

• We could then update cookbook docs and add the example config that uses GitHub Actions job summaries.

• In the installation guide, we could let users know that they can check the cookbook if they want.

story

https://trello.com/c/B5TOBMTM

[ArturT]

@singhprd If you would like to publish on our blog we are open for Pull Request. Some of our users already published their solutions.