Cucumber Ltd / cucumber-pro

Version test summaryfeatures/test_results/version_test_summary.feature

Release Dasher II (patch 1) (2018-12-13)

Romain Gérard

Currently viewing

Feature: Version Test Summary

Concrete example:

Build 6285 on circle CI of revision 9e34b6d of the corinto.patch.3 tag version contains 3 runs of cucumber: - Profile cucumber-fakes-ci produced a resultset with 65 passed scenarios - Profile cucumber-integrated-no-ui-ci produced a resultset with 65 passed scenarios - Profile cucumber-ui-fakes-ci produced a resultset with 64 passed scenarios

=> Overall bar with 194 tests, all green

Edge case: two builds, same revision, different results (flicker!)

Scenario: See summary for latest run of current version

  • Given Danny has created test-project
  • And Danny has uploaded his public key
  • And Danny has added some documents to test-project
  • And Bob is watching test-project's documentation outline
  • When Cucumber-Java JSON test results failing-passing-pending-scenarios.feature are received for test-project:
    CIRCLE_BRANCHmaster
    CIRCLE_BUILD_NUM2
    CIRCLE_BUILD_URLhttps://circleci.com/...
  • Then Bob should see a test results summary with:
    failed1
    passed1
    pending1

Scenario: Don't see test results for another version

    • @todo-ui

    Scenario: Don't see test results for older revision of same version

    • Given Danny has created test-project
    • And Danny has uploaded his public key
    • And Bob is watching test-project's documentation outline
    • And Danny has added some documents to test-project
    • When Cucumber-Java JSON test results failing-passing-pending-scenarios.feature are received for test-project:
      CIRCLE_BRANCHmaster
      CIRCLE_BUILD_NUM2
      CIRCLE_BUILD_URLhttps://circleci.com/...
    • And Danny adds some more documents to test-project
    • Then Bob should see that the test results summary is out of date

    Scenario: Docs received after results

    • Given Danny has created test-project
    • And Danny has uploaded his public key
    • And Bob is watching test-project's documentation outline
    • And Cucumber-Java JSON test results failing-passing-pending-scenarios.feature exist for test-project:
      CIRCLE_BRANCHmaster
      CIRCLE_BUILD_NUM2
      CIRCLE_BUILD_URLhttps://circleci.com/...
    • When Danny adds some documents to test-project
    • Then Bob should see a test results summary with:
      failed1
      passed1
      pending1
      skipped0
      undefined0

    Scenario: Results received but no docs

      Scenario: Un-processable results received