Test report

When running pytest --sharpit, sharpit writes a consolidated machine-readable JSON report and optional JUnit XML. The JSON report follows the published schema sharpit-report/1.

The formal JSON Schema definition is available in the repository at docs/reference/report.schema.json.

The sharpit-report/1 schema

The report emits all contract keys (including nulls) in camelCase, with epoch millisecond timestamps and no outer envelope.

Schema fields

• schema: Schema identifier string, e.g. "sharpit-report/1".
• tool: Object with tool metadata (name and version).
• generatedAt: Generation timestamp in epoch milliseconds.
• status: Overall session outcome: "pass", "fail", or "unknown".
• statusReasons: Array of human-readable strings explaining why the session failed or was unknown.
• warnings: Array of masked warning messages (such as settings fallbacks or source limitations).
• gates: The configured thresholds for count gates and advisor severity.
• sources: List of capture sources (sqlalchemy, sqlite3, exceptions) with attachment status and limitations.
• workers: Worker status list for parallel execution (e.g. gw0, gw1).
• severityCounts: Finding counts grouped by severity in standard order (CRITICAL, HIGH, MEDIUM, LOW, INFO).
• totals: Aggregated counts of tests, queries, filtered queries, unattributed queries, unattributed share percentage, and exceptions.
• tests: Array of per-test records detailing nodeId, worker, attempt, outcome, query counts across setup, call, and teardown phases, detected N+1 queries, and finding fingerprints.
• unattributed: Summary of queries not linked to a specific test and their top normalized SQL statements.
• filtered: Count of queries filtered from gating and the active filter rules.
• advisors: Results from offline advisor scans per application snapshot.
• findings: Flat list of all findings generated by count gates and advisors.

Example JSON report

Here is an example sharpit-report.json produced by a session:

{
  "schema": "sharpit-report/1",
  "tool": {"name": "sharpit", "version": "0.3.0"},
  "generatedAt": 1761955200000,
  "status": "fail",
  "statusReasons": ["2 tests failed sharpit gates"],
  "warnings": [],
  "gates": {
    "maxQueriesPerTest": null,
    "nPlusOneThreshold": 5,
    "failOnSeverity": "HIGH",
    "allowUnknown": false,
    "maxUnattributedPercent": 10
  },
  "sources": [
    {"name": "sqlalchemy", "attached": true, "reason": null, "limitations": []},
    {"name": "sqlite3", "attached": true, "reason": null, "limitations": ["connections opened before capture started are not traced"]},
    {"name": "exceptions", "attached": true, "reason": null, "limitations": []}
  ],
  "workers": [{"id": "gw0", "status": "ok"}],
  "severityCounts": [
    {"severity": "CRITICAL", "count": 0},
    {"severity": "HIGH", "count": 2},
    {"severity": "MEDIUM", "count": 0},
    {"severity": "LOW", "count": 0},
    {"severity": "INFO", "count": 0}
  ],
  "totals": {
    "tests": 120,
    "queries": 840,
    "filtered": 96,
    "unattributed": 12,
    "unattributedShare": 1.6,
    "exceptions": 1
  },
  "tests": [
    {
      "nodeId": "tests/test_owners.py::test_list",
      "worker": "gw0",
      "attempt": 1,
      "outcome": "failed",
      "budget": {"maxQueries": 5, "allowNPlusOne": false, "maxExceptions": 0},
      "phases": {
        "setup": {"queries": 3, "filtered": 2, "exceptions": 0},
        "call": {"queries": 12, "filtered": 0, "exceptions": 0},
        "teardown": {"queries": 1, "filtered": 1, "exceptions": 0}
      },
      "nPlusOne": [
        {"sql": "SELECT ... FROM pets WHERE owner_id = ?", "count": 10}
      ],
      "findings": ["test-runtime:sharpit.n-plus-one:3f9a..."]
    }
  ],
  "unattributed": {
    "queries": 12,
    "topStatements": [
      {"sql": "SELECT ... FROM jobs WHERE ...", "count": 12}
    ]
  },
  "filtered": {
    "count": 96,
    "rules": ["PRAGMA", "sqlite_schema", "migration tables", "transaction control"]
  },
  "advisors": [
    {
      "advisor": "rest-api",
      "status": "PASSED",
      "reason": null,
      "apps": [
        {
          "app": "Owners API#1c2d3e4f",
          "status": "PASSED",
          "reason": null,
          "evidence": {"usable": true, "coverageComplete": true, "limitations": []}
        }
      ],
      "findings": []
    }
  ],
  "findings": [
    {
      "ruleId": "sharpit.n-plus-one",
      "advisor": "test-runtime",
      "severity": "HIGH",
      "target": "SELECT ... FROM pets WHERE owner_id = ?",
      "message": "10 executions of one SELECT in the call phase",
      "fingerprint": "test-runtime:sharpit.n-plus-one:3f9a...",
      "testNodeId": "tests/test_owners.py::test_list",
      "dismissed": false
    }
  ]
}

Schema versioning

The schema string is versioned as sharpit-report/<major>.

• Minor changes within /1: Adding new fields or keys is backwards-compatible. Consumers must ignore unknown properties (additionalProperties: true).
• Breaking changes (/2): Renaming or removing existing keys, or changing their types and semantic meanings, will bump the major version to sharpit-report/2.

BootUI-style CI gating with jq

The severityCounts list mirrors BootUI's order (CRITICAL, HIGH, MEDIUM, LOW, INFO), with all severities present. Counts reflect non-dismissed findings.

You can gate CI pipelines using jq directly against severityCounts. For example, to fail if any CRITICAL or HIGH findings exist:

jq -e '([.severityCounts[] | select(.severity == "CRITICAL" or .severity == "HIGH") | .count] | add) == 0' sharpit-report.json

Or to verify that the overall report status passed:

jq -e '.status == "pass"' sharpit-report.json

JUnit XML report

Specify --sharpit-junit=PATH to generate a standard JUnit XML file representing findings.

• Contains a single <testsuite name="sharpit">.
• Each finding in the report appears as a <testcase> with a severity property (classname is the advisor, name is the finding fingerprint). Only non-dismissed findings at or above --sharpit-fail-on carry a <failure type="<ruleId>" message="...">; dismissed and lower-severity findings are passing cases.
• A <testcase name="sharpit.status"> records overall session state, failing if the run status is fail or unknown (without --sharpit-allow-unknown).
• The JUnit file contains findings only and does not duplicate the application tests (standard pytest --junitxml covers individual test results).