Skip to content

feat: gh aw audit diff — compare firewall behavior across runs#22996

Open
Copilot wants to merge 3 commits intomainfrom
copilot/add-gh-aw-audit-diff-command
Open

feat: gh aw audit diff — compare firewall behavior across runs#22996
Copilot wants to merge 3 commits intomainfrom
copilot/add-gh-aw-audit-diff-command

Conversation

Copy link
Contributor

Copilot AI commented Mar 25, 2026

Adds gh aw audit diff <run-id-1> <run-id-2> to compare firewall behavior across two workflow runs. Detects policy regressions, new unauthorized domains, and behavioral drift.

gh aw audit diff 12345 12346                    # pretty console output
gh aw audit diff 12345 12346 --format markdown  # for PR comments
gh aw audit diff 12345 12346 --json             # for CI integration

Diff categories

  • New domains — domains in run-2 not in run-1
  • Removed domains — domains in run-1 not in run-2
  • Status changes — domains that flipped allowed↔denied
  • Volume changes — >100% request count delta per domain
  • Anomaly flags — new denied domains, status flips

Implementation

  • audit_diff.gocomputeFirewallDiff() operates on DomainRequestStats maps from both runs. loadFirewallAnalysisForRun() tries cached RunSummary first, falls back to artifact download + firewall log parsing.
  • audit_diff_command.go — Cobra subcommand registered under existing audit command. Backward compatible — audit <run-id> still works unchanged.
  • audit_diff_render.go — Three formatters: pretty (stderr), markdown (stdout), JSON (stdout).
  • audit_diff_test.go — 14 unit tests covering all diff categories, nil/empty edge cases, anomaly detection, JSON roundtrip, and helpers.
  • audit.go — Single AddCommand call to register the subcommand.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • https://api.github.com/graphql
    • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GOMOD GOMODCACHE go env on&#39; --ignore-patGOINSECURE GO111MODULE x_amd64/compile GOINSECURE GOMOD GOMODCACHE x_amd64/compile (http block)
    • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GOMOD GOMODCACHE go env on&#39; --ignore-patGOINSECURE GO111MODULE x_amd64/link GOINSECURE GOMOD GOMODCACHE x_amd64/link (http block)
  • https://api.github.com/orgs/test-owner/actions/secrets
    • Triggering command: /usr/bin/gh gh api /orgs/test-owner/actions/secrets --jq .secrets[].name -json GO111MODULE ache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env h ../../../.prettierignore GO111MODULE ache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE node (http block)
  • https://api.github.com/repos/actions/ai-inference/git/ref/tags/v1
    • Triggering command: /usr/bin/gh gh api /repos/actions/ai-inference/git/ref/tags/v1 --jq .object.sha --show-toplevel go /usr/bin/git json&#39; --ignore-pgit GO111MODULE /opt/hostedtoolc--show-toplevel git rev-�� --show-toplevel go /usr/bin/gh th .prettierignogit GO111MODULE er: String!, $na--show-toplevel gh (http block)
  • https://api.github.com/repos/actions/checkout/git/ref/tags/v3
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v3 --jq .object.sha &#34;prettier&#34; --write &#39;../../../**/*.json&#39; &#39;!../../../pkg/workflow/-s go /opt/hostedtoolcache/go/1.25.0/x64/pkg/tool/linux_amd64/compile heck &#39;**/*.cjs&#39; git GO111MODULE 64/bin/go /opt/hostedtoolcache/go/1.25.0/x64/pkg/tool/linux_amd64/compile -o /tmp/go-build3189605158/b417/_pkg_.a -trimpath /usr/bin/git -p github.com/githurev-parse -lang=go1.25 git (http block)
  • https://api.github.com/repos/actions/checkout/git/ref/tags/v5
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq .object.sha k/gh-aw/gh-aw/.github/workflows GOPROXY 86_64/sh GOSUMDB GOWORK DiscussionsEnabl--get-regexp git -C *.json&#39; &#39;!../../../pkg/workflow/js/**/*.json&#39; ---p config /usr/bin/git remote.origin.urgit GO111MODULE $name) { has--show-toplevel git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq .object.sha list --json /usr/bin/git --workflow nonexistent-workrev-parse --limit git rev-�� --show-toplevel go /usr/bin/git -json GO111MODULE 0/x64/bin/npx git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq .object.sha --show-toplevel git /usr/bin/git te &#39;scripts/**/*git rev-parse k/gh-aw/gh-aw/no--show-toplevel git rev-�� --show-toplevel infocmp /usr/bin/git xterm-color on rkflow/js/**/*.j--show-toplevel git (http block)
  • https://api.github.com/repos/actions/checkout/git/ref/tags/v6
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq .object.sha k/gh-aw/gh-aw/.github/workflows/auto-triage-issues.md node /usr/bin/git js/**/*.json&#39; --git **/*.cjs ache/go/1.25.0/x--show-toplevel git chec�� .github/workflows/test.md go /usr/bin/git ithub/workflows GO111MODULE _modules/.bin/no--show-toplevel git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq .object.sha --show-toplevel node /usr/bin/git js/**/*.json&#39; --git **/*.cjs ache/go/1.25.0/x--show-toplevel git rev-�� --git-dir go /usr/bin/git -json GO111MODULE /opt/hostedtoolc--show-toplevel git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq .object.sha --show-toplevel 64/pkg/tool/linux_amd64/compile /usr/bin/git g_.a x_amd64/cgo /usr/bin/git git rev-�� --show-toplevel git /usr/bin/git 5342-41882/test-git rev-parse k/node_modules/.--show-toplevel git (http block)
  • https://api.github.com/repos/actions/github-script/git/ref/tags/v8
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v8 --jq .object.sha -json GO111MODULE r: $owner, name: $name) { hasDiscussionsEnabled } } GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE ache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE node (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v8 --jq .object.sha ath ../../../.pr**/*.json GO111MODULE ache/go/1.25.0/x../../../.prettierignore GOINSECURE GOMOD GOMODCACHE sh er ithub/workflows GOPROXY ache/go/1.25.0/x64/bin/go GOSUMDB GOWORK 64/bin/go sh (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v8 --jq .object.sha -json GO111MODULE docker-compose GOINSECURE GOMOD GOMODCACHE docker-compose -c npx prettier --check &#39;**/*.cjs&#39; -f GOPROXY ache/go/1.25.0/x64/bin/go GOSUMDB GOWORK 64/bin/go node (http block)
  • https://api.github.com/repos/actions/setup-go/git/ref/tags/v4
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-go/git/ref/tags/v4 --jq .object.sha -test.paniconexit0 -test.v=true /usr/bin/git -test.timeout=10git -test.run=^Test -test.short=true--show-toplevel git -C /tmp/gh-aw-test-runs/20260325-225342-41882/test-1988847378 status /usr/bin/git .github/workflowgit GO111MODULE /opt/hostedtoolc--show-toplevel git (http block)
  • https://api.github.com/repos/actions/setup-node/git/ref/tags/v4
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v4 --jq .object.sha k/gh-aw/gh-aw/.github/workflows/blog-auditor.md node /usr/bin/git js/**/*.json&#39; --git **/*.cjs ache/go/1.25.0/x--show-toplevel git rev-�� --git-dir go /usr/bin/git -json GO111MODULE node git (http block)
  • https://api.github.com/repos/actions/upload-artifact/git/ref/tags/v4
    • Triggering command: /usr/bin/gh gh api /repos/actions/upload-artifact/git/ref/tags/v4 --jq .object.sha prettier --write /bin/sh **/*.ts **/*.json --ignore-path /bin/sh -c runs/20260325-225342-41882/test-202902941/custom/workflows go /home/REDACTED/work/_temp/ghcca-node/node/bin/bash tierignore GO111MODULE 64/bin/go bash (http block)
  • https://api.github.com/repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b
    • Triggering command: /usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq .object.sha on&#39; --ignore-patGOINSECURE GO111MODULE x_amd64/link GOINSECURE GOMOD GOMODCACHE x_amd64/link env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/github/gh-aw
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw --jq .visibility &#39;**/*.ts&#39; &#39;**/*.json&#39; --ignore-pGOINSECURE GO111MODULE ules/.bin/node GOINSECURE GOMOD GOMODCACHE go 8d51�� -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/github/gh-aw-actions/git/ref/tags/v0
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v0 --jq .object.sha on&#39; --ignore-patGOINSECURE GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/github/gh-aw-actions/git/ref/tags/v0.1.2
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v0.1.2 --jq .object.sha -test.paniconexit0 -test.v=true (http block)
  • https://api.github.com/repos/github/gh-aw-actions/git/ref/tags/v1.0.0
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v1.0.0 --jq .object.sha ring530894474/001/test2.md go /opt/hostedtoolcache/node/24.14.0/x64/lib/node_modules/npm/node_modules/@npmcli/run-script/lib/n--show-toplevel -json GO111MODULE run-script/lib/n--show-toplevel sh -c npx prettier --write &#39;../../../**/*.json&#39; &#39;!../../../pkg/workflow/js/**/*.json&#39; --ignore-path go 0/x64/bin/node -json GO111MODULE 64/bin/go ache/go/1.25.0/x64/pkg/tool/linux_amd64/link (http block)
  • https://api.github.com/repos/github/gh-aw-actions/git/ref/tags/v1.2.3
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v1.2.3 --jq .object.sha jpq_JbW5I go /home/node_modules/.bin/sh l GO111MODULE 64/bin/go ortcfg -c g/timeutil/format.go g/timeutil/format_test.go ache/go/1.25.0/x64/pkg/tool/linux_amd64/compile -json GO111MODULE 64/bin/go ache/go/1.25.0/x64/pkg/tool/linux_amd64/compile (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/1/artifacts
    • Triggering command: /usr/bin/gh gh run download 1 --dir test-logs/run-1 GO111MODULE x_amd64/compile GOINSECURE GOMOD ed } } x_amd64/compile env &#39;**/*.ts&#39; &#39;**/*.json&#39; --ignore-path ../../../.pr**/*.json GO111MODULE /usr/bin/git GOINSECURE GOMOD ed } } git (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/12345/artifacts
    • Triggering command: /usr/bin/gh gh run download 12345 --dir test-logs/run-12345 GO111MODULE bin/sh l GOMOD DiscussionsEnabl--show-toplevel go env &#39;**/*.ts&#39; &#39;**/*.json&#39; --ignore-p0 GO111MODULE 0/x64/bin/sh GOINSECURE GOMOD DiscussionsEnabl--show-toplevel /usr/bin/gh (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/12346/artifacts
    • Triggering command: /usr/bin/gh gh run download 12346 --dir test-logs/run-12346 GO111MODULE h GOINSECURE GOMOD GOMODCACHE go env &#39;**/*.ts&#39; &#39;**/*.json&#39; --ignore-premote.origin.url GO111MODULE de/node/bin/sh GOINSECURE GOMOD GOMODCACHE infocmp (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/2/artifacts
    • Triggering command: /usr/bin/gh gh run download 2 --dir test-logs/run-2 GO111MODULE x_amd64/compile GOINSECURE GOMOD GOMODCACHE x_amd64/compile --no�� &#39;**/*.ts&#39; &#39;**/*.json&#39; --ignore-path ../../../.pr**/*.json GOPROXY 8d519d9/node_modules/.bin/prettier l GOWORK 64/bin/go infocmp (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/3/artifacts
    • Triggering command: /usr/bin/gh gh run download 3 --dir test-logs/run-3 GO111MODULE 0/x64/bin/sh l GOMOD GOMODCACHE go estl�� &#39;**/*.ts&#39; &#39;**/*.json&#39; --ignore-premote.origin.url GO111MODULE (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/4/artifacts
    • Triggering command: /usr/bin/gh gh run download 4 --dir test-logs/run-4 GO111MODULE x_amd64/compile GOINSECURE GOMOD DiscussionsEnabl--show-toplevel x_amd64/compile --no�� &#39;**/*.ts&#39; &#39;**/*.json&#39; --ignore-path ../../../.pr**/*.json GOPROXY /usr/bin/gh GOSUMDB GOWORK 64/bin/go /usr/bin/gh (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/5/artifacts
    • Triggering command: /usr/bin/gh gh run download 5 --dir test-logs/run-5 GO111MODULE me: String!) { -nilfunc GOINSECURE GOMOD GOMODCACHE bash --no�� &#39;**/*.ts&#39; &#39;**/*.json&#39; --ignore-path ../../../.pr**/*.json GOPROXY x_amd64/compile GOSUMDB GOWORK DiscussionsEnabled } } x_amd64/compile (http block)
  • https://api.github.com/repos/github/gh-aw/actions/workflows
    • Triggering command: /usr/bin/gh gh workflow list --json name,state,path 570155/b373/_pkg_.a GO111MODULE repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } GOINSECURE b/gh-aw/pkg/cli GOMODCACHE go env h ../../../.pret.prettierignore GO111MODULE ache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE 570155/b373/impo../../../**/*.json (http block)
    • Triggering command: /usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --workflow nonexistent-workflow-12345 --limit 100 GOMOD GOMODCACHE node /hom�� js/**/*.json&#39; --- **/*.cjs ache/go/1.25.0/x64/bin/go **/*.json --ignore-path ../../../.pretti/tmp/gh-aw-test-runs/20260325-225342-41882/test-4122226171/.github/workflows go (http block)
    • Triggering command: /usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --workflow nonexistent-workflow-12345 --limit 6 GOMOD DiscussionsEnabl--show-toplevel go env k/gh-aw/gh-aw/.github/workflows GO111MODULE 86_64/node GOINSECURE GOMOD DiscussionsEnabl--show-toplevel git (http block)
  • https://api.github.com/repos/github/gh-aw/git/ref/tags/v1.0.0
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v1.0.0 --jq .object.sha ithub/workflows GO111MODULE tions/setup/js/node_modules/.bin-nilfunc GOINSECURE GOMOD GOMODCACHE git tion�� k/gh-aw/gh-aw/.github/workflows --always 64/bin/node GOSUMDB GOWORK DiscussionsEnabl--show-toplevel git (http block)
  • https://api.github.com/repos/github/gh-aw/git/ref/tags/v1.2.3
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v1.2.3 --jq .object.sha ithub/workflows GO111MODULE ache/go/1.25.0/x../../../.prettierignore GOINSECURE GOMOD GOMODCACHE node /hom�� js/**/*.json&#39; ---errorsas **/*.cjs ache/go/1.25.0/x-nilfunc **/*.json --ignore-path ../../../.pretti/tmp/TestGuardPolicyMinIntegrityOnlymin-integrity_only_defaults_repo2614931207/001 go (http block)
  • https://api.github.com/repos/github/gh-aw/git/ref/tags/v2.0.0
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v2.0.0 --jq .object.sha re --log-level=e!../../../pkg/workflow/js/**/*.json GO111MODULE ache/go/1.25.0/x../../../.prettierignore GOINSECURE GOMOD GOMODCACHE node /opt�� js/**/*.json&#39; ---errorsas --check ache/go/1.25.0/x-nilfunc --ignore-path .prettierignore 64/bin/go go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v2.0.0 --jq .object.sha re --log-level=e!../../../pkg/workflow/js/**/*.json GO111MODULE ache/go/1.25.0/x../../../.prettierignore GOINSECURE GOMOD GOMODCACHE node /opt�� js/**/*.json&#39; ---p --check repository(owne-lang=go1.25 --ignore-path .prettierignore 64/bin/go go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v2.0.0 --jq .object.sha re --log-level=e!../../../pkg/workflow/js/**/*.json GOPROXY /node GOSUMDB GOWORK 64/bin/go sh /pre�� ithub/workflows go _modules/.bin/sh-nilfunc -json GO111MODULE 64/bin/go go (http block)
  • https://api.github.com/repos/github/gh-aw/git/ref/tags/v3.0.0
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v3.0.0 --jq .object.sha re --log-level=e!../../../pkg/workflow/js/**/*.json GO111MODULE me: String!) { ../../../.prettierignore GOINSECURE GOMOD GOMODCACHE node /opt�� js/**/*.json&#39; ---s --check repository(owne-buildmode=exe --ignore-path .prettierignore 64/bin/go go (http block)
  • https://api.github.com/repos/githubnext/agentics/git/ref/tags/
    • Triggering command: /usr/bin/gh gh api /repos/githubnext/agentics/git/ref/tags/# --jq .object.sha -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE ache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/nonexistent/action/git/ref/tags/v999.999.999
    • Triggering command: /usr/bin/gh gh api /repos/nonexistent/action/git/ref/tags/v999.999.999 --jq .object.sha ithub/workflows GO111MODULE tions/node_modules/.bin/node GOINSECURE GOMOD GOMODCACHE go tion�� ithout_min-integrity81874459/001 -s -w -X main.version=5705a3e-dirty tions/setup/js/node_modules/.bin/sh gh-aw ./cmd/gh-aw 64/bin/go infocmp (http block)
  • https://api.github.com/repos/nonexistent/repo/actions/runs/12345
    • Triggering command: /usr/bin/gh gh run view 12345 --repo nonexistent/repo --json status,conclusion l GOMOD GOMODCACHE bash --no�� 512253920/.github/workflows GOPROXY ules/.bin/sh GOSUMDB GOWORK DiscussionsEnabl--show-toplevel /usr/bin/gh (http block)
  • https://api.github.com/repos/owner/repo/actions/workflows
    • Triggering command: /usr/bin/gh gh workflow list --json name,state,path --repo owner/repo repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } GOINSECURE GOMOD GOMODCACHE go env h ../../../.pret.prettierignore GO111MODULE ache/go/1.25.0/x64/bin/go GOINSECURE 115b47a6 GOMODCACHE sh (http block)
    • Triggering command: /usr/bin/gh gh workflow list --json name,state,path --repo owner/repo repository(owne--ignore-path GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linuscripts/**/*.js env h ../../../.pret.prettierignore GO111MODULE ache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE sh (http block)
  • https://api.github.com/repos/owner/repo/contents/file.md
    • Triggering command: /tmp/go-build3189605158/b403/cli.test /tmp/go-build3189605158/b403/cli.test -test.testlogfile=/tmp/go-build3189605158/b403/testlog.txt -test.paniconexit0 -test.v=true -test.parallel=4 -test.timeout=10m0s -test.run=^Test -test.short=true GOINSECURE GOMOD GOMODCACHE go env ath ../../../.pr**/*.json GO111MODULE ache/go/1.25.0/x../../../.prettierignore GOINSECURE GOMOD GOMODCACHE sh (http block)
  • https://api.github.com/repos/test-owner/test-repo/actions/secrets
    • Triggering command: /usr/bin/gh gh api /repos/test-owner/test-repo/actions/secrets --jq .secrets[].name -json GO111MODULE repository(owner: $owner, name:-f GOINSECURE GOMOD GOMODCACHE go env h ../../../.prettierignore GO111MODULE ache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE node (http block)

If you need me to access, download, or install something from one of these locations, you can either:


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.


Changeset

  • Type: patch
  • Description: Added gh aw audit diff <run-id-1> <run-id-2> to compare firewall behavior across runs, including new/removed domains, status flips, request-volume drift, and anomaly flags across pretty, markdown, and JSON outputs.

Generated by Changeset Generator for issue #22996 ·



✨ PR Review Safe Output Test - Run 23568997760

Note

🔒 Integrity filter blocked 1 item

The following item were blocked because they don't meet the GitHub integrity level.

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

💥 [THE END] — Illustrated by Smoke Claude ·

Copilot AI changed the title [WIP] Add gh aw audit diff command to compare firewall behavior feat: gh aw audit diff — compare firewall behavior across runs Mar 25, 2026
Copilot AI requested a review from Mossaka March 25, 2026 22:58
@Mossaka Mossaka marked this pull request as ready for review March 25, 2026 23:10
Copilot AI review requested due to automatic review settings March 25, 2026 23:10
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new gh aw audit diff <run-id-1> <run-id-2> subcommand to compare firewall behavior between two workflow runs, surfacing domain-level changes and anomalies across runs.

Changes:

  • Introduces diff computation logic over per-domain firewall request stats (new/removed domains, status flips, and high-volume deltas).
  • Adds a Cobra subcommand (audit diff) with pretty/markdown/JSON output modes.
  • Adds unit tests for diff computation helpers and JSON roundtrip.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
pkg/cli/audit_diff.go Implements computeFirewallDiff plus cached analysis loading via run summaries/artifacts.
pkg/cli/audit_diff_command.go Adds audit diff Cobra subcommand and orchestrates loading + rendering.
pkg/cli/audit_diff_render.go Adds pretty/markdown/JSON renderers and helper utilities.
pkg/cli/audit_diff_test.go Adds unit tests for diff categories, helper functions, and JSON serialization.
pkg/cli/audit.go Registers the new audit diff subcommand under audit.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +53 to +63
var owner, repo, hostname string
if repoFlag != "" {
parts := strings.SplitN(repoFlag, "/", 2)
if len(parts) != 2 || parts[0] == "" || parts[1] == "" {
return fmt.Errorf("invalid repository format '%s': expected 'owner/repo'", repoFlag)
}
owner = parts[0]
repo = parts[1]
}

return RunAuditDiff(cmd.Context(), runID1, runID2, owner, repo, hostname, outputDir, verbose, jsonOutput, format)
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--repo is documented as [HOST/]owner/repo, but this parsing assumes only two path segments and never derives hostname from the flag. If a user passes a host-qualified repo that differs from the git remote host, RunAuditDiff will auto-detect hostname from the remote and downloadRunArtifacts will build an incorrect -R <hostname>/<owner>/<repo> (e.g. ghes.example.com/github.com/owner/repo). Consider parsing repoFlag into hostname/owner/repo when 3 segments are provided (or otherwise letting repoFlag drive the hostname override) and adjust the validation/error message accordingly.

Copilot uses AI. Check for mistakes.
Comment on lines +49 to +70
jsonOutput, _ := cmd.Flags().GetBool("json")
format, _ := cmd.Flags().GetString("format")
repoFlag, _ := cmd.Flags().GetString("repo")

var owner, repo, hostname string
if repoFlag != "" {
parts := strings.SplitN(repoFlag, "/", 2)
if len(parts) != 2 || parts[0] == "" || parts[1] == "" {
return fmt.Errorf("invalid repository format '%s': expected 'owner/repo'", repoFlag)
}
owner = parts[0]
repo = parts[1]
}

return RunAuditDiff(cmd.Context(), runID1, runID2, owner, repo, hostname, outputDir, verbose, jsonOutput, format)
},
}

addOutputFlag(cmd, defaultLogsOutputDir)
addJSONFlag(cmd)
addRepoFlag(cmd)
cmd.Flags().String("format", "pretty", "Output format: pretty, markdown, json")
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

format is user-facing but isn’t validated. As written, any unknown value silently falls back to the pretty renderer, which makes typos hard to notice (e.g. --format makrdown). Consider validating format against the supported set (pretty, markdown, json) and returning a clear error on invalid values; also consider detecting/handling conflicts when both --json and --format are set.

Copilot uses AI. Check for mistakes.
}
fmt.Fprintln(os.Stderr, console.FormatInfoMessage("Changes: "+strings.Join(parts, ", ")))
if diff.Summary.HasAnomalies {
fmt.Fprintln(os.Stderr, console.FormatWarningMessage(fmt.Sprintf("⚠️ %d anomalies detected", diff.Summary.AnomalyCount)))
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

console.FormatWarningMessage already prefixes messages with a warning icon. Including "⚠️" inside the message here will result in a duplicated icon in output (e.g. ⚠ ⚠️ 3 anomalies detected). Consider removing the embedded emoji and letting FormatWarningMessage provide the icon/styling.

Suggested change
fmt.Fprintln(os.Stderr, console.FormatWarningMessage(fmt.Sprintf("⚠️ %d anomalies detected", diff.Summary.AnomalyCount)))
fmt.Fprintln(os.Stderr, console.FormatWarningMessage(fmt.Sprintf("%d anomalies detected", diff.Summary.AnomalyCount)))

Copilot uses AI. Check for mistakes.
Comment on lines +109 to +114
// Anomaly: new denied domain
if stats2.Blocked > 0 {
entry.IsAnomaly = true
entry.AnomalyNote = "new denied domain"
anomalyCount++
}
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New-domain anomalies are flagged whenever stats2.Blocked > 0, but domainStatus(stats2) can be mixed (both allowed+blocked). In that case the anomaly note "new denied domain" is inaccurate and over-flags domains that are not fully denied. Consider flagging only when the new domain’s status is denied (e.g., stats2.Allowed == 0 && stats2.Blocked > 0) or updating the note/logic to distinguish mixed from denied.

Copilot uses AI. Check for mistakes.
@Mossaka Mossaka added the smoke label Mar 25, 2026
@github-actions
Copy link
Contributor

github-actions bot commented Mar 25, 2026

🎬 THE ENDSmoke Claude MISSION: ACCOMPLISHED! The hero saves the day! ✨

@github-actions
Copy link
Contributor

github-actions bot commented Mar 25, 2026

✅ All tools validated successfully! Agent Container Smoke Test confirms agent container is ready.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 25, 2026

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

@github-actions
Copy link
Contributor

📰 BREAKING: Smoke Copilot is now investigating this pull request. Sources say the story is developing...

@github-actions github-actions bot removed the smoke label Mar 25, 2026
@github-actions
Copy link
Contributor

Agent Container Tool Check

Tool Status Version
bash 5.2.21
sh available
git 2.53.0
jq 1.7
yq 4.52.4
curl 8.5.0
gh 2.88.1
node 20.20.1
python3 3.12.3
go 1.24.13
java 21.0.10
dotnet 10.0.201

Result: 12/12 tools available ✅

Overall Status: PASS

🔧 Tool validation by Agent Container Smoke Test ·

@github-actions
Copy link
Contributor

Commit pushed: c80a754

Generated by Changeset Generator

@github-actions
Copy link
Contributor

Smoke Test Results — Run §23568997793

Test Status
GitHub MCP
MCP Scripts GH CLI
Serena MCP
Playwright
Web Fetch
File Writing + Bash
Discussion Interaction
Build gh-aw
Discussion Creation
Workflow Dispatch
PR Review

Overall: ⚠️ PARTIAL PASS — PR author: @Copilot, assignees: @pelikhan, @Copilot

Note

🔒 Integrity filter blocked 1 item

The following item were blocked because they don't meet the GitHub integrity level.

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

📰 BREAKING: Report filed by Smoke Copilot ·

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quick review of gh aw audit diff — clean implementation overall. Two minor observations on the logger alias and volume-change threshold left as inline comments.

Note

🔒 Integrity filter blocked 1 item

The following item were blocked because they don't meet the GitHub integrity level.

  • #22996 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

📰 BREAKING: Report filed by Smoke Copilot

)

var auditDiffLog = auditLog

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reusing auditLog via var auditDiffLog = auditLog means both commands share the same logger namespace. Consider using logger.New("cli:audit_diff") for a distinct logger so debug filtering can target this subcommand independently.


// volumeChangeThresholdPercent is the minimum percentage increase to flag as a volume change.
// >100% increase means the request count more than doubled.
const volumeChangeThresholdPercent = 100.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The threshold 100.0 (>100% increase) is quite high — a domain that goes from 1 request to 5 requests (+400%) might slip through without a volume-change flag if the absolute counts are low. Worth noting whether a minimum absolute delta should also be considered here.

@github-actions
Copy link
Contributor

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

@github-actions
Copy link
Contributor

Smoke Test 23568997760 — PARTIAL (skips only)

Core: 1✅ 2✅ 3✅ 4✅ 5✅ 6✅ 7✅ 8✅ 9✅ 10✅ 11✅
PR Review: 12✅ 13✅ 14✅ 15⚠️ 16✅ 17⚠️ 18⚠️

Skips: #15 DIFC policy blocked thread reads · #17 avoided test commit on feature PR · #18 no test PR

Note

🔒 Integrity filter blocked 1 item

The following item were blocked because they don't meet the GitHub integrity level.

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

💥 [THE END] — Illustrated by Smoke Claude ·

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💥 Automated smoke test review - all systems nominal!

Note

🔒 Integrity filter blocked 1 item

The following item were blocked because they don't meet the GitHub integrity level.

  • #22996 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

💥 [THE END] — Illustrated by Smoke Claude

return fmt.Errorf("invalid run ID %q: must be a numeric run ID", args[1])
}

outputDir, _ := cmd.Flags().GetString("output")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error from cmd.Flags().GetString("output") is silently discarded with _. While flag retrieval rarely fails, it's worth using cmd.Flags().GetString with proper error handling for consistency with the rest of the CLI codebase, or at minimum using cmd.Flags().Lookup("output").Value.String() which is guaranteed not to fail.


// volumeChangeThresholdPercent is the minimum percentage increase to flag as a volume change.
// >100% increase means the request count more than doubled.
const volumeChangeThresholdPercent = 100.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider making volumeChangeThresholdPercent configurable via a --threshold flag rather than a hardcoded constant. Different teams may have different sensitivity requirements for volume change detection.

@github-actions
Copy link
Contributor

Smoke Test: Codex

🔮 The oracle has spoken through Smoke Codex ·

Copy link
Collaborator

@Mossaka Mossaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed changes and verified smoke tests.

Code review:

  • Clean command structure: audit_diff.go (core diff logic, 254 lines), audit_diff_command.go (cobra subcommand, 146 lines), audit_diff_render.go (3 formatters: pretty/markdown/JSON, 182 lines), audit_diff_test.go (14 tests, 324 lines)
  • computeFirewallDiff() handles nil/empty inputs, deterministic output (sorted domains), and anomaly detection (new denied domains, status flips)
  • loadFirewallAnalysisForRun() correctly uses cached RunSummary first, falls back to artifact download
  • Registered as subcommand under existing audit — backward compatible (audit <run-id> unchanged)
  • No compiler changes, no lock files affected

CI: All 51 checks pass.

Smoke tests: All passed:

  • Smoke Claude: success
  • Smoke Copilot: success
  • Smoke Codex: success
  • Agent Container Smoke Test: success

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: gh aw audit diff — compare firewall behavior across runs

3 participants