Skip to content

Add changeset creation skill#26825

Open
tylerbutler wants to merge 3 commits intomicrosoft:mainfrom
tylerbutler:feat/changeset-skill
Open

Add changeset creation skill#26825
tylerbutler wants to merge 3 commits intomicrosoft:mainfrom
tylerbutler:feat/changeset-skill

Conversation

@tylerbutler
Copy link
Member

@tylerbutler tylerbutler commented Mar 24, 2026

Summary

  • Adds a Claude Code skill (changeset/SKILL.md) that automates creating changeset files for the FluidFramework repo
  • The skill analyzes git diffs, determines affected packages (including fluid-framework re-export rules), selects the appropriate release notes section, and writes properly formatted changeset files

What the skill does

  1. Reads the branch diff against main to understand the change
  2. Identifies affected packages (only those meaningful to consumers)
  3. Applies the fluid-framework re-export rule (e.g., tree changes get both @fluidframework/tree and fluid-framework)
  4. Chooses the correct __section (feature, tree, fix, breaking, deprecation, legacy, other)
  5. Writes a changeset with proper format: present tense summary, no trailing period, no backticks in summary, sentence-boundary line wrapping, level 4+ headings, code examples for features/breaking/deprecations

Evaluation results

The skill was evaluated across 3 test scenarios (tree feature, bug fix, deprecation) comparing with-skill vs. without-skill (baseline) runs:

Metric With Skill Without Skill (baseline)
Assertion pass rate 100% 97%
Avg tokens 16.8K 18.3K
Avg time 33.6s 42.8s

Key findings

  • 100% format compliance with the skill vs. 97% without — the baseline consistently fails on present perfect tense for bug fix summaries (writes "now include" instead of "have been fixed")
  • 21% faster and 8% fewer tokens — the skill provides clear instructions so agents don't need to explore the repo to discover formatting conventions
  • Correct "future release" language for deprecations — the baseline writes "future major release" which is incorrect since some APIs are removed in minor releases
  • Includes links to relevant docs (e.g., MDN for standard API equivalents) which the baseline omits
  • The baseline performed well overall (97%) because the existing .changeset/README.md provides good guidance — the skill's value is in the nuances that the README alone doesn't convey reliably

Qualitative observations

  • With-skill outputs hit the right level of detail: specific methods/types called out, practical benefits explained, but not overly verbose
  • The skill's examples serve as strong few-shot prompts that shape output quality

Add a Claude Code skill that automates creating changeset files for the
FluidFramework repo. The skill analyzes git diffs, determines affected
packages (including fluid-framework re-export rules), selects the
appropriate release notes section, and writes properly formatted
changeset files following the project's writing guidelines.
Copilot AI review requested due to automatic review settings March 24, 2026 23:48
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new Claude Code skill for authoring FluidFramework changesets, aiming to standardize package selection (including fluid-framework re-export considerations) and enforce changeset formatting/sectioning conventions.

Changes:

  • Introduces a new changeset skill with step-by-step guidance for analyzing branch diffs and selecting affected packages.
  • Documents how to choose __section/__highlight and how to write well-formatted summary/body content.
  • Provides example changesets for fix/feature/breaking/tree scenarios and some edge cases.

@tylerbutler tylerbutler self-assigned this Mar 25, 2026
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Tyler Butler <tyler@tylerbutler.com>
@github-actions
Copy link
Contributor

🔗 No broken links found! ✅

Your attention to detail is admirable.

linkcheck output


> fluid-framework-docs-site@0.0.0 ci:check-links /home/runner/work/FluidFramework/FluidFramework/docs
> start-server-and-test "npm run serve -- --no-open" 3000 check-links

1: starting server using command "npm run serve -- --no-open"
and when url "[ 'http://127.0.0.1:3000' ]" is responding with HTTP status code 200
running tests using command "npm run check-links"


> fluid-framework-docs-site@0.0.0 serve
> docusaurus serve --no-open

[SUCCESS] Serving "build" directory at: http://localhost:3000/

> fluid-framework-docs-site@0.0.0 check-links
> linkcheck http://localhost:3000 --skip-file skipped-urls.txt

Crawling...

Stats:
  272202 links
    1863 destination URLs
    2108 URLs ignored
       0 warnings
       0 errors


Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants