enhancement(transforms): dynamic rate for sample by jh7459-gh · Pull Request #25035 · vectordotdev/vector

jh7459-gh · 2026-03-24T17:01:07Z

Summary

This PR improves the sample transform for per-event sampling use cases by adding dynamic field-driven sampling while preserving safe fallback behavior.

Changes included:

Add ratio_field support for per-event ratio sampling ((0, 1]).
Add rate_field support for per-event 1/N sampling (positive integer).
Requires static sampling configuration (rate or ratio) as a default value (dynamic-only mode is not allowed).
Disallow configuring ratio_field and rate_field together (mirrors existing rate vs ratio exclusivity).
Keeps sample_rate_key populated with the effective sampling value used for each sampled event.
Add validation and transform tests for fallback behavior and invalid configs.
Update generated transform reference metadata and changelog fragment.

Vector configuration

[transforms.sample_dynamic]
type = "sample"
inputs = ["in"]

# required static fallback (exactly one of these)
ratio = 0.01
# rate = 100

# optional dynamic field (choose one)
ratio_field = "sample_ratio"
# rate_field = "sample_rate_n"

# optional deterministic/keyed sampling controls
key_field = "trace_id"
group_by = "{{ service }}"

How did you test this PR?

Ran locally:

cargo fmt --all
cargo test --no-default-features --features transforms-sample sample:: --lib
cargo clippy --no-default-features --features transforms-sample --lib --tests -- -D warnings
./scripts/check_changelog_fragments.sh

Test coverage added/updated includes:

reject dynamic ratio-only config
reject dynamic rate-only config
reject config that sets both dynamic fields
accept static + dynamic config
dynamic ratio overrides static
dynamic rate overrides static
dynamic ratio fallback to static ratio/rate when missing
dynamic rate fallback to static ratio when missing

Change Type

Is this a breaking change?

Yes
No

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

References

Related: dynamic per-event sampling use case for sample transform

github-actions · 2026-03-24T17:01:27Z

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

jh7459-gh · 2026-03-24T17:10:00Z

I have read the CLA Document and I hereby sign the CLA

jh7459-gh · 2026-03-24T17:10:15Z

recheck

jh7459-gh · 2026-03-24T17:25:06Z

recheck

jh7459-gh · 2026-03-24T17:32:34Z

recheck

jh7459-gh · 2026-03-24T17:34:09Z

recheck

jh7459-gh · 2026-03-24T17:35:10Z

recheck

jh7459-gh · 2026-03-24T17:36:12Z

recheck

jh7459-gh · 2026-03-24T17:37:14Z

recheck

jh7459-gh · 2026-03-24T17:38:16Z

recheck

jh7459-gh · 2026-03-24T17:39:17Z

recheck

jh7459-gh · 2026-03-24T17:40:19Z

recheck

jh7459-gh · 2026-03-24T17:41:21Z

recheck

jh7459-gh · 2026-03-24T17:42:23Z

recheck

jh7459-gh · 2026-03-24T17:43:25Z

recheck

jh7459-gh · 2026-03-24T17:44:27Z

recheck

jh7459-gh · 2026-03-24T17:45:29Z

recheck

jh7459-gh · 2026-03-24T17:46:31Z

recheck

jh7459-gh · 2026-03-24T17:47:32Z

recheck

jh7459-gh · 2026-03-24T17:48:34Z

recheck

jh7459-gh · 2026-03-24T17:49:36Z

recheck

jh7459-gh · 2026-03-24T17:50:38Z

recheck

jh7459-gh · 2026-03-24T17:51:39Z

recheck

jh7459-gh · 2026-03-24T17:52:41Z

recheck

jh7459-gh · 2026-03-24T17:53:43Z

recheck

chatgpt-codex-connector · 2026-03-24T19:06:45Z

Codex Review: Didn't find any major issues. 🎉

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

.vscode/settings.json

pront · 2026-03-25T13:02:57Z

@codex review

chatgpt-codex-connector · 2026-03-25T13:28:57Z

Codex Review: Didn't find any major issues. 🚀

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

pront · 2026-03-25T16:33:19Z

src/transforms/sample/transform.rs

+                self.increment_dynamic_rate(rate, group_by_key, value)
+            }
+            None => self.static_mode.increment(group_by_key, value),
+        };


Take the following with a grain of salt since it is AI generated but I do think we should make this stateless. Free free to discuss more in this thread.

Dynamic state is stale when per-event value changes across a group bucket

The counter-based accumulators (dynamic_values, dynamic_counters) are the root cause of the stale-state bug — each event's outcome is tied
to the history of previous events in the same bucket. For a truly per-event dynamic path, don't accumulate anything. Each event should be
decided independently.

The two cases

When key_field is set: already stateless — the hash of the key is compared against the threshold. No change needed.

When key_field is NOT set: currently uses the accumulator. Replace with rand::random:

fn sample_with_dynamic_ratio(ratio: f64, value: Option<&Value>) -> bool { match value { Some(v) => SampleMode::hash_with_ratio(v.to_string_lossy().as_bytes(), ratio), None => rand::random::<f64>() < ratio, } } fn sample_with_dynamic_rate(rate: u64, value: Option<&Value>) -> bool { match value { Some(v) => seahash::hash(v.to_string_lossy().as_bytes()).is_multiple_of(rate), None => rand::random::<f64>() < (1.0 / rate as f64), } }

These become plain associated fn with no &mut self and no group_by_key parameter.

What to remove

dynamic_values: HashMap<Option, f64> — delete from struct

dynamic_counters: HashMap<Option, u64> — delete from struct

The HashMap::default() initializers in new_with_dynamic

rand is already a workspace dependency so no new dep needed

Updated call site in transform()

let should_sample = match event_sample_mode { Some(EventSampleMode::Ratio(ratio)) => Self::sample_with_dynamic_ratio(ratio, value), Some(EventSampleMode::Rate(rate)) => Self::sample_with_dynamic_rate(rate, value), None => self.static_mode.increment(group_by_key, value), };

group_by_key is now only passed to the static path. &mut self on transform is still needed for the static path's counters.

Trade-off

The counter-based approach guaranteed exactly the configured ratio over a window (e.g. exactly 1-in-10 for ratio=0.1). Random sampling gives the
correct ratio in expectation but with variance — noticeable on low-traffic streams. If deterministic throughput matters for the dynamic path,
the alternative fix is to key state by (group_by_key, ratio_or_rate) instead of just group_by_key. But stateless random is simpler and
consistent with how key_field-based sampling already works.

👍

removed dynamic accumulator, made dynamic sampling stateless for keyless events (rand::random)

keeps hashed behavior when a key value is present

pront · 2026-03-25T16:36:34Z

src/transforms/sample/config.rs

+    /// static sampling settings (`rate` or `ratio`) are used as a fallback.
+    /// This option cannot be used together with `ratio_field`.
+    #[configurable(metadata(docs::examples = "sample_rate_n"))]
+    pub rate_field: Option<String>,


With dynamic fields, the hash for field_key key is checked against a different threshold per event (driven by ratio_field/rate_field). Two events sharing a key_field value but with different dynamic values can now be split, silently violating the existing contract.

Something to consider:

Document that key_field only maintains its coherence guarantee when the dynamic field value is constant within that key

or reject the combination at config-build time.

👍

chose the strict option

config now rejects key_field with ratio_field/rate_field + explicit validation error

jh7459-gh · 2026-03-25T20:00:15Z

recheck

jh7459-gh · 2026-03-25T22:20:14Z

I have read the CLA Document and I hereby sign the CLA

jh7459-gh · 2026-03-25T22:20:19Z

recheck

enhancement(transforms): dynamic rate for sample

122ccd5

jh7459-gh requested review from a team as code owners March 24, 2026 17:01

github-actions bot added domain: transforms Anything related to Vector's transform components domain: external docs Anything related to Vector's external, public documentation labels Mar 24, 2026

This comment was marked as spam.

Sign in to view

thomasqueirozb reviewed Mar 24, 2026

View reviewed changes

.vscode/settings.json Outdated Show resolved Hide resolved

chore: remove accidental VSCode settings file

90d6ccb

urseberry self-assigned this Mar 24, 2026

urseberry approved these changes Mar 24, 2026

View reviewed changes

Merge branch 'master' into enhancement/sample-rate-field-clean2

f27df4d

pront reviewed Mar 25, 2026

View reviewed changes

jh7459-gh added 3 commits March 25, 2026 15:33

fix(sample): make dynamic sampling stateless and reject key_field combo

bd5c1aa

Merge branch 'master' into enhancement/sample-rate-field-clean2

a25708d

chore(docs): regenerate component cue docs

be4db6e

Conversation

jh7459-gh commented Mar 24, 2026

Summary

Vector configuration

How did you test this PR?

Change Type

Is this a breaking change?

Does this PR include user facing changes?

References

Uh oh!

github-actions bot commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jh7459-gh commented Mar 24, 2026

Uh oh!

jh7459-gh commented Mar 24, 2026

Uh oh!

jh7459-gh commented Mar 24, 2026

Uh oh!

jh7459-gh commented Mar 24, 2026

Uh oh!

jh7459-gh commented Mar 24, 2026

Uh oh!

jh7459-gh commented Mar 24, 2026

Uh oh!

jh7459-gh commented Mar 24, 2026

Uh oh!

jh7459-gh commented Mar 24, 2026

Uh oh!

jh7459-gh commented Mar 24, 2026

Uh oh!

jh7459-gh commented Mar 24, 2026

Uh oh!

jh7459-gh commented Mar 24, 2026

Uh oh!

jh7459-gh commented Mar 24, 2026

Uh oh!

jh7459-gh commented Mar 24, 2026

Uh oh!

jh7459-gh commented Mar 24, 2026

Uh oh!

jh7459-gh commented Mar 24, 2026

Uh oh!

jh7459-gh commented Mar 24, 2026

Uh oh!

jh7459-gh commented Mar 24, 2026

Uh oh!

jh7459-gh commented Mar 24, 2026

Uh oh!

jh7459-gh commented Mar 24, 2026

Uh oh!

jh7459-gh commented Mar 24, 2026

Uh oh!

jh7459-gh commented Mar 24, 2026

Uh oh!

jh7459-gh commented Mar 24, 2026

Uh oh!

jh7459-gh commented Mar 24, 2026

Uh oh!

jh7459-gh commented Mar 24, 2026

Uh oh!

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

chatgpt-codex-connector bot commented Mar 24, 2026

Uh oh!

Uh oh!

pront commented Mar 25, 2026

github-actions bot commented Mar 24, 2026 •

edited

Loading

jh7459-gh Mar 25, 2026 •

edited

Loading