Skip to content

Add callback_source to context to differentiate DAG vs task callbacks#61643

Draft
vasanthrpjan1-boop wants to merge 1 commit intoapache:mainfrom
vasanthrpjan1-boop:add-callback-source-to-context
Draft

Add callback_source to context to differentiate DAG vs task callbacks#61643
vasanthrpjan1-boop wants to merge 1 commit intoapache:mainfrom
vasanthrpjan1-boop:add-callback-source-to-context

Conversation

@vasanthrpjan1-boop
Copy link

@vasanthrpjan1-boop vasanthrpjan1-boop commented Feb 8, 2026

Summary

Adds a callback key to the callback context that carries a CallbackMeta named tuple, letting shared callbacks tell whether they were triggered at the DAG level or the task level.

  • New CallbackSource str enum ("dag" / "task") and CallbackMeta named tuple in airflow.sdk.definitions.context
  • Injects context["callback"] = CallbackMeta(source=...) in all four callback invocation paths
  • The Context TypedDict field is NotRequired because it is only set during callback execution, not during normal task template rendering
  • Both classes are exported from airflow.sdk for user access

Usage example

from airflow.sdk import CallbackMeta, CallbackSource

def my_callback(context):
    if context["callback"].source == CallbackSource.DAG:
        notify_team(f"DAG run finished: {context['reason']}")
    else:
        notify_team(f"Task {context['ti'].task_id} finished")

Using a CallbackMeta named tuple (instead of a bare enum) makes it easy to add more metadata fields in the future without breaking existing callbacks.

Test plan

  • New test test_task_callback_context_has_callback_meta covers success and failure task callbacks
  • Assertions added to existing DAG callback processor tests (with and without context_from_server)
  • Assertion added to existing task callback processor test

Gen-AI disclosure

This PR was developed with the assistance of a generative AI coding tool (Cursor). All code was reviewed, understood, and validated by me before inclusion. I take full responsibility for the changes.

Closes #61119

@boring-cyborg
Copy link

boring-cyborg bot commented Feb 8, 2026

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@boring-cyborg boring-cyborg bot added area:DAG-processing area:dev-tools area:task-sdk backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch labels Feb 8, 2026
@Asquator
Copy link
Contributor

Amazing! Exactly the thing I was planning to do.

@Asquator
Copy link
Contributor

Maybe it's worth wrapping CallbackSource into a generic CallbackMeta struct?

What if we want to add more meta information, like the type of a callback that was called?

class Context(TypedDict, total=False):
"""Jinja2 template context for task rendering."""

callback_source: NotRequired[CallbackSource]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it NotRequired? Doesn't every callback have a source?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or is it because we inject it just before the execution?

@potiuk potiuk marked this pull request as draft March 11, 2026 15:51
@potiuk
Copy link
Member

potiuk commented Mar 11, 2026

@vasanthrpjan1-boop This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

  • Merge conflicts: This PR has merge conflicts with the main branch. Your branch is 824 commits behind main. Please rebase your branch (git fetch origin && git rebase origin/main), resolve the conflicts, and push again. See contributing quick start.

Note: Your branch is 824 commits behind main. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.

What to do next:

  • The comment informs you what you need to do.
  • Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
  • Maintainers will then proceed with a normal review.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. If you have questions, feel free to ask on the Airflow Slack.

@vasanthrpjan1-boop vasanthrpjan1-boop force-pushed the add-callback-source-to-context branch from c9fe979 to 26b49f7 Compare March 12, 2026 08:04
@vasanthrpjan1-boop vasanthrpjan1-boop marked this pull request as ready for review March 12, 2026 08:19
@potiuk potiuk marked this pull request as draft March 13, 2026 13:48
@potiuk
Copy link
Member

potiuk commented Mar 13, 2026

@vasanthrpjan1-boop This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

  • Pre-commit / static checks: Failing: CI image checks / Static checks. Run prek run --from-ref main locally to find and fix issues. See Pre-commit / static checks docs.

Note: Your branch is 47 commits behind main. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.

What to do next:

  • The comment informs you what you need to do.
  • Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
  • Maintainers will then proceed with a normal review.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. If you have questions, feel free to ask on the Airflow Slack.

Currently a callback defined at the DAG level receives a context that
is nearly identical to a task-level callback context, making it
impossible to tell which level triggered the call.

This adds a `CallbackSource` enum ("dag" / "task") and injects it as
`context["callback_source"]` in every code path that invokes callbacks:

- task_runner._run_task_state_change_callbacks  (in-worker task path)
- processor._execute_dag_callbacks              (DAG processor path)
- processor._execute_task_callbacks             (DAG processor path)
- DagRun.handle_dag_callback                    (dag.test() path)

Closes: apache#61119
Co-authored-by: Cursor <cursoragent@cursor.com>
@vasanthrpjan1-boop vasanthrpjan1-boop force-pushed the add-callback-source-to-context branch from 26b49f7 to 7f05f97 Compare March 26, 2026 02:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:DAG-processing area:dev-tools area:task-sdk backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a way to differentiate between a task/DAG-level callback by context

3 participants