Skip to content

fix(celery): Propagate user-set headers #5581

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sentrivana merged 5 commits into from
Mar 3, 2026
Merged

fix(celery): Propagate user-set headers #5581

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
1a31c9b
fix(celery): Propagate user-set headers
sentrivana Mar 3, 2026
9e59ee5
Update sentry_sdk/integrations/celery/__init__.py
sentrivana Mar 3, 2026
73a96b2
.
sentrivana Mar 3, 2026
0187bfb
.
sentrivana Mar 3, 2026
fde36c9
.
sentrivana Mar 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions sentry_sdk/integrations/celery/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,12 @@ def _update_celery_task_headers(
if key.startswith("sentry-"):
updated_headers["headers"][key] = value

# Preserve user-provided custom headers in the inner "headers" dict
# so they survive to task.request.headers on the worker (celery#4875).
for key, value in original_headers.items():
if key != "headers" and key not in updated_headers["headers"]:
Copy link

@sentry sentry bot Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: A KeyError occurs when using custom headers with apply_async if the SDK client's propagate_traces option is disabled, as updated_headers["headers"] is accessed before it is created.
Severity: HIGH

Suggested Fix

Ensure the updated_headers["headers"] dictionary exists before the new loop that attempts to access it. This can be done by calling updated_headers.setdefault("headers", {}) unconditionally before the loop that preserves original headers.

Prompt for AI Agent 
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.

Location: sentry_sdk/integrations/celery/__init__.py#L237

Potential issue: A `KeyError` will occur when calling a Celery task's `apply_async` with
custom headers under specific configuration. The new code attempts to access
`updated_headers["headers"]` to preserve user-provided headers. However, this dictionary
key is only created if Sentry trace headers are generated. If the Sentry SDK client has
`propagate_traces=False` while the Celery integration has `propagate_traces=True` (the
default), no Sentry headers are generated, the `updated_headers["headers"]` key is never
created, and the new code will raise a `KeyError` when it tries to access it, causing
the task submission to fail.

Did we get this right? 👍 / 👎 to inform future reviews.

updated_headers["headers"][key] = value
Copy link

@cursor cursor bot Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

User header propagation skipped when tracing is inactive

Medium Severity

The new loop that copies user-provided headers into the inner updated_headers["headers"] dict is placed inside the if headers: block (line 191), so it only runs when Sentry produces non-empty trace propagation headers (i.e., when an active Sentry transaction exists). When tracing is disabled or there is no active transaction, headers is an empty dict, the block is skipped entirely, and user custom headers are never copied to the inner dict — leaving the original issue (#5566) unfixed for non-tracing deployments.

Fix in Cursor Fix in Web

Copy link
Contributor

@alexander-alderman-webb alexander-alderman-webb Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please address this as well if applicable 🙏 @sentrivana

Copy link
Contributor Author

@sentrivana sentrivana Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I can tell this is ok.

Celery itself has logic in there that actually moves the headers to the correct location as long as there are no headers. If tracing is disabled in the SDK, the SDK won't pre-populate the headers, and the default Celery logic will kick in and take care of it.

Only the case where the SDK actually populates some of the headers is problematic, since it causes Celery to skip the moving logic entirely. So we need to make sure that we either populate all or nothing.


return updated_headers


Expand Down
29 changes: 29 additions & 0 deletions tests/integrations/celery/test_celery.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -847,6 +847,35 @@ def test_send_task_wrapped(
assert span["trace_id"] == kwargs["headers"]["sentry-trace"].split("-")[0]


def test_user_custom_headers_accessible_in_task(init_celery):
"""
Regression test for https://github.com/getsentry/sentry-python/issues/5566

User-provided custom headers passed to apply_async() must be accessible
via task.request.headers on the worker side.
"""
celery = init_celery(traces_sample_rate=1.0)

@celery.task(name="custom_headers_task", bind=True)
def custom_headers_task(self):
return dict(self.request.headers or {})

custom_headers = {
"my_custom_key": "my_value",
"correlation_id": "abc-123",
"tenant_id": "tenant-42",
}

with start_transaction(name="test"):
result = custom_headers_task.apply_async(headers=custom_headers)

received_headers = result.get()
for key, value in custom_headers.items():
assert received_headers.get(key) == value, (
f"Custom header {key!r} not found in task.request.headers"
)


@pytest.mark.skip(reason="placeholder so that forked test does not come last")
def test_placeholder():
"""Forked tests must not come last in the module.
Expand Down
Loading