Traceable Configuration Reload

[brbzull0]

New Configuration Reload Framework

TL;DR

NOTE: Backward compatible: The existing traffic_ctl config reload command works exactly as before —
same syntax, same behavior from the operator's perspective. Internally, it now fires the new reload
logic, which means every reload is automatically tracked, timed, and queryable.

New traffic_ctl commands:

# Basic reload — works exactly as before, but now returns a token for tracking
$ traffic_ctl config reload
✔ Reload scheduled [rldtk-1739808000000]

  Monitor : traffic_ctl config reload -t rldtk-1739808000000 -m
  Details : traffic_ctl config reload -t rldtk-1739808000000 -s -l

# Monitor mode with a custom token
$ traffic_ctl config reload -t deploy-v2.1 -m
✔ Reload scheduled [deploy-v2.1]
✔ [deploy-v2.1] ████████████████████ 11/11  success  (245ms)

# Full status report
$ traffic_ctl config status -t deploy-v2.1
✔ Reload [success] — deploy-v2.1
  Started : 2025 Feb 17 12:00:00.123
  Finished: 2025 Feb 17 12:00:00.368
  Duration: 245ms

  ✔ 11 success  ◌ 0 in-progress  ✗ 0 failed  (11 total)

  Tasks:
   ✔ logging.yaml ··························· 120ms
   ✔ ip_allow.yaml ·························· 18ms
   ✔ remap.config ··························· 42ms
   ✔ ssl_client_coordinator ················· 35ms
   ├─ ✔ sni.yaml ··························· 20ms
   └─ ✔ ssl_multicert.config ··············· 15ms
   ...

Failed reload — monitor mode:

$ traffic_ctl config reload -t hotfix-ssl-cert -m
✔ Reload scheduled [hotfix-ssl-cert]
✗ [hotfix-ssl-cert] ██████████████░░░░░░ 9/11  fail  (310ms)

  Details : traffic_ctl config status -t hotfix-ssl-cert

Failed reload — status report:

$ traffic_ctl config status -t hotfix-ssl-cert
✗ Reload [fail] — hotfix-ssl-cert
  Started : 2025 Feb 17 14:30:10.500
  Finished: 2025 Feb 17 14:30:10.810
  Duration: 310ms

  ✔ 9 success  ◌ 0 in-progress  ✗ 2 failed  (11 total)

  Tasks:
   ✔ ip_allow.yaml ·························· 18ms
   ✔ remap.config ··························· 42ms
   ✗ logging.yaml ·························· 120ms  ✗ FAIL
   ✗ ssl_client_coordinator ················· 85ms  ✗ FAIL
   ├─ ✔ sni.yaml ··························· 20ms
   └─ ✗ ssl_multicert.config ··············· 65ms  ✗ FAIL
   ...

Inline YAML reload (runtime only, not persisted to disk):

Note: Inline YAML reload is currently disabled — no config handler supports ConfigSource::FileAndRpc
yet. The infrastructure is in place and will be enabled as handlers are migrated. See TODO below.

$ traffic_ctl config reload -d @ip_allow_new.yaml -t update-ip-rules -m
✔ Reload scheduled [update-ip-rules]
✔ [update-ip-rules] ████████████████████ 1/1  success  (18ms)

Note: Inline configuration is NOT persisted to disk.
      Server restart will revert to file-based configuration.

The -d flag accepts @filename to read from a file, or @- to read from stdin. The YAML file
uses registry keys as top-level keys — the key string passed as the first argument to
register_config() or register_record_config(). The content under each key is the actual YAML
that the config file normally contains — it is passed as-is to the handler via ctx.supplied_yaml().
A single file can target multiple handlers:

# reload_rules.yaml — multiple configs in one file
# Each top-level key is a registry key (as declared in register_config()).
# The value is the full config content, exactly as it appears in the config file.
ip_allow:
  - apply: in
    ip_addrs: 0.0.0.0/0
    action: allow
    methods: ALL
sni:
  - fqdn: "*.example.com"
    verify_client: NONE

# From file — reloads both ip_allow and sni handlers
$ traffic_ctl config reload -d @reload_rules.yaml -t update-rules -m

# From stdin — pipe YAML directly into ATS
$ cat reload_rules.yaml | traffic_ctl config reload -d @- -m

New traffic_ctl Commands

Command	Description
traffic_ctl config reload	Trigger a file-based reload. Shows token and next-step hints.
traffic_ctl config reload -m	Trigger and monitor with a live progress bar.
traffic_ctl config reload -s -l	Trigger and immediately show detailed report with logs.
traffic_ctl config reload -t <token>	Reload with a custom token.
traffic_ctl config reload -d @file.yaml	Inline reload from file (runtime only, not persisted).
traffic_ctl config reload -d @-	Inline reload from stdin.
traffic_ctl config reload --force	Force a new reload even if one is in progress.
traffic_ctl config status	Show the last reload status.
traffic_ctl config status -t <token>	Show status of a specific reload.
traffic_ctl config status -c all	Show full reload history.

All commands support --format json to output the raw JSONRPC response instead of human-readable
text. This is useful for automation, CI pipelines, or any tool that consumes structured output
directly:

$ traffic_ctl config status -t reload1 --format json
{"tasks": [{"config_token": "reload1", "status": "success", "description": "Main reload task - 2026 Feb 18 19:46:02", ...}]}

Full JSON output

{
  "tasks": [
    {
      "config_token": "reload1",
      "status": "success",
      "description": "Main reload task - 2026 Feb 18 20:03:10",
      "filename": "",
      "meta": {
        "created_time_ms": "1771444990585",
        "last_updated_time_ms": "1771444991015",
        "main_task": "true"
      },
      "log": [],
      "sub_tasks": [
        {
          "config_token": "reload1",
          "status": "success",
          "description": "ip_allow",
          "filename": "/opt/ats/etc/trafficserver/ip_allow.yaml",
          "meta": {
            "created_time_ms": "1771444991013",
            "last_updated_time_ms": "1771444991015",
            "main_task": "false"
          },
          "log": [],
          "logs": [
            "Finished loading"
          ],
          "sub_tasks": []
        },
        {
          "config_token": "reload1",
          "status": "success",
          "description": "ssl_ticket_key",
          "filename": "",
          "meta": {
            "created_time_ms": "1771444991015",
            "last_updated_time_ms": "1771444991015",
            "main_task": "false"
          },
          "log": [],
          "logs": [
            "SSL ticket key reloaded"
          ],
          "sub_tasks": []
        }
      ]
    }
  ]
}

New JSONRPC APIs

Method	Description
admin_config_reload	Unified reload — file-based (default) or inline when configs param is present. Params: token, force, configs.
get_reload_config_status	Query reload status by token or get the last N reloads via count.

Inline reload RPC example:

jsonrpc: "2.0"
method: "admin_config_reload"
params:
  token: "update-ip-and-sni"
  configs:
    ip_allow:
      - apply: in
        ip_addrs: 0.0.0.0/0
        action: allow
        methods: ALL
    sni:
      - fqdn: "*.example.com"
        verify_client: NONE

---

Background: Issues with the Previous Reload Mechanism

The previous configuration reload relied on a loose collection of independent record callbacks
(RecRegisterConfigUpdateCb) wired through FileManager and AddConfigFilesHere.cc. Each config
module registered its file independently, and reloads were fire-and-forget:

• No visibility — There was no way to know whether a reload succeeded or failed, which handlers
  ran, or how long each one took.
• No coordination — Each handler ran independently with no shared context. There was no concept of
  a "reload session" grouping all config updates triggered by a single request.
• No inline content — Configuration could only be reloaded from files on disk. There was no way
  to push YAML content at runtime through the RPC or CLI.
• Scattered registration — File registrations were split between AddConfigFilesHere.cc (for
  FileManager) and individual modules (for record callbacks), making it hard to reason about which
  files were tracked and which records triggered reloads.
• No token tracking — There was no identifier for a reload operation, so you couldn't query the
  status of a specific reload or distinguish between overlapping reloads.

What the New Design Solves

1. Full reload traceability — Every reload gets a token. Each config handler reports its status
   (in_progress, success, fail) through a ConfigContext. Results are aggregated into a task
   tree with per-handler timings and logs.
2. Centralized registration — ConfigRegistry is the single source of truth for all config files,
   their filename records, trigger records, and reload handlers.
3. Inline YAML injection — Handlers that opt in (ConfigSource::FileAndRpc) can receive YAML
   content directly through the RPC, without writing to disk. This is runtime-only — the content
   lives in memory and is lost on restart.
4. Coordinated reload sessions — ReloadCoordinator manages the lifecycle of each reload:
   token generation, concurrency control (--force to override), timeout detection, and history.
5. CLI observability — traffic_ctl config reload -m shows a live progress bar.
   traffic_ctl config status provides a full post-mortem with task tree, durations, and failure
   details.

Basic Design

┌─────────────┐   JSONRPC    ┌────────────────┐
│ traffic_ctl  │────────────►│  RPC Handler   │
│ config reload│             │  reload_config  │
└─────────────┘             └───────┬────────┘
                                    │
          ┌─────────────────────────┼─────────────────────────┐
          │                         ▼                         │
          │              ┌──────────────────┐                 │
          │              │ ReloadCoordinator │                 │
          │              │  - prepare_reload │                 │
          │              │  - token tracking │                 │
          │              │  - history        │                 │
          │              └────────┬─────────┘                 │
          │                       │                           │
          │           ┌───────────┴──────────┐                │
          │           ▼                      ▼                │
          │   ┌──────────────┐      ┌──────────────────┐      │
          │   │ File-based   │      │ Inline mode      │      │
          │   │ FileManager  │      │ set_passed_config │      │
          │   │ rereadConfig │      │ schedule_reload   │      │
          │   └──────┬───────┘      └────────┬─────────┘      │
          │          │                       │                │
          │          └───────────┬───────────┘                │
          │                      ▼                            │
          │            ┌──────────────────┐                   │
          │            │  ConfigRegistry  │                   │
          │            │  execute_reload  │                   │
          │            └────────┬─────────┘                   │
          │                     ▼                             │
          │           ┌──────────────────┐                    │
          │           │  ConfigContext   │                    │
          │           │  - in_progress() │                    │
          │           │  - log()         │                    │
          │           │  - complete()    │                    │
          │           │  - fail()        │                    │
          │           │  - supplied_yaml()│                   │
          │           └────────┬─────────┘                    │
          │                    ▼                              │
          │           ┌──────────────────┐                    │
          │           │  Handler         │                    │
          │           │  (IpAllow, SNI,  │                    │
          │           │   remap, etc.)   │                    │
          │           └──────────────────┘                    │
          └───────────────────────────────────────────────────┘

Key components:

Component	Role
ConfigRegistry	Singleton registry mapping config keys to handlers, filenames, trigger records. Self-registers with FileManager.
ReloadCoordinator	Manages reload sessions: token generation, concurrency, timeout detection, history.
ConfigReloadTask	Tracks a single reload operation as a tree of sub-tasks with status, timings, and logs.
ConfigContext	Lightweight context passed to handlers. Provides in_progress(), complete(), fail(), log(), supplied_yaml(), and add_dependent_ctx(). Safe no-op at startup (no active reload task).
ConfigReloadProgress	Periodic checker that detects stuck tasks and marks them as TIMEOUT.

Stuck reload checker:
ConfigReloadProgress is a periodic continuation scheduled on ET_TASK. It monitors active reload
tasks and marks any that exceed the configured timeout as TIMEOUT. This acts as a safety net for
handlers that fail to call ctx.complete() or ctx.fail() — for example, if a handler crashes,
deadlocks, or its deferred thread never executes. The checker reads proxy.config.admin.reload.timeout
dynamically at each interval, so the timeout can be adjusted at runtime without a restart. This is
a simple record read (RecGetRecordString), not an expensive operation. Setting the
timeout to "0" disables it (tasks will run indefinitely until completion).
The checker is not a global poller — a new instance is created per-reload and self-terminates once
the task reaches a terminal state. No idle polling when no reload is in progress.

How Handlers Work

Before — scattered registration (ip_allow example):

Registration was split across multiple files with no centralized tracking:

// 1. AddConfigFilesHere.cc — register with FileManager for mtime detection
registerFile("proxy.config.cache.ip_allow.filename", ts::filename::IP_ALLOW, NOT_REQUIRED);
registerFile("proxy.config.cache.ip_categories.filename", ts::filename::IP_CATEGORIES, NOT_REQUIRED);

// 2. IPAllow.cc — attach record callback (fire-and-forget, no status tracking)
ConfigUpdateHandler<IpAllow> *ipAllowUpdate = new ConfigUpdateHandler<IpAllow>("ip_allow");
ipAllowUpdate->attach("proxy.config.cache.ip_allow.filename");

// 3. IpAllow::reconfigure() — no context, no status, no tracing
void IpAllow::reconfigure() {
    // ... load config from disk, no way to report success/failure ...
}

Now — each module self-registers with full tracing:

Each module registers itself directly with ConfigRegistry. No more separate AddConfigFilesHere.cc
entry — the registry handles FileManager registration, record callbacks, and status tracking
automatically:

// IPAllow.cc — one call replaces all three steps above
config::ConfigRegistry::Get_Instance().register_config(
    "ip_allow",                                           // registry key
    ts::filename::IP_ALLOW,                               // default filename
    "proxy.config.cache.ip_allow.filename",               // record holding filename
    [](ConfigContext ctx) { IpAllow::reconfigure(ctx); }, // handler with context
    config::ConfigSource::FileOnly,                       // content source
    {"proxy.config.cache.ip_allow.filename"});            // trigger records

// Auxiliary file — attach ip_categories as a dependency (changes trigger ip_allow reload)
config::ConfigRegistry::Get_Instance().add_file_dependency(
    "ip_allow",                                  // config key to attach to
    "proxy.config.cache.ip_categories.filename", // record holding the filename
    ts::filename::IP_CATEGORIES,                 // default filename
    false);                                      // not required

Additional triggers can be attached from any module at any time:

// From another module — attach an extra record trigger
config::ConfigRegistry::Get_Instance().attach("ip_allow", "proxy.config.some.extra.record");

Composite configs can declare file dependencies and dependency keys. For example, SSLClientCoordinator
owns sni.yaml and ssl_multicert.config as children:

// Main registration (no file of its own — it's a pure coordinator)
// From SSLClientCoordinator::startup()
config::ConfigRegistry::Get_Instance().register_record_config(
    "ssl_client_coordinator",                                          // registry key
    [](ConfigContext ctx) { SSLClientCoordinator::reconfigure(ctx); }, // reload handler
    {"proxy.config.ssl.client.cert.path",                              // trigger records
     "proxy.config.ssl.client.cert.filename", "proxy.config.ssl.client.private_key.path",
     "proxy.config.ssl.client.private_key.filename", "proxy.config.ssl.keylog_file",
     "proxy.config.ssl.server.cert.path", "proxy.config.ssl.server.private_key.path",
     "proxy.config.ssl.server.cert_chain.filename",
     "proxy.config.ssl.server.session_ticket.enable"});

// Track sni.yaml — FileManager watches for mtime changes, record wired to trigger reload
config::ConfigRegistry::Get_Instance().add_file_and_node_dependency(
    "ssl_client_coordinator", "sni",
    "proxy.config.ssl.servername.filename", ts::filename::SNI, false);

// Track ssl_multicert.config — same pattern
config::ConfigRegistry::Get_Instance().add_file_and_node_dependency(
    "ssl_client_coordinator", "ssl_multicert",
    "proxy.config.ssl.server.multicert.filename", ts::filename::SSL_MULTICERT, false);

Handler interaction with ConfigContext:

Each config module implements a C++ reload handler — the callback passed to register_config().
The handler reports progress through the ConfigContext:

void IpAllow::reconfigure(ConfigContext ctx) {
    ctx.in_progress();

    // ... load config from disk ...

    ctx.complete("Loaded successfully");
    // or on error:
    // ctx.fail(errata, "Failed to load");
}

When a reload fires, the handler receives a ConfigContext:

• File source — ctx.supplied_yaml() is undefined; the handler reads from its registered file on disk.
• RPC source — ctx.supplied_yaml() contains the YAML node passed via --data / RPC.
  The content is runtime-only and is never written to disk.

Handlers report progress:

ctx.in_progress("Parsing ip_allow.yaml");
ctx.log("Loaded 42 rules");
ctx.complete("Finished loading");
// or on error:
ctx.fail(errata, "Failed to load ip_allow.yaml");

Supplied YAML — inline content via -d / RPC:

Note: The infrastructure for RPC-supplied YAML is fully implemented, but no handler currently
opts into ConfigSource::FileAndRpc. File-based handlers use ConfigSource::FileOnly, and
record-only handlers use ConfigSource::RecordOnly (implicitly via register_record_config()).

When a handler opts into ConfigSource::FileAndRpc, it can receive YAML content directly instead
of reading from disk. The handler checks ctx.supplied_yaml() to determine the source:

void IpAllow::reconfigure(ConfigContext ctx) {
    ctx.in_progress();

    YAML::Node root;
    if (auto yaml = ctx.supplied_yaml()) {
        // Inline mode: YAML supplied via -d flag or JSONRPC.
        // Not persisted to disk — runtime only.
        root = yaml;
    } else {
        // File mode: read from the registered config file on disk.
        root = YAML::LoadFile(config_filename);
    }

    // ... parse and apply config ...

    ctx.complete("Loaded successfully");
}

For composite configs (e.g., SSLClientCoordinator), handlers create child contexts to track
each sub-config independently. From SSLClientCoordinator::reconfigure():

SSLConfig::reconfigure(reconf_ctx.add_dependent_ctx("SSLConfig"));
SNIConfig::reconfigure(reconf_ctx.add_dependent_ctx("SNIConfig"));
SSLCertificateConfig::reconfigure(reconf_ctx.add_dependent_ctx("SSLCertificateConfig"));
reconf_ctx.complete("SSL configs reloaded");

The parent task automatically aggregates status from its children. In traffic_ctl config status,
this renders as a tree:

   ✔ ssl_client_coordinator ················· 35ms
   ├─ ✔ SSLConfig ·························· 10ms
   ├─ ✔ SNIConfig ·························· 12ms
   └─ ✔ SSLCertificateConfig ·············· 13ms

Design Challenges

1. Handlers must reach a terminal state — or the task hangs

The entire tracing model relies on handlers calling ctx.complete() or ctx.fail() before
returning. If a handler returns without reaching a terminal state, the task stays IN_PROGRESS
indefinitely until the timeout checker marks it as TIMEOUT.

After execute_reload() calls the handler, it checks ctx.is_terminal() and emits a warning
if the handler left the task in a non-terminal state:

entry_copy.handler(ctx);
if (!ctx.is_terminal()) {
    Warning("Config '%s' handler returned without reaching a terminal state. "
            "If the handler deferred work to another thread, ensure ctx.complete() or ctx.fail() "
            "is called when processing finishes; otherwise the task will remain in progress "
            "until the timeout checker marks it as TIMEOUT.",
            entry_copy.key.c_str());
}

The safety net: ConfigReloadProgress runs periodically on ET_TASK and marks stuck tasks as
TIMEOUT after the configured duration (proxy.config.admin.reload.timeout, default: 1h).

2. Parent status aggregation from sub-tasks

Parent tasks do not track their own status directly — they derive it from their children.
When a child calls complete() or fail(), it notifies its parent, which re-evaluates:

• Any child failed or timed out → parent is FAIL
• Any child still in progress → parent stays IN_PROGRESS
• All children succeeded → parent is SUCCESS

This aggregation is recursive: a sub-task can have its own children (e.g.,
ssl_client_coordinator → sni + ssl_multicert), and status bubbles up through the tree.

One subtle issue: if a handler creates child contexts but forgets to call complete() or
fail() on one of them, that child stays CREATED and the parent never reaches SUCCESS.
It is the handler developer's responsibility to ensure every ConfigContext (and its children)
reaches a terminal state (complete() or fail()). The timeout checker is the ultimate safety
net for cases where this is not properly handled.

3. Startup vs. reload — same handler, different context

Handlers are called both at startup (initial config load) and during runtime reloads. At startup,
there is no active ReloadCoordinator task, so all ConfigContext operations (in_progress(),
complete(), fail(), log()) are safe no-ops — they check _task.lock() and return
immediately if the weak pointer is expired or empty.

This avoids having two separate code paths for startup vs. reload. The handler logic is identical
in both cases:

void IpAllow::reconfigure(ConfigContext ctx) {
    ctx.in_progress();  // no-op at startup, tracks progress during reload
    // ... load config ...
    ctx.complete();     // no-op at startup, marks task as SUCCESS during reload
}

4. Known issue: ssl_client_coordinator may appear twice in reload status

ssl_client_coordinator registers multiple trigger records and file dependencies, each wiring an
independent on_record_change callback with no deduplication. When several of these fire during
the same reload (e.g., both sni.yaml and ssl_multicert.config changed), the handler executes
more than once, producing duplicate entries in the reload status output.

This is a pre-existing issue present on master — see #11724.

5. Plugin support

Plugins are not supported by ConfigRegistry in this PR. The legacy reload notification
mechanism (TSMgmtUpdateRegister) still works — plugins registered through it will continue
to be invoked via FileManager::invokeConfigPluginCallbacks() during every reload cycle.
A dedicated plugin API to let plugins register their own config handlers and participate in
the reload framework will be addressed in a separate PR.

---

Configs Migrated to ConfigRegistry

Config	Key	File
IP Allow	ip_allow	ip_allow.yaml
IP Categories	(dependency of ip_allow)	ip_categories.yaml
Cache Control	cache_control	cache.config
Cache Hosting	cache_hosting	hosting.config
Parent Selection	parent_proxy	parent.config
Split DNS	split_dns	splitdns.config
Remap	remap	remap.config ¹
Logging	logging	logging.yaml
SSL/TLS (coordinator)	ssl_client_coordinator	—
SNI	(dependency of ssl_client_coordinator)	sni.yaml
SSL Multicert	(dependency of ssl_client_coordinator)	ssl_multicert.config
SSL Ticket Key	ssl_ticket_key	(record-only, no file)

¹ Remap migration will be refactored after #12813 (remap.yaml)
and #12669 (virtual hosts) land.

New Configuration Records

records:
  admin:
    reload:
      # Maximum time a reload task can run before being marked as TIMEOUT.
      # Supports duration strings: "30s", "5min", "1h". Set to "0" to disable.
      # Default: 1h. Updateable at runtime (RECU_DYNAMIC).
      timeout: 1h

      # How often the progress checker polls for stuck tasks (minimum: 1s).
      # Supports duration strings: "1s", "5s", "30s".
      # Default: 2s. Updateable at runtime (RECU_DYNAMIC).
      check_interval: 2s

---

TODO

• Clean up - Run some clean up on this code.
• Documentation — Add user-facing docs for the new traffic_ctl config reload / config status commands and the JSONRPC APIs.
• Improve error reporting — All config loaders are migrated to ConfigRegistry. Remaining work: fully log detailed errors via ctx.log(), ctx.fail(), etc.
• Enable inline YAML for more handlers — Currently file-based handlers use ConfigSource::FileOnly and record-only handlers use ConfigSource::RecordOnly. Migrate file-based handlers to FileAndRpc so they can read YAML directly from the RPC (via ctx.supplied_yaml()).
• Remove legacy reload infrastructure — Most config loaders are migrated. Remove ConfigUpdateHandler/ConfigUpdateContinuation and the remaining registerFile() calls in AddConfigFilesHere.cc.
• Consolidate AddConfigFilesHere.cc into ConfigRegistry — Remaining static files (storage.config, volume.config, plugin.config, etc.) can be registered in ConfigRegistry as inventory-only entries (no handler, no reload) to fully retire AddConfigFilesHere.cc.
• Autest reload extension — Implement an autest extension that checks reload success/failure via the JSONRPC status API (traffic_ctl config status -t <token>) instead of grepping log files.
• Trace record-triggered reloads — Record-based reloads (via trigger_records / RecRegisterConfigUpdateCb) are not currently tracked. Create a main task with a synthetic token so they appear in traffic_ctl config status.
• Expose ConfigRegistry to plugins — Add a plugin API so plugins can register their own config handlers and participate in the reload framework. This will be in a separate PR.
• Additional tests — Expand autest coverage.

---

Dependencies and Related Issues

Fixes #12324 — Improving traffic_ctl config reload.

This PR will likely land after:

• #12813
• #12669

There should be no major conflicts with those PRs. Conversation and coordination needs to be done before merging.