Ledger Chunk download API

[achamayou]

Adding an API aiming to allow a ledger backup process that does not have access to the ledger storage directories to efficiently fetch committed ledger chunks for archival/retention purposes.

HEAD/GET /node/ledger-chunk
HEAD/GET /node/ledger-chunk/{chunk_name}

Typical Scenario

sequenceDiagram
  Note over Client: Client asks for chunk starting at index
  Client->>+Backup: GET /node/ledger-chunk?since=index
  Backup->>-Client: 308 Location: /node/ledger-chunk/ledger_startIndex_endIndex.committed
  Note over Backup: Backup node has that chunk
  Client->>+Backup: GET /node/ledger-chunk/ledger_startIndex_endIndex.committed
  Backup->>-Client: 200 <Chunk Contents>
  Client->>+Backup: GET /node/ledger-chunk?since=endIndex+1
  Note over Backup: Backup node does not yet have a committed chunk starting at endIndex+1
  Backup->>-Client: 308 Location: https://primary/node/ledger-chunk?since=endIndex+1
  Client->>+Primary: GET /node/ledger-chunk?since=endIndex+1
  Primary->>-Client: 308 Location: /node/ledger-chunk/ledger_endIndex+1_nextEndIndex.committed
  Client->>+Primary: GET /node/ledger-chunk/ledger_startIndex_endIndex.committed
  Note over Primary: But the Primary node has the most recent chunk already
  Primary->>-Client: 200 <Chunk Contents>

Loading

Alternative Scenario

The initial node (Primary in this case) that client hits has started from a snapshot, and does not have some past chunks. To make this more readable, let's say that Primary started from snapshot_100.committed and locally has:

ledger_1-50.committed
ledger_101-150.committed

Backup has:

ledger_1-50.committed
ledger_51-100.committed

sequenceDiagram
  Client->>+Primary: GET /node/ledger-chunk?since=51
  Primary->>-Client: 308 Location: https://backup/node/ledger-chunk?since=51
  Client->>+Backup: GET /node/ledger-chunk?since=51
  Backup->>-Client: 308 Location: /node/ledger-chunk/ledger_51-100.committed
  Client->>+Backup: GET /node/ledger-chunk/ledger_51-100.committed
  Backup->>-Client: 200 <Chunk Contents>
  Client->>+Backup: GET /node/ledger-chunk?since=101
  Note over Backup: Backup node does not have 101-150
  Backup->>-Client: 308 Location: https://primary/node/ledger-chunk?since=51
  Client->>+Primary: GET /node/ledger-chunk?since=101

Loading

• Serialise all access to Ledger
• Check performance impact of serialisation
• Add fetch file API
• Test fetch file API
• File API redirects to primary if requests go beyond the last known local committed file
• Test primary redirection
• Separate LedgerChunkRead interface feature
• Factor out pass with the snapshot handlers
• /ledger-chunk?since=index needs to redirect to the next node in a stable order when a file is not found and index is strictly less than locally known start index (i.e. the startup snapshot version)
• Documentation/Changelog for fetch file API
• Committed chunks need to be fsync()'ed before they are closed
• Final performance check

[achamayou]

This is blocked on #7576 and #7578 being merged and backported.

[Copilot]

Pull request overview

This PR introduces a new API for downloading committed ledger chunks, designed to enable ledger backup processes that don't have direct access to ledger storage directories. The implementation adds HTTP endpoints /node/ledger-chunk and /node/ledger-chunk/{chunk_name} with intelligent redirection logic to handle scenarios where nodes may not have all historical chunks (e.g., when started from a snapshot).

Changes:

• New ledger chunk download API with HEAD/GET endpoints supporting byte-range requests
• Serialization of ledger access using mutexes to ensure thread-safe concurrent access
• fsync of committed chunks before closing to ensure file integrity for remote serving
• New ReadLedgerSubsystem interface for accessing ledger metadata
• Comprehensive end-to-end tests covering chunk access, redirection, and gap scenarios

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/schema.py	Added test registration for ledger chunk download
tests/infra/remote.py	Added helper to get main ledger directory
tests/infra/node.py	Added version check for LedgerChunkRead feature and helper method
tests/infra/interfaces.py	Enabled LedgerChunkRead operator feature on file serving interface
tests/e2e_operations.py	Added comprehensive tests for chunk access, redirection scenarios, and gap handling
src/node/rpc/node_frontend.h	Updated API version to 5.0.1
src/node/rpc/ledger_subsystem.h	New subsystem implementation for ledger read operations
src/node/rpc/ledger_interface.h	New interface definition for ledger subsystem
src/node/rpc/file_serving_handlers.h	Implemented ledger chunk endpoints with redirection logic and byte-range support
src/host/test/ledger.cpp	Updated ledger test capture to include file sizes
src/host/run.cpp	Added ledger parameter to enclave creation
src/host/ledger.h	Added state_lock for thread safety, fsync for committed files, new query methods
src/enclave/main.cpp	Updated to pass ledger reference to enclave
src/enclave/entry_points.h	Updated signature to accept ledger reference
src/enclave/enclave.h	Installed ReadLedgerSubsystem in node context
include/ccf/http_consts.h	Added CCF_LEDGER_CHUNK_NAME header constant
doc/schemas/node_openapi.json	Added OpenAPI schema for new endpoints, updated version

[eddyashton]

LGTM!

The consolidation of helper functions in file_serving_handlers.h is great. I think we could go slightly further and factor out the ?since= param lookup, and possibly standardise the pattern for subsystem lookup, but these can be deferred to a future PR. I'm a little suspicious of the concept of init_idx, but it looks like its populated and tested correctly so I'm happy.

[achamayou]

I think we could go slightly further and factor out the ?since= param lookup, and possibly standardise the pattern for subsystem lookup, but these can be deferred to a future PR.

I had a go, but it's mandatory in one case and optional in the other, and so it's a little awkward. I'll try again.

[achamayou]

I'm a little suspicious of the concept of init_idx, but it looks like its populated and tested correctly so I'm happy.

I am also unhappy with that because LedgerFiles clearly has too much state already, and this is more state. But on the other hand, this is the right cutoff, and there is no other obvious way to get it.