Skip to content

Split CI script into separate GitHub Actions steps #4357

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
joostjager wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Split CI script into separate GitHub Actions steps #4357

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 57 additions & 3 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,9 +88,63 @@ jobs:
run: |
echo "BITCOIND_EXE=$( pwd )/bin/bitcoind-${{ runner.os }}-${{ runner.arch }}" >> "$GITHUB_ENV"
echo "ELECTRS_EXE=$( pwd )/bin/electrs-${{ runner.os }}-${{ runner.arch }}" >> "$GITHUB_ENV"
- name: Run CI script
shell: bash # Default on Winblows is powershell
run: CI_ENV=1 CI_MINIMIZE_DISK_USAGE=1 ./ci/ci-tests.sh
- name: Check workspace (except lightning-transaction-sync)
shell: bash
run: CI_ENV=1 CI_MINIMIZE_DISK_USAGE=1 ./ci/ci-tests.sh check-workspace
- name: Test workspace (except lightning-transaction-sync)
shell: bash
run: CI_ENV=1 CI_MINIMIZE_DISK_USAGE=1 ./ci/ci-tests.sh test-workspace
- name: Test upgrade from prior LDK versions
shell: bash
run: CI_ENV=1 CI_MINIMIZE_DISK_USAGE=1 ./ci/ci-tests.sh test-ldk-upgrade
- name: Check and build docs for workspace members
shell: bash
run: CI_ENV=1 CI_MINIMIZE_DISK_USAGE=1 ./ci/ci-tests.sh check-workspace-members
- name: Test lightning with dnssec feature
shell: bash
run: CI_ENV=1 CI_MINIMIZE_DISK_USAGE=1 ./ci/ci-tests.sh test-lightning-dnssec
- name: Test Block Sync Clients with features
shell: bash
run: CI_ENV=1 CI_MINIMIZE_DISK_USAGE=1 ./ci/ci-tests.sh test-lightning-block-sync
- name: Check Transaction Sync Clients
shell: bash
run: CI_ENV=1 CI_MINIMIZE_DISK_USAGE=1 ./ci/ci-tests.sh check-lightning-transaction-sync
- name: Test Transaction Sync Clients
shell: bash
run: CI_ENV=1 CI_MINIMIZE_DISK_USAGE=1 ./ci/ci-tests.sh test-lightning-transaction-sync
- name: Test lightning-persister with tokio
shell: bash
run: CI_ENV=1 CI_MINIMIZE_DISK_USAGE=1 ./ci/ci-tests.sh test-lightning-persister
- name: Test Custom Message Macros
shell: bash
run: CI_ENV=1 CI_MINIMIZE_DISK_USAGE=1 ./ci/ci-tests.sh test-lightning-custom-message
- name: Test backtrace-debug builds
shell: bash
run: CI_ENV=1 CI_MINIMIZE_DISK_USAGE=1 ./ci/ci-tests.sh test-lightning-backtrace
- name: Test no_std builds
shell: bash
run: CI_ENV=1 CI_MINIMIZE_DISK_USAGE=1 ./ci/ci-tests.sh test-no-std
- name: Test c_bindings builds
shell: bash
run: CI_ENV=1 CI_MINIMIZE_DISK_USAGE=1 ./ci/ci-tests.sh test-c-bindings
- name: Test other crate-specific builds
shell: bash
run: CI_ENV=1 CI_MINIMIZE_DISK_USAGE=1 ./ci/ci-tests.sh test-crate-specific
- name: Check no_std downstream crate
shell: bash
run: CI_ENV=1 CI_MINIMIZE_DISK_USAGE=1 ./ci/ci-tests.sh check-no-std
- name: Check MSRV with release pins only
shell: bash
run: CI_ENV=1 CI_MINIMIZE_DISK_USAGE=1 ./ci/ci-tests.sh check-msrv-no-dev-deps
- name: Build no_std for ARM Embedded
shell: bash
run: CI_ENV=1 CI_MINIMIZE_DISK_USAGE=1 ./ci/ci-tests.sh build-no-std-arm
- name: Test cfg-flag builds
shell: bash
run: CI_ENV=1 CI_MINIMIZE_DISK_USAGE=1 ./ci/ci-tests.sh test-cfg-flags
- name: Verify all CI steps ran
shell: bash
run: ./ci/ci-tests.sh --verify-complete

coverage:
needs: fuzz
Expand Down
101 changes: 99 additions & 2 deletions ci/ci-tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,30 +24,89 @@ PIN_RELEASE_DEPS # pin the release dependencies in our main workspace

export RUST_BACKTRACE=1

echo -e "\n\nChecking the workspace, except lightning-transaction-sync."
cargo check --quiet --color always
# All steps in order, matching the original script flow
ALL_STEPS="
check-workspace
test-workspace
test-ldk-upgrade
check-workspace-members
test-lightning-dnssec
test-lightning-block-sync
check-lightning-transaction-sync
test-lightning-transaction-sync
test-lightning-persister
test-lightning-custom-message
test-lightning-backtrace
test-no-std
test-c-bindings
test-crate-specific
check-no-std
check-msrv-no-dev-deps
build-no-std-arm
test-cfg-flags
"

# If a step name is passed, run just that step. Otherwise run all.
if [ -n "$1" ]; then
STEPS_TO_RUN="$1"
else
STEPS_TO_RUN="$ALL_STEPS"
fi

WORKSPACE_MEMBERS=( $(cat Cargo.toml | tr '\n' '\r' | sed 's/\r //g' | tr '\r' '\n' | grep '^members =' | sed 's/members.*=.*\[//' | tr -d '"' | tr ',' ' ') )

# Verify that all steps were executed (called at the end of CI)
if [ "$1" = "--verify-complete" ]; then
MISSING_STEPS=""
for STEP in $ALL_STEPS; do
if [[ ! " $CI_COMPLETED_STEPS " == *" $STEP "* ]]; then
MISSING_STEPS="$MISSING_STEPS $STEP"
fi
done
if [ -n "$MISSING_STEPS" ]; then
echo "ERROR: The following CI steps were not executed:$MISSING_STEPS"
exit 1
fi
echo "All CI steps were executed successfully."
exit 0
fi

for STEP in $STEPS_TO_RUN; do
case "$STEP" in

check-workspace)
echo -e "\n\nChecking the workspace, except lightning-transaction-sync."
cargo check --quiet --color always
;;

test-workspace)
echo -e "\n\nTesting the workspace, except lightning-transaction-sync."
cargo test --quiet --color always
;;

test-ldk-upgrade)
echo -e "\n\nTesting upgrade from prior versions of LDK"
pushd lightning-tests
cargo test --quiet
popd
;;

check-workspace-members)
echo -e "\n\nChecking and building docs for all workspace members individually..."
for DIR in "${WORKSPACE_MEMBERS[@]}"; do
cargo check -p "$DIR" --quiet --color always
cargo doc -p "$DIR" --quiet --document-private-items
done
;;

test-lightning-dnssec)
echo -e "\n\nChecking and testing lightning with features"
cargo test -p lightning --quiet --color always --features dnssec
cargo check -p lightning --quiet --color always --features dnssec
cargo doc -p lightning --quiet --document-private-items --features dnssec
;;

test-lightning-block-sync)
echo -e "\n\nChecking and testing Block Sync Clients with features"

cargo test -p lightning-block-sync --quiet --color always --features rest-client
Expand All @@ -58,13 +117,17 @@ cargo test -p lightning-block-sync --quiet --color always --features rpc-client,
cargo check -p lightning-block-sync --quiet --color always --features rpc-client,rest-client
cargo test -p lightning-block-sync --quiet --color always --features rpc-client,rest-client,tokio
cargo check -p lightning-block-sync --quiet --color always --features rpc-client,rest-client,tokio
;;

check-lightning-transaction-sync)
echo -e "\n\nChecking Transaction Sync Clients with features."
cargo check -p lightning-transaction-sync --quiet --color always --features esplora-blocking
cargo check -p lightning-transaction-sync --quiet --color always --features esplora-async
cargo check -p lightning-transaction-sync --quiet --color always --features esplora-async-https
cargo check -p lightning-transaction-sync --quiet --color always --features electrum
;;

test-lightning-transaction-sync)
if [ -z "$CI_ENV" ] && [[ -z "$BITCOIND_EXE" || -z "$ELECTRS_EXE" ]]; then
echo -e "\n\nSkipping testing Transaction Sync Clients due to BITCOIND_EXE or ELECTRS_EXE being unset."
cargo check -p lightning-transaction-sync --tests
Expand All @@ -75,27 +138,37 @@ else
cargo test -p lightning-transaction-sync --quiet --color always --features esplora-async-https
cargo test -p lightning-transaction-sync --quiet --color always --features electrum
fi
;;

test-lightning-persister)
echo -e "\n\nChecking and testing lightning-persister with features"
cargo test -p lightning-persister --quiet --color always --features tokio
cargo check -p lightning-persister --quiet --color always --features tokio
cargo doc -p lightning-persister --quiet --document-private-items --features tokio
;;

test-lightning-custom-message)
echo -e "\n\nTest Custom Message Macros"
cargo test -p lightning-custom-message --quiet --color always
[ "$CI_MINIMIZE_DISK_USAGE" != "" ] && cargo clean
;;

test-lightning-backtrace)
echo -e "\n\nTest backtrace-debug builds"
cargo test -p lightning --quiet --color always --features backtrace
;;

test-no-std)
echo -e "\n\nTesting no_std builds"
for DIR in lightning-invoice lightning-rapid-gossip-sync lightning-liquidity; do
cargo test -p $DIR --quiet --color always --no-default-features
done

cargo test -p lightning --quiet --color always --no-default-features
cargo test -p lightning-background-processor --quiet --color always --no-default-features
;;

test-c-bindings)
echo -e "\n\nTesting c_bindings builds"
# Note that because `$RUSTFLAGS` is not passed through to doctest builds we cannot selectively
# disable doctests in `c_bindings` so we skip doctests entirely here.
Expand All @@ -110,35 +183,45 @@ done
# disable doctests in `c_bindings` so we skip doctests entirely here.
RUSTFLAGS="$RUSTFLAGS --cfg=c_bindings" cargo test -p lightning-background-processor --quiet --color always --no-default-features --lib --bins --tests
RUSTFLAGS="$RUSTFLAGS --cfg=c_bindings" cargo test -p lightning --quiet --color always --no-default-features --lib --bins --tests
;;

test-crate-specific)
echo -e "\n\nTesting other crate-specific builds"
# Note that outbound_commitment_test only runs in this mode because of hardcoded signature values
RUSTFLAGS="$RUSTFLAGS --cfg=ldk_test_vectors" cargo test -p lightning --quiet --color always --no-default-features --features=std
# This one only works for lightning-invoice
# check that compile with no_std and serde works in lightning-invoice
cargo test -p lightning-invoice --quiet --color always --no-default-features --features serde
;;

check-no-std)
echo -e "\n\nTesting no_std build on a downstream no-std crate"
# check no-std compatibility across dependencies
pushd no-std-check
cargo check --quiet --color always
[ "$CI_MINIMIZE_DISK_USAGE" != "" ] && cargo clean
popd
;;

check-msrv-no-dev-deps)
# Test that we can build downstream code with only the "release pins".
pushd msrv-no-dev-deps-check
PIN_RELEASE_DEPS
cargo check --quiet
[ "$CI_MINIMIZE_DISK_USAGE" != "" ] && cargo clean
popd
;;

build-no-std-arm)
if [ -f "$(which arm-none-eabi-gcc)" ]; then
pushd no-std-check
cargo build --quiet --target=thumbv7m-none-eabi
[ "$CI_MINIMIZE_DISK_USAGE" != "" ] && cargo clean
popd
fi
;;

test-cfg-flags)
echo -e "\n\nTest cfg-flag builds"
RUSTFLAGS="--cfg=taproot" cargo test --quiet --color always -p lightning
[ "$CI_MINIMIZE_DISK_USAGE" != "" ] && cargo clean
Expand All @@ -147,3 +230,17 @@ RUSTFLAGS="--cfg=simple_close" cargo test --quiet --color always -p lightning
RUSTFLAGS="--cfg=lsps1_service" cargo test --quiet --color always -p lightning-liquidity
[ "$CI_MINIMIZE_DISK_USAGE" != "" ] && cargo clean
RUSTFLAGS="--cfg=peer_storage" cargo test --quiet --color always -p lightning
;;

*)
echo "Unknown step: $STEP"
exit 1
;;

esac

# Log the completed step to GITHUB_ENV for the verification step
if [ -n "$GITHUB_ENV" ]; then
echo "CI_COMPLETED_STEPS=${CI_COMPLETED_STEPS:-} $STEP" >> "$GITHUB_ENV"
Copy link
Contributor

@tnull tnull Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: It's a bit weird to modify the external env variable here.

Copy link
Contributor Author

@joostjager joostjager Feb 5, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you have a suggestion how to do it otherwise? We need to write the modified CI_COMPLETED_STEPS var to the github file in order for it to carry over to the next step, and ultimately to the verification step that checks that we didn't forget anything.

fi
done
Loading