Gossmap: compaction support#8869
Merged
rustyrussell merged 1 commit intoElementsProject:masterfrom Feb 9, 2026
Merged
Conversation
rustyrussell
force-pushed
the
gossmap-compact
branch
7 times, most recently
from
e967d40 to
2639a79
Compare
rustyrussell
force-pushed
the
gossmap-compact
branch
6 times, most recently
from
9024e31 to
cc2fade
Compare
Collaborator
|
I have tested cc2fade and i get a crash with this test: def test_gossip(node_factory, bitcoind):
l1, l2, l3, l4 = node_factory.get_nodes(
4,
opts=[
{},
{},
{},
{},
],
)
nodes = [l1, l2, l3, l4]
l1.fundwallet(10_000_000)
l2.fundwallet(10_000_000)
l3.fundwallet(10_000_000)
l4.fundwallet(10_000_000)
l2.rpc.fundchannel(
l4.info["id"] + "@localhost:" + str(l4.port),
1_000_000,
)
l3.rpc.fundchannel(
l4.info["id"] + "@localhost:" + str(l4.port),
1_000_000,
)
l4.rpc.fundchannel(
l1.info["id"] + "@localhost:" + str(l1.port),
1_000_000,
)
bitcoind.generate_block(1)
sync_blockheight(bitcoind, nodes)
l4.rpc.fundchannel(
l1.info["id"] + "@localhost:" + str(l1.port),
1_000_000,
)
bitcoind.generate_block(6)
sync_blockheight(bitcoind, nodes)
wait_for(lambda: len(l1.rpc.listpeerchannels()["channels"]) == 2)
l4_chans = l1.rpc.listpeerchannels(l4.info["id"])["channels"]
scid_l1_l4_1 = l4_chans[0]["short_channel_id"]
scid_l2_l4 = l2.rpc.listpeerchannels(l4.info["id"])["channels"][0][
"short_channel_id"
]
wait_for(lambda: len(l1.rpc.listchannels()["channels"]) == 8)
l1.rpc.close(scid_l1_l4_1)
bitcoind.generate_block(6, wait_for_mempool=1)
sync_blockheight(bitcoind, nodes)
# wait_for(lambda: len(l1.rpc.listchannels()["channels"]) == 6)
l1.restart()
l1.rpc.connect(l2.info["id"] + "@localhost:" + str(l2.port))
l1.rpc.connect(l3.info["id"] + "@localhost:" + str(l3.port))
l1.rpc.connect(l4.info["id"] + "@localhost:" + str(l4.port))
# wait_for(lambda: len(l1.rpc.listchannels()["channels"]) == 6)
l2.rpc.close(scid_l2_l4)
bitcoind.generate_block(1, wait_for_mempool=1)
sync_blockheight(bitcoind, nodes)
# wait_for(lambda: len(l1.rpc.listchannels()["channels"]) == 4)
l1.rpc.call("dev-compact-gossip-store", [])
# wait_for(lambda: len(l1.rpc.listpeerchannels()["channels"]) == 2)
# wait_for(lambda: len(l1.rpc.listchannels()["channels"]) == 4)
l4.rpc.fundchannel(
l1.info["id"] + "@localhost:" + str(l1.port),
1_000_000,
)
bitcoind.generate_block(6)
sync_blockheight(bitcoind, nodes)
wait_for(lambda: len(l1.rpc.listchannels()["channels"]) == 6)It's a test from one of my plugins where i removed all my plugin specific lines. Also i would really like it if the commented |
Contributor
Author
Nice catch! It's because I didn't correctly reset the "dying_chans" array. Fixed now, and added a reduced version of this test to be sure. The reason your wait_for lines didn't work is that we don't immediately consider a channel closed: we give it 12 blocks before stopping its use. That allows us to keep using the channel if it turns out to be a splice. We make an exception for our own channels (in that case, we will know about the splice). |
rustyrussell
force-pushed
the
gossmap-compact
branch
2 times, most recently
from
dd4c02f to
4b347dc
Compare
In November 2022 we seemed to increase parallelism from 2 and 3 to 10! That is a huge load for these CI boxes, and does explain some of our flakes. We only run in parallel because some tests sleep, but it's diminishing returns (GH runners have 4 VCPUs, 16GB RAM). This reduces it so: - Normal runs are -n 4 - Valgrind runs are -n 2 - Sanitizer runs are -n 3 If I use my beefy build box (64BG RAM) but reduce it to 4 CPUs: Time for pytest -n 5: Time for pytest -n 4: Time for pytest -n 3: Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell
force-pushed
the
gossmap-compact
branch
from
4b347dc to
7645019
Compare
Collaborator
|
What happened here? Where is the gossip compaction code? I can't find it on master. |
Contributor
Author
|
Something went badly wrong, I just noticed the same thing! |
Contributor
Author
|
I will open a new PR. |
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Based on #8878-- MergedWe used to do compaction of the gossip store, but it was buggy. So now we do it on every startup (which is slow). Better is to do it when required, using a separate process.