currencyrate: new rust plugin to provide the currencyconvert API

[daywalker90]

Fixes #8825

I described my approach for doing API calls in the issue but i repeat it here:
I added 7 default price sources of which one (bitstamp) never seems to work (HTTP 403) when using a tor proxy. Out of these 6-7 CONVERT_SOURCES_COUNT (3) are chosen at random when currencyconvert is called, unless we are inside the cache time window.

I like this approach because:

• always fresh data, maximum staleness is CACHE_DURATIONS_SECS (10)
• median of CONVERT_SOURCES_COUNT (3) sources aslong as enough are configured and return a valid result
• not hitting the API's rapidly within the cache time window CACHE_DURATIONS_SECS (10)
• not hitting all the API's every time, assuming you have more than CONVERT_SOURCES_COUNT (3)

What i don't like about it:

• outside of the cache time window the worst case response time is SOURCE_TIMEOUT_SECS (5) * n (where n is >1 if any of the initial 3 fail but is at most the number of sources configured - 2) and i expect this to be called right before doing a payment, delaying it by up to that much. If it's only used for recurring payments i would assume response time is not that critical, so i found this acceptable

To always instantly have a response, we would either have to look for API endpoints that return all currency pairs and cache them or ask the user what currency they are interested in or use historic user queries to know wich currency to track.

I also tried my best to be compatible with the python version of currencyrate, meaning i left the options and rpc methods the same, except for the msat suffix. But i can see if you want to break this since the option names inside CLN are a little under-descriptive (add-source, disable-source). I also added support for positional members when using add-source (kraken needs it).

Please let me know:

• are you happy with the API call approach or do you prefer a different one, if so please let me know what you value more response time, data age, API rate limit etc.

• where and how you want this documented

• should i mark the tests as flaky since they make real-live API calls and they can easily fail if one of them doesn't return a valid result

• The changelog has been updated in the relevant commit(s) according to the guidelines.

• Tests have been added or modified to reflect the changes.

• Documentation has been reviewed and updated as needed.

• Related issues have been listed and linked, including any that this PR closes.

• Important All PRs must consider how to reverse any persistent changes for tools/lightning-downgrade

[daywalker90]

Fixed the check-manpage error and disabled the plugin in test_pay.py, as it assumed we didn't have such a plugin.

[madelinevibes]

@daywalker90 I've had various conversations in Blockstream and confirmed we don't have a Blockstream-API for prices

[rustyrussell]

This is awesome!

I've rebased, and I'm making the following trivial changes:

1. Options now called currencyrate-add-source / currencyrate-disable-source (as you suggested)
2. Renaming the plugin go cln-currencyrate. I prefer this namign for new plugins, as they give you a clue when you see them in top or whatever.

[rustyrussell]

OK, I love this, but I am going to ask for more changes :)

I plan on using this inside bookkeeper, aggressively (when configured). So I think we need to buff up the algorithm. Here's my thinking:

On request, we should return "immediately" if we have fresh enough results (say, < 1 hour, and algo below is not fetching more), and start refresh in the background.

I think we should refresh lazily: if they're all over 1 hour old, pick one to refresh. If it has changed from its last value significantly, refresh others. Handwave over how this would work! But we should try for polling all the sources, even if we only do it very occasionally. If a source fails, don't randomly select it for some time (unless they all fail!).

[daywalker90]

Thank you!

The algorithm is now like this:

• On first request per currency fetch from all sources (in parallel) and start a background task
• background task fetches from sources, which are ordered from oldest to freshest
• background task fetches from 1 source (the oldest) per hour unless the price changed more than 1% then it keeps fetching until 2 sources agree on a price
• background task exits if there were no requests for that currency in 3 hours

This enables currencyconvert to immediately return on subsequent requests, but the price could be an hour old.

currencyrates fetches from all sources where the cached price is older than an hour.

There is a backoff function for sources and if all sources are backed off of we just try all of them.