bulk updates for performance

[rnewson]

Overview

Nouveau switched from ibrowse to gun (and http/1.1 to http/2) in 3.5.0, in order to reduce the large number of connections made between couchdb and nouveau server. A user has found, on a larger test than I performed during code development, a significant indexing speed regression.

Before the ibrowse to gun transition the update requests to nouveau server used http pipelining (that is, multiple requests were made to the server, in order, without waiting for the responses). This was a significant optimization. With gun this was not possible (as http/2 uses multiplexing instead). The difference turns out much more significant that expected.

This PR adds a new endpoint on the nouveau server that supports bulk update. Each item in the bulk list is a document update or delete request.

This has demonstrated a substantial performance improvement.

The single doc update and delete endpoints will be removed in a future release but will remain for a time for backward compatibility.

Testing recommendations

Will be covered by automated tests

Related Issues or Pull Requests

#5894

Checklist

• This is my own work, I did not use AI, LLM's or similar technology
• Code is written and works correctly
• Changes are covered by tests
• Any new configurable parameters are documented in rel/overlay/etc/default.ini
• Documentation changes were made in the src/docs folder
• Documentation changes were backported (separated PR) to affected branches

[rnewson]

noting that the java side is less cooperative (won't start the response until the request is complete, so can't add flow control), but I am grateful as it has forced me to a tidier solution. I will add a bulk doc update endpoint in the JAX-RS fashion, a domain object that represents a small group of updates to make, and have the erlang side accumulate a batch of docs then issue the POST update and wait for its successful response. We get flow control, we avoid introducing json-seq and I can remove the ugly code in this PR too. Will try to get a version of that pushed to this PR over the weekend. The batch size will be configurable but I should be able to find a decent default as I can induce the problem locally quite easily.

[rnewson]

pushed an update, not quite done yet ("purge with conflicts" test fails atm but the rest pass).

[rnewson]

all tests passing now (was a silly mistake in purge_index)

[rnewson]

built 1 million doc index locally in 1m21s which I think is good