perf: reduce relay traffic for holepunches in the multipath world

[matheus23]

Anecdotally (I don't remember the exact numbers right now), iroh 0.35 holepunched with ~3-4kB of relay traffic under normal conditions (let's say, 40-50ms relay RTT, somewhere between 10-100ms RTT between the endpoints that want to holepunch).

In some dogfooding tests recently, the new multipath- and quic-nat-traversal-based hole-punching iroh sends more data via the relay:

Connected to 49975e383823c4ae09ae196fafa45b9e24f9e2f2d8b1312f5c2a2dfee2eb655c
Sent: "be12a8cde3 is saying hello!"
[49975e3838] Connection type changed to: Relay(RelayUrl("https://staging-euw1-1.relay.iroh.network./")) (RTT: 189.01663ms)
[49975e3838] Connection type changed to: Ip(176.199.209.244:26743) (RTT: 49.69848ms)
Received 10.00 MiB in 8.8536s (1.13 MiB/s, time to first byte 0.063306306s, 3681 chunks) (Shutdown took 0.0000s)
Path stats:
  Relay(RelayUrl("https://staging-euw1-1.relay.iroh.network./")): RTT 132.205998ms, tx=10385, rx=35209
  Ip(176.199.209.244:26743): RTT 96.374115ms, tx=96839, rx=10707656

In this instance, even though most of the traffic was sent via the hole-punched path, 10KB were still sent via the relay, and 35KB were received.

We can and should strive to reduce those numbers.

[matheus23]

I didn't investigate this yet at all, but I just want to write down some thoughts on what this might be/possible improvement areas:

• We're sending tons of NEW_PATH_CONNECTION_ID frames initially, potentially sending another couple of datagrams via the relay that we might not all need to holepunch, instead we should only send as many CIDs as we need at each point in time. We've discussed this idea before.
• We only start holepunching once the connection is established, because only then we exchange addresses via ADD_ADDRESS and REACH_OUT frames (after multipath & QNT are negotiated). We could consider creating QUIC extensions that allow us to put equivalent data into transport parameters.
• For the server to start sending via a hole-punched path, it has to:
  1. Receive a REACH_OUT frame
  2. Send random data to that address
  3. Wait for the client's PATH_CHALLENGE to arrive
  4. Respond with a PATH_RESPONSE and send a PATH_CHALLENGE of its own
  5. Wait for the client's PATH_RESPONSE to arrive.
  6. Switch to that path.
     Due to sending random data instead of a path challenge immediately, the path takes a whole round-trip longer to be validated after the client side's firewall is opened than if we sent a path challenge from the server immediately. This should be fixed by use PATH_CHALLENGE for hole punching quinn#231

Pretty likely there are other things I didn't think of yet.

I think once we've investigated and identified what the issues are, we should probably close this in favor of the actual sub-issues, or make the actual issues sub-issues of this issue.

[flub]

Great summary so far!

We only start holepunching once the connection is established, because only then we exchange addresses via ADD_ADDRESS and REACH_OUT frames (after multipath & QNT are negotiated). We could consider creating QUIC extensions that allow us to put equivalent data into transport parameters.

I've looked at this before as well, and there's nothing to be gained here as this is already happening in the fewest roundtrips: i.e. in 0.5 RTT.

The When the server has processed the client's Initial packet with it's transport parameters it already knows all the negotiated transport parameters. So it also knows that QNT is negotiated at that time. It then immediately creates the Handshake and Data space keys. This means it can already send 1-RTT packets in the Data space. And indeed it does so: it responds sending to the client at least 3 packets:

1. Initial for the handshake
2. Handshake for the handshake
3. Data for CIDs and a few other things, this could be more than 1 packet/datagram.

Note that not everything can be sent in the data space at this time. E.g. application data still has to wait until the handshake is confirmed.

But also note that unfortunately there is still room for improvement. 1 & 2 do get coalesced into the same datagram but because of implementation reasons 3 does not. Especially the CIDs get sent in a separate datagram which is sent a little later (but before a response from the client).

We can at least make sure that ADD_ADDRESS frames end up coalesced into the first datagram. And with a bit more work we should also be able to make sure that we can start sending CIDs in the coalesced datagram.

[matheus23]

Ah this is super helpful.
Compared to what we had before, this still leaves something on the table: Only clients can start holepunching, but the client is the last one to know all of the addresses to do so.
Should perhaps the server start holepunching already when it receives the ADD_ADDRESS frames? As opposed to only after the client receives the addresses and sends REACH_OUT frames.
This would obviously require changes to the QUIC holepunching protocol. But as far as I see, it could be achieved by simply switching the roles of who is the one sending REACH_OUTs.

[flub]

Ah this is super helpful. Compared to what we had before, this still leaves something on the table: Only clients can start holepunching, but the client is the last one to know all of the addresses to do so. Should perhaps the server start holepunching already when it receives the ADD_ADDRESS frames? As opposed to only after the client receives the addresses and sends REACH_OUT frames. This would obviously require changes to the QUIC holepunching protocol. But as far as I see, it could be achieved by simply switching the roles of who is the one sending REACH_OUTs.

The server never receives ADD_ADDRESS frames in QNT. The reason is that in multipath only the client open new paths so QNT also gives all control to the client and all the info is being collected at the client side to do.

I think you're suggesting something rather differently then the current design. The way I can make your suggestions works is by something that does:

1. Client sends holepunching candidates in the transport parameters.
2. Server immediately sends PATH_CHALLENGE to all these to start holepunching.
3. Server sends REACH_OUT (or equivalent) to client together in the first datagram(s) it sends (0.5 RTT)
4. Client sends PATH_CHALLENGEs on path_ids to open the paths

It's interesting, but very different from the current impl. 2 is a difficult since you do not have any unused CIDs. You'd have to use the client's initial CID and then all paths are linkable.

And also the same linkability issue exists with 1 because the client's transport parameters are not encrypted.

[flub]

See also n0-computer/quinn#283