Worker network stream optimizations #283

gabotechs · 2026-01-09T08:21:52Z

Previously, when two workers needed communication, one gRPC stream per partition was used for moving per-partition data from one worker to the other, something like:

  ┌─────────────────────────┐ 
  │        Worker A         │ 
  └──┬───┬───┬───┬───┬───┬──┘ 
     │   │   │   │   │   │    
    0│  1│  2│  3│  4│  5│    
     │   │   │   │   │   │    
     │   │   │   │   │   │    
  ┌──▼───▼───▼───▼───▼───▼──┐ 
  │        Worker B         │ 
  └─────────────────────────┘

At first sight, one could think that it's problematic to open so many streams, but fortunately the way gRPC and Tonic work is that as long as we reuse the same Channel (which we do), Tonic will reuse the same underlaying TCP connection, interleaving HTTP2 frames in a very efficient manner.

There do is a problem though... because of how the project works, we do have some extra overhead in opening multiple gRPC streams that Tonic is not going to save us from:

For each gRPC stream, we need to send the serialized plan over the wire. For example, for a UNION with hundreds of children, where each children has tens of partitions, we are sending a serialized plan with hundreds of children over the wire hundreds * tens times (one per partition).

This PR reworks the communication between workers by introducing some new WorkerConnectionPool and WorkerConnection structs that automatically handle gRPC stream batching, which results in:

Cleaner, less duplicated code, as a lot of code is now reused in the different network boundaries
More efficient communication, avoiding the overhead of opening multiple gRPC streams

┌─────────────────────────┐
│        Worker A         │
└────────────┬────────────┘
             │             
        0,1,2,3,4,5        
             │             
             │             
┌────────────▼────────────┐
│        Worker B         │
└─────────────────────────┘

The results of executing the following query over TPCH SF1 data on the remote cluster are a ~1.5x speedup:

UNION of multiple children

-- Custom TPC-H Query: Massive UNION across 300 subqueries
-- This query performs a UNION ALL operation with 300 separate scans of the lineitem table
-- Each subquery filters on a different l_orderkey range to simulate 300 different partitions

SELECT 1 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 0
UNION ALL
SELECT 2 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 1
UNION ALL
SELECT 3 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 2
UNION ALL
SELECT 4 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 3
UNION ALL
SELECT 5 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 4
UNION ALL
SELECT 6 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 5
UNION ALL
SELECT 7 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 6
UNION ALL
SELECT 8 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 7
UNION ALL
SELECT 9 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 8
UNION ALL
SELECT 10 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 9
UNION ALL
SELECT 11 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 10
UNION ALL
SELECT 12 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 11
UNION ALL
SELECT 13 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 12
UNION ALL
SELECT 14 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 13
UNION ALL
SELECT 15 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 14
UNION ALL
SELECT 16 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 15
UNION ALL
SELECT 17 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 16
UNION ALL
SELECT 18 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 17
UNION ALL
SELECT 19 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 18
UNION ALL
SELECT 20 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 19
UNION ALL
SELECT 21 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 20
UNION ALL
SELECT 22 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 21
UNION ALL
SELECT 23 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 22
UNION ALL
SELECT 24 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 23
UNION ALL
SELECT 25 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 24
UNION ALL
SELECT 26 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 25
UNION ALL
SELECT 27 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 26
UNION ALL
SELECT 28 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 27
UNION ALL
SELECT 29 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 28
UNION ALL
SELECT 30 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 29
UNION ALL
SELECT 31 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 30
UNION ALL
SELECT 32 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 31
UNION ALL
SELECT 33 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 32
UNION ALL
SELECT 34 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 33
UNION ALL
SELECT 35 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 34
UNION ALL
SELECT 36 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 35
UNION ALL
SELECT 37 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 36
UNION ALL
SELECT 38 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 37
UNION ALL
SELECT 39 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 38
UNION ALL
SELECT 40 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 39
UNION ALL
SELECT 41 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 40
UNION ALL
SELECT 42 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 41
UNION ALL
SELECT 43 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 42
UNION ALL
SELECT 44 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 43
UNION ALL
SELECT 45 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 44
UNION ALL
SELECT 46 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 45
UNION ALL
SELECT 47 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 46
UNION ALL
SELECT 48 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 47
UNION ALL
SELECT 49 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 48
UNION ALL
SELECT 50 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 49
UNION ALL
SELECT 51 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 50
UNION ALL
SELECT 52 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 51
UNION ALL
SELECT 53 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 52
UNION ALL
SELECT 54 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 53
UNION ALL
SELECT 55 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 54
UNION ALL
SELECT 56 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 55
UNION ALL
SELECT 57 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 56
UNION ALL
SELECT 58 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 57
UNION ALL
SELECT 59 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 58
UNION ALL
SELECT 60 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 59
UNION ALL
SELECT 61 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 60
UNION ALL
SELECT 62 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 61
UNION ALL
SELECT 63 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 62
UNION ALL
SELECT 64 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 63
UNION ALL
SELECT 65 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 64
UNION ALL
SELECT 66 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 65
UNION ALL
SELECT 67 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 66
UNION ALL
SELECT 68 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 67
UNION ALL
SELECT 69 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 68
UNION ALL
SELECT 70 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 69
UNION ALL
SELECT 71 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 70
UNION ALL
SELECT 72 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 71
UNION ALL
SELECT 73 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 72
UNION ALL
SELECT 74 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 73
UNION ALL
SELECT 75 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 74
UNION ALL
SELECT 76 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 75
UNION ALL
SELECT 77 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 76
UNION ALL
SELECT 78 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 77
UNION ALL
SELECT 79 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 78
UNION ALL
SELECT 80 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 79
UNION ALL
SELECT 81 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 80
UNION ALL
SELECT 82 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 81
UNION ALL
SELECT 83 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 82
UNION ALL
SELECT 84 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 83
UNION ALL
SELECT 85 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 84
UNION ALL
SELECT 86 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 85
UNION ALL
SELECT 87 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 86
UNION ALL
SELECT 88 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 87
UNION ALL
SELECT 89 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 88
UNION ALL
SELECT 90 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 89
UNION ALL
SELECT 91 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 90
UNION ALL
SELECT 92 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 91
UNION ALL
SELECT 93 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 92
UNION ALL
SELECT 94 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 93
UNION ALL
SELECT 95 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 94
UNION ALL
SELECT 96 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 95
UNION ALL
SELECT 97 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 96
UNION ALL
SELECT 98 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 97
UNION ALL
SELECT 99 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 98
UNION ALL
SELECT 100 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 99
UNION ALL
SELECT 101 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 100
UNION ALL
SELECT 102 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 101
UNION ALL
SELECT 103 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 102
UNION ALL
SELECT 104 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 103
UNION ALL
SELECT 105 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 104
UNION ALL
SELECT 106 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 105
UNION ALL
SELECT 107 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 106
UNION ALL
SELECT 108 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 107
UNION ALL
SELECT 109 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 108
UNION ALL
SELECT 110 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 109
UNION ALL
SELECT 111 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 110
UNION ALL
SELECT 112 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 111
UNION ALL
SELECT 113 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 112
UNION ALL
SELECT 114 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 113
UNION ALL
SELECT 115 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 114
UNION ALL
SELECT 116 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 115
UNION ALL
SELECT 117 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 116
UNION ALL
SELECT 118 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 117
UNION ALL
SELECT 119 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 118
UNION ALL
SELECT 120 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 119
UNION ALL
SELECT 121 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 120
UNION ALL
SELECT 122 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 121
UNION ALL
SELECT 123 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 122
UNION ALL
SELECT 124 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 123
UNION ALL
SELECT 125 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 124
UNION ALL
SELECT 126 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 125
UNION ALL
SELECT 127 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 126
UNION ALL
SELECT 128 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 127
UNION ALL
SELECT 129 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 128
UNION ALL
SELECT 130 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 129
UNION ALL
SELECT 131 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 130
UNION ALL
SELECT 132 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 131
UNION ALL
SELECT 133 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 132
UNION ALL
SELECT 134 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 133
UNION ALL
SELECT 135 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 134
UNION ALL
SELECT 136 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 135
UNION ALL
SELECT 137 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 136
UNION ALL
SELECT 138 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 137
UNION ALL
SELECT 139 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 138
UNION ALL
SELECT 140 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 139
UNION ALL
SELECT 141 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 140
UNION ALL
SELECT 142 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 141
UNION ALL
SELECT 143 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 142
UNION ALL
SELECT 144 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 143
UNION ALL
SELECT 145 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 144
UNION ALL
SELECT 146 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 145
UNION ALL
SELECT 147 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 146
UNION ALL
SELECT 148 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 147
UNION ALL
SELECT 149 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 148
UNION ALL
SELECT 150 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 149
UNION ALL
SELECT 151 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 150
UNION ALL
SELECT 152 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 151
UNION ALL
SELECT 153 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 152
UNION ALL
SELECT 154 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 153
UNION ALL
SELECT 155 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 154
UNION ALL
SELECT 156 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 155
UNION ALL
SELECT 157 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 156
UNION ALL
SELECT 158 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 157
UNION ALL
SELECT 159 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 158
UNION ALL
SELECT 160 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 159
UNION ALL
SELECT 161 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 160
UNION ALL
SELECT 162 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 161
UNION ALL
SELECT 163 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 162
UNION ALL
SELECT 164 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 163
UNION ALL
SELECT 165 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 164
UNION ALL
SELECT 166 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 165
UNION ALL
SELECT 167 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 166
UNION ALL
SELECT 168 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 167
UNION ALL
SELECT 169 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 168
UNION ALL
SELECT 170 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 169
UNION ALL
SELECT 171 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 170
UNION ALL
SELECT 172 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 171
UNION ALL
SELECT 173 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 172
UNION ALL
SELECT 174 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 173
UNION ALL
SELECT 175 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 174
UNION ALL
SELECT 176 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 175
UNION ALL
SELECT 177 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 176
UNION ALL
SELECT 178 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 177
UNION ALL
SELECT 179 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 178
UNION ALL
SELECT 180 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 179
UNION ALL
SELECT 181 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 180
UNION ALL
SELECT 182 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 181
UNION ALL
SELECT 183 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 182
UNION ALL
SELECT 184 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 183
UNION ALL
SELECT 185 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 184
UNION ALL
SELECT 186 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 185
UNION ALL
SELECT 187 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 186
UNION ALL
SELECT 188 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 187
UNION ALL
SELECT 189 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 188
UNION ALL
SELECT 190 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 189
UNION ALL
SELECT 191 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 190
UNION ALL
SELECT 192 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 191
UNION ALL
SELECT 193 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 192
UNION ALL
SELECT 194 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 193
UNION ALL
SELECT 195 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 194
UNION ALL
SELECT 196 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 195
UNION ALL
SELECT 197 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 196
UNION ALL
SELECT 198 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 197
UNION ALL
SELECT 199 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 198
UNION ALL
SELECT 200 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 199
UNION ALL
SELECT 201 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 200
UNION ALL
SELECT 202 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 201
UNION ALL
SELECT 203 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 202
UNION ALL
SELECT 204 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 203
UNION ALL
SELECT 205 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 204
UNION ALL
SELECT 206 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 205
UNION ALL
SELECT 207 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 206
UNION ALL
SELECT 208 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 207
UNION ALL
SELECT 209 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 208
UNION ALL
SELECT 210 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 209
UNION ALL
SELECT 211 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 210
UNION ALL
SELECT 212 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 211
UNION ALL
SELECT 213 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 212
UNION ALL
SELECT 214 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 213
UNION ALL
SELECT 215 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 214
UNION ALL
SELECT 216 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 215
UNION ALL
SELECT 217 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 216
UNION ALL
SELECT 218 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 217
UNION ALL
SELECT 219 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 218
UNION ALL
SELECT 220 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 219
UNION ALL
SELECT 221 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 220
UNION ALL
SELECT 222 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 221
UNION ALL
SELECT 223 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 222
UNION ALL
SELECT 224 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 223
UNION ALL
SELECT 225 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 224
UNION ALL
SELECT 226 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 225
UNION ALL
SELECT 227 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 226
UNION ALL
SELECT 228 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 227
UNION ALL
SELECT 229 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 228
UNION ALL
SELECT 230 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 229
UNION ALL
SELECT 231 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 230
UNION ALL
SELECT 232 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 231
UNION ALL
SELECT 233 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 232
UNION ALL
SELECT 234 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 233
UNION ALL
SELECT 235 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 234
UNION ALL
SELECT 236 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 235
UNION ALL
SELECT 237 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 236
UNION ALL
SELECT 238 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 237
UNION ALL
SELECT 239 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 238
UNION ALL
SELECT 240 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 239
UNION ALL
SELECT 241 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 240
UNION ALL
SELECT 242 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 241
UNION ALL
SELECT 243 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 242
UNION ALL
SELECT 244 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 243
UNION ALL
SELECT 245 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 244
UNION ALL
SELECT 246 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 245
UNION ALL
SELECT 247 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 246
UNION ALL
SELECT 248 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 247
UNION ALL
SELECT 249 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 248
UNION ALL
SELECT 250 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 249
UNION ALL
SELECT 251 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 250
UNION ALL
SELECT 252 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 251
UNION ALL
SELECT 253 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 252
UNION ALL
SELECT 254 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 253
UNION ALL
SELECT 255 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 254
UNION ALL
SELECT 256 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 255
UNION ALL
SELECT 257 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 256
UNION ALL
SELECT 258 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 257
UNION ALL
SELECT 259 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 258
UNION ALL
SELECT 260 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 259
UNION ALL
SELECT 261 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 260
UNION ALL
SELECT 262 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 261
UNION ALL
SELECT 263 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 262
UNION ALL
SELECT 264 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 263
UNION ALL
SELECT 265 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 264
UNION ALL
SELECT 266 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 265
UNION ALL
SELECT 267 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 266
UNION ALL
SELECT 268 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 267
UNION ALL
SELECT 269 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 268
UNION ALL
SELECT 270 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 269
UNION ALL
SELECT 271 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 270
UNION ALL
SELECT 272 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 271
UNION ALL
SELECT 273 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 272
UNION ALL
SELECT 274 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 273
UNION ALL
SELECT 275 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 274
UNION ALL
SELECT 276 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 275
UNION ALL
SELECT 277 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 276
UNION ALL
SELECT 278 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 277
UNION ALL
SELECT 279 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 278
UNION ALL
SELECT 280 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 279
UNION ALL
SELECT 281 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 280
UNION ALL
SELECT 282 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 281
UNION ALL
SELECT 283 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 282
UNION ALL
SELECT 284 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 283
UNION ALL
SELECT 285 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 284
UNION ALL
SELECT 286 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 285
UNION ALL
SELECT 287 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 286
UNION ALL
SELECT 288 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 287
UNION ALL
SELECT 289 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 288
UNION ALL
SELECT 290 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 289
UNION ALL
SELECT 291 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 290
UNION ALL
SELECT 292 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 291
UNION ALL
SELECT 293 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 292
UNION ALL
SELECT 294 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 293
UNION ALL
SELECT 295 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 294
UNION ALL
SELECT 296 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 295
UNION ALL
SELECT 297 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 296
UNION ALL
SELECT 298 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 297
UNION ALL
SELECT 299 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 298
UNION ALL
SELECT 300 as partition_id, count(*) as row_count FROM lineitem WHERE l_orderkey % 300 = 299
ORDER BY partition_id;

The result of running any other benchmark is pretty much the same as before, as the size of the data in the benchmarks completely dominates the size of the plans themselves

==== Comparison with previous run ====
      q1: prev= 703 ms, new= 549 ms, 1.28x faster ✅
 custom1: prev=10221 ms, new=7642 ms, 1.34x faster ✅
      q2: prev= 512 ms, new= 546 ms, 1.07x slower ✖
      q3: prev= 773 ms, new= 810 ms, 1.05x slower ✖
      q4: prev= 514 ms, new= 477 ms, 1.08x faster ✔
      q5: prev= 861 ms, new= 717 ms, 1.20x faster ✅
      q6: prev= 484 ms, new= 422 ms, 1.15x faster ✔
      q7: prev= 864 ms, new= 768 ms, 1.13x faster ✔
      q8: prev= 964 ms, new= 990 ms, 1.03x slower ✖
      q9: prev=1184 ms, new=1164 ms, 1.02x faster ✔
     q10: prev= 580 ms, new= 624 ms, 1.08x slower ✖
     q11: prev= 420 ms, new= 442 ms, 1.05x slower ✖
     q12: prev= 565 ms, new= 484 ms, 1.17x faster ✔
     q13: prev= 540 ms, new= 430 ms, 1.26x faster ✅
     q14: prev= 492 ms, new= 418 ms, 1.18x faster ✔
     q15: prev= 693 ms, new= 814 ms, 1.17x slower ✖
     q16: prev= 341 ms, new= 397 ms, 1.16x slower ✖
     q17: prev= 863 ms, new= 849 ms, 1.02x faster ✔
     q18: prev= 751 ms, new= 649 ms, 1.16x faster ✔
     q19: prev= 541 ms, new= 557 ms, 1.03x slower ✖
     q20: prev= 756 ms, new= 712 ms, 1.06x faster ✔
     q21: prev=1200 ms, new=1330 ms, 1.11x slower ✖
     q22: prev= 381 ms, new= 388 ms, 1.02x slower ✖

==== Comparison with previous run ====
      q1: prev= 808 ms, new= 756 ms, 1.07x faster ✔
      q2: prev= 621 ms, new= 592 ms, 1.05x faster ✔
      q3: prev= 464 ms, new= 352 ms, 1.32x faster ✅
      q4: prev=6541 ms, new=6897 ms, 1.05x slower ✖
      q5: prev= 873 ms, new=1057 ms, 1.21x slower ❌
      q6: prev=1736 ms, new=1835 ms, 1.06x slower ✖
      q7: prev= 698 ms, new= 708 ms, 1.01x slower ✖
      q8: prev= 722 ms, new= 833 ms, 1.15x slower ✖
     q10: prev=1159 ms, new=1287 ms, 1.11x slower ✖
     q11: prev=4198 ms, new=4386 ms, 1.04x slower ✖
     q12: prev= 591 ms, new= 475 ms, 1.24x faster ✅
     q13: prev=1260 ms, new=1247 ms, 1.01x faster ✔
     q14: prev=2172 ms, new=2198 ms, 1.01x slower ✖
     q15: prev= 623 ms, new= 506 ms, 1.23x faster ✅
     q16: prev=1054 ms, new= 845 ms, 1.25x faster ✅
     q17: prev= 752 ms, new= 782 ms, 1.04x slower ✖
     q18: prev= 887 ms, new= 748 ms, 1.19x faster ✔
     q19: prev= 933 ms, new= 826 ms, 1.13x faster ✔
     q20: prev= 480 ms, new= 470 ms, 1.02x faster ✔
     q21: prev= 682 ms, new= 541 ms, 1.26x faster ✅
     q22: prev= 799 ms, new= 770 ms, 1.04x faster ✔
     q23: prev=2202 ms, new=2347 ms, 1.07x slower ✖
     q24: prev=1258 ms, new=1251 ms, 1.01x faster ✔
     q25: prev= 697 ms, new= 697 ms, 1.00x slower ✖
     q26: prev= 616 ms, new= 484 ms, 1.27x faster ✅
     q27: prev=1475 ms, new=1656 ms, 1.12x slower ✖
     q28: prev= 706 ms, new= 705 ms, 1.00x faster ✔
     q29: prev= 637 ms, new= 668 ms, 1.05x slower ✖
     q31: prev= 971 ms, new= 992 ms, 1.02x slower ✖
     q32: prev= 568 ms, new= 516 ms, 1.10x faster ✔
     q33: prev= 768 ms, new= 802 ms, 1.04x slower ✖
     q34: prev= 681 ms, new= 664 ms, 1.03x faster ✔
     q35: prev=1135 ms, new=1126 ms, 1.01x faster ✔
     q36: prev= 944 ms, new=1120 ms, 1.19x slower ✖
     q37: prev=1140 ms, new=1254 ms, 1.10x slower ✖
     q38: prev= 798 ms, new= 842 ms, 1.06x slower ✖
     q39: prev=1420 ms, new=1358 ms, 1.05x faster ✔
     q40: prev= 613 ms, new= 651 ms, 1.06x slower ✖
     q41: prev= 398 ms, new= 307 ms, 1.30x faster ✅
     q42: prev= 400 ms, new= 366 ms, 1.09x faster ✔
     q43: prev= 410 ms, new= 434 ms, 1.06x slower ✖
     q44: prev= 748 ms, new= 693 ms, 1.08x faster ✔
     q45: prev= 758 ms, new= 564 ms, 1.34x faster ✅
     q46: prev=1696 ms, new=1201 ms, 1.41x faster ✅
     q47: prev=1878 ms, new=1977 ms, 1.05x slower ✖
     q48: prev=1088 ms, new=1107 ms, 1.02x slower ✖
     q49: prev= 679 ms, new= 714 ms, 1.05x slower ✖
     q50: prev= 665 ms, new= 659 ms, 1.01x faster ✔
     q51: prev= 990 ms, new=1044 ms, 1.05x slower ✖
     q52: prev= 439 ms, new= 375 ms, 1.17x faster ✔
     q53: prev= 603 ms, new= 603 ms, 1.00x slower ✖
     q54: prev=1375 ms, new=1476 ms, 1.07x slower ✖
     q55: prev= 734 ms, new= 633 ms, 1.16x faster ✔
     q56: prev= 836 ms, new= 836 ms, 1.00x slower ✖
     q57: prev=1125 ms, new=1117 ms, 1.01x faster ✔
     q58: prev= 986 ms, new= 952 ms, 1.04x faster ✔
     q59: prev= 669 ms, new= 534 ms, 1.25x faster ✅
     q60: prev= 840 ms, new= 737 ms, 1.14x faster ✔
     q61: prev=1630 ms, new=1733 ms, 1.06x slower ✖
     q62: prev=2259 ms, new=2441 ms, 1.08x slower ✖
     q63: prev= 615 ms, new= 631 ms, 1.03x slower ✖
     q64: prev=3147 ms, new=3193 ms, 1.01x slower ✖
     q65: prev= 944 ms, new= 946 ms, 1.00x slower ✖
     q66: prev=1099 ms, new=1310 ms, 1.19x slower ✖
     q67: prev=2201 ms, new=2133 ms, 1.03x faster ✔
     q68: prev=1110 ms, new=1104 ms, 1.01x faster ✔
     q69: prev=1340 ms, new=1185 ms, 1.13x faster ✔
     q70: prev=1179 ms, new= 997 ms, 1.18x faster ✔
     q71: prev= 988 ms, new= 781 ms, 1.27x faster ✅
     q72: prev=67824 ms, new=69807 ms, 1.03x slower ✖
     q73: prev= 736 ms, new= 717 ms, 1.03x faster ✔
     q74: prev=2090 ms, new=2348 ms, 1.12x slower ✖
     q75: prev=1248 ms, new=1214 ms, 1.03x faster ✔
     q76: prev= 545 ms, new= 509 ms, 1.07x faster ✔
     q77: prev=1194 ms, new=1465 ms, 1.23x slower ❌
     q78: prev=1162 ms, new=1059 ms, 1.10x faster ✔
     q79: prev= 843 ms, new= 954 ms, 1.13x slower ✖
     q80: prev=1464 ms, new=1272 ms, 1.15x faster ✔
     q81: prev= 796 ms, new= 840 ms, 1.06x slower ✖
     q82: prev=1200 ms, new=1307 ms, 1.09x slower ✖
     q83: prev= 892 ms, new= 901 ms, 1.01x slower ✖
     q84: prev= 888 ms, new=1113 ms, 1.25x slower ❌
     q85: prev=1152 ms, new=1554 ms, 1.35x slower ❌
     q86: prev= 684 ms, new= 732 ms, 1.07x slower ✖
     q87: prev= 814 ms, new= 826 ms, 1.01x slower ✖
     q88: prev=1343 ms, new=1333 ms, 1.01x faster ✔
     q89: prev= 500 ms, new= 531 ms, 1.06x slower ✖
     q90: prev= 606 ms, new= 632 ms, 1.04x slower ✖
     q91: prev= 616 ms, new= 607 ms, 1.01x faster ✔
     q92: prev= 643 ms, new= 638 ms, 1.01x faster ✔
     q93: prev= 522 ms, new= 512 ms, 1.02x faster ✔
     q94: prev= 808 ms, new= 837 ms, 1.04x slower ✖
     q95: prev= 830 ms, new= 940 ms, 1.13x slower ✖
     q96: prev= 536 ms, new= 588 ms, 1.10x slower ✖
     q97: prev= 830 ms, new= 783 ms, 1.06x faster ✔
     q98: prev= 579 ms, new= 579 ms, 1.00x slower ✖
     q99: prev=29283 ms, new=29848 ms, 1.02x slower ✖

gabotechs · 2026-01-09T08:24:34Z

src/flight_service/spawn_select_all.rs

+
+/// Consumes all the provided streams in parallel sending their produced messages to a single
+/// queue in random order. The resulting queue is returned as a stream.
+pub(crate) fn spawn_select_all<T, El, Err>(


Same as before, just moved closer to where its used now

…on batching

…o gabrielmusat/worker-network-stream-optimizations # Conflicts: # src/execution_plans/network_coalesce.rs # src/execution_plans/network_shuffle.rs # tests/tpch_explain_analyze.rs # tests/tpch_plans_test.rs

…ions # Conflicts: # tests/tpch_explain_analyze.rs # tests/tpch_plans_test.rs

jayshrivastava

Nice PR!

Interestingly we had to do the exact same optimization in cockroachdb: cockroachdb/cockroach#75581

I really like all the simplification you've been doing lately. The network_*.rs files are much easier to read now. Plus no more MetricsCollectingStream 👍🏽

src/flight_service/worker_connection_pool.rs

src/protobuf/app_metadata.rs

src/flight_service/do_get.rs

gabotechs added 2 commits January 7, 2026 17:49

Allow passing any query id for benchmarks

b17e15a

Allow passing a custom query

ea02ab4

gabotechs force-pushed the gabrielmusat/worker-network-stream-optimizations branch 2 times, most recently from b7727f5 to 57c5182 Compare January 9, 2026 08:25

gabotechs commented Jan 9, 2026

View reviewed changes

gabotechs force-pushed the gabrielmusat/worker-network-stream-optimizations branch from 57c5182 to 6af968b Compare January 9, 2026 08:29

Tidy up benchmarking code

00dca96

gabotechs force-pushed the gabrielmusat/adapt-benchmarks-to-pass-any-query-id branch from 0b29cb3 to 00dca96 Compare January 9, 2026 09:20

gabotechs added 2 commits January 9, 2026 10:21

Allow passing a custom query

114fc49

Introduce worker_connection_pool.rs for handing inter-worker connecti…

bdef4cb

…on batching

gabotechs force-pushed the gabrielmusat/worker-network-stream-optimizations branch from 6af968b to bdef4cb Compare January 9, 2026 09:21

gabotechs added 3 commits January 13, 2026 10:44

Allow passing any query id for benchmarks

e15bdd6

Allow passing a custom query

25b81ab

Tidy up benchmarking code

66fa3a7

gabotechs force-pushed the gabrielmusat/adapt-benchmarks-to-pass-any-query-id branch from 00dca96 to 66fa3a7 Compare January 13, 2026 09:44

Merge branch 'gabrielmusat/adapt-benchmarks-to-pass-any-query-id' int…

10d9f05

…o gabrielmusat/worker-network-stream-optimizations # Conflicts: # src/execution_plans/network_coalesce.rs # src/execution_plans/network_shuffle.rs # tests/tpch_explain_analyze.rs # tests/tpch_plans_test.rs

gabotechs mentioned this pull request Jan 13, 2026

Coalesce partitions before and after NetworkCoalesceExec #285

Closed

Base automatically changed from gabrielmusat/adapt-benchmarks-to-pass-any-query-id to main January 13, 2026 17:09

Merge branch 'main' into gabrielmusat/worker-network-stream-optimizat…

9b72d79

…ions # Conflicts: # tests/tpch_explain_analyze.rs # tests/tpch_plans_test.rs

jayshrivastava reviewed Jan 13, 2026

View reviewed changes

jayshrivastava approved these changes Jan 13, 2026

View reviewed changes

src/protobuf/app_metadata.rs Show resolved Hide resolved

src/protobuf/app_metadata.rs Show resolved Hide resolved

src/flight_service/do_get.rs Show resolved Hide resolved

gabotechs added 2 commits January 14, 2026 07:52

Use ranges for the worker connection pool

a73cb52

Better debug messages

c29f52c

gabotechs merged commit d4212bb into main Jan 14, 2026
7 checks passed

gabotechs deleted the gabrielmusat/worker-network-stream-optimizations branch January 14, 2026 07:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Worker network stream optimizations #283

Worker network stream optimizations #283

Uh oh!

gabotechs commented Jan 9, 2026 •

edited

Loading

Uh oh!

gabotechs Jan 9, 2026

Uh oh!

jayshrivastava left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Worker network stream optimizations #283

Worker network stream optimizations #283

Uh oh!

Conversation

gabotechs commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gabotechs Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

jayshrivastava left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gabotechs commented Jan 9, 2026 •

edited

Loading