Commit 5c5c1cf
authored
chore(refactor): use memdb for flag storage (#1697)
This PR contains no behavioral changes, and no breaking changes.
It lays the groundwork for some of the upcoming improvements we want in
flagd, as specified in recent ADRs. It does this by re-implementing our
storage layer to use [go-memdb](https://github.com/hashicorp/go-memdb),
an open source in-memory database developed by Hashicorp and used in
[Consul](https://developer.hashicorp.com/consul) (as well as MANY other
projects).
### Why?
This will allow us to easily support:
- duplicate flag keys (namespaced by flagSetId), as mentioned in [this
ADR](https://github.com/open-feature/flagd/blob/main/docs/architecture-decisions/duplicate-flag-keys.md)
- robust flag selectors, as mentioned in [this
ADR](#1644), by supporting
"watchers" which allow us to "listen" to change in flags returned by a
given query 🕺
- robust (and possibly, in the future, even customizable) indexing on
arbitrary attributes (to easily support fetching "all boolean flags", or
"flags with metadata = xyz", etc)
- cross-"table" transactions
I have already PoC'd that these are all practical.
### Changes in implementation
- the `store` package's `State` is no longer just a struct; it's a
object with methods wrapping the internal db
- the `store` package's `State` was renamed to `Store` and the file was
renamed from `flags.go` to `store.go`, since it's ceased to be a simple
stateful object, and for consistency
- a non-serialized (used only internally) `Key` field was added to the
flag type (for indexing)
- a new constructor for the `Store` was added which takes a logger and
returns an error, the old was deprecated to avoid breaking changes in
consumers (the Go flagd provider, mostly)
Note that the go-memdb dependency is MPL2, which is not allowed by the
CNCF, however, go-memdb is already used in CNCF projects and has a
[special
exception](https://github.com/cncf/foundation/blob/347a55dc0a6c6dc3d55a9a782e5701080a8ec43b/license-exceptions/cncf-exceptions-2023-08-31.json#L7-L11).
### Perfromance
There was no significant change in performance, see benchmark diff vs
main below:
<details>
<summary>Benchmark diff vs main</summary>
```diff
-BenchmarkFractionalEvaluation/[email protected] 559051 13832 ns/op 7229 B/op 135 allocs/op
-BenchmarkFractionalEvaluation/[email protected] 611665 13106 ns/op 7229 B/op 135 allocs/op
-BenchmarkFractionalEvaluation/[email protected] 383074 13433 ns/op 7229 B/op 135 allocs/op
-BenchmarkFractionalEvaluation/[email protected] 529185 12190 ns/op 7229 B/op 135 allocs/op
-BenchmarkResolveBooleanValue/test_staticBoolFlag-16 3929409 1712 ns/op 1008 B/op 11 allocs/op
-BenchmarkResolveBooleanValue/test_targetingBoolFlag-16 812671 10276 ns/op 6065 B/op 87 allocs/op
-BenchmarkResolveBooleanValue/test_staticObjectFlag-16 4398327 1700 ns/op 1008 B/op 11 allocs/op
-BenchmarkResolveBooleanValue/test_missingFlag-16 4541409 1494 ns/op 784 B/op 12 allocs/op
-BenchmarkResolveBooleanValue/test_disabledFlag-16 2998599 1815 ns/op 1072 B/op 13 allocs/op
-BenchmarkResolveStringValue/test_staticStringFlag-16 4378830 1698 ns/op 1040 B/op 13 allocs/op
-BenchmarkResolveStringValue/test_targetingStringFlag-16 849668 9165 ns/op 6097 B/op 89 allocs/op
-BenchmarkResolveStringValue/test_staticObjectFlag-16 4560192 1363 ns/op 1008 B/op 11 allocs/op
-BenchmarkResolveStringValue/test_missingFlag-16 5283511 1196 ns/op 784 B/op 12 allocs/op
-BenchmarkResolveStringValue/test_disabledFlag-16 4393116 1446 ns/op 1072 B/op 13 allocs/op
-BenchmarkResolveFloatValue/test:_staticFloatFlag-16 4264772 1501 ns/op 1024 B/op 13 allocs/op
-BenchmarkResolveFloatValue/test:_targetingFloatFlag-16 776436 8191 ns/op 6081 B/op 89 allocs/op
-BenchmarkResolveFloatValue/test:_staticObjectFlag-16 4685841 1285 ns/op 1008 B/op 11 allocs/op
-BenchmarkResolveFloatValue/test:_missingFlag-16 5001636 1376 ns/op 784 B/op 12 allocs/op
-BenchmarkResolveFloatValue/test:_disabledFlag-16 3707120 1897 ns/op 1072 B/op 13 allocs/op
-BenchmarkResolveIntValue/test_staticIntFlag-16 3770362 1677 ns/op 1008 B/op 11 allocs/op
-BenchmarkResolveIntValue/test_targetingNumberFlag-16 739861 11142 ns/op 6065 B/op 87 allocs/op
-BenchmarkResolveIntValue/test_staticObjectFlag-16 4221418 1913 ns/op 1008 B/op 11 allocs/op
-BenchmarkResolveIntValue/test_missingFlag-16 4289516 1488 ns/op 768 B/op 12 allocs/op
-BenchmarkResolveIntValue/test_disabledFlag-16 4027533 1829 ns/op 1072 B/op 13 allocs/op
-BenchmarkResolveObjectValue/test_staticObjectFlag-16 1588855 3880 ns/op 2243 B/op 37 allocs/op
-BenchmarkResolveObjectValue/test_targetingObjectFlag-16 562364 11580 ns/op 7283 B/op 109 allocs/op
-BenchmarkResolveObjectValue/test_staticBoolFlag-16 5026976 1791 ns/op 1008 B/op 11 allocs/op
-BenchmarkResolveObjectValue/test_missingFlag-16 4254043 1553 ns/op 784 B/op 12 allocs/op
-BenchmarkResolveObjectValue/test_disabledFlag-16 3051976 2250 ns/op 1072 B/op 13 allocs/op
+BenchmarkFractionalEvaluation/[email protected] 478593 14527 ns/op 7467 B/op 143 allocs/op
+BenchmarkFractionalEvaluation/[email protected] 429560 14728 ns/op 7467 B/op 143 allocs/op
+BenchmarkFractionalEvaluation/[email protected] 574078 14230 ns/op 7467 B/op 143 allocs/op
+BenchmarkFractionalEvaluation/[email protected] 411690 15296 ns/op 7467 B/op 143 allocs/op
+BenchmarkResolveBooleanValue/test_staticBoolFlag-16 4133443 1973 ns/op 960 B/op 18 allocs/op
+BenchmarkResolveBooleanValue/test_targetingBoolFlag-16 822934 10981 ns/op 6033 B/op 94 allocs/op
+BenchmarkResolveBooleanValue/test_staticObjectFlag-16 3955728 1964 ns/op 976 B/op 18 allocs/op
+BenchmarkResolveBooleanValue/test_missingFlag-16 3068791 2294 ns/op 1064 B/op 21 allocs/op
+BenchmarkResolveBooleanValue/test_disabledFlag-16 3500334 2225 ns/op 1024 B/op 20 allocs/op
+BenchmarkResolveStringValue/test_staticStringFlag-16 3935048 1781 ns/op 1008 B/op 20 allocs/op
+BenchmarkResolveStringValue/test_targetingStringFlag-16 770565 10765 ns/op 6065 B/op 96 allocs/op
+BenchmarkResolveStringValue/test_staticObjectFlag-16 3896060 2084 ns/op 976 B/op 18 allocs/op
+BenchmarkResolveStringValue/test_missingFlag-16 3103950 2141 ns/op 1064 B/op 21 allocs/op
+BenchmarkResolveStringValue/test_disabledFlag-16 3717013 2092 ns/op 1024 B/op 20 allocs/op
+BenchmarkResolveFloatValue/test:_staticFloatFlag-16 3971438 2003 ns/op 976 B/op 20 allocs/op
+BenchmarkResolveFloatValue/test:_targetingFloatFlag-16 782996 10153 ns/op 6049 B/op 96 allocs/op
+BenchmarkResolveFloatValue/test:_staticObjectFlag-16 3469644 2115 ns/op 976 B/op 18 allocs/op
+BenchmarkResolveFloatValue/test:_missingFlag-16 3376167 2157 ns/op 1064 B/op 21 allocs/op
+BenchmarkResolveFloatValue/test:_disabledFlag-16 3610095 2032 ns/op 1024 B/op 20 allocs/op
+BenchmarkResolveIntValue/test_staticIntFlag-16 3883299 1941 ns/op 960 B/op 18 allocs/op
+BenchmarkResolveIntValue/test_targetingNumberFlag-16 823038 10725 ns/op 6033 B/op 94 allocs/op
+BenchmarkResolveIntValue/test_staticObjectFlag-16 3697454 2028 ns/op 976 B/op 18 allocs/op
+BenchmarkResolveIntValue/test_missingFlag-16 3326895 1986 ns/op 1048 B/op 21 allocs/op
+BenchmarkResolveIntValue/test_disabledFlag-16 3327046 2142 ns/op 1024 B/op 20 allocs/op
+BenchmarkResolveObjectValue/test_staticObjectFlag-16 1534627 4885 ns/op 2211 B/op 44 allocs/op
+BenchmarkResolveObjectValue/test_targetingObjectFlag-16 509614 14640 ns/op 7251 B/op 116 allocs/op
+BenchmarkResolveObjectValue/test_staticBoolFlag-16 3871867 1978 ns/op 960 B/op 18 allocs/op
+BenchmarkResolveObjectValue/test_missingFlag-16 3484065 2080 ns/op 1064 B/op 21 allocs/op
+BenchmarkResolveObjectValue/test_disabledFlag-16 4013230 2158 ns/op 1024 B/op 20 allocs/op
PASS
-ok github.com/open-feature/flagd/core/pkg/evaluator 233.286s
+ok github.com/open-feature/flagd/core/pkg/evaluator 261.212s
? github.com/open-feature/flagd/core/pkg/evaluator/mock [no test files]
PASS
-ok github.com/open-feature/flagd/core/pkg/logger 0.003s
+ok github.com/open-feature/flagd/core/pkg/logger 0.002s
? github.com/open-feature/flagd/core/pkg/model [no test files]
? github.com/open-feature/flagd/core/pkg/service [no test files]
PASS
-ok github.com/open-feature/flagd/core/pkg/service/ofrep 0.003s
+ok github.com/open-feature/flagd/core/pkg/service/ofrep 0.002s
PASS
ok github.com/open-feature/flagd/core/pkg/store 0.002s
? github.com/open-feature/flagd/core/pkg/sync [no test files]
@@ -51,9 +51,9 @@ PASS
ok github.com/open-feature/flagd/core/pkg/sync/builder 0.020s
? github.com/open-feature/flagd/core/pkg/sync/builder/mock [no test files]
PASS
-ok github.com/open-feature/flagd/core/pkg/sync/file 1.007s
+ok github.com/open-feature/flagd/core/pkg/sync/file 1.006s
PASS
-ok github.com/open-feature/flagd/core/pkg/sync/grpc 8.014s
+ok github.com/open-feature/flagd/core/pkg/sync/grpc 8.013s
PASS
ok github.com/open-feature/flagd/core/pkg/sync/grpc/credentials 0.004s
? github.com/open-feature/flagd/core/pkg/sync/grpc/credentials/mock [no test files]
@@ -61,10 +61,10 @@ ok github.com/open-feature/flagd/core/pkg/sync/grpc/credentials 0.004s
PASS
ok github.com/open-feature/flagd/core/pkg/sync/grpc/nameresolvers 0.002s
PASS
-ok github.com/open-feature/flagd/core/pkg/sync/http 4.008s
+ok github.com/open-feature/flagd/core/pkg/sync/http 4.007s
? github.com/open-feature/flagd/core/pkg/sync/http/mock [no test files]
PASS
-ok github.com/open-feature/flagd/core/pkg/sync/kubernetes 0.015s
+ok github.com/open-feature/flagd/core/pkg/sync/kubernetes 0.016s
? github.com/open-feature/flagd/core/pkg/sync/testing [no test files]
PASS
ok github.com/open-feature/flagd/core/pkg/telemetry 0.016s
```
</details>
---------
Signed-off-by: Todd Baert <[email protected]>1 parent 2f49ea7 commit 5c5c1cf
File tree
21 files changed
+465
-462
lines changed- core/pkg
- evaluator
- model
- store
- flagd
- pkg
- runtime
- service/flag-sync
- test/integration
21 files changed
+465
-462
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
54 | 59 | | |
55 | 60 | | |
56 | 61 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
459 | 459 | | |
460 | 460 | | |
461 | 461 | | |
462 | | - | |
| 462 | + | |
463 | 463 | | |
464 | 464 | | |
465 | 465 | | |
| |||
587 | 587 | | |
588 | 588 | | |
589 | 589 | | |
590 | | - | |
| 590 | + | |
| 591 | + | |
591 | 592 | | |
592 | 593 | | |
593 | 594 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
64 | 64 | | |
65 | 65 | | |
66 | 66 | | |
67 | | - | |
| 67 | + | |
68 | 68 | | |
69 | 69 | | |
70 | 70 | | |
71 | 71 | | |
72 | 72 | | |
73 | | - | |
| 73 | + | |
74 | 74 | | |
75 | 75 | | |
76 | 76 | | |
| |||
118 | 118 | | |
119 | 119 | | |
120 | 120 | | |
121 | | - | |
| 121 | + | |
122 | 122 | | |
123 | 123 | | |
124 | 124 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1377 | 1377 | | |
1378 | 1378 | | |
1379 | 1379 | | |
1380 | | - | |
1381 | | - | |
| 1380 | + | |
| 1381 | + | |
1382 | 1382 | | |
1383 | 1383 | | |
1384 | 1384 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
273 | 273 | | |
274 | 274 | | |
275 | 275 | | |
276 | | - | |
| 276 | + | |
277 | 277 | | |
278 | 278 | | |
279 | 279 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
923 | 923 | | |
924 | 924 | | |
925 | 925 | | |
926 | | - | |
| 926 | + | |
927 | 927 | | |
928 | 928 | | |
929 | 929 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
186 | 186 | | |
187 | 187 | | |
188 | 188 | | |
189 | | - | |
| 189 | + | |
190 | 190 | | |
191 | 191 | | |
192 | 192 | | |
| |||
383 | 383 | | |
384 | 384 | | |
385 | 385 | | |
386 | | - | |
387 | | - | |
| 386 | + | |
388 | 387 | | |
389 | 388 | | |
390 | 389 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
| |||
0 commit comments