Security implications of count optimized containers

[nrktkt]

The current spec says

This allows the parser to pre-size any internal construct used for parsing

While useful for optimization and essential for the way UBJ byte arrays would work, this requires implementations carefully allow for configuration of limits on collection size. Consider the simple memory exhaustion from careless parsing of this partial payload

[[][$][L][#][H][i][5][1e+12] // A terrabyte of data
    [0]
    [0]

The spec should have a section dedicated to outlining prevention of resource exhaustion. Methods to do this would include:

• A parsing mode which doesn't preallocate memory, and instead uses an incrementally growing data structure (such as a Java ArrayList with a small initial size). This would lean on external systems to limit size (like max entity size in a HTTP library) as well as force the attacker to actually send the full declared size to consume the memory.
• A limit on the number of elements in a single collection. eg. an array of bytes could have no more than 500000000 elements.
• A limit on the total number of elements in a collection and its nested collections combined. So a limit of 500000000 elements would prevent a 2d array of 400000000 by 400000000 elements.

[lightmare]

Allocating the exact size is not mandatory, implementation can take that as a hint.

All three bullets sound like implementation details. Checking size limits is not something a spec should require. You're assuming the whole payload needs to fit in memory of some arbitrary device.

[nrktkt]

You're right that it's an implementation detail and not something the spec should require, I'm not saying there's anything wrong with the way the spec works. But this is definitely something the spec should acknowledge and advise implementations on how to deal with. The language might look something like

Implementations SHOULD ensure the declared size of a collection is within appropriate limits before pre-sizing any internal constructs used for parsing using one of the following methods ...

I'm using SHOULD in the RFC meaning

This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.