Ramin Gharib created FLINK-41110:
------------------------------------
Summary: Find VARIANT object fields by comparing UTF-8 bytes
Key: FLINK-41110
URL: https://issues.apache.org/jira/browse/FLINK-41110
Project: Flink
Issue Type: Improvement
Components: API / Core
Reporter: Ramin Gharib
Fix For: 2.4.0
{{Variant.getField}} decodes every metadata key it compares into a new
{{String}}:
{code:java}
// BinaryVariant.getFieldByKey, linear search for objects below 32 fields
if (key.equals(getMetadataKey(metadata, id))) {
{code}
{{v['a']}} in SQL calls {{getField}} once per record. One lookup in a 13-key
object can create 13 throwaway strings.
h3. Proposal
For objects below {{BINARY_SEARCH_THRESHOLD}} fields, encode the wanted key to
UTF-8 once. Compare it with the key bytes in the metadata dictionary. The
builder stores keys as {{key.getBytes(UTF_8)}}, so the lookup encodes the same
way.
The binary search for larger objects stays as it is. It depends on the order in
which the builder sorts keys. That order is {{String.compareTo}}, not the UTF-8
byte order the Parquet Variant spec asks for. Fixing the order is a separate
ticket.
JMH on JDK 17, 4 lookups per row of 13 keys:
|| ||master||patch||
|time per row|285 ns|132 ns|
|allocated per row|1368 B|296 B|
h3. Behavior changes below the threshold
||Lookup||master||patch||
|{{getField("a\uD800")}} on a field written with that key|null|found|
|{{getField("\uFFFD")}} on a key whose bytes are not valid UTF-8|found|null|
Flink never writes keys that are not valid UTF-8. Two distinct malformed keys
already decoded to the same name.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
