Neil Ramaswamy created SPARK-47746:
--------------------------------------
Summary: Use column ordinals instead of prefix ordering columns in
the range scan encoder
Key: SPARK-47746
URL: https://issues.apache.org/jira/browse/SPARK-47746
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 4.0.0
Reporter: Neil Ramaswamy
Currently, the State V2 implementations do projections in their state managers,
and then provide some prefix (ordering) columns to the RocksDBStateEncoder.
However, we can avoid doing extra projection by just reading the ordinals we
need, in the order we need, in the state encoder.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
