[ https://issues.apache.org/jira/browse/SPARK-48742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jing Zhan updated SPARK-48742: ------------------------------ Description: Introducing virtual column family to RocksDB. We attach an 8-byte-Id prefix as column family identifier for each of the key row that is put into RocksDB. The encoding and decoding of the virtual column family prefix happens at the {{RocksDBKeyEncoder}} layer as we can pre-allocate extra 8 bytes and avoid additional memcpy. > Virtual Column Family for RocksDB > --------------------------------- > > Key: SPARK-48742 > URL: https://issues.apache.org/jira/browse/SPARK-48742 > Project: Spark > Issue Type: Task > Components: Structured Streaming > Affects Versions: 4.0.0 > Reporter: Jing Zhan > Priority: Major > > Introducing virtual column family to RocksDB. We attach an 8-byte-Id prefix > as column family identifier for each of the key row that is put into RocksDB. > The encoding and decoding of the virtual column family prefix happens at the > {{RocksDBKeyEncoder}} layer as we can pre-allocate extra 8 bytes and avoid > additional memcpy. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org