Neil Ramaswamy created SPARK-48707: -------------------------------------- Summary: RocksDB setLevelCompactionDynamicLevelBytes should be set to false Key: SPARK-48707 URL: https://issues.apache.org/jira/browse/SPARK-48707 Project: Spark Issue Type: Improvement Components: Structured Streaming Affects Versions: 4.0.0 Reporter: Neil Ramaswamy
In RocksDB 8.4.0, they changed the default of `level_compaction_dynamic_level_bytes` to true. This means that any query that runs with this enabled will have an L6 in the LSM tree, so if a user rolls back to a previous version of Spark that doesn't have RocksDB 8.4.0, then their L6 of the LSM tree will not be compacted. We should set this conf to be false so that this rolling back works in Spark. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org