Read the link carefully, This solution is available (*only*) in Databricks Runtime.
You can enable RockDB-based state management by setting the following configuration in the SparkSession before starting the streaming query. spark.conf.set( "spark.sql.streaming.stateStore.providerClass", "com.databricks.sql.streaming.state.RocksDBStateStoreProvider") On Sun, 10 Mar 2019 at 11:54, Lian Jiang <jiangok2...@gmail.com> wrote: > Hi, > > I have a very simple SSS pipeline which does: > > val query = df > .dropDuplicates(Array("Id", "receivedAt")) > .withColumn(timePartitionCol, timestamp_udfnc(col("receivedAt"))) > .writeStream > .format("parquet") > .partitionBy("availabilityDomain", timePartitionCol) > .trigger(Trigger.ProcessingTime(5, TimeUnit.MINUTES)) > .option("path", "/data") > .option("checkpointLocation", "/data_checkpoint") > .start() > > After ingesting 2T records, the state under checkpoint folder on HDFS > (replicator factor 2) grows to 2T bytes. > My cluster has only 2T bytes which means the cluster can barely handle > further data growth. > > Online spark documents > (https://docs.databricks.com/spark/latest/structured-streaming/production.html) > says using rocksdb help SSS job reduce JVM memory overhead. But I cannot find > any document how > > to setup rocksdb for SSS. Spark class CheckpointReader seems to only handle > HDFS. > > Any suggestions? Thanks! > > > >