[ https://issues.apache.org/jira/browse/SPARK-45558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jungtaek Lim reassigned SPARK-45558: ------------------------------------ Assignee: Chaoqin Li > Introduce a metadata file for streaming stateful operator > --------------------------------------------------------- > > Key: SPARK-45558 > URL: https://issues.apache.org/jira/browse/SPARK-45558 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming > Affects Versions: 3.5.0 > Reporter: Chaoqin Li > Assignee: Chaoqin Li > Priority: Major > Labels: pull-request-available > > The information to store in the metadata file: > * operator name (no need to be unique among stateful operators in the query) > * state store name > * numColumnsPrefixKey: > 0 if prefix scan is enabled, 0 otherwise > The body of metadata file will be in json format. The metadata file will be > versioned just as other streaming metadata file to be future proof. > The metadata file will improve expose more information about the state store, > improves debugability and facilitate the development of state related feature > such as reading and writing state and state repartitioning. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org