[
https://issues.apache.org/jira/browse/SPARK-38056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiyi Kong updated SPARK-38056:
-------------------------------
Description:
In
[SPARK-31953|https://github.com/apache/spark/commit/4f9667035886a67e6c9a4e8fad2efa390e87ca68],
structured streaming support is added to history server. However this does not
work when spark.history.store.path is set to save app info using LevelDB.
This is because one of the keys of StreamingQueryData, runId, is UUID type,
which is not supported by LevelDB. When replaying event log file in history
server, StreamingQueryStatusListener will throw an exception when writing info
to the store, saying "java.lang.IllegalArgumentException: Type java.util.UUID
not allowed as key.".
Example event log is provided in attachments. When opening it in history server
with spark.history.store.path set to somewhere, no structured streaming info is
available.
was:
In
[SPARK-31953|https://github.com/apache/spark/commit/4f9667035886a67e6c9a4e8fad2efa390e87ca68],
structured streaming support is added to history server. However this does not
work when spark.history.store.path is set to save app info using LevelDB.
This is because one of the keys of StreamingQueryData, runId, is UUID type,
which is not supported by LevelDB. When replaying event log file in history
server, StreamingQueryStatusListener will throw an exception when writing info
to the store, saying "java.lang.IllegalArgumentException: Type java.util.UUID
not allowed as key.".
Example event log is provided in attachments. When opening it in history server
with spark.history.store.path set to somewhere, no structured info is available.
> Structured streaming not working in history server when using LevelDB
> ---------------------------------------------------------------------
>
> Key: SPARK-38056
> URL: https://issues.apache.org/jira/browse/SPARK-38056
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming, Web UI
> Affects Versions: 3.1.2, 3.2.0
> Reporter: Weiyi Kong
> Priority: Major
> Attachments: local-1643373518829
>
>
> In
> [SPARK-31953|https://github.com/apache/spark/commit/4f9667035886a67e6c9a4e8fad2efa390e87ca68],
> structured streaming support is added to history server. However this does
> not work when spark.history.store.path is set to save app info using LevelDB.
> This is because one of the keys of StreamingQueryData, runId, is UUID type,
> which is not supported by LevelDB. When replaying event log file in history
> server, StreamingQueryStatusListener will throw an exception when writing
> info to the store, saying "java.lang.IllegalArgumentException: Type
> java.util.UUID not allowed as key.".
> Example event log is provided in attachments. When opening it in history
> server with spark.history.store.path set to somewhere, no structured
> streaming info is available.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
