Jungtaek Lim created SPARK-41152:
------------------------------------
Summary: Print out schema as tree string format for state store
schema checker
Key: SPARK-41152
URL: https://issues.apache.org/jira/browse/SPARK-41152
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 3.4.0
Reporter: Jungtaek Lim
When the state store schema checker detects the incompatibility and produces
the error, the error log is not human friendly for schema comparison.
e.g.
{code:java}
valueSchema:
StructType(StructField(field1,StringType,true),StructField(field2,StringType,true),StructField(field3,StringType,true),StructField(_rescued_data,StringType,true),StructField(_metadata,StructType(StructField(file_path,StringType,true),StructField(file_name,StringType,true),StructField(file_size,LongType,true),StructField(file_modification_time,TimestampType,true)),false))
{code}
It should be much easier to compare if we print the schema out for tree string
format. It will be multi-line which regress the UX, but the entire error
message is already multi-line so should not a big deal.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
