[ https://issues.apache.org/jira/browse/SPARK-31480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17096930#comment-17096930 ]
Apache Spark commented on SPARK-31480: -------------------------------------- User 'dilipbiswal' has created a pull request for this issue: https://github.com/apache/spark/pull/28425 > Improve the EXPLAIN FORMATTED's output for DSV2's Scan Node > ----------------------------------------------------------- > > Key: SPARK-31480 > URL: https://issues.apache.org/jira/browse/SPARK-31480 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.0.0 > Reporter: Xiao Li > Priority: Major > > Below is the EXPLAIN OUTPUT when using the *DSV2* > *Output of EXPLAIN EXTENDED* > {code:java} > +- BatchScan[col.dots#39L] JsonScan DataFilters: [isnotnull(col.dots#39L), > (col.dots#39L = 500)], Location: > InMemoryFileIndex[file:/private/var/folders/nr/j6hw4kr51wv0zynvr6srwgr00000gp/T/spark-7dad6f63-dc..., > PartitionFilters: [], ReadSchema: struct<col.dots:bigint> > {code} > *Output of EXPLAIN FORMATTED* > {code:java} > (1) BatchScan > Output [1]: [col.dots#39L] > Arguments: [col.dots#39L], > JsonScan(org.apache.spark.sql.test.TestSparkSession@45eab322,org.apache.spark.sql.execution.datasources.InMemoryFileIndex@72065f16,StructType(StructField(col.dots,LongType,true)),StructType(StructField(col.dots,LongType,true)),StructType(),org.apache.spark.sql.util.CaseInsensitiveStringMap@8822c5e0,Vector(),List(isnotnull(col.dots#39L), > (col.dots#39L = 500))) > {code} > When using *DSV1*, the output is much cleaner than the output of DSV2, > especially for EXPLAIN FORMATTED. > *Output of EXPLAIN EXTENDED* > {code:java} > +- FileScan json [col.dots#37L] Batched: false, DataFilters: > [isnotnull(col.dots#37L), (col.dots#37L = 500)], Format: JSON, Location: > InMemoryFileIndex[file:/private/var/folders/nr/j6hw4kr51wv0zynvr6srwgr00000gp/T/spark-89021d76-59..., > PartitionFilters: [], PushedFilters: [IsNotNull(`col.dots`), > EqualTo(`col.dots`,500)], ReadSchema: struct<col.dots:bigint> > {code} > *Output of EXPLAIN FORMATTED* > {code:java} > (1) Scan json > Output [1]: [col.dots#37L] > Batched: false > Location: InMemoryFileIndex > [file:/private/var/folders/nr/j6hw4kr51wv0zynvr6srwgr00000gp/T/spark-89021d76-5971-4a96-bf10-0730873f6ce0] > PushedFilters: [IsNotNull(`col.dots`), EqualTo(`col.dots`,500)] > ReadSchema: struct<col.dots:bigint>{code} > > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org