[jira] [Comment Edited] (SPARK-25966) "EOF Reached the end of stream with bytes left to read" while reading/writing to Parquets
[
https://issues.apache.org/jira/browse/SPARK-25966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16678542#comment-16678542
]
Cheng Lian edited comment on SPARK-25966 at 11/7/18 5:34 PM:
-
Hey, [~andrioni], if you still have the original (potentially) corrupted
Parquet files at hand, could you please try reading them again with Spark 2.4
but with {{spark.sql.parquet.enableVectorizedReader}} set to {{false}}? In this
way, we fall back to the vanilla {{parquet-mr}} 1.10 Parquet reader. If it
works fine, it might be an issue in the vectorized reader.
Also, any chances that you can share a sample problematic file?
Since the same workload worked fine with Spark 2.2.1, I doubt whether this is
really a file corruption issue. Unless somehow Spark 2.4 is reading more
column(s)/row group(s) than Spark 2.2.1 for the same job, and those extra
column(s)/row group(s) happened to contain some corrupted data, which would
also indicate an optimizer side issue (predicate push-down and column pruning).
was (Author: lian cheng):
Hey, [~andrioni], if you still have the original (potentially) corrupted
Parquet files at hand, could you please try reading them again with Spark 2.4
but with {{spark.sql.parquet.enableVectorizedReader}} set to {{false}}? In this
way, we fall back to the vanilla {{parquet-mr}} 1.10 Parquet reader. If it
works fine, it might be an issue in the vectorized reader.
Also, any chances that you can share a sample problematic file?
Since the same workload worked fine with Spark 2.2.1, I doubt whether this is
really a file corruption issue. Unless somehow Spark 2.4 is reading more
columns/row groups than Spark 2.2.1 for the same job, which would also indicate
an optimizer side issue (predicate push-down and column pruning).
> "EOF Reached the end of stream with bytes left to read" while reading/writing
> to Parquets
> -
>
> Key: SPARK-25966
> URL: https://issues.apache.org/jira/browse/SPARK-25966
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.0
> Environment: Spark 2.4.0 (built from RC5 tag) running Hadoop 3.1.1 on
> top of a Mesos cluster. Both input and output Parquet files are on S3.
>Reporter: Alessandro Andrioni
>Priority: Major
>
> I was persistently getting the following exception while trying to run one
> Spark job we have using Spark 2.4.0. It went away after I regenerated from
> scratch all the input Parquet files (generated by another Spark job also
> using Spark 2.4.0).
> Is there a chance that Spark is writing (quite rarely) corrupted Parquet
> files?
> {code:java}
> org.apache.spark.SparkException: Job aborted.
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:196)
> at
> org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:159)
> at
> org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
> at
> org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)
> at
> org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
> at
> org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)
> at
> org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)
> at
> org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
> at
> org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
> at
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
> at
> org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:668)
> at
> org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:276)
> at
[jira] [Comment Edited] (SPARK-25966) "EOF Reached the end of stream with bytes left to read" while reading/writing to Parquets
[
https://issues.apache.org/jira/browse/SPARK-25966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16678542#comment-16678542
]
Cheng Lian edited comment on SPARK-25966 at 11/7/18 5:34 PM:
-
Hey, [~andrioni], if you still have the original (potentially) corrupted
Parquet files at hand, could you please try reading them again with Spark 2.4
but with {{spark.sql.parquet.enableVectorizedReader}} set to {{false}}? In this
way, we fall back to the vanilla {{parquet-mr}} 1.10 Parquet reader. If it
works fine, it might be an issue in the vectorized reader.
Also, any chances that you can share a sample problematic file?
Since the same workload worked fine with Spark 2.2.1, I doubt whether this is
really a file corruption issue. Unless somehow Spark 2.4 is reading more
columns/row groups than Spark 2.2.1 for the same job, and those extra
columns/row groups happened to contain some corrupted data, which would also
indicate an optimizer side issue (predicate push-down and column pruning).
was (Author: lian cheng):
Hey, [~andrioni], if you still have the original (potentially) corrupted
Parquet files at hand, could you please try reading them again with Spark 2.4
but with {{spark.sql.parquet.enableVectorizedReader}} set to {{false}}? In this
way, we fall back to the vanilla {{parquet-mr}} 1.10 Parquet reader. If it
works fine, it might be an issue in the vectorized reader.
Also, any chances that you can share a sample problematic file?
Since the same workload worked fine with Spark 2.2.1, I doubt whether this is
really a file corruption issue. Unless somehow Spark 2.4 is reading more
column(s)/row group(s) than Spark 2.2.1 for the same job, and those extra
column(s)/row group(s) happened to contain some corrupted data, which would
also indicate an optimizer side issue (predicate push-down and column pruning).
> "EOF Reached the end of stream with bytes left to read" while reading/writing
> to Parquets
> -
>
> Key: SPARK-25966
> URL: https://issues.apache.org/jira/browse/SPARK-25966
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.0
> Environment: Spark 2.4.0 (built from RC5 tag) running Hadoop 3.1.1 on
> top of a Mesos cluster. Both input and output Parquet files are on S3.
>Reporter: Alessandro Andrioni
>Priority: Major
>
> I was persistently getting the following exception while trying to run one
> Spark job we have using Spark 2.4.0. It went away after I regenerated from
> scratch all the input Parquet files (generated by another Spark job also
> using Spark 2.4.0).
> Is there a chance that Spark is writing (quite rarely) corrupted Parquet
> files?
> {code:java}
> org.apache.spark.SparkException: Job aborted.
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:196)
> at
> org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:159)
> at
> org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
> at
> org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)
> at
> org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
> at
> org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)
> at
> org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)
> at
> org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
> at
> org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
> at
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
> at
> org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:668)
> at
>
