372242283 commented on issue #3466:
URL: https://github.com/apache/iceberg/issues/3466#issuecomment-1418562275
I have the same problem, which happens occasionally,big guys, help me
analyze it Thank you:
version:iceberg1.1.0,spark3.1.3
CALL hive_catalog.system.rewrite_manifests('xx.tableA', false)
ERROR Msg：
23/02/05 21:12:04 INFO
KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each
executor to shut down
23/02/05 21:12:04 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client
has been closed (this is expected if the application is shutting down.)
23/02/05 21:12:05 INFO MapOutputTrackerMasterEndpoint:
MapOutputTrackerMasterEndpoint stopped!
23/02/05 21:12:05 INFO MemoryStore: MemoryStore cleared
23/02/05 21:12:05 INFO BlockManager: BlockManager stopped
23/02/05 21:12:05 INFO BlockManagerMaster: BlockManagerMaster stopped
23/02/05 21:12:05 INFO
OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
OutputCommitCoordinator stopped!
23/02/05 21:12:05 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main"
org.apache.iceberg.exceptions.ValidationException: Manifest is missing:
hdfs://xxx:8020/user/hive/warehouse/xx.db/tableA/metadata/d9c19289-fbf1-48a8-90be-a8eecdf088b4-m0.avro
at
org.apache.iceberg.BaseRewriteManifests.lambda$validateDeletedManifests$7(BaseRewriteManifests.java:299)
at java.base/java.util.Optional.ifPresent(Unknown Source)
at
org.apache.iceberg.BaseRewriteManifests.validateDeletedManifests(BaseRewriteManifests.java:297)
at
org.apache.iceberg.BaseRewriteManifests.apply(BaseRewriteManifests.java:195)
at org.apache.iceberg.SnapshotProducer.apply(SnapshotProducer.java:223)
at
org.apache.iceberg.BaseRewriteManifests.apply(BaseRewriteManifests.java:50)
at
org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:369)
at
org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:402)
at
org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:212)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:189)
at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:367)
at
org.apache.iceberg.BaseRewriteManifests.commit(BaseRewriteManifests.java:50)
at
org.apache.iceberg.spark.actions.BaseSnapshotUpdateSparkAction.commit(BaseSnapshotUpdateSparkAction.java:43)
at
org.apache.iceberg.spark.actions.BaseRewriteManifestsSparkAction.replaceManifests(BaseRewriteManifestsSparkAction.java:312)
at
org.apache.iceberg.spark.actions.BaseRewriteManifestsSparkAction.doExecute(BaseRewriteManifestsSparkAction.java:185)
at
org.apache.iceberg.spark.actions.BaseSparkAction.withJobGroupInfo(BaseSparkAction.java:97)
at
org.apache.iceberg.spark.actions.BaseRewriteManifestsSparkAction.execute(BaseRewriteManifestsSparkAction.java:150)
at
org.apache.iceberg.spark.actions.BaseRewriteManifestsSparkAction.execute(BaseRewriteManifestsSparkAction.java:80)
at
org.apache.iceberg.spark.procedures.RewriteManifestsProcedure.lambda$call$0(RewriteManifestsProcedure.java:97)
at
org.apache.iceberg.spark.procedures.BaseProcedure.execute(BaseProcedure.java:86)
at
org.apache.iceberg.spark.procedures.BaseProcedure.modifyIcebergTable(BaseProcedure.java:74)
at
org.apache.iceberg.spark.procedures.RewriteManifestsProcedure.call(RewriteManifestsProcedure.java:88)
at
org.apache.spark.sql.execution.datasources.v2.CallExec.run(CallExec.scala:33)
at
org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:40)
at
org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:40)
at
org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:46)
at
org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:228)
at
org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3700)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
at
org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
