Sagar Sumit created HUDI-8519:
---------------------------------
Summary: Update failed with mismatched partition path if mutliple
secondary indexes present
Key: HUDI-8519
URL: https://issues.apache.org/jira/browse/HUDI-8519
Project: Apache Hudi
Issue Type: Task
Reporter: Sagar Sumit
Assignee: Sagar Sumit
Fix For: 1.0.0
After building an index on {{rider}} and {{driver}} columns, an update with
fails with {{mismatched partition path, record partition:}}
{code:java}
24/11/13 21:12:27 ERROR SparkSQLDriver: Failed in [
update hudi_table set rider = 'rider-E' where rider = 'rider-F']
org.apache.hudi.exception.HoodieException: Failed to update metadata
at
org.apache.hudi.client.BaseHoodieClient.writeTableMetadata(BaseHoodieClient.java:295)
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:294)
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:244)
at
org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:93)
at
org.apache.hudi.HoodieSparkSqlWriterInternal.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:991)
at
org.apache.hudi.HoodieSparkSqlWriterInternal.writeInternal(HoodieSparkSqlWriter.scala:533)
at
org.apache.hudi.HoodieSparkSqlWriterInternal.$anonfun$write$1(HoodieSparkSqlWriter.scala:190)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:125)
...
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:248)
at
org.apache.spark.sql.hudi.command.UpdateHoodieTableCommand.run(UpdateHoodieTableCommand.scala:79)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
...
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure:
Task 5 in stage 154.0 failed 1 times, most recent failure: Lost task 5.0 in
stage 154.0 (TID 495) (192.168.68.107 executor driver):
org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType
UPDATE for partition :5
at
org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:317)
at
org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.lambda$mapPartitionsAsRDD$a3ab3c4$1(BaseSparkCommitActionExecutor.java:250)
at
org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1(JavaRDDLike.scala:102)
at
org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1$adapted(JavaRDDLike.scala:102)
...
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hudi.exception.HoodieException: Error writing record
HoodieRecord{key=HoodieKey { recordKey=driver-P$trip4
partitionPath=secondary_index_idx_driver},
currentLocation='HoodieRecordLocation {instantTime=20241113211226184,
fileId=secondary-index-0004-0, position=-1}', newLocation='null'}
at
org.apache.hudi.client.FailOnFirstErrorWriteStatus.markFailure(FailOnFirstErrorWriteStatus.java:45)
at
org.apache.hudi.io.HoodieAppendHandle.writeToBuffer(HoodieAppendHandle.java:601)
at
org.apache.hudi.io.HoodieAppendHandle.doAppend(HoodieAppendHandle.java:469)
at
org.apache.hudi.table.action.deltacommit.BaseSparkDeltaCommitActionExecutor.handleUpdate(BaseSparkDeltaCommitActionExecutor.java:83)
at
org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:310)
... 33 more
Caused by: org.apache.hudi.exception.HoodieUpsertException: mismatched
partition path, record partition: secondary_index_idx_driver but trying to
insert into partition: secondary_index_idx_rider
at
org.apache.hudi.io.HoodieAppendHandle.writeToBuffer(HoodieAppendHandle.java:600)
... 36 more
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
