eshu opened a new issue, #6283:
URL: https://github.com/apache/hudi/issues/6283
* Hudi version : 0.11.1
* Spark version : 3.1.1
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
* Environment: Glue 3
Issue happen quite rarely while writing, but if it happen, it reproduces
persistently.
As far as I understood from the second part of the stacktrace, hudi tries to
find files that contain `.marker` in their name, but there are no such files in
the path. Files in `.hoodie/.temp/20220802132553801/` have the name pattern
`MARKERS\d+` (MARKERS0, MARKERS1, MARKERS2, ..., MARKERS19).
**Stacktrace**
```
Caused by: org.apache.hudi.exception.HoodieRollbackException: Error rolling
back using marker files written for [==>20220802132553801__compaction__INFLIGHT]
at
org.apache.hudi.table.action.rollback.MarkerBasedRollbackStrategy.getRollbackRequests(MarkerBasedRollbackStrategy.java:103)
at
org.apache.hudi.table.action.rollback.BaseRollbackPlanActionExecutor.requestRollback(BaseRollbackPlanActionExecutor.java:109)
at
org.apache.hudi.table.action.rollback.BaseRollbackPlanActionExecutor.execute(BaseRollbackPlanActionExecutor.java:132)
at
org.apache.hudi.table.HoodieSparkMergeOnReadTable.scheduleRollback(HoodieSparkMergeOnReadTable.java:161)
at
org.apache.hudi.table.HoodieTable.rollbackInflightCompaction(HoodieTable.java:551)
...
Caused by: java.lang.IllegalArgumentException
at
org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:31)
at
org.apache.hudi.common.util.MarkerUtils.stripMarkerFolderPrefix(MarkerUtils.java:67)
at
org.apache.hudi.table.marker.DirectWriteMarkers.lambda$allMarkerFilePaths$0(DirectWriteMarkers.java:136)
at org.apache.hudi.common.fs.FSUtils.processFiles(FSUtils.java:277)
at
org.apache.hudi.table.marker.DirectWriteMarkers.allMarkerFilePaths(DirectWriteMarkers.java:135)
at
org.apache.hudi.table.marker.MarkerBasedRollbackUtils.getAllMarkerPaths(MarkerBasedRollbackUtils.java:62)
at
org.apache.hudi.table.action.rollback.MarkerBasedRollbackStrategy.getRollbackRequests(MarkerBasedRollbackStrategy.java:76)
... 80 more
```
Is it possible to recover the dataset?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
