deepakpanda93 commented on issue #7613: URL: https://github.com/apache/hudi/issues/7613#issuecomment-4844735311
Closing this as stale / needs-info — it was never narrowed to a reproducible defect, and the relevant areas have since improved. This is almost certainly a cross-engine config issue rather than a bug in delete itself: for the Spark delete to find the Flink-written records, the record key, partition field, and key generator must match what the Flink job persisted. If they differ, the index can't locate the records and the delete is a no-op (your config1); the rollback + empty-parquet behavior in config2 is typically an index miss or concurrent Flink + Spark writers without a lock (which is why @danny0405 asked whether the Flink job was stopped during the Spark delete). On recent 1.x, much of this is handled automatically: the key generator type is inferred (HUDI-5929 [#8176](https://github.com/apache/hudi/pull/8176)) and writers reuse the persisted table configs (HUDI-5665 [#7901](https://github.com/apache/hudi/pull/7901)), so a Spark delete inherits the Flink table's record-key/partition/key-gen settings instead of relying on you to match them. Cross-engine read/merge is also standardized via the engine-agnostic FileGroup reader. Recommendations: - On 1.x, let the writer inherit key config from the table (hoodie.properties) and confirm the Spark delete's record-key/partition match the Flink table. - Don't run concurrent Flink + Spark writers without a lock provider + OCC/NBCC. If you can still reproduce a failed delete on 1.x with matching key config and no concurrent writer, please reopen with the table's hoodie.properties and a minimal repro, and we'll dig in. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
