Re: [I] update cannot change partition path value [hudi]
rangareddy closed issue #17311: update cannot change partition path value URL: https://github.com/apache/hudi/issues/17311 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] update cannot change partition path value [hudi]
rangareddy commented on issue #17311:
URL: https://github.com/apache/hudi/issues/17311#issuecomment-5507012017
Closing this as resolved, under a different ticket rather than this one.
The behaviour reported here, an UPDATE silently leaving the partition column
at its old value, cannot occur on `master`. HUDI-8823 ("Ban update query from
changing partition column value", **Closed/Fixed**) settled it by rejecting the
statement outright, and that landed in `6b4c8f3308dd` ("[HUDI-8823] Ban update
from updating primary key and partition key", #12587, 2025-01-14), six weeks
after this ticket was filed.
On current `master` the statement in the description throws
`HoodieAnalysisException` during analysis, before any write:
https://github.com/apache/hudi/blob/a2788eef922e2375432ead3fe50af7fe5f55faf7/hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/hudi/command/UpdateHoodieTableCommand.scala#L132-L139
The Spark 4 copy carries the same check, and `TestUpdateTable.scala:419-428`
pins the behaviour, so the silent-wrong-result path is gone rather than merely
unobserved.
To be precise about what was and was not resolved: the data-correctness bug
is fixed, the capability is not. Moving a record between partitions from an
UPDATE statement is still unsupported by design. Anyone who wants that should
open a **new feature request** referencing HUDI-8823, so a reviewer can see the
ban is being lifted deliberately rather than treat it as a regression fix.
Reopening this issue for that purpose would obscure the distinction.
Two notes for anyone arriving here:
- Cross-partition record movement from SQL is supported today via MERGE INTO
with a global index and the matching `update.partition.path` config enabled.
See `MergeIntoHoodieTableCommand.scala:306-317` and the index-to-config map at
`:1197-1201`.
- The rejection message ends with "Please remove the assignment clause to
avoid the error", which does not name that alternative. Worth a small follow-up
so the next person does not need this thread.
HUDI-8568 in JIRA is still Open and assigned; it needs resolving there as
well, pointing at HUDI-8823.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] update cannot change partition path value [hudi]
rangareddy commented on issue #17311:
URL: https://github.com/apache/hudi/issues/17311#issuecomment-5504704565
Re-checked against `master` at `a2788eef922e`. **Recommend closing this as
superseded by HUDI-8823.**
The behaviour reported here cannot happen any more. HUDI-8568 was filed
2024-11-23 describing a silent wrong result: update the partition column, get
the old value back with no error. Six weeks later HUDI-8823, "Ban update query
from changing partition column value", resolved that by rejecting the statement
outright. It is **Closed/Fixed**, and it landed in `6b4c8f3308dd` ("[HUDI-8823]
Ban update from updating primary key and partition key", #12587, 2025-01-14).
On current `master`, an UPDATE that assigns to a partition field throws
`HoodieAnalysisException` before any write happens:
https://github.com/apache/hudi/blob/a2788eef922e2375432ead3fe50af7fe5f55faf7/hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/hudi/command/UpdateHoodieTableCommand.scala#L132-L139
The Spark 4 copy carries the same check, and the behaviour is pinned by a
test at `TestUpdateTable.scala:419-428`, so the repro in the description now
fails fast rather than returning `(1,1,"a")`.
Two consequences worth stating explicitly, since they affect what should
happen to this ticket rather than to the code:
- Patching this issue as written would mean removing a deliberate, tested
restriction. That is a design reversal, not a bug fix, and belongs on `dev@`
rather than in a drive-by PR. If cross-partition UPDATE is still wanted, it is
cleaner to reopen it as a feature request that references HUDI-8823, so the
reviewer sees the ban is being lifted on purpose.
- Moving a record across partitions from SQL is already supported today via
MERGE INTO with a global index and the matching `update.partition.path` config
enabled. See `MergeIntoHoodieTableCommand.scala:306-317` and the
index-to-config map at `:1197-1201`.
One small piece of residue if this stays open in any form: the rejection
message says "Please remove the assignment clause to avoid the error", which
does not tell a user what to do instead. Naming the MERGE INTO route there
would save the next person the trip to this ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] update cannot change partition path value [hudi]
rangareddy commented on issue #17311: URL: https://github.com/apache/hudi/issues/17311#issuecomment-5365820861 This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-8568). **Findings: needs a reproduction against a current build before it can be dispositioned.** The report is concrete and actionable as written: with a global index, SQL UPDATE setting the partition key leaves the row in its original partition - starting from (1, a), UPDATE ... SET partitionKey=b WHERE primaryKey=1 yields (1, a) rather than (1, b). I have deliberately not called this one from the source. The surrounding code has been substantially rewritten since it was filed - table versions 8 and 9, the `BufferedRecordMerger` family replacing payload-based merging, the file-group-reader migration, and the `hudi-common` package reorganisations in #19193 / #19195. With that much churn, inferring either "still broken" or "silently fixed" would be a guess rather than a finding, and both errors are costly here. The good news is that reproducing it is cheap: the description includes a full Scala test case parameterised over cow and mor with hoodie.index.type=GLOBAL_SIMPLE. There is also no commit anywhere in the tree referencing HUDI-8568, so if this was fixed it was incidental to another change rather than deliberate - which is another reason to verify rather than assume. Keeping this open. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
