cloud-fan commented on pull request #28707:
URL: https://github.com/apache/spark/pull/28707#issuecomment-646450708
thanks, merging to master! (I think this patch is too big to backport)
This is an automated message from the A
cloud-fan commented on pull request #28707:
URL: https://github.com/apache/spark/pull/28707#issuecomment-644635176
I guess we may need to change the state store API to allow Spark to do
validation easily. I'm OK to put the validation logic in the state store
implementation for now, and fig
cloud-fan commented on pull request #28707:
URL: https://github.com/apache/spark/pull/28707#issuecomment-638626662
Yea we need this PR to unblock backporting SPARK-28067 to 3.0.
> the mechanism is a validation of the UnsafeRow which can be applied to all
stateful operations.
W
cloud-fan commented on pull request #28707:
URL: https://github.com/apache/spark/pull/28707#issuecomment-638166050
> Having a util function that does this check inside of Spark would be very
handy for future low-level debugging / investigations.
+1. How about put the validation code
cloud-fan commented on pull request #28707:
URL: https://github.com/apache/spark/pull/28707#issuecomment-637992953
I think this PR and SPARK-27237 are orthogonal, and we should have both.
SPARK-27237 is a bit hard to be merged as it changes the checkpoint. We may
need more reviews to see i
cloud-fan commented on pull request #28707:
URL: https://github.com/apache/spark/pull/28707#issuecomment-637991937
@skambha it doesn't fix the issue, it gives a better error message when we
hit the issue.
This is an automat
