MaxGekk commented on PR #59225: URL: https://github.com/apache/spark/pull/59225#issuecomment-6027547185
Thank you for the second pass, @dongjoon-hyun. I checked all 15 against the code, and every one holds; they are addressed in 2344945b0a4, and I reply inline with what changed for each. Items 1 and 2: the decision memo is a plain Guava cache within sql/core again, with the deciders serialized by core's Guava-free `KeyLock`, so nothing passes a Guava type to core. Lambda variables are now matched without the pruning, and the test's join key holds a real `transform`, with the unsplit method checked to be between 8000 and 65535 bytes. Items 3 and 4 went into `chooseCode` together. It now decides from how many blocks the split generation turned into methods, not from comparing the code, so a split that splits nothing is never compiled. A split that fails to compile is warned about with its error, or thrown under testing, and `SplitFailed` is gone. For item 5, I ran your UNPIVOT query: with the conf on the stage compiles and answers right, and with it off compiling the stage fails with "')' expected instead of '['". A test pins it, and the docs and description say so. I also checked that the new tests catch what they are for. With the old pruned transform restored, the lambda test fails; with the old code comparison restored, the new no-block-split test fails with two compiles. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
