viirya commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested
fields from Generate
URL: https://github.com/apache/spark/pull/24637#issuecomment-572310533
Re-checked the current rule. Still cannot find clue from it and and above
query plan. At first glance, I suspect if
viirya commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested
fields from Generate
URL: https://github.com/apache/spark/pull/24637#issuecomment-572152187
Is `flattenRuns` (generatorOutput) also not in the parent Project?
viirya commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested
fields from Generate
URL: https://github.com/apache/spark/pull/24637#issuecomment-571947114
@cloud-fan Yea, will look at it tomorrow. Do you have test case? If no, I
may try to reproduce it.
viirya commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested
fields from Generate
URL: https://github.com/apache/spark/pull/24637#issuecomment-513047838
I'm fine to add a white-list, thro I think this approach is not
generator-specific. It is more conservative and safer,
viirya commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested
fields from Generate
URL: https://github.com/apache/spark/pull/24637#issuecomment-512694668
retest this please
This is an automated message from
viirya commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested
fields from Generate
URL: https://github.com/apache/spark/pull/24637#issuecomment-512682054
@dongjoon-hyun I added more generators. I think existing generators should
be found in the test.
viirya commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested
fields from Generate
URL: https://github.com/apache/spark/pull/24637#issuecomment-510705961
Thanks @dongjoon-hyun for the advice! I will add more few test cases
targeting other Generator.
viirya commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested
fields from Generate in explode
URL: https://github.com/apache/spark/pull/24637#issuecomment-510371170
retest this please
This is an automated
viirya commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested
fields from Generate in explode
URL: https://github.com/apache/spark/pull/24637#issuecomment-510286138
@dongjoon-hyun oh, yes, the new config should be enabled. Let me add it.
Would you like to re-submit a PR
viirya commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested
fields from Generate to address performance issue in explode
URL: https://github.com/apache/spark/pull/24637#issuecomment-493733142
retest this please.
viirya commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested
fields from Generate
URL: https://github.com/apache/spark/pull/24637#issuecomment-493715754
cc @uzadude @cloud-fan @dongjoon-hyun
This is an
11 matches
Mail list logo
