Re: [PR] [SPARK-47241][SQL] Fix rule order issues for ExtractGenerator [spark]

2024-05-02 Thread via GitHub
cloud-fan commented on PR #45350: URL: https://github.com/apache/spark/pull/45350#issuecomment-2090237466 > SELECT col_1, EXPLODE(MAP_KEYS(map_str_col)) AS key, map_str_col[key] AS value FROM nestedTable1; I think this can be supported with LCA. cc @anchovYu -- This is an

Re: [PR] [SPARK-47241][SQL] Fix rule order issues for ExtractGenerator [spark]

2024-05-02 Thread via GitHub
rajatrj20 commented on PR #45350: URL: https://github.com/apache/spark/pull/45350#issuecomment-2089760654 @cloud-fan This change broke an existing behaviour. When a aliased generator field A is referenced in some another field B in project list, it will create a situation where the B will

Re: [PR] [SPARK-47241][SQL] Fix rule order issues for ExtractGenerator [spark]

2024-03-07 Thread via GitHub
cloud-fan closed pull request #45350: [SPARK-47241][SQL] Fix rule order issues for ExtractGenerator URL: https://github.com/apache/spark/pull/45350 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [SPARK-47241][SQL] Fix rule order issues for ExtractGenerator [spark]

2024-03-07 Thread via GitHub
cloud-fan commented on PR #45350: URL: https://github.com/apache/spark/pull/45350#issuecomment-1983026959 thanks for the review, merging to master/3.5! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [SPARK-47241][SQL] Fix rule order issues for ExtractGenerator [spark]

2024-03-06 Thread via GitHub
cloud-fan commented on code in PR #45350: URL: https://github.com/apache/spark/pull/45350#discussion_r1515378900 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala: ## @@ -2876,28 +2876,36 @@ class Analyzer(override val catalogManager:

Re: [PR] [SPARK-47241][SQL] Fix rule order issues for ExtractGenerator [spark]

2024-03-06 Thread via GitHub
viirya commented on code in PR #45350: URL: https://github.com/apache/spark/pull/45350#discussion_r1515378800 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala: ## @@ -2876,28 +2876,36 @@ class Analyzer(override val catalogManager:

Re: [PR] [SPARK-47241][SQL] Fix rule order issues for ExtractGenerator [spark]

2024-03-06 Thread via GitHub
cloud-fan commented on code in PR #45350: URL: https://github.com/apache/spark/pull/45350#discussion_r1515378900 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala: ## @@ -2876,28 +2876,36 @@ class Analyzer(override val catalogManager:

Re: [PR] [SPARK-47241][SQL] Fix rule order issues for ExtractGenerator [spark]

2024-03-06 Thread via GitHub
cloud-fan commented on code in PR #45350: URL: https://github.com/apache/spark/pull/45350#discussion_r1515379202 ## sql/core/src/test/scala/org/apache/spark/sql/GeneratorFunctionSuite.scala: ## @@ -553,6 +552,32 @@ class GeneratorFunctionSuite extends QueryTest with

Re: [PR] [SPARK-47241][SQL] Fix rule order issues for ExtractGenerator [spark]

2024-03-06 Thread via GitHub
cloud-fan commented on code in PR #45350: URL: https://github.com/apache/spark/pull/45350#discussion_r1515379047 ## sql/core/src/test/scala/org/apache/spark/sql/GeneratorFunctionSuite.scala: ## @@ -553,6 +552,32 @@ class GeneratorFunctionSuite extends QueryTest with

Re: [PR] [SPARK-47241][SQL] Fix rule order issues for ExtractGenerator [spark]

2024-03-06 Thread via GitHub
cloud-fan commented on code in PR #45350: URL: https://github.com/apache/spark/pull/45350#discussion_r1515378012 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala: ## @@ -2876,28 +2876,36 @@ class Analyzer(override val catalogManager:

Re: [PR] [SPARK-47241][SQL] Fix rule order issues for ExtractGenerator [spark]

2024-03-06 Thread via GitHub
viirya commented on code in PR #45350: URL: https://github.com/apache/spark/pull/45350#discussion_r1514904624 ## sql/core/src/test/scala/org/apache/spark/sql/GeneratorFunctionSuite.scala: ## @@ -553,6 +552,32 @@ class GeneratorFunctionSuite extends QueryTest with

Re: [PR] [SPARK-47241][SQL] Fix rule order issues for ExtractGenerator [spark]

2024-03-06 Thread via GitHub
viirya commented on code in PR #45350: URL: https://github.com/apache/spark/pull/45350#discussion_r1514903490 ## sql/core/src/test/scala/org/apache/spark/sql/GeneratorFunctionSuite.scala: ## @@ -553,6 +552,32 @@ class GeneratorFunctionSuite extends QueryTest with

Re: [PR] [SPARK-47241][SQL] Fix rule order issues for ExtractGenerator [spark]

2024-03-06 Thread via GitHub
viirya commented on code in PR #45350: URL: https://github.com/apache/spark/pull/45350#discussion_r1514897535 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala: ## @@ -2876,28 +2876,36 @@ class Analyzer(override val catalogManager:

Re: [PR] [SPARK-47241][SQL] Fix rule order issues for ExtractGenerator [spark]

2024-03-06 Thread via GitHub
cloud-fan commented on PR #45350: URL: https://github.com/apache/spark/pull/45350#issuecomment-1981124178 cc @MaxGekk @yaooqinn @dongjoon-hyun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [SPARK-47241][SQL] Fix rule order issues for ExtractGenerator [spark]

2024-02-29 Thread via GitHub
cloud-fan commented on PR #45350: URL: https://github.com/apache/spark/pull/45350#issuecomment-1972615533 cc @viirya @gengliangwang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[PR] [SPARK-47241][SQL] Fix rule order issues for ExtractGenerator [spark]

2024-02-29 Thread via GitHub
cloud-fan opened a new pull request, #45350: URL: https://github.com/apache/spark/pull/45350 ### What changes were proposed in this pull request? The rule `ExtractGenerator` does not define any trigger condition when rewriting generator functions in `Project`, which makes the