[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19389 **[Test build #84051 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84051/testReport)** for PR 19389 at commit [`07bcf36`](https://github.com/apache/spark/commit/07bcf36a387e3b7fe2164b610ed1beaabcedb4e6). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19389 I have just made a table to check the diff easily: **Before**: |InputA \ InputB|`NullType`|`IntegerType`|`LongType`|`DecimalType(38,0)`|`DoubleType`|`DateType`|`TimestampType`|`StringType`| ||--|--|--|--|--|--|--|--| |**`NullType`**|`StringType`|||`StringType`||`StringType`|`StringType`|| |**`IntegerType`**`IntegerType`||`IntegerType`|`IntegerType`|| |**`LongType`**`LongType`|`DoubleType`|`LongType`|`LongType`|| |**`DecimalType(38,0)`**|`StringType`|`IntegerType`|`LongType`||`DoubleType`|`DecimalType(38,0)`|`DecimalType(38,0)`|| |**`DoubleType`**|||`DoubleType`|`DoubleType`||`DoubleType`|`DoubleType`|| |**`DateType`**|`StringType`|`IntegerType`|`LongType`|`DateType`|`DoubleType`||`DateType`|| |**`TimestampType`**|`StringType`|`IntegerType`|`LongType`|`TimestampType`|`DoubleType` |**`StringType`**| **After**: |InputA \ InputB|`NullType`|`IntegerType`|`LongType`|`DecimalType(38,0)`|`DoubleType`|`DateType`|`TimestampType`|`StringType`| ||--|--|--|--|--|--|--|--| |**`NullType`**|`NullType`|||`DecimalType(38,0)`||`DateType`|`TimestampType`|| |**`IntegerType`**`DecimalType(38,0)`||`StringType`|`StringType`|| |**`LongType`**`DecimalType(38,0)`|`StringType`|`StringType`|`StringType`|| |**`DecimalType(38,0)`**|`DecimalType(38,0)`|`DecimalType(38,0)`|`DecimalType(38,0)`||`StringType`|`StringType`|`StringType`|| |**`DoubleType`**|||`StringType`|`StringType`||`StringType`|`StringType`|| |**`DateType`**|`DateType`|`StringType`|`StringType`|`StringType`|`StringType`||`TimestampType`|| |**`TimestampType`**|`TimestampType`|`StringType`|`StringType`|`StringType`|`StringType` |**`StringType`**| --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19389 **[Test build #84049 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84049/testReport)** for PR 19389 at commit [`a1f1c3a`](https://github.com/apache/spark/commit/a1f1c3a0aa5f80b3d7a129ab7e8c20fafa8e5a13). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19389 Maybe, what do you think about opening a discussion in the mailing list? If I understood correctly, committers have a different opinion on this (did I understand correctly?). That should deduplicate a discussion about it. I am willing to actively join in it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19389 Generally, the current type inference/coercion rules are messy and random. We have to seriously revisit our type coercion. After thinking it more, I think this change in this PR is pretty risky. It just introduces new type inference behaviors, although I do not like the previous one neither. These changes could easily cause new regression when our users upgrading their Spark versions. For making the migration more smooth, my general proposal is to introduce a conf for each one when we change something like this, if we believe this is a bug fix. Remove or deprecate the internal conf in the next release (or after a few releases) if nobody raises the issue after a major release (around half a year). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19389 Also cc @rxin @cloud-fan @sameeragarwal --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19389 Hi @gatorsmile, could you please review this when you have some time? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19389 Thank you so much @gatorsmile. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19389 Will review it this weekend. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19389 @cloud-fan, could you take a look when you have some time please? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19389 ping? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19389 ping @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19389 Do you mean before / after in PR description? They are bugs to fix, aren't they? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19389 @gatorsmile, could you elaborate which behaviour changes you mean? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19389 Please ensure no behavior change is introduced when fixing such issues. Also cc @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19389 This PR introduces the behavior changes. We are unable to do this. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19389 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82345/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19389 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19389 **[Test build #82345 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82345/testReport)** for PR 19389 at commit [`52d0cc8`](https://github.com/apache/spark/commit/52d0cc83cb67d9dc82b8e0bc1e8e411d75739356). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19389 > why the tests are all going into `ParquetPartitionDiscoverySuite` -- this doesn't seem specific to parquet, and in fact I wonder if it will be different in parquet since the non-partition columns have schemas specified in the data. I'm just surprised this isn't tested across more formats. Yea, I agree since this problem is not specific to Parquet. Here, such changes and test cases look added in this file so far and I simply just decided to follow it, rather than including changes here restructuing or moving the test cases, partly for the easy of backporting (we should backport this into branch-2.2 and 2.1) and partly to reduce reviewing cost. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19389 **[Test build #82345 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82345/testReport)** for PR 19389 at commit [`52d0cc8`](https://github.com/apache/spark/commit/52d0cc83cb67d9dc82b8e0bc1e8e411d75739356). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19389 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19389 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82310/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19389 **[Test build #82310 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82310/testReport)** for PR 19389 at commit [`1e10336`](https://github.com/apache/spark/commit/1e10336128bf1e78a889ee4438e4519bb12bd84a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19389 **[Test build #82310 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82310/testReport)** for PR 19389 at commit [`1e10336`](https://github.com/apache/spark/commit/1e10336128bf1e78a889ee4438e4519bb12bd84a). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19389 **[Test build #82307 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82307/testReport)** for PR 19389 at commit [`1e10336`](https://github.com/apache/spark/commit/1e10336128bf1e78a889ee4438e4519bb12bd84a). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19389 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19389 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19389 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82307/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19389 **[Test build #82307 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82307/testReport)** for PR 19389 at commit [`1e10336`](https://github.com/apache/spark/commit/1e10336128bf1e78a889ee4438e4519bb12bd84a). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19389: [SPARK-22165][SQL] Resolve type conflicts between decima...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19389 cc @cloud-fan (I believe my similar PR was reviewed by you before), @ueshin and @squito. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
