[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18986 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18986 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18986 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18986 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user stanzhai commented on the issue: https://github.com/apache/spark/pull/18986 @gatorsmile @DonnyZone When comparing a string to a int in Hive, it will cast string type to double. ``` hive> select * from tb; 0 0 0.1 0 true0 19157170390056971 0 hive> select * from tb where a = 0; 0 0 hive> select * from tb where a = 19157170390056973L; WARNING: Comparing a bigint and a string may result in a loss of precision. 19157170390056973 0 hive> select 1 = 'true'; NULL hive> select 19157170390056973L = '19157170390056971'; WARNING: Comparing a bigint and a string may result in a loss of precision. true ``` So, I think that cast a string to double type when compare with a numeric is more reasonable. Actually, my usage scenarios are for Spark compatibility. The problem I found when I upgraded Spark to 2.2.0, and lots of SQL's results are wrong. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user stanzhai commented on the issue: https://github.com/apache/spark/pull/18986 @DonnyZone @gatorsmile @cloud-fan PostgreSQL will throw an error when comparing a string to a int. ``` postgres=# select * from tb; a | b --+--- 0.1 | 1 a| 1 true | 1 (3 rows) postgres=# select * from tb where a>0; ERROR: operator does not exist: character varying > integer LINE 1: select * from tb where a>0; ^ HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts. ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user DonnyZone commented on the issue:
https://github.com/apache/spark/pull/18986
@gatorsmile For this issue, I think the behevior in PromoteStrings rule is
reasonable, but there are problems in underlying converter UTF8String.
As described in PR-15880 (https://github.com/apache/spark/pull/15880):
> It's more reasonable to follow postgres, i.e. cast string to the type of
the other side, but return null if the string is not castable to keep hive
compatibility.
However, the underlying UTF8String still returns true for cases that are
not castable. From the below code, we can get res=true and wrapper.value=0.
Consequently, it results in wrong answer in SPARK-21774.
```
val x = UTF8String.fromString("0.1")
val wrapper = new IntWrapper
val res = x.toInt(wrapper)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18986 Yea, since this topic is important for some users, I mean we better move the doc into `./docs/` ( I feel novices dont seem to check the code documents). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18986 @maropu Previously I did it in https://github.com/apache/spark/blob/master/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala#L34-L59 We can follow what MySQL did https://dev.mysql.com/doc/refman/5.7/en/type-conversion.html. It looks more user friendly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18986 Either way, we couldn't touch on this behaviour until the next major release? (It seems we've already fixed this policy for the 2.x releases in #15880... maybe). Btw, I feel we first should document this implicit casting behaviour somewhere like https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-AllowedImplicitConversions cuz this makes users much confused. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18986 In Hive, `1 = 'true'` will return `true`? `19157170390056973L = '19157170390056971'` will return `true`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/18986
@gatorsmile Seems cast to `double` is correct.
```
hive> create table spark_21646(c1 string, c2 string);
hive> insert into spark_21646
values('922337203685477580719223372036854775807192233720368547758071', 'a');
hive> insert into spark_21646 values('21474836471', 'b');
hive> insert into spark_21646 values('10', 'c');
hive> insert into spark_21646 values('0.125', 'd');
hive> insert into spark_21646 values('1.25', 'd');
hive> select * from spark_21646 where c1 > 0;
922337203685477580719223372036854775807192233720368547758071a
0.125 d
21474836471 b
1.25d
10 c
```
```
spark-sql> select * from spark_21646 where c1 > 0;
1.25d
10 c
spark-sql> select * from spark_21646 where c1 > 0L;
21474836471 b
1.25d
10 c
spark-sql> select * from spark_21646 where c1 > 0D;
922337203685477580719223372036854775807192233720368547758071a
0.125 d
21474836471 b
1.25d
10 c
spark-sql>
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18986 @stanzhai @wangyum How about Hive? I think your usage scenarios are for hive compatibility, right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/18986 There is a issue if a string value out of double range, see: [SPARK-21646](https://issues.apache.org/jira/browse/SPARK-21646). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18986 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18986 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80868/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18986 **[Test build #80868 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80868/testReport)** for PR 18986 at commit [`1647447`](https://github.com/apache/spark/commit/1647447b4e29f43e8bbb13ff5eb97f55e7b2ea99). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18986 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80866/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18986 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18986 **[Test build #80866 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80866/testReport)** for PR 18986 at commit [`627116a`](https://github.com/apache/spark/commit/627116a39e5a3cc5c2479dda98f794120aa2db52). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18986 **[Test build #80868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80868/testReport)** for PR 18986 at commit [`1647447`](https://github.com/apache/spark/commit/1647447b4e29f43e8bbb13ff5eb97f55e7b2ea99). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user stanzhai commented on the issue: https://github.com/apache/spark/pull/18986 In MySQL conversion of values from one string type to numeric, will be compared as floating-point (real) numbers. [](https://dev.mysql.com/doc/refman/5.7/en/type-conversion.html) The following rules describe how conversion occurs for comparison operations: - If one or both arguments are NULL, the result of the comparison is NULL, except for the NULL-safe <=> equality comparison operator. For NULL <=> NULL, the result is true. No conversion is needed. - If both arguments in a comparison operation are strings, they are compared as strings. - If both arguments are integers, they are compared as integers. - Hexadecimal values are treated as binary strings if not compared to a number. - If one of the arguments is a TIMESTAMP or DATETIME column and the other argument is a constant, the constant is converted to a timestamp before the comparison is performed. This is done to be more ODBC-friendly. > Note that this is not done for the arguments to IN()! To be safe, always use complete datetime, date, or time strings when doing comparisons. For example, to achieve best results when using BETWEEN with date or time values, use CAST() to explicitly convert the values to the desired data type. - A single-row subquery from a table or tables is not considered a constant. For example, if a subquery returns an integer to be compared to a DATETIME value, the comparison is done as two integers. The integer is not converted to a temporal value. To compare the operands as DATETIME values, use CAST() to explicitly convert the subquery value to DATETIME. - If one of the arguments is a decimal value, comparison depends on the other argument. The arguments are compared as decimal values if the other argument is a decimal or integer value, or as floating-point values if the other argument is a floating-point value. - In all other cases, the arguments are compared as floating-point (real) numbers. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18986 **[Test build #80866 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80866/testReport)** for PR 18986 at commit [`627116a`](https://github.com/apache/spark/commit/627116a39e5a3cc5c2479dda98f794120aa2db52). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18986 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80849/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18986 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18986 **[Test build #80849 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80849/testReport)** for PR 18986 at commit [`fd77e1e`](https://github.com/apache/spark/commit/fd77e1ed671f1e112a6c5de7786343b70029f9a3). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18986 @stanzhai Implicit type casting has to follow some rules. If you think this is a bug, could you do a complete investigation about this issue? For example, how Hive handles the type promotion? If needed, we can consider supporting different type promotion/casting modes. (e.g., one is Hive compatible) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18986 **[Test build #80849 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80849/testReport)** for PR 18986 at commit [`fd77e1e`](https://github.com/apache/spark/commit/fd77e1ed671f1e112a6c5de7786343b70029f9a3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18986 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18986 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
