[jira] [Commented] (SPARK-7286) Precedence of operator not behaving properly
[ https://issues.apache.org/jira/browse/SPARK-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15185924#comment-15185924 ] Apache Spark commented on SPARK-7286: - User 'jodersky' has created a pull request for this issue: https://github.com/apache/spark/pull/11588 > Precedence of operator not behaving properly > > > Key: SPARK-7286 > URL: https://issues.apache.org/jira/browse/SPARK-7286 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.3.1 > Environment: Linux >Reporter: DevilJetha >Priority: Critical > > The precedence of the operators ( especially with !== and && ) in Dataframe > Columns seems to be messed up. > Example Snippet > .where( $"col1" === "val1" && ($"col2" !== "val2") ) works fine. > whereas .where( $"col1" === "val1" && $"col2" !== "val2" ) > evaluates as ( $"col1" === "val1" && $"col2" ) !== "val2" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7286) Precedence of operator not behaving properly
[ https://issues.apache.org/jira/browse/SPARK-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15023569#comment-15023569 ] Apache Spark commented on SPARK-7286: - User 'jodersky' has created a pull request for this issue: https://github.com/apache/spark/pull/9925 > Precedence of operator not behaving properly > > > Key: SPARK-7286 > URL: https://issues.apache.org/jira/browse/SPARK-7286 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.3.1 > Environment: Linux >Reporter: DevilJetha >Priority: Critical > > The precedence of the operators ( especially with !== and && ) in Dataframe > Columns seems to be messed up. > Example Snippet > .where( $"col1" === "val1" && ($"col2" !== "val2") ) works fine. > whereas .where( $"col1" === "val1" && $"col2" !== "val2" ) > evaluates as ( $"col1" === "val1" && $"col2" ) !== "val2" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7286) Precedence of operator not behaving properly
[ https://issues.apache.org/jira/browse/SPARK-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15020246#comment-15020246 ] Xiao Li commented on SPARK-7286: : ) That is perfect! The problem is solved! See you next Monday and we can discuss the PR issue. > Precedence of operator not behaving properly > > > Key: SPARK-7286 > URL: https://issues.apache.org/jira/browse/SPARK-7286 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.3.1 > Environment: Linux >Reporter: DevilJetha >Priority: Critical > > The precedence of the operators ( especially with !== and && ) in Dataframe > Columns seems to be messed up. > Example Snippet > .where( $"col1" === "val1" && ($"col2" !== "val2") ) works fine. > whereas .where( $"col1" === "val1" && $"col2" !== "val2" ) > evaluates as ( $"col1" === "val1" && $"col2" ) !== "val2" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7286) Precedence of operator not behaving properly
[ https://issues.apache.org/jira/browse/SPARK-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15020244#comment-15020244 ] Jakob Odersky commented on SPARK-7286: -- Hey @xiao, I think we had a misunderstanding, =!= solves all the problems, I was suggesting that <> would not solve all problems but that =!= doesn't "look" as good (my subjective opinion ;)) > Precedence of operator not behaving properly > > > Key: SPARK-7286 > URL: https://issues.apache.org/jira/browse/SPARK-7286 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.3.1 > Environment: Linux >Reporter: DevilJetha >Priority: Critical > > The precedence of the operators ( especially with !== and && ) in Dataframe > Columns seems to be messed up. > Example Snippet > .where( $"col1" === "val1" && ($"col2" !== "val2") ) works fine. > whereas .where( $"col1" === "val1" && $"col2" !== "val2" ) > evaluates as ( $"col1" === "val1" && $"col2" ) !== "val2" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7286) Precedence of operator not behaving properly
[ https://issues.apache.org/jira/browse/SPARK-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15020243#comment-15020243 ] Xiao Li commented on SPARK-7286: Last week, [~jodersky] and I had a discussion. It sounds like their precedence are still different, even if we use =!= > Precedence of operator not behaving properly > > > Key: SPARK-7286 > URL: https://issues.apache.org/jira/browse/SPARK-7286 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.3.1 > Environment: Linux >Reporter: DevilJetha >Priority: Critical > > The precedence of the operators ( especially with !== and && ) in Dataframe > Columns seems to be messed up. > Example Snippet > .where( $"col1" === "val1" && ($"col2" !== "val2") ) works fine. > whereas .where( $"col1" === "val1" && $"col2" !== "val2" ) > evaluates as ( $"col1" === "val1" && $"col2" ) !== "val2" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7286) Precedence of operator not behaving properly
[
https://issues.apache.org/jira/browse/SPARK-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15015418#comment-15015418
]
Iulian Dragos commented on SPARK-7286:
--
I agree with Jakob's analysis. For reference, Slick uses {{=!=}}, which
wouldn't be a bad option and gives you the same precedence.
> Precedence of operator not behaving properly
>
>
> Key: SPARK-7286
> URL: https://issues.apache.org/jira/browse/SPARK-7286
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.3.1
> Environment: Linux
>Reporter: DevilJetha
>Priority: Critical
>
> The precedence of the operators ( especially with !== and && ) in Dataframe
> Columns seems to be messed up.
> Example Snippet
> .where( $"col1" === "val1" && ($"col2" !== "val2") ) works fine.
> whereas .where( $"col1" === "val1" && $"col2" !== "val2" )
> evaluates as ( $"col1" === "val1" && $"col2" ) !== "val2"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7286) Precedence of operator not behaving properly
[ https://issues.apache.org/jira/browse/SPARK-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014647#comment-15014647 ] Jakob Odersky commented on SPARK-7286: -- I just realized that <> would also have a different precedence that === > Precedence of operator not behaving properly > > > Key: SPARK-7286 > URL: https://issues.apache.org/jira/browse/SPARK-7286 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.3.1 > Environment: Linux >Reporter: DevilJetha >Priority: Critical > > The precedence of the operators ( especially with !== and && ) in Dataframe > Columns seems to be messed up. > Example Snippet > .where( $"col1" === "val1" && ($"col2" !== "val2") ) works fine. > whereas .where( $"col1" === "val1" && $"col2" !== "val2" ) > evaluates as ( $"col1" === "val1" && $"col2" ) !== "val2" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7286) Precedence of operator not behaving properly
[ https://issues.apache.org/jira/browse/SPARK-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014613#comment-15014613 ] Jakob Odersky commented on SPARK-7286: -- If we want the same precedence of equals and non-equals operators, they should both start with a symbol of same-precedence. = has the same precedence as !, however as per spec, if the operator also ends with a = and does not start with =, it is considered an assignment operator with a lower precedence than others. So I see a couple of options: 1) abandon the wish of same precedence and accept the status-quo 2) adopt Reynold's solution or define another operator 3) remove non-equals comparison altogether, instead rely on negating equals comparison. I.e (A !== B becomes !(A === B)) Personally, I am against choosing option 1 as it leads to confusing behaviour (like this JIRA demonstrates). Not sure what I prefer between option 2 or 3 though > Precedence of operator not behaving properly > > > Key: SPARK-7286 > URL: https://issues.apache.org/jira/browse/SPARK-7286 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.3.1 > Environment: Linux >Reporter: DevilJetha >Priority: Critical > > The precedence of the operators ( especially with !== and && ) in Dataframe > Columns seems to be messed up. > Example Snippet > .where( $"col1" === "val1" && ($"col2" !== "val2") ) works fine. > whereas .where( $"col1" === "val1" && $"col2" !== "val2" ) > evaluates as ( $"col1" === "val1" && $"col2" ) !== "val2" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7286) Precedence of operator not behaving properly
[ https://issues.apache.org/jira/browse/SPARK-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014581#comment-15014581 ] Reynold Xin commented on SPARK-7286: cc [~dragos] If this is indeed a problem, we should deprecate "!==" and create a new function "<>". > Precedence of operator not behaving properly > > > Key: SPARK-7286 > URL: https://issues.apache.org/jira/browse/SPARK-7286 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.3.1 > Environment: Linux >Reporter: DevilJetha >Priority: Critical > > The precedence of the operators ( especially with !== and && ) in Dataframe > Columns seems to be messed up. > Example Snippet > .where( $"col1" === "val1" && ($"col2" !== "val2") ) works fine. > whereas .where( $"col1" === "val1" && $"col2" !== "val2" ) > evaluates as ( $"col1" === "val1" && $"col2" ) !== "val2" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7286) Precedence of operator not behaving properly
[ https://issues.apache.org/jira/browse/SPARK-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014575#comment-15014575 ] Xiao Li commented on SPARK-7286: It works! Do you want we fix this by PR? > Precedence of operator not behaving properly > > > Key: SPARK-7286 > URL: https://issues.apache.org/jira/browse/SPARK-7286 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.3.1 > Environment: Linux >Reporter: DevilJetha >Priority: Critical > > The precedence of the operators ( especially with !== and && ) in Dataframe > Columns seems to be messed up. > Example Snippet > .where( $"col1" === "val1" && ($"col2" !== "val2") ) works fine. > whereas .where( $"col1" === "val1" && $"col2" !== "val2" ) > evaluates as ( $"col1" === "val1" && $"col2" ) !== "val2" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7286) Precedence of operator not behaving properly
[ https://issues.apache.org/jira/browse/SPARK-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014545#comment-15014545 ] Xiao Li commented on SPARK-7286: Let me try it > Precedence of operator not behaving properly > > > Key: SPARK-7286 > URL: https://issues.apache.org/jira/browse/SPARK-7286 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.3.1 > Environment: Linux >Reporter: DevilJetha >Priority: Critical > > The precedence of the operators ( especially with !== and && ) in Dataframe > Columns seems to be messed up. > Example Snippet > .where( $"col1" === "val1" && ($"col2" !== "val2") ) works fine. > whereas .where( $"col1" === "val1" && $"col2" !== "val2" ) > evaluates as ( $"col1" === "val1" && $"col2" ) !== "val2" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7286) Precedence of operator not behaving properly
[ https://issues.apache.org/jira/browse/SPARK-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014544#comment-15014544 ] Reynold Xin commented on SPARK-7286: Is <> taken? > Precedence of operator not behaving properly > > > Key: SPARK-7286 > URL: https://issues.apache.org/jira/browse/SPARK-7286 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.3.1 > Environment: Linux >Reporter: DevilJetha >Priority: Critical > > The precedence of the operators ( especially with !== and && ) in Dataframe > Columns seems to be messed up. > Example Snippet > .where( $"col1" === "val1" && ($"col2" !== "val2") ) works fine. > whereas .where( $"col1" === "val1" && $"col2" !== "val2" ) > evaluates as ( $"col1" === "val1" && $"col2" ) !== "val2" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7286) Precedence of operator not behaving properly
[ https://issues.apache.org/jira/browse/SPARK-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014540#comment-15014540 ] Xiao Li commented on SPARK-7286: [~marmbrus][~rxin] Is that possible we can change !== to =!= in Spark 2.0? Thank you! > Precedence of operator not behaving properly > > > Key: SPARK-7286 > URL: https://issues.apache.org/jira/browse/SPARK-7286 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.3.1 > Environment: Linux >Reporter: DevilJetha >Priority: Critical > > The precedence of the operators ( especially with !== and && ) in Dataframe > Columns seems to be messed up. > Example Snippet > .where( $"col1" === "val1" && ($"col2" !== "val2") ) works fine. > whereas .where( $"col1" === "val1" && $"col2" !== "val2" ) > evaluates as ( $"col1" === "val1" && $"col2" ) !== "val2" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7286) Precedence of operator not behaving properly
[ https://issues.apache.org/jira/browse/SPARK-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15012643#comment-15012643 ] Jakob Odersky commented on SPARK-7286: -- Going through the code, I saw that catalyst also defines !== in its dsl, so it seems this operator has quite wide-spread usage. Would deprecating it in favor of something else be a viable option? > Precedence of operator not behaving properly > > > Key: SPARK-7286 > URL: https://issues.apache.org/jira/browse/SPARK-7286 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.3.1 > Environment: Linux >Reporter: DevilJetha >Priority: Critical > > The precedence of the operators ( especially with !== and && ) in Dataframe > Columns seems to be messed up. > Example Snippet > .where( $"col1" === "val1" && ($"col2" !== "val2") ) works fine. > whereas .where( $"col1" === "val1" && $"col2" !== "val2" ) > evaluates as ( $"col1" === "val1" && $"col2" ) !== "val2" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7286) Precedence of operator not behaving properly
[ https://issues.apache.org/jira/browse/SPARK-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15012532#comment-15012532 ] Jakob Odersky commented on SPARK-7286: -- The problem is that !== is recognized as an assignment operator (according to ยง6.12.4 of the scala specification http://www.scala-lang.org/docu/files/ScalaReference.pdf) and thus has lower precedence than any other operator. A potential fix could be to rename !== to =!= > Precedence of operator not behaving properly > > > Key: SPARK-7286 > URL: https://issues.apache.org/jira/browse/SPARK-7286 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.3.1 > Environment: Linux >Reporter: DevilJetha >Priority: Critical > > The precedence of the operators ( especially with !== and && ) in Dataframe > Columns seems to be messed up. > Example Snippet > .where( $"col1" === "val1" && ($"col2" !== "val2") ) works fine. > whereas .where( $"col1" === "val1" && $"col2" !== "val2" ) > evaluates as ( $"col1" === "val1" && $"col2" ) !== "val2" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7286) Precedence of operator not behaving properly
[ https://issues.apache.org/jira/browse/SPARK-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15009898#comment-15009898 ] Xiao Li commented on SPARK-7286: This is affected by the operator precedence in Scala. There does not exist a simple fix to resolve this issue. For another example, the Scala compiler is unable to recognize the following syntax: where ( $"col1" !== "val1" && $"col2" === "val2" ) > Precedence of operator not behaving properly > > > Key: SPARK-7286 > URL: https://issues.apache.org/jira/browse/SPARK-7286 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.3.1 > Environment: Linux >Reporter: DevilJetha >Priority: Critical > > The precedence of the operators ( especially with !== and && ) in Dataframe > Columns seems to be messed up. > Example Snippet > .where( $"col1" === "val1" && ($"col2" !== "val2") ) works fine. > whereas .where( $"col1" === "val1" && $"col2" !== "val2" ) > evaluates as ( $"col1" === "val1" && $"col2" ) !== "val2" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7286) Precedence of operator not behaving properly
[ https://issues.apache.org/jira/browse/SPARK-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15009701#comment-15009701 ] Xiao Li commented on SPARK-7286: I can reproduce it. Let me investigate it. > Precedence of operator not behaving properly > > > Key: SPARK-7286 > URL: https://issues.apache.org/jira/browse/SPARK-7286 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.3.1 > Environment: Linux >Reporter: DevilJetha >Priority: Critical > > The precedence of the operators ( especially with !== and && ) in Dataframe > Columns seems to be messed up. > Example Snippet > .where( $"col1" === "val1" && ($"col2" !== "val2") ) works fine. > whereas .where( $"col1" === "val1" && $"col2" !== "val2" ) > evaluates as ( $"col1" === "val1" && $"col2" ) !== "val2" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
