Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-146688197
open a new one is also OK.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not hav
Github user dilipbiswal commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-146685208
@cloud-fan
I didn't see a re-open option on this pull request. Do i have to create a
new pull request ?
Please let me know ..
---
If your project is set up
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-146682761
just reopen this PR and we will trigger a test on our jenkin for it.
For local test, you can do `./build/sbt catalyst/test`
---
If your project is set up for
Github user dilipbiswal commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-146682265
@marmbrus .. sorry about it. Is there a way i can look at the list of
failures ?
I had run :
build/mvn -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver
-Dhadoo
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-146680375
Sorry, this never passed tests and broke something. I'm going to revert.
Please reopen the PR.
---
If your project is set up for it, you can reply to this email and
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-146679919
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feat
Github user dilipbiswal commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-146645486
Thanks a lot @marmbrus .
Many thanks to @cloud-fan for his help.
---
If your project is set up for it, you can reply to this email and have your
reply appea
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/8983
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enab
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-146634337
Thanks, merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-146631777
I only found one rule in `Optimizer` to optimize `In` to `InSet` when the
values in `list` are all literals.
I think it's better to add a case in `NullPropagation`
Github user dilipbiswal commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-146470802
@cloud-fan
Thanks a lot. I have implemented the review comments. Please take a look. I
looked at the optimizer code,. we already seem to be transforming NULL in
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/8983#discussion_r41475635
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
---
@@ -135,4 +135,26 @@ class AnalysisSuite extends Analy
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/8983#discussion_r41475573
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
---
@@ -305,12 +305,22 @@ object HiveTypeCoercion {
Github user dilipbiswal commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-146404345
@cloud-fan
Hi Wenchen, can you please take a look at the changes and let me know what
you think..
---
If your project is set up for it, you can reply to this em
Github user dilipbiswal commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-146287283
Thank you @marmbrus @rick-ibm @cloud-fan
I checked the behavior of db2. It also raises an error if the in list types
are not compatible.
db2 => sel
Github user rick-ibm commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-146280859
I don't find any guidance in the Standard for what should be done if the
left side of the IN operator is an untyped NULL literal. Technically, there is
no such thing in
Github user rick-ibm commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-146271123
Hi Michael,
Postgres and Derby raise an error if the expressions in the IN list can't
be implicitly cast to a common type. MySQL is more forgiving.
Tha
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-146266287
I think we are all in agreement that `null IN (...)` should return null.
The only question here is if we should through an error when the stuff in
`(...)` can
Github user rick-ibm commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-146244545
According to my reading of the SQL Standard,
NULL IN (expr1, ...)
should always evaluate to NULL. Here is my reasoning:
The 2011 SQL Standard
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-146077599
if `null in (true, array(2,3))` returns null in hive, then we probably
should first cast `key` and all values in `list` to a common type. If there is
no such a common
Github user dilipbiswal commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-146047895
@marmbrus
Thanks a lot michael for looking into this. I debugged hive to understand
the
behaviour and would like to share my findings. I wanted to make
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-146016743
Lets follow hive.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this fea
Github user dilipbiswal commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-146003985
Very good point..Thanks.. Actually Hive reports an error in this case.
hive> select * from tnull where array(2,3) in (1, array(2,3));
FAILED: SemanticExce
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-145977094
yup, another JIRA please.
You need ask some committers like @marmbrus to review your PR to get it
merged .
My final thoughts on this PR:
When the
Github user dilipbiswal commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-145958925
Thanks a LOT @cloud-fan. Sure.. i will look into it. When you say another
PR,
do we mean another JIRA ?
Asking as i am new to the process.
On
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-145953466
ok misleaded by the imperative code style, sorry for that...
So we will return null if the `key` is null, or we can't find a value in
`list` matching our `key`, and
Github user dilipbiswal commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-145948447
@cloud-fan , please confirm my understanding of the code (fairly new to the
codebase..:-)
In the code we go through the entire in list and run evaluate the
expre
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-145930866
Looks like our
[implementation](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala#L124-
Github user dilipbiswal commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-145734618
Can you please help clarify ? Are you referring to the case when one of the
value in the
in list is literal null like 1 in (1, NULL) ? If so, i don't think we c
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-145717934
I checked our
[implementation](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala#L126-L
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-145718166
btw you can send another PR to add a rule in `Optimizer` so that if `value`
is literal null, we can just return null.
---
If your project is set up for it, you can re
Github user dilipbiswal commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-145713214
hive> select null in (1,2,null) from tnull;
OK
NULL
NULL
NULL
Time taken: 0.139 seconds, Fetched: 3 row(s)
---
If your project is set up for it
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-145712756
can you try `select null in (1,2,null)`? I wanna make sure hive doesn't
execute the `In` operation when `value` is null type.
---
If your project is set up for it, yo
Github user dilipbiswal commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-145711558
Hi Wenchen,
Here is the link i could find where its a bit confusing on the equality
operator.
https://cwiki.apache.org/confluence/display/Hive/Lang
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/8983#discussion_r41212281
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
---
@@ -305,12 +305,17 @@ object HiveTypeCoercion {
Github user dilipbiswal commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-145696762
Thanks Wenchen. You are right that not all types can be casted to boolean.
However, in this case, we are not trying to cast the in list types to the
LHS typ
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-145689983
if you change the `value` type to boolean, then we have to change each
element in `list` to boolean type too, which maybe dangerous as a lot of types
can't be casted t
Github user dilipbiswal commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-145664718
Thanks !!
Do we need to look at the in list types in this case ? The in list types
could be literals of different types , right ? for example NULL not in (1
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/8983#discussion_r41188938
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
---
@@ -305,12 +305,17 @@ object HiveTypeCoercion {
Github user dilipbiswal commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-145639934
Thanks for reviewing the code Wenchen. I was trying to model the test case
based on what was put in the JIRA which did a caseInsensitiveAnalyze. I have
fixed it now.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/8983#discussion_r41183045
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
---
@@ -135,4 +135,11 @@ class AnalysisSuite extends Analy
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-145630551
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your pr
GitHub user dilipbiswal opened a pull request:
https://github.com/apache/spark/pull/8983
[SPARK-8654][SQL] Fix Analysis exception when using NULL IN (...)
In the analysis phase , while processing the rules for IN predicate, we
compare the in-list types to the lhs expression type
43 matches
Mail list logo
