Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22029
@cloud-fan do you think we can go ahead with this or change EQUAL behavior
for structs? thanks.
---
-
To unsubscribe, e-mail:
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22029
I think so, in that case we wouldn't need to distinguish between a struct
and a mutli-value IN.
---
-
To unsubscribe, e-mail:
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22029
If we decide to follow PostgreSQL about the EQUAL behavior eventually, then
it will be much easier to fix the IN behavior, right?
---
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22029
Oracle doesn't support EQUAL between structs: `(1, 'a') = (1, 'b')` doesn't
work on Oracle. Postgres is the only one returning NULL in the case `(1, 'a') =
(1, null)`.
My main reason here is
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22029
If we want to follow PostgreSQL/Oracle for the IN behavior, why don't we
follow the EQUAL behavior as well?
---
-
To
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22029
@cloud-fan yes, but the behavior of EQUAL is not so consistent among the
different DBs. In Hive EQUAL on struct behaves like Spark as of now, in Presto
throws exception if there is a null,
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22029
Another point: I think it's also important to make the behavior of IN be
consistent with EQUAL. I tried PostgreSQL and `(1, 2) = (3, null)` returns null.
Shall we update EQUAL first? The
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22029
@cloud-fan sorry, I mistyped. I meant `(a, b) IN (select (x, y) ...)` is a
valid subquery, but probably my understanding that it was a multivalued
subquery is wrong, as I realize now that `(x, y)`
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22029
which Presto version did you test? I tried 0.203 and it fails
```
presto:default> select * from t2 where (1, 2) in (select x, y from t);
Query 20181101_085707_00012_n644a failed: line
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22029
@cloud-fan Presto doesn't have structs. Presto has `ROW`. `(a, b) IN
(select x, y ...)` is a valid mutli-value subquery in Presto, while for a
struct (ie. a row) the valid syntax is `row_col IN
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22029
Do you know how Presto supports multi-value in subquery? By reading the PR
description, it seems impossible if Preso treats `(a, b)` as a struct value.
How Preso distinguish `(a, b) IN (select
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98318/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22029
**[Test build #98318 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98318/testReport)**
for PR 22029 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22029
**[Test build #98318 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98318/testReport)**
for PR 22029 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22029
anymore comments on this @cloud-fan ? Thanks.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98089/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22029
**[Test build #98089 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98089/testReport)**
for PR 22029 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22029
**[Test build #98089 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98089/testReport)**
for PR 22029 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98084/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22029
**[Test build #98084 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98084/testReport)**
for PR 22029 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22029
**[Test build #98084 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98084/testReport)**
for PR 22029 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98027/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22029
**[Test build #98027 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98027/testReport)**
for PR 22029 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22029
**[Test build #98027 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98027/testReport)**
for PR 22029 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22029
thanks @cloud-fan and @juliuszsompolski. I changed to an approach similar
to the one used in #21403. I think we can also move to have the default value
for the config switched to the new behavior,
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22029
**[Test build #97800 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97800/testReport)**
for PR 22029 at commit
Github user juliuszsompolski commented on the issue:
https://github.com/apache/spark/pull/22029
IMHO if a new wrapper was justifiable for the IN-subquery in #21403, then
it is also justifiable to add one here for the IN-literal-list case.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97733/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97712/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22029
**[Test build #97733 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97733/testReport)**
for PR 22029 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22029
**[Test build #97712 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97712/testReport)**
for PR 22029 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22029
**[Test build #97733 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97733/testReport)**
for PR 22029 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22029
**[Test build #97712 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97712/testReport)**
for PR 22029 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22029
I don't have a strong opinion. cc @gatorsmile @hvanhovell @juliuszsompolski
---
-
To unsubscribe, e-mail:
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22029
Yes, I see your point. Unfortunately, the only way to address it is to
introduce a new wrapper as it was done previously in #21403 in order to be able
to differentiate the cases. We can do that,
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22029
So, the goal here is to make the behavior consistent between multi-column
IN-subquery and multi-column normal IN for Spark.
That said, I feel it's reasonable to change the behavior of
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22029
Sorry, I'll clarify:
- Oracle doesn't support `input_struct_col in (struct_col1, struct_col2,
...)` (same as Postgres);
- Hive returns `false` in that case.
---
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22029
> Oracle does the same.
So what's the behavior? return false?
> Hive behaves like Spark now (before and after the PR) for this case.
Again what's the behavior? return
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22029
Yes, I summarized earlier the behavior for `(a, b) in (struct_col1,
struct_col2, ...)`. You can also check the PR description in "Summarizing:".
Instead, `input_struct_col in (struct_col1,
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22029
Do you know how other databases behave for `(a, b) in (struct_col1,
struct_col2, ...)` and `input_struct_col in (struct_col1, struct_col2, ...)`?
Since `(a, b)` may be treated specially, we need
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22029
> It looks to me that this is another instance of special handling (a, b,
..), like #21403
Yes, we might say so. The problem is specific to how `=` is implemented for
struct when one or
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22029
It looks to me that this is another instance of special handling `(a, b,
..)`, like https://github.com/apache/spark/pull/21403
`(a, b) in (struct_col1, struct_col2, ...)` is different
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22029
Thanks for you comment @HyukjinKwon. Yes sure, I'll update the PR
description, thanks.
Yes, the previous behavior is the same as Hive behavior. What I wanted to
highlight there, though,
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22029
@mgaido91, mind if I ask add few notes about
https://github.com/apache/spark/pull/22029#issuecomment-411473227? For
instance, which DBMS follows the current change. And sounds the previous
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22029
kindly ping @cloud-fan @dongjoon-hyun @gatorsmile
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97150/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22029
**[Test build #97150 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97150/testReport)**
for PR 22029 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22029
**[Test build #97150 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97150/testReport)**
for PR 22029 at commit
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22029
@mgaido91 . The failure is relevant to this PR. Could you take a look?
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97110/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22029
**[Test build #97110 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97110/testReport)**
for PR 22029 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22029
**[Test build #97110 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97110/testReport)**
for PR 22029 at commit
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22029
@cloud-fan @dongjoon-hyun @gatorsmile anymore comments on this?
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96035/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22029
**[Test build #96035 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96035/testReport)**
for PR 22029 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22029
**[Test build #96035 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96035/testReport)**
for PR 22029 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22029
kindly ping @cloud-fan @gatorsmile
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22029
kindly ping @cloud-fan @gatorsmile
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22029
@cloud-fan @gatorsmile any thoughts on this?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22029
**[Test build #94482 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94482/testReport)**
for PR 22029 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94482/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94480/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22029
**[Test build #94480 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94480/testReport)**
for PR 22029 at commit
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22029
@cloud-fan if we consider only the expression IN with literals, yes, the
behavior is very similar, with the following difference: Presto throws
exception when null is present on both sides.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22029
@mgaido91 do you mean the current behavior is same with Hive and Presto?
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22029
**[Test build #94482 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94482/testReport)**
for PR 22029 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22029
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22029
**[Test build #94480 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94480/testReport)**
for PR 22029 at commit
1 - 100 of 124 matches
Mail list logo
