[GitHub] [spark] cloud-fan closed pull request #29318: [SPARK-32509][SQL] Ignore unused DPP True Filter in Canonicalization

2020-08-02 Thread GitBox
cloud-fan closed pull request #29318: URL: https://github.com/apache/spark/pull/29318 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [spark] cloud-fan commented on pull request #29318: [SPARK-32509][SQL] Ignore unused DPP True Filter in Canonicalization

2020-08-02 Thread GitBox
cloud-fan commented on pull request #29318: URL: https://github.com/apache/spark/pull/29318#issuecomment-667783276 thanks, merging to master/3.0! This is an automated message from the Apache Git Service. To respond to the mes

[GitHub] [spark] leanken commented on pull request #29304: [SPARK-32494][SQL] Null Aware Anti Join Optimize Support Multi-Column

2020-08-02 Thread GitBox
leanken commented on pull request #29304: URL: https://github.com/apache/spark/pull/29304#issuecomment-667783096 ![image](https://user-images.githubusercontent.com/17242071/89143099-fb2dd500-d57b-11ea-881e-9d248403db9d.png) -

[GitHub] [spark] leanken edited a comment on pull request #29304: [SPARK-32494][SQL] Null Aware Anti Join Optimize Support Multi-Column

2020-08-02 Thread GitBox
leanken edited a comment on pull request #29304: URL: https://github.com/apache/spark/pull/29304#issuecomment-667781762 > > Step 2: Say there is a right (build) side row (1, null, 3). It should be counted as a match against a row on the left side (1, 2, 3). What makes this tricky is that s

[GitHub] [spark] cloud-fan commented on pull request #29317: [SPARK-32510][SQL] Check duplicate nested columns in read from JDBC datasource

2020-08-02 Thread GitBox
cloud-fan commented on pull request #29317: URL: https://github.com/apache/spark/pull/29317#issuecomment-667781962 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [spark] cloud-fan closed pull request #29317: [SPARK-32510][SQL] Check duplicate nested columns in read from JDBC datasource

2020-08-02 Thread GitBox
cloud-fan closed pull request #29317: URL: https://github.com/apache/spark/pull/29317 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [spark] leanken commented on pull request #29304: [SPARK-32494][SQL] Null Aware Anti Join Optimize Support Multi-Column

2020-08-02 Thread GitBox
leanken commented on pull request #29304: URL: https://github.com/apache/spark/pull/29304#issuecomment-667781762 > > Step 2: Say there is a right (build) side row (1, null, 3). It should be counted as a match against a row on the left side (1, 2, 3). What makes this tricky is that say say

[GitHub] [spark] cloud-fan closed pull request #29067: [SPARK-32274][SQL] Make SQL cache serialization pluggable

2020-08-02 Thread GitBox
cloud-fan closed pull request #29067: URL: https://github.com/apache/spark/pull/29067 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [spark] cloud-fan commented on pull request #29067: [SPARK-32274][SQL] Make SQL cache serialization pluggable

2020-08-02 Thread GitBox
cloud-fan commented on pull request #29067: URL: https://github.com/apache/spark/pull/29067#issuecomment-667780468 github action passes, I'm merging to master, thanks! This is an automated message from the Apache Git Service.

[GitHub] [spark] cloud-fan commented on a change in pull request #29146: [SPARK-32257][SQL] Reports explicit errors for invalid usage of SET/RESET command

2020-08-02 Thread GitBox
cloud-fan commented on a change in pull request #29146: URL: https://github.com/apache/spark/pull/29146#discussion_r464172835 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/SparkSqlParserSuite.scala ## @@ -61,6 +64,80 @@ class SparkSqlParserSuite extends

[GitHub] [spark] cloud-fan commented on a change in pull request #29146: [SPARK-32257][SQL] Reports explicit errors for invalid usage of SET/RESET command

2020-08-02 Thread GitBox
cloud-fan commented on a change in pull request #29146: URL: https://github.com/apache/spark/pull/29146#discussion_r464172446 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ## @@ -66,17 +68,29 @@ class SparkSqlAstBuilder(conf: SQLConf

[GitHub] [spark] cloud-fan commented on a change in pull request #29146: [SPARK-32257][SQL] Reports explicit errors for invalid usage of SET/RESET command

2020-08-02 Thread GitBox
cloud-fan commented on a change in pull request #29146: URL: https://github.com/apache/spark/pull/29146#discussion_r464172133 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ## @@ -66,17 +68,29 @@ class SparkSqlAstBuilder(conf: SQLConf

[GitHub] [spark] cloud-fan commented on a change in pull request #29146: [SPARK-32257][SQL] Reports explicit errors for invalid usage of SET/RESET command

2020-08-02 Thread GitBox
cloud-fan commented on a change in pull request #29146: URL: https://github.com/apache/spark/pull/29146#discussion_r464172042 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ## @@ -66,17 +68,29 @@ class SparkSqlAstBuilder(conf: SQLConf

[GitHub] [spark] cloud-fan commented on a change in pull request #29146: [SPARK-32257][SQL] Reports explicit errors for invalid usage of SET/RESET command

2020-08-02 Thread GitBox
cloud-fan commented on a change in pull request #29146: URL: https://github.com/apache/spark/pull/29146#discussion_r464171781 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ## @@ -66,17 +68,29 @@ class SparkSqlAstBuilder(conf: SQLConf

[GitHub] [spark] cloud-fan commented on a change in pull request #29146: [SPARK-32257][SQL] Reports explicit errors for invalid usage of SET/RESET command

2020-08-02 Thread GitBox
cloud-fan commented on a change in pull request #29146: URL: https://github.com/apache/spark/pull/29146#discussion_r464171523 ## File path: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ## @@ -246,11 +246,17 @@ statement | SET TIME ZONE inte

[GitHub] [spark] cloud-fan commented on a change in pull request #27066: [SPARK-31317][SQL] Add withField method to Column

2020-08-02 Thread GitBox
cloud-fan commented on a change in pull request #27066: URL: https://github.com/apache/spark/pull/27066#discussion_r464170447 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Column.scala ## @@ -871,6 +871,72 @@ class Column(val expr: Expression) extends Logging {

[GitHub] [spark] agrawaldevesh commented on pull request #29304: [SPARK-32494][SQL] Null Aware Anti Join Optimize Support Multi-Column

2020-08-02 Thread GitBox
agrawaldevesh commented on pull request #29304: URL: https://github.com/apache/spark/pull/29304#issuecomment-667773518 > Step 2: Say there is a right (build) side row (1, null, 3). It should be counted as a match against a row on the left side (1, 2, 3). What makes this tricky is that say

[GitHub] [spark] yaooqinn commented on pull request #28527: [SPARK-31709][SQL] Proper base path for database/table location when it is a relative path

2020-08-02 Thread GitBox
yaooqinn commented on pull request #28527: URL: https://github.com/apache/spark/pull/28527#issuecomment-667772325 gentle ping @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, pl

[GitHub] [spark] maropu commented on pull request #29192: [SPARK-32393][SQL] Support PostgreSQL `bpchar` array

2020-08-02 Thread GitBox
maropu commented on pull request #29192: URL: https://github.com/apache/spark/pull/29192#issuecomment-667772167 kindly ping. This is an automated message from the Apache Git Service. To respond to the message, please log on t

[GitHub] [spark] wangshisan commented on pull request #29266: [SPARK-32464][SQL] Support skew handling on join that has one side wi…

2020-08-02 Thread GitBox
wangshisan commented on pull request #29266: URL: https://github.com/apache/spark/pull/29266#issuecomment-667772074 @cloud-fan @JkSelf Could you have a look? This is an automated message from the Apache Git Service. To respo

[GitHub] [spark] HyukjinKwon commented on pull request #29320: [WIP][SPARK-32507][DOCS][PYTHON] Add main page for PySpark documentation

2020-08-02 Thread GitBox
HyukjinKwon commented on pull request #29320: URL: https://github.com/apache/spark/pull/29320#issuecomment-667772011 > What's docs/img/pyspark-components.pptx for? It is for the image I used in the main page in case some people want to edit. There are other pptx files in `docs/img` a

[GitHub] [spark] yaooqinn commented on a change in pull request #29303: [SPARK-32492][SQL] Fulfill missing column meta information COLUMN_SIZE /DECIMAL_DIGITS/NUM_PREC_RADIX/ORDINAL_POSITION for thrif

2020-08-02 Thread GitBox
yaooqinn commented on a change in pull request #29303: URL: https://github.com/apache/spark/pull/29303#discussion_r464165400 ## File path: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetColumnsOperation.scala ## @@ -126,12 +124,52 @@ privat

[GitHub] [spark] agrawaldevesh commented on a change in pull request #29304: [SPARK-32494][SQL] Null Aware Anti Join Optimize Support Multi-Column

2020-08-02 Thread GitBox
agrawaldevesh commented on a change in pull request #29304: URL: https://github.com/apache/spark/pull/29304#discussion_r464165213 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala ## @@ -327,11 +327,27 @@ private[joins] object Unsa

[GitHub] [spark] HyukjinKwon commented on pull request #29323: [SPARK-32513][SQL] Rename classes/files with the Jdbc prefix to JDBC

2020-08-02 Thread GitBox
HyukjinKwon commented on pull request #29323: URL: https://github.com/apache/spark/pull/29323#issuecomment-667770803 @dongjoon-hyun, as you said `JdbcUtils` isn't an API and under the private package. I don't think we should block a PR for this reason. But sure I don't have a strong feelin

[GitHub] [spark] yanlin-Lynn commented on pull request #23369: [SPARK-26429][SS]add jdbc sink for Structured Streaming.

2020-08-02 Thread GitBox
yanlin-Lynn commented on pull request #23369: URL: https://github.com/apache/spark/pull/23369#issuecomment-667769454 > Hi @yanlin-Lynn is there a plan to support JDBC as a source type for the Structured Streaming API? No. I have added it in bahir project. --

[GitHub] [spark] leanken commented on a change in pull request #29304: [SPARK-32494][SQL] Null Aware Anti Join Optimize Support Multi-Column

2020-08-02 Thread GitBox
leanken commented on a change in pull request #29304: URL: https://github.com/apache/spark/pull/29304#discussion_r464159019 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala ## @@ -327,11 +327,27 @@ private[joins] object UnsafeHash

[GitHub] [spark] leanken commented on a change in pull request #29304: [SPARK-32494][SQL] Null Aware Anti Join Optimize Support Multi-Column

2020-08-02 Thread GitBox
leanken commented on a change in pull request #29304: URL: https://github.com/apache/spark/pull/29304#discussion_r464159019 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala ## @@ -327,11 +327,27 @@ private[joins] object UnsafeHash

[GitHub] [spark] maropu commented on a change in pull request #29303: [SPARK-32492][SQL] Fulfill missing column meta information COLUMN_SIZE /DECIMAL_DIGITS/NUM_PREC_RADIX/ORDINAL_POSITION for thrifts

2020-08-02 Thread GitBox
maropu commented on a change in pull request #29303: URL: https://github.com/apache/spark/pull/29303#discussion_r463592697 ## File path: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerWithSparkContextSuite.scala ## @@ -101,6 +104,79 @@

[GitHub] [spark] yaooqinn commented on a change in pull request #29303: [SPARK-32492][SQL] Fulfill missing column meta information COLUMN_SIZE /DECIMAL_DIGITS/NUM_PREC_RADIX/ORDINAL_POSITION for thrif

2020-08-02 Thread GitBox
yaooqinn commented on a change in pull request #29303: URL: https://github.com/apache/spark/pull/29303#discussion_r464157516 ## File path: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerWithSparkContextSuite.scala ## @@ -101,6 +104,79 @

[GitHub] [spark] leanken commented on a change in pull request #29304: [SPARK-32494][SQL] Null Aware Anti Join Optimize Support Multi-Column

2020-08-02 Thread GitBox
leanken commented on a change in pull request #29304: URL: https://github.com/apache/spark/pull/29304#discussion_r464157833 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastHashJoinExec.scala ## @@ -245,7 +244,7 @@ case class BroadcastHashJoi

[GitHub] [spark] leanken edited a comment on pull request #29304: [SPARK-32494][SQL] Null Aware Anti Join Optimize Support Multi-Column

2020-08-02 Thread GitBox
leanken edited a comment on pull request #29304: URL: https://github.com/apache/spark/pull/29304#issuecomment-667760805 > > let's say there is a record > > (1, null, 3) in probe side, if there is a (1,2,3) in build side, it's counted as `MATCH` in comparison. basically if i want to avoid

[GitHub] [spark] gatorsmile commented on a change in pull request #27066: [SPARK-31317][SQL] Add withField method to Column

2020-08-02 Thread GitBox
gatorsmile commented on a change in pull request #27066: URL: https://github.com/apache/spark/pull/27066#discussion_r464156494 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Column.scala ## @@ -871,6 +871,72 @@ class Column(val expr: Expression) extends Logging {

[GitHub] [spark] gatorsmile commented on a change in pull request #27066: [SPARK-31317][SQL] Add withField method to Column

2020-08-02 Thread GitBox
gatorsmile commented on a change in pull request #27066: URL: https://github.com/apache/spark/pull/27066#discussion_r464156306 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Column.scala ## @@ -871,6 +871,72 @@ class Column(val expr: Expression) extends Logging {

[GitHub] [spark] leanken edited a comment on pull request #29304: [SPARK-32494][SQL] Null Aware Anti Join Optimize Support Multi-Column

2020-08-02 Thread GitBox
leanken edited a comment on pull request #29304: URL: https://github.com/apache/spark/pull/29304#issuecomment-667760805 > > let's say there is a record > > (1, null, 3) in probe side, if there is a (1,2,3) in build side, it's counted as `MATCH` in comparison. basically if i want to avoid

[GitHub] [spark] AngersZhuuuu commented on pull request #29199: [SPARK-32403][SQL] Refactor current ScriptTransformationExec

2020-08-02 Thread GitBox
AngersZh commented on pull request #29199: URL: https://github.com/apache/spark/pull/29199#issuecomment-667762041 It‘s ok to merge this and start to work on follow step ? This is an automated message from the Apache Git S

[GitHub] [spark] leanken edited a comment on pull request #29304: [SPARK-32494][SQL] Null Aware Anti Join Optimize Support Multi-Column

2020-08-02 Thread GitBox
leanken edited a comment on pull request #29304: URL: https://github.com/apache/spark/pull/29304#issuecomment-667760805 > > let's say there is a record > > (1, null, 3) in probe side, if there is a (1,2,3) in build side, it's counted as `MATCH` in comparison. basically if i want to avoid

[GitHub] [spark] leanken commented on pull request #29304: [SPARK-32494][SQL] Null Aware Anti Join Optimize Support Multi-Column

2020-08-02 Thread GitBox
leanken commented on pull request #29304: URL: https://github.com/apache/spark/pull/29304#issuecomment-667760805 > > let's say there is a record > > (1, null, 3) in probe side, if there is a (1,2,3) in build side, it's counted as `MATCH` in comparison. basically if i want to avoid 0(M*N)

[GitHub] [spark] leanken commented on a change in pull request #29304: [SPARK-32494][SQL] Null Aware Anti Join Optimize Support Multi-Column

2020-08-02 Thread GitBox
leanken commented on a change in pull request #29304: URL: https://github.com/apache/spark/pull/29304#discussion_r464154490 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala ## @@ -391,35 +393,41 @@ object PhysicalWindow { } }

[GitHub] [spark] leanken commented on a change in pull request #29304: [SPARK-32494][SQL] Null Aware Anti Join Optimize Support Multi-Column

2020-08-02 Thread GitBox
leanken commented on a change in pull request #29304: URL: https://github.com/apache/spark/pull/29304#discussion_r464154277 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala ## @@ -391,35 +393,41 @@ object PhysicalWindow { } }

[GitHub] [spark] AmplabJenkins commented on pull request #29324: [SPARK-32402][SQL] Implement ALTER TABLE in JDBC Table Catalog

2020-08-02 Thread GitBox
AmplabJenkins commented on pull request #29324: URL: https://github.com/apache/spark/pull/29324#issuecomment-667757606 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29324: [SPARK-32402][SQL] Implement ALTER TABLE in JDBC Table Catalog

2020-08-02 Thread GitBox
AmplabJenkins removed a comment on pull request #29324: URL: https://github.com/apache/spark/pull/29324#issuecomment-667757606 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA removed a comment on pull request #29324: [SPARK-32402][SQL] Implement ALTER TABLE in JDBC Table Catalog

2020-08-02 Thread GitBox
SparkQA removed a comment on pull request #29324: URL: https://github.com/apache/spark/pull/29324#issuecomment-667726106 **[Test build #126946 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126946/testReport)** for PR 29324 at commit [`3672250`](https://gi

[GitHub] [spark] SparkQA commented on pull request #29324: [SPARK-32402][SQL] Implement ALTER TABLE in JDBC Table Catalog

2020-08-02 Thread GitBox
SparkQA commented on pull request #29324: URL: https://github.com/apache/spark/pull/29324#issuecomment-667757139 **[Test build #126946 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126946/testReport)** for PR 29324 at commit [`3672250`](https://github.co

[GitHub] [spark] maropu commented on a change in pull request #29304: [SPARK-32494][SQL] Null Aware Anti Join Optimize Support Multi-Column

2020-08-02 Thread GitBox
maropu commented on a change in pull request #29304: URL: https://github.com/apache/spark/pull/29304#discussion_r464149263 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala ## @@ -391,35 +393,41 @@ object PhysicalWindow { } }

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29322: [SPARK-32511][SQL] Add dropFields method to Column class

2020-08-02 Thread GitBox
AmplabJenkins removed a comment on pull request #29322: URL: https://github.com/apache/spark/pull/29322#issuecomment-667749023 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29322: [SPARK-32511][SQL] Add dropFields method to Column class

2020-08-02 Thread GitBox
AmplabJenkins commented on pull request #29322: URL: https://github.com/apache/spark/pull/29322#issuecomment-667749023 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #29322: [SPARK-32511][SQL] Add dropFields method to Column class

2020-08-02 Thread GitBox
SparkQA commented on pull request #29322: URL: https://github.com/apache/spark/pull/29322#issuecomment-667748670 **[Test build #126947 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126947/testReport)** for PR 29322 at commit [`19587e8`](https://github.com

[GitHub] [spark] imback82 edited a comment on pull request #29328: [WIP][SPARK-32516][SQL] 'path' option should be treated consistently when loading dataframes for different APIs

2020-08-02 Thread GitBox
imback82 edited a comment on pull request #29328: URL: https://github.com/apache/spark/pull/29328#issuecomment-667748032 FYI, the existing test `"SPARK-11544 test pathfilter"` fails if I change ``` assert(spark.read.options(extraOptions).json(path).count() === 2) ``` to ```

[GitHub] [spark] imback82 commented on pull request #29328: [WIP][SPARK-32516][SQL] 'path' option should be treated consistently when loading dataframes for different APIs

2020-08-02 Thread GitBox
imback82 commented on pull request #29328: URL: https://github.com/apache/spark/pull/29328#issuecomment-667748032 The existing test `"SPARK-11544 test pathfilter"` fails if I change ``` assert(spark.read.options(extraOptions).json(path).count() === 2) ``` to ``` assert(spar

[GitHub] [spark] maropu commented on pull request #29323: [SPARK-32513][SQL] Rename classes/files with the Jdbc prefix to JDBC

2020-08-02 Thread GitBox
maropu commented on pull request #29323: URL: https://github.com/apache/spark/pull/29323#issuecomment-667747580 Ah, I missed that consideration..., thanks, @dongjoon-hyun. As you suggested above, we cannot rename the class names in the `main` codebase.

[GitHub] [spark] fqaiser94 commented on pull request #29322: [SPARK-32511][SQL] Add dropFields method to Column class

2020-08-02 Thread GitBox
fqaiser94 commented on pull request #29322: URL: https://github.com/apache/spark/pull/29322#issuecomment-667747506 retest this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] github-actions[bot] closed pull request #28278: "[SPARK-29274][SQL] Fix compare bigdecimal and string, string cast to…

2020-08-02 Thread GitBox
github-actions[bot] closed pull request #28278: URL: https://github.com/apache/spark/pull/28278 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

[GitHub] [spark] github-actions[bot] closed pull request #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty

2020-08-02 Thread GitBox
github-actions[bot] closed pull request #27861: URL: https://github.com/apache/spark/pull/27861 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

[GitHub] [spark] dongjoon-hyun commented on pull request #29327: [SPARK-28818][SQL][2.4] Respect source column nullability in the arrays created by `freqItems()`

2020-08-02 Thread GitBox
dongjoon-hyun commented on pull request #29327: URL: https://github.com/apache/spark/pull/29327#issuecomment-667746538 Sure, @srowen . We respect both efforts. That's the reason why we introduced the `lead-author` and `co-author` into our official `merge-script`. Now, we have a better way

[GitHub] [spark] dongjoon-hyun commented on pull request #29327: [SPARK-28818][SQL][2.4] Respect source column nullability in the arrays created by `freqItems()`

2020-08-02 Thread GitBox
dongjoon-hyun commented on pull request #29327: URL: https://github.com/apache/spark/pull/29327#issuecomment-667745701 Thank you, @maropu . I checked. This is an automated message from the Apache Git Service. To respond to th

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29323: [SPARK-32513][SQL] Rename classes/files with the Jdbc prefix to JDBC

2020-08-02 Thread GitBox
dongjoon-hyun commented on a change in pull request #29323: URL: https://github.com/apache/spark/pull/29323#discussion_r464140666 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCUtils.scala ## @@ -44,7 +44,7 @@ import org.apache.spark.

[GitHub] [spark] maropu commented on pull request #25575: [SPARK-28818][SQL] Respect source column nullability in the arrays created by `freqItems()`

2020-08-02 Thread GitBox
maropu commented on pull request #25575: URL: https://github.com/apache/spark/pull/25575#issuecomment-667742753 @dot-vlad I've merged this fix into branch-2.4, so it will be included in the next 2.4.7 release. Enjoy! This is

[GitHub] [spark] maropu commented on pull request #29327: [SPARK-28818][SQL][2.4] Respect source column nullability in the arrays created by `freqItems()`

2020-08-02 Thread GitBox
maropu commented on pull request #29327: URL: https://github.com/apache/spark/pull/29327#issuecomment-667742152 Merged to branch-2.4 (NOTE: I checked that the lead author in the commit has been rewritten) This is an automate

[GitHub] [spark] srowen commented on pull request #29327: [SPARK-28818][SQL][2.4] Respect source column nullability in the arrays created by `freqItems()`

2020-08-02 Thread GitBox
srowen commented on pull request #29327: URL: https://github.com/apache/spark/pull/29327#issuecomment-667742024 OK, I mean, I'd also consider @maropu the author of this _backport_. Sure cherry-pick takes care of that but it also kind of loses some info about who did the conflict resolution

[GitHub] [spark] maropu edited a comment on pull request #29327: [SPARK-28818][SQL][2.4] Respect source column nullability in the arrays created by `freqItems()`

2020-08-02 Thread GitBox
maropu edited a comment on pull request #29327: URL: https://github.com/apache/spark/pull/29327#issuecomment-667740060 I have no intention of being a main-author since this is the same with the original one, so I'll rewrite the author in this commit and merge this into branch-2.4. --

[GitHub] [spark] maropu closed pull request #29327: [SPARK-28818][SQL][2.4] Respect source column nullability in the arrays created by `freqItems()`

2020-08-02 Thread GitBox
maropu closed pull request #29327: URL: https://github.com/apache/spark/pull/29327 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [spark] maropu commented on pull request #29327: [SPARK-28818][SQL][2.4] Respect source column nullability in the arrays created by `freqItems()`

2020-08-02 Thread GitBox
maropu commented on pull request #29327: URL: https://github.com/apache/spark/pull/29327#issuecomment-667741185 okay, thanks~, @dongjoon-hyun This is an automated message from the Apache Git Service. To respond to the messag

[GitHub] [spark] dongjoon-hyun closed pull request #29299: [SPARK-32490][BUILD] Upgrade netty-all to 4.1.51.Final

2020-08-02 Thread GitBox
dongjoon-hyun closed pull request #29299: URL: https://github.com/apache/spark/pull/29299 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29328: [WIP][SPARK-32516][SQL] 'path' option should be treated consistently when loading dataframes for different APIs

2020-08-02 Thread GitBox
AmplabJenkins removed a comment on pull request #29328: URL: https://github.com/apache/spark/pull/29328#issuecomment-667738245 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #29327: [SPARK-28818][SQL][2.4] Respect source column nullability in the arrays created by `freqItems()`

2020-08-02 Thread GitBox
dongjoon-hyun edited a comment on pull request #29327: URL: https://github.com/apache/spark/pull/29327#issuecomment-667740151 Yep, but please make it sure next time that every committer can merge your PR without noticing the authorship change. -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29328: [WIP][SPARK-32516][SQL] 'path' option should be treated consistently when loading dataframes for different APIs

2020-08-02 Thread GitBox
AmplabJenkins removed a comment on pull request #29328: URL: https://github.com/apache/spark/pull/29328#issuecomment-667738241 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] dongjoon-hyun commented on pull request #29327: [SPARK-28818][SQL][2.4] Respect source column nullability in the arrays created by `freqItems()`

2020-08-02 Thread GitBox
dongjoon-hyun commented on pull request #29327: URL: https://github.com/apache/spark/pull/29327#issuecomment-667740151 Yep, but please make it sure. Every committer can merge your PR without noticing the authorship change. T

[GitHub] [spark] SparkQA removed a comment on pull request #29328: [WIP][SPARK-32516][SQL] 'path' option should be treated consistently when loading dataframes for different APIs

2020-08-02 Thread GitBox
SparkQA removed a comment on pull request #29328: URL: https://github.com/apache/spark/pull/29328#issuecomment-667726098 **[Test build #126945 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126945/testReport)** for PR 29328 at commit [`63f3d60`](https://gi

[GitHub] [spark] maropu commented on pull request #29327: [SPARK-28818][SQL][2.4] Respect source column nullability in the arrays created by `freqItems()`

2020-08-02 Thread GitBox
maropu commented on pull request #29327: URL: https://github.com/apache/spark/pull/29327#issuecomment-667740060 I have no intention of being a co-author since this is the same with the original one, so I'll rewrite the author in this commit and merge this into branch-2.4. ---

[GitHub] [spark] dongjoon-hyun commented on pull request #29327: [SPARK-28818][SQL][2.4] Respect source column nullability in the arrays created by `freqItems()`

2020-08-02 Thread GitBox
dongjoon-hyun commented on pull request #29327: URL: https://github.com/apache/spark/pull/29327#issuecomment-667739987 Yes. That's the correct way which I expected in order to prevent any accidental merging. > Ur...did I have to cherry-pick the original commit and resolve the conflict t

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #29327: [SPARK-28818][SQL][2.4] Respect source column nullability in the arrays created by `freqItems()`

2020-08-02 Thread GitBox
dongjoon-hyun edited a comment on pull request #29327: URL: https://github.com/apache/spark/pull/29327#issuecomment-667739987 Yes. That's the correct way which I expected in order to prevent any accidental merging. > Ur...did I have to cherry-pick the original commit and resolve the con

[GitHub] [spark] maropu commented on pull request #29327: [SPARK-28818][SQL][2.4] Respect source column nullability in the arrays created by `freqItems()`

2020-08-02 Thread GitBox
maropu commented on pull request #29327: URL: https://github.com/apache/spark/pull/29327#issuecomment-667739838 Ur...did I have to cherry-pick the original commit and resolve the conflict to keep the original author? I might misunderstant it though. ---

[GitHub] [spark] dongjoon-hyun commented on pull request #29327: [SPARK-28818][SQL][2.4] Respect source column nullability in the arrays created by `freqItems()`

2020-08-02 Thread GitBox
dongjoon-hyun commented on pull request #29327: URL: https://github.com/apache/spark/pull/29327#issuecomment-667739730 That's right~ Please do that, @maropu . This is an automated message from the Apache Git Service. To resp

[GitHub] [spark] maropu edited a comment on pull request #29327: [SPARK-28818][SQL][2.4] Respect source column nullability in the arrays created by `freqItems()`

2020-08-02 Thread GitBox
maropu edited a comment on pull request #29327: URL: https://github.com/apache/spark/pull/29327#issuecomment-667739006 Thanks for the check, @dongjoon-hyun and @srowen . I was thinking we can rewrite the author by the merge scprit. Am I wrong? So, I wrote `Note: This is the backport PR of

[GitHub] [spark] maropu commented on pull request #29327: [SPARK-28818][SQL][2.4] Respect source column nullability in the arrays created by `freqItems()`

2020-08-02 Thread GitBox
maropu commented on pull request #29327: URL: https://github.com/apache/spark/pull/29327#issuecomment-667739006 Thanks for the check, @dongjoon-hyun and @srowen . I was thinking we can rewrite the author by the merge scprit. Am I wrong?

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #29327: [SPARK-28818][SQL][2.4] Respect source column nullability in the arrays created by `freqItems()`

2020-08-02 Thread GitBox
dongjoon-hyun edited a comment on pull request #29327: URL: https://github.com/apache/spark/pull/29327#issuecomment-667738644 The commit of this PR, @srowen . - https://github.com/apache/spark/pull/29327/commits/44d8afea926c4a2137815d5ed880588987fd0f84 If you merge this PR, the me

[GitHub] [spark] dongjoon-hyun commented on pull request #29327: [SPARK-28818][SQL][2.4] Respect source column nullability in the arrays created by `freqItems()`

2020-08-02 Thread GitBox
dongjoon-hyun commented on pull request #29327: URL: https://github.com/apache/spark/pull/29327#issuecomment-667738644 The commit of this PR, @srowen . This is an automated message from the Apache Git Service. To respond to t

[GitHub] [spark] maropu commented on a change in pull request #29304: [SPARK-32494][SQL] Null Aware Anti Join Optimize Support Multi-Column

2020-08-02 Thread GitBox
maropu commented on a change in pull request #29304: URL: https://github.com/apache/spark/pull/29304#discussion_r464135912 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastHashJoinExec.scala ## @@ -245,7 +244,7 @@ case class BroadcastHashJoin

[GitHub] [spark] agrawaldevesh commented on a change in pull request #29304: [SPARK-32494][SQL] Null Aware Anti Join Optimize Support Multi-Column

2020-08-02 Thread GitBox
agrawaldevesh commented on a change in pull request #29304: URL: https://github.com/apache/spark/pull/29304#discussion_r464135838 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala ## @@ -327,11 +327,27 @@ private[joins] object Unsa

[GitHub] [spark] AmplabJenkins commented on pull request #29328: [SPARK-32516][SQL] 'path' option should be treated consistently when loading dataframes for different APIs

2020-08-02 Thread GitBox
AmplabJenkins commented on pull request #29328: URL: https://github.com/apache/spark/pull/29328#issuecomment-667738241 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #29328: [SPARK-32516][SQL] 'path' option should be treated consistently when loading dataframes for different APIs

2020-08-02 Thread GitBox
SparkQA commented on pull request #29328: URL: https://github.com/apache/spark/pull/29328#issuecomment-667738197 **[Test build #126945 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126945/testReport)** for PR 29328 at commit [`63f3d60`](https://github.co

[GitHub] [spark] maropu commented on a change in pull request #29304: [SPARK-32494][SQL] Null Aware Anti Join Optimize Support Multi-Column

2020-08-02 Thread GitBox
maropu commented on a change in pull request #29304: URL: https://github.com/apache/spark/pull/29304#discussion_r464134743 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala ## @@ -327,11 +327,27 @@ private[joins] object UnsafeHashe

[GitHub] [spark] maropu commented on a change in pull request #29304: [SPARK-32494][SQL] Null Aware Anti Join Optimize Support Multi-Column

2020-08-02 Thread GitBox
maropu commented on a change in pull request #29304: URL: https://github.com/apache/spark/pull/29304#discussion_r464134743 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala ## @@ -327,11 +327,27 @@ private[joins] object UnsafeHashe

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29322: [SPARK-32511][SQL] Add dropFields method to Column class

2020-08-02 Thread GitBox
AmplabJenkins removed a comment on pull request #29322: URL: https://github.com/apache/spark/pull/29322#issuecomment-667734634 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29322: [SPARK-32511][SQL] Add dropFields method to Column class

2020-08-02 Thread GitBox
AmplabJenkins removed a comment on pull request #29322: URL: https://github.com/apache/spark/pull/29322#issuecomment-667734633 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] SparkQA removed a comment on pull request #29322: [SPARK-32511][SQL] Add dropFields method to Column class

2020-08-02 Thread GitBox
SparkQA removed a comment on pull request #29322: URL: https://github.com/apache/spark/pull/29322#issuecomment-667711910 **[Test build #126944 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126944/testReport)** for PR 29322 at commit [`19587e8`](https://gi

[GitHub] [spark] AmplabJenkins commented on pull request #29322: [SPARK-32511][SQL] Add dropFields method to Column class

2020-08-02 Thread GitBox
AmplabJenkins commented on pull request #29322: URL: https://github.com/apache/spark/pull/29322#issuecomment-667734633 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #29322: [SPARK-32511][SQL] Add dropFields method to Column class

2020-08-02 Thread GitBox
SparkQA commented on pull request #29322: URL: https://github.com/apache/spark/pull/29322#issuecomment-667734521 **[Test build #126944 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126944/testReport)** for PR 29322 at commit [`19587e8`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

2020-08-02 Thread GitBox
SparkQA removed a comment on pull request #28715: URL: https://github.com/apache/spark/pull/28715#issuecomment-667706833 **[Test build #126942 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126942/testReport)** for PR 28715 at commit [`d6f8459`](https://gi

[GitHub] [spark] AmplabJenkins removed a comment on pull request #27507: [SPARK-24884][SQL] Support regexp function regexp_extract_all

2020-08-02 Thread GitBox
AmplabJenkins removed a comment on pull request #27507: URL: https://github.com/apache/spark/pull/27507#issuecomment-667733395 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

2020-08-02 Thread GitBox
AmplabJenkins removed a comment on pull request #28715: URL: https://github.com/apache/spark/pull/28715#issuecomment-667731372 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA removed a comment on pull request #27507: [SPARK-24884][SQL] Support regexp function regexp_extract_all

2020-08-02 Thread GitBox
SparkQA removed a comment on pull request #27507: URL: https://github.com/apache/spark/pull/27507#issuecomment-667707851 **[Test build #126943 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126943/testReport)** for PR 27507 at commit [`2125bff`](https://gi

[GitHub] [spark] AmplabJenkins commented on pull request #27507: [SPARK-24884][SQL] Support regexp function regexp_extract_all

2020-08-02 Thread GitBox
AmplabJenkins commented on pull request #27507: URL: https://github.com/apache/spark/pull/27507#issuecomment-667733392 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #27507: [SPARK-24884][SQL] Support regexp function regexp_extract_all

2020-08-02 Thread GitBox
SparkQA commented on pull request #27507: URL: https://github.com/apache/spark/pull/27507#issuecomment-667733197 **[Test build #126943 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126943/testReport)** for PR 27507 at commit [`2125bff`](https://github.co

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

2020-08-02 Thread GitBox
AmplabJenkins commented on pull request #28715: URL: https://github.com/apache/spark/pull/28715#issuecomment-667731372 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

2020-08-02 Thread GitBox
SparkQA commented on pull request #28715: URL: https://github.com/apache/spark/pull/28715#issuecomment-667731207 **[Test build #126942 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126942/testReport)** for PR 28715 at commit [`d6f8459`](https://github.co

[GitHub] [spark] imback82 edited a comment on pull request #29328: [SPARK-32516][SQL] 'path' option should be treated consistently when loading dataframes for different APIs

2020-08-02 Thread GitBox
imback82 edited a comment on pull request #29328: URL: https://github.com/apache/spark/pull/29328#issuecomment-667725762 I also see that `path` is treated different depending on the number of paths. For example, ``` // For single path, "path" option is overridden with path given in "l

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29328: [SPARK-32516][SQL] 'path' option should be treated consistently when loading dataframes for different APIs

2020-08-02 Thread GitBox
AmplabJenkins removed a comment on pull request #29328: URL: https://github.com/apache/spark/pull/29328#issuecomment-667726332 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29324: [SPARK-32402][SQL] Implement ALTER TABLE in JDBC Table Catalog

2020-08-02 Thread GitBox
AmplabJenkins removed a comment on pull request #29324: URL: https://github.com/apache/spark/pull/29324#issuecomment-667726306 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] srowen commented on pull request #29327: [SPARK-28818][SQL][2.4] Respect source column nullability in the arrays created by `freqItems()`

2020-08-02 Thread GitBox
srowen commented on pull request #29327: URL: https://github.com/apache/spark/pull/29327#issuecomment-667726530 What are you referring to here @dongjoon-hyun ... the requester and author do not seem to be available to do this work. The JIRA remains credited to the origin author. The origin

<    1   2   3   4   >