[GitHub] [spark] AmplabJenkins removed a comment on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

AmplabJenkins removed a comment on issue #25322: [SPARK-28586][INFRA] Make 
merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517147701
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

AmplabJenkins removed a comment on issue #25322: [SPARK-28586][INFRA] Make 
merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517147716
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13600/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

AmplabJenkins commented on issue #25322: [SPARK-28586][INFRA] Make 
merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517147716
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13600/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

AmplabJenkins commented on issue #25322: [SPARK-28586][INFRA] Make 
merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517147701
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] cloud-fan closed pull request #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result

[GitBox]

cloud-fan closed pull request #21599: [SPARK-26218][SQL] Overflow on arithmetic 
operations returns incorrect result
URL: https://github.com/apache/spark/pull/21599
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

SparkQA commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr 
script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517145794
 
 
   **[Test build #108506 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108506/testReport)**
 for PR 25322 at commit 
[`11c3c7f`](https://github.com/apache/spark/commit/11c3c7fbc52b5a678f19e7509f402d0a2b210ac7).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] cloud-fan commented on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result

[GitBox]

cloud-fan commented on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic 
operations returns incorrect result
URL: https://github.com/apache/spark/pull/21599#issuecomment-517145656
 
 
   thanks, merging to master!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25235: [SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks blocking on BarrierTaskContext.barrier()

[GitBox]

SparkQA commented on issue #25235: [SPARK-28483][Core] Fix canceling a spark 
job using barrier mode but barrier tasks blocking on 
BarrierTaskContext.barrier()
URL: https://github.com/apache/spark/pull/25235#issuecomment-517143546
 
 
   **[Test build #108505 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108505/testReport)**
 for PR 25235 at commit 
[`2eda187`](https://github.com/apache/spark/commit/2eda1876337e65915f03464076a772ea809bd361).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25235: [SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks blocking on BarrierTaskContext.barrier()

[GitBox]

AmplabJenkins removed a comment on issue #25235: [SPARK-28483][Core] Fix 
canceling a spark job using barrier mode but barrier tasks blocking on 
BarrierTaskContext.barrier()
URL: https://github.com/apache/spark/pull/25235#issuecomment-517142959
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13599/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25235: [SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks blocking on BarrierTaskContext.barrier()

[GitBox]

AmplabJenkins removed a comment on issue #25235: [SPARK-28483][Core] Fix 
canceling a spark job using barrier mode but barrier tasks blocking on 
BarrierTaskContext.barrier()
URL: https://github.com/apache/spark/pull/25235#issuecomment-517142954
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25235: [SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks blocking on BarrierTaskContext.barrier()

[GitBox]

AmplabJenkins commented on issue #25235: [SPARK-28483][Core] Fix canceling a 
spark job using barrier mode but barrier tasks blocking on 
BarrierTaskContext.barrier()
URL: https://github.com/apache/spark/pull/25235#issuecomment-517142959
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13599/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25235: [SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks blocking on BarrierTaskContext.barrier()

[GitBox]

AmplabJenkins commented on issue #25235: [SPARK-28483][Core] Fix canceling a 
spark job using barrier mode but barrier tasks blocking on 
BarrierTaskContext.barrier()
URL: https://github.com/apache/spark/pull/25235#issuecomment-517142954
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

dongjoon-hyun commented on a change in pull request #25322: 
[SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#discussion_r309543065
 
 

 ##
 File path: dev/merge_spark_pr.py
 ##
 @@ -92,9 +100,9 @@ def fail(msg):
 def run_cmd(cmd):
 print(cmd)
 if isinstance(cmd, list):
-return subprocess.check_output(cmd)
+return subprocess.check_output(cmd).decode(sys.stdout.encoding)
 else:
-return subprocess.check_output(cmd.split(" "))
+return subprocess.check_output(cmd.split(" 
")).decode(sys.stdout.encoding)
 
 Review comment:
   Got it~


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

dongjoon-hyun commented on a change in pull request #25322: 
[SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#discussion_r309542929
 
 

 ##
 File path: dev/merge_spark_pr.py
 ##
 @@ -31,7 +31,15 @@
 import subprocess
 import sys
 import traceback
-import urllib2
+try:
+from urllib2 import urlopen
+from urllib2 import Request
+from urllib2 import HTTPError
+except ImportError:
+# Python 3
+from urllib.request import urlopen
+from urllib.request import Request
+from urllib.error import HTTPError
 
 Review comment:
   Instead of `try`, shall we move this to `if` statement at [line 
50](https://github.com/apache/spark/pull/25322/files#diff-c321b6c82ebb21d8fd225abea9b7b74cR50)
 since we already have a branch for versioning?
   ```python
   if sys.version < '3':
   input = raw_input  # noqa
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

dongjoon-hyun commented on a change in pull request #25322: 
[SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#discussion_r309542929
 
 

 ##
 File path: dev/merge_spark_pr.py
 ##
 @@ -31,7 +31,15 @@
 import subprocess
 import sys
 import traceback
-import urllib2
+try:
+from urllib2 import urlopen
+from urllib2 import Request
+from urllib2 import HTTPError
+except ImportError:
+# Python 3
+from urllib.request import urlopen
+from urllib.request import Request
+from urllib.error import HTTPError
 
 Review comment:
   Instead of `try`, shall we move this to `if` statement at line 50 since we 
already have a branch for versioning?
   ```python
   if sys.version < '3':
   input = raw_input  # noqa
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] MaxGekk commented on a change in pull request #25310: [SPARK-28578][INFRA] Improve Github pull request template

[GitBox]

MaxGekk commented on a change in pull request #25310: [SPARK-28578][INFRA] 
Improve Github pull request template
URL: https://github.com/apache/spark/pull/25310#discussion_r309541321
 
 

 ##
 File path: .github/PULL_REQUEST_TEMPLATE
 ##
 @@ -1,10 +1,32 @@
-## What changes were proposed in this pull request?
+
 
-(Please fill in changes proposed in this fix)
+### What changes were proposed in this pull request?
 
-## How was this patch tested?
+### Why changes are needed?
 
-(Please explain how this patch was tested. E.g. unit tests, integration tests, 
manual tests)
-(If this patch involves UI changes, please attach a screenshot; otherwise, 
remove this)
+### Special notes for your reviewer
+

[GitHub] [spark] MaxGekk commented on a change in pull request #25310: [SPARK-28578][INFRA] Improve Github pull request template

[GitBox]

MaxGekk commented on a change in pull request #25310: [SPARK-28578][INFRA] 
Improve Github pull request template
URL: https://github.com/apache/spark/pull/25310#discussion_r309540255
 
 

 ##
 File path: .github/PULL_REQUEST_TEMPLATE
 ##
 @@ -1,10 +1,32 @@
-## What changes were proposed in this pull request?
+
 
-(Please fill in changes proposed in this fix)
+### What changes were proposed in this pull request?
 
-## How was this patch tested?
+### Why changes are needed?
 
-(Please explain how this patch was tested. E.g. unit tests, integration tests, 
manual tests)
-(If this patch involves UI changes, please attach a screenshot; otherwise, 
remove this)
+### Special notes for your reviewer
+
 
-Please review https://spark.apache.org/contributing.html before opening a pull 
request.
+### Does this PR introduce any user-facing change?
+
+
+### How was this patch tested?
+

[GitHub] [spark] cloud-fan commented on issue #25264: [SPARK-28213][SQL][followup] code cleanup and bug fix for columnar execution framework

[GitBox]

cloud-fan commented on issue #25264: [SPARK-28213][SQL][followup] code cleanup 
and bug fix for columnar execution framework
URL: https://github.com/apache/spark/pull/25264#issuecomment-517139578
 
 
   @tgravescs no I'm not going to. It's correct that `InputAdapter` can support 
columnar execution if its child supports, by calling `child.doExecuteColumnar`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

AmplabJenkins removed a comment on issue #25322: [SPARK-28586][INFRA] Make 
merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517138769
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13598/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

AmplabJenkins removed a comment on issue #25322: [SPARK-28586][INFRA] Make 
merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517138763
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

AmplabJenkins commented on issue #25322: [SPARK-28586][INFRA] Make 
merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517138763
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

AmplabJenkins commented on issue #25322: [SPARK-28586][INFRA] Make 
merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517138769
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13598/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

HyukjinKwon commented on a change in pull request #25322: [SPARK-28586][INFRA] 
Make merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#discussion_r309538697
 
 

 ##
 File path: dev/merge_spark_pr.py
 ##
 @@ -92,9 +100,9 @@ def fail(msg):
 def run_cmd(cmd):
 print(cmd)
 if isinstance(cmd, list):
-return subprocess.check_output(cmd)
+return subprocess.check_output(cmd).decode(sys.stdout.encoding)
 else:
-return subprocess.check_output(cmd.split(" "))
+return subprocess.check_output(cmd.split(" 
")).decode(sys.stdout.encoding)
 
 Review comment:
   Yup, to use the system default encoding rather than utf-8 just in case some 
machines have other encodings. I roughly tested this too. (most of guys' 
default encoding will be utf-8 and mind too).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

HyukjinKwon commented on a change in pull request #25322: [SPARK-28586][INFRA] 
Make merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#discussion_r309538697
 
 

 ##
 File path: dev/merge_spark_pr.py
 ##
 @@ -92,9 +100,9 @@ def fail(msg):
 def run_cmd(cmd):
 print(cmd)
 if isinstance(cmd, list):
-return subprocess.check_output(cmd)
+return subprocess.check_output(cmd).decode(sys.stdout.encoding)
 else:
-return subprocess.check_output(cmd.split(" "))
+return subprocess.check_output(cmd.split(" 
")).decode(sys.stdout.encoding)
 
 Review comment:
   Yup, to use the system default encoding rather than utf-8 just in case some 
machines have other encodings. I roughly tested this too. (most of guys' 
default encoding will be utf-8 and mine too).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

HyukjinKwon commented on a change in pull request #25322: [SPARK-28586][INFRA] 
Make merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#discussion_r309538890
 
 

 ##
 File path: dev/merge_spark_pr.py
 ##
 @@ -92,9 +100,9 @@ def fail(msg):
 def run_cmd(cmd):
 print(cmd)
 if isinstance(cmd, list):
-return subprocess.check_output(cmd)
+return subprocess.check_output(cmd).decode(sys.stdout.encoding)
 else:
-return subprocess.check_output(cmd.split(" "))
+return subprocess.check_output(cmd.split(" 
")).decode(sys.stdout.encoding)
 
 Review comment:
   Like, if a machine or OS has a different encoding, its standard output 
shouldn't be decoded into unicodes by utf-8. I think here is only the 
OS-specific code one.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

HyukjinKwon commented on a change in pull request #25322: [SPARK-28586][INFRA] 
Make merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#discussion_r309538697
 
 

 ##
 File path: dev/merge_spark_pr.py
 ##
 @@ -92,9 +100,9 @@ def fail(msg):
 def run_cmd(cmd):
 print(cmd)
 if isinstance(cmd, list):
-return subprocess.check_output(cmd)
+return subprocess.check_output(cmd).decode(sys.stdout.encoding)
 else:
-return subprocess.check_output(cmd.split(" "))
+return subprocess.check_output(cmd.split(" 
")).decode(sys.stdout.encoding)
 
 Review comment:
   Yup, to use the system default encoding rather than utf-8 just in case some 
machines have other encodings. I roughly tested this too.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

dongjoon-hyun commented on a change in pull request #25322: 
[SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#discussion_r309537961
 
 

 ##
 File path: dev/merge_spark_pr.py
 ##
 @@ -92,9 +100,9 @@ def fail(msg):
 def run_cmd(cmd):
 print(cmd)
 if isinstance(cmd, list):
-return subprocess.check_output(cmd)
+return subprocess.check_output(cmd).decode(sys.stdout.encoding)
 else:
-return subprocess.check_output(cmd.split(" "))
+return subprocess.check_output(cmd.split(" 
")).decode(sys.stdout.encoding)
 
 Review comment:
   Is this changed?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

SparkQA commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr 
script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517137356
 
 
   **[Test build #108504 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108504/testReport)**
 for PR 25322 at commit 
[`9e139db`](https://github.com/apache/spark/commit/9e139dbaa1ddd44a712d018556925c18462541af).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25261: [SPARK-28521][SQL] Fix error message for built-in functions

[GitBox]

AmplabJenkins removed a comment on issue #25261: [SPARK-28521][SQL] Fix error 
message for built-in functions
URL: https://github.com/apache/spark/pull/25261#issuecomment-517135232
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108500/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25261: [SPARK-28521][SQL] Fix error message for built-in functions

[GitBox]

AmplabJenkins removed a comment on issue #25261: [SPARK-28521][SQL] Fix error 
message for built-in functions
URL: https://github.com/apache/spark/pull/25261#issuecomment-517135229
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25261: [SPARK-28521][SQL] Fix error message for built-in functions

[GitBox]

AmplabJenkins commented on issue #25261: [SPARK-28521][SQL] Fix error message 
for built-in functions
URL: https://github.com/apache/spark/pull/25261#issuecomment-517135229
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25261: [SPARK-28521][SQL] Fix error message for built-in functions

[GitBox]

AmplabJenkins commented on issue #25261: [SPARK-28521][SQL] Fix error message 
for built-in functions
URL: https://github.com/apache/spark/pull/25261#issuecomment-517135232
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108500/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #25261: [SPARK-28521][SQL] Fix error message for built-in functions

[GitBox]

SparkQA removed a comment on issue #25261: [SPARK-28521][SQL] Fix error message 
for built-in functions
URL: https://github.com/apache/spark/pull/25261#issuecomment-517092086
 
 
   **[Test build #108500 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108500/testReport)**
 for PR 25261 at commit 
[`3c46a84`](https://github.com/apache/spark/commit/3c46a84a4a847f0eb2e68b3f978dfeb5ec61f4df).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25261: [SPARK-28521][SQL] Fix error message for built-in functions

[GitBox]

SparkQA commented on issue #25261: [SPARK-28521][SQL] Fix error message for 
built-in functions
URL: https://github.com/apache/spark/pull/25261#issuecomment-517134804
 
 
   **[Test build #108500 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108500/testReport)**
 for PR 25261 at commit 
[`3c46a84`](https://github.com/apache/spark/commit/3c46a84a4a847f0eb2e68b3f978dfeb5ec61f4df).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

dongjoon-hyun commented on issue #25322: [SPARK-28586][INFRA] Make 
merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517133463
 
 
   Nice, @HyukjinKwon !


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] xuanyuanking commented on a change in pull request #24382: [SPARK-27330][SS] support task abort in foreach writer

[GitBox]

xuanyuanking commented on a change in pull request #24382: [SPARK-27330][SS] 
support task abort in foreach writer
URL: https://github.com/apache/spark/pull/24382#discussion_r309533983
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/ForeachWriterTable.scala
 ##
 @@ -141,17 +143,33 @@ class ForeachDataWriter[T](
   writer.process(rowConverter(record))
 } catch {
   case t: Throwable =>
-writer.close(t)
+closeWriter(t)
 throw t
 }
   }
 
   override def commit(): WriterCommitMessage = {
-writer.close(null)
+closeWriter(null)
 ForeachWriterCommitMessage
   }
 
-  override def abort(): Unit = {}
+  override def abort(): Unit = {
 
 Review comment:
   Ah I see, thanks for your explanation Ryan.
   Gives a scenario of `DataWritingSparkTask`, both dataWriter.wirte, 
dataWriter.commit, and Spark task self could throw different exceptions. In the 
close API of DataWriter, I think `CommitDeniedException` and the exception 
threw by `writer.commit` can deal with separately. Please correct me if I'm 
wrong, Thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon edited a comment on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

HyukjinKwon edited a comment on issue #25322: [SPARK-28586][INFRA] Make 
merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517131664
 
 
   Let me cc Spark committers to inform although it might be a bit of noise .. 
because I know it's annoying to use this script with Python 2 in Mac due to 
some pre-installed lib issues. After this PR, we can now use Python 3 for this 
script. cc @apache/spark-committers.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

AmplabJenkins removed a comment on issue #25322: [SPARK-28586][INFRA] Make 
merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517131144
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13597/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

AmplabJenkins removed a comment on issue #25322: [SPARK-28586][INFRA] Make 
merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517131141
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

HyukjinKwon commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr 
script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517131664
 
 
   Let me cc Spark committers to inform although it might be a bit of noise .. 
because I know it's annoying to use Python 2 in Mac due to some pre-installed 
lib issues. After this PR, we can now use Python 3 for this script. cc 
@apache/spark-committers.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

SparkQA commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr 
script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517131560
 
 
   **[Test build #108503 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108503/testReport)**
 for PR 25322 at commit 
[`0d0fdc9`](https://github.com/apache/spark/commit/0d0fdc99b82e650a4fbc30dd5edbca5e48e6621a).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

AmplabJenkins commented on issue #25322: [SPARK-28586][INFRA] Make 
merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517131141
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

HyukjinKwon commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr 
script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517131263
 
 
   Yea, it's deprecated but not removed out yet.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

AmplabJenkins commented on issue #25322: [SPARK-28586][INFRA] Make 
merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517131144
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13597/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] maropu commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

maropu commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr 
script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517130856
 
 
   We still need to keep the Python2 support?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon opened a new pull request #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

[GitBox]

HyukjinKwon opened a new pull request #25322: [SPARK-28586][INFRA] Make 
merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322
 
 
   ## What changes were proposed in this pull request?
   
   This PR proposes to make `merge_spark_pr.py` script Python 3 compatible.
   
   ## How was this patch tested?
   
   Manually tested against my forked remote with the PR and JIRA below:
   
   https://github.com/apache/spark/pull/25321
   https://github.com/apache/spark/pull/25286
   https://issues.apache.org/jira/browse/SPARK-28153


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #24938: [SPARK-27946][SQL] Hive DDL to Spark DDL conversion USING "show create table"

[GitBox]

maropu commented on a change in pull request #24938: [SPARK-27946][SQL] Hive 
DDL to Spark DDL conversion USING "show create table"
URL: https://github.com/apache/spark/pull/24938#discussion_r309530837
 
 

 ##
 File path: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveShowCreateTableSuite.scala
 ##
 @@ -195,4 +203,214 @@ class HiveShowCreateTableSuite extends 
ShowCreateTableSuite with TestHiveSinglet
 
hiveContext.sharedState.externalCatalog.unwrapped.asInstanceOf[HiveExternalCatalog]
   .client.runSqlHive(ddl)
   }
+
+  private def checkCreateSparkTable(tableName: String): Unit = {
+val table = TableIdentifier(tableName, Some("default"))
+val db = table.database.get
+val hiveTable = spark.sharedState.externalCatalog.getTable(db, table.table)
+val shownSparkDDL = sql(s"SHOW CREATE TABLE ${table.quotedString} AS 
SPARK").head().getString(0)
+// Drops original Hive table.
+sql(s"DROP TABLE ${table.quotedString}")
+
+try {
+  sql(shownSparkDDL)
+  val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+  val shownDDL = sql(s"SHOW CREATE TABLE 
${table.quotedString}").head().getString(0)
+
+  // Drops created Spark table using `SHOW CREATE TABLE AS SPARK`.
+  sql(s"DROP TABLE ${table.quotedString}")
+
+  sql(shownDDL)
+  val expected = spark.sharedState.externalCatalog.getTable(db, 
table.table)
+
+  checkCatalogTables(expected, actual)
+  checkHiveCatalogTables(hiveTable, actual)
+} finally {
+  sql(s"DROP TABLE IF EXISTS ${table.table}")
+}
+  }
+
+  private def checkHiveCatalogTables(expected: CatalogTable, actual: 
CatalogTable): Unit = {
+def normalize(table: CatalogTable): CatalogTable = {
+  val nondeterministicProps = Set(
+"CreateTime",
+"transient_lastDdlTime",
+"grantTime",
+"lastUpdateTime",
+"last_modified_by",
+"last_modified_time",
+"Owner:",
+// The following are hive specific schema parameters which we do not 
need to match exactly.
+"totalNumberFiles",
+"maxFileSize",
+"minFileSize"
+  )
+
+  table.copy(
+createTime = 0L,
+lastAccessTime = 0L,
+properties = 
table.properties.filterKeys(!nondeterministicProps.contains(_)),
+stats = None,
+ignoredProperties = Map.empty,
+storage = CatalogStorageFormat.empty,
+provider = None,
+tracksPartitionsInCatalog = false
+  )
+}
+assert(normalize(actual) == normalize(expected))
+  }
+
+  test("simple hive table as spark") {
+withTable("t1") {
+  sql(
+s"""CREATE TABLE t1 (
 
 Review comment:
   nit: format issue? 
https://github.com/apache/spark/pull/25204#discussion_r307234253


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] gatorsmile commented on a change in pull request #25107: [SPARK-28344][SQL] detect ambiguous self-join and fail the query

[GitBox]

gatorsmile commented on a change in pull request #25107: [SPARK-28344][SQL] 
detect ambiguous self-join and fail the query
URL: https://github.com/apache/spark/pull/25107#discussion_r309530391
 
 

 ##
 File path: sql/core/src/main/scala/org/apache/spark/sql/Column.scala
 ##
 @@ -144,11 +153,16 @@ class Column(val expr: Expression) extends Logging {
   override def toString: String = toPrettySQL(expr)
 
   override def equals(that: Any): Boolean = that match {
-case that: Column => that.expr.equals(this.expr)
+case that: Column => that.normalizedExpr().equals(this.normalizedExpr())
 case _ => false
   }
 
-  override def hashCode: Int = this.expr.hashCode()
+  override def hashCode: Int = this.normalizedExpr().hashCode()
+
+  private def normalizedExpr(): Expression = expr match {
 
 Review comment:
   When we add the metadata in addDataFrameIdToCol, we use transform. Do we 
need to use it here when removing it? 
   
   Our test case can cover it?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] gatorsmile commented on issue #24938: [SPARK-27946][SQL] Hive DDL to Spark DDL conversion USING "show create table"

[GitBox]

gatorsmile commented on issue #24938: [SPARK-27946][SQL] Hive DDL to Spark DDL 
conversion USING "show create table"
URL: https://github.com/apache/spark/pull/24938#issuecomment-517125964
 
 
   cc @cloud-fan @gengliangwang
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] gatorsmile commented on a change in pull request #25294: [SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION

[GitBox]

gatorsmile commented on a change in pull request #25294: [SPARK-28476][SQL] 
Support ALTER DATABASE SET LOCATION
URL: https://github.com/apache/spark/pull/25294#discussion_r309527041
 
 

 ##
 File path: 
sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLParserSuite.scala
 ##
 @@ -184,6 +184,15 @@ class DDLParserSuite extends AnalysisTest with 
SharedSQLContext {
   containsThesePhrases = Seq("key_without_value"))
   }
 
+  test("alter database set location") {
+// ALTER (DATABASE|SCHEMA) database_name SET LOCATION
+val sql1 = "ALTER DATABASE database_name SET LOCATION '/home/user/db'"
 
 Review comment:
   1. A database that does not exist before issuing this command?
   2. Setting an illegal path that could trigger an exception inside the 
function `stringToURI`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on issue #25286: [SPARK-25584][ML][DOC] datasource for libsvm user guide

[GitBox]

HyukjinKwon commented on issue #25286: [SPARK-25584][ML][DOC] datasource for 
libsvm user guide
URL: https://github.com/apache/spark/pull/25286#issuecomment-517121115
 
 
   (ignore the logs above. I was testing merge script Python 3 compatibility)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] rdblue commented on a change in pull request #24382: [SPARK-27330][SS] support task abort in foreach writer

[GitBox]

rdblue commented on a change in pull request #24382: [SPARK-27330][SS] support 
task abort in foreach writer
URL: https://github.com/apache/spark/pull/24382#discussion_r309521630
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/ForeachWriterTable.scala
 ##
 @@ -141,17 +143,33 @@ class ForeachDataWriter[T](
   writer.process(rowConverter(record))
 } catch {
   case t: Throwable =>
-writer.close(t)
+closeWriter(t)
 throw t
 }
   }
 
   override def commit(): WriterCommitMessage = {
-writer.close(null)
+closeWriter(null)
 ForeachWriterCommitMessage
   }
 
-  override def abort(): Unit = {}
+  override def abort(): Unit = {
 
 Review comment:
   I can see that the exception is passed to close. My question is: what does 
the writer do differently based on the exception? If this is to satisfy an API 
and you can pass any exception, then I don't think it matters. There's effort 
required to get the exception to pass through and without a reasonable use case 
I'm wondering why it is necessary to do it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

[GitBox]

HyukjinKwon commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use 
AtomicReference at InputFileBlockHolder (to support input_file_name with Python 
UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517117629
 
 
   I am testing mergine script Python 3 compatibility. please ignore the noise 
above.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

[GitBox]

SparkQA commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use 
AtomicReference at InputFileBlockHolder (to support input_file_name with Python 
UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517113023
 
 
   **[Test build #108502 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108502/testReport)**
 for PR 25321 at commit 
[`4275f82`](https://github.com/apache/spark/commit/4275f8284d13c49a46a15e48aed08a4114201e7e).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25262: [SPARK-28486][CORE][PYTHON] Map PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC

[GitBox]

AmplabJenkins removed a comment on issue #25262: [SPARK-28486][CORE][PYTHON] 
Map PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC
URL: https://github.com/apache/spark/pull/25262#issuecomment-517112556
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

[GitBox]

AmplabJenkins removed a comment on issue #25321: 
[SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder 
(to support input_file_name with Python UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517112639
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13596/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

[GitBox]

AmplabJenkins commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use 
AtomicReference at InputFileBlockHolder (to support input_file_name with Python 
UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517112639
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13596/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

[GitBox]

AmplabJenkins removed a comment on issue #25321: 
[SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder 
(to support input_file_name with Python UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517112634
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

[GitBox]

AmplabJenkins commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use 
AtomicReference at InputFileBlockHolder (to support input_file_name with Python 
UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517112634
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25262: [SPARK-28486][CORE][PYTHON] Map PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC

[GitBox]

AmplabJenkins removed a comment on issue #25262: [SPARK-28486][CORE][PYTHON] 
Map PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC
URL: https://github.com/apache/spark/pull/25262#issuecomment-517112559
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108499/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25262: [SPARK-28486][CORE][PYTHON] Map PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC

[GitBox]

AmplabJenkins commented on issue #25262: [SPARK-28486][CORE][PYTHON] Map 
PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC
URL: https://github.com/apache/spark/pull/25262#issuecomment-517112556
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25262: [SPARK-28486][CORE][PYTHON] Map PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC

[GitBox]

AmplabJenkins commented on issue #25262: [SPARK-28486][CORE][PYTHON] Map 
PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC
URL: https://github.com/apache/spark/pull/25262#issuecomment-517112559
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108499/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #25262: [SPARK-28486][CORE][PYTHON] Map PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC

[GitBox]

SparkQA removed a comment on issue #25262: [SPARK-28486][CORE][PYTHON] Map 
PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC
URL: https://github.com/apache/spark/pull/25262#issuecomment-517082869
 
 
   **[Test build #108499 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108499/testReport)**
 for PR 25262 at commit 
[`5cdbab3`](https://github.com/apache/spark/commit/5cdbab3659edd219bff462ce4d004724cff68e10).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

[GitBox]

HyukjinKwon commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use 
AtomicReference at InputFileBlockHolder (to support input_file_name with Python 
UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517112094
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25262: [SPARK-28486][CORE][PYTHON] Map PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC

[GitBox]

SparkQA commented on issue #25262: [SPARK-28486][CORE][PYTHON] Map 
PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC
URL: https://github.com/apache/spark/pull/25262#issuecomment-517112137
 
 
   **[Test build #108499 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108499/testReport)**
 for PR 25262 at commit 
[`5cdbab3`](https://github.com/apache/spark/commit/5cdbab3659edd219bff462ce4d004724cff68e10).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] beliefer commented on a change in pull request #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array

[GitBox]

beliefer commented on a change in pull request #25172: [SPARK-28412][SQL] ANSI 
SQL: OVERLAY function support byte array
URL: https://github.com/apache/spark/pull/25172#discussion_r309516995
 
 

 ##
 File path: 
sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala
 ##
 @@ -141,6 +141,19 @@ class StringFunctionsSuite extends QueryTest with 
SharedSQLContext {
 // scalastyle:on
   }
 
+  test("binary overlay function") {
+// non ascii characters are not allowed in the code, so we disable the 
scalastyle here.
+val df = Seq((Array[Byte](1, 2, 3, 4, 5, 6, 7, 8, 9))).toDF("a")
+checkAnswer(df.select(overlay(
+  $"a", Array[Byte](-1), 6)), Row(Array[Byte](1, 2, 3, 4, 5, -1, 7, 8, 9)))
+checkAnswer(df.select(overlay(
+  $"a", Array[Byte](-1, -1, -1, -1), 7)), Row(Array[Byte](1, 2, 3, 4, 5, 
6, -1, -1, -1, -1)))
+checkAnswer(df.select(overlay(
+  $"a", Array[Byte](-1, -1), 7, 0)), Row(Array[Byte](1, 2, 3, 4, 5, 6, -1, 
-1, 7, 8, 9)))
+checkAnswer(df.select(overlay($"a", Array[Byte](-1, -1, -1, -1, -1), 2, 
4)),
+  Row(Array[Byte](1, -1, -1, -1, -1, -1, 6, 7, 8, 9)))
+  }
 
 Review comment:
   OK. Thanks for the decision. @maropu 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

[GitBox]

AmplabJenkins removed a comment on issue #25321: 
[SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder 
(to support input_file_name with Python UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517110949
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108497/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

[GitBox]

SparkQA removed a comment on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] 
Use AtomicReference at InputFileBlockHolder (to support input_file_name with 
Python UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517070221
 
 
   **[Test build #108497 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108497/testReport)**
 for PR 25321 at commit 
[`4275f82`](https://github.com/apache/spark/commit/4275f8284d13c49a46a15e48aed08a4114201e7e).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

[GitBox]

AmplabJenkins removed a comment on issue #25321: 
[SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder 
(to support input_file_name with Python UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517110943
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

[GitBox]

AmplabJenkins commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use 
AtomicReference at InputFileBlockHolder (to support input_file_name with Python 
UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517110949
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108497/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

[GitBox]

AmplabJenkins commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use 
AtomicReference at InputFileBlockHolder (to support input_file_name with Python 
UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517110943
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

[GitBox]

SparkQA commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use 
AtomicReference at InputFileBlockHolder (to support input_file_name with Python 
UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517110757
 
 
   **[Test build #108497 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108497/testReport)**
 for PR 25321 at commit 
[`4275f82`](https://github.com/apache/spark/commit/4275f8284d13c49a46a15e48aed08a4114201e7e).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] beliefer commented on a change in pull request #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array

[GitBox]

beliefer commented on a change in pull request #25172: [SPARK-28412][SQL] ANSI 
SQL: OVERLAY function support byte array
URL: https://github.com/apache/spark/pull/25172#discussion_r309515031
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
 ##
 @@ -496,19 +509,39 @@ case class Overlay(input: Expression, replace: 
Expression, pos: Expression, len:
 this(str, replace, pos, Literal.create(-1, IntegerType))
   }
 
-  override def dataType: DataType = StringType
+  override def dataType: DataType = input.dataType
 
-  override def inputTypes: Seq[AbstractDataType] =
-Seq(StringType, StringType, IntegerType, IntegerType)
+  override def inputTypes: Seq[AbstractDataType] = 
Seq(TypeCollection(StringType, BinaryType),
+TypeCollection(StringType, BinaryType), IntegerType, IntegerType)
 
   override def children: Seq[Expression] = input :: replace :: pos :: len :: 
Nil
 
+  override def checkInputDataTypes(): TypeCheckResult = {
+val (inputType, replaceType, posType, lenType) =
+  (input.dataType, replace.dataType, pos.dataType, len.dataType)
+(inputType, replaceType) match {
+  case (StringType, StringType) | (BinaryType, BinaryType) =>
+  case (inputType, replaceType) =>
+return TypeCheckResult.TypeCheckFailure(s"Invalid data type 
${inputType.simpleString}" +
+  s" and ${replaceType.simpleString}. The result type of the input 
expression and the" +
+  "replace expression are either String or Byte Array.")
+}
+if (IntegerType.acceptsType(posType) && IntegerType.acceptsType(lenType)) {
+  TypeCheckResult.TypeCheckSuccess
+} else {
+  TypeCheckResult.TypeCheckFailure("Pos expression and len expression must 
be integer type.")
+}
+  }
+
   override def nullSafeEval(inputEval: Any, replaceEval: Any, posEval: Any, 
lenEval: Any): Any = {
-val inputStr = inputEval.asInstanceOf[UTF8String]
-val replaceStr = replaceEval.asInstanceOf[UTF8String]
 val position = posEval.asInstanceOf[Int]
 val length = lenEval.asInstanceOf[Int]
-Overlay.calculate(inputStr, replaceStr, position, length)
+input.dataType match {
+  case StringType => Overlay.calculate(inputEval.asInstanceOf[UTF8String],
+replaceEval.asInstanceOf[UTF8String], position, length)
+  case BinaryType => Overlay.calculate(inputEval.asInstanceOf[Array[Byte]],
+replaceEval.asInstanceOf[Array[Byte]], position, length)
+}
 
 Review comment:
   IMHO, it seems no difference and more code. Why we need this change?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] viirya commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template

[GitBox]

viirya commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull 
request template
URL: https://github.com/apache/spark/pull/25310#issuecomment-517108202
 
 
   Thanks.
   
   Don't have stable Internet connection right now.
   
   May look into it and provide feedback tomorrow or later.
   
   On Wed, Jul 31, 2019, 20:50 Hyukjin Kwon  wrote:
   
   > Let me cc some more active contributors who didn't put input here for
   > better visibility and to collect more feedback. cc @viirya
   >  @mgaido91 
   > @MaxGekk  @HeartSaVioR
   >  @gaborgsomogyi
   >  @dilipbiswal
   > 
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or mute the thread
   > 

   > .
   >
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template

[GitBox]

SparkQA commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull 
request template
URL: https://github.com/apache/spark/pull/25310#issuecomment-517107671
 
 
   **[Test build #108501 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108501/testReport)**
 for PR 25310 at commit 
[`eb76d8d`](https://github.com/apache/spark/commit/eb76d8d947147b9c89f53ab609c49b06b9a66b9d).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template

[GitBox]

AmplabJenkins removed a comment on issue #25310: [SPARK-28578][INFRA] Improve 
Github pull request template
URL: https://github.com/apache/spark/pull/25310#issuecomment-517107269
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13595/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template

[GitBox]

AmplabJenkins removed a comment on issue #25310: [SPARK-28578][INFRA] Improve 
Github pull request template
URL: https://github.com/apache/spark/pull/25310#issuecomment-517107264
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template

[GitBox]

HyukjinKwon commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull 
request template
URL: https://github.com/apache/spark/pull/25310#issuecomment-517107290
 
 
   Let me cc some more active contributors who didn't put input here for better 
visibility and to collect more feedback. cc @viirya @mgaido91 @MaxGekk 
@HeartSaVioR @gaborgsomogyi @dilipbiswal 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template

[GitBox]

AmplabJenkins commented on issue #25310: [SPARK-28578][INFRA] Improve Github 
pull request template
URL: https://github.com/apache/spark/pull/25310#issuecomment-517107269
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13595/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template

[GitBox]

AmplabJenkins commented on issue #25310: [SPARK-28578][INFRA] Improve Github 
pull request template
URL: https://github.com/apache/spark/pull/25310#issuecomment-517107264
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array

[GitBox]

maropu commented on a change in pull request #25172: [SPARK-28412][SQL] ANSI 
SQL: OVERLAY function support byte array
URL: https://github.com/apache/spark/pull/25172#discussion_r309512454
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
 ##
 @@ -496,19 +509,39 @@ case class Overlay(input: Expression, replace: 
Expression, pos: Expression, len:
 this(str, replace, pos, Literal.create(-1, IntegerType))
   }
 
-  override def dataType: DataType = StringType
+  override def dataType: DataType = input.dataType
 
-  override def inputTypes: Seq[AbstractDataType] =
-Seq(StringType, StringType, IntegerType, IntegerType)
+  override def inputTypes: Seq[AbstractDataType] = 
Seq(TypeCollection(StringType, BinaryType),
+TypeCollection(StringType, BinaryType), IntegerType, IntegerType)
 
   override def children: Seq[Expression] = input :: replace :: pos :: len :: 
Nil
 
+  override def checkInputDataTypes(): TypeCheckResult = {
+val (inputType, replaceType, posType, lenType) =
+  (input.dataType, replace.dataType, pos.dataType, len.dataType)
+(inputType, replaceType) match {
+  case (StringType, StringType) | (BinaryType, BinaryType) =>
+  case (inputType, replaceType) =>
+return TypeCheckResult.TypeCheckFailure(s"Invalid data type 
${inputType.simpleString}" +
+  s" and ${replaceType.simpleString}. The result type of the input 
expression and the" +
+  "replace expression are either String or Byte Array.")
+}
+if (IntegerType.acceptsType(posType) && IntegerType.acceptsType(lenType)) {
+  TypeCheckResult.TypeCheckSuccess
+} else {
+  TypeCheckResult.TypeCheckFailure("Pos expression and len expression must 
be integer type.")
+}
+  }
+
   override def nullSafeEval(inputEval: Any, replaceEval: Any, posEval: Any, 
lenEval: Any): Any = {
-val inputStr = inputEval.asInstanceOf[UTF8String]
-val replaceStr = replaceEval.asInstanceOf[UTF8String]
 val position = posEval.asInstanceOf[Int]
 val length = lenEval.asInstanceOf[Int]
-Overlay.calculate(inputStr, replaceStr, position, length)
+input.dataType match {
+  case StringType => Overlay.calculate(inputEval.asInstanceOf[UTF8String],
+replaceEval.asInstanceOf[UTF8String], position, length)
+  case BinaryType => Overlay.calculate(inputEval.asInstanceOf[Array[Byte]],
+replaceEval.asInstanceOf[Array[Byte]], position, length)
+}
 
 Review comment:
   ```
 private lazy val replaceFunc = input.dataType match {
   case StringType =>
 (inputEval: Any, replaceEval: Any, posEval: Int, lenEval: Int) => {
   Overlay.calculate(
 inputEval.asInstanceOf[UTF8String],
 replaceEval.asInstanceOf[UTF8String],
 posEval, lenEval)
 }
   case BinaryType =>
 (inputEval: Any, replaceEval: Any, posEval: Int, lenEval: Int) => {
   Overlay.calculate(
 inputEval.asInstanceOf[Array[Byte]],
 replaceEval.asInstanceOf[Array[Byte]],
 posEval, lenEval)
 }
 }
   
 override def nullSafeEval(inputEval: Any, replaceEval: Any, posEval: Any, 
lenEval: Any): Any = {
   replaceFunc(inputEval, replaceEval, posEval.asInstanceOf[Int], 
lenEval.asInstanceOf[Int])
 }
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template

[GitBox]

HyukjinKwon commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull 
request template
URL: https://github.com/apache/spark/pull/25310#issuecomment-517105598
 
 
   I think basically JIRA describes what issue and PR describes how it is fixed.
   
   Each focuses on each's purpose but other information can be duplicated for 
better and faster reviews. I think it's case-by-case.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array

[GitBox]

maropu commented on a change in pull request #25172: [SPARK-28412][SQL] ANSI 
SQL: OVERLAY function support byte array
URL: https://github.com/apache/spark/pull/25172#discussion_r309511156
 
 

 ##
 File path: 
sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala
 ##
 @@ -141,6 +141,19 @@ class StringFunctionsSuite extends QueryTest with 
SharedSQLContext {
 // scalastyle:on
   }
 
+  test("binary overlay function") {
+// non ascii characters are not allowed in the code, so we disable the 
scalastyle here.
+val df = Seq((Array[Byte](1, 2, 3, 4, 5, 6, 7, 8, 9))).toDF("a")
+checkAnswer(df.select(overlay(
+  $"a", Array[Byte](-1), 6)), Row(Array[Byte](1, 2, 3, 4, 5, -1, 7, 8, 9)))
+checkAnswer(df.select(overlay(
+  $"a", Array[Byte](-1, -1, -1, -1), 7)), Row(Array[Byte](1, 2, 3, 4, 5, 
6, -1, -1, -1, -1)))
+checkAnswer(df.select(overlay(
+  $"a", Array[Byte](-1, -1), 7, 0)), Row(Array[Byte](1, 2, 3, 4, 5, 6, -1, 
-1, 7, 8, 9)))
+checkAnswer(df.select(overlay($"a", Array[Byte](-1, -1, -1, -1, -1), 2, 
4)),
+  Row(Array[Byte](1, -1, -1, -1, -1, -1, 6, 7, 8, 9)))
+  }
 
 Review comment:
   Ur, I forgot the discussion. ok, its ok to keep as it is. If necessary to 
support that, we can make a follow-up pr for that.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #25310: [SPARK-28578][INFRA] Improve Github pull request template

[GitBox]

HyukjinKwon commented on a change in pull request #25310: [SPARK-28578][INFRA] 
Improve Github pull request template
URL: https://github.com/apache/spark/pull/25310#discussion_r309511122
 
 

 ##
 File path: .github/PULL_REQUEST_TEMPLATE
 ##
 @@ -1,10 +1,31 @@
-## What changes were proposed in this pull request?
+

[GitHub] [spark] HyukjinKwon commented on a change in pull request #25310: [SPARK-28578][INFRA] Improve Github pull request template

[GitBox]

HyukjinKwon commented on a change in pull request #25310: [SPARK-28578][INFRA] 
Improve Github pull request template
URL: https://github.com/apache/spark/pull/25310#discussion_r309511122
 
 

 ##
 File path: .github/PULL_REQUEST_TEMPLATE
 ##
 @@ -1,10 +1,31 @@
-## What changes were proposed in this pull request?
+

[GitHub] [spark] WeichenXu123 commented on a change in pull request #25315: [SPARK-28582][PYSPARK] Fix pyspark daemon exit failed when receive SIGTERM on Python 3.7

[GitBox]

WeichenXu123 commented on a change in pull request #25315: 
[SPARK-28582][PYSPARK] Fix pyspark daemon exit failed when receive SIGTERM on 
Python 3.7
URL: https://github.com/apache/spark/pull/25315#discussion_r309503553
 
 

 ##
 File path: python/pyspark/daemon.py
 ##
 @@ -102,7 +102,7 @@ def shutdown(code):
 signal.signal(SIGTERM, SIG_DFL)
 # Send SIGHUP to notify workers of shutdown
 os.kill(0, SIGHUP)
-sys.exit(code)
+os._exit(code)
 
 Review comment:
   This line `os.kill(0, SIGHUP)` kill child processes by sending SIGHUP to 
them.
   so looks like calling sys.exit is the last thing to do in daemon. We have 
already done necessary clean before calling exit.
   
   So I prefer `os._exit` rather than `sys.exit` here, because `sys.exit` will 
throw `SystemExit` exception in main thread. If this `SystemExit` exception is 
swallowed by user code somewhere or in finally block run into some blocking 
code, then the kill will fail.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] xuanyuanking commented on a change in pull request #24382: [SPARK-27330][SS] support task abort in foreach writer

[GitBox]

xuanyuanking commented on a change in pull request #24382: [SPARK-27330][SS] 
support task abort in foreach writer
URL: https://github.com/apache/spark/pull/24382#discussion_r309508694
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/ForeachWriterTable.scala
 ##
 @@ -141,17 +143,33 @@ class ForeachDataWriter[T](
   writer.process(rowConverter(record))
 } catch {
   case t: Throwable =>
-writer.close(t)
+closeWriter(t)
 throw t
 }
   }
 
   override def commit(): WriterCommitMessage = {
-writer.close(null)
+closeWriter(null)
 ForeachWriterCommitMessage
   }
 
-  override def abort(): Unit = {}
+  override def abort(): Unit = {
 
 Review comment:
   In this case, we need to call `writer.close` in `abort`, the exception is 
necessary for the close API. 
https://github.com/apache/spark/blob/b3ffd8be14779cbb824d14b409f0a6eab93444ba/sql/core/src/main/scala/org/apache/spark/sql/ForeachWriter.scala#L129-L141


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25294: [SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION

[GitBox]

AmplabJenkins removed a comment on issue #25294: [SPARK-28476][SQL] Support 
ALTER DATABASE SET LOCATION
URL: https://github.com/apache/spark/pull/25294#issuecomment-517100795
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108498/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25294: [SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION

[GitBox]

AmplabJenkins removed a comment on issue #25294: [SPARK-28476][SQL] Support 
ALTER DATABASE SET LOCATION
URL: https://github.com/apache/spark/pull/25294#issuecomment-517100787
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25294: [SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION

[GitBox]

SparkQA commented on issue #25294: [SPARK-28476][SQL] Support ALTER DATABASE 
SET LOCATION
URL: https://github.com/apache/spark/pull/25294#issuecomment-517100666
 
 
   **[Test build #108498 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108498/testReport)**
 for PR 25294 at commit 
[`93d76d6`](https://github.com/apache/spark/commit/93d76d68c493074e568cae1661d0b314a1948f4a).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #25294: [SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION

[GitBox]

SparkQA removed a comment on issue #25294: [SPARK-28476][SQL] Support ALTER 
DATABASE SET LOCATION
URL: https://github.com/apache/spark/pull/25294#issuecomment-517082849
 
 
   **[Test build #108498 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108498/testReport)**
 for PR 25294 at commit 
[`93d76d6`](https://github.com/apache/spark/commit/93d76d68c493074e568cae1661d0b314a1948f4a).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25294: [SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION

[GitBox]

AmplabJenkins commented on issue #25294: [SPARK-28476][SQL] Support ALTER 
DATABASE SET LOCATION
URL: https://github.com/apache/spark/pull/25294#issuecomment-517100795
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108498/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25294: [SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION

[GitBox]

AmplabJenkins commented on issue #25294: [SPARK-28476][SQL] Support ALTER 
DATABASE SET LOCATION
URL: https://github.com/apache/spark/pull/25294#issuecomment-517100787
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] WeichenXu123 commented on a change in pull request #25315: [SPARK-28582][PYSPARK] Fix pyspark daemon exit failed when receive SIGTERM on Python 3.7

[GitBox]

WeichenXu123 commented on a change in pull request #25315: 
[SPARK-28582][PYSPARK] Fix pyspark daemon exit failed when receive SIGTERM on 
Python 3.7
URL: https://github.com/apache/spark/pull/25315#discussion_r309503553
 
 

 ##
 File path: python/pyspark/daemon.py
 ##
 @@ -102,7 +102,7 @@ def shutdown(code):
 signal.signal(SIGTERM, SIG_DFL)
 # Send SIGHUP to notify workers of shutdown
 os.kill(0, SIGHUP)
-sys.exit(code)
+os._exit(code)
 
 Review comment:
   This line `os.kill(0, SIGHUP)` kill child processes by sending SIGHUP to 
them.
   And I prefer `os._exit` rather than `sys.exit` here, because `sys.exit` will 
throw `SystemExit` exception in main thread. If this `SystemExit` exception is 
caught by user code then the kill will fail.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] WeichenXu123 commented on a change in pull request #25315: [SPARK-28582][PYSPARK] Fix pyspark daemon exit failed when receive SIGTERM on Python 3.7

[GitBox]

WeichenXu123 commented on a change in pull request #25315: 
[SPARK-28582][PYSPARK] Fix pyspark daemon exit failed when receive SIGTERM on 
Python 3.7
URL: https://github.com/apache/spark/pull/25315#discussion_r309503553
 
 

 ##
 File path: python/pyspark/daemon.py
 ##
 @@ -102,7 +102,7 @@ def shutdown(code):
 signal.signal(SIGTERM, SIG_DFL)
 # Send SIGHUP to notify workers of shutdown
 os.kill(0, SIGHUP)
-sys.exit(code)
+os._exit(code)
 
 Review comment:
   This line `os.kill(0, SIGHUP)` kill child processes by sending SIGHUP to 
them.
   And I prefer `os._exit` rather than `sys.exit` here, because `sys.exit` will 
throw `SystemExit` exception in main thread. If this `SystemExit` exception is 
caught by user code or there're a dead loop in user code then kill will fail.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25261: [SPARK-28521][SQL] Fix error message for built-in functions

[GitBox]

SparkQA commented on issue #25261: [SPARK-28521][SQL] Fix error message for 
built-in functions
URL: https://github.com/apache/spark/pull/25261#issuecomment-517092086
 
 
   **[Test build #108500 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108500/testReport)**
 for PR 25261 at commit 
[`3c46a84`](https://github.com/apache/spark/commit/3c46a84a4a847f0eb2e68b3f978dfeb5ec61f4df).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25261: [SPARK-28521][SQL] Fix error message for built-in functions

[GitBox]

AmplabJenkins removed a comment on issue #25261: [SPARK-28521][SQL] Fix error 
message for built-in functions
URL: https://github.com/apache/spark/pull/25261#issuecomment-517091636
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25261: [SPARK-28521][SQL] Fix error message for built-in functions

[GitBox]

AmplabJenkins removed a comment on issue #25261: [SPARK-28521][SQL] Fix error 
message for built-in functions
URL: https://github.com/apache/spark/pull/25261#issuecomment-517091643
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13594/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25261: [SPARK-28521][SQL] Fix error message for built-in functions

[GitBox]

AmplabJenkins commented on issue #25261: [SPARK-28521][SQL] Fix error message 
for built-in functions
URL: https://github.com/apache/spark/pull/25261#issuecomment-517091636
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25261: [SPARK-28521][SQL] Fix error message for built-in functions

[GitBox]

AmplabJenkins commented on issue #25261: [SPARK-28521][SQL] Fix error message 
for built-in functions
URL: https://github.com/apache/spark/pull/25261#issuecomment-517091643
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13594/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org