[jira] [Commented] (SPARK-34684) Hadoop config could not be successfully serilized from driver pods to executor pods

2021-03-23 Thread shanyu zhao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307375#comment-17307375 ] shanyu zhao commented on SPARK-34684: - [~attilapiros] What about we want to connect to HDFS HA with

[jira] [Commented] (SPARK-30536) Sort-merge join operator spilling performance improvements

2020-03-08 Thread shanyu zhao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17054555#comment-17054555 ] shanyu zhao commented on SPARK-30536: - Uploaded two slides to explain the optimization idea of this

[jira] [Updated] (SPARK-30536) Sort-merge join operator spilling performance improvements

2020-03-08 Thread shanyu zhao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated SPARK-30536: Attachment: spark-30536-explained.pdf > Sort-merge join operator spilling performance

[jira] [Updated] (SPARK-31029) Occasional class not found error in user's Future code using global ExecutionContext

2020-03-04 Thread shanyu zhao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated SPARK-31029: Description: *Problem:* When running tpc-ds test (https://github.com/databricks/spark-sql-perf),

[jira] [Created] (SPARK-31029) Occasional class not found error in user's Future code using global ExecutionContext

2020-03-03 Thread shanyu zhao (Jira)
shanyu zhao created SPARK-31029: --- Summary: Occasional class not found error in user's Future code using global ExecutionContext Key: SPARK-31029 URL: https://issues.apache.org/jira/browse/SPARK-31029

[jira] [Updated] (SPARK-31028) Add "-XX:ActiveProcessorCount" to Spark driver and executor in Yarn mode

2020-03-03 Thread shanyu zhao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated SPARK-31028: Description: When starting Spark driver and executors on Yarn cluster, the JVM process can

[jira] [Created] (SPARK-31028) Add "-XX:ActiveProcessorCount" to Spark driver and executor in Yarn mode

2020-03-03 Thread shanyu zhao (Jira)
shanyu zhao created SPARK-31028: --- Summary: Add "-XX:ActiveProcessorCount" to Spark driver and executor in Yarn mode Key: SPARK-31028 URL: https://issues.apache.org/jira/browse/SPARK-31028 Project:

[jira] [Created] (SPARK-30845) spark-submit pyspark app on yarn uploads local pyspark archives

2020-02-16 Thread shanyu zhao (Jira)
shanyu zhao created SPARK-30845: --- Summary: spark-submit pyspark app on yarn uploads local pyspark archives Key: SPARK-30845 URL: https://issues.apache.org/jira/browse/SPARK-30845 Project: Spark

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

2020-02-04 Thread shanyu zhao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030114#comment-17030114 ] shanyu zhao commented on SPARK-30602: - Thanks for the effort Min! Riffle seems to only do map side

[jira] [Commented] (SPARK-29003) Spark history server startup hang due to deadlock

2019-09-06 Thread shanyu zhao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924567#comment-16924567 ] shanyu zhao commented on SPARK-29003: - Please see the full jstack attached. > Spark history server

[jira] [Updated] (SPARK-29003) Spark history server startup hang due to deadlock

2019-09-06 Thread shanyu zhao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated SPARK-29003: Attachment: sparkhistory-jstack.log > Spark history server startup hang due to deadlock >

[jira] [Updated] (SPARK-29003) Spark history server startup hang due to deadlock

2019-09-05 Thread shanyu zhao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated SPARK-29003: Description: Occasionally when starting Spark History Server, the service process will hang

[jira] [Created] (SPARK-29003) Spark history server startup hang due to deadlock

2019-09-05 Thread shanyu zhao (Jira)
shanyu zhao created SPARK-29003: --- Summary: Spark history server startup hang due to deadlock Key: SPARK-29003 URL: https://issues.apache.org/jira/browse/SPARK-29003 Project: Spark Issue Type:

[jira] [Updated] (SPARK-12312) JDBC connection to Kerberos secured databases fails on remote executors

2019-05-03 Thread shanyu zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated SPARK-12312: Affects Version/s: 2.4.2 > JDBC connection to Kerberos secured databases fails on remote

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

2019-04-22 Thread shanyu zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823458#comment-16823458 ] shanyu zhao commented on SPARK-18673: - Ping. What is the verdict here for users want to use Spark

[jira] [Updated] (SPARK-26011) pyspark app with "spark.jars.packages" config does not work

2018-11-11 Thread shanyu zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated SPARK-26011: Description: Command "pyspark --packages" works as expected, but if submitting a livy pyspark

[jira] [Created] (SPARK-26011) pyspark app with "spark.jars.packages" config does not work

2018-11-11 Thread shanyu zhao (JIRA)
shanyu zhao created SPARK-26011: --- Summary: pyspark app with "spark.jars.packages" config does not work Key: SPARK-26011 URL: https://issues.apache.org/jira/browse/SPARK-26011 Project: Spark

[jira] [Commented] (SPARK-25999) make-distribution.sh failure with --r and -Phadoop-provided

2018-11-09 Thread shanyu zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682089#comment-16682089 ] shanyu zhao commented on SPARK-25999: - patch attached. Basically it creates an optional project that

[jira] [Updated] (SPARK-25999) make-distribution.sh failure with --r and -Phadoop-provided

2018-11-09 Thread shanyu zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated SPARK-25999: Attachment: SPARK-25999.patch > make-distribution.sh failure with --r and -Phadoop-provided >

[jira] [Updated] (SPARK-25999) make-distribution.sh failure with --r and -Phadoop-provided

2018-11-09 Thread shanyu zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated SPARK-25999: Summary: make-distribution.sh failure with --r and -Phadoop-provided (was: Spark

[jira] [Created] (SPARK-25999) Spark make-distribution failure with --r and -Phadoop-provided

2018-11-09 Thread shanyu zhao (JIRA)
shanyu zhao created SPARK-25999: --- Summary: Spark make-distribution failure with --r and -Phadoop-provided Key: SPARK-25999 URL: https://issues.apache.org/jira/browse/SPARK-25999 Project: Spark

[jira] [Created] (SPARK-24975) Spark history server REST API /api/v1/version returns error 404

2018-07-30 Thread shanyu zhao (JIRA)
shanyu zhao created SPARK-24975: --- Summary: Spark history server REST API /api/v1/version returns error 404 Key: SPARK-24975 URL: https://issues.apache.org/jira/browse/SPARK-24975 Project: Spark

[jira] [Commented] (SPARK-9514) Add EventHubsReceiver to support Spark Streaming using Azure EventHubs

2015-08-03 Thread shanyu zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14652883#comment-14652883 ] shanyu zhao commented on SPARK-9514: Thanks [~CodingCat], I've created the pull

[jira] [Updated] (SPARK-9514) Add EventHubsReceiver to support Spark Streaming using Azure EventHubs

2015-08-01 Thread shanyu zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated SPARK-9514: --- Attachment: SPARK-9514.patch Patch attached. I put EventHubsReceiver in external folder and added an

[jira] [Updated] (SPARK-9514) Add EventHubsReceiver to support Spark Streaming using Azure EventHubs

2015-07-31 Thread shanyu zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated SPARK-9514: --- Shepherd: shanyu zhao Add EventHubsReceiver to support Spark Streaming using Azure EventHubs

[jira] [Created] (SPARK-9514) Add EventHubsReceiver to support Spark Streaming using Azure EventHubs

2015-07-31 Thread shanyu zhao (JIRA)
shanyu zhao created SPARK-9514: -- Summary: Add EventHubsReceiver to support Spark Streaming using Azure EventHubs Key: SPARK-9514 URL: https://issues.apache.org/jira/browse/SPARK-9514 Project: Spark