[jira] [Commented] (SPARK-27937) Revert changes introduced as a part of Automatic namespace discovery [SPARK-24149]

[ https://issues.apache.org/jira/browse/SPARK-27937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16859001#comment-16859001 ] Dhruve Ashar commented on SPARK-27937: -- The exception that we started encountering is while spark

[jira] [Comment Edited] (SPARK-27937) Revert changes introduced as a part of Automatic namespace discovery [SPARK-24149]

[ https://issues.apache.org/jira/browse/SPARK-27937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16859001#comment-16859001 ] Dhruve Ashar edited comment on SPARK-27937 at 6/7/19 9:27 PM: -- The

[jira] [Created] (SPARK-27937) Revert changes introduced as a part of Automatic namespace discovery [SPARK-24149]

Dhruve Ashar created SPARK-27937: Summary: Revert changes introduced as a part of Automatic namespace discovery [SPARK-24149] Key: SPARK-27937 URL: https://issues.apache.org/jira/browse/SPARK-27937

[jira] [Commented] (SPARK-24149) Automatic namespaces discovery in HDFS federation

[ https://issues.apache.org/jira/browse/SPARK-24149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852365#comment-16852365 ] Dhruve Ashar commented on SPARK-24149: -- IMHO having a consistent way to reason about a behavior is

[jira] [Commented] (SPARK-24149) Automatic namespaces discovery in HDFS federation

[ https://issues.apache.org/jira/browse/SPARK-24149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850096#comment-16850096 ] Dhruve Ashar commented on SPARK-24149: -- Thanks for the missing context. The current behavior

[jira] [Commented] (SPARK-24149) Automatic namespaces discovery in HDFS federation

[ https://issues.apache.org/jira/browse/SPARK-24149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847929#comment-16847929 ] Dhruve Ashar commented on SPARK-24149: -- I also think that HDFS client shouldn't be figuring out if

[jira] [Commented] (SPARK-24149) Automatic namespaces discovery in HDFS federation

[ https://issues.apache.org/jira/browse/SPARK-24149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847923#comment-16847923 ] Dhruve Ashar commented on SPARK-24149: --   Spark should be agnostic about the namenodes for which

[jira] [Commented] (SPARK-24149) Automatic namespaces discovery in HDFS federation

[ https://issues.apache.org/jira/browse/SPARK-24149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847854#comment-16847854 ] Dhruve Ashar commented on SPARK-24149: -- [~mgaido] [~vanzin], The change this PR introduced is

[jira] [Commented] (SPARK-27112) Spark Scheduler encounters two independent Deadlocks when trying to kill executors either due to dynamic allocation or blacklisting

[ https://issues.apache.org/jira/browse/SPARK-27112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797253#comment-16797253 ] Dhruve Ashar commented on SPARK-27112: -- [~irashid] - I think this is a critical bug and since it is

[jira] [Commented] (SPARK-27107) Spark SQL Job failing because of Kryo buffer overflow with ORC

[ https://issues.apache.org/jira/browse/SPARK-27107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790911#comment-16790911 ] Dhruve Ashar commented on SPARK-27107: -- I verified the changes and we are no longer seeing the

[jira] [Commented] (SPARK-27107) Spark SQL Job failing because of Kryo buffer overflow with ORC

[ https://issues.apache.org/jira/browse/SPARK-27107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790610#comment-16790610 ] Dhruve Ashar commented on SPARK-27107: -- Update: The PR was merged in the orc repository. My

[jira] [Updated] (SPARK-27107) Spark SQL Job failing because of Kryo buffer overflow with ORC

[ https://issues.apache.org/jira/browse/SPARK-27107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dhruve Ashar updated SPARK-27107: - Description: The issue occurs while trying to read ORC data and setting the SearchArgument.

[jira] [Commented] (SPARK-27107) Spark SQL Job failing because of Kryo buffer overflow with ORC

[ https://issues.apache.org/jira/browse/SPARK-27107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789668#comment-16789668 ] Dhruve Ashar commented on SPARK-27107: -- This is something we can consistently reproduce every

[jira] [Commented] (SPARK-27107) Spark SQL Job failing because of Kryo buffer overflow with ORC

[ https://issues.apache.org/jira/browse/SPARK-27107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788021#comment-16788021 ] Dhruve Ashar commented on SPARK-27107: -- [~dongjoon] can you review the PR for ORC to fix this

[jira] [Created] (SPARK-27107) Spark SQL Job failing because of Kryo buffer overflow with ORC

Dhruve Ashar created SPARK-27107: Summary: Spark SQL Job failing because of Kryo buffer overflow with ORC Key: SPARK-27107 URL: https://issues.apache.org/jira/browse/SPARK-27107 Project: Spark

[jira] [Created] (ORC-476) Make SearchAgument kryo buffer size configurable

Dhruve Ashar created ORC-476: Summary: Make SearchAgument kryo buffer size configurable Key: ORC-476 URL: https://issues.apache.org/jira/browse/ORC-476 Project: ORC Issue Type: Improvement

[jira] [Comment Edited] (SPARK-26827) Support importing python modules having shared objects(.so)

[ https://issues.apache.org/jira/browse/SPARK-26827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761072#comment-16761072 ] Dhruve Ashar edited comment on SPARK-26827 at 2/5/19 6:08 PM: -- Resolution :

[jira] [Resolved] (SPARK-26827) Support importing python modules having shared objects(.so)

[ https://issues.apache.org/jira/browse/SPARK-26827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dhruve Ashar resolved SPARK-26827. -- Resolution: Workaround Pass the same archive with py-files and archives option. > Support

[jira] [Commented] (SPARK-26827) Support importing python modules having shared objects(.so)

[ https://issues.apache.org/jira/browse/SPARK-26827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761069#comment-16761069 ] Dhruve Ashar commented on SPARK-26827: -- Thanks for the response [~irashid] and [~hyukjin.kwon].

[jira] [Commented] (SPARK-26827) Support importing python modules having shared objects(.so)

[ https://issues.apache.org/jira/browse/SPARK-26827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761034#comment-16761034 ] Dhruve Ashar commented on SPARK-26827: -- [~holden.ka...@gmail.com] , [~irashid] any thoughts on this

[jira] [Created] (SPARK-26827) Support importing python modules having shared objects(.so)

Dhruve Ashar created SPARK-26827: Summary: Support importing python modules having shared objects(.so) Key: SPARK-26827 URL: https://issues.apache.org/jira/browse/SPARK-26827 Project: Spark

[jira] [Commented] (SPARK-26801) Spark unable to read valid avro types

[ https://issues.apache.org/jira/browse/SPARK-26801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760178#comment-16760178 ] Dhruve Ashar commented on SPARK-26801: -- I have given a short summary of the issue in the PR

[jira] [Created] (SPARK-26801) Spark unable to read valid avro types

Dhruve Ashar created SPARK-26801: Summary: Spark unable to read valid avro types Key: SPARK-26801 URL: https://issues.apache.org/jira/browse/SPARK-26801 Project: Spark Issue Type: Bug

***UNCHECKED*** [jira] [Updated] (SPARK-25468) Highlight current page index in the history server

[ https://issues.apache.org/jira/browse/SPARK-25468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dhruve Ashar updated SPARK-25468: - Attachment: SparkHistoryServer.png > Highlight current page index in the history server >

[jira] [Updated] (SPARK-25468) Highlight current page index in the history server

[ https://issues.apache.org/jira/browse/SPARK-25468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dhruve Ashar updated SPARK-25468: - Description: Spark History Server Web UI should highlight the current page index selected for

[jira] [Updated] (SPARK-25468) Highlight current page index in the history server

[ https://issues.apache.org/jira/browse/SPARK-25468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dhruve Ashar updated SPARK-25468: - Description: Spark History Server Web UI should highlight the current page index selected for

[jira] [Created] (SPARK-25468) Highlight current page index in the history server

Dhruve Ashar created SPARK-25468: Summary: Highlight current page index in the history server Key: SPARK-25468 URL: https://issues.apache.org/jira/browse/SPARK-25468 Project: Spark Issue

[jira] [Created] (SPARK-24610) wholeTextFiles broken for small files

Dhruve Ashar created SPARK-24610: Summary: wholeTextFiles broken for small files Key: SPARK-24610 URL: https://issues.apache.org/jira/browse/SPARK-24610 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114628#comment-16114628 ] Dhruve Ashar commented on SPARK-20589: -- I have a patch for this and would like to have some

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114626#comment-16114626 ] Dhruve Ashar commented on SPARK-20589: -- Spark defines stages based on the shuffle dependencies and

[jira] [Created] (SPARK-21500) Update Shuffle Fetch bookkeeping immediately after receiving remote block

Dhruve Ashar created SPARK-21500: Summary: Update Shuffle Fetch bookkeeping immediately after receiving remote block Key: SPARK-21500 URL: https://issues.apache.org/jira/browse/SPARK-21500 Project:

[jira] [Commented] (SPARK-21460) Spark dynamic allocation breaks when ListenerBus event queue runs full

[ https://issues.apache.org/jira/browse/SPARK-21460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094689#comment-16094689 ] Dhruve Ashar commented on SPARK-21460: -- [~Tagar] Can you attach the driver logs so that it helps in

[jira] [Created] (SPARK-21243) Limit the number of maps in a single shuffle fetch

Dhruve Ashar created SPARK-21243: Summary: Limit the number of maps in a single shuffle fetch Key: SPARK-21243 URL: https://issues.apache.org/jira/browse/SPARK-21243 Project: Spark Issue

[jira] [Created] (SPARK-21181) Suppress memory leak errors reported by netty

Dhruve Ashar created SPARK-21181: Summary: Suppress memory leak errors reported by netty Key: SPARK-21181 URL: https://issues.apache.org/jira/browse/SPARK-21181 Project: Spark Issue Type:

[jira] [Updated] (SPARK-17417) Fix sorting of part files while reconstructing RDD/partition from checkpointed files.

[ https://issues.apache.org/jira/browse/SPARK-17417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dhruve Ashar updated SPARK-17417: - Summary: Fix sorting of part files while reconstructing RDD/partition from checkpointed files.

[jira] [Commented] (SPARK-17417) Fix # of partitions for RDD while checkpointing - Currently limited by 10000(%05d)

[ https://issues.apache.org/jira/browse/SPARK-17417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546792#comment-15546792 ] Dhruve Ashar commented on SPARK-17417: -- [~srowen] AFAIU the checkpointing mechanism in spark core,

[jira] [Commented] (SPARK-17649) Log how many Spark events got dropped in LiveListenerBus

[ https://issues.apache.org/jira/browse/SPARK-17649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531080#comment-15531080 ] Dhruve Ashar commented on SPARK-17649: -- I get the idea. While you certainly cannot filter out a

[jira] [Commented] (SPARK-17649) Log how many Spark events got dropped in LiveListenerBus

[ https://issues.apache.org/jira/browse/SPARK-17649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526426#comment-15526426 ] Dhruve Ashar commented on SPARK-17649: -- [~zsxwing] Since we are logging the drop event count, it

[jira] [Commented] (SPARK-16441) Spark application hang when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-16441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488528#comment-15488528 ] Dhruve Ashar commented on SPARK-16441: -- So all of these are related in one way or the other. I will

[jira] [Commented] (SPARK-16441) Spark application hang when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-16441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15487224#comment-15487224 ] Dhruve Ashar commented on SPARK-16441: -- There was a patch which was recently contributed which

[jira] [Commented] (SPARK-17417) Fix # of partitions for RDD while checkpointing - Currently limited by 10000(%05d)

[ https://issues.apache.org/jira/browse/SPARK-17417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15468622#comment-15468622 ] Dhruve Ashar commented on SPARK-17417: -- Thanks for the suggestion. I'll work on the changes and

[jira] [Created] (SPARK-17417) Fix # of partitions for RDD while checkpointing - Currently limited by 10000(%05d)

Dhruve Ashar created SPARK-17417: Summary: Fix # of partitions for RDD while checkpointing - Currently limited by 1(%05d) Key: SPARK-17417 URL: https://issues.apache.org/jira/browse/SPARK-17417

[jira] [Created] (SPARK-17365) Kill multiple executors together to reduce lock contention

Dhruve Ashar created SPARK-17365: Summary: Kill multiple executors together to reduce lock contention Key: SPARK-17365 URL: https://issues.apache.org/jira/browse/SPARK-17365 Project: Spark

[jira] [Updated] (SPARK-16441) Spark application hang when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-16441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dhruve Ashar updated SPARK-16441: - Affects Version/s: 2.0.0 > Spark application hang when dynamic allocation is enabled >

[jira] [Updated] (SPARK-16441) Spark application hang when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-16441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dhruve Ashar updated SPARK-16441: - External issue URL: (was: https://issues.apache.org/jira/browse/SPARK-15703) > Spark

[jira] [Updated] (SPARK-16441) Spark application hang when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-16441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dhruve Ashar updated SPARK-16441: - External issue URL: https://issues.apache.org/jira/browse/SPARK-15703 > Spark application hang

[jira] [Commented] (SPARK-16441) Spark application hang when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-16441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392049#comment-15392049 ] Dhruve Ashar commented on SPARK-16441: -- How large is the data being sent to driver? And is this

[jira] [Commented] (SPARK-16441) Spark application hang when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-16441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384732#comment-15384732 ] Dhruve Ashar commented on SPARK-16441: -- [~cenyuhai] have you started looking at this one? If not

[jira] [Commented] (SPARK-16441) Spark application hang when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-16441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384573#comment-15384573 ] Dhruve Ashar commented on SPARK-16441: -- Does the application hang consistently when dynamic

[jira] [Updated] (SPARK-15703) Spark UI doesn't show all tasks as completed when it should

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dhruve Ashar updated SPARK-15703: - Component/s: Scheduler > Spark UI doesn't show all tasks as completed when it should >

[jira] [Commented] (SPARK-15703) Spark UI doesn't show all tasks as completed when it should

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384331#comment-15384331 ] Dhruve Ashar commented on SPARK-15703: -- Here are some of the findings: LiveListenerBus replaces

[jira] [Updated] (SPARK-15703) Spark UI doesn't show all tasks as completed when it should

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dhruve Ashar updated SPARK-15703: - Attachment: SparkListenerBus .png spark-dynamic-executor-allocation.png > Spark

[jira] [Created] (SPARK-16018) Shade netty for shuffle to work on YARN

Dhruve Ashar created SPARK-16018: Summary: Shade netty for shuffle to work on YARN Key: SPARK-16018 URL: https://issues.apache.org/jira/browse/SPARK-16018 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-15966) Fix markdown for Spark Monitoring

[ https://issues.apache.org/jira/browse/SPARK-15966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dhruve Ashar updated SPARK-15966: - Description: The markdown for Spark monitoring needs to be fixed.

[jira] [Created] (SPARK-15966) Fix markdown for Spark Monitoring

Dhruve Ashar created SPARK-15966: Summary: Fix markdown for Spark Monitoring Key: SPARK-15966 URL: https://issues.apache.org/jira/browse/SPARK-15966 Project: Spark Issue Type: Documentation

[jira] [Created] (SPARK-14572) Update Config Doc to specify -Xms in extraJavaOptions

Dhruve Ashar created SPARK-14572: Summary: Update Config Doc to specify -Xms in extraJavaOptions Key: SPARK-14572 URL: https://issues.apache.org/jira/browse/SPARK-14572 Project: Spark Issue