[jira] [Commented] (SPARK-13877) Consider removing Kafka modules from Spark / Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-13877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200236#comment-15200236 ] Kostas Sakellis commented on SPARK-13877: - Why have we not more seriously conside

[jira] [Commented] (SPARK-13877) Consider removing Kafka modules from Spark / Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-13877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200252#comment-15200252 ] Kostas Sakellis commented on SPARK-13877: - How is this any different than creatin

[jira] [Commented] (SPARK-9197) Cached RDD partitions are lost when executors are dynamically deallocated

[ https://issues.apache.org/jira/browse/SPARK-9197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14634748#comment-14634748 ] Kostas Sakellis commented on SPARK-9197: I think this is a duplicate of: https://i

[jira] [Updated] (SPARK-7161) Provide REST api to download event logs from History Server

[ https://issues.apache.org/jira/browse/SPARK-7161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kostas Sakellis updated SPARK-7161: --- Component/s: (was: Streaming) Spark Core > Provide REST api to download e

[jira] [Commented] (SPARK-6506) python support yarn cluster mode requires SPARK_HOME to be set

[ https://issues.apache.org/jira/browse/SPARK-6506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14484250#comment-14484250 ] Kostas Sakellis commented on SPARK-6506: Here is the exception I saw when I ran th

[jira] [Commented] (SPARK-6506) python support yarn cluster mode requires SPARK_HOME to be set

[ https://issues.apache.org/jira/browse/SPARK-6506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14482414#comment-14482414 ] Kostas Sakellis commented on SPARK-6506: I ran into this issue too by running: bq.

[jira] [Commented] (SPARK-1239) Don't fetch all map output statuses at each reducer during shuffles

[ https://issues.apache.org/jira/browse/SPARK-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14352306#comment-14352306 ] Kostas Sakellis commented on SPARK-1239: How many reduce side tasks do you have? C

[jira] [Created] (SPARK-5721) Propagate missing external shuffle service errors to client

Kostas Sakellis created SPARK-5721: -- Summary: Propagate missing external shuffle service errors to client Key: SPARK-5721 URL: https://issues.apache.org/jira/browse/SPARK-5721 Project: Spark

[jira] [Commented] (SPARK-5647) Output metrics do not show up for older hadoop versions (< 2.5)

[ https://issues.apache.org/jira/browse/SPARK-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312818#comment-14312818 ] Kostas Sakellis commented on SPARK-5647: I'm not sure if this is possible with old

[jira] [Commented] (SPARK-5679) Flaky tests in InputOutputMetricsSuite: input metrics with interleaved reads and input metrics with mixed read method

[ https://issues.apache.org/jira/browse/SPARK-5679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312600#comment-14312600 ] Kostas Sakellis commented on SPARK-5679: I have tried to repo this in a number of

[jira] [Created] (SPARK-5674) Spark Job Explain Plan Proof of Concept

Kostas Sakellis created SPARK-5674: -- Summary: Spark Job Explain Plan Proof of Concept Key: SPARK-5674 URL: https://issues.apache.org/jira/browse/SPARK-5674 Project: Spark Issue Type: Improve

[jira] [Created] (SPARK-5647) Output metrics do not show up for older hadoop versions (< 2.5)

Kostas Sakellis created SPARK-5647: -- Summary: Output metrics do not show up for older hadoop versions (< 2.5) Key: SPARK-5647 URL: https://issues.apache.org/jira/browse/SPARK-5647 Project: Spark

[jira] [Created] (SPARK-5646) Record output metrics for cache

Kostas Sakellis created SPARK-5646: -- Summary: Record output metrics for cache Key: SPARK-5646 URL: https://issues.apache.org/jira/browse/SPARK-5646 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-5645) Track local bytes read for shuffles - update UI

Kostas Sakellis created SPARK-5645: -- Summary: Track local bytes read for shuffles - update UI Key: SPARK-5645 URL: https://issues.apache.org/jira/browse/SPARK-5645 Project: Spark Issue Type:

[jira] [Commented] (SPARK-5557) Servlet API classes now missing after jetty shading

[ https://issues.apache.org/jira/browse/SPARK-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308374#comment-14308374 ] Kostas Sakellis commented on SPARK-5557: [~pwendell] recommended this which did th

[jira] [Commented] (SPARK-5081) Shuffle write increases

[ https://issues.apache.org/jira/browse/SPARK-5081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14307584#comment-14307584 ] Kostas Sakellis commented on SPARK-5081: Can you add a sample of the code too? >

[jira] [Commented] (SPARK-4630) Dynamically determine optimal number of partitions

[ https://issues.apache.org/jira/browse/SPARK-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14284341#comment-14284341 ] Kostas Sakellis commented on SPARK-4630: I agree that this should be build not ass

[jira] [Updated] (SPARK-5225) Support coalesed Input Metrics from different sources

[ https://issues.apache.org/jira/browse/SPARK-5225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kostas Sakellis updated SPARK-5225: --- Description: Currently, If task reads data from more than one block and it is from different

[jira] [Created] (SPARK-5225) Support coalesed Input Metrics from different sources

Kostas Sakellis created SPARK-5225: -- Summary: Support coalesed Input Metrics from different sources Key: SPARK-5225 URL: https://issues.apache.org/jira/browse/SPARK-5225 Project: Spark Issue

[jira] [Created] (SPARK-5190) Allow spark listeners to be added before spark context gets initialized.

Kostas Sakellis created SPARK-5190: -- Summary: Allow spark listeners to be added before spark context gets initialized. Key: SPARK-5190 URL: https://issues.apache.org/jira/browse/SPARK-5190 Project: S

[jira] [Commented] (SPARK-4874) Report number of records read/written in a task

[ https://issues.apache.org/jira/browse/SPARK-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14250304#comment-14250304 ] Kostas Sakellis commented on SPARK-4874: I'm working on this. > Report number of

[jira] [Created] (SPARK-4874) Report number of records read/written in a task

Kostas Sakellis created SPARK-4874: -- Summary: Report number of records read/written in a task Key: SPARK-4874 URL: https://issues.apache.org/jira/browse/SPARK-4874 Project: Spark Issue Type:

[jira] [Created] (SPARK-4857) Add Executor Events to SparkListener

Kostas Sakellis created SPARK-4857: -- Summary: Add Executor Events to SparkListener Key: SPARK-4857 URL: https://issues.apache.org/jira/browse/SPARK-4857 Project: Spark Issue Type: Improvemen

[jira] [Commented] (SPARK-4857) Add Executor Events to SparkListener

[ https://issues.apache.org/jira/browse/SPARK-4857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14247578#comment-14247578 ] Kostas Sakellis commented on SPARK-4857: I'll work on this. > Add Executor Events

[jira] [Commented] (SPARK-4843) Squash ExecutorRunnable and ExecutorRunnableUtil hierarchy in yarn module

[ https://issues.apache.org/jira/browse/SPARK-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14246254#comment-14246254 ] Kostas Sakellis commented on SPARK-4843: I'm working on this. > Squash ExecutorRu

[jira] [Created] (SPARK-4843) Squash ExecutorRunnable and ExecutorRunnableUtil hierarchy in yarn module

Kostas Sakellis created SPARK-4843: -- Summary: Squash ExecutorRunnable and ExecutorRunnableUtil hierarchy in yarn module Key: SPARK-4843 URL: https://issues.apache.org/jira/browse/SPARK-4843 Project:

[jira] [Updated] (SPARK-4774) Make HiveFromSpark example more portable

[ https://issues.apache.org/jira/browse/SPARK-4774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kostas Sakellis updated SPARK-4774: --- Component/s: SQL > Make HiveFromSpark example more portable >

[jira] [Created] (SPARK-4774) Make HiveFromSpark example more portable

Kostas Sakellis created SPARK-4774: -- Summary: Make HiveFromSpark example more portable Key: SPARK-4774 URL: https://issues.apache.org/jira/browse/SPARK-4774 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-4630) Dynamically determine optimal number of partitions

Kostas Sakellis created SPARK-4630: -- Summary: Dynamically determine optimal number of partitions Key: SPARK-4630 URL: https://issues.apache.org/jira/browse/SPARK-4630 Project: Spark Issue Ty

[jira] [Commented] (SPARK-2450) Provide link to YARN executor logs on UI

[ https://issues.apache.org/jira/browse/SPARK-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209350#comment-14209350 ] Kostas Sakellis commented on SPARK-2450: I'll pick up the work on this. > Provide

[jira] [Commented] (SPARK-4092) Input metrics don't work for coalesce()'d RDD's

[ https://issues.apache.org/jira/browse/SPARK-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14207190#comment-14207190 ] Kostas Sakellis commented on SPARK-4092: [~aash], yes this should solve a superset

[jira] [Commented] (SPARK-4079) Snappy bundled with Spark does not work on older Linux distributions

[ https://issues.apache.org/jira/browse/SPARK-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14192188#comment-14192188 ] Kostas Sakellis commented on SPARK-4079: yes, I'm taking this over from Marcelo.

[jira] [Commented] (SPARK-4079) Snappy bundled with Spark does not work on older Linux distributions

[ https://issues.apache.org/jira/browse/SPARK-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191013#comment-14191013 ] Kostas Sakellis commented on SPARK-4079: [~pwendell] I'm in favor of checking if S

[jira] [Commented] (SPARK-4092) Input metrics don't work for coalesce()'d RDD's

[ https://issues.apache.org/jira/browse/SPARK-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190974#comment-14190974 ] Kostas Sakellis commented on SPARK-4092: The current patch I'm working on does the

[jira] [Resolved] (SPARK-3053) Reconcile spark.files.userClassPathFirst with spark.yarn.user.classpath.first

[ https://issues.apache.org/jira/browse/SPARK-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kostas Sakellis resolved SPARK-3053. Resolution: Duplicate > Reconcile spark.files.userClassPathFirst with spark.yarn.user.classp

[jira] [Commented] (SPARK-1239) Don't fetch all map output statuses at each reducer during shuffles

[ https://issues.apache.org/jira/browse/SPARK-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182110#comment-14182110 ] Kostas Sakellis commented on SPARK-1239: Apologies for not commenting on this JIRA

[jira] [Commented] (SPARK-1239) Don't fetch all map output statuses at each reducer during shuffles

[ https://issues.apache.org/jira/browse/SPARK-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14123883#comment-14123883 ] Kostas Sakellis commented on SPARK-1239: [~pwendell] I'd like to take a crack at t