[jira] [Commented] (SPARK-13877) Consider removing Kafka modules from Spark / Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-13877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15200306#comment-15200306 ] Hari Shreedharan commented on SPARK-13877: -- You could have separate repos and separate releases,

[jira] [Comment Edited] (SPARK-13877) Consider removing Kafka modules from Spark / Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-13877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15196805#comment-15196805 ] Hari Shreedharan edited comment on SPARK-13877 at 3/16/16 5:49 AM: ---

[jira] [Commented] (SPARK-13877) Consider removing Kafka modules from Spark / Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-13877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15196805#comment-15196805 ] Hari Shreedharan commented on SPARK-13877: -- [~c...@koeninger.org] - Sure. I agree with having

[jira] [Commented] (SPARK-13877) Consider removing Kafka modules from Spark / Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-13877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15195897#comment-15195897 ] Hari Shreedharan commented on SPARK-13877: -- [~c...@koeninger.org] - As long as they are

[jira] [Commented] (SPARK-13877) Consider removing Kafka modules from Spark / Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-13877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15195843#comment-15195843 ] Hari Shreedharan commented on SPARK-13877: -- I think we could easily make this a sub-project and

[jira] [Created] (SPARK-13877) Consider removing Kafka modules from Spark

Hari Shreedharan created SPARK-13877: Summary: Consider removing Kafka modules from Spark Key: SPARK-13877 URL: https://issues.apache.org/jira/browse/SPARK-13877 Project: Spark Issue

[jira] [Commented] (SPARK-12316) Stack overflow with endless call of `Delegation token thread` when application end.

[ https://issues.apache.org/jira/browse/SPARK-12316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140024#comment-15140024 ] Hari Shreedharan commented on SPARK-12316: -- So the issue is that the driver got rid of the

[jira] [Commented] (SPARK-13216) Spark streaming application not honoring --num-executors in restarting of an application from a checkpoint

[ https://issues.apache.org/jira/browse/SPARK-13216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137797#comment-15137797 ] Hari Shreedharan commented on SPARK-13216: -- I disagree that checkpointing is only for failed

[jira] [Commented] (SPARK-13216) Spark streaming application not honoring --num-executors in restarting of an application from a checkpoint

[ https://issues.apache.org/jira/browse/SPARK-13216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135233#comment-15135233 ] Hari Shreedharan commented on SPARK-13216: -- I think this brings up a broader question - I think

[jira] [Resolved] (SPARK-4122) Add library to write data back to Kafka

[ https://issues.apache.org/jira/browse/SPARK-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan resolved SPARK-4122. - Resolution: Won't Fix I pushed it here: https://github.com/cloudera/spark-kafka-writer

[jira] [Commented] (SPARK-4122) Add library to write data back to Kafka

[ https://issues.apache.org/jira/browse/SPARK-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990007#comment-14990007 ] Hari Shreedharan commented on SPARK-4122: - I kept it updated for a while and didn't get reviews.

[jira] [Updated] (SPARK-11249) [Launcher] Launcher library fails is app resource is not added

[ https://issues.apache.org/jira/browse/SPARK-11249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated SPARK-11249: - Component/s: Spark Submit > [Launcher] Launcher library fails is app resource is not

[jira] [Updated] (SPARK-11249) [Launcher] Launcher library fails is app resource is not added

[ https://issues.apache.org/jira/browse/SPARK-11249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated SPARK-11249: - Affects Version/s: 1.5.1 > [Launcher] Launcher library fails is app resource is not

[jira] [Created] (SPARK-11249) [Launcher] Launcher library fails is app resource is not added

Hari Shreedharan created SPARK-11249: Summary: [Launcher] Launcher library fails is app resource is not added Key: SPARK-11249 URL: https://issues.apache.org/jira/browse/SPARK-11249 Project:

[jira] [Commented] (SPARK-11201) StreamContext.getOrCreate is broken is yarn-client mode

[ https://issues.apache.org/jira/browse/SPARK-11201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965533#comment-14965533 ] Hari Shreedharan commented on SPARK-11201: -- Yep, closed the PR. So I will close this jira as a

[jira] [Closed] (SPARK-11201) StreamContext.getOrCreate is broken is yarn-client mode

[ https://issues.apache.org/jira/browse/SPARK-11201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan closed SPARK-11201. Resolution: Duplicate > StreamContext.getOrCreate is broken is yarn-client mode >

[jira] [Updated] (SPARK-11201) StreamContext.getOrCreate is broken is yarn-client mode

[ https://issues.apache.org/jira/browse/SPARK-11201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated SPARK-11201: - Description: If {{StreamingContext.getOrCreate}} (or the constructors that create the

[jira] [Created] (SPARK-11201) StreamContext.getOrCreate is broken is yarn-client mode

Hari Shreedharan created SPARK-11201: Summary: StreamContext.getOrCreate is broken is yarn-client mode Key: SPARK-11201 URL: https://issues.apache.org/jira/browse/SPARK-11201 Project: Spark

[jira] [Created] (SPARK-11019) [streaming] [flume] Gracefully shutdown Flume receiver threads

Hari Shreedharan created SPARK-11019: Summary: [streaming] [flume] Gracefully shutdown Flume receiver threads Key: SPARK-11019 URL: https://issues.apache.org/jira/browse/SPARK-11019 Project:

[jira] [Created] (SPARK-10955) Disable dynamic allocation for Streaming jobs

Hari Shreedharan created SPARK-10955: Summary: Disable dynamic allocation for Streaming jobs Key: SPARK-10955 URL: https://issues.apache.org/jira/browse/SPARK-10955 Project: Spark Issue

[jira] [Commented] (SPARK-10557) Publish Spark 1.5.0 on Maven central

[ https://issues.apache.org/jira/browse/SPARK-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741202#comment-14741202 ] Hari Shreedharan commented on SPARK-10557: -- Filed:

[jira] [Created] (SPARK-8988) Driver logs links is missing on secure cluster

Hari Shreedharan created SPARK-8988: --- Summary: Driver logs links is missing on secure cluster Key: SPARK-8988 URL: https://issues.apache.org/jira/browse/SPARK-8988 Project: Spark Issue

[jira] [Created] (SPARK-8851) in Yarn client mode, Client.scala does not login even when credentials are specified

Hari Shreedharan created SPARK-8851: --- Summary: in Yarn client mode, Client.scala does not login even when credentials are specified Key: SPARK-8851 URL: https://issues.apache.org/jira/browse/SPARK-8851

[jira] [Commented] (SPARK-8851) in Yarn client mode, Client.scala does not login even when credentials are specified

[ https://issues.apache.org/jira/browse/SPARK-8851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14615895#comment-14615895 ] Hari Shreedharan commented on SPARK-8851: - I am currently confirming that this is

[jira] [Commented] (SPARK-8405) Show executor logs on Web UI when Yarn log aggregation is enabled

[ https://issues.apache.org/jira/browse/SPARK-8405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14603269#comment-14603269 ] Hari Shreedharan commented on SPARK-8405: - Actually I think this is a config issue

[jira] [Commented] (SPARK-8405) Show executor logs on Web UI when Yarn log aggregation is enabled

[ https://issues.apache.org/jira/browse/SPARK-8405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14603262#comment-14603262 ] Hari Shreedharan commented on SPARK-8405: - If you wait for a few seconds, that

[jira] [Updated] (SPARK-8619) Can't find the keytab file when recovering the streaming application.

[ https://issues.apache.org/jira/browse/SPARK-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated SPARK-8619: Affects Version/s: (was: 1.5.0) 1.4.0 Target Version/s:

[jira] [Created] (SPARK-8533) Bump Flume version to 1.6.0

Hari Shreedharan created SPARK-8533: --- Summary: Bump Flume version to 1.6.0 Key: SPARK-8533 URL: https://issues.apache.org/jira/browse/SPARK-8533 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-8483) Remove commons-lang3 depedency from flume-sink

[ https://issues.apache.org/jira/browse/SPARK-8483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594252#comment-14594252 ] Hari Shreedharan commented on SPARK-8483: - Well, we aren't adding a dependency -

[jira] [Created] (SPARK-8483) Remove commons-lang3 depedency from flume-sink

Hari Shreedharan created SPARK-8483: --- Summary: Remove commons-lang3 depedency from flume-sink Key: SPARK-8483 URL: https://issues.apache.org/jira/browse/SPARK-8483 Project: Spark Issue

[jira] [Commented] (SPARK-6892) Recovery from checkpoint will also reuse the application id when write eventLog in yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14584086#comment-14584086 ] Hari Shreedharan commented on SPARK-6892: - This should have been fixed by

[jira] [Created] (SPARK-8136) AM link download test can be flaky

Hari Shreedharan created SPARK-8136: --- Summary: AM link download test can be flaky Key: SPARK-8136 URL: https://issues.apache.org/jira/browse/SPARK-8136 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-7955) Ensure executors that have cached blocks are not removed when dynamic allocation is enabled

Hari Shreedharan created SPARK-7955: --- Summary: Ensure executors that have cached blocks are not removed when dynamic allocation is enabled Key: SPARK-7955 URL: https://issues.apache.org/jira/browse/SPARK-7955

[jira] [Resolved] (SPARK-7928) Yarn App Master Logs are not displayed in the spark historyserver UI

[ https://issues.apache.org/jira/browse/SPARK-7928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan resolved SPARK-7928. - Resolution: Duplicate This issue was fixed by https://github.com/apache/spark/pull/6166

[jira] [Commented] (SPARK-7038) [Streaming] Spark Sink requires spark assembly in classpath

[ https://issues.apache.org/jira/browse/SPARK-7038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14563985#comment-14563985 ] Hari Shreedharan commented on SPARK-7038: - [~vanzin] - Does adding the shade

[jira] [Commented] (SPARK-7908) PySpark Streaming tests are flaky.

[ https://issues.apache.org/jira/browse/SPARK-7908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14562112#comment-14562112 ] Hari Shreedharan commented on SPARK-7908: - Note that {{test_window}} also failed

[jira] [Updated] (SPARK-7908) PySpark Streaming tests are flaky.

[ https://issues.apache.org/jira/browse/SPARK-7908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated SPARK-7908: Priority: Critical (was: Major) PySpark Streaming tests are flaky.

[jira] [Created] (SPARK-7908) PySpark Streaming tests are flaky.

Hari Shreedharan created SPARK-7908: --- Summary: PySpark Streaming tests are flaky. Key: SPARK-7908 URL: https://issues.apache.org/jira/browse/SPARK-7908 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-7908) PySpark Streaming tests are flaky.

[ https://issues.apache.org/jira/browse/SPARK-7908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14561972#comment-14561972 ] Hari Shreedharan commented on SPARK-7908: - /cc [~tdas], [~davies] PySpark

[jira] [Created] (SPARK-7750) Rename json endpoints to api endpoints

Hari Shreedharan created SPARK-7750: --- Summary: Rename json endpoints to api endpoints Key: SPARK-7750 URL: https://issues.apache.org/jira/browse/SPARK-7750 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-7657) [YARN] Show driver link in Spark UI

Hari Shreedharan created SPARK-7657: --- Summary: [YARN] Show driver link in Spark UI Key: SPARK-7657 URL: https://issues.apache.org/jira/browse/SPARK-7657 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-7356) Flaky test: o.a.s.streaming.flume.FlumePollingStreamSuite

[ https://issues.apache.org/jira/browse/SPARK-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542971#comment-14542971 ] Hari Shreedharan commented on SPARK-7356: - Thanks [~andrewor14]. If this test

[jira] [Commented] (SPARK-7356) Flaky test: o.a.s.streaming.flume.FlumePollingStreamSuite

[ https://issues.apache.org/jira/browse/SPARK-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527891#comment-14527891 ] Hari Shreedharan commented on SPARK-7356: - I instrumented the SparkSinkSuite

[jira] [Commented] (SPARK-7356) Flaky test: o.a.s.streaming.flume.FlumePollingStreamSuite

[ https://issues.apache.org/jira/browse/SPARK-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527958#comment-14527958 ] Hari Shreedharan commented on SPARK-7356: - Perhaps in a separate patch? I will

[jira] [Commented] (SPARK-5342) Allow long running Spark apps to run on secure YARN/HDFS

[ https://issues.apache.org/jira/browse/SPARK-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522368#comment-14522368 ] Hari Shreedharan commented on SPARK-5342: - Ah, missed the hadoop-1 build. I will

[jira] [Created] (SPARK-7252) Add support for creating new Hive and HBase delegation tokens

Hari Shreedharan created SPARK-7252: --- Summary: Add support for creating new Hive and HBase delegation tokens Key: SPARK-7252 URL: https://issues.apache.org/jira/browse/SPARK-7252 Project: Spark

[jira] [Created] (SPARK-7161) Provide REST api to download event logs from History Server

Hari Shreedharan created SPARK-7161: --- Summary: Provide REST api to download event logs from History Server Key: SPARK-7161 URL: https://issues.apache.org/jira/browse/SPARK-7161 Project: Spark

[jira] [Created] (SPARK-7038) [Streaming] Spark Sink requires spark assembly in classpath

Hari Shreedharan created SPARK-7038: --- Summary: [Streaming] Spark Sink requires spark assembly in classpath Key: SPARK-7038 URL: https://issues.apache.org/jira/browse/SPARK-7038 Project: Spark

[jira] [Created] (SPARK-7040) Explore receiver-less DStream for Flume

Hari Shreedharan created SPARK-7040: --- Summary: Explore receiver-less DStream for Flume Key: SPARK-7040 URL: https://issues.apache.org/jira/browse/SPARK-7040 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-6415) Spark Streaming fail-fast: Stop scheduling jobs when a batch fails, and kills the app

Hari Shreedharan created SPARK-6415: --- Summary: Spark Streaming fail-fast: Stop scheduling jobs when a batch fails, and kills the app Key: SPARK-6415 URL: https://issues.apache.org/jira/browse/SPARK-6415

[jira] [Commented] (SPARK-6222) [STREAMING] All data may not be recovered from WAL when driver is killed

[ https://issues.apache.org/jira/browse/SPARK-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355731#comment-14355731 ] Hari Shreedharan commented on SPARK-6222: - In the direct connector for Kafka, we

[jira] [Commented] (SPARK-6222) [STREAMING] All data may not be recovered from WAL when driver is killed

[ https://issues.apache.org/jira/browse/SPARK-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355652#comment-14355652 ] Hari Shreedharan commented on SPARK-6222: - I could not generate TRACE level logs

[jira] [Commented] (SPARK-6222) [STREAMING] All data may not be recovered from WAL when driver is killed

[ https://issues.apache.org/jira/browse/SPARK-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355965#comment-14355965 ] Hari Shreedharan commented on SPARK-6222: - Another option is to change the way we

[jira] [Commented] (SPARK-6222) [STREAMING] All data may not be recovered from WAL when driver is killed

[ https://issues.apache.org/jira/browse/SPARK-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355996#comment-14355996 ] Hari Shreedharan commented on SPARK-6222: - Thinking about it again -

[jira] [Updated] (SPARK-6222) [STREAMING] All data may not be recovered from WAL when driver is killed

[ https://issues.apache.org/jira/browse/SPARK-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated SPARK-6222: Description: When testing for our next release, our internal tests written by [~wypoon]

[jira] [Commented] (SPARK-6222) [STREAMING] All data may not be recovered from WAL when driver is killed

[ https://issues.apache.org/jira/browse/SPARK-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353867#comment-14353867 ] Hari Shreedharan commented on SPARK-6222: - [~srowen] This patch is actually not

[jira] [Commented] (SPARK-6222) [STREAMING] All data may not be recovered from WAL when driver is killed

[ https://issues.apache.org/jira/browse/SPARK-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353842#comment-14353842 ] Hari Shreedharan commented on SPARK-6222: - The one on the jira. [STREAMING] All

[jira] [Updated] (SPARK-6222) [STREAMING] All data may not be recovered from WAL when driver is killed

[ https://issues.apache.org/jira/browse/SPARK-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated SPARK-6222: Attachment: AfterPatch.txt CleanWithoutPatch.txt Here are the logs:

[jira] [Created] (SPARK-6222) [STREAMING] All data may not be recovered from WAL when driver is killed

Hari Shreedharan created SPARK-6222: --- Summary: [STREAMING] All data may not be recovered from WAL when driver is killed Key: SPARK-6222 URL: https://issues.apache.org/jira/browse/SPARK-6222

[jira] [Created] (SPARK-5937) [YARN] ClientSuite must set YARN mode to true to ensure correct SparkHadoopUtil implementation is used.

Hari Shreedharan created SPARK-5937: --- Summary: [YARN] ClientSuite must set YARN mode to true to ensure correct SparkHadoopUtil implementation is used. Key: SPARK-5937 URL:

[jira] [Commented] (SPARK-4905) Flaky FlumeStreamSuite test: org.apache.spark.streaming.flume.FlumeStreamSuite.flume input stream

[ https://issues.apache.org/jira/browse/SPARK-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14313037#comment-14313037 ] Hari Shreedharan commented on SPARK-4905: - Thanks [~joshrosen]! Flaky

[jira] [Commented] (SPARK-4905) Flaky FlumeStreamSuite test: org.apache.spark.streaming.flume.FlumeStreamSuite.flume input stream

[ https://issues.apache.org/jira/browse/SPARK-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303766#comment-14303766 ] Hari Shreedharan commented on SPARK-4905: - Is there a recent instance of failure?

[jira] [Resolved] (SPARK-5545) [STREAMING] DStream#saveAs**Files can fail after app restarts

[ https://issues.apache.org/jira/browse/SPARK-5545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan resolved SPARK-5545. - Resolution: Duplicate [STREAMING] DStream#saveAs**Files can fail after app restarts

[jira] [Commented] (SPARK-5545) [STREAMING] DStream#saveAs**Files can fail after app restarts

[ https://issues.apache.org/jira/browse/SPARK-5545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302794#comment-14302794 ] Hari Shreedharan commented on SPARK-5545: - I had not seen that one before. It

[jira] [Updated] (SPARK-5545) [STREAMING] DStream#saveAs**Files can fail after app restarts

[ https://issues.apache.org/jira/browse/SPARK-5545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated SPARK-5545: Priority: Critical (was: Major) [STREAMING] DStream#saveAs**Files can fail after app

[jira] [Created] (SPARK-5545) [STREAMING] DStream#saveAs**Files can fail after app restarts

Hari Shreedharan created SPARK-5545: --- Summary: [STREAMING] DStream#saveAs**Files can fail after app restarts Key: SPARK-5545 URL: https://issues.apache.org/jira/browse/SPARK-5545 Project: Spark

[jira] [Commented] (SPARK-2345) ForEachDStream should have an option of running the foreachfunc on Spark

[ https://issues.apache.org/jira/browse/SPARK-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14288013#comment-14288013 ] Hari Shreedharan commented on SPARK-2345: - +1. Please do. ForEachDStream should

[jira] [Commented] (SPARK-5342) Allow long running Spark apps to run on secure YARN/HDFS

[ https://issues.apache.org/jira/browse/SPARK-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14286543#comment-14286543 ] Hari Shreedharan commented on SPARK-5342: - Looks like SPARK-3883 is adding SSL

[jira] [Commented] (SPARK-5342) Allow long running Spark apps to run on secure YARN/HDFS

[ https://issues.apache.org/jira/browse/SPARK-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14287018#comment-14287018 ] Hari Shreedharan commented on SPARK-5342: - I am considering just copying the

[jira] [Commented] (SPARK-5342) Allow long running Spark apps to run on secure YARN/HDFS

[ https://issues.apache.org/jira/browse/SPARK-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14284908#comment-14284908 ] Hari Shreedharan commented on SPARK-5342: - [~pwendell], [~tgraves], [~vanzin],

[jira] [Updated] (SPARK-5342) Allow long running Spark apps to run on secure YARN/HDFS

[ https://issues.apache.org/jira/browse/SPARK-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated SPARK-5342: Attachment: SparkYARN.pdf Design doc with proposed design. Original design doc with

[jira] [Commented] (SPARK-5342) Allow long running Spark apps to run on secure YARN/HDFS

[ https://issues.apache.org/jira/browse/SPARK-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14284914#comment-14284914 ] Hari Shreedharan commented on SPARK-5342: - Thanks [~adhoot] for helping with

[jira] [Created] (SPARK-5342) Allow long running Spark apps to run on secure YARN/HDFS

Hari Shreedharan created SPARK-5342: --- Summary: Allow long running Spark apps to run on secure YARN/HDFS Key: SPARK-5342 URL: https://issues.apache.org/jira/browse/SPARK-5342 Project: Spark

[jira] [Comment Edited] (SPARK-5342) Allow long running Spark apps to run on secure YARN/HDFS

[ https://issues.apache.org/jira/browse/SPARK-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14284908#comment-14284908 ] Hari Shreedharan edited comment on SPARK-5342 at 1/21/15 12:46 AM:

[jira] [Updated] (SPARK-5342) Allow long running Spark apps to run on secure YARN/HDFS

[ https://issues.apache.org/jira/browse/SPARK-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated SPARK-5342: Attachment: SparkYARN.pdf Minor updates. Allow long running Spark apps to run on secure

[jira] [Updated] (SPARK-5342) Allow long running Spark apps to run on secure YARN/HDFS

[ https://issues.apache.org/jira/browse/SPARK-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated SPARK-5342: Attachment: (was: SparkYARN.pdf) Allow long running Spark apps to run on secure

[jira] [Updated] (SPARK-5342) Allow long running Spark apps to run on secure YARN/HDFS

[ https://issues.apache.org/jira/browse/SPARK-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated SPARK-5342: Attachment: SparkYARN.pdf Allow long running Spark apps to run on secure YARN/HDFS

[jira] [Commented] (SPARK-5342) Allow long running Spark apps to run on secure YARN/HDFS

[ https://issues.apache.org/jira/browse/SPARK-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14285266#comment-14285266 ] Hari Shreedharan commented on SPARK-5342: - Looks like we cannot write to the cache

[jira] [Updated] (SPARK-5342) Allow long running Spark apps to run on secure YARN/HDFS

[ https://issues.apache.org/jira/browse/SPARK-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated SPARK-5342: Attachment: (was: SparkYARN.pdf) Allow long running Spark apps to run on secure

[jira] [Commented] (SPARK-4122) Add library to write data back to Kafka

[ https://issues.apache.org/jira/browse/SPARK-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270373#comment-14270373 ] Hari Shreedharan commented on SPARK-4122: - The current design doc talks only about

[jira] [Commented] (SPARK-4905) Flaky FlumeStreamSuite test: org.apache.spark.streaming.flume.FlumeStreamSuite.flume input stream

[ https://issues.apache.org/jira/browse/SPARK-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265228#comment-14265228 ] Hari Shreedharan commented on SPARK-4905: - I can't reproduce this - but once

[jira] [Commented] (SPARK-4905) Flaky FlumeStreamSuite test: org.apache.spark.streaming.flume.FlumeStreamSuite.flume input stream

[ https://issues.apache.org/jira/browse/SPARK-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265322#comment-14265322 ] Hari Shreedharan commented on SPARK-4905: - I am not sure. It might have something

[jira] [Commented] (SPARK-4905) Flaky FlumeStreamSuite test: org.apache.spark.streaming.flume.FlumeStreamSuite.flume input stream

[ https://issues.apache.org/jira/browse/SPARK-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14264936#comment-14264936 ] Hari Shreedharan commented on SPARK-4905: - Taking a look now. Flaky

[jira] [Commented] (SPARK-2757) Add Mima test for Spark Sink after 1.10 is released

[ https://issues.apache.org/jira/browse/SPARK-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260531#comment-14260531 ] Hari Shreedharan commented on SPARK-2757: - Actually yes, the SparkSink and the

[jira] [Commented] (SPARK-4960) Interceptor pattern in receivers

[ https://issues.apache.org/jira/browse/SPARK-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258639#comment-14258639 ] Hari Shreedharan commented on SPARK-4960: - Awesome! One idea I had is to call it

[jira] [Commented] (SPARK-3146) Improve the flexibility of Spark Streaming Kafka API to offer user the ability to process message before storing into BM

[ https://issues.apache.org/jira/browse/SPARK-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14254123#comment-14254123 ] Hari Shreedharan commented on SPARK-3146: - For now, I am ok with just adding it to

[jira] [Commented] (SPARK-3146) Improve the flexibility of Spark Streaming Kafka API to offer user the ability to process message before storing into BM

[ https://issues.apache.org/jira/browse/SPARK-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252113#comment-14252113 ] Hari Shreedharan commented on SPARK-3146: - I am a +1 for using something similar

[jira] [Commented] (SPARK-3146) Improve the flexibility of Spark Streaming Kafka API to offer user the ability to process message before storing into BM

[ https://issues.apache.org/jira/browse/SPARK-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252160#comment-14252160 ] Hari Shreedharan commented on SPARK-3146: - The difference between interceptor

[jira] [Commented] (SPARK-3146) Improve the flexibility of Spark Streaming Kafka API to offer user the ability to process message before storing into BM

[ https://issues.apache.org/jira/browse/SPARK-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252256#comment-14252256 ] Hari Shreedharan commented on SPARK-3146: - I understand what you mean by flatMap

[jira] [Commented] (SPARK-3146) Improve the flexibility of Spark Streaming Kafka API to offer user the ability to process message before storing into BM

[ https://issues.apache.org/jira/browse/SPARK-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252319#comment-14252319 ] Hari Shreedharan commented on SPARK-3146: - If you look at the addDataWithCallback

[jira] [Commented] (SPARK-4790) Flaky test in ReceivedBlockTrackerSuite: block addition, block to batch allocation, and cleanup with write ahead log

[ https://issues.apache.org/jira/browse/SPARK-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14250525#comment-14250525 ] Hari Shreedharan commented on SPARK-4790: - Ah, so this is the issue:

[jira] [Commented] (SPARK-4826) Possible flaky tests in WriteAheadLogBackedBlockRDDSuite: java.lang.IllegalStateException: File exists and there is no append support!

[ https://issues.apache.org/jira/browse/SPARK-4826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14246041#comment-14246041 ] Hari Shreedharan commented on SPARK-4826: - It looks like there is some issue with

[jira] [Commented] (SPARK-4826) Possible flaky tests in WriteAheadLogBackedBlockRDDSuite: java.lang.IllegalStateException: File exists and there is no append support!

[ https://issues.apache.org/jira/browse/SPARK-4826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14246270#comment-14246270 ] Hari Shreedharan commented on SPARK-4826: - I suspect that the nextString is

[jira] [Commented] (SPARK-4707) Reliable Kafka Receiver can lose data if the block generator fails to store data

[ https://issues.apache.org/jira/browse/SPARK-4707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14240496#comment-14240496 ] Hari Shreedharan commented on SPARK-4707: - TD and I discussed this and decided

[jira] [Created] (SPARK-4707) Reliable Kafka Receiver can lose data if the block generator fails to store data

Hari Shreedharan created SPARK-4707: --- Summary: Reliable Kafka Receiver can lose data if the block generator fails to store data Key: SPARK-4707 URL: https://issues.apache.org/jira/browse/SPARK-4707

[jira] [Commented] (SPARK-4174) Streaming: Optionally provide notifications to Receivers when DStream has been generated

[ https://issues.apache.org/jira/browse/SPARK-4174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14220196#comment-14220196 ] Hari Shreedharan commented on SPARK-4174: - [~tdas] Comments? If we can agree on

[jira] [Commented] (SPARK-4174) Streaming: Optionally provide notifications to Receivers when DStream has been generated

[ https://issues.apache.org/jira/browse/SPARK-4174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14197464#comment-14197464 ] Hari Shreedharan commented on SPARK-4174: - I will write up a doc soon, but here

[jira] [Created] (SPARK-4174) Optionally provide notifications to Receivers when DStream has been generated

Hari Shreedharan created SPARK-4174: --- Summary: Optionally provide notifications to Receivers when DStream has been generated Key: SPARK-4174 URL: https://issues.apache.org/jira/browse/SPARK-4174

[jira] [Updated] (SPARK-4174) Streaming: Optionally provide notifications to Receivers when DStream has been generated

[ https://issues.apache.org/jira/browse/SPARK-4174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated SPARK-4174: Summary: Streaming: Optionally provide notifications to Receivers when DStream has been

[jira] [Updated] (SPARK-4174) Streaming: Optionally provide notifications to Receivers when DStream has been generated

[ https://issues.apache.org/jira/browse/SPARK-4174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated SPARK-4174: Issue Type: Improvement (was: Bug) Streaming: Optionally provide notifications to

  1   2   >