[jira] [Commented] (SPARK-8360) Streaming DataFrames

[ https://issues.apache.org/jira/browse/SPARK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190414#comment-15190414 ] Tathagata Das commented on SPARK-8360: -- This is still highly WIP, and not ready for even experimental

[jira] [Created] (SPARK-13146) API for managing streaming dataframes

Tathagata Das created SPARK-13146: - Summary: API for managing streaming dataframes Key: SPARK-13146 URL: https://issues.apache.org/jira/browse/SPARK-13146 Project: Spark Issue Type: Sub-task

[jira] [Resolved] (SPARK-12847) Remove StreamingListenerBus and post all Streaming events to the same thread as Spark events

[ https://issues.apache.org/jira/browse/SPARK-12847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-12847. --- Resolution: Fixed Fix Version/s: 2.0.0 > Remove StreamingListenerBus and post all

[jira] [Updated] (SPARK-7799) Move "StreamingContext.actorStream" to a separate project and deprecate it in StreamingContext

[ https://issues.apache.org/jira/browse/SPARK-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-7799: - Assignee: Shixiong Zhu (was: Tathagata Das) > Move "StreamingContext.actorStream" to a separate

[jira] [Assigned] (SPARK-7799) Move "StreamingContext.actorStream" to a separate project and deprecate it in StreamingContext

[ https://issues.apache.org/jira/browse/SPARK-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-7799: Assignee: Tathagata Das > Move "StreamingContext.actorStream" to a separate project and

[jira] [Commented] (SPARK-12140) Support Streaming UI in HistoryServer

[ https://issues.apache.org/jira/browse/SPARK-12140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15109316#comment-15109316 ] Tathagata Das commented on SPARK-12140: --- BTW, a major concern in supporting streaming stuff in

[jira] [Resolved] (SPARK-7799) Move "StreamingContext.actorStream" to a separate project and deprecate it in StreamingContext

[ https://issues.apache.org/jira/browse/SPARK-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-7799. -- Resolution: Fixed Fix Version/s: 2.0.0 > Move "StreamingContext.actorStream" to a

[jira] [Resolved] (SPARK-12786) Actor demo does not demonstrate usable code

[ https://issues.apache.org/jira/browse/SPARK-12786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-12786. --- Resolution: Fixed Assignee: Tathagata Das Fix Version/s: 2.0.0 > Actor demo

[jira] [Resolved] (SPARK-12814) Add deploy instructions for Python in flume integration doc

[ https://issues.apache.org/jira/browse/SPARK-12814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-12814. --- Resolution: Fixed Fix Version/s: 2.0.0 1.6.1 > Add deploy

[jira] [Resolved] (SPARK-12894) Add deploy instructions for Python in Kinesis integration doc

[ https://issues.apache.org/jira/browse/SPARK-12894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-12894. --- Resolution: Fixed Fix Version/s: 2.0.0 1.6.1 > Add deploy

[jira] [Updated] (SPARK-12591) NullPointerException using checkpointed mapWithState with KryoSerializer

[ https://issues.apache.org/jira/browse/SPARK-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-12591: -- Fix Version/s: 1.6.1 > NullPointerException using checkpointed mapWithState with

[jira] [Resolved] (SPARK-12507) Expose closeFileAfterWrite and allowBatching configurations for Streaming

[ https://issues.apache.org/jira/browse/SPARK-12507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-12507. --- Resolution: Fixed Fix Version/s: 2.0.0 1.6.1 > Expose

[jira] [Resolved] (SPARK-12591) NullPointerException using checkpointed mapWithState with KryoSerializer

[ https://issues.apache.org/jira/browse/SPARK-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-12591. --- Resolution: Fixed Fix Version/s: 2.0.0 > NullPointerException using checkpointed

[jira] [Updated] (SPARK-12591) NullPointerException using checkpointed mapWithState with KryoSerializer

[ https://issues.apache.org/jira/browse/SPARK-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-12591: -- Assignee: Shixiong Zhu > NullPointerException using checkpointed mapWithState with

[jira] [Updated] (SPARK-12429) Update documentation to show how to use accumulators and broadcasts with Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-12429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-12429: -- Assignee: Shixiong Zhu (was: Apache Spark) > Update documentation to show how to use

[jira] [Resolved] (SPARK-12429) Update documentation to show how to use accumulators and broadcasts with Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-12429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-12429. --- Resolution: Fixed Fix Version/s: 1.6.0 > Update documentation to show how to use

[jira] [Resolved] (SPARK-12487) Add docs for Kafka message handler

[ https://issues.apache.org/jira/browse/SPARK-12487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-12487. --- Resolution: Fixed Fix Version/s: 1.6.0 > Add docs for Kafka message handler >

[jira] [Created] (SPARK-12244) Rename trackStateByKey to mapWithState

Tathagata Das created SPARK-12244: - Summary: Rename trackStateByKey to mapWithState Key: SPARK-12244 URL: https://issues.apache.org/jira/browse/SPARK-12244 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-12244) Rename trackStateByKey to mapWithState

[ https://issues.apache.org/jira/browse/SPARK-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-12244: -- Priority: Blocker (was: Major) > Rename trackStateByKey to mapWithState >

[jira] [Created] (SPARK-12245) Change the simple tracking function signature to include key in the params

Tathagata Das created SPARK-12245: - Summary: Change the simple tracking function signature to include key in the params Key: SPARK-12245 URL: https://issues.apache.org/jira/browse/SPARK-12245

[jira] [Updated] (SPARK-2629) Improved state management for Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-2629: - Description: Current updateStateByKey provides stateful processing in Spark Streaming. It allows

[jira] [Updated] (SPARK-2629) Improved state management for Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-2629: - Description: Current updateStateByKey provides stateful processing in Spark Streaming. It

[jira] [Resolved] (SPARK-12106) Flaky Test: BatchedWriteAheadLog - name log with aggregated entries with the timestamp of last entry

[ https://issues.apache.org/jira/browse/SPARK-12106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-12106. --- Resolution: Fixed Assignee: Burak Yavuz Fix Version/s: 1.6.0 > Flaky Test:

[jira] [Resolved] (SPARK-11932) trackStateByKey throws java.lang.IllegalArgumentException: requirement failed on restarting from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-11932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11932. --- Resolution: Fixed Fix Version/s: 1.6.0 > trackStateByKey throws

[jira] [Resolved] (SPARK-12122) Recovered streaming context can sometimes run a batch twice

[ https://issues.apache.org/jira/browse/SPARK-12122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-12122. --- Resolution: Fixed Fix Version/s: 1.6.0 > Recovered streaming context can sometimes

[jira] [Resolved] (SPARK-12058) Fix KinesisStreamTests in python/pyspark/streaming/tests.py and enable it

[ https://issues.apache.org/jira/browse/SPARK-12058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-12058. --- Resolution: Fixed Fix Version/s: 1.6.0 > Fix KinesisStreamTests in

[jira] [Created] (SPARK-12122) Recovered streaming context can sometimes run a batch twice

Tathagata Das created SPARK-12122: - Summary: Recovered streaming context can sometimes run a batch twice Key: SPARK-12122 URL: https://issues.apache.org/jira/browse/SPARK-12122 Project: Spark

[jira] [Created] (SPARK-12087) DStream.saveAsHadoopFiles can throw ConcurrentModificationException

Tathagata Das created SPARK-12087: - Summary: DStream.saveAsHadoopFiles can throw ConcurrentModificationException Key: SPARK-12087 URL: https://issues.apache.org/jira/browse/SPARK-12087 Project: Spark

[jira] [Issue Comment Deleted] (SPARK-11932) trackStateByKey throws java.lang.IllegalArgumentException: requirement failed on restarting from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-11932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-11932: -- Comment: was deleted (was: User 'tmnd1991' has created a pull request for this issue:

[jira] [Created] (SPARK-12004) RDD checkpointing does not preserve partitioner

Tathagata Das created SPARK-12004: - Summary: RDD checkpointing does not preserve partitioner Key: SPARK-12004 URL: https://issues.apache.org/jira/browse/SPARK-12004 Project: Spark Issue

[jira] [Commented] (SPARK-11932) trackStateByKey throws java.lang.IllegalArgumentException: requirement failed on restarting from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-11932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027937#comment-15027937 ] Tathagata Das commented on SPARK-11932: --- The reason why trackStateByKey throws this exception is

[jira] [Updated] (SPARK-11932) trackStateByKey throws java.lang.IllegalArgumentException: requirement failed on restarting from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-11932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-11932: -- Description: The problem is that Code {code} StreamingContext.getOrCreate(".", () =>

[jira] [Comment Edited] (SPARK-11932) trackStateByKey throws java.lang.IllegalArgumentException: requirement failed on restarting from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-11932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027937#comment-15027937 ] Tathagata Das edited comment on SPARK-11932 at 11/26/15 1:07 AM: - The

[jira] [Comment Edited] (SPARK-11932) trackStateByKey throws java.lang.IllegalArgumentException: requirement failed on restarting from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-11932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027937#comment-15027937 ] Tathagata Das edited comment on SPARK-11932 at 11/26/15 2:06 AM: - The

[jira] [Comment Edited] (SPARK-11932) trackStateByKey throws java.lang.IllegalArgumentException: requirement failed on restarting from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-11932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027937#comment-15027937 ] Tathagata Das edited comment on SPARK-11932 at 11/26/15 2:04 AM: - The

[jira] [Updated] (SPARK-11932) trackStateByKey throws java.lang.IllegalArgumentException: requirement failed on restarting from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-11932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-11932: -- Description: The problem is that when recovering a streaming application using

[jira] [Comment Edited] (SPARK-11932) trackStateByKey throws java.lang.IllegalArgumentException: requirement failed on restarting from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-11932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027937#comment-15027937 ] Tathagata Das edited comment on SPARK-11932 at 11/26/15 1:07 AM: - The

[jira] [Updated] (SPARK-11935) Send the Python exceptions in TransformFunction and TransformFunctionSerializer to Java

[ https://issues.apache.org/jira/browse/SPARK-11935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-11935: -- Assignee: Shixiong Zhu > Send the Python exceptions in TransformFunction and >

[jira] [Resolved] (SPARK-11935) Send the Python exceptions in TransformFunction and TransformFunctionSerializer to Java

[ https://issues.apache.org/jira/browse/SPARK-11935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11935. --- Resolution: Fixed Fix Version/s: 1.6.0 > Send the Python exceptions in

[jira] [Created] (SPARK-11979) Empty TrackStateRDD cannot be checkpointed and recovered from checkpoint file

Tathagata Das created SPARK-11979: - Summary: Empty TrackStateRDD cannot be checkpointed and recovered from checkpoint file Key: SPARK-11979 URL: https://issues.apache.org/jira/browse/SPARK-11979

[jira] [Updated] (SPARK-11979) Empty TrackStateRDD cannot be checkpointed and recovered from checkpoint file

[ https://issues.apache.org/jira/browse/SPARK-11979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-11979: -- Description: {code} Job aborted due to stage failure: Task 0 in stage 6.0 failed 1 times, most

[jira] [Updated] (SPARK-11932) trackStateByKey throws java.lang.IllegalArgumentException: requirement failed on restarting from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-11932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-11932: -- Description: Code {code} StreamingContext.getOrCreate(".", () => createContext(args)) ...

[jira] [Updated] (SPARK-11932) trackStateByKey throws java.lang.IllegalArgumentException: requirement failed on restarting from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-11932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-11932: -- Description: Code {{{ StreamingContext.getOrCreate(".", () => createContext(args)) ...

[jira] [Created] (SPARK-11932) trackStateByKey throws java.lang.IllegalArgumentException: requirement failed on restarting from checkpoint

Tathagata Das created SPARK-11932: - Summary: trackStateByKey throws java.lang.IllegalArgumentException: requirement failed on restarting from checkpoint Key: SPARK-11932 URL:

[jira] [Resolved] (SPARK-11812) pyspark reduceByKeyAndWindow does not handle unspecified invFunc (invFunc=None)

[ https://issues.apache.org/jira/browse/SPARK-11812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11812. --- Resolution: Fixed Fix Version/s: 1.6.0 1.5.3

[jira] [Updated] (SPARK-11812) pyspark reduceByKeyAndWindow does not handle unspecified invFunc (invFunc=None)

[ https://issues.apache.org/jira/browse/SPARK-11812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-11812: -- Assignee: David Tolpin > pyspark reduceByKeyAndWindow does not handle unspecified invFunc >

[jira] [Updated] (SPARK-11812) pyspark reduceByKeyAndWindow does not handle unspecified invFunc (invFunc=None)

[ https://issues.apache.org/jira/browse/SPARK-11812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-11812: -- Shepherd: Shixiong(Ryan) Zhu > pyspark reduceByKeyAndWindow does not handle unspecified

[jira] [Resolved] (SPARK-4557) Spark Streaming' foreachRDD method should accept a VoidFunction<...>, not a Function<..., Void>

[ https://issues.apache.org/jira/browse/SPARK-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-4557. -- Resolution: Fixed Assignee: Bryan Cutler Fix Version/s: 1.6.0 > Spark

[jira] [Updated] (SPARK-4557) Spark Streaming' foreachRDD method should accept a VoidFunction<...>, not a Function<..., Void>

[ https://issues.apache.org/jira/browse/SPARK-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-4557: - Target Version/s: (was: 2.0.0) > Spark Streaming' foreachRDD method should accept a

[jira] [Created] (SPARK-11814) Set better default DStream checkpoint interval

Tathagata Das created SPARK-11814: - Summary: Set better default DStream checkpoint interval Key: SPARK-11814 URL: https://issues.apache.org/jira/browse/SPARK-11814 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-11814) Set better default DStream checkpoint interval

[ https://issues.apache.org/jira/browse/SPARK-11814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11814. --- Resolution: Fixed Fix Version/s: 1.6.0 > Set better default DStream checkpoint

[jira] [Resolved] (SPARK-11791) Flaky test: org.apache.spark.streaming.util.BatchedWriteAheadLogSuite.BatchedWriteAheadLog

[ https://issues.apache.org/jira/browse/SPARK-11791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11791. --- Resolution: Fixed Assignee: Burak Yavuz Fix Version/s: 1.6.0 > Flaky test:

[jira] [Updated] (SPARK-11845) Add unit tests to verify correct checkpointing of TrackStateRDD

[ https://issues.apache.org/jira/browse/SPARK-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-11845: -- Target Version/s: 1.6.0 > Add unit tests to verify correct checkpointing of TrackStateRDD >

[jira] [Created] (SPARK-11845) Add unit tests to verify correct checkpointing of TrackStateRDD

Tathagata Das created SPARK-11845: - Summary: Add unit tests to verify correct checkpointing of TrackStateRDD Key: SPARK-11845 URL: https://issues.apache.org/jira/browse/SPARK-11845 Project: Spark

[jira] [Updated] (SPARK-11845) Add unit tests to verify correct checkpointing of TrackStateRDD

[ https://issues.apache.org/jira/browse/SPARK-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-11845: -- Issue Type: Test (was: Bug) > Add unit tests to verify correct checkpointing of TrackStateRDD

[jira] [Resolved] (SPARK-11740) Fix DStream checkpointing logic to prevent failures during checkpoint recovery

[ https://issues.apache.org/jira/browse/SPARK-11740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11740. --- Resolution: Fixed Assignee: Shixiong Zhu Fix Version/s: 1.6.0 > Fix DStream

[jira] [Resolved] (SPARK-9065) Add the ability to specify message handler function in python similar to Scala/Java

[ https://issues.apache.org/jira/browse/SPARK-9065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-9065. -- Resolution: Fixed Fix Version/s: 1.6.0 > Add the ability to specify message handler

[jira] [Updated] (SPARK-2629) Improved state management for Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-2629: - Target Version/s: (was: 1.6.0) > Improved state management for Spark Streaming >

[jira] [Resolved] (SPARK-9573) Forward exceptions in batch jobs to the awaitTermination thread

[ https://issues.apache.org/jira/browse/SPARK-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-9573. -- Resolution: Invalid > Forward exceptions in batch jobs to the awaitTermination thread >

[jira] [Resolved] (SPARK-11761) Prevent the call to StreamingContext#stop() in the listener bus's thread

[ https://issues.apache.org/jira/browse/SPARK-11761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11761. --- Resolution: Fixed Assignee: Ted Yu Fix Version/s: 1.6.0 > Prevent the call

[jira] [Updated] (SPARK-7341) Fix the flaky test: org.apache.spark.streaming.InputStreamsSuite.socket input stream

[ https://issues.apache.org/jira/browse/SPARK-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-7341: - Target Version/s: (was: 1.6.0) > Fix the flaky test:

[jira] [Resolved] (SPARK-6328) Python API for StreamingListener

[ https://issues.apache.org/jira/browse/SPARK-6328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-6328. -- Resolution: Fixed Fix Version/s: 1.6.0 > Python API for StreamingListener >

[jira] [Resolved] (SPARK-11731) Enable batching on Driver WriteAheadLog by default

[ https://issues.apache.org/jira/browse/SPARK-11731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11731. --- Resolution: Fixed Assignee: Burak Yavuz Fix Version/s: 1.6.0 > Enable

[jira] [Updated] (SPARK-6328) Python API for StreamingListener

[ https://issues.apache.org/jira/browse/SPARK-6328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-6328: - Assignee: Yifan Wang > Python API for StreamingListener > > >

[jira] [Resolved] (SPARK-11742) Show batch failures in the Streaming UI landing page

[ https://issues.apache.org/jira/browse/SPARK-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11742. --- Resolution: Fixed Fix Version/s: 1.6.0 > Show batch failures in the Streaming UI

[jira] [Updated] (SPARK-11742) Show batch failures in the Streaming UI landing page

[ https://issues.apache.org/jira/browse/SPARK-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-11742: -- Assignee: Shixiong Zhu > Show batch failures in the Streaming UI landing page >

[jira] [Resolved] (SPARK-11706) Streaming Python tests cannot report failures

[ https://issues.apache.org/jira/browse/SPARK-11706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11706. --- Resolution: Fixed Assignee: Shixiong Zhu Fix Version/s: 1.6.0 > Streaming

[jira] [Resolved] (SPARK-11663) Add Java API for trackStateByKey

[ https://issues.apache.org/jira/browse/SPARK-11663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11663. --- Resolution: Fixed Fix Version/s: 1.6.0 > Add Java API for trackStateByKey >

[jira] [Resolved] (SPARK-11681) Fix state timestamp update logic in TrackStateRDD

[ https://issues.apache.org/jira/browse/SPARK-11681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11681. --- Resolution: Fixed Fix Version/s: 1.6.0 > Fix state timestamp update logic in

[jira] [Resolved] (SPARK-11419) WriteAheadLog recovery improvements for when closeFileAfterWrite is enabled

[ https://issues.apache.org/jira/browse/SPARK-11419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11419. --- Resolution: Fixed Assignee: Burak Yavuz Fix Version/s: 1.6.0 > WriteAheadLog

[jira] [Created] (SPARK-11681) Fix state timestamp update logic in TrackStateRDD

Tathagata Das created SPARK-11681: - Summary: Fix state timestamp update logic in TrackStateRDD Key: SPARK-11681 URL: https://issues.apache.org/jira/browse/SPARK-11681 Project: Spark Issue

[jira] [Resolved] (SPARK-11639) Flaky test: BatchedWriteAheadLog - name log with aggregated entries with the timestamp of last entry

[ https://issues.apache.org/jira/browse/SPARK-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11639. --- Resolution: Fixed Assignee: Burak Yavuz Fix Version/s: 1.6.0 > Flaky test:

[jira] [Created] (SPARK-11663) Add Java API for trackStateByKey

Tathagata Das created SPARK-11663: - Summary: Add Java API for trackStateByKey Key: SPARK-11663 URL: https://issues.apache.org/jira/browse/SPARK-11663 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-11335) Update documentation on accessing Kafka offsets from Python

[ https://issues.apache.org/jira/browse/SPARK-11335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11335. --- Resolution: Fixed Assignee: Nick Evans Fix Version/s: 1.6.0 > Update

[jira] [Updated] (SPARK-2629) Improved state management for Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-2629: - Epic Name: Improved State Management > Improved state management for Spark Streaming >

[jira] [Resolved] (SPARK-11290) Implement trackStateByKey for improved state management

[ https://issues.apache.org/jira/browse/SPARK-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11290. --- Resolution: Fixed Fix Version/s: 1.6.0 > Implement trackStateByKey for improved state

[jira] [Resolved] (SPARK-11462) Add JavaStreamingListener

[ https://issues.apache.org/jira/browse/SPARK-11462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11462. --- Resolution: Fixed Assignee: Shixiong Zhu Fix Version/s: 1.6.0 > Add

[jira] [Resolved] (SPARK-11141) Batching of ReceivedBlockTrackerLogEvents for efficient WAL writes

[ https://issues.apache.org/jira/browse/SPARK-11141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11141. --- Resolution: Fixed Assignee: Burak Yavuz Fix Version/s: 1.6.0 > Batching of

[jira] [Resolved] (SPARK-11333) Add the receiver's executor information to UI

[ https://issues.apache.org/jira/browse/SPARK-11333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11333. --- Resolution: Fixed Fix Version/s: 1.6.0 > Add the receiver's executor information to

[jira] [Resolved] (SPARK-11359) Kinesis receiver does not checkpoint to DynamoDB if there is no new data.

[ https://issues.apache.org/jira/browse/SPARK-11359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11359. --- Resolution: Fixed Fix Version/s: 1.6.0 > Kinesis receiver does not checkpoint to

[jira] [Resolved] (SPARK-11198) Support record de-aggregation in KinesisReceiver

[ https://issues.apache.org/jira/browse/SPARK-11198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11198. --- Resolution: Fixed Fix Version/s: 1.6.0 > Support record de-aggregation in

[jira] [Updated] (SPARK-11378) StreamingContext.awaitTerminationOrTimeout does not return

[ https://issues.apache.org/jira/browse/SPARK-11378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-11378: -- Assignee: Nick Evans > StreamingContext.awaitTerminationOrTimeout does not return >

[jira] [Updated] (SPARK-11378) StreamingContext.awaitTerminationOrTimeout does not return

[ https://issues.apache.org/jira/browse/SPARK-11378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-11378: -- Target Version/s: 1.6.0 > StreamingContext.awaitTerminationOrTimeout does not return >

[jira] [Resolved] (SPARK-11270) Add improved equality testing for TopicAndPartition from the Kafka Streaming API

[ https://issues.apache.org/jira/browse/SPARK-11270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11270. --- Resolution: Fixed Fix Version/s: 1.6.0 1.5.3 > Add improved

[jira] [Resolved] (SPARK-11324) Flag to close Write Ahead Log after writing

[ https://issues.apache.org/jira/browse/SPARK-11324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11324. --- Resolution: Fixed Assignee: Burak Yavuz Fix Version/s: 1.6.0 > Flag to close

[jira] [Resolved] (SPARK-11212) Make RDD's preferred locations support the executor location and fix ReceiverTracker for multiple executors in a host

[ https://issues.apache.org/jira/browse/SPARK-11212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11212. --- Resolution: Fixed Assignee: Shixiong Zhu Fix Version/s: 1.6.0 > Make RDD's

[jira] [Created] (SPARK-11359) Kinesis receiver does not checkpoint to DynamoDB if there is no new data.

Tathagata Das created SPARK-11359: - Summary: Kinesis receiver does not checkpoint to DynamoDB if there is no new data. Key: SPARK-11359 URL: https://issues.apache.org/jira/browse/SPARK-11359

[jira] [Updated] (SPARK-11361) Show scopes of RDD operations inside DStream.foreachRDD and DStream.transform in DAG viz

[ https://issues.apache.org/jira/browse/SPARK-11361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-11361: -- Description: Currently, when a DStream sets the scope for RDD generated by it, that scope is

[jira] [Updated] (SPARK-11333) Add the receiver's executor information to UI

[ https://issues.apache.org/jira/browse/SPARK-11333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-11333: -- Assignee: Shixiong Zhu > Add the receiver's executor information to UI >

[jira] [Commented] (SPARK-5206) Accumulators are not re-registered during recovering from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-5206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975572#comment-14975572 ] Tathagata Das commented on SPARK-5206: -- Same singleton idea. Have a singleton reference to broadcast

[jira] [Resolved] (SPARK-10891) Add MessageHandler to KinesisUtils.createStream similar to Direct Kafka

[ https://issues.apache.org/jira/browse/SPARK-10891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-10891. --- Resolution: Fixed Assignee: Burak Yavuz Fix Version/s: 1.6.0 > Add

[jira] [Resolved] (SPARK-11127) Upgrade Kinesis Client Library to the latest stable version

[ https://issues.apache.org/jira/browse/SPARK-11127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11127. --- Resolution: Fixed Fix Version/s: 1.6.0 > Upgrade Kinesis Client Library to the latest

[jira] [Created] (SPARK-11290) Implement trackStateByKey for improved state management

Tathagata Das created SPARK-11290: - Summary: Implement trackStateByKey for improved state management Key: SPARK-11290 URL: https://issues.apache.org/jira/browse/SPARK-11290 Project: Spark

[jira] [Updated] (SPARK-2629) Improved state management for Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-2629: - Issue Type: Epic (was: Improvement) > Improved state management for Spark Streaming >

[jira] [Updated] (SPARK-2629) Improved state management for Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-2629: - Description: Current updateStateByKey provides stateful processing in Spark Streaming. It allows

[jira] [Updated] (SPARK-2629) Improved state management for Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-2629: - Description: Current updateStateByKey provides stateful processing in Spark Streaming. It allows

[jira] [Updated] (SPARK-2629) Improved state management for Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-2629: - Summary: Improved state management for Spark Streaming (was: Improve performance of

[jira] [Updated] (SPARK-2629) Improved state management for Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-2629: - Description: Current updateStateByKey provides stateful processing in Spark Streaming. It allows

[jira] [Created] (SPARK-11291) Implement trackStateByKey for improvement state management

Tathagata Das created SPARK-11291: - Summary: Implement trackStateByKey for improvement state management Key: SPARK-11291 URL: https://issues.apache.org/jira/browse/SPARK-11291 Project: Spark

[jira] [Updated] (SPARK-10955) Warn if dynamic allocation is enabled for Streaming jobs

[ https://issues.apache.org/jira/browse/SPARK-10955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-10955: -- Summary: Warn if dynamic allocation is enabled for Streaming jobs (was: Disable dynamic

<    1   2   3   4   5   6   7   8   9   10   >