[jira] [Updated] (SPARK-24987) Kafka Cached Consumer Leaking File Descriptors

[ https://issues.apache.org/jira/browse/SPARK-24987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-24987: Fix Version/s: 2.3.2 > Kafka Cached Consumer Leaking File Descriptors > --

[jira] [Comment Edited] (SPARK-24987) Kafka Cached Consumer Leaking File Descriptors

[ https://issues.apache.org/jira/browse/SPARK-24987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16569432#comment-16569432 ] Yuval Itzchakov edited comment on SPARK-24987 at 8/5/18 10:58 AM:

[jira] [Updated] (SPARK-24987) Kafka Cached Consumer Leaking File Descriptors

[ https://issues.apache.org/jira/browse/SPARK-24987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-24987: Shepherd: (was: Tathagata Das) > Kafka Cached Consumer Leaking File Descriptors > --

[jira] [Comment Edited] (SPARK-24987) Kafka Cached Consumer Leaking File Descriptors

[ https://issues.apache.org/jira/browse/SPARK-24987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16569432#comment-16569432 ] Yuval Itzchakov edited comment on SPARK-24987 at 8/5/18 10:54 AM:

[jira] [Commented] (SPARK-24987) Kafka Cached Consumer Leaking File Descriptors

[ https://issues.apache.org/jira/browse/SPARK-24987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16569432#comment-16569432 ] Yuval Itzchakov commented on SPARK-24987: - Is there any chance this will make it

[jira] [Updated] (SPARK-24987) Kafka Cached Consumer Leaking File Descriptors

[ https://issues.apache.org/jira/browse/SPARK-24987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-24987: Shepherd: Shixiong Zhu (was: Tathagata Das) > Kafka Cached Consumer Leaking File Descript

[jira] [Updated] (SPARK-24987) Kafka Cached Consumer Leaking File Descriptors

[ https://issues.apache.org/jira/browse/SPARK-24987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-24987: Shepherd: Tathagata Das (was: Shixiong Zhu) > Kafka Cached Consumer Leaking File Descript

[jira] [Updated] (SPARK-24987) Kafka Cached Consumer Leaking File Descriptors

[ https://issues.apache.org/jira/browse/SPARK-24987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-24987: Summary: Kafka Cached Consumer Leaking File Descriptors (was: Kafka Cached Consumer Leaki

[jira] [Updated] (SPARK-24987) Kafka Cached Consumer Leaking Consumers

[ https://issues.apache.org/jira/browse/SPARK-24987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-24987: Description: Setup: * Spark 2.3.1 * Java 1.8.0 (112) * Standalone Cluster Manager * 3

[jira] [Updated] (SPARK-24987) Kafka Cached Consumer Leaking Consumers

[ https://issues.apache.org/jira/browse/SPARK-24987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-24987: Description: Spark 2.3.0 introduced a new mechanism for caching Kafka consumers (https://

[jira] [Updated] (SPARK-24987) Kafka Cached Consumer Leaking Consumers

[ https://issues.apache.org/jira/browse/SPARK-24987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-24987: Description: Spark 2.3.0 introduced a new mechanism for caching Kafka consumers (https://

[jira] [Updated] (SPARK-24987) Kafka Cached Consumer Leaking Consumers

[ https://issues.apache.org/jira/browse/SPARK-24987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-24987: Shepherd: Tathagata Das > Kafka Cached Consumer Leaking Consumers > --

[jira] [Updated] (SPARK-24987) Kafka Cached Consumer Leaking Consumers

[ https://issues.apache.org/jira/browse/SPARK-24987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-24987: Description: Spark 2.3.0 introduced a new mechanism for caching Kafka consumers (https://

[jira] [Updated] (SPARK-24987) Kafka Cached Consumer Leaking Consumers

[ https://issues.apache.org/jira/browse/SPARK-24987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-24987: Environment: Spark 2.3.1 Java(TM) SE Runtime Environment (build 1.8.0_112-b15) Java HotS

[jira] [Updated] (SPARK-24987) Kafka Cached Consumer Leaking Consumers

[ https://issues.apache.org/jira/browse/SPARK-24987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-24987: Description: Spark 2.3.0 introduced a new mechanism for caching Kafka consumers (https://

[jira] [Updated] (SPARK-24987) Kafka Cached Consumer Leaking Consumers

[ https://issues.apache.org/jira/browse/SPARK-24987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-24987: Environment: Spark 2.3.1 Java(TM) SE Runtime Environment (build 1.8.0_112-b15) Java HotS

[jira] [Created] (SPARK-24987) Kafka Cached Consumer Leaking Consumers

Yuval Itzchakov created SPARK-24987: --- Summary: Kafka Cached Consumer Leaking Consumers Key: SPARK-24987 URL: https://issues.apache.org/jira/browse/SPARK-24987 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-22305) HDFSBackedStateStoreProvider fails with StackOverflowException when attempting to recover state

[ https://issues.apache.org/jira/browse/SPARK-22305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221982#comment-16221982 ] Yuval Itzchakov commented on SPARK-22305: - [~zsxwing] A typical query takes ~ 0.5

[jira] [Commented] (SPARK-22305) HDFSBackedStateStoreProvider fails with StackOverflowException when attempting to recover state

[ https://issues.apache.org/jira/browse/SPARK-22305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212608#comment-16212608 ] Yuval Itzchakov commented on SPARK-22305: - [~tdas] What do you think? > HDFSBack

[jira] [Comment Edited] (SPARK-22304) HDFSBackedStateStoreProvider stackoverflow due to reloading of very old state

[ https://issues.apache.org/jira/browse/SPARK-22304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209128#comment-16209128 ] Yuval Itzchakov edited comment on SPARK-22304 at 10/18/17 10:41 AM: ---

[jira] [Resolved] (SPARK-22304) HDFSBackedStateStoreProvider stackoverflow due to reloading of very old state

[ https://issues.apache.org/jira/browse/SPARK-22304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov resolved SPARK-22304. - Resolution: Duplicate Accidently opened twice (SPARK-22304) > HDFSBackedStateStoreProvid

[jira] [Updated] (SPARK-22305) HDFSBackedStateStoreProvider fails with StackOverflowException when attempting to recover state

[ https://issues.apache.org/jira/browse/SPARK-22305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-22305: Description: Environment: Spark: 2.2.0 Java version: 1.8.0_112 spark.sql.streaming.minBatc

[jira] [Updated] (SPARK-22305) HDFSBackedStateStoreProvider fails with StackOverflowException when attempting to recover state

[ https://issues.apache.org/jira/browse/SPARK-22305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-22305: Description: Environment: Spark: 2.2.0 Java version: 1.8.0_112 spark.sql.streaming.stateSt

[jira] [Created] (SPARK-22305) HDFSBackedStateStoreProvider fails with StackOverflowException when attempting to recover state

Yuval Itzchakov created SPARK-22305: --- Summary: HDFSBackedStateStoreProvider fails with StackOverflowException when attempting to recover state Key: SPARK-22305 URL: https://issues.apache.org/jira/browse/SPARK-22

[jira] [Created] (SPARK-22304) HDFSBackedStateStoreProvider stackoverflow due to reloading of very old state

Yuval Itzchakov created SPARK-22304: --- Summary: HDFSBackedStateStoreProvider stackoverflow due to reloading of very old state Key: SPARK-22304 URL: https://issues.apache.org/jira/browse/SPARK-22304 P

[jira] [Updated] (SPARK-21873) CachedKafkaConsumer throws NonLocalReturnControl during fetching from Kafka

[ https://issues.apache.org/jira/browse/SPARK-21873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-21873: Description: In Scala, using `return` inside a function causes a `NonLocalReturnControl` e

[jira] [Created] (SPARK-21873) CachedKafkaConsumer throws NonLocalReturnControl during fetching from Kafka

Yuval Itzchakov created SPARK-21873: --- Summary: CachedKafkaConsumer throws NonLocalReturnControl during fetching from Kafka Key: SPARK-21873 URL: https://issues.apache.org/jira/browse/SPARK-21873 Pro

[jira] [Commented] (SPARK-19067) mapGroupsWithState - arbitrary stateful operations with Structured Streaming (similar to DStream.mapWithState)

[ https://issues.apache.org/jira/browse/SPARK-19067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024602#comment-16024602 ] Yuval Itzchakov commented on SPARK-19067: - [~marmbrus] [~tdas] Is there any reas

[jira] [Commented] (SPARK-19067) mapGroupsWithState - arbitrary stateful operations with Structured Streaming (similar to DStream.mapWithState)

[ https://issues.apache.org/jira/browse/SPARK-19067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15961844#comment-15961844 ] Yuval Itzchakov commented on SPARK-19067: - Is this going to make it into 2.1.1?

[jira] [Comment Edited] (SPARK-15406) Structured streaming support for consuming from Kafka

[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15686297#comment-15686297 ] Yuval Itzchakov edited comment on SPARK-15406 at 11/22/16 10:13 AM: ---

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15686297#comment-15686297 ] Yuval Itzchakov commented on SPARK-15406: - [~ankit.kothana], Did you read the doc

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15553493#comment-15553493 ] Yuval Itzchakov commented on SPARK-15406: - We'ee using Kafka in a combination wit

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15553391#comment-15553391 ] Yuval Itzchakov commented on SPARK-15406: - @Michael Perhaps I need to read that P

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15553301#comment-15553301 ] Yuval Itzchakov commented on SPARK-15406: - As someone using Spark Streaming and K

[jira] [Commented] (SPARK-12591) NullPointerException using checkpointed mapWithState with KryoSerializer

[ https://issues.apache.org/jira/browse/SPARK-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235294#comment-15235294 ] Yuval Itzchakov commented on SPARK-12591: - Send me a DM on twitter if you need ad

[jira] [Commented] (SPARK-12591) NullPointerException using checkpointed mapWithState with KryoSerializer

[ https://issues.apache.org/jira/browse/SPARK-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235286#comment-15235286 ] Yuval Itzchakov commented on SPARK-12591: - No, there's no need to register parent

[jira] [Commented] (SPARK-12591) NullPointerException using checkpointed mapWithState with KryoSerializer

[ https://issues.apache.org/jira/browse/SPARK-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235239#comment-15235239 ] Yuval Itzchakov commented on SPARK-12591: - Try registering your classes via `Spar

[jira] [Commented] (SPARK-12591) NullPointerException using checkpointed mapWithState with KryoSerializer

[ https://issues.apache.org/jira/browse/SPARK-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235170#comment-15235170 ] Yuval Itzchakov commented on SPARK-12591: - That's weird, I'm using mapWithState w

[jira] [Commented] (SPARK-12591) NullPointerException using checkpointed mapWithState with KryoSerializer

[ https://issues.apache.org/jira/browse/SPARK-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234880#comment-15234880 ] Yuval Itzchakov commented on SPARK-12591: - This workaround is already builtin Spa

[jira] [Comment Edited] (SPARK-12591) NullPointerException using checkpointed mapWithState with KryoSerializer

[ https://issues.apache.org/jira/browse/SPARK-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15195508#comment-15195508 ] Yuval Itzchakov edited comment on SPARK-12591 at 3/15/16 3:46 PM: -

[jira] [Comment Edited] (SPARK-12591) NullPointerException using checkpointed mapWithState with KryoSerializer

[ https://issues.apache.org/jira/browse/SPARK-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15195508#comment-15195508 ] Yuval Itzchakov edited comment on SPARK-12591 at 3/15/16 3:43 PM: -

[jira] [Comment Edited] (SPARK-12591) NullPointerException using checkpointed mapWithState with KryoSerializer

[ https://issues.apache.org/jira/browse/SPARK-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15195508#comment-15195508 ] Yuval Itzchakov edited comment on SPARK-12591 at 3/15/16 3:41 PM: -

[jira] [Commented] (SPARK-12591) NullPointerException using checkpointed mapWithState with KryoSerializer

[ https://issues.apache.org/jira/browse/SPARK-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15195508#comment-15195508 ] Yuval Itzchakov commented on SPARK-12591: - After applying the fix, I'm still seei

[jira] [Created] (SPARK-13743) Adding configurable support for Spark Streaming gracefull timeout

Yuval Itzchakov created SPARK-13743: --- Summary: Adding configurable support for Spark Streaming gracefull timeout Key: SPARK-13743 URL: https://issues.apache.org/jira/browse/SPARK-13743 Project: Spar

[jira] [Comment Edited] (SPARK-12591) NullPointerException using checkpointed mapWithState with KryoSerializer

[ https://issues.apache.org/jira/browse/SPARK-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136208#comment-15136208 ] Yuval Itzchakov edited comment on SPARK-12591 at 2/7/16 10:47 AM: -

[jira] [Comment Edited] (SPARK-12591) NullPointerException using checkpointed mapWithState with KryoSerializer

[ https://issues.apache.org/jira/browse/SPARK-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136208#comment-15136208 ] Yuval Itzchakov edited comment on SPARK-12591 at 2/7/16 10:04 AM: -

[jira] [Commented] (SPARK-12591) NullPointerException using checkpointed mapWithState with KryoSerializer

[ https://issues.apache.org/jira/browse/SPARK-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136208#comment-15136208 ] Yuval Itzchakov commented on SPARK-12591: - I'm seeing this error as well running

[jira] [Comment Edited] (SPARK-12591) NullPointerException using checkpointed mapWithState with KryoSerializer

[ https://issues.apache.org/jira/browse/SPARK-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136208#comment-15136208 ] Yuval Itzchakov edited comment on SPARK-12591 at 2/7/16 10:03 AM: -

[jira] [Updated] (SPARK-13195) PairDStreamFunctions.mapWithState fails in case timeout is set without updating State[S]

[ https://issues.apache.org/jira/browse/SPARK-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-13195: Description: Using the new spark mapWithState API, I've encountered a bug when setting a t

[jira] [Updated] (SPARK-13195) PairDStreamFunctions.mapWithState fails in case timeout is set without updating State[S]

[ https://issues.apache.org/jira/browse/SPARK-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-13195: Description: Using the new spark mapWithState API, I've encountered a bug when setting a t

[jira] [Updated] (SPARK-13195) PairDStreamFunctions.mapWithState fails in case timeout is set without updating State[S]

[ https://issues.apache.org/jira/browse/SPARK-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-13195: Description: Using the new spark mapWithState API, I've encountered a bug when setting a t

[jira] [Updated] (SPARK-13195) PairDStreamFunctions.mapWithState fails in case timeout is set without updating State[S]

[ https://issues.apache.org/jira/browse/SPARK-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-13195: Description: Using the new spark mapWithState API, I've encountered a bug when setting a t

[jira] [Updated] (SPARK-13195) PairDStreamFunctions.mapWithState fails in case timeout is set without updating State[S]

[ https://issues.apache.org/jira/browse/SPARK-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-13195: Summary: PairDStreamFunctions.mapWithState fails in case timeout is set without updating St

[jira] [Updated] (SPARK-13195) PairDStreamFunctions.mapWithState fails in case timeout is set without state update

[ https://issues.apache.org/jira/browse/SPARK-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-13195: Summary: PairDStreamFunctions.mapWithState fails in case timeout is set without state updat

[jira] [Updated] (SPARK-13195) mapWithState fails in case of timeout set without state update

[ https://issues.apache.org/jira/browse/SPARK-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-13195: Description: Using the new spark mapWithState API, I've encountered a bug when setting a t

[jira] [Updated] (SPARK-13195) PairDStreamFunctions.mapWithState fails in case of timeout set without state update

[ https://issues.apache.org/jira/browse/SPARK-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-13195: Summary: PairDStreamFunctions.mapWithState fails in case of timeout set without state updat

[jira] [Updated] (SPARK-13195) mapWithState fails in case of timeout set without state update

[ https://issues.apache.org/jira/browse/SPARK-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-13195: Description: Using the new spark mapWithState API, I've encountered a bug when setting a t

[jira] [Updated] (SPARK-13195) mapWithState fails in case of timeout set without state update

[ https://issues.apache.org/jira/browse/SPARK-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-13195: Description: Using the new spark mapWithState API, I've encountered a bug when setting a t

[jira] [Updated] (SPARK-13195) mapWithState fails in case of timeout set without state update

[ https://issues.apache.org/jira/browse/SPARK-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-13195: Description: Using the new spark mapWithState API, I've encountered a bug when setting a t

[jira] [Updated] (SPARK-13195) mapWithState fails in case of timeout set without state update

[ https://issues.apache.org/jira/browse/SPARK-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-13195: Description: Using the new spark mapWithState API, I've encountered a bug when setting a t

[jira] [Updated] (SPARK-13195) mapWithState fails in case of timeout set without state update

[ https://issues.apache.org/jira/browse/SPARK-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-13195: Description: Using the new spark mapWithState API, I've encountered a bug when setting a t

[jira] [Updated] (SPARK-13195) mapWithState fails in case of timeout set without state update

[ https://issues.apache.org/jira/browse/SPARK-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuval Itzchakov updated SPARK-13195: Description: Using the new spark mapWithState API, I've encountered a bug when setting a t

[jira] [Created] (SPARK-13195) mapWithState fails in case of timeout set without state update

Yuval Itzchakov created SPARK-13195: --- Summary: mapWithState fails in case of timeout set without state update Key: SPARK-13195 URL: https://issues.apache.org/jira/browse/SPARK-13195 Project: Spark