[jira] [Commented] (SPARK-19067) mapGroupsWithState - arbitrary stateful operations with Structured Streaming (similar to DStream.mapWithState)

[ https://issues.apache.org/jira/browse/SPARK-19067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958403#comment-15958403 ] Amit Sela commented on SPARK-19067: --- [~tdas] should I open a ticket, for future improvements, about

[jira] [Comment Edited] (SPARK-19067) mapGroupsWithState - arbitrary stateful operations with Structured Streaming (similar to DStream.mapWithState)

[ https://issues.apache.org/jira/browse/SPARK-19067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903532#comment-15903532 ] Amit Sela edited comment on SPARK-19067 at 3/9/17 6:15 PM: --- [~tdas] I just read

[jira] [Commented] (SPARK-19067) mapGroupsWithState - arbitrary stateful operations with Structured Streaming (similar to DStream.mapWithState)

[ https://issues.apache.org/jira/browse/SPARK-19067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903532#comment-15903532 ] Amit Sela commented on SPARK-19067: --- [~tdas] I just read the PR, and I'm very excited for Spark to

[jira] [Commented] (SPARK-19067) mapGroupsWithState - arbitrary stateful operations with Structured Streaming (similar to DStream.mapWithState)

[ https://issues.apache.org/jira/browse/SPARK-19067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897868#comment-15897868 ] Amit Sela commented on SPARK-19067: --- Sweet! I will make time tomorrow to go over the PR thoroughly

[jira] [Commented] (SPARK-19067) mapGroupsWithState - arbitrary stateful operations with Structured Streaming (similar to DStream.mapWithState)

[ https://issues.apache.org/jira/browse/SPARK-19067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896866#comment-15896866 ] Amit Sela commented on SPARK-19067: --- It depends, will those timers be "resetable" ? So that once I

[jira] [Commented] (SPARK-19067) mapGroupsWithState - arbitrary stateful operations with Structured Streaming (similar to DStream.mapWithState)

[ https://issues.apache.org/jira/browse/SPARK-19067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859228#comment-15859228 ] Amit Sela commented on SPARK-19067: --- [~tdas] I would like to add another feature to consider, that

[jira] [Commented] (SPARK-10816) EventTime based sessionization

[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674147#comment-15674147 ] Amit Sela commented on SPARK-10816: --- [~rxin] it might be worth taking into account a more generic

[jira] [Commented] (SPARK-16607) Aggregator with null initialisation will result in null

[ https://issues.apache.org/jira/browse/SPARK-16607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382602#comment-15382602 ] Amit Sela commented on SPARK-16607: --- Copy of the thread discussing this in the user mailing list:

[jira] [Commented] (SPARK-15810) Aggregator doesn't play nice with Option

[ https://issues.apache.org/jira/browse/SPARK-15810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382586#comment-15382586 ] Amit Sela commented on SPARK-15810: --- opened https://issues.apache.org/jira/browse/SPARK-16607 >

[jira] [Updated] (SPARK-16607) Aggregator with null initialisation will result in null

[ https://issues.apache.org/jira/browse/SPARK-16607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated SPARK-16607: -- Description: Java code example: {code} SparkSession session = SparkSession.builder()

[jira] [Created] (SPARK-16607) Aggregator with null initialisation will result in null

Amit Sela created SPARK-16607: - Summary: Aggregator with null initialisation will result in null Key: SPARK-16607 URL: https://issues.apache.org/jira/browse/SPARK-16607 Project: Spark Issue

[jira] [Commented] (SPARK-15810) Aggregator doesn't play nice with Option

[ https://issues.apache.org/jira/browse/SPARK-15810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15370494#comment-15370494 ] Amit Sela commented on SPARK-15810: --- [~rxin] what's your take on the Java API issue with null ? does

[jira] [Closed] (SPARK-16474) Global Aggregation doesn't seem to work at all

[ https://issues.apache.org/jira/browse/SPARK-16474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela closed SPARK-16474. - Resolution: Not A Problem It seems as if the right way to use the agg() API directly on

[jira] [Commented] (SPARK-16474) Global Aggregation doesn't seem to work at all

[ https://issues.apache.org/jira/browse/SPARK-16474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369829#comment-15369829 ] Amit Sela commented on SPARK-16474: --- I thought the bufferEncoder is supposed to take care of that..

[jira] [Commented] (SPARK-16474) Global Aggregation doesn't seem to work at all

[ https://issues.apache.org/jira/browse/SPARK-16474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369706#comment-15369706 ] Amit Sela commented on SPARK-16474: --- Thanks [~koert] that works. > Global Aggregation doesn't seem to

[jira] [Comment Edited] (SPARK-15810) Aggregator doesn't play nice with Option

[ https://issues.apache.org/jira/browse/SPARK-15810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369496#comment-15369496 ] Amit Sela edited comment on SPARK-15810 at 7/10/16 2:53 PM: Just ran this

[jira] [Created] (SPARK-16474) Global Aggregation doesn't seem to work at all

Amit Sela created SPARK-16474: - Summary: Global Aggregation doesn't seem to work at all Key: SPARK-16474 URL: https://issues.apache.org/jira/browse/SPARK-16474 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-15810) Aggregator doesn't play nice with Option

[ https://issues.apache.org/jira/browse/SPARK-15810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369509#comment-15369509 ] Amit Sela edited comment on SPARK-15810 at 7/10/16 9:01 AM: Running the (sort

[jira] [Comment Edited] (SPARK-15810) Aggregator doesn't play nice with Option

[ https://issues.apache.org/jira/browse/SPARK-15810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369509#comment-15369509 ] Amit Sela edited comment on SPARK-15810 at 7/10/16 8:59 AM: Running the (sort

[jira] [Comment Edited] (SPARK-15810) Aggregator doesn't play nice with Option

[ https://issues.apache.org/jira/browse/SPARK-15810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369509#comment-15369509 ] Amit Sela edited comment on SPARK-15810 at 7/10/16 8:59 AM: Running the (sort

[jira] [Commented] (SPARK-15810) Aggregator doesn't play nice with Option

[ https://issues.apache.org/jira/browse/SPARK-15810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369509#comment-15369509 ] Amit Sela commented on SPARK-15810: --- Running the (sort of) same Java code: {code} SparkSession

[jira] [Comment Edited] (SPARK-15810) Aggregator doesn't play nice with Option

[ https://issues.apache.org/jira/browse/SPARK-15810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369496#comment-15369496 ] Amit Sela edited comment on SPARK-15810 at 7/10/16 8:28 AM: Just ran this

[jira] [Comment Edited] (SPARK-15810) Aggregator doesn't play nice with Option

[ https://issues.apache.org/jira/browse/SPARK-15810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369496#comment-15369496 ] Amit Sela edited comment on SPARK-15810 at 7/10/16 8:28 AM: Just ran this

[jira] [Commented] (SPARK-15810) Aggregator doesn't play nice with Option

[ https://issues.apache.org/jira/browse/SPARK-15810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369496#comment-15369496 ] Amit Sela commented on SPARK-15810: --- Just ran this exact code, prefixed by: {code} val session =

[jira] [Commented] (SPARK-15489) Dataset kryo encoder won't load custom user settings

[ https://issues.apache.org/jira/browse/SPARK-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305997#comment-15305997 ] Amit Sela commented on SPARK-15489: --- [~marmbrus] you can assign this one to me. > Dataset kryo encoder

[jira] [Updated] (SPARK-15489) Dataset kryo encoder won't load custom user settings

[ https://issues.apache.org/jira/browse/SPARK-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated SPARK-15489: -- Description: When setting a custom "spark.kryo.registrator" (or any other configuration for that

[jira] [Updated] (SPARK-15489) Dataset kryo encoder won't load custom user settings

[ https://issues.apache.org/jira/browse/SPARK-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated SPARK-15489: -- Description: When setting a custom "spark.kryo.registrator" (or any other configuration for that

[jira] [Updated] (SPARK-15489) Dataset kryo encoder won't load custom user settings

[ https://issues.apache.org/jira/browse/SPARK-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated SPARK-15489: -- Summary: Dataset kryo encoder won't load custom user settings (was: Dataset kryo encoder fails on

[jira] [Commented] (SPARK-15489) Dataset kryo encoder fails on Collections$UnmodifiableCollection

[ https://issues.apache.org/jira/browse/SPARK-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304893#comment-15304893 ] Amit Sela commented on SPARK-15489: --- The issue here is the fact that setting the SparkConf does not

[jira] [Commented] (SPARK-15489) Dataset kryo encoder fails on Collections$UnmodifiableCollection

[ https://issues.apache.org/jira/browse/SPARK-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304886#comment-15304886 ] Amit Sela commented on SPARK-15489: --- Got it! So I wasn't using the custom registrator correctly, it

[jira] [Commented] (SPARK-15489) Dataset kryo encoder fails on Collections$UnmodifiableCollection

[ https://issues.apache.org/jira/browse/SPARK-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304222#comment-15304222 ] Amit Sela commented on SPARK-15489: --- I would expect this to be related to KryoSerializer not

[jira] [Commented] (SPARK-15489) Dataset kryo encoder fails on Collections$UnmodifiableCollection

[ https://issues.apache.org/jira/browse/SPARK-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15298847#comment-15298847 ] Amit Sela commented on SPARK-15489: --- Well it looks like the codegen creates a ```java new

[jira] [Comment Edited] (SPARK-15489) Dataset kryo encoder fails on Collections$UnmodifiableCollection

[ https://issues.apache.org/jira/browse/SPARK-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15298847#comment-15298847 ] Amit Sela edited comment on SPARK-15489 at 5/24/16 8:38 PM: Well it looks

[jira] [Commented] (SPARK-15489) Dataset kryo encoder fails on Collections$UnmodifiableCollection

[ https://issues.apache.org/jira/browse/SPARK-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15297087#comment-15297087 ] Amit Sela commented on SPARK-15489: --- This is my registrator: public class ImmutableRegistrator

[jira] [Commented] (SPARK-15489) Dataset kryo encoder fails on Collections$UnmodifiableCollection

[ https://issues.apache.org/jira/browse/SPARK-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15296989#comment-15296989 ] Amit Sela commented on SPARK-15489: --- In 2.0.0-SNAPSHOT it manifests as: java.lang.AssertionError:

[jira] [Commented] (SPARK-15489) Dataset kryo encoder fails on Collections$UnmodifiableCollection

[ https://issues.apache.org/jira/browse/SPARK-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15296950#comment-15296950 ] Amit Sela commented on SPARK-15489: --- I've tried registering `UnmodifiableCollectionsSerializer` and

[jira] [Commented] (SPARK-15489) Dataset kryo encoder fails on Collections$UnmodifiableCollection

[ https://issues.apache.org/jira/browse/SPARK-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15296951#comment-15296951 ] Amit Sela commented on SPARK-15489: --- Serialization trace: tupleTags

[jira] [Created] (SPARK-15489) Dataset kryo encoder fails on Collections$UnmodifiableCollection

Amit Sela created SPARK-15489: - Summary: Dataset kryo encoder fails on Collections$UnmodifiableCollection Key: SPARK-15489 URL: https://issues.apache.org/jira/browse/SPARK-15489 Project: Spark