[jira] [Commented] (SPARK-13027) Add API for updateStateByKey to provide batch time as input
[ https://issues.apache.org/jira/browse/SPARK-13027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149126#comment-15149126 ] Aaditya Ramesh commented on SPARK-13027: Hi [~zsxwing] sorry to bump this again, I've submitted a new patch. Could you take a look when you get a chance? > Add API for updateStateByKey to provide batch time as input > --- > > Key: SPARK-13027 > URL: https://issues.apache.org/jira/browse/SPARK-13027 > Project: Spark > Issue Type: Improvement > Components: Streaming >Affects Versions: 1.6.0 >Reporter: Aaditya Ramesh > > The StateDStream currently does not provide the batch time as input to the > state update function. This is required in cases where the behavior depends > on the batch start time. > We (Conviva) have been patching it manually for the past several Spark > versions but we thought it might be useful for others as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13027) Add API for updateStateByKey to provide batch time as input
[ https://issues.apache.org/jira/browse/SPARK-13027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15139898#comment-15139898 ] Aaditya Ramesh commented on SPARK-13027: [~zsxwing] I've submitted a pull request. Could you take a look when you get a chance? > Add API for updateStateByKey to provide batch time as input > --- > > Key: SPARK-13027 > URL: https://issues.apache.org/jira/browse/SPARK-13027 > Project: Spark > Issue Type: Improvement > Components: Streaming >Affects Versions: 1.6.0 >Reporter: Aaditya Ramesh > > The StateDStream currently does not provide the batch time as input to the > state update function. This is required in cases where the behavior depends > on the batch start time. > We (Conviva) have been patching it manually for the past several Spark > versions but we thought it might be useful for others as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13027) Add API for updateStateByKey to provide batch time as input
[ https://issues.apache.org/jira/browse/SPARK-13027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137858#comment-15137858 ] Apache Spark commented on SPARK-13027: -- User 'aramesh117' has created a pull request for this issue: https://github.com/apache/spark/pull/11122 > Add API for updateStateByKey to provide batch time as input > --- > > Key: SPARK-13027 > URL: https://issues.apache.org/jira/browse/SPARK-13027 > Project: Spark > Issue Type: Improvement > Components: Streaming >Affects Versions: 1.6.0 >Reporter: Aaditya Ramesh > > The StateDStream currently does not provide the batch time as input to the > state update function. This is required in cases where the behavior depends > on the batch start time. > We (Conviva) have been patching it manually for the past several Spark > versions but we thought it might be useful for others as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13027) Add API for updateStateByKey to provide batch time as input
[ https://issues.apache.org/jira/browse/SPARK-13027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133087#comment-15133087 ] Aaditya Ramesh commented on SPARK-13027: [~zsxwing] Sorry for the late response, definitely, I have the patch ready, I'm just reviewing it with coworkers before submitting the PR. [~mariobriggs] thank you for the offer. :) I actually already have the patch ready and it would be very helpful if you could review it. > Add API for updateStateByKey to provide batch time as input > --- > > Key: SPARK-13027 > URL: https://issues.apache.org/jira/browse/SPARK-13027 > Project: Spark > Issue Type: Improvement > Components: Streaming >Affects Versions: 1.6.0 >Reporter: Aaditya Ramesh > > The StateDStream currently does not provide the batch time as input to the > state update function. This is required in cases where the behavior depends > on the batch start time. > We (Conviva) have been patching it manually for the past several Spark > versions but we thought it might be useful for others as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13027) Add API for updateStateByKey to provide batch time as input
[ https://issues.apache.org/jira/browse/SPARK-13027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133643#comment-15133643 ] Mario Briggs commented on SPARK-13027: -- [~rameshaaditya117] sure, more than happy to review > Add API for updateStateByKey to provide batch time as input > --- > > Key: SPARK-13027 > URL: https://issues.apache.org/jira/browse/SPARK-13027 > Project: Spark > Issue Type: Improvement > Components: Streaming >Affects Versions: 1.6.0 >Reporter: Aaditya Ramesh > > The StateDStream currently does not provide the batch time as input to the > state update function. This is required in cases where the behavior depends > on the batch start time. > We (Conviva) have been patching it manually for the past several Spark > versions but we thought it might be useful for others as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13027) Add API for updateStateByKey to provide batch time as input
[ https://issues.apache.org/jira/browse/SPARK-13027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15132641#comment-15132641 ] Mario Briggs commented on SPARK-13027: -- [~rameshaaditya117] if you are tied up on something else, i could take a shot at it > Add API for updateStateByKey to provide batch time as input > --- > > Key: SPARK-13027 > URL: https://issues.apache.org/jira/browse/SPARK-13027 > Project: Spark > Issue Type: Improvement > Components: Streaming >Affects Versions: 1.6.0 >Reporter: Aaditya Ramesh > > The StateDStream currently does not provide the batch time as input to the > state update function. This is required in cases where the behavior depends > on the batch start time. > We (Conviva) have been patching it manually for the past several Spark > versions but we thought it might be useful for others as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13027) Add API for updateStateByKey to provide batch time as input
[ https://issues.apache.org/jira/browse/SPARK-13027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122169#comment-15122169 ] Shixiong Zhu commented on SPARK-13027: -- Could you submit a PR? > Add API for updateStateByKey to provide batch time as input > --- > > Key: SPARK-13027 > URL: https://issues.apache.org/jira/browse/SPARK-13027 > Project: Spark > Issue Type: Improvement > Components: Streaming >Affects Versions: 1.6.0 >Reporter: Aaditya Ramesh > > The StateDStream currently does not provide the batch time as input to the > state update function. This is required in cases where the behavior depends > on the batch start time. > We (Conviva) have been patching it manually for the past several Spark > versions but we thought it might be useful for others as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
