[jira] [Commented] (SPARK-8360) Streaming DataFrames
[ https://issues.apache.org/jira/browse/SPARK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15194296#comment-15194296 ] Reynold Xin commented on SPARK-8360: Note that there was an old streaming dataframe doc that was fairly incomplete. I moved it to SPARK-13875. > Streaming DataFrames > > > Key: SPARK-8360 > URL: https://issues.apache.org/jira/browse/SPARK-8360 > Project: Spark > Issue Type: Umbrella > Components: SQL, Streaming >Reporter: Reynold Xin > Attachments: > StructuredStreamingProgrammingAbstractionSemanticsandAPIs-ApacheJIRA.pdf > > > Umbrella ticket to track what's needed to make streaming DataFrame a reality. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8360) Streaming DataFrames
[ https://issues.apache.org/jira/browse/SPARK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15194265#comment-15194265 ] Reynold Xin commented on SPARK-8360: I've uploaded the first major design doc for this task -- covering api and semantics. This is not set in stone, and we'd love to get some feedback and iterate on the model as well as the explanation of it. The best way is to comment on it directly in Google Docs. > Streaming DataFrames > > > Key: SPARK-8360 > URL: https://issues.apache.org/jira/browse/SPARK-8360 > Project: Spark > Issue Type: Umbrella > Components: SQL, Streaming >Reporter: Reynold Xin > Attachments: StreamingDataFrameProposal.pdf, > StructuredStreamingProgrammingAbstractionSemanticsandAPIs-ApacheJIRA.pdf > > > Umbrella ticket to track what's needed to make streaming DataFrame a reality. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8360) Streaming DataFrames
[ https://issues.apache.org/jira/browse/SPARK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193596#comment-15193596 ] Praveen Devarao commented on SPARK-8360: Hi [~tdas] Thanks for the update. I would like to join hands with you guys contributing to the dev efforts of structured streaming...could you let me know how can I be of help? We (@ IBM) are looking at contributing Complex Event Processing (CEP) feature to spark streaming and have done some initial work on supporting the same in Spark 1.6now that we learn of structured streaming we want to ensure we get the CEP feature enabled on Spark 2.0 (as it makes more sense). For this we would like be part of the structured streaming efforts so that we get to understand it better and contributions will be inline with design. Let me know if you will need more information. We would be happy to have a call too for discussion on CEP (if you think that is better to start with). Thanks Praveen > Streaming DataFrames > > > Key: SPARK-8360 > URL: https://issues.apache.org/jira/browse/SPARK-8360 > Project: Spark > Issue Type: Umbrella > Components: SQL, Streaming >Reporter: Reynold Xin > Attachments: StreamingDataFrameProposal.pdf > > > Umbrella ticket to track what's needed to make streaming DataFrame a reality. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8360) Streaming DataFrames
[ https://issues.apache.org/jira/browse/SPARK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190414#comment-15190414 ] Tathagata Das commented on SPARK-8360: -- This is still highly WIP, and not ready for even experimental consumption. So please sit tight until we have something ready by Spark 2.0 release. > Streaming DataFrames > > > Key: SPARK-8360 > URL: https://issues.apache.org/jira/browse/SPARK-8360 > Project: Spark > Issue Type: Umbrella > Components: SQL, Streaming >Reporter: Reynold Xin > Attachments: StreamingDataFrameProposal.pdf > > > Umbrella ticket to track what's needed to make streaming DataFrame a reality. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8360) Streaming DataFrames
[ https://issues.apache.org/jira/browse/SPARK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15185168#comment-15185168 ] Praveen Devarao commented on SPARK-8360: Hi [~tdas],[~marmbrus],[~rxin] Any docs on how to consume the new APIs (if they are available)? Or any pointers in the current code which I can go through to play around with the new feature. I see some test cases related to using the streams on read() but don't find any pointers to which class can be used in .format(). The test suite is working out of the DefaultSource class defined within the DataFrameReaderWriterSuite but I suppose there is would be something consumable in the source. Thanks Praveen > Streaming DataFrames > > > Key: SPARK-8360 > URL: https://issues.apache.org/jira/browse/SPARK-8360 > Project: Spark > Issue Type: Umbrella > Components: SQL, Streaming >Reporter: Reynold Xin > Attachments: StreamingDataFrameProposal.pdf > > > Umbrella ticket to track what's needed to make streaming DataFrame a reality. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8360) Streaming DataFrames
[ https://issues.apache.org/jira/browse/SPARK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15158377#comment-15158377 ] Linbo commented on SPARK-8360: -- Spark 2.0 on April/May > Streaming DataFrames > > > Key: SPARK-8360 > URL: https://issues.apache.org/jira/browse/SPARK-8360 > Project: Spark > Issue Type: Umbrella > Components: SQL, Streaming >Reporter: Reynold Xin > Attachments: StreamingDataFrameProposal.pdf > > > Umbrella ticket to track what's needed to make streaming DataFrame a reality. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8360) Streaming DataFrames
[ https://issues.apache.org/jira/browse/SPARK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089035#comment-15089035 ] Ricky Yang commented on SPARK-8360: --- Which version will release this feature?I'd like to help test this feature in product > Streaming DataFrames > > > Key: SPARK-8360 > URL: https://issues.apache.org/jira/browse/SPARK-8360 > Project: Spark > Issue Type: Umbrella > Components: SQL, Streaming >Reporter: Reynold Xin > Attachments: StreamingDataFrameProposal.pdf > > > Umbrella ticket to track what's needed to make streaming DataFrame a reality. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8360) Streaming DataFrames
[ https://issues.apache.org/jira/browse/SPARK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035335#comment-15035335 ] Cheng Hao commented on SPARK-8360: -- Add some thoughts on StreamingSQL. https://docs.google.com/document/d/1ZgmDsNLT2XltMI317nNtkUkxBFwaaCkLHtPXjQE6JBY/edit > Streaming DataFrames > > > Key: SPARK-8360 > URL: https://issues.apache.org/jira/browse/SPARK-8360 > Project: Spark > Issue Type: Umbrella > Components: SQL, Streaming >Reporter: Reynold Xin > > Umbrella ticket to track what's needed to make streaming DataFrame a reality. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8360) Streaming DataFrames
[ https://issues.apache.org/jira/browse/SPARK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035379#comment-15035379 ] Xiao Li commented on SPARK-8360: "You need permission to access this published document." I got this message when accessing it. Could you make it publicly available? Thank you! > Streaming DataFrames > > > Key: SPARK-8360 > URL: https://issues.apache.org/jira/browse/SPARK-8360 > Project: Spark > Issue Type: Umbrella > Components: SQL, Streaming >Reporter: Reynold Xin > > Umbrella ticket to track what's needed to make streaming DataFrame a reality. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8360) Streaming DataFrames
[ https://issues.apache.org/jira/browse/SPARK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15015280#comment-15015280 ] Xiao Li commented on SPARK-8360: Streaming Dataframe will be built on Dataset APIs https://issues.apache.org/jira/browse/SPARK-? > Streaming DataFrames > > > Key: SPARK-8360 > URL: https://issues.apache.org/jira/browse/SPARK-8360 > Project: Spark > Issue Type: Umbrella > Components: SQL, Streaming >Reporter: Reynold Xin > > Umbrella ticket to track what's needed to make streaming DataFrame a reality. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8360) Streaming DataFrames
[ https://issues.apache.org/jira/browse/SPARK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14712377#comment-14712377 ] Adrian Wang commented on SPARK-8360: https://github.com/intel-bigdata/spark-streamingsql Our streaming sql project is highly related to this jira ticket. Streaming DataFrames Key: SPARK-8360 URL: https://issues.apache.org/jira/browse/SPARK-8360 Project: Spark Issue Type: Umbrella Components: SQL, Streaming Reporter: Reynold Xin Umbrella ticket to track what's needed to make streaming DataFrame a reality. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8360) Streaming DataFrames
[ https://issues.apache.org/jira/browse/SPARK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649729#comment-14649729 ] Joseph Batchik commented on SPARK-8360: --- Would streaming DataFrames replace streaming RDDs or coincide with it? Streaming DataFrames Key: SPARK-8360 URL: https://issues.apache.org/jira/browse/SPARK-8360 Project: Spark Issue Type: Umbrella Components: SQL, Streaming Reporter: Reynold Xin Umbrella ticket to track what's needed to make streaming DataFrame a reality. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
