[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-21 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510787#comment-15510787 ] Michael Armbrust commented on SPARK-16407: -- I'm still a little unclear on the use cases we are

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-20 Thread Frederick Reiss (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508260#comment-15508260 ] Frederick Reiss commented on SPARK-16407: - It looks like there is a general agreement here that

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-19 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504301#comment-15504301 ] Michael Armbrust commented on SPARK-16407: -- You are taking an experimental interface, which

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-19 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504303#comment-15504303 ] Michael Armbrust commented on SPARK-16407: -- You are taking an experimental interface, which

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-17 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498394#comment-15498394 ] holdenk commented on SPARK-16407: - I'm not sure I understand - this really isn't exposing a new sink

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-17 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498389#comment-15498389 ] holdenk commented on SPARK-16407: - The current sink interface output depends on DataFrames - although

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495006#comment-15495006 ] Reynold Xin commented on SPARK-16407: - The source/sink interface currently depends on RDDs, doesn't

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-15 Thread Frederick Reiss (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15494795#comment-15494795 ] Frederick Reiss commented on SPARK-16407: - With respect, I'm not seeing a whole lot of flux in

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15494475#comment-15494475 ] Michael Armbrust commented on SPARK-16407: -- Sure, but the bar for compatibility is different for

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-15 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15493936#comment-15493936 ] holdenk commented on SPARK-16407: - I think its important to keep in mind that these APIs are already

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-14 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491819#comment-15491819 ] Michael Armbrust commented on SPARK-16407: -- I think it is likely that we will want to make it

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-14 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491787#comment-15491787 ] holdenk commented on SPARK-16407: - That's part of why I decided to just use the ForeachRDD sink as

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-14 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491784#comment-15491784 ] holdenk commented on SPARK-16407: - It's true it doesn't work in SQL - but I don't think the current

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-14 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491779#comment-15491779 ] Reynold Xin commented on SPARK-16407: - Actually I spoke too soon. I only read the code change and

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-14 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491772#comment-15491772 ] holdenk commented on SPARK-16407: - Right the simplest example where you need to use the typed API is with

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-14 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491752#comment-15491752 ] Reynold Xin commented on SPARK-16407: - This doesn't work in SQL, Python, etc. I like the general

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-14 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491739#comment-15491739 ] Shixiong Zhu commented on SPARK-16407: -- Right now we don't want to add such typed API to

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-08-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425243#comment-15425243 ] Apache Spark commented on SPARK-16407: -- User 'holdenk' has created a pull request for this issue: