+1 to add the source to Bahir That is an easier way to iterate fast on this and release quickly
On Wed, Oct 12, 2016 at 10:38 AM, Robert Metzger <rmetz...@apache.org> wrote: > Just a quick update on this one: The bahir community started already > discussing the first bahir-flink release. I expect it to happen soon. > > I would really like to see the netty source in Bahir. > > On Wed, Sep 28, 2016 at 3:18 PM, Stephan Ewen <se...@apache.org> wrote: > > > The Bahir-Flink stuff is fairly new - the first release has not yet > > happened. > > Robert is pushing that, will probably happen once he is back. > > > > On Tue, Sep 27, 2016 at 3:31 PM, shijinkui <shijin...@huawei.com> wrote: > > > > > It's nice. Will present flink source connector be pushed to > bahir-flink? > > > I can add netty-source to bahir-flink. > > > > > > Maven repository have no bahir-flink's. > > > https://mvnrepository.com/artifact/org.apache.bahir > > > > > > -----邮件原件----- > > > 发件人: Greg Hogan [mailto:c...@greghogan.com] > > > 发送时间: 2016年9月27日 20:58 > > > 收件人: dev@flink.apache.org > > > 主题: Re: 答复: [DISCUSS] add netty tcp/restful pushed source support > > > > > > Apache Bahir's website only suggests support for additional frameworks, > > > but there is a Flink repository at > > > https://github.com/apache/bahir-flink > > > > > > On Tue, Sep 27, 2016 at 8:38 AM, shijinkui <shijin...@huawei.com> > wrote: > > > > > > > Hey, Stephan Ewen > > > > > > > > 1. bahir's target is spark. The contributer are rxin, srowen, > > tdas, > > > > mateiz and so on. > > > > If we want bahir used by flink, we can suggest bahir provide > > > > streaming connecter interface, such as store(), start(), stop(), > > > > restart(), receiving(Any)... > > > > Then same streaming connector can be implemented by spark and > > > > flink. But I think this is impossible, as bahir depend > spark-streaming > > > > and spark sql. > > > > 2. About connector fault tolerance. Bahir's mqtt and akka > > connector > > > > are themselves' storage. But netty have no persist data feature. > > > > I think we can append data to a ringbuffer. When > SourceContext > > > > collect() throw error, then write message to inform client to stop > > > > send message. When flink SourceContext is normal, then write the > > > > ringbuffer data to flink, inform client to go on. > > > > Because pushing mode is hard to control the flow throughput, > > > > the upstream client can. > > > > This netty connector's purpose is end-to-end streaming, > > > > minimum time delay. > > > > 3. Later on, We can provide http protocol ability, put, post, > get, > > > > even websocket or jersey restful based on netty. > > > > > > > > -----邮件原件----- > > > > 发件人: Stephan Ewen [mailto:se...@apache.org] > > > > 发送时间: 2016年9月27日 16:54 > > > > 收件人: dev@flink.apache.org > > > > 主题: Re: [DISCUSS] add netty tcp/restful pushed source support > > > > > > > > I think that could be an interesting source. Two quick questions to > > > > move forward > > > > > > > > - To keep the Flink code base from becoming too big (hard to > > > > maintain and > > > > test) we started working with Apache Bahir as a project dedicated to > > > > streaming connectors. Would that be a good target for the connector? > > > > > > > > - What are your thoughts on fault tolerance for that connector? > > > > > > > > On Mon, Sep 26, 2016 at 3:01 PM, shijinkui <shijin...@huawei.com> > > wrote: > > > > > > > > > Hi, all > > > > > > > > > > 1. In order to support end-to-end pushed source, I create > > > FLINK-4630< > > > > > https://issues.apache.org/jira/browse/FLINK-4630>. I want to know > > > > > whether is this idea worth? > > > > > > > > > > ------------------- > > > > > When source stream get start, listen a provided tcp port, receive > > > > > stream data from user data source. > > > > > This netty tcp source is keepping alive and end-to-end, that is > from > > > > > business system to flink worker directly. > > > > > > > > > > user app push -----> netty server source of Flink > > > > > > > > > > describe the source in detail below: > > > > > > > > > > 1. source run as a netty tcp server > > > > > 2. user provide a tcp port, if the port is in used, increace the > > > port > > > > > number between 1024 to 65535. Source can parallel. > > > > > 3. callback the provided url to report the real port to listen > > > > > 4. user push streaming data to netty server, then collect the > data > > > to > > > > > flink > > > > > > > > > > > > > > > Thanks > > > > > > > > > > Jinkui Shi > > > > > > > > > > > > > > > > > > > >
