Hey, Stephan Ewen
1. bahir's target is spark. The contributer are rxin, srowen, tdas, mateiz
and so on.
If we want bahir used by flink, we can suggest bahir provide streaming
connecter interface, such as store(), start(), stop(), restart(),
receiving(Any)...
Then same streaming connector can be implemented by spark and flink.
But I think this is impossible, as bahir depend spark-streaming and spark sql.
2. About connector fault tolerance. Bahir's mqtt and akka connector are
themselves' storage. But netty have no persist data feature.
I think we can append data to a ringbuffer. When SourceContext
collect() throw error, then write message to inform client to stop send
message. When flink SourceContext is normal, then write the ringbuffer data to
flink, inform client to go on.
Because pushing mode is hard to control the flow throughput, the
upstream client can.
This netty connector's purpose is end-to-end streaming, minimum time
delay.
3. Later on, We can provide http protocol ability, put, post, get, even
websocket or jersey restful based on netty.
-----邮件原件-----
发件人: Stephan Ewen [mailto:se...@apache.org]
发送时间: 2016年9月27日 16:54
收件人: dev@flink.apache.org
主题: Re: [DISCUSS] add netty tcp/restful pushed source support
I think that could be an interesting source. Two quick questions to move forward
- To keep the Flink code base from becoming too big (hard to maintain and
test) we started working with Apache Bahir as a project dedicated to streaming
connectors. Would that be a good target for the connector?
- What are your thoughts on fault tolerance for that connector?
On Mon, Sep 26, 2016 at 3:01 PM, shijinkui <shijin...@huawei.com> wrote:
> Hi, all
>
> 1. In order to support end-to-end pushed source, I create FLINK-4630<
> https://issues.apache.org/jira/browse/FLINK-4630>. I want to know
> whether is this idea worth?
>
> -------------------
> When source stream get start, listen a provided tcp port, receive
> stream data from user data source.
> This netty tcp source is keepping alive and end-to-end, that is from
> business system to flink worker directly.
>
> user app push -----> netty server source of Flink
>
> describe the source in detail below:
>
> 1. source run as a netty tcp server
> 2. user provide a tcp port, if the port is in used, increace the port
> number between 1024 to 65535. Source can parallel.
> 3. callback the provided url to report the real port to listen
> 4. user push streaming data to netty server, then collect the data to
> flink
>
>
> Thanks
>
> Jinkui Shi
>
>
