Re: Custom Unbounded Sources for 2.4.0 SDK

[Jean-Baptiste Onofré]

Ok, so, you should not use native source as it would break the portability.

Instead, just use KafkaIO !

Regards
JB

On 09/06/2018 22:12, Abdul Qadeer wrote:
> Alternate way while using FlinkKafkaConsumer* (:
> Will the Beam Flink runner code need several changes for this?
> 
> On Sat, 9 Jun 2018 at 12:42, Abdul Qadeer  > wrote:
> 
> I want to use 'FlinkKafkaConsumer' with 'UnboundedFlinkSource' as
> present in 0.6.0 SDK and it's examples (
> 
> https://github.com/dataArtisans/flink-dataflow/blob/master/examples/src/main/java/com/dataartisans/flink/dataflow/examples/streaming/KafkaWindowedWordCountExample.java
> ). 
> 
> As I understand you are saying this support is not there now,
> correct? How else could I use a 'FlinkKafkaConsumer' at source
> level? Is there an alternate you would suggest if not possible?
> 
> On Sat, 9 Jun 2018 at 07:23, Jean-Baptiste Onofré  > wrote:
> 
> By the away, if you mean that your custom source is implemented for
> Flink, it's not supported. I meant the Beam source.
> 
> On 09/06/2018 09:31, Abdul Qadeer wrote:
> > Hi!
> >
> > I would like to know if there is any way I can use 2.4.0
> Beam's Source
> > API for Flink 1.4.0 runner? I have a custom unbounded source
> implemented
> > for Flink runner but I can not find the documentation to use
> it for 2.x
> > Beam SDK. Looks like it was only supported in 0.x SDK? Any
> help appreciated.
> 
> -- 
> Jean-Baptiste Onofré
> jbono...@apache.org 
> http://blog.nanthrax.net
> Talend - http://www.talend.com
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Custom Unbounded Sources for 2.4.0 SDK

[Abdul Qadeer]

Alternate way while using FlinkKafkaConsumer* (:
Will the Beam Flink runner code need several changes for this?

On Sat, 9 Jun 2018 at 12:42, Abdul Qadeer  wrote:

> I want to use 'FlinkKafkaConsumer' with 'UnboundedFlinkSource' as present
> in 0.6.0 SDK and it's examples (
>
> https://github.com/dataArtisans/flink-dataflow/blob/master/examples/src/main/java/com/dataartisans/flink/dataflow/examples/streaming/KafkaWindowedWordCountExample.java
> ).
>
> As I understand you are saying this support is not there now, correct? How
> else could I use a 'FlinkKafkaConsumer' at source level? Is there an
> alternate you would suggest if not possible?
>
> On Sat, 9 Jun 2018 at 07:23, Jean-Baptiste Onofré  wrote:
>
>> By the away, if you mean that your custom source is implemented for
>> Flink, it's not supported. I meant the Beam source.
>>
>> On 09/06/2018 09:31, Abdul Qadeer wrote:
>> > Hi!
>> >
>> > I would like to know if there is any way I can use 2.4.0 Beam's Source
>> > API for Flink 1.4.0 runner? I have a custom unbounded source implemented
>> > for Flink runner but I can not find the documentation to use it for 2.x
>> > Beam SDK. Looks like it was only supported in 0.x SDK? Any help
>> appreciated.
>>
>> --
>> Jean-Baptiste Onofré
>> jbono...@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>

Re: Custom Unbounded Sources for 2.4.0 SDK

[Abdul Qadeer]

I want to use 'FlinkKafkaConsumer' with 'UnboundedFlinkSource' as present
in 0.6.0 SDK and it's examples (
https://github.com/dataArtisans/flink-dataflow/blob/master/examples/src/main/java/com/dataartisans/flink/dataflow/examples/streaming/KafkaWindowedWordCountExample.java
).

As I understand you are saying this support is not there now, correct? How
else could I use a 'FlinkKafkaConsumer' at source level? Is there an
alternate you would suggest if not possible?

On Sat, 9 Jun 2018 at 07:23, Jean-Baptiste Onofré  wrote:

> By the away, if you mean that your custom source is implemented for
> Flink, it's not supported. I meant the Beam source.
>
> On 09/06/2018 09:31, Abdul Qadeer wrote:
> > Hi!
> >
> > I would like to know if there is any way I can use 2.4.0 Beam's Source
> > API for Flink 1.4.0 runner? I have a custom unbounded source implemented
> > for Flink runner but I can not find the documentation to use it for 2.x
> > Beam SDK. Looks like it was only supported in 0.x SDK? Any help
> appreciated.
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: Custom Unbounded Sources for 2.4.0 SDK

[Jean-Baptiste Onofré]

By the away, if you mean that your custom source is implemented for
Flink, it's not supported. I meant the Beam source.

On 09/06/2018 09:31, Abdul Qadeer wrote:
> Hi!
> 
> I would like to know if there is any way I can use 2.4.0 Beam's Source
> API for Flink 1.4.0 runner? I have a custom unbounded source implemented
> for Flink runner but I can not find the documentation to use it for 2.x
> Beam SDK. Looks like it was only supported in 0.x SDK? Any help appreciated.

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Custom Unbounded Sources for 2.4.0 SDK

[Jean-Baptiste Onofré]

Hi,

Not sure I understand your question.

You can use any source with Flink runner, all in Beam 2.4.0
(IOs/SDK/runner).

You can see usage of different IOs (sources) with different runner in
the beam-samples:

https://github.com/jbonofre/beam-samples

Regards
JB

On 09/06/2018 09:31, Abdul Qadeer wrote:
> Hi!
> 
> I would like to know if there is any way I can use 2.4.0 Beam's Source
> API for Flink 1.4.0 runner? I have a custom unbounded source implemented
> for Flink runner but I can not find the documentation to use it for 2.x
> Beam SDK. Looks like it was only supported in 0.x SDK? Any help appreciated.

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Custom Unbounded Sources for 2.4.0 SDK

[Abdul Qadeer]

Hi!

I would like to know if there is any way I can use 2.4.0 Beam's Source API
for Flink 1.4.0 runner? I have a custom unbounded source implemented for
Flink runner but I can not find the documentation to use it for 2.x Beam
SDK. Looks like it was only supported in 0.x SDK? Any help appreciated.