Re: Modular IO presentation at Apachecon

[Alexey Romanenko]

Interesting talk, I’ve had a chance to see this in person. 
Many thanks to Ismael and Eugene!

> On 27 Sep 2018, at 21:34, Eugene Kirpichov  wrote:
> 
> Thanks Ismael and everyone else! Unfortunately I do not believe that this 
> session was recorded on video :(
> Juan - yes, this is some of the important future work, and I think it's not 
> hard to add to many connectors; contributions would be welcome.
> In terms of a "per-key" Wait transform, yeah, that definitely needs to be 
> figured out too. The presentation considers only the non-per-key case but I 
> think it should not be hard to add a per-key one. If you need to do something 
> directly with the results, you can use Combine.perKey().
> 
> On Thu, Sep 27, 2018 at 10:10 AM Pablo Estrada  > wrote:
> I'll take this chance to plug in my little directory of Beam tools/materials: 
> https://github.com/pabloem/awesome-beam 
> 
> 
> Please feel free to send PRs : )
> 
> 
> On Wed, Sep 26, 2018 at 10:29 PM Ankur Goenka  > wrote:
> Thanks for sharing. Great slides and looking for the recorded session.
> 
> Do we have a central location where we link all the beam presentations for 
> discoverability?
> 
> On Wed, Sep 26, 2018 at 9:35 PM Thomas Weise  > wrote:
> Thanks for sharing. I'm looking forward to see the recording of the talk 
> (hopefully!).
> 
> This will be very helpful for Beam users. IO still is typically the 
> unexpectedly hard and time consuming part of authoring pipelines.
> 
> 
> On Wed, Sep 26, 2018 at 2:48 PM Alan Myrvold  > wrote:
> Thanks for the slides.
> Really enjoyed the talk in person, especially the concept that IO is a 
> transformation, and a source or sink are not special and the splittable DoFn 
> explanation.
> 
> On Wed, Sep 26, 2018 at 2:17 PM Ismaël Mejía  > wrote:
> Hello, today Eugene and me did a talk about about modular APIs for IO
> at ApacheCon. This talk introduces some common patterns that we have
> found while creating IO connectors and also presents recent ideas like
> dynamic destinations, sequential writes among others using FileIO as a
> use case.
> 
> In case you guys want to take a look, here is a copy of the slides, we
> will probably add this to the IO authoring documentation too.
> 
> https://s.apache.org/beam-modular-io-talk 
> 

Re: Modular IO presentation at Apachecon

[Eugene Kirpichov]

Thanks Ismael and everyone else! Unfortunately I do not believe that this
session was recorded on video :(
Juan - yes, this is some of the important future work, and I think it's not
hard to add to many connectors; contributions would be welcome.
In terms of a "per-key" Wait transform, yeah, that definitely needs to be
figured out too. The presentation considers only the non-per-key case but I
think it should not be hard to add a per-key one. If you need to do
something directly with the results, you can use Combine.perKey().

On Thu, Sep 27, 2018 at 10:10 AM Pablo Estrada  wrote:

> I'll take this chance to plug in my little directory of Beam
> tools/materials: https://github.com/pabloem/awesome-beam
>
> Please feel free to send PRs : )
>
>
> On Wed, Sep 26, 2018 at 10:29 PM Ankur Goenka  wrote:
>
>> Thanks for sharing. Great slides and looking for the recorded session.
>>
>> Do we have a central location where we link all the beam presentations
>> for discoverability?
>>
>> On Wed, Sep 26, 2018 at 9:35 PM Thomas Weise  wrote:
>>
>>> Thanks for sharing. I'm looking forward to see the recording of the talk
>>> (hopefully!).
>>>
>>> This will be very helpful for Beam users. IO still is typically the
>>> unexpectedly hard and time consuming part of authoring pipelines.
>>>
>>>
>>> On Wed, Sep 26, 2018 at 2:48 PM Alan Myrvold 
>>> wrote:
>>>
 Thanks for the slides.
 Really enjoyed the talk in person, especially the concept that IO is a
 transformation, and a source or sink are not special and the splittable
 DoFn explanation.

 On Wed, Sep 26, 2018 at 2:17 PM Ismaël Mejía  wrote:

> Hello, today Eugene and me did a talk about about modular APIs for IO
> at ApacheCon. This talk introduces some common patterns that we have
> found while creating IO connectors and also presents recent ideas like
> dynamic destinations, sequential writes among others using FileIO as a
> use case.
>
> In case you guys want to take a look, here is a copy of the slides, we
> will probably add this to the IO authoring documentation too.
>
> https://s.apache.org/beam-modular-io-talk
>

Re: Modular IO presentation at Apachecon

[Pablo Estrada]

I'll take this chance to plug in my little directory of Beam
tools/materials: https://github.com/pabloem/awesome-beam

Please feel free to send PRs : )

On Wed, Sep 26, 2018 at 10:29 PM Ankur Goenka  wrote:

> Thanks for sharing. Great slides and looking for the recorded session.
>
> Do we have a central location where we link all the beam presentations for
> discoverability?
>
> On Wed, Sep 26, 2018 at 9:35 PM Thomas Weise  wrote:
>
>> Thanks for sharing. I'm looking forward to see the recording of the talk
>> (hopefully!).
>>
>> This will be very helpful for Beam users. IO still is typically the
>> unexpectedly hard and time consuming part of authoring pipelines.
>>
>>
>> On Wed, Sep 26, 2018 at 2:48 PM Alan Myrvold  wrote:
>>
>>> Thanks for the slides.
>>> Really enjoyed the talk in person, especially the concept that IO is a
>>> transformation, and a source or sink are not special and the splittable
>>> DoFn explanation.
>>>
>>> On Wed, Sep 26, 2018 at 2:17 PM Ismaël Mejía  wrote:
>>>
 Hello, today Eugene and me did a talk about about modular APIs for IO
 at ApacheCon. This talk introduces some common patterns that we have
 found while creating IO connectors and also presents recent ideas like
 dynamic destinations, sequential writes among others using FileIO as a
 use case.

 In case you guys want to take a look, here is a copy of the slides, we
 will probably add this to the IO authoring documentation too.

 https://s.apache.org/beam-modular-io-talk

>>>

Re: Modular IO presentation at Apachecon

[Chamikara Jayalath]

Thanks, it was a great talk. Modular and composable IO FTW!

On Thu, Sep 27, 2018 at 1:30 AM Juan Carlos Garcia 
wrote:

> Im really looking forward for a way to monitor the results(like which
> batch of elements were written per destination if possible 🙆 ) of an IO
> Module in a consistent way.
>
> Nice presentation.
>
> Thomas Weise  schrieb am Do., 27. Sep. 2018, 06:35:
>
>> Thanks for sharing. I'm looking forward to see the recording of the talk
>> (hopefully!).
>>
>> This will be very helpful for Beam users. IO still is typically the
>> unexpectedly hard and time consuming part of authoring pipelines.
>>
>>
>> On Wed, Sep 26, 2018 at 2:48 PM Alan Myrvold  wrote:
>>
>>> Thanks for the slides.
>>> Really enjoyed the talk in person, especially the concept that IO is a
>>> transformation, and a source or sink are not special and the splittable
>>> DoFn explanation.
>>>
>>> On Wed, Sep 26, 2018 at 2:17 PM Ismaël Mejía  wrote:
>>>
 Hello, today Eugene and me did a talk about about modular APIs for IO
 at ApacheCon. This talk introduces some common patterns that we have
 found while creating IO connectors and also presents recent ideas like
 dynamic destinations, sequential writes among others using FileIO as a
 use case.

 In case you guys want to take a look, here is a copy of the slides, we
 will probably add this to the IO authoring documentation too.

 https://s.apache.org/beam-modular-io-talk

>>>

Re: Modular IO presentation at Apachecon

[Juan Carlos Garcia]

Im really looking forward for a way to monitor the results(like which batch
of elements were written per destination if possible 🙆 ) of an IO Module
in a consistent way.

Nice presentation.

Thomas Weise  schrieb am Do., 27. Sep. 2018, 06:35:

> Thanks for sharing. I'm looking forward to see the recording of the talk
> (hopefully!).
>
> This will be very helpful for Beam users. IO still is typically the
> unexpectedly hard and time consuming part of authoring pipelines.
>
>
> On Wed, Sep 26, 2018 at 2:48 PM Alan Myrvold  wrote:
>
>> Thanks for the slides.
>> Really enjoyed the talk in person, especially the concept that IO is a
>> transformation, and a source or sink are not special and the splittable
>> DoFn explanation.
>>
>> On Wed, Sep 26, 2018 at 2:17 PM Ismaël Mejía  wrote:
>>
>>> Hello, today Eugene and me did a talk about about modular APIs for IO
>>> at ApacheCon. This talk introduces some common patterns that we have
>>> found while creating IO connectors and also presents recent ideas like
>>> dynamic destinations, sequential writes among others using FileIO as a
>>> use case.
>>>
>>> In case you guys want to take a look, here is a copy of the slides, we
>>> will probably add this to the IO authoring documentation too.
>>>
>>> https://s.apache.org/beam-modular-io-talk
>>>
>>

Re: Modular IO presentation at Apachecon

[Ankur Goenka]

Thanks for sharing. Great slides and looking for the recorded session.

Do we have a central location where we link all the beam presentations for
discoverability?

On Wed, Sep 26, 2018 at 9:35 PM Thomas Weise  wrote:

> Thanks for sharing. I'm looking forward to see the recording of the talk
> (hopefully!).
>
> This will be very helpful for Beam users. IO still is typically the
> unexpectedly hard and time consuming part of authoring pipelines.
>
>
> On Wed, Sep 26, 2018 at 2:48 PM Alan Myrvold  wrote:
>
>> Thanks for the slides.
>> Really enjoyed the talk in person, especially the concept that IO is a
>> transformation, and a source or sink are not special and the splittable
>> DoFn explanation.
>>
>> On Wed, Sep 26, 2018 at 2:17 PM Ismaël Mejía  wrote:
>>
>>> Hello, today Eugene and me did a talk about about modular APIs for IO
>>> at ApacheCon. This talk introduces some common patterns that we have
>>> found while creating IO connectors and also presents recent ideas like
>>> dynamic destinations, sequential writes among others using FileIO as a
>>> use case.
>>>
>>> In case you guys want to take a look, here is a copy of the slides, we
>>> will probably add this to the IO authoring documentation too.
>>>
>>> https://s.apache.org/beam-modular-io-talk
>>>
>>

Re: Modular IO presentation at Apachecon

[Thomas Weise]

Thanks for sharing. I'm looking forward to see the recording of the talk
(hopefully!).

This will be very helpful for Beam users. IO still is typically the
unexpectedly hard and time consuming part of authoring pipelines.


On Wed, Sep 26, 2018 at 2:48 PM Alan Myrvold  wrote:

> Thanks for the slides.
> Really enjoyed the talk in person, especially the concept that IO is a
> transformation, and a source or sink are not special and the splittable
> DoFn explanation.
>
> On Wed, Sep 26, 2018 at 2:17 PM Ismaël Mejía  wrote:
>
>> Hello, today Eugene and me did a talk about about modular APIs for IO
>> at ApacheCon. This talk introduces some common patterns that we have
>> found while creating IO connectors and also presents recent ideas like
>> dynamic destinations, sequential writes among others using FileIO as a
>> use case.
>>
>> In case you guys want to take a look, here is a copy of the slides, we
>> will probably add this to the IO authoring documentation too.
>>
>> https://s.apache.org/beam-modular-io-talk
>>
>

Re: Modular IO presentation at Apachecon

[Alan Myrvold]

Thanks for the slides.
Really enjoyed the talk in person, especially the concept that IO is a
transformation, and a source or sink are not special and the splittable
DoFn explanation.

On Wed, Sep 26, 2018 at 2:17 PM Ismaël Mejía  wrote:

> Hello, today Eugene and me did a talk about about modular APIs for IO
> at ApacheCon. This talk introduces some common patterns that we have
> found while creating IO connectors and also presents recent ideas like
> dynamic destinations, sequential writes among others using FileIO as a
> use case.
>
> In case you guys want to take a look, here is a copy of the slides, we
> will probably add this to the IO authoring documentation too.
>
> https://s.apache.org/beam-modular-io-talk
>

Modular IO presentation at Apachecon

[Ismaël Mejía]

Hello, today Eugene and me did a talk about about modular APIs for IO
at ApacheCon. This talk introduces some common patterns that we have
found while creating IO connectors and also presents recent ideas like
dynamic destinations, sequential writes among others using FileIO as a
use case.

In case you guys want to take a look, here is a copy of the slides, we
will probably add this to the IO authoring documentation too.

https://s.apache.org/beam-modular-io-talk