Re: Contribution of Asgarde: Error Handling for Beam?

[Austin Bennett]

It sounds like there is sufficient support for adding error-handling/DLQ
into Beam; that will be a great boost for users.

I'll start to take a look at formalizing some sort of BIP / Design doc, and
once that takes shape, will surface on list [ am out of my normal routine,
traveling for work, so that might take some time ... so anyone sufficiently
interested, do not feel blocked by me ].

I'm wondering about even intermixing (a) and (b).  Ex: short-run we could
look to define an API surface in the Beam Core API that we are happy with,
which *could* rely on asgarde [ in github.com/apache/beam-asgarde ] in the
short run.



On Wed, Sep 13, 2023 at 5:48 PM Alexey Romanenko 
wrote:

> I agree with Cham on these two options.
>
> In the end, it would be great to have such functionality (error handling /
> DLQ) integrated into Beam core API, but it will require, for sure, some
> technical discussions and reviews before - so it will take more time.
>
> Though, to make it available for users soon as a part of Beam
> distribution, adding this as an extension looks very feasible for me.
>
> —
> Alexey
>
> On 12 Sep 2023, at 19:44, Chamikara Jayalath via dev 
> wrote:
>
> Thanks Mazlum, this sounds great. I think there are two ways we can
> proceed if we decide to integrate the Asgarde library into Beam.
>
> (1) Directly import the code into Beam without significant modifications
> and/or a review (though we may add tests).
>
> (2) Go through a design/code review to determine whether this is the best
> approach for implementing error handling / DLQ in Beam transforms or
> whether there are other alternatives/modifications to Asgarde we want to
> consider.
>
> If we do (1) I prefer adding Asgarde as a separate Gradle module in Beam.
> We can later integrate it into the core module after a design/code review.
>
> Thank,
> Cham
>
>
>
> On Tue, Sep 12, 2023 at 10:26 AM Mazlum TOSUN 
> wrote:
>
>> Hello Austin and everyone,
>>
>> I am open for discussion.
>>
>> My first intention with Asgarde was to help the Beam community, because
>> Dead Letter Queue is so important in Beam and all the data pipeline
>> frameworks.
>> When I worked with Beam on production with my customers, we needed to
>> catch errors with side outputs and dead letter queue.
>>
>> This library really helped us to keep a less verbose code while applying
>> all the error handling logic, that is error prone and verbose if it is
>> repeated.
>>
>> As Kennet said, my intention was to stay as close as possible to Beam,
>> with a Wrapper and a Failure Monad on top of a PCollection, to handle all
>> the code and complexity for try catch blocks and side output.
>>
>> For the governance, even if I am the creator of this library, the most
>> important isn't me but the community and to help the community.
>> If the best solution to help the community is including the library
>> directly on Beam, we can go in this direction, with of course your reviews
>> and recommendations.
>>
>> Then the library will belong to the community and we will continue to
>> improve it.
>>
>> For the decision about the best place, I will comply with the majority.
>>
>> Best regards,
>>
>> Mazlum
>>
>> On Mon, Sep 11, 2023 at 11:15 PM Austin Bennett 
>> wrote:
>>
>>> @Mazlum TOSUN  --  you and I have spoken a few
>>> times about this.  it'd be good for you to comment here on list, on any of
>>> your concerns with governance, and/or other thoughts.  Ex: if you think
>>> contributing asgarde directly is the thing [ or perhaps expressing any
>>> interest helping write/contribute the relevant functionality into beam ...
>>> it is possible that by adding the actual functionality into beam - like
>>> Kenn's mentioned 'other place' we could make asgarde as an separate add-on
>>> obsolete ].
>>>
>>>
>>>
>>> On Fri, Sep 8, 2023 at 8:55 AM Kenneth Knowles  wrote:
>>>
 For anyone who hasn't clicked over the Asgarde, my TL;DR description of
 it is that it adds the "failure monad" aka "andThen" style error/result
 handling on top of chaining of PCollections. So it is at a similar level of
 abstraction of our basic transforms and generally useful for chaining
 dead-letter side outputs. It is no more or less appropriate for the core
 SDK than, say, the Project/Filter/Join transforms, or Watch, etc. If we
 actually aspired to have a thin core with the accessories like that in
 another place, then it should go to that other place.

 Kenn

 On Fri, Sep 8, 2023 at 11:24 AM Daniel Collins via dev <
 dev@beam.apache.org> wrote:

> > until we *require* Asgard on a core transform, it shouldn't be in
> the main repo
>
> I don't think this is necessarily true if it solves end user use
> cases. If there is a specific transform that solves a specific use case, 
> we
> could include it in the transforms folder for end-users, even if it isn't
> utilized in the I/Os at present. Hence the suggestion to take the most
> promising 

Re: Contribution of Asgarde: Error Handling for Beam?

[Alexey Romanenko]

I agree with Cham on these two options. 

In the end, it would be great to have such functionality (error handling / DLQ) 
integrated into Beam core API, but it will require, for sure, some technical 
discussions and reviews before - so it will take more time. 

Though, to make it available for users soon as a part of Beam distribution, 
adding this as an extension looks very feasible for me.   

—
Alexey

> On 12 Sep 2023, at 19:44, Chamikara Jayalath via dev  
> wrote:
> 
> Thanks Mazlum, this sounds great. I think there are two ways we can proceed 
> if we decide to integrate the Asgarde library into Beam.
> 
> (1) Directly import the code into Beam without significant modifications 
> and/or a review (though we may add tests).
> 
> (2) Go through a design/code review to determine whether this is the best 
> approach for implementing error handling / DLQ in Beam transforms or whether 
> there are other alternatives/modifications to Asgarde we want to consider.
> 
> If we do (1) I prefer adding Asgarde as a separate Gradle module in Beam. We 
> can later integrate it into the core module after a design/code review.
> 
> Thank,
> Cham
> 
> 
> 
> On Tue, Sep 12, 2023 at 10:26 AM Mazlum TOSUN  > wrote:
>> Hello Austin and everyone,
>> 
>> I am open for discussion.
>> 
>> My first intention with Asgarde was to help the Beam community, because Dead 
>> Letter Queue is so important in Beam and all the data pipeline frameworks.
>> When I worked with Beam on production with my customers, we needed to catch 
>> errors with side outputs and dead letter queue.
>> 
>> This library really helped us to keep a less verbose code while applying all 
>> the error handling logic, that is error prone and verbose if it is repeated.
>> 
>> As Kennet said, my intention was to stay as close as possible to Beam, with 
>> a Wrapper and a Failure Monad on top of a PCollection, to handle all the 
>> code and complexity for try catch blocks and side output.
>> 
>> For the governance, even if I am the creator of this library, the most 
>> important isn't me but the community and to help the community.
>> If the best solution to help the community is including the library directly 
>> on Beam, we can go in this direction, with of course your reviews and 
>> recommendations.
>> 
>> Then the library will belong to the community and we will continue to 
>> improve it.
>> 
>> For the decision about the best place, I will comply with the majority.
>> 
>> Best regards,
>> 
>> Mazlum
>> 
>> On Mon, Sep 11, 2023 at 11:15 PM Austin Bennett > > wrote:
>>> @Mazlum TOSUN  --  you and I have spoken a 
>>> few times about this.  it'd be good for you to comment here on list, on any 
>>> of your concerns with governance, and/or other thoughts.  Ex: if you think 
>>> contributing asgarde directly is the thing [ or perhaps expressing any 
>>> interest helping write/contribute the relevant functionality into beam ... 
>>> it is possible that by adding the actual functionality into beam - like 
>>> Kenn's mentioned 'other place' we could make asgarde as an separate add-on 
>>> obsolete ].  
>>> 
>>> 
>>> 
>>> On Fri, Sep 8, 2023 at 8:55 AM Kenneth Knowles >> > wrote:
 For anyone who hasn't clicked over the Asgarde, my TL;DR description of it 
 is that it adds the "failure monad" aka "andThen" style error/result 
 handling on top of chaining of PCollections. So it is at a similar level 
 of abstraction of our basic transforms and generally useful for chaining 
 dead-letter side outputs. It is no more or less appropriate for the core 
 SDK than, say, the Project/Filter/Join transforms, or Watch, etc. If we 
 actually aspired to have a thin core with the accessories like that in 
 another place, then it should go to that other place.
 
 Kenn
 
 On Fri, Sep 8, 2023 at 11:24 AM Daniel Collins via dev 
 mailto:dev@beam.apache.org>> wrote:
> > until we *require* Asgard on a core transform, it shouldn't be in the 
> > main repo
> 
> I don't think this is necessarily true if it solves end user use cases. 
> If there is a specific transform that solves a specific use case, we 
> could include it in the transforms folder for end-users, even if it isn't 
> utilized in the I/Os at present. Hence the suggestion to take the most 
> promising transforms and propose adding them with documentation, apis and 
> rationale.
> 
> -Daniel
> 
> On Fri, Sep 8, 2023 at 11:20 AM Robert Burke  > wrote:
>> I would say until we *require* Asgard on a core transform, it shouldn't 
>> be in the main repo. 
>> 
>> Incorporating something before there's a need for it is premature 
>> abstraction. We can't do things because they *might* be useful. Let's 
>> see concrete places where they are useful, or we're already having 

Re: Contribution of Asgarde: Error Handling for Beam?

[Chamikara Jayalath via dev]

Thanks Mazlum, this sounds great. I think there are two ways we can proceed
if we decide to integrate the Asgarde library into Beam.

(1) Directly import the code into Beam without significant modifications
and/or a review (though we may add tests).

(2) Go through a design/code review to determine whether this is the best
approach for implementing error handling / DLQ in Beam transforms or
whether there are other alternatives/modifications to Asgarde we want to
consider.

If we do (1) I prefer adding Asgarde as a separate Gradle module in Beam.
We can later integrate it into the core module after a design/code review.

Thank,
Cham



On Tue, Sep 12, 2023 at 10:26 AM Mazlum TOSUN 
wrote:

> Hello Austin and everyone,
>
> I am open for discussion.
>
> My first intention with Asgarde was to help the Beam community, because
> Dead Letter Queue is so important in Beam and all the data pipeline
> frameworks.
> When I worked with Beam on production with my customers, we needed to
> catch errors with side outputs and dead letter queue.
>
> This library really helped us to keep a less verbose code while applying
> all the error handling logic, that is error prone and verbose if it is
> repeated.
>
> As Kennet said, my intention was to stay as close as possible to Beam,
> with a Wrapper and a Failure Monad on top of a PCollection, to handle all
> the code and complexity for try catch blocks and side output.
>
> For the governance, even if I am the creator of this library, the most
> important isn't me but the community and to help the community.
> If the best solution to help the community is including the library
> directly on Beam, we can go in this direction, with of course your reviews
> and recommendations.
>
> Then the library will belong to the community and we will continue to
> improve it.
>
> For the decision about the best place, I will comply with the majority.
>
> Best regards,
>
> Mazlum
>
> On Mon, Sep 11, 2023 at 11:15 PM Austin Bennett  wrote:
>
>> @Mazlum TOSUN  --  you and I have spoken a few
>> times about this.  it'd be good for you to comment here on list, on any of
>> your concerns with governance, and/or other thoughts.  Ex: if you think
>> contributing asgarde directly is the thing [ or perhaps expressing any
>> interest helping write/contribute the relevant functionality into beam ...
>> it is possible that by adding the actual functionality into beam - like
>> Kenn's mentioned 'other place' we could make asgarde as an separate add-on
>> obsolete ].
>>
>>
>>
>> On Fri, Sep 8, 2023 at 8:55 AM Kenneth Knowles  wrote:
>>
>>> For anyone who hasn't clicked over the Asgarde, my TL;DR description of
>>> it is that it adds the "failure monad" aka "andThen" style error/result
>>> handling on top of chaining of PCollections. So it is at a similar level of
>>> abstraction of our basic transforms and generally useful for chaining
>>> dead-letter side outputs. It is no more or less appropriate for the core
>>> SDK than, say, the Project/Filter/Join transforms, or Watch, etc. If we
>>> actually aspired to have a thin core with the accessories like that in
>>> another place, then it should go to that other place.
>>>
>>> Kenn
>>>
>>> On Fri, Sep 8, 2023 at 11:24 AM Daniel Collins via dev <
>>> dev@beam.apache.org> wrote:
>>>
 > until we *require* Asgard on a core transform, it shouldn't be in the
 main repo

 I don't think this is necessarily true if it solves end user use cases.
 If there is a specific transform that solves a specific use case, we could
 include it in the transforms folder for end-users, even if it isn't
 utilized in the I/Os at present. Hence the suggestion to take the most
 promising transforms and propose adding them with documentation, apis and
 rationale.

 -Daniel

 On Fri, Sep 8, 2023 at 11:20 AM Robert Burke 
 wrote:

> I would say until we *require* Asgard on a core transform, it
> shouldn't be in the main repo.
>
> Incorporating something before there's a need for it is premature
> abstraction. We can't do things because they *might* be useful. Let's see
> concrete places where they are useful, or we're already having a similar
> need solved a different way.
>
> Beam is complicated by itself, and we do encourage multiple ways of
> solving problems, but that says to me that having an out of repo ecosystem
> is the right path, rather than incorporation.
>
> On Fri, Sep 8, 2023, 8:14 AM Daniel Collins via dev <
> dev@beam.apache.org> wrote:
>
>> I think there are a lot of interesting and relatively isolated
>> components of the project, it might make sense to write per-transform one
>> pagers for isolated things like the most useful pieces (just basically
>> copying the documentation and justifying the API) instead of doing a
>> one-shot import or having it live forever in an external project.
>>
>> -Daniel
>>
>> On Fri, Sep 

Re: Contribution of Asgarde: Error Handling for Beam?

[Mazlum TOSUN]

Hello Austin and everyone,

I am open for discussion.

My first intention with Asgarde was to help the Beam community, because
Dead Letter Queue is so important in Beam and all the data pipeline
frameworks.
When I worked with Beam on production with my customers, we needed to catch
errors with side outputs and dead letter queue.

This library really helped us to keep a less verbose code while applying
all the error handling logic, that is error prone and verbose if it is
repeated.

As Kennet said, my intention was to stay as close as possible to Beam, with
a Wrapper and a Failure Monad on top of a PCollection, to handle all the
code and complexity for try catch blocks and side output.

For the governance, even if I am the creator of this library, the most
important isn't me but the community and to help the community.
If the best solution to help the community is including the library
directly on Beam, we can go in this direction, with of course your reviews
and recommendations.

Then the library will belong to the community and we will continue to
improve it.

For the decision about the best place, I will comply with the majority.

Best regards,

Mazlum

On Mon, Sep 11, 2023 at 11:15 PM Austin Bennett  wrote:

> @Mazlum TOSUN  --  you and I have spoken a few
> times about this.  it'd be good for you to comment here on list, on any of
> your concerns with governance, and/or other thoughts.  Ex: if you think
> contributing asgarde directly is the thing [ or perhaps expressing any
> interest helping write/contribute the relevant functionality into beam ...
> it is possible that by adding the actual functionality into beam - like
> Kenn's mentioned 'other place' we could make asgarde as an separate add-on
> obsolete ].
>
>
>
> On Fri, Sep 8, 2023 at 8:55 AM Kenneth Knowles  wrote:
>
>> For anyone who hasn't clicked over the Asgarde, my TL;DR description of
>> it is that it adds the "failure monad" aka "andThen" style error/result
>> handling on top of chaining of PCollections. So it is at a similar level of
>> abstraction of our basic transforms and generally useful for chaining
>> dead-letter side outputs. It is no more or less appropriate for the core
>> SDK than, say, the Project/Filter/Join transforms, or Watch, etc. If we
>> actually aspired to have a thin core with the accessories like that in
>> another place, then it should go to that other place.
>>
>> Kenn
>>
>> On Fri, Sep 8, 2023 at 11:24 AM Daniel Collins via dev <
>> dev@beam.apache.org> wrote:
>>
>>> > until we *require* Asgard on a core transform, it shouldn't be in the
>>> main repo
>>>
>>> I don't think this is necessarily true if it solves end user use cases.
>>> If there is a specific transform that solves a specific use case, we could
>>> include it in the transforms folder for end-users, even if it isn't
>>> utilized in the I/Os at present. Hence the suggestion to take the most
>>> promising transforms and propose adding them with documentation, apis and
>>> rationale.
>>>
>>> -Daniel
>>>
>>> On Fri, Sep 8, 2023 at 11:20 AM Robert Burke  wrote:
>>>
 I would say until we *require* Asgard on a core transform, it shouldn't
 be in the main repo.

 Incorporating something before there's a need for it is premature
 abstraction. We can't do things because they *might* be useful. Let's see
 concrete places where they are useful, or we're already having a similar
 need solved a different way.

 Beam is complicated by itself, and we do encourage multiple ways of
 solving problems, but that says to me that having an out of repo ecosystem
 is the right path, rather than incorporation.

 On Fri, Sep 8, 2023, 8:14 AM Daniel Collins via dev <
 dev@beam.apache.org> wrote:

> I think there are a lot of interesting and relatively isolated
> components of the project, it might make sense to write per-transform one
> pagers for isolated things like the most useful pieces (just basically
> copying the documentation and justifying the API) instead of doing a
> one-shot import or having it live forever in an external project.
>
> -Daniel
>
> On Fri, Sep 8, 2023 at 11:10 AM Kenneth Knowles 
> wrote:
>
>> I agree with everyone about "not everything has to be in the Beam
>> repo". I really like the idea of having a clearer "ecosystem" section of
>> the website, which is sort of started at
>> https://beam.apache.org/community/integrations/ but that is not very
>> prominent.
>>
>> Agree with John though. The transforms in Asgarde could potentially
>> be used in Beam. Potentially best accomplished by just adding them as
>> transforms to the core Java SDK?
>>
>> Kenn
>>
>> On Wed, Sep 6, 2023 at 1:46 PM John Casey via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Agreed on documentation and on keeping it in a separate repo.
>>>
>>> We have a few pretty significant beam extensions (scio and 

Re: Contribution of Asgarde: Error Handling for Beam?

[Austin Bennett]

@Mazlum TOSUN  --  you and I have spoken a few
times about this.  it'd be good for you to comment here on list, on any of
your concerns with governance, and/or other thoughts.  Ex: if you think
contributing asgarde directly is the thing [ or perhaps expressing any
interest helping write/contribute the relevant functionality into beam ...
it is possible that by adding the actual functionality into beam - like
Kenn's mentioned 'other place' we could make asgarde as an separate add-on
obsolete ].



On Fri, Sep 8, 2023 at 8:55 AM Kenneth Knowles  wrote:

> For anyone who hasn't clicked over the Asgarde, my TL;DR description of it
> is that it adds the "failure monad" aka "andThen" style error/result
> handling on top of chaining of PCollections. So it is at a similar level of
> abstraction of our basic transforms and generally useful for chaining
> dead-letter side outputs. It is no more or less appropriate for the core
> SDK than, say, the Project/Filter/Join transforms, or Watch, etc. If we
> actually aspired to have a thin core with the accessories like that in
> another place, then it should go to that other place.
>
> Kenn
>
> On Fri, Sep 8, 2023 at 11:24 AM Daniel Collins via dev <
> dev@beam.apache.org> wrote:
>
>> > until we *require* Asgard on a core transform, it shouldn't be in the
>> main repo
>>
>> I don't think this is necessarily true if it solves end user use cases.
>> If there is a specific transform that solves a specific use case, we could
>> include it in the transforms folder for end-users, even if it isn't
>> utilized in the I/Os at present. Hence the suggestion to take the most
>> promising transforms and propose adding them with documentation, apis and
>> rationale.
>>
>> -Daniel
>>
>> On Fri, Sep 8, 2023 at 11:20 AM Robert Burke  wrote:
>>
>>> I would say until we *require* Asgard on a core transform, it shouldn't
>>> be in the main repo.
>>>
>>> Incorporating something before there's a need for it is premature
>>> abstraction. We can't do things because they *might* be useful. Let's see
>>> concrete places where they are useful, or we're already having a similar
>>> need solved a different way.
>>>
>>> Beam is complicated by itself, and we do encourage multiple ways of
>>> solving problems, but that says to me that having an out of repo ecosystem
>>> is the right path, rather than incorporation.
>>>
>>> On Fri, Sep 8, 2023, 8:14 AM Daniel Collins via dev 
>>> wrote:
>>>
 I think there are a lot of interesting and relatively isolated
 components of the project, it might make sense to write per-transform one
 pagers for isolated things like the most useful pieces (just basically
 copying the documentation and justifying the API) instead of doing a
 one-shot import or having it live forever in an external project.

 -Daniel

 On Fri, Sep 8, 2023 at 11:10 AM Kenneth Knowles 
 wrote:

> I agree with everyone about "not everything has to be in the Beam
> repo". I really like the idea of having a clearer "ecosystem" section of
> the website, which is sort of started at
> https://beam.apache.org/community/integrations/ but that is not very
> prominent.
>
> Agree with John though. The transforms in Asgarde could potentially be
> used in Beam. Potentially best accomplished by just adding them as
> transforms to the core Java SDK?
>
> Kenn
>
> On Wed, Sep 6, 2023 at 1:46 PM John Casey via dev 
> wrote:
>
>> Agreed on documentation and on keeping it in a separate repo.
>>
>> We have a few pretty significant beam extensions (scio and Dataflow
>> Templates also come to mind) that Beam should highlight, but are separate
>> repos for their own governance, contributions, and release reasons.
>>
>> The difference with Asgarde is that we might want to use it in Beam
>> itself, which makes it more reasonable to include in the main repo.
>>
>> On Tue, Sep 5, 2023 at 8:36 PM Robert Bradshaw via dev <
>> dev@beam.apache.org> wrote:
>>
>>> I think this is a great library. I'm on the fence of whether it
>>> makes sense to include with Beam proper vs. be a library that builds on 
>>> top
>>> of Beam. (Would there be benefits of tighter integration? There is the
>>> maintenance/loss of governance issue.) I am definitely not on the side 
>>> that
>>> the entire Beam ecosystem needs to be distributed/maintained by Beam
>>> itself.
>>>
>>> Regardless of the direction we go, I think it could make a lot of
>>> sense to put pointers to it in our documentation.
>>>
>>>
>>> On Tue, Sep 5, 2023 at 7:21 AM Danny McCormick via dev <
>>> dev@beam.apache.org> wrote:
>>>
 I think my only concerns here are around the toil we'll be taking
 on, and will we be leaving the asgarde project in a better or worse 
 place.

 From a release standpoint, we would need to release it 

Re: Contribution of Asgarde: Error Handling for Beam?

[Kenneth Knowles]

For anyone who hasn't clicked over the Asgarde, my TL;DR description of it
is that it adds the "failure monad" aka "andThen" style error/result
handling on top of chaining of PCollections. So it is at a similar level of
abstraction of our basic transforms and generally useful for chaining
dead-letter side outputs. It is no more or less appropriate for the core
SDK than, say, the Project/Filter/Join transforms, or Watch, etc. If we
actually aspired to have a thin core with the accessories like that in
another place, then it should go to that other place.

Kenn

On Fri, Sep 8, 2023 at 11:24 AM Daniel Collins via dev 
wrote:

> > until we *require* Asgard on a core transform, it shouldn't be in the
> main repo
>
> I don't think this is necessarily true if it solves end user use cases. If
> there is a specific transform that solves a specific use case, we could
> include it in the transforms folder for end-users, even if it isn't
> utilized in the I/Os at present. Hence the suggestion to take the most
> promising transforms and propose adding them with documentation, apis and
> rationale.
>
> -Daniel
>
> On Fri, Sep 8, 2023 at 11:20 AM Robert Burke  wrote:
>
>> I would say until we *require* Asgard on a core transform, it shouldn't
>> be in the main repo.
>>
>> Incorporating something before there's a need for it is premature
>> abstraction. We can't do things because they *might* be useful. Let's see
>> concrete places where they are useful, or we're already having a similar
>> need solved a different way.
>>
>> Beam is complicated by itself, and we do encourage multiple ways of
>> solving problems, but that says to me that having an out of repo ecosystem
>> is the right path, rather than incorporation.
>>
>> On Fri, Sep 8, 2023, 8:14 AM Daniel Collins via dev 
>> wrote:
>>
>>> I think there are a lot of interesting and relatively isolated
>>> components of the project, it might make sense to write per-transform one
>>> pagers for isolated things like the most useful pieces (just basically
>>> copying the documentation and justifying the API) instead of doing a
>>> one-shot import or having it live forever in an external project.
>>>
>>> -Daniel
>>>
>>> On Fri, Sep 8, 2023 at 11:10 AM Kenneth Knowles  wrote:
>>>
 I agree with everyone about "not everything has to be in the Beam
 repo". I really like the idea of having a clearer "ecosystem" section of
 the website, which is sort of started at
 https://beam.apache.org/community/integrations/ but that is not very
 prominent.

 Agree with John though. The transforms in Asgarde could potentially be
 used in Beam. Potentially best accomplished by just adding them as
 transforms to the core Java SDK?

 Kenn

 On Wed, Sep 6, 2023 at 1:46 PM John Casey via dev 
 wrote:

> Agreed on documentation and on keeping it in a separate repo.
>
> We have a few pretty significant beam extensions (scio and Dataflow
> Templates also come to mind) that Beam should highlight, but are separate
> repos for their own governance, contributions, and release reasons.
>
> The difference with Asgarde is that we might want to use it in Beam
> itself, which makes it more reasonable to include in the main repo.
>
> On Tue, Sep 5, 2023 at 8:36 PM Robert Bradshaw via dev <
> dev@beam.apache.org> wrote:
>
>> I think this is a great library. I'm on the fence of whether it makes
>> sense to include with Beam proper vs. be a library that builds on top of
>> Beam. (Would there be benefits of tighter integration? There is the
>> maintenance/loss of governance issue.) I am definitely not on the side 
>> that
>> the entire Beam ecosystem needs to be distributed/maintained by Beam
>> itself.
>>
>> Regardless of the direction we go, I think it could make a lot of
>> sense to put pointers to it in our documentation.
>>
>>
>> On Tue, Sep 5, 2023 at 7:21 AM Danny McCormick via dev <
>> dev@beam.apache.org> wrote:
>>
>>> I think my only concerns here are around the toil we'll be taking
>>> on, and will we be leaving the asgarde project in a better or worse 
>>> place.
>>>
>>> From a release standpoint, we would need to release it with the same
>>> cadence as Beam. Adding asgarde into our standard release process seems
>>> fairly straightforward, though, so I'm not too worried about it - looks
>>> like it's basically (1) add a commit like this
>>> ,
>>> (2) run this workflow
>>> ,
>>> and (3) tag/mark the release as released on GitHub.
>>>
>>> In terms of bug fixes and improvements, though, I'm a little worried
>>> that we might be leaving things in a worse state since Mazlum has been 
>>> the
>>> only 

Re: Contribution of Asgarde: Error Handling for Beam?

[Daniel Collins via dev]

> until we *require* Asgard on a core transform, it shouldn't be in the
main repo

I don't think this is necessarily true if it solves end user use cases. If
there is a specific transform that solves a specific use case, we could
include it in the transforms folder for end-users, even if it isn't
utilized in the I/Os at present. Hence the suggestion to take the most
promising transforms and propose adding them with documentation, apis and
rationale.

-Daniel

On Fri, Sep 8, 2023 at 11:20 AM Robert Burke  wrote:

> I would say until we *require* Asgard on a core transform, it shouldn't be
> in the main repo.
>
> Incorporating something before there's a need for it is premature
> abstraction. We can't do things because they *might* be useful. Let's see
> concrete places where they are useful, or we're already having a similar
> need solved a different way.
>
> Beam is complicated by itself, and we do encourage multiple ways of
> solving problems, but that says to me that having an out of repo ecosystem
> is the right path, rather than incorporation.
>
> On Fri, Sep 8, 2023, 8:14 AM Daniel Collins via dev 
> wrote:
>
>> I think there are a lot of interesting and relatively isolated components
>> of the project, it might make sense to write per-transform one pagers for
>> isolated things like the most useful pieces (just basically copying the
>> documentation and justifying the API) instead of doing a one-shot import or
>> having it live forever in an external project.
>>
>> -Daniel
>>
>> On Fri, Sep 8, 2023 at 11:10 AM Kenneth Knowles  wrote:
>>
>>> I agree with everyone about "not everything has to be in the Beam repo".
>>> I really like the idea of having a clearer "ecosystem" section of the
>>> website, which is sort of started at
>>> https://beam.apache.org/community/integrations/ but that is not very
>>> prominent.
>>>
>>> Agree with John though. The transforms in Asgarde could potentially be
>>> used in Beam. Potentially best accomplished by just adding them as
>>> transforms to the core Java SDK?
>>>
>>> Kenn
>>>
>>> On Wed, Sep 6, 2023 at 1:46 PM John Casey via dev 
>>> wrote:
>>>
 Agreed on documentation and on keeping it in a separate repo.

 We have a few pretty significant beam extensions (scio and Dataflow
 Templates also come to mind) that Beam should highlight, but are separate
 repos for their own governance, contributions, and release reasons.

 The difference with Asgarde is that we might want to use it in Beam
 itself, which makes it more reasonable to include in the main repo.

 On Tue, Sep 5, 2023 at 8:36 PM Robert Bradshaw via dev <
 dev@beam.apache.org> wrote:

> I think this is a great library. I'm on the fence of whether it makes
> sense to include with Beam proper vs. be a library that builds on top of
> Beam. (Would there be benefits of tighter integration? There is the
> maintenance/loss of governance issue.) I am definitely not on the side 
> that
> the entire Beam ecosystem needs to be distributed/maintained by Beam
> itself.
>
> Regardless of the direction we go, I think it could make a lot of
> sense to put pointers to it in our documentation.
>
>
> On Tue, Sep 5, 2023 at 7:21 AM Danny McCormick via dev <
> dev@beam.apache.org> wrote:
>
>> I think my only concerns here are around the toil we'll be taking on,
>> and will we be leaving the asgarde project in a better or worse place.
>>
>> From a release standpoint, we would need to release it with the same
>> cadence as Beam. Adding asgarde into our standard release process seems
>> fairly straightforward, though, so I'm not too worried about it - looks
>> like it's basically (1) add a commit like this
>> ,
>> (2) run this workflow
>> ,
>> and (3) tag/mark the release as released on GitHub.
>>
>> In terms of bug fixes and improvements, though, I'm a little worried
>> that we might be leaving things in a worse state since Mazlum has been 
>> the
>> only contributor thus far, and he would lose some governance (and 
>> possibly
>> the ability to commit code on his own). An extra motivated community 
>> member
>> or two could change the math a bit, but I'm not sure if there are 
>> actually
>> clear advantages to including it in Apache other than visibility. Would
>> adding links to our docs calling Asgarde out as an option accomplish the
>> same purpose?
>>
>> > Let's be careful about whether these tests are included in our
>> presubmits. Contrib code with flaky tests has been a major pain point in
>> the past.
>>
>> +1 - I think if we do this I'd vote that it be in a separate repo (
>> github.com/apache/beam-asgarde made sense to me).

Re: Contribution of Asgarde: Error Handling for Beam?

[Robert Burke]

I would say until we *require* Asgard on a core transform, it shouldn't be
in the main repo.

Incorporating something before there's a need for it is premature
abstraction. We can't do things because they *might* be useful. Let's see
concrete places where they are useful, or we're already having a similar
need solved a different way.

Beam is complicated by itself, and we do encourage multiple ways of solving
problems, but that says to me that having an out of repo ecosystem is the
right path, rather than incorporation.

On Fri, Sep 8, 2023, 8:14 AM Daniel Collins via dev 
wrote:

> I think there are a lot of interesting and relatively isolated components
> of the project, it might make sense to write per-transform one pagers for
> isolated things like the most useful pieces (just basically copying the
> documentation and justifying the API) instead of doing a one-shot import or
> having it live forever in an external project.
>
> -Daniel
>
> On Fri, Sep 8, 2023 at 11:10 AM Kenneth Knowles  wrote:
>
>> I agree with everyone about "not everything has to be in the Beam repo".
>> I really like the idea of having a clearer "ecosystem" section of the
>> website, which is sort of started at
>> https://beam.apache.org/community/integrations/ but that is not very
>> prominent.
>>
>> Agree with John though. The transforms in Asgarde could potentially be
>> used in Beam. Potentially best accomplished by just adding them as
>> transforms to the core Java SDK?
>>
>> Kenn
>>
>> On Wed, Sep 6, 2023 at 1:46 PM John Casey via dev 
>> wrote:
>>
>>> Agreed on documentation and on keeping it in a separate repo.
>>>
>>> We have a few pretty significant beam extensions (scio and Dataflow
>>> Templates also come to mind) that Beam should highlight, but are separate
>>> repos for their own governance, contributions, and release reasons.
>>>
>>> The difference with Asgarde is that we might want to use it in Beam
>>> itself, which makes it more reasonable to include in the main repo.
>>>
>>> On Tue, Sep 5, 2023 at 8:36 PM Robert Bradshaw via dev <
>>> dev@beam.apache.org> wrote:
>>>
 I think this is a great library. I'm on the fence of whether it makes
 sense to include with Beam proper vs. be a library that builds on top of
 Beam. (Would there be benefits of tighter integration? There is the
 maintenance/loss of governance issue.) I am definitely not on the side that
 the entire Beam ecosystem needs to be distributed/maintained by Beam
 itself.

 Regardless of the direction we go, I think it could make a lot of sense
 to put pointers to it in our documentation.


 On Tue, Sep 5, 2023 at 7:21 AM Danny McCormick via dev <
 dev@beam.apache.org> wrote:

> I think my only concerns here are around the toil we'll be taking on,
> and will we be leaving the asgarde project in a better or worse place.
>
> From a release standpoint, we would need to release it with the same
> cadence as Beam. Adding asgarde into our standard release process seems
> fairly straightforward, though, so I'm not too worried about it - looks
> like it's basically (1) add a commit like this
> ,
> (2) run this workflow
> ,
> and (3) tag/mark the release as released on GitHub.
>
> In terms of bug fixes and improvements, though, I'm a little worried
> that we might be leaving things in a worse state since Mazlum has been the
> only contributor thus far, and he would lose some governance (and possibly
> the ability to commit code on his own). An extra motivated community 
> member
> or two could change the math a bit, but I'm not sure if there are actually
> clear advantages to including it in Apache other than visibility. Would
> adding links to our docs calling Asgarde out as an option accomplish the
> same purpose?
>
> > Let's be careful about whether these tests are included in our
> presubmits. Contrib code with flaky tests has been a major pain point in
> the past.
>
> +1 - I think if we do this I'd vote that it be in a separate repo (
> github.com/apache/beam-asgarde made sense to me).
>
> ---
>
> Overall, I'm probably a slight -1 to adding this to the Apache
> workspace, but +1 to at least adding links from the Beam docs to Asgarde.
>
> Thanks,
> Danny
>
>
>
> On Tue, Sep 5, 2023 at 12:03 AM Reuven Lax via dev <
> dev@beam.apache.org> wrote:
>
>> Let's be careful about whether these tests are included in our
>> presubmits. Contrib code with flaky tests has been a major pain point in
>> the past.
>>
>> On Sat, Sep 2, 2023 at 12:02 PM Austin Bennett 
>> wrote:
>>
>>> Wanting us to not miss this. @Mazlum 

Re: Contribution of Asgarde: Error Handling for Beam?

[Daniel Collins via dev]

I think there are a lot of interesting and relatively isolated components
of the project, it might make sense to write per-transform one pagers for
isolated things like the most useful pieces (just basically copying the
documentation and justifying the API) instead of doing a one-shot import or
having it live forever in an external project.

-Daniel

On Fri, Sep 8, 2023 at 11:10 AM Kenneth Knowles  wrote:

> I agree with everyone about "not everything has to be in the Beam repo". I
> really like the idea of having a clearer "ecosystem" section of the
> website, which is sort of started at
> https://beam.apache.org/community/integrations/ but that is not very
> prominent.
>
> Agree with John though. The transforms in Asgarde could potentially be
> used in Beam. Potentially best accomplished by just adding them as
> transforms to the core Java SDK?
>
> Kenn
>
> On Wed, Sep 6, 2023 at 1:46 PM John Casey via dev 
> wrote:
>
>> Agreed on documentation and on keeping it in a separate repo.
>>
>> We have a few pretty significant beam extensions (scio and Dataflow
>> Templates also come to mind) that Beam should highlight, but are separate
>> repos for their own governance, contributions, and release reasons.
>>
>> The difference with Asgarde is that we might want to use it in Beam
>> itself, which makes it more reasonable to include in the main repo.
>>
>> On Tue, Sep 5, 2023 at 8:36 PM Robert Bradshaw via dev <
>> dev@beam.apache.org> wrote:
>>
>>> I think this is a great library. I'm on the fence of whether it makes
>>> sense to include with Beam proper vs. be a library that builds on top of
>>> Beam. (Would there be benefits of tighter integration? There is the
>>> maintenance/loss of governance issue.) I am definitely not on the side that
>>> the entire Beam ecosystem needs to be distributed/maintained by Beam
>>> itself.
>>>
>>> Regardless of the direction we go, I think it could make a lot of sense
>>> to put pointers to it in our documentation.
>>>
>>>
>>> On Tue, Sep 5, 2023 at 7:21 AM Danny McCormick via dev <
>>> dev@beam.apache.org> wrote:
>>>
 I think my only concerns here are around the toil we'll be taking on,
 and will we be leaving the asgarde project in a better or worse place.

 From a release standpoint, we would need to release it with the same
 cadence as Beam. Adding asgarde into our standard release process seems
 fairly straightforward, though, so I'm not too worried about it - looks
 like it's basically (1) add a commit like this
 ,
 (2) run this workflow
 ,
 and (3) tag/mark the release as released on GitHub.

 In terms of bug fixes and improvements, though, I'm a little worried
 that we might be leaving things in a worse state since Mazlum has been the
 only contributor thus far, and he would lose some governance (and possibly
 the ability to commit code on his own). An extra motivated community member
 or two could change the math a bit, but I'm not sure if there are actually
 clear advantages to including it in Apache other than visibility. Would
 adding links to our docs calling Asgarde out as an option accomplish the
 same purpose?

 > Let's be careful about whether these tests are included in our
 presubmits. Contrib code with flaky tests has been a major pain point in
 the past.

 +1 - I think if we do this I'd vote that it be in a separate repo (
 github.com/apache/beam-asgarde made sense to me).

 ---

 Overall, I'm probably a slight -1 to adding this to the Apache
 workspace, but +1 to at least adding links from the Beam docs to Asgarde.

 Thanks,
 Danny



 On Tue, Sep 5, 2023 at 12:03 AM Reuven Lax via dev 
 wrote:

> Let's be careful about whether these tests are included in our
> presubmits. Contrib code with flaky tests has been a major pain point in
> the past.
>
> On Sat, Sep 2, 2023 at 12:02 PM Austin Bennett 
> wrote:
>
>> Wanting us to not miss this. @Mazlum TOSUN  is
>> happy to donate Asgarde to our project.
>>
>> It looks like he'd need a SGA and CCLA [ 1 ] on file; anything else?
>>
>> I recalled the donation of Euphoria [ 2 ] , so I looked at those
>> threads [ 3 ]  for insights into the process.  It didn't look like there
>> was a needed VOTE, so mostly a matter of ensuring necessary signatures, 
>> and
>> ideally some sort of consensus [ or non-opposition ] to the donation.
>>
>>
>> [ 1 ] https://www.apache.org/licenses/contributor-agreements.html
>> [ 2 ] https://beam.apache.org/documentation/sdks/java/euphoria/
>> [ 3 ]
>> https://lists.apache.org/thread/xzlx4rm2tvc36mmwvhyvtdvsw7bnjscp
>>
>>

Re: Contribution of Asgarde: Error Handling for Beam?

[Kenneth Knowles]

I agree with everyone about "not everything has to be in the Beam repo". I
really like the idea of having a clearer "ecosystem" section of the
website, which is sort of started at
https://beam.apache.org/community/integrations/ but that is not very
prominent.

Agree with John though. The transforms in Asgarde could potentially be used
in Beam. Potentially best accomplished by just adding them as transforms to
the core Java SDK?

Kenn

On Wed, Sep 6, 2023 at 1:46 PM John Casey via dev 
wrote:

> Agreed on documentation and on keeping it in a separate repo.
>
> We have a few pretty significant beam extensions (scio and Dataflow
> Templates also come to mind) that Beam should highlight, but are separate
> repos for their own governance, contributions, and release reasons.
>
> The difference with Asgarde is that we might want to use it in Beam
> itself, which makes it more reasonable to include in the main repo.
>
> On Tue, Sep 5, 2023 at 8:36 PM Robert Bradshaw via dev <
> dev@beam.apache.org> wrote:
>
>> I think this is a great library. I'm on the fence of whether it makes
>> sense to include with Beam proper vs. be a library that builds on top of
>> Beam. (Would there be benefits of tighter integration? There is the
>> maintenance/loss of governance issue.) I am definitely not on the side that
>> the entire Beam ecosystem needs to be distributed/maintained by Beam
>> itself.
>>
>> Regardless of the direction we go, I think it could make a lot of sense
>> to put pointers to it in our documentation.
>>
>>
>> On Tue, Sep 5, 2023 at 7:21 AM Danny McCormick via dev <
>> dev@beam.apache.org> wrote:
>>
>>> I think my only concerns here are around the toil we'll be taking on,
>>> and will we be leaving the asgarde project in a better or worse place.
>>>
>>> From a release standpoint, we would need to release it with the same
>>> cadence as Beam. Adding asgarde into our standard release process seems
>>> fairly straightforward, though, so I'm not too worried about it - looks
>>> like it's basically (1) add a commit like this
>>> ,
>>> (2) run this workflow
>>> ,
>>> and (3) tag/mark the release as released on GitHub.
>>>
>>> In terms of bug fixes and improvements, though, I'm a little worried
>>> that we might be leaving things in a worse state since Mazlum has been the
>>> only contributor thus far, and he would lose some governance (and possibly
>>> the ability to commit code on his own). An extra motivated community member
>>> or two could change the math a bit, but I'm not sure if there are actually
>>> clear advantages to including it in Apache other than visibility. Would
>>> adding links to our docs calling Asgarde out as an option accomplish the
>>> same purpose?
>>>
>>> > Let's be careful about whether these tests are included in our
>>> presubmits. Contrib code with flaky tests has been a major pain point in
>>> the past.
>>>
>>> +1 - I think if we do this I'd vote that it be in a separate repo (
>>> github.com/apache/beam-asgarde made sense to me).
>>>
>>> ---
>>>
>>> Overall, I'm probably a slight -1 to adding this to the Apache
>>> workspace, but +1 to at least adding links from the Beam docs to Asgarde.
>>>
>>> Thanks,
>>> Danny
>>>
>>>
>>>
>>> On Tue, Sep 5, 2023 at 12:03 AM Reuven Lax via dev 
>>> wrote:
>>>
 Let's be careful about whether these tests are included in our
 presubmits. Contrib code with flaky tests has been a major pain point in
 the past.

 On Sat, Sep 2, 2023 at 12:02 PM Austin Bennett 
 wrote:

> Wanting us to not miss this. @Mazlum TOSUN  is
> happy to donate Asgarde to our project.
>
> It looks like he'd need a SGA and CCLA [ 1 ] on file; anything else?
>
> I recalled the donation of Euphoria [ 2 ] , so I looked at those
> threads [ 3 ]  for insights into the process.  It didn't look like there
> was a needed VOTE, so mostly a matter of ensuring necessary signatures, 
> and
> ideally some sort of consensus [ or non-opposition ] to the donation.
>
>
> [ 1 ] https://www.apache.org/licenses/contributor-agreements.html
> [ 2 ] https://beam.apache.org/documentation/sdks/java/euphoria/
> [ 3 ] https://lists.apache.org/thread/xzlx4rm2tvc36mmwvhyvtdvsw7bnjscp
>
>
>
> On Thu, Jun 15, 2023 at 7:05 AM Kerry Donny-Clark via dev <
> dev@beam.apache.org> wrote:
>
>> This looks like an excellent contribution. I can easily understand
>> the motivation, and I think Beam would benefit from a higher level
>> abstraction for error handling.
>> Kerry
>>
>> On Wed, Jun 14, 2023, 6:31 PM Austin Bennett 
>> wrote:
>>
>>> Hi Beam Devs,
>>>
>>> @Mazlum  was
>>> suggested to consider donating 

Re: Contribution of Asgarde: Error Handling for Beam?

[John Casey via dev]

Agreed on documentation and on keeping it in a separate repo.

We have a few pretty significant beam extensions (scio and Dataflow
Templates also come to mind) that Beam should highlight, but are separate
repos for their own governance, contributions, and release reasons.

The difference with Asgarde is that we might want to use it in Beam itself,
which makes it more reasonable to include in the main repo.

On Tue, Sep 5, 2023 at 8:36 PM Robert Bradshaw via dev 
wrote:

> I think this is a great library. I'm on the fence of whether it makes
> sense to include with Beam proper vs. be a library that builds on top of
> Beam. (Would there be benefits of tighter integration? There is the
> maintenance/loss of governance issue.) I am definitely not on the side that
> the entire Beam ecosystem needs to be distributed/maintained by Beam
> itself.
>
> Regardless of the direction we go, I think it could make a lot of sense to
> put pointers to it in our documentation.
>
>
> On Tue, Sep 5, 2023 at 7:21 AM Danny McCormick via dev <
> dev@beam.apache.org> wrote:
>
>> I think my only concerns here are around the toil we'll be taking on, and
>> will we be leaving the asgarde project in a better or worse place.
>>
>> From a release standpoint, we would need to release it with the same
>> cadence as Beam. Adding asgarde into our standard release process seems
>> fairly straightforward, though, so I'm not too worried about it - looks
>> like it's basically (1) add a commit like this
>> ,
>> (2) run this workflow
>> ,
>> and (3) tag/mark the release as released on GitHub.
>>
>> In terms of bug fixes and improvements, though, I'm a little worried that
>> we might be leaving things in a worse state since Mazlum has been the only
>> contributor thus far, and he would lose some governance (and possibly the
>> ability to commit code on his own). An extra motivated community member or
>> two could change the math a bit, but I'm not sure if there are actually
>> clear advantages to including it in Apache other than visibility. Would
>> adding links to our docs calling Asgarde out as an option accomplish the
>> same purpose?
>>
>> > Let's be careful about whether these tests are included in our
>> presubmits. Contrib code with flaky tests has been a major pain point in
>> the past.
>>
>> +1 - I think if we do this I'd vote that it be in a separate repo (
>> github.com/apache/beam-asgarde made sense to me).
>>
>> ---
>>
>> Overall, I'm probably a slight -1 to adding this to the Apache workspace,
>> but +1 to at least adding links from the Beam docs to Asgarde.
>>
>> Thanks,
>> Danny
>>
>>
>>
>> On Tue, Sep 5, 2023 at 12:03 AM Reuven Lax via dev 
>> wrote:
>>
>>> Let's be careful about whether these tests are included in our
>>> presubmits. Contrib code with flaky tests has been a major pain point in
>>> the past.
>>>
>>> On Sat, Sep 2, 2023 at 12:02 PM Austin Bennett 
>>> wrote:
>>>
 Wanting us to not miss this. @Mazlum TOSUN  is
 happy to donate Asgarde to our project.

 It looks like he'd need a SGA and CCLA [ 1 ] on file; anything else?

 I recalled the donation of Euphoria [ 2 ] , so I looked at those
 threads [ 3 ]  for insights into the process.  It didn't look like there
 was a needed VOTE, so mostly a matter of ensuring necessary signatures, and
 ideally some sort of consensus [ or non-opposition ] to the donation.


 [ 1 ] https://www.apache.org/licenses/contributor-agreements.html
 [ 2 ] https://beam.apache.org/documentation/sdks/java/euphoria/
 [ 3 ] https://lists.apache.org/thread/xzlx4rm2tvc36mmwvhyvtdvsw7bnjscp



 On Thu, Jun 15, 2023 at 7:05 AM Kerry Donny-Clark via dev <
 dev@beam.apache.org> wrote:

> This looks like an excellent contribution. I can easily understand the
> motivation, and I think Beam would benefit from a higher level abstraction
> for error handling.
> Kerry
>
> On Wed, Jun 14, 2023, 6:31 PM Austin Bennett 
> wrote:
>
>> Hi Beam Devs,
>>
>> @Mazlum  was
>> suggested to consider donating Asgarde
>>  to Beam for Java/Kotlin error
>> handling to Beam [ see:
>> https://2022.beamsummit.org/sessions/error-handling-asgarde/ for
>> last year's Beam Summit talk ], he is also the author of Pasgard
>> e [ for Python ] and Milgard [
>> for a simplified Kotlin API ].
>>
>> Would Asgarde be a good contribution, something the Beam community
>> would be willing to accept?  I imagine we might want it to live at
>> github.com/apache/beam-asgarde ?  Or perhaps there is a good place
>> in github.com/apache/beam ??
>>

Re: Contribution of Asgarde: Error Handling for Beam?

[Robert Bradshaw via dev]

I think this is a great library. I'm on the fence of whether it makes sense
to include with Beam proper vs. be a library that builds on top of Beam.
(Would there be benefits of tighter integration? There is the
maintenance/loss of governance issue.) I am definitely not on the side that
the entire Beam ecosystem needs to be distributed/maintained by Beam
itself.

Regardless of the direction we go, I think it could make a lot of sense to
put pointers to it in our documentation.


On Tue, Sep 5, 2023 at 7:21 AM Danny McCormick via dev 
wrote:

> I think my only concerns here are around the toil we'll be taking on, and
> will we be leaving the asgarde project in a better or worse place.
>
> From a release standpoint, we would need to release it with the same
> cadence as Beam. Adding asgarde into our standard release process seems
> fairly straightforward, though, so I'm not too worried about it - looks
> like it's basically (1) add a commit like this
> ,
> (2) run this workflow
> ,
> and (3) tag/mark the release as released on GitHub.
>
> In terms of bug fixes and improvements, though, I'm a little worried that
> we might be leaving things in a worse state since Mazlum has been the only
> contributor thus far, and he would lose some governance (and possibly the
> ability to commit code on his own). An extra motivated community member or
> two could change the math a bit, but I'm not sure if there are actually
> clear advantages to including it in Apache other than visibility. Would
> adding links to our docs calling Asgarde out as an option accomplish the
> same purpose?
>
> > Let's be careful about whether these tests are included in our
> presubmits. Contrib code with flaky tests has been a major pain point in
> the past.
>
> +1 - I think if we do this I'd vote that it be in a separate repo (
> github.com/apache/beam-asgarde made sense to me).
>
> ---
>
> Overall, I'm probably a slight -1 to adding this to the Apache workspace,
> but +1 to at least adding links from the Beam docs to Asgarde.
>
> Thanks,
> Danny
>
>
>
> On Tue, Sep 5, 2023 at 12:03 AM Reuven Lax via dev 
> wrote:
>
>> Let's be careful about whether these tests are included in our
>> presubmits. Contrib code with flaky tests has been a major pain point in
>> the past.
>>
>> On Sat, Sep 2, 2023 at 12:02 PM Austin Bennett  wrote:
>>
>>> Wanting us to not miss this. @Mazlum TOSUN  is
>>> happy to donate Asgarde to our project.
>>>
>>> It looks like he'd need a SGA and CCLA [ 1 ] on file; anything else?
>>>
>>> I recalled the donation of Euphoria [ 2 ] , so I looked at those threads
>>> [ 3 ]  for insights into the process.  It didn't look like there was a
>>> needed VOTE, so mostly a matter of ensuring necessary signatures, and
>>> ideally some sort of consensus [ or non-opposition ] to the donation.
>>>
>>>
>>> [ 1 ] https://www.apache.org/licenses/contributor-agreements.html
>>> [ 2 ] https://beam.apache.org/documentation/sdks/java/euphoria/
>>> [ 3 ] https://lists.apache.org/thread/xzlx4rm2tvc36mmwvhyvtdvsw7bnjscp
>>>
>>>
>>>
>>> On Thu, Jun 15, 2023 at 7:05 AM Kerry Donny-Clark via dev <
>>> dev@beam.apache.org> wrote:
>>>
 This looks like an excellent contribution. I can easily understand the
 motivation, and I think Beam would benefit from a higher level abstraction
 for error handling.
 Kerry

 On Wed, Jun 14, 2023, 6:31 PM Austin Bennett  wrote:

> Hi Beam Devs,
>
> @Mazlum  was
> suggested to consider donating Asgarde
>  to Beam for Java/Kotlin error
> handling to Beam [ see:
> https://2022.beamsummit.org/sessions/error-handling-asgarde/ for last
> year's Beam Summit talk ], he is also the author of Pasgard
> e [ for Python ] and Milgard [
> for a simplified Kotlin API ].
>
> Would Asgarde be a good contribution, something the Beam community
> would be willing to accept?  I imagine we might want it to live at
> github.com/apache/beam-asgarde ?  Or perhaps there is a good place in
> github.com/apache/beam ??
>
> Especially once/if officially part of Beam, I imagine we'd add
> follow-up items like getting onto the website/docs, and related.
>
> Cheers,
> Austin
>
>
> P.S.  This might warrant separate/additional conversations for his
> other libraries, but let's focus any discussion on Asgarde for now?
>

Re: Contribution of Asgarde: Error Handling for Beam?

[Danny McCormick via dev]

I think my only concerns here are around the toil we'll be taking on, and
will we be leaving the asgarde project in a better or worse place.

>From a release standpoint, we would need to release it with the same
cadence as Beam. Adding asgarde into our standard release process seems
fairly straightforward, though, so I'm not too worried about it - looks
like it's basically (1) add a commit like this
,
(2) run this workflow
,
and (3) tag/mark the release as released on GitHub.

In terms of bug fixes and improvements, though, I'm a little worried that
we might be leaving things in a worse state since Mazlum has been the only
contributor thus far, and he would lose some governance (and possibly the
ability to commit code on his own). An extra motivated community member or
two could change the math a bit, but I'm not sure if there are actually
clear advantages to including it in Apache other than visibility. Would
adding links to our docs calling Asgarde out as an option accomplish the
same purpose?

> Let's be careful about whether these tests are included in our
presubmits. Contrib code with flaky tests has been a major pain point in
the past.

+1 - I think if we do this I'd vote that it be in a separate repo (
github.com/apache/beam-asgarde made sense to me).

---

Overall, I'm probably a slight -1 to adding this to the Apache workspace,
but +1 to at least adding links from the Beam docs to Asgarde.

Thanks,
Danny



On Tue, Sep 5, 2023 at 12:03 AM Reuven Lax via dev 
wrote:

> Let's be careful about whether these tests are included in our presubmits.
> Contrib code with flaky tests has been a major pain point in the past.
>
> On Sat, Sep 2, 2023 at 12:02 PM Austin Bennett  wrote:
>
>> Wanting us to not miss this. @Mazlum TOSUN  is
>> happy to donate Asgarde to our project.
>>
>> It looks like he'd need a SGA and CCLA [ 1 ] on file; anything else?
>>
>> I recalled the donation of Euphoria [ 2 ] , so I looked at those threads
>> [ 3 ]  for insights into the process.  It didn't look like there was a
>> needed VOTE, so mostly a matter of ensuring necessary signatures, and
>> ideally some sort of consensus [ or non-opposition ] to the donation.
>>
>>
>> [ 1 ] https://www.apache.org/licenses/contributor-agreements.html
>> [ 2 ] https://beam.apache.org/documentation/sdks/java/euphoria/
>> [ 3 ] https://lists.apache.org/thread/xzlx4rm2tvc36mmwvhyvtdvsw7bnjscp
>>
>>
>>
>> On Thu, Jun 15, 2023 at 7:05 AM Kerry Donny-Clark via dev <
>> dev@beam.apache.org> wrote:
>>
>>> This looks like an excellent contribution. I can easily understand the
>>> motivation, and I think Beam would benefit from a higher level abstraction
>>> for error handling.
>>> Kerry
>>>
>>> On Wed, Jun 14, 2023, 6:31 PM Austin Bennett  wrote:
>>>
 Hi Beam Devs,

 @Mazlum  was
 suggested to consider donating Asgarde
  to Beam for Java/Kotlin error
 handling to Beam [ see:
 https://2022.beamsummit.org/sessions/error-handling-asgarde/ for last
 year's Beam Summit talk ], he is also the author of Pasgard
 e [ for Python ] and Milgard [
 for a simplified Kotlin API ].

 Would Asgarde be a good contribution, something the Beam community
 would be willing to accept?  I imagine we might want it to live at
 github.com/apache/beam-asgarde ?  Or perhaps there is a good place in
 github.com/apache/beam ??

 Especially once/if officially part of Beam, I imagine we'd add
 follow-up items like getting onto the website/docs, and related.

 Cheers,
 Austin


 P.S.  This might warrant separate/additional conversations for his
 other libraries, but let's focus any discussion on Asgarde for now?

>>>

Re: Contribution of Asgarde: Error Handling for Beam?

[Reuven Lax via dev]

Let's be careful about whether these tests are included in our presubmits.
Contrib code with flaky tests has been a major pain point in the past.

On Sat, Sep 2, 2023 at 12:02 PM Austin Bennett  wrote:

> Wanting us to not miss this. @Mazlum TOSUN  is
> happy to donate Asgarde to our project.
>
> It looks like he'd need a SGA and CCLA [ 1 ] on file; anything else?
>
> I recalled the donation of Euphoria [ 2 ] , so I looked at those threads [
> 3 ]  for insights into the process.  It didn't look like there was a needed
> VOTE, so mostly a matter of ensuring necessary signatures, and ideally some
> sort of consensus [ or non-opposition ] to the donation.
>
>
> [ 1 ] https://www.apache.org/licenses/contributor-agreements.html
> [ 2 ] https://beam.apache.org/documentation/sdks/java/euphoria/
> [ 3 ] https://lists.apache.org/thread/xzlx4rm2tvc36mmwvhyvtdvsw7bnjscp
>
>
>
> On Thu, Jun 15, 2023 at 7:05 AM Kerry Donny-Clark via dev <
> dev@beam.apache.org> wrote:
>
>> This looks like an excellent contribution. I can easily understand the
>> motivation, and I think Beam would benefit from a higher level abstraction
>> for error handling.
>> Kerry
>>
>> On Wed, Jun 14, 2023, 6:31 PM Austin Bennett  wrote:
>>
>>> Hi Beam Devs,
>>>
>>> @Mazlum  was
>>> suggested to consider donating Asgarde
>>>  to Beam for Java/Kotlin error
>>> handling to Beam [ see:
>>> https://2022.beamsummit.org/sessions/error-handling-asgarde/ for last
>>> year's Beam Summit talk ], he is also the author of Pasgard
>>> e [ for Python ] and Milgard [
>>> for a simplified Kotlin API ].
>>>
>>> Would Asgarde be a good contribution, something the Beam community would
>>> be willing to accept?  I imagine we might want it to live at
>>> github.com/apache/beam-asgarde ?  Or perhaps there is a good place in
>>> github.com/apache/beam ??
>>>
>>> Especially once/if officially part of Beam, I imagine we'd add follow-up
>>> items like getting onto the website/docs, and related.
>>>
>>> Cheers,
>>> Austin
>>>
>>>
>>> P.S.  This might warrant separate/additional conversations for his other
>>> libraries, but let's focus any discussion on Asgarde for now?
>>>
>>

Contribution of Asgarde: Error Handling for Beam?

[Austin Bennett]

Wanting us to not miss this. @Mazlum TOSUN  is
happy to donate Asgarde to our project.

It looks like he'd need a SGA and CCLA [ 1 ] on file; anything else?

I recalled the donation of Euphoria [ 2 ] , so I looked at those threads [
3 ]  for insights into the process.  It didn't look like there was a needed
VOTE, so mostly a matter of ensuring necessary signatures, and ideally some
sort of consensus [ or non-opposition ] to the donation.


[ 1 ] https://www.apache.org/licenses/contributor-agreements.html
[ 2 ] https://beam.apache.org/documentation/sdks/java/euphoria/
[ 3 ] https://lists.apache.org/thread/xzlx4rm2tvc36mmwvhyvtdvsw7bnjscp



On Thu, Jun 15, 2023 at 7:05 AM Kerry Donny-Clark via dev <
dev@beam.apache.org> wrote:

> This looks like an excellent contribution. I can easily understand the
> motivation, and I think Beam would benefit from a higher level abstraction
> for error handling.
> Kerry
>
> On Wed, Jun 14, 2023, 6:31 PM Austin Bennett  wrote:
>
>> Hi Beam Devs,
>>
>> @Mazlum  was
>> suggested to consider donating Asgarde
>>  to Beam for Java/Kotlin error
>> handling to Beam [ see:
>> https://2022.beamsummit.org/sessions/error-handling-asgarde/ for last
>> year's Beam Summit talk ], he is also the author of Pasgard
>> e [ for Python ] and Milgard [ for
>> a simplified Kotlin API ].
>>
>> Would Asgarde be a good contribution, something the Beam community would
>> be willing to accept?  I imagine we might want it to live at
>> github.com/apache/beam-asgarde ?  Or perhaps there is a good place in
>> github.com/apache/beam ??
>>
>> Especially once/if officially part of Beam, I imagine we'd add follow-up
>> items like getting onto the website/docs, and related.
>>
>> Cheers,
>> Austin
>>
>>
>> P.S.  This might warrant separate/additional conversations for his other
>> libraries, but let's focus any discussion on Asgarde for now?
>>
>

Re: Asgarde: Error Handling for Beam?

[Kerry Donny-Clark via dev]

This looks like an excellent contribution. I can easily understand the
motivation, and I think Beam would benefit from a higher level abstraction
for error handling.
Kerry

On Wed, Jun 14, 2023, 6:31 PM Austin Bennett  wrote:

> Hi Beam Devs,
>
> @Mazlum  was
> suggested to consider donating Asgarde
>  to Beam for Java/Kotlin error
> handling to Beam [ see:
> https://2022.beamsummit.org/sessions/error-handling-asgarde/ for last
> year's Beam Summit talk ], he is also the author of Pasgard
> e [ for Python ] and Milgard [ for
> a simplified Kotlin API ].
>
> Would Asgarde be a good contribution, something the Beam community would
> be willing to accept?  I imagine we might want it to live at
> github.com/apache/beam-asgarde ?  Or perhaps there is a good place in
> github.com/apache/beam ??
>
> Especially once/if officially part of Beam, I imagine we'd add follow-up
> items like getting onto the website/docs, and related.
>
> Cheers,
> Austin
>
>
> P.S.  This might warrant separate/additional conversations for his other
> libraries, but let's focus any discussion on Asgarde for now?
>

Asgarde: Error Handling for Beam?

[Austin Bennett]

Hi Beam Devs,

@Mazlum  was suggested
to consider donating Asgarde  to
Beam for Java/Kotlin error handling to Beam [ see:
https://2022.beamsummit.org/sessions/error-handling-asgarde/ for last
year's Beam Summit talk ], he is also the author of Pasgard
e [ for Python ] and Milgard [ for a
simplified Kotlin API ].

Would Asgarde be a good contribution, something the Beam community would be
willing to accept?  I imagine we might want it to live at
github.com/apache/beam-asgarde ?  Or perhaps there is a good place in
github.com/apache/beam ??

Especially once/if officially part of Beam, I imagine we'd add follow-up
items like getting onto the website/docs, and related.

Cheers,
Austin


P.S.  This might warrant separate/additional conversations for his other
libraries, but let's focus any discussion on Asgarde for now?