For anyone who hasn't clicked over the Asgarde, my TL;DR description of it is that it adds the "failure monad" aka "andThen" style error/result handling on top of chaining of PCollections. So it is at a similar level of abstraction of our basic transforms and generally useful for chaining dead-letter side outputs. It is no more or less appropriate for the core SDK than, say, the Project/Filter/Join transforms, or Watch, etc. If we actually aspired to have a thin core with the accessories like that in another place, then it should go to that other place.
Kenn On Fri, Sep 8, 2023 at 11:24 AM Daniel Collins via dev <dev@beam.apache.org> wrote: > > until we *require* Asgard on a core transform, it shouldn't be in the > main repo > > I don't think this is necessarily true if it solves end user use cases. If > there is a specific transform that solves a specific use case, we could > include it in the transforms folder for end-users, even if it isn't > utilized in the I/Os at present. Hence the suggestion to take the most > promising transforms and propose adding them with documentation, apis and > rationale. > > -Daniel > > On Fri, Sep 8, 2023 at 11:20 AM Robert Burke <rob...@frantil.com> wrote: > >> I would say until we *require* Asgard on a core transform, it shouldn't >> be in the main repo. >> >> Incorporating something before there's a need for it is premature >> abstraction. We can't do things because they *might* be useful. Let's see >> concrete places where they are useful, or we're already having a similar >> need solved a different way. >> >> Beam is complicated by itself, and we do encourage multiple ways of >> solving problems, but that says to me that having an out of repo ecosystem >> is the right path, rather than incorporation. >> >> On Fri, Sep 8, 2023, 8:14 AM Daniel Collins via dev <dev@beam.apache.org> >> wrote: >> >>> I think there are a lot of interesting and relatively isolated >>> components of the project, it might make sense to write per-transform one >>> pagers for isolated things like the most useful pieces (just basically >>> copying the documentation and justifying the API) instead of doing a >>> one-shot import or having it live forever in an external project. >>> >>> -Daniel >>> >>> On Fri, Sep 8, 2023 at 11:10 AM Kenneth Knowles <k...@apache.org> wrote: >>> >>>> I agree with everyone about "not everything has to be in the Beam >>>> repo". I really like the idea of having a clearer "ecosystem" section of >>>> the website, which is sort of started at >>>> https://beam.apache.org/community/integrations/ but that is not very >>>> prominent. >>>> >>>> Agree with John though. The transforms in Asgarde could potentially be >>>> used in Beam. Potentially best accomplished by just adding them as >>>> transforms to the core Java SDK? >>>> >>>> Kenn >>>> >>>> On Wed, Sep 6, 2023 at 1:46 PM John Casey via dev <dev@beam.apache.org> >>>> wrote: >>>> >>>>> Agreed on documentation and on keeping it in a separate repo. >>>>> >>>>> We have a few pretty significant beam extensions (scio and Dataflow >>>>> Templates also come to mind) that Beam should highlight, but are separate >>>>> repos for their own governance, contributions, and release reasons. >>>>> >>>>> The difference with Asgarde is that we might want to use it in Beam >>>>> itself, which makes it more reasonable to include in the main repo. >>>>> >>>>> On Tue, Sep 5, 2023 at 8:36 PM Robert Bradshaw via dev < >>>>> dev@beam.apache.org> wrote: >>>>> >>>>>> I think this is a great library. I'm on the fence of whether it makes >>>>>> sense to include with Beam proper vs. be a library that builds on top of >>>>>> Beam. (Would there be benefits of tighter integration? There is the >>>>>> maintenance/loss of governance issue.) I am definitely not on the side >>>>>> that >>>>>> the entire Beam ecosystem needs to be distributed/maintained by Beam >>>>>> itself. >>>>>> >>>>>> Regardless of the direction we go, I think it could make a lot of >>>>>> sense to put pointers to it in our documentation. >>>>>> >>>>>> >>>>>> On Tue, Sep 5, 2023 at 7:21 AM Danny McCormick via dev < >>>>>> dev@beam.apache.org> wrote: >>>>>> >>>>>>> I think my only concerns here are around the toil we'll be taking >>>>>>> on, and will we be leaving the asgarde project in a better or worse >>>>>>> place. >>>>>>> >>>>>>> From a release standpoint, we would need to release it with the same >>>>>>> cadence as Beam. Adding asgarde into our standard release process seems >>>>>>> fairly straightforward, though, so I'm not too worried about it - looks >>>>>>> like it's basically (1) add a commit like this >>>>>>> <https://github.com/tosun-si/asgarde/commit/432de527d67dc71f06507328319b466b6d0fb56a>, >>>>>>> (2) run this workflow >>>>>>> <https://github.com/tosun-si/asgarde/blob/main/.github/workflows/publish-project.yml>, >>>>>>> and (3) tag/mark the release as released on GitHub. >>>>>>> >>>>>>> In terms of bug fixes and improvements, though, I'm a little worried >>>>>>> that we might be leaving things in a worse state since Mazlum has been >>>>>>> the >>>>>>> only contributor thus far, and he would lose some governance (and >>>>>>> possibly >>>>>>> the ability to commit code on his own). An extra motivated community >>>>>>> member >>>>>>> or two could change the math a bit, but I'm not sure if there are >>>>>>> actually >>>>>>> clear advantages to including it in Apache other than visibility. Would >>>>>>> adding links to our docs calling Asgarde out as an option accomplish the >>>>>>> same purpose? >>>>>>> >>>>>>> > Let's be careful about whether these tests are included in our >>>>>>> presubmits. Contrib code with flaky tests has been a major pain point in >>>>>>> the past. >>>>>>> >>>>>>> +1 - I think if we do this I'd vote that it be in a separate repo ( >>>>>>> github.com/apache/beam-asgarde made sense to me). >>>>>>> >>>>>>> --------------------------------------- >>>>>>> >>>>>>> Overall, I'm probably a slight -1 to adding this to the Apache >>>>>>> workspace, but +1 to at least adding links from the Beam docs to >>>>>>> Asgarde. >>>>>>> >>>>>>> Thanks, >>>>>>> Danny >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Sep 5, 2023 at 12:03 AM Reuven Lax via dev < >>>>>>> dev@beam.apache.org> wrote: >>>>>>> >>>>>>>> Let's be careful about whether these tests are included in our >>>>>>>> presubmits. Contrib code with flaky tests has been a major pain point >>>>>>>> in >>>>>>>> the past. >>>>>>>> >>>>>>>> On Sat, Sep 2, 2023 at 12:02 PM Austin Bennett <aus...@apache.org> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Wanting us to not miss this. @Mazlum TOSUN >>>>>>>>> <mazlum.to...@gmail.com> is happy to donate Asgarde to >>>>>>>>> our project. >>>>>>>>> >>>>>>>>> It looks like he'd need a SGA and CCLA [ 1 ] on file; anything >>>>>>>>> else? >>>>>>>>> >>>>>>>>> I recalled the donation of Euphoria [ 2 ] , so I looked at those >>>>>>>>> threads [ 3 ] for insights into the process. It didn't look like >>>>>>>>> there >>>>>>>>> was a needed VOTE, so mostly a matter of ensuring necessary >>>>>>>>> signatures, and >>>>>>>>> ideally some sort of consensus [ or non-opposition ] to the donation. >>>>>>>>> >>>>>>>>> >>>>>>>>> [ 1 ] https://www.apache.org/licenses/contributor-agreements.html >>>>>>>>> [ 2 ] https://beam.apache.org/documentation/sdks/java/euphoria/ >>>>>>>>> [ 3 ] >>>>>>>>> https://lists.apache.org/thread/xzlx4rm2tvc36mmwvhyvtdvsw7bnjscp >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Jun 15, 2023 at 7:05 AM Kerry Donny-Clark via dev < >>>>>>>>> dev@beam.apache.org> wrote: >>>>>>>>> >>>>>>>>>> This looks like an excellent contribution. I can easily >>>>>>>>>> understand the motivation, and I think Beam would benefit from a >>>>>>>>>> higher >>>>>>>>>> level abstraction for error handling. >>>>>>>>>> Kerry >>>>>>>>>> >>>>>>>>>> On Wed, Jun 14, 2023, 6:31 PM Austin Bennett <aus...@apache.org> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Beam Devs, >>>>>>>>>>> >>>>>>>>>>> @Mazlum <https://www.linkedin.com/in/mazlum-tosun-900b1812/> >>>>>>>>>>> was suggested to consider donating Asgarde >>>>>>>>>>> <https://github.com/tosun-si/asgarde> to Beam for Java/Kotlin >>>>>>>>>>> error handling to Beam [ see: >>>>>>>>>>> https://2022.beamsummit.org/sessions/error-handling-asgarde/ >>>>>>>>>>> for last year's Beam Summit talk ], he is also the author of >>>>>>>>>>> Pasgard <https://github.com/tosun-si/pasgarde>e [ for Python ] >>>>>>>>>>> and Milgard [ for a simplified Kotlin API ]. >>>>>>>>>>> >>>>>>>>>>> Would Asgarde be a good contribution, something the Beam >>>>>>>>>>> community would be willing to accept? I imagine we might want it >>>>>>>>>>> to live >>>>>>>>>>> at github.com/apache/beam-asgarde ? Or perhaps there is a good >>>>>>>>>>> place in github.com/apache/beam ?? >>>>>>>>>>> >>>>>>>>>>> Especially once/if officially part of Beam, I imagine we'd add >>>>>>>>>>> follow-up items like getting onto the website/docs, and related. >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> Austin >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> P.S. This might warrant separate/additional conversations for >>>>>>>>>>> his other libraries, but let's focus any discussion on Asgarde for >>>>>>>>>>> now? >>>>>>>>>>> >>>>>>>>>>
