Re: Join a meeting to help coordinate implementing a Dask Runner for Beam
I wanted to share that Ryan gave a presentation about his (and Charles') work on Pangeo Forge at Scipy 2022 (in Austin just before Beam Summit!), with a couple mentions of their transition to Beam [1]. There were also a couple of other talks about Pangeo [2,3] with some Beam/xarray-beam references in there. [1] https://www.youtube.com/watch?v=sY20UpYCAEE=PLYx7XA2nY5Gde0WF1yswQw5InhmSNED8o=9 [2] https://www.youtube.com/watch?v=7niNfs3ZpfQ=PLYx7XA2nY5Gfb0tQyezb4Gsf1nVsy86zt=2 [3] https://www.youtube.com/watch?v=ftlgOESINvo=PLYx7XA2nY5Gfb0tQyezb4Gsf1nVsy86zt=3 On Tue, Jun 21, 2022 at 9:29 AM Ahmet Altay wrote: > Were you able to meet? If yes, I would be very interested in a summary if > someone would like to share that :) > > On Mon, Jun 13, 2022 at 9:16 AM Pablo Estrada wrote: > >> Also added my availability... please do invite me as well : ) >> -P. >> >> On Mon, Jun 13, 2022 at 6:57 AM Kenneth Knowles wrote: >> >>> I would love to try to join any meetings if you add me. My calendar is >>> too chaotic to be useful on the when2meet :-) but I can often move things >>> around. >>> >>> Kenn >>> >>> On Wed, Jun 8, 2022 at 2:50 PM Brian Hulette >>> wrote: >>> Thanks for reaching out, Ryan, this sounds really cool. I added my availability to the calendar since I'm interested in this space, but I'm not sure I can offer much help - I don't have any experience building a runner, to date I've worked exclusively on the SDK side of Beam. So I hope some other folks can join as well :) @Pablo Estrada might have some useful insight - he's been working on a spike to build a Ray runner. On Wed, Jun 8, 2022 at 12:53 PM Robert Bradshaw wrote: > This sounds like a great project. Unfortunately I wouldn't be able to > meet next week, but would be happy to meet some other time and if that > doesn't work answer questions over email, etc. Looking forward to a > Dask runner. > > On Wed, Jun 8, 2022 at 9:04 AM Ryan Abernathey > wrote: > > > > Dear Beamer, > > > > Thank you for all of your work on this amazing project. I am new to > Beam and am quite excited about its potential to help with some data > processing challenges in my field of climate science. > > > > Our community is interested in running Beam on Dask Distributed > clusters, which we already know how to deploy. This has been discussed at > https://issues.apache.org/jira/browse/BEAM-5336 and > https://github.com/apache/beam/issues/18962. It seems technically > feasible. > > > > We are trying to organize a meeting next week to kickstart and > coordinate this effort. It would be great if we could entrain some Beam > maintainers into this meeting. If you have interest in this topic and are > available next week, please share your availability here - > https://www.when2meet.com/?15861604-jLnA4 > > > > Alternatively, if you have any guidance or suggestions you wish to > provide by email or GitHub discussion, we welcome your input. > > > > Thanks again for your open source work. > > > > Best, > > Ryan Abernathey > > >
Re: Join a meeting to help coordinate implementing a Dask Runner for Beam
+Brian Hulette and +Kenneth Knowles were curious about this effort. Alex can you elaborate on what happened with GH actions? We can help you figure out the CI story. On Tue, Jun 21, 2022 at 11:06 AM Alex Merose wrote: > We had a great meeting last week on this topic! Here is a proposal / > meeting notes doc: > > https://docs.google.com/document/d/1Awj_eNmH-WRSte3bKcCcUlQDiZ5mMKmCO_xV-mHWAak/edit#heading=h.y0pwg4polebc > > Tomorrow, another engineer (https://github.com/cisaacstern) and I are > meeting to create an initial prototype of the Dask runner in the main Beam > repo. Let us know if you'd like to help out in any way. We'll post updates > in this mailing list + the above doc. > > Best, > Alex Merose > > On 2022/06/08 14:22:41 Ryan Abernathey wrote: > > Dear Beamer, > > > > Thank you for all of your work on this amazing project. I am new to Beam > > and am quite excited about its potential to help with some data > processing > > challenges in my field of climate science. > > > > Our community is interested in running Beam on Dask Distributed clusters, > > which we already know how to deploy. This has been discussed at > > https://issues.apache.org/jira/browse/BEAM-5336 and > > https://github.com/apache/beam/issues/18962. It seems technically > feasible. > > > > We are trying to organize a meeting next week to kickstart and coordinate > > this effort. It would be great if we could entrain some Beam maintainers > > into this meeting. If you have interest in this topic and are available > > next week, please share your availability here - > > https://www.when2meet.com/?15861604-jLnA4 > > > > Alternatively, if you have any guidance or suggestions you wish to > provide > > by email or GitHub discussion, we welcome your input. > > > > Thanks again for your open source work. > > > > Best, > > Ryan Abernathey > > >
RE: Join a meeting to help coordinate implementing a Dask Runner for Beam
Dear Beam & Dask communities, Together with Pablo and Charles, I've hacked together an initial prototype of a Dask runner for Beam. I'm happy to announce that I have minimum viable working version in a fork here: https://github.com/alxmrs/beam/pull/1 There's definitely more work to do here – more operations to implement, tests to write, style guides to follow, etc. However, I'm pleased that there are enough operations implemented to run test pipelines with assertions. >From here, what are good next steps? Best, Alex PS – Meeting / design notes are available in this doc: https://docs.google.com/document/d/1Awj_eNmH-WRSte3bKcCcUlQDiZ5mMKmCO_xV-mHWAak/edit#heading=h.y0pwg4polebc On 2022/06/08 14:22:41 Ryan Abernathey wrote: > Dear Beamer, > > Thank you for all of your work on this amazing project. I am new to Beam > and am quite excited about its potential to help with some data processing > challenges in my field of climate science. > > Our community is interested in running Beam on Dask Distributed clusters, > which we already know how to deploy. This has been discussed at > https://issues.apache.org/jira/browse/BEAM-5336 and > https://github.com/apache/beam/issues/18962. It seems technically feasible. > > We are trying to organize a meeting next week to kickstart and coordinate > this effort. It would be great if we could entrain some Beam maintainers > into this meeting. If you have interest in this topic and are available > next week, please share your availability here - > https://www.when2meet.com/?15861604-jLnA4 > > Alternatively, if you have any guidance or suggestions you wish to provide > by email or GitHub discussion, we welcome your input. > > Thanks again for your open source work. > > Best, > Ryan Abernathey >
Re: Join a meeting to help coordinate implementing a Dask Runner for Beam
Looks/Sounds great! On Tue, Jun 21, 2022 at 11:06 AM Alex Merose wrote: > We had a great meeting last week on this topic! Here is a proposal / > meeting notes doc: > > https://docs.google.com/document/d/1Awj_eNmH-WRSte3bKcCcUlQDiZ5mMKmCO_xV-mHWAak/edit#heading=h.y0pwg4polebc > > Tomorrow, another engineer (https://github.com/cisaacstern) and I are > meeting to create an initial prototype of the Dask runner in the main Beam > repo. Let us know if you'd like to help out in any way. We'll post updates > in this mailing list + the above doc. > > Best, > Alex Merose > > On 2022/06/08 14:22:41 Ryan Abernathey wrote: > > Dear Beamer, > > > > Thank you for all of your work on this amazing project. I am new to Beam > > and am quite excited about its potential to help with some data > processing > > challenges in my field of climate science. > > > > Our community is interested in running Beam on Dask Distributed clusters, > > which we already know how to deploy. This has been discussed at > > https://issues.apache.org/jira/browse/BEAM-5336 and > > https://github.com/apache/beam/issues/18962. It seems technically > feasible. > > > > We are trying to organize a meeting next week to kickstart and coordinate > > this effort. It would be great if we could entrain some Beam maintainers > > into this meeting. If you have interest in this topic and are available > > next week, please share your availability here - > > https://www.when2meet.com/?15861604-jLnA4 > > > > Alternatively, if you have any guidance or suggestions you wish to > provide > > by email or GitHub discussion, we welcome your input. > > > > Thanks again for your open source work. > > > > Best, > > Ryan Abernathey > > >
Re: Join a meeting to help coordinate implementing a Dask Runner for Beam
I would love to try to join any meetings if you add me. My calendar is too chaotic to be useful on the when2meet :-) but I can often move things around. Kenn On Wed, Jun 8, 2022 at 2:50 PM Brian Hulette wrote: > Thanks for reaching out, Ryan, this sounds really cool. I added my > availability to the calendar since I'm interested in this space, but I'm > not sure I can offer much help - I don't have any experience building a > runner, to date I've worked exclusively on the SDK side of Beam. So I hope > some other folks can join as well :) > > @Pablo Estrada might have some useful insight - he's > been working on a spike to build a Ray runner. > > > On Wed, Jun 8, 2022 at 12:53 PM Robert Bradshaw > wrote: > >> This sounds like a great project. Unfortunately I wouldn't be able to >> meet next week, but would be happy to meet some other time and if that >> doesn't work answer questions over email, etc. Looking forward to a >> Dask runner. >> >> On Wed, Jun 8, 2022 at 9:04 AM Ryan Abernathey >> wrote: >> > >> > Dear Beamer, >> > >> > Thank you for all of your work on this amazing project. I am new to >> Beam and am quite excited about its potential to help with some data >> processing challenges in my field of climate science. >> > >> > Our community is interested in running Beam on Dask Distributed >> clusters, which we already know how to deploy. This has been discussed at >> https://issues.apache.org/jira/browse/BEAM-5336 and >> https://github.com/apache/beam/issues/18962. It seems technically >> feasible. >> > >> > We are trying to organize a meeting next week to kickstart and >> coordinate this effort. It would be great if we could entrain some Beam >> maintainers into this meeting. If you have interest in this topic and are >> available next week, please share your availability here - >> https://www.when2meet.com/?15861604-jLnA4 >> > >> > Alternatively, if you have any guidance or suggestions you wish to >> provide by email or GitHub discussion, we welcome your input. >> > >> > Thanks again for your open source work. >> > >> > Best, >> > Ryan Abernathey >> > >> >
Join a meeting to help coordinate implementing a Dask Runner for Beam
Dear Beamer, Thank you for all of your work on this amazing project. I am new to Beam and am quite excited about its potential to help with some data processing challenges in my field of climate science. Our community is interested in running Beam on Dask Distributed clusters, which we already know how to deploy. This has been discussed at https://issues.apache.org/jira/browse/BEAM-5336 and https://github.com/apache/beam/issues/18962. It seems technically feasible. We are trying to organize a meeting next week to kickstart and coordinate this effort. It would be great if we could entrain some Beam maintainers into this meeting. If you have interest in this topic and are available next week, please share your availability here - https://www.when2meet.com/?15861604-jLnA4 Alternatively, if you have any guidance or suggestions you wish to provide by email or GitHub discussion, we welcome your input. Thanks again for your open source work. Best, Ryan Abernathey
