DynamicMessage in protobufs for re-usable beam pipelines

Hi, I intend to use Protobuf options to trigger different transforms and use metadata from storage proto options for sink partitioning etc.. and also allow different protobuf message types flowing via the same pipeline, running as different instances of the pipeline. I am able to parse descript

Beam Dependency Check Report (2020-06-15)

ERROR: File 'src/build/dependencyUpdates/beam-dependency-check-report.html' does not exist

Beam Dependency Check Report (2020-06-15)

ERROR: File 'src/build/dependencyUpdates/beam-dependency-check-report.html' does not exist

Re: Beam Jenkins Migration

Great. Nice to see the detailed plan, too! Kenn On Fri, Jun 12, 2020 at 10:14 AM Udi Meiri wrote: > This is great! Looking forward to it. > > Would any metrics need to be migrated over to the new Jenkins? > http://metrics.beam.apache.org/ > > > > On Fri, Jun 12, 2020 at 9:57 AM Tyson Hamilton

Subscribe Apache Beam Dev maillist

Subscribe Apache Beam Dev maillist

Re: Python Cross-language wrappers for Java IOs

Right now I'm working on JdbcIO and I'm using Row and Schema protobuffs. I'm figuring out how to use them properly. Thanks for the article - for sure it will be helpful! On 2020/06/12 20:32:16, Brian Hulette wrote: > Thanks! I see there are jiras for SpannerIO and JdbcIO as part of that. Are >

Re: [Discuss] Build Kafka read transform on top of SplittableDoFn

Hi Boyuan, Is the implementation (even if incomplete) open source / available at this moment? Trying to implement here an IO to a custom source here using SplittableDoFn, and it would be helpful to see more examples :) Thanks, Pedro On 2020/05/29 02:16:49, Boyuan Zhang wrote: > Hi team, >

Re: [Discuss] Build Kafka read transform on top of SplittableDoFn

Hi Pedro, Boyuan shared her prototype implementation in [1]. If you're coding a SplittableDoFn, I'd guess the relevant piece of code is ReadViaSDF.java Best -P. [1] https://github.com/apache/beam/pull/11749/files On Mon, Jun 15, 2020 at 10:00 AM Pedro H S Teixeira wrote: > Hi Boyuan, > > Is the

Re: Python Cross-language wrappers for Java IOs

Thanks. +1 for using RowCoder. We should try to use standard coders [1] in the x-lang SDK boundaries. If we use other coders (for example, ProtoCoder) it may or may not work depending on how various runners implement support for x-lang. This might require slightly updating existing transforms or a

Re: [Discuss] Build Kafka read transform on top of SplittableDoFn

Thanks Pablo! Hi Pedro, as Pablo mentioned, the core PTransform is ReadViaSDF, and the core DoFn is ReadFromKafkaDoFn. We also have some other IOs in SDF: HBaseIO

Re: Python Cross-language wrappers for Java IOs

Thanks Cham. Standard coder is a good point. Does it mean non-standard coder doesn't work when crossing language boundaries even if it is implemented in both Java and Python sdk? On Mon, Jun 15, 2020 at 10:08 AM Chamikara Jayalath wrote: > Thanks. +1 for using RowCoder. We should try to use stan

Re: Python Cross-language wrappers for Java IOs

It may or may not work. Some runners may have additional optimizations/rewiring and runners are not aware whether two arbitrary coders implemented in different languages are equivalent or not unless they use the same URN defined in standard coders. It's safe to stick to standard coders. We already

Re: Beam Jenkins Migration

Great! Thank you for working on this and letting us know. > On 12 Jun 2020, at 16:58, Damian Gadomski wrote: > > Hello, > > During the last few days, I was preparing for the Beam Jenkins migration from > builds.apache.org to ci-beam.apache.org >

Re: Python Cross-language wrappers for Java IOs

On Fri, Jun 12, 2020 at 4:12 PM Brian Hulette wrote: > > are unknown fields propagated through if the user only reads/modifies a > row? > I'm not sure I understand this question. Are you asking about handling > schema changes? > The wire format includes the number of fields in the schema, specifi

Re: DynamicMessage in protobufs for re-usable beam pipelines

I don't think I can help with your specific issue, but I can point you to some potentially useful code. +Alex Van Boxel was working on a very similar strategy and added a lot of code for mapping protobufs to Beam schemas which you may be able to take advantage of. He added options to Beam schemas

Re: Python Cross-language wrappers for Java IOs

On Mon, Jun 15, 2020 at 11:12 AM Robert Bradshaw wrote: > On Fri, Jun 12, 2020 at 4:12 PM Brian Hulette wrote: > >> > are unknown fields propagated through if the user only reads/modifies a >> row? >> I'm not sure I understand this question. Are you asking about handling >> schema changes? >> Th

Re: Python Cross-language wrappers for Java IOs

The change should be schema change, mostly adding new fields. On Mon, Jun 15, 2020 at 11:32 AM Brian Hulette wrote: > > > On Mon, Jun 15, 2020 at 11:12 AM Robert Bradshaw > wrote: > >> On Fri, Jun 12, 2020 at 4:12 PM Brian Hulette >> wrote: >> >>> > are unknown fields propagated through if the

Re: Access to Beam issue tracker

Thank you, Pablo :) On 2020/06/12 19:22:05, Pablo Estrada wrote: > Hello Jie!> > Welcome. I've added you as a contributor!> > Best> > -P.> > > On Fri, Jun 12, 2020 at 12:10 PM Jie Fan wrote:> > > > Hi,> > >> > > This is Jie from Google, I started to look into making contributions to> > > the Go

Re: Beam Summit Status Report - 6/10

Great, if worried about publicly archiving things -- it seems that PMC/etc is starting to view this effort even more as a community effort of the project!? That's my hope. As opposed to an external, yet somewhat condoned, effort. That said, have we worked out the parts where such efforts need to

[PROPOSAL] Preparing for Beam 2.23.0 release

Hi all, According to the Beam release calendar [1], the next (2.23.0) release branch cut is scheduled for July 1. I would be happy to help with this release and volunteer myself to be the next release manager. As usual, the plan is to cut the branch on that date, and cherrypick release -blocking

Re: Subscribe Apache Beam Dev maillist

You want dev-subscr...@beam.apache.org On Mon, Jun 15, 2020 at 9:59 AM 核动力潜艇 <554979...@qq.com> wrote: > Subscribe Apache Beam Dev maillist >

Re: [PROPOSAL] Preparing for Beam 2.23.0 release

Sounds good, thanks Valentyn! On Mon, Jun 15, 2020 at 12:31 PM Valentyn Tymofieiev wrote: > Hi all, > > According to the Beam release calendar [1], the next (2.23.0) release > branch cut is scheduled for July 1. > > I would be happy to help with this release and volunteer myself to be the > next

Re: [PROPOSAL] Preparing for Beam 2.23.0 release

Thanks Valentyn! On Mon, Jun 15, 2020 at 12:41 PM Kyle Weaver wrote: > Sounds good, thanks Valentyn! > > On Mon, Jun 15, 2020 at 12:31 PM Valentyn Tymofieiev > wrote: > >> Hi all, >> >> According to the Beam release calendar [1], the next (2.23.0) release >> branch cut is scheduled for July 1.

Re: [PROPOSAL] Preparing for Beam 2.23.0 release

Thank you Valentyn! On Mon, Jun 15, 2020 at 12:46 PM Ankur Goenka wrote: > Thanks Valentyn! > > On Mon, Jun 15, 2020 at 12:41 PM Kyle Weaver wrote: > >> Sounds good, thanks Valentyn! >> >> On Mon, Jun 15, 2020 at 12:31 PM Valentyn Tymofieiev >> wrote: >> >>> Hi all, >>> >>> According to the Be

Re: [PROPOSAL] Preparing for Beam 2.23.0 release

Thank you Valentyn! On Mon, Jun 15, 2020 at 1:08 PM Ahmet Altay wrote: > Thank you Valentyn! > > On Mon, Jun 15, 2020 at 12:46 PM Ankur Goenka wrote: > >> Thanks Valentyn! >> >> On Mon, Jun 15, 2020 at 12:41 PM Kyle Weaver wrote: >> >>> Sounds good, thanks Valentyn! >>> >>> On Mon, Jun 15, 202

Re: python precommit error - google-auth depenedency?

FYI, Beam dependency checker is still complaining currently since google-auth currently sets the restriction to rsa<4.1 for Py2. There are 2 independent changes in flight, which should fix this: https://github.com/googleapis/google-auth-library-python/pull/535, https://github.com/apache/beam/pull/

Re: Beam Jenkins Migration

Happy to see your positive response :) @Udi Meiri, Thanks for pointing that out. I've checked it and indeed it needs some attention. There are two things basing on my research: - data uploaded by performance and load tests by the jobs, directly to the influx DB - that should be handled aut

Re: python precommit error - google-auth depenedency?

Thank you all for working on this. I have a curiosity question. rsa package released 8 versions in the past 5 days after 2 years. Does anyone have visibility into what is happening? Which version is stable? On Mon, Jun 15, 2020 at 1:32 PM Valentyn Tymofieiev wrote: > FYI, Beam dependency checke

Re: Python2.7 Beam End-of-Life Date

Another input here: If you opened a Python PR in the last few days, you probably noticed that our test suites were broken by a transitive dependency of Beam that dropped python 2 support, but did not declare python_requires>=3 in its setup.py [1]. This temporarily broke a subset of Beam Py2 users

Re: python precommit error - google-auth depenedency?

They have been fixing an unclean turndown of Py2 support, there are some details in https://github.com/sybrenstuvel/python-rsa/issues/152. On Mon, Jun 15, 2020 at 1:59 PM Ahmet Altay wrote: > Thank you all for working on this. > > I have a curiosity question. rsa package released 8 versions in t

Re: python precommit error - google-auth depenedency?

Also: https://github.com/sybrenstuvel/python-rsa/blob/master/CHANGELOG.md On Mon, Jun 15, 2020 at 2:10 PM Valentyn Tymofieiev wrote: > They have been fixing an unclean turndown of Py2 support, there are some > details in https://github.com/sybrenstuvel/python-rsa/issues/152. > > On Mon, Jun 15,

Re: Automation for Jira

One thing I noticed is that links being added to issues automatically (e.g. a PR is opened that tags something) doesn't reset the activity counter so things are marked stale even though there are PRs opened for the issue recently. On Thu, Jun 11, 2020 at 10:37 AM Kenneth Knowles wrote: > Yes, my

Re: Python2.7 Beam End-of-Life Date

As a concrete proposal, could we commit to removing python 2 support by 2.24? In other words, mark the next release 2.23 as the last python 2 compatible Beam version. On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev wrote: > Another input here: > > If you opened a Python PR in the last few da

Re: Python2.7 Beam End-of-Life Date

+1 On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay wrote: > As a concrete proposal, could we commit to removing python 2 support by > 2.24? In other words, mark the next release 2.23 as the last python 2 > compatible Beam version. > > On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev > wrote: > >

Re: Automation for Jira

Interesting: you could consider the JIRA as active as long as the linked PRs are open. On Mon, Jun 15, 2020 at 2:28 PM Luke Cwik wrote: > One thing I noticed is that links being added to issues automatically > (e.g. a PR is opened that tags something) doesn't reset the activity > counter so thin