Re: Live fixing of a Beam bug on July 25 at 3:30pm-4:30pm PST
Hello all, This will be streamed on youtube on this link: https://www.youtube.com/watch?v=xpIpEO4PUDo I think there will be a live chat, so I will hopefully be available to answer questions. To be honest, my workflow is not super efficient, but... oh well, hopefully it will be at least somewhat helpful to others : ) Best -P. On Thu, Jul 18, 2019 at 12:59 AM Tim Sell wrote: > +1, I'd love to see this as a recording. Will you stick it up on youtube > afterwards? > > On Thu, Jul 18, 2019 at 4:00 AM sridhar inuog > wrote: > >> Thanks, Pablo! Looking forward to it! Hopefully, it will also be recorded >> as well. >> >> On Wed, Jul 17, 2019 at 2:50 PM Pablo Estrada wrote: >> >>> Yes! So I will be working on a small feature request for Java's >>> BigQueryIO: https://issues.apache.org/jira/browse/BEAM-7607 >>> >>> Maybe I'll do something for Python next month. : ) >>> Best >>> -P. >>> >>> On Wed, Jul 17, 2019 at 12:32 PM Rakesh Kumar >>> wrote: >>> +1, I really appreciate this initiative. It would be really helpful newbies like me. Is it possible to list out what are the things that you are planning to cover? On Tue, Jul 16, 2019 at 11:19 AM Yichi Zhang wrote: > Thanks for organizing this Pablo, it'll be very helpful! > > On Tue, Jul 16, 2019 at 10:57 AM Pablo Estrada > wrote: > >> Hello all, >> I'll be having a session where I live-fix a Beam bug for 1 hour next >> week. Everyone is invited. >> >> It will be on July 25, between 3:30pm and 4:30pm PST. Hopefully I >> will finish a full change in that time frame, but we'll see. >> >> I have not yet decided if I will do this via hangouts, or via a >> youtube livestream. In any case, I will share the link here in the next >> few >> days. >> >> I will most likely work on the Java SDK (I have a little feature >> request in mind). >> >> Thanks! >> -P. >> >
Re: Beam release 2.5.0 tag SNAPSHOT version
Thank you Kenneth! I thought Maven was removed starting 2.6.0. I will change my build tool to Gradle. On Fri, 19 Jul 2019 at 12:13, Kenneth Knowles wrote: > Good catch. > > The release 2.5.0 was built with gradle, so that pom is left over. The > gradle release plugin does not edit poms, so it did not change that. > Instead, the pom is generated and you can find them on maven central like > https://repo1.maven.org/maven2/org/apache/beam/beam-runners-direct-java/2.5.0/beam-runners-direct-java-2.5.0.pom. > Today > the pom has been removed and it is generated at run time. > > Since you are digging into details, you may also be interested in the RC > vote and verification thread: > https://lists.apache.org/thread.html/dac093378fb27331c5d9d86a3bd03397f7e186d482f0fc8c8ef88feb@%3Cdev.beam.apache.org%3E > > To answer your original question, I do believe the tag v2.5.0 and > v2.5.0-RC2 point to the commit where the released artifacts were built. > > Kenn > > On Wed, Jul 17, 2019 at 10:54 PM Abdul Qadeer > wrote: > >> Hi Kenneth! >> >> But even the tag v2.5.0 points to SNAPSHOT version for Maven: >> https://github.com/apache/beam/blob/v2.5.0/pom.xml#L37 >> >> The same file in v2.4.0: >> https://github.com/apache/beam/blob/v2.4.0/pom.xml#L37 >> >> I am looking to publish v2.5.0 artifacts via maven to a private >> artifactory, and without right versions it will publish it as a SNAPSHOT >> version. Shall I raise a PR for this? >> >> On Wed, Jul 17, 2019 at 3:41 PM Kenneth Knowles wrote: >> >>> What you have pointed to is the tip of the release-2.5.0 branch. The >>> gradle release plugin copies the maven release plugin. So it has rolled >>> back the version change so the branch is always at a snapshot version. >>> >>> The commit before that is tag v2.5.0 and that is the final tag. Here is >>> the gradle properties: >>> https://github.com/apache/beam/blob/v2.5.0/gradle.properties >>> >>> I do believe this should be 2.5.0, not 2.5.0-RC2. But anyhow I think >>> that is the commit that was used to build 2.5.0. >>> >>> Kenn >>> >>> On Wed, Jul 17, 2019 at 3:36 PM Kenneth Knowles wrote: >>> I take that back - misclicked on 0.5.0 (which has a correct tag). On Wed, Jul 17, 2019 at 3:34 PM Kenneth Knowles wrote: > Looks like it is this: > https://github.com/apache/beam/tree/4838ae16c172252bc0a15e3a984e085f82e25c2d > > I believe the release manager created the tag to point to the tip of > the release branch after the maven release plugin rolled that change back > (this is how the maven release plugin works since it does not utilize > git's > branching model). > > I will fix the tag. > > Kenn > > On Wed, Jul 17, 2019 at 3:32 PM Abdul Qadeer > wrote: > >> Hi! >> >> Why is v2.5.0 tag for Beam 2.5.0 release SNAPSHOT version here? >> https://github.com/apache/beam/blob/v2.5.0/pom.xml#L37 >> >> https://github.com/apache/beam/blob/release-2.5.0/gradle.properties#L25 >> >> Please let me know where to find final release 2.5.0 commit. >> >
Portability framework: multiple environments in one pipeline
Hi all, I'm interested to know if others on the list would find value in the ability to use multiple environments (e.g. docker images) within a single pipeline, using some mechanism to identify the environment(s) that a transform should use. It would be quite useful for us, since our transforms can have conflicting python requirements, or worse, conflicting interpreter requirements. Currently to solve this we have to break the pipeline up into multiple pipelines and use pubsub to communicate between them, which is not ideal. -chad
