Re: Beam release 2.5.0 tag SNAPSHOT version

[Abdul Qadeer]

Hi Kenneth!

But even the tag v2.5.0 points to SNAPSHOT version for Maven:
https://github.com/apache/beam/blob/v2.5.0/pom.xml#L37

The same file in v2.4.0:
https://github.com/apache/beam/blob/v2.4.0/pom.xml#L37

I am looking to publish v2.5.0 artifacts via maven to a private
artifactory, and without right versions it will publish it as a SNAPSHOT
version. Shall I raise a PR for this?

On Wed, Jul 17, 2019 at 3:41 PM Kenneth Knowles  wrote:

> What you have pointed to is the tip of the release-2.5.0 branch. The
> gradle release plugin copies the maven release plugin. So it has rolled
> back the version change so the branch is always at a snapshot version.
>
> The commit before that is tag v2.5.0 and that is the final tag. Here is
> the gradle properties:
> https://github.com/apache/beam/blob/v2.5.0/gradle.properties
>
> I do believe this should be 2.5.0, not 2.5.0-RC2. But anyhow I think that
> is the commit that was used to build 2.5.0.
>
> Kenn
>
> On Wed, Jul 17, 2019 at 3:36 PM Kenneth Knowles  wrote:
>
>> I take that back - misclicked on 0.5.0 (which has a correct tag).
>>
>> On Wed, Jul 17, 2019 at 3:34 PM Kenneth Knowles  wrote:
>>
>>> Looks like it is this:
>>> https://github.com/apache/beam/tree/4838ae16c172252bc0a15e3a984e085f82e25c2d
>>>
>>> I believe the release manager created the tag to point to the tip of the
>>> release branch after the maven release plugin rolled that change back (this
>>> is how the maven release plugin works since it does not utilize git's
>>> branching model).
>>>
>>> I will fix the tag.
>>>
>>> Kenn
>>>
>>> On Wed, Jul 17, 2019 at 3:32 PM Abdul Qadeer 
>>> wrote:
>>>
 Hi!

 Why is v2.5.0 tag for Beam 2.5.0 release SNAPSHOT version here?
 https://github.com/apache/beam/blob/v2.5.0/pom.xml#L37
 https://github.com/apache/beam/blob/release-2.5.0/gradle.properties#L25

 Please let me know where to find final release 2.5.0 commit.

>>>

Re: Beam release 2.5.0 tag SNAPSHOT version

[Kenneth Knowles]

What you have pointed to is the tip of the release-2.5.0 branch. The gradle
release plugin copies the maven release plugin. So it has rolled back the
version change so the branch is always at a snapshot version.

The commit before that is tag v2.5.0 and that is the final tag. Here is the
gradle properties:
https://github.com/apache/beam/blob/v2.5.0/gradle.properties

I do believe this should be 2.5.0, not 2.5.0-RC2. But anyhow I think that
is the commit that was used to build 2.5.0.

Kenn

On Wed, Jul 17, 2019 at 3:36 PM Kenneth Knowles  wrote:

> I take that back - misclicked on 0.5.0 (which has a correct tag).
>
> On Wed, Jul 17, 2019 at 3:34 PM Kenneth Knowles  wrote:
>
>> Looks like it is this:
>> https://github.com/apache/beam/tree/4838ae16c172252bc0a15e3a984e085f82e25c2d
>>
>> I believe the release manager created the tag to point to the tip of the
>> release branch after the maven release plugin rolled that change back (this
>> is how the maven release plugin works since it does not utilize git's
>> branching model).
>>
>> I will fix the tag.
>>
>> Kenn
>>
>> On Wed, Jul 17, 2019 at 3:32 PM Abdul Qadeer 
>> wrote:
>>
>>> Hi!
>>>
>>> Why is v2.5.0 tag for Beam 2.5.0 release SNAPSHOT version here?
>>> https://github.com/apache/beam/blob/v2.5.0/pom.xml#L37
>>> https://github.com/apache/beam/blob/release-2.5.0/gradle.properties#L25
>>>
>>> Please let me know where to find final release 2.5.0 commit.
>>>
>>

Re: Beam release 2.5.0 tag SNAPSHOT version

[Kenneth Knowles]

I take that back - misclicked on 0.5.0 (which has a correct tag).

On Wed, Jul 17, 2019 at 3:34 PM Kenneth Knowles  wrote:

> Looks like it is this:
> https://github.com/apache/beam/tree/4838ae16c172252bc0a15e3a984e085f82e25c2d
>
> I believe the release manager created the tag to point to the tip of the
> release branch after the maven release plugin rolled that change back (this
> is how the maven release plugin works since it does not utilize git's
> branching model).
>
> I will fix the tag.
>
> Kenn
>
> On Wed, Jul 17, 2019 at 3:32 PM Abdul Qadeer 
> wrote:
>
>> Hi!
>>
>> Why is v2.5.0 tag for Beam 2.5.0 release SNAPSHOT version here?
>> https://github.com/apache/beam/blob/v2.5.0/pom.xml#L37
>> https://github.com/apache/beam/blob/release-2.5.0/gradle.properties#L25
>>
>> Please let me know where to find final release 2.5.0 commit.
>>
>

Re: Beam release 2.5.0 tag SNAPSHOT version

[Kenneth Knowles]

Looks like it is this:
https://github.com/apache/beam/tree/4838ae16c172252bc0a15e3a984e085f82e25c2d

I believe the release manager created the tag to point to the tip of the
release branch after the maven release plugin rolled that change back (this
is how the maven release plugin works since it does not utilize git's
branching model).

I will fix the tag.

Kenn

On Wed, Jul 17, 2019 at 3:32 PM Abdul Qadeer  wrote:

> Hi!
>
> Why is v2.5.0 tag for Beam 2.5.0 release SNAPSHOT version here?
> https://github.com/apache/beam/blob/v2.5.0/pom.xml#L37
> https://github.com/apache/beam/blob/release-2.5.0/gradle.properties#L25
>
> Please let me know where to find final release 2.5.0 commit.
>

Beam release 2.5.0 tag SNAPSHOT version

[Abdul Qadeer]

Hi!

Why is v2.5.0 tag for Beam 2.5.0 release SNAPSHOT version here?
https://github.com/apache/beam/blob/v2.5.0/pom.xml#L37
https://github.com/apache/beam/blob/release-2.5.0/gradle.properties#L25

Please let me know where to find final release 2.5.0 commit.

Re: Live fixing of a Beam bug on July 25 at 3:30pm-4:30pm PST

[sridhar inuog]

Thanks, Pablo! Looking forward to it! Hopefully, it will also be recorded
as well.

On Wed, Jul 17, 2019 at 2:50 PM Pablo Estrada  wrote:

> Yes! So I will be working on a small feature request for Java's
> BigQueryIO: https://issues.apache.org/jira/browse/BEAM-7607
>
> Maybe I'll do something for Python next month. : )
> Best
> -P.
>
> On Wed, Jul 17, 2019 at 12:32 PM Rakesh Kumar 
> wrote:
>
>> +1, I really appreciate this initiative. It would be really helpful
>> newbies like me.
>>
>> Is it possible to list out what are the things that you are planning to
>> cover?
>>
>>
>>
>>
>> On Tue, Jul 16, 2019 at 11:19 AM Yichi Zhang  wrote:
>>
>>> Thanks for organizing this Pablo, it'll be very helpful!
>>>
>>> On Tue, Jul 16, 2019 at 10:57 AM Pablo Estrada 
>>> wrote:
>>>
 Hello all,
 I'll be having a session where I live-fix a Beam bug for 1 hour next
 week. Everyone is invited.

 It will be on July 25, between 3:30pm and 4:30pm PST. Hopefully I will
 finish a full change in that time frame, but we'll see.

 I have not yet decided if I will do this via hangouts, or via a youtube
 livestream. In any case, I will share the link here in the next few days.

 I will most likely work on the Java SDK (I have a little feature
 request in mind).

 Thanks!
 -P.

>>>

Re: Live fixing of a Beam bug on July 25 at 3:30pm-4:30pm PST

[Pablo Estrada]

Yes! So I will be working on a small feature request for Java's BigQueryIO:
https://issues.apache.org/jira/browse/BEAM-7607

Maybe I'll do something for Python next month. : )
Best
-P.

On Wed, Jul 17, 2019 at 12:32 PM Rakesh Kumar  wrote:

> +1, I really appreciate this initiative. It would be really helpful
> newbies like me.
>
> Is it possible to list out what are the things that you are planning to
> cover?
>
>
>
>
> On Tue, Jul 16, 2019 at 11:19 AM Yichi Zhang  wrote:
>
>> Thanks for organizing this Pablo, it'll be very helpful!
>>
>> On Tue, Jul 16, 2019 at 10:57 AM Pablo Estrada 
>> wrote:
>>
>>> Hello all,
>>> I'll be having a session where I live-fix a Beam bug for 1 hour next
>>> week. Everyone is invited.
>>>
>>> It will be on July 25, between 3:30pm and 4:30pm PST. Hopefully I will
>>> finish a full change in that time frame, but we'll see.
>>>
>>> I have not yet decided if I will do this via hangouts, or via a youtube
>>> livestream. In any case, I will share the link here in the next few days.
>>>
>>> I will most likely work on the Java SDK (I have a little feature request
>>> in mind).
>>>
>>> Thanks!
>>> -P.
>>>
>>

Re: pubsub -> IO

[Eugene Kirpichov]

I think full-blown SDF is not needed for this - someone just needs to
implement a MongoDbIO.readAll() variant, using a composite transform. The
regular pattern for this sort of thing will do (ParDo split, reshuffle,
ParDo read).
Whether it's worth replacing MongoDbIO.read() with a redirect to readAll()
is another matter - size estimation, available only in BoundedSource for
now, may or may not be important.

On Wed, Jul 17, 2019 at 2:39 AM Ryan Skraba  wrote:

> Hello!  To clarify, you want to do something like this?
>
> PubSubIO.read() -> extract mongodb collection and range ->
> MongoDbIO.read(collection, range) -> ...
>
> If I'm not mistaken, it isn't possible with the implementation of
> MongoDbIO (based on BoundedSource interface, requiring the collection to be
> specified once at pipeline construction time).
>
> BUT -- this is a good candidate for an improvement in composability, and
> the ongoing work to prefer the SDF for these types of use cases.   Maybe
> raise a JIRA for an improvement?
>
> All my best, Ryan
>
>
> On Wed, Jul 17, 2019 at 9:35 AM Chaim Turkel  wrote:
>
>> any ideas?
>>
>> On Mon, Jul 15, 2019 at 11:04 PM Rui Wang  wrote:
>> >
>> > +user@beam.apache.org
>> >
>> >
>> > -Rui
>> >
>> > On Mon, Jul 15, 2019 at 6:55 AM Chaim Turkel  wrote:
>> >>
>> >> Hi,
>> >>   I am looking to write a pipeline that read from a mongo collection.
>> >>   I would like to listen to a pubsub that will have a object that will
>> >> tell me which collection and which time frame.
>> >>   Is there a way to do this?
>> >>
>> >> Chaim
>> >>
>> >> --
>> >>
>> >>
>> >> Loans are funded by
>> >> FinWise Bank, a Utah-chartered bank located in Sandy,
>> >> Utah, member FDIC, Equal
>> >> Opportunity Lender. Merchant Cash Advances are
>> >> made by Behalf. For more
>> >> information on ECOA, click here
>> >> . For important information about
>> >> opening a new
>> >> account, review Patriot Act procedures here
>> >> .
>> >> Visit Legal
>> >>  to
>> >> review our comprehensive program terms,
>> >> conditions, and disclosures.
>>
>> --
>>
>>
>> Loans are funded by
>> FinWise Bank, a Utah-chartered bank located in Sandy,
>> Utah, member FDIC, Equal
>> Opportunity Lender. Merchant Cash Advances are
>> made by Behalf. For more
>> information on ECOA, click here
>> . For important information about
>> opening a new
>> account, review Patriot Act procedures here
>> .
>> Visit Legal
>>  to
>> review our comprehensive program terms,
>> conditions, and disclosures.
>>
>

Re: pubsub -> IO

[Ryan Skraba]

Hello!  To clarify, you want to do something like this?

PubSubIO.read() -> extract mongodb collection and range ->
MongoDbIO.read(collection, range) -> ...

If I'm not mistaken, it isn't possible with the implementation of MongoDbIO
(based on BoundedSource interface, requiring the collection to be specified
once at pipeline construction time).

BUT -- this is a good candidate for an improvement in composability, and
the ongoing work to prefer the SDF for these types of use cases.   Maybe
raise a JIRA for an improvement?

All my best, Ryan


On Wed, Jul 17, 2019 at 9:35 AM Chaim Turkel  wrote:

> any ideas?
>
> On Mon, Jul 15, 2019 at 11:04 PM Rui Wang  wrote:
> >
> > +user@beam.apache.org
> >
> >
> > -Rui
> >
> > On Mon, Jul 15, 2019 at 6:55 AM Chaim Turkel  wrote:
> >>
> >> Hi,
> >>   I am looking to write a pipeline that read from a mongo collection.
> >>   I would like to listen to a pubsub that will have a object that will
> >> tell me which collection and which time frame.
> >>   Is there a way to do this?
> >>
> >> Chaim
> >>
> >> --
> >>
> >>
> >> Loans are funded by
> >> FinWise Bank, a Utah-chartered bank located in Sandy,
> >> Utah, member FDIC, Equal
> >> Opportunity Lender. Merchant Cash Advances are
> >> made by Behalf. For more
> >> information on ECOA, click here
> >> . For important information about
> >> opening a new
> >> account, review Patriot Act procedures here
> >> .
> >> Visit Legal
> >>  to
> >> review our comprehensive program terms,
> >> conditions, and disclosures.
>
> --
>
>
> Loans are funded by
> FinWise Bank, a Utah-chartered bank located in Sandy,
> Utah, member FDIC, Equal
> Opportunity Lender. Merchant Cash Advances are
> made by Behalf. For more
> information on ECOA, click here
> . For important information about
> opening a new
> account, review Patriot Act procedures here
> .
> Visit Legal
>  to
> review our comprehensive program terms,
> conditions, and disclosures.
>