Understanding watermark

[Cam Mach]

Hello Beam expert,

We have a pipeline (run on Flink) that read both bounded and unbounded
sources and our understanding is that when the bounded sources complete
they should get a watermark of +inf and then we should be able to take a
savepoint and safely restart the pipeline. However, we have source that
never get watermarks and we are confused as to what we are seeing and what
we should expect

Thanks,

Re: [DISCUSS] AWS IOs V1 Deprecation Plan

[Cam Mach]

It's been for a week now, so it's time for us to conclude this discussion.
A summary:
1) We have discussed about removing or keeping the existing V1 IOs. And, we
decided to keep them, but not support and maintain, except for security
reasons
2) We are going to mark the V1 IOs as deprecated and document to encourage
forks to use V2.

Thanks again for your contributions to this thread

Cam Mach



On Fri, Nov 29, 2019 at 4:41 AM Cam Mach  wrote:

> Thanks Alex for the summary, and all for your contributes.
> I have been waiting for a couple of days, and so far we don't have any
> objections. So I guess we can move forward with this approach.
> We can, of course, wait until next week to see if anyone else has any
> ideas, and then make a final decision to close this discussion :-)
>
> Thanks,
> Cam
>
>
>
>
> On Thu, Nov 28, 2019 at 7:47 AM Alexey Romanenko 
> wrote:
>
>> Going back to main subject of this thread, I just wanted to make things
>> clear for all.
>>
>> Seems like that everybody is agree that we will *just deprecate* AWS SDK
>> V1 connectors once the alternative will be available, *don’t remove*
>> them and still *distribute artifacts* [1] with new releases along with
>> artifacts with IOs based on AWS SDK V2 [2].
>>
>> Do we need to vote for this or we can accept it without voting if there
>> are no objections?
>>
>> [1]
>> https://search.maven.org/search?q=a:beam-sdks-java-io-amazon-web-services
>> [2]
>> https://search.maven.org/search?q=a:beam-sdks-java-io-amazon-web-services2
>>
>>
>> On 27 Nov 2019, at 18:44, Kenneth Knowles  wrote:
>>
>>
>> On Tue, Nov 26, 2019 at 7:00 PM Chamikara Jayalath 
>> wrote:
>>
>>>
>>>
>>> On Tue, Nov 26, 2019 at 6:17 PM Reza Rokni  wrote:
>>>
>>>> With regards to @Experimental there are a couple of discussions around
>>>> its usage ( or rather over usage! ) on dev@. It is something that we
>>>> need to clean up ( some of those IO are now being used on production env
>>>> for years!).
>>>>
>>>
>>> I agree that we should move some IO connectors out of experimental state
>>> and probably this should be a separate discussion. Also, this issue is
>>> probably more than for IO connectors since there are other parts of the
>>> code that is marked as experimental as well, sometimes for a good reason
>>> (for example, SDF).
>>>
>>
>> Yes, let's have a separate thread on @Experimental. There are a ton of
>> threads that start talking about it, and they all seem to agree it isn't
>> working. Only one direct thread* that was about something a bit more
>> specific
>> https://lists.apache.org/thread.html/302bd51c77feb5c9ce39882316d391535a0fc92e7608a623d9139160@%3Cdev.beam.apache.org%3E
>>
>>
>>
>>
>>> On Tue, Nov 26, 2019 at 8:21 AM Alexey Romanenko <
>>>>>> aromanenko@gmail.com> wrote:
>>>>>>
>>>>>>> AFAICT, all AWS SDK V1 IOs (SnsIO, SqsIO, DynamoDBIO, KinesisIO) are
>>>>>>> marked as "Experimental". So, it should not be a problem to gracefully
>>>>>>> deprecate and finally remove them. We already did the similar procedure 
>>>>>>> for
>>>>>>> “HadoopInputFormatIO”, which was renamed to just “HadoopFormatIO” 
>>>>>>> (since it
>>>>>>> started to support HadoopOutputFormatI as well). Old 
>>>>>>> “HadoopInputFormatIO”
>>>>>>> was deprecated and removed after *3 consecutive* Beam releases (as
>>>>>>> we agreed on mailing list).
>>>>>>>
>>>>>>> In the same time, some users for some reasons would not be able or
>>>>>>> to want to move on AWS SDK V2. So, I’d prefer to just deprecate AWS SDK 
>>>>>>> V1
>>>>>>> IOs and accept new features/fixes *only* for V2 IOs.
>>>>>>>
>>>>>>
>>> +1 for deprecating AWS V1 IO connectors as opposed to removing as well
>>> unless we can confirm that usage is extremely limited.
>>>
>>
>> +1 to deprecate as soon as there is an alternative.
>>
>> Trying to measure usage is a good idea, but hard. If the maven
>> coordinates were different we could look at download numbers and
>> dependencies.
>>
>>
>> Talking about “Experimental” annotation. Sorry in advance If I missed
>>>>>>> that and switch a subject a bit, but do we have clear rules or an 
&

Re: [DISCUSS] AWS IOs V1 Deprecation Plan

[Cam Mach]

Thanks Alex for the summary, and all for your contributes.
I have been waiting for a couple of days, and so far we don't have any
objections. So I guess we can move forward with this approach.
We can, of course, wait until next week to see if anyone else has any
ideas, and then make a final decision to close this discussion :-)

Thanks,
Cam




On Thu, Nov 28, 2019 at 7:47 AM Alexey Romanenko 
wrote:

> Going back to main subject of this thread, I just wanted to make things
> clear for all.
>
> Seems like that everybody is agree that we will *just deprecate* AWS SDK
> V1 connectors once the alternative will be available, *don’t remove* them
> and still *distribute artifacts* [1] with new releases along with
> artifacts with IOs based on AWS SDK V2 [2].
>
> Do we need to vote for this or we can accept it without voting if there
> are no objections?
>
> [1]
> https://search.maven.org/search?q=a:beam-sdks-java-io-amazon-web-services
> [2]
> https://search.maven.org/search?q=a:beam-sdks-java-io-amazon-web-services2
>
>
> On 27 Nov 2019, at 18:44, Kenneth Knowles  wrote:
>
>
> On Tue, Nov 26, 2019 at 7:00 PM Chamikara Jayalath 
> wrote:
>
>>
>>
>> On Tue, Nov 26, 2019 at 6:17 PM Reza Rokni  wrote:
>>
>>> With regards to @Experimental there are a couple of discussions around
>>> its usage ( or rather over usage! ) on dev@. It is something that we
>>> need to clean up ( some of those IO are now being used on production env
>>> for years!).
>>>
>>
>> I agree that we should move some IO connectors out of experimental state
>> and probably this should be a separate discussion. Also, this issue is
>> probably more than for IO connectors since there are other parts of the
>> code that is marked as experimental as well, sometimes for a good reason
>> (for example, SDF).
>>
>
> Yes, let's have a separate thread on @Experimental. There are a ton of
> threads that start talking about it, and they all seem to agree it isn't
> working. Only one direct thread* that was about something a bit more
> specific
> https://lists.apache.org/thread.html/302bd51c77feb5c9ce39882316d391535a0fc92e7608a623d9139160@%3Cdev.beam.apache.org%3E
>
>
>
>
>> On Tue, Nov 26, 2019 at 8:21 AM Alexey Romanenko <
>>>>> aromanenko@gmail.com> wrote:
>>>>>
>>>>>> AFAICT, all AWS SDK V1 IOs (SnsIO, SqsIO, DynamoDBIO, KinesisIO) are
>>>>>> marked as "Experimental". So, it should not be a problem to gracefully
>>>>>> deprecate and finally remove them. We already did the similar procedure 
>>>>>> for
>>>>>> “HadoopInputFormatIO”, which was renamed to just “HadoopFormatIO” (since 
>>>>>> it
>>>>>> started to support HadoopOutputFormatI as well). Old 
>>>>>> “HadoopInputFormatIO”
>>>>>> was deprecated and removed after *3 consecutive* Beam releases (as
>>>>>> we agreed on mailing list).
>>>>>>
>>>>>> In the same time, some users for some reasons would not be able or to
>>>>>> want to move on AWS SDK V2. So, I’d prefer to just deprecate AWS SDK V1 
>>>>>> IOs
>>>>>> and accept new features/fixes *only* for V2 IOs.
>>>>>>
>>>>>
>> +1 for deprecating AWS V1 IO connectors as opposed to removing as well
>> unless we can confirm that usage is extremely limited.
>>
>
> +1 to deprecate as soon as there is an alternative.
>
> Trying to measure usage is a good idea, but hard. If the maven coordinates
> were different we could look at download numbers and dependencies.
>
>
> Talking about “Experimental” annotation. Sorry in advance If I missed that
>>>>>> and switch a subject a bit, but do we have clear rules or an agreement 
>>>>>> when
>>>>>> IO becomes stable and should not be marked as experimental anymore?
>>>>>> *Most* of our Java IOs are marked as Experimental but many of them
>>>>>> were using in production by real users under real load. Does it mean that
>>>>>> they are ready to be stable in terms of API? Perhaps, this topic 
>>>>>> deserves a
>>>>>> new discussion if there are several opinions on that.
>>>>>>
>>>>>
>> Probably, decision to move component APIs (for example, an IO connector)
>> out of experimental state should be done on a case-by-case basis.
>>
>
> Let's repeat these good points on a dedicated thread.
>
> Kenn
>
>
>
>>

Re: [DISCUSS] AWS IOs V1 Deprecation Plan

[Cam Mach]

Thank you, Alex for sharing the information, and Luke for the questions.
I like the idea that just depreciate the V1 IOs, and just maintain V2 IOs,
so we can support whoever want continue with V1.
Just as Alex said, a lot of users, including my teams :-) , use the V1 IOs
in production for real workload. So it'll be hard to remove V1 IOs or force
them migrate to V2. But let hear if there are any other ideas?

Btw, making V1 a wrapper around V2 is not very positive, code will get more
complicated since V2 API is very different from V1's.

Thanks,



On Tue, Nov 26, 2019 at 8:21 AM Alexey Romanenko 
wrote:

> AFAICT, all AWS SDK V1 IOs (SnsIO, SqsIO, DynamoDBIO, KinesisIO) are
> marked as "Experimental". So, it should not be a problem to gracefully
> deprecate and finally remove them. We already did the similar procedure for
> “HadoopInputFormatIO”, which was renamed to just “HadoopFormatIO” (since it
> started to support HadoopOutputFormatI as well). Old “HadoopInputFormatIO”
> was deprecated and removed after *3 consecutive* Beam releases (as we
> agreed on mailing list).
>
> In the same time, some users for some reasons would not be able or to want
> to move on AWS SDK V2. So, I’d prefer to just deprecate AWS SDK V1 IOs and
> accept new features/fixes *only* for V2 IOs.
>
> Talking about “Experimental” annotation. Sorry in advance If I missed that
> and switch a subject a bit, but do we have clear rules or an agreement when
> IO becomes stable and should not be marked as experimental anymore? *Most*
> of our Java IOs are marked as Experimental but many of them were using in
> production by real users under real load. Does it mean that they are ready
> to be stable in terms of API? Perhaps, this topic deserves a new discussion
> if there are several opinions on that.
>
> On 26 Nov 2019, at 00:39, Luke Cwik  wrote:
>
> Phase I sounds fine.
>
> Apache Beam follows semantic versioning and I believe removing the IOs
> will be a backwards incompatible change unless they were marked
> experimental which will be a problem for Phase 2.
>
> What is the feasibility of making the V1 transforms wrappers around V2?
>
> On Mon, Nov 25, 2019 at 1:46 PM Cam Mach  wrote:
>
>> Hello Beam Devs,
>>
>> I have been working on the migration of Amazon Web Services IO connectors
>> into the new AWS SDK for Java V2. The goal is to have an updated
>> implementation aligned with the most recent AWS improvements. So far we
>> have already migrated the connectors for AWS SNS, SQS and  DynamoDB.
>>
>> In the meantime some contributions are still going on V1 IOs. So far we
>> have dealt with those by porting (or asking contributors) to port the
>> changes into V2 IOs too because we don’t want features of both versions to
>> be unaligned but this may quickly become a maintenance issue, so we want to
>> discuss a plan to stop supporting (deprecate) V1 IOs and encourage users to
>> move to V2.
>>
>> Phase I (ASAP):
>>
>>- Mark migrated AWS V1 IOs as deprecated
>>- Document migration path to V2
>>
>> Phase II (end of 2020):
>>
>>- Decide a date or Beam release to remove the V1 IOs
>>- Send a notification to the community 3 months before we remove them
>>- Completely get rid of V1 IOs
>>
>>
>> Please let me know what you think or if you see any potential issues?
>>
>> Thanks,
>> Cam Mach
>>
>>
>

[DISCUSS] AWS IOs V1 Deprecation Plan

[Cam Mach]

Hello Beam Devs,

I have been working on the migration of Amazon Web Services IO connectors
into the new AWS SDK for Java V2. The goal is to have an updated
implementation aligned with the most recent AWS improvements. So far we
have already migrated the connectors for AWS SNS, SQS and  DynamoDB.

In the meantime some contributions are still going on V1 IOs. So far we
have dealt with those by porting (or asking contributors) to port the
changes into V2 IOs too because we don’t want features of both versions to
be unaligned but this may quickly become a maintenance issue, so we want to
discuss a plan to stop supporting (deprecate) V1 IOs and encourage users to
move to V2.

Phase I (ASAP):

   -

   Mark migrated AWS V1 IOs as deprecated
   -

   Document migration path to V2

Phase II (end of 2020):

   -

   Decide a date or Beam release to remove the V1 IOs
   -

   Send a notification to the community 3 months before we remove them
   -

   Completely get rid of V1 IOs


Please let me know what you think or if you see any potential issues?

Thanks,
Cam Mach

Re: DynamoDBIO related issue

[Cam Mach]

Hi Pradeep,

What is your enhancement? Can you create a ticket and describe it?

Thanks,
Cam



On Thu, Oct 24, 2019 at 1:58 PM Pradeep Bhosale <
bhosale.pradeep1...@gmail.com> wrote:

> Hi,
>
> This is Pradeep. I am using DynamoDB IO to write data to dynamo DB.
> I would like to report one enhancement.
>
> Please let me know how can I achieve that.
> I don't have *create issue* access on beam JIRA.
>
> https://issues.apache.org/jira/projects/BEAM/issues/BEAM-8368?filter=allopenissues
>
>

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

[Cam Mach]

+1



On Tue, Oct 1, 2019 at 9:44 AM Udi Meiri  wrote:

> +1
>
> On Tue, Oct 1, 2019 at 3:22 AM Łukasz Gajowy 
> wrote:
>
>> +1
>>
>> wt., 1 paź 2019 o 11:29 Maximilian Michels  napisał(a):
>>
>>> +1
>>>
>>> On 30.09.19 23:03, Reza Rokni wrote:
>>> > +1
>>> >
>>> > On Tue, 1 Oct 2019 at 13:54, Tanay Tummalapalli >> > > wrote:
>>> >
>>> > +1
>>> >
>>> > On Tue, Oct 1, 2019 at 8:19 AM Suneel Marthi >> > > wrote:
>>> >
>>> > +1
>>> >
>>> > On Mon, Sep 30, 2019 at 10:33 PM Manu Zhang
>>> > mailto:owenzhang1...@gmail.com>>
>>> wrote:
>>> >
>>> > +1
>>> >
>>> > On Tue, Oct 1, 2019 at 9:44 AM Austin Bennett
>>> > >> > > wrote:
>>> >
>>> > +1
>>> >
>>> > On Mon, Sep 30, 2019 at 5:22 PM Valentyn Tymofieiev
>>> > mailto:valen...@google.com>>
>>> wrote:
>>> >
>>> > Hi everyone,
>>> >
>>> > Please vote whether to sign a pledge on behalf of
>>> > Apache Beam to sunset Beam Python 2 offering (in
>>> new
>>> > releases) in 2020 on http://python3stament.org as
>>> > follows:
>>> >
>>> > [ ] +1: Sign a pledge to discontinue support of
>>> > Python 2 in Beam in 2020.
>>> > [ ] -1: Do not sign a pledge to discontinue support
>>> > of Python 2 in Beam in 2020.
>>> >
>>> > The motivation and details for this vote were
>>> > discussed in [1, 2]. Please follow up in [2] if you
>>> > have any questions.
>>> >
>>> > This is a procedural vote [3] that will follow the
>>> > majority approval rules and will be open for at
>>> > least 72 hours.
>>> >
>>> > Thanks,
>>> > Valentyn
>>> >
>>> > [1]
>>> >
>>> https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
>>> > [2]
>>> >
>>> https://lists.apache.org/thread.html/456631fe1a696c537ef8ebfee42cd3ea8121bf7c639c52da5f7032e7@%3Cdev.beam.apache.org%3E
>>> > [3] https://www.apache.org/foundation/voting.html
>>> >
>>> >
>>> >
>>> > --
>>> >
>>> > This email may be confidential and privileged. If you received this
>>> > communication by mistake, please don't forward it to anyone else,
>>> please
>>> > erase all copies and attachments, and please let me know that it has
>>> > gone to the wrong person.
>>> >
>>> > The above terms reflect a potential business arrangement, are provided
>>> > solely as a basis for further discussion, and are not intended to be
>>> and
>>> > do not constitute a legally binding obligation. No legally binding
>>> > obligations will be created, implied, or inferred until an agreement
>>> in
>>> > final form is executed in writing by all parties involved.
>>> >
>>>
>>

Beam KinesisIO Migration V1 to V2

[Cam Mach]

Hello Beam Dev,

I have discussed with a couple of Beam dev regarding this topic. We found
something interesting in the new AWS Kinesis SDK and Libraries V2, so like
to propose a design for this migration.

Here is the design doc:
https://docs.google.com/document/d/1XeIVbiDHBReZY8rEI2OWA3cTEQuaR7RPdwGAup6S1DM


I would love to hear from you, your feedback and comments

Thanks,
Cam

Re: How to run DynamoDBIOTest?

[Cam Mach]

Hi Anton,

It should not be a machine issue, since you have the container got started.
I would say give it around 5 - 10 minutes, it would throw out connection
time out error log. This usually happens when the integration tests can't
connect to the container. As you may notice, in the tests we don't hard
code the container IP and port, and we get it dynamically from the
TestContainer framework. Which mean, for some reason, the framework
couldn't allocate IP and port for the container. At this moment I don't
have a quick fix, but if it happens to several dev, we can file a Jira to
their dashboard, or get rid of it.

P.S. try this gradlew `./gradlew :sdks:java:io:amazon-web-services:test`

Thanks,



On Sat, Jul 20, 2019 at 6:33 AM Elliotte Rusty Harold 
wrote:

> I've seen this myself trying to run tests in Linux (Goobuntu).
>
> On Fri, Jul 19, 2019 at 1:15 PM Anton Kedin  wrote:
> >
> > Hi dev@,
> >
> > Does anyone know if there's anything extra needed to run
> `DynamoDBIOTest`? If I do `./graldew
> :sdks:java:io:amazon-web-services:build --debug` it passes few tests during
> `:test` but then seems to sit on `DynamoDBIOTest` forever. No errors, last
> meaningful log is `INFO: Container localstack/localstack:0.8.6 started`.
> Happens on different machines, both on master and release-2.14.0 branches.
> >
> > Any pointers?
> >
> > Regards,
> > Anton
>
>
>
> --
> Elliotte Rusty Harold
> elh...@ibiblio.org
>