Re: Java Jenkins tests not running

[Tyson Hamilton]

Yes, there is a backup of ~14 builds in queue.

On Fri, Aug 7, 2020 at 10:54 AM Andrew Pilloud  wrote:

> Now it is going, just slow to start. There must be a backlog built up from
> the outage.
>
> On Fri, Aug 7, 2020 at 10:44 AM Andrew Pilloud 
> wrote:
>
>> It appears Jenkins is not working again. It is not running on the PR to
>> fix build breaks introduced while it was down:
>> https://github.com/apache/beam/pull/12500
>>
>> On Fri, Aug 7, 2020 at 10:29 AM Luke Cwik  wrote:
>>
>>> I can confirm that it is working again.
>>>
>>> On Fri, Aug 7, 2020 at 4:43 AM Damian Gadomski <
>>> damian.gadom...@polidea.com> wrote:
>>>
 Hey,

 We knew about the issue. All Jenkins builds were not being triggered
 from the pull requests. There's a ticket for that if you're curious about
 details: INF 
 RA-20649
 
 Seems that we've just fixed it. I've also retriggered the tests from
 your PR.

 Regards,
 Damian


 On Fri, Aug 7, 2020 at 9:11 AM Reuven Lax  wrote:

> Does anyone know why Jenkins is not triggering any Java tests for
> pr/12474? It is only triggering python tasks, which is odd considering 
> that
> this PR doesn't touch any python files.
>
> Reuven
>

Re: Java Jenkins tests not running

[Andrew Pilloud]

Now it is going, just slow to start. There must be a backlog built up from
the outage.

On Fri, Aug 7, 2020 at 10:44 AM Andrew Pilloud  wrote:

> It appears Jenkins is not working again. It is not running on the PR to
> fix build breaks introduced while it was down:
> https://github.com/apache/beam/pull/12500
>
> On Fri, Aug 7, 2020 at 10:29 AM Luke Cwik  wrote:
>
>> I can confirm that it is working again.
>>
>> On Fri, Aug 7, 2020 at 4:43 AM Damian Gadomski <
>> damian.gadom...@polidea.com> wrote:
>>
>>> Hey,
>>>
>>> We knew about the issue. All Jenkins builds were not being triggered
>>> from the pull requests. There's a ticket for that if you're curious about
>>> details: INF RA-20649
>>> 
>>> Seems that we've just fixed it. I've also retriggered the tests from
>>> your PR.
>>>
>>> Regards,
>>> Damian
>>>
>>>
>>> On Fri, Aug 7, 2020 at 9:11 AM Reuven Lax  wrote:
>>>
 Does anyone know why Jenkins is not triggering any Java tests for
 pr/12474? It is only triggering python tasks, which is odd considering that
 this PR doesn't touch any python files.

 Reuven

>>>

Re: Java Jenkins tests not running

[Andrew Pilloud]

It appears Jenkins is not working again. It is not running on the PR to fix
build breaks introduced while it was down:
https://github.com/apache/beam/pull/12500

On Fri, Aug 7, 2020 at 10:29 AM Luke Cwik  wrote:

> I can confirm that it is working again.
>
> On Fri, Aug 7, 2020 at 4:43 AM Damian Gadomski <
> damian.gadom...@polidea.com> wrote:
>
>> Hey,
>>
>> We knew about the issue. All Jenkins builds were not being triggered from
>> the pull requests. There's a ticket for that if you're curious about
>> details: INF RA-20649
>> 
>> Seems that we've just fixed it. I've also retriggered the tests from your
>> PR.
>>
>> Regards,
>> Damian
>>
>>
>> On Fri, Aug 7, 2020 at 9:11 AM Reuven Lax  wrote:
>>
>>> Does anyone know why Jenkins is not triggering any Java tests for
>>> pr/12474? It is only triggering python tasks, which is odd considering that
>>> this PR doesn't touch any python files.
>>>
>>> Reuven
>>>
>>

Re: Java Jenkins tests not running

[Luke Cwik]

I can confirm that it is working again.

On Fri, Aug 7, 2020 at 4:43 AM Damian Gadomski 
wrote:

> Hey,
>
> We knew about the issue. All Jenkins builds were not being triggered from
> the pull requests. There's a ticket for that if you're curious about
> details: INF RA-20649
> 
> Seems that we've just fixed it. I've also retriggered the tests from your
> PR.
>
> Regards,
> Damian
>
>
> On Fri, Aug 7, 2020 at 9:11 AM Reuven Lax  wrote:
>
>> Does anyone know why Jenkins is not triggering any Java tests for
>> pr/12474? It is only triggering python tasks, which is odd considering that
>> this PR doesn't touch any python files.
>>
>> Reuven
>>
>

Re: Stateful Pardo Question

[Reza Ardeshir Rokni]

Hi,

One possible approach ( have not tried it out, so might be missing cases..)
but you can reset the timer from within the OnTimer code.

So maybe you start the timer on the onprocess to go off at
current+requiredGap. Then OnTimer, you check the list of elements and
output a session if nothing new. Then reset the timer to go off either at
latestTimestampValue+requiredGap if there was new elements or at
currentEventTime+requiredGap. If a timer fires and there are no elements in
the bag then you don't rest.

You will need to keep state to know you have a timer firing so as not to
set it again in OnProcess as there is no read() for timers.

Also we don't have a sorted map state, so you will take a performance hit
as you will need to keep sorting the events every OnTimer...

Cheers
Reza


On Fri, 7 Aug 2020 at 14:57, Reuven Lax  wrote:

>
>
> On Tue, Aug 4, 2020 at 1:08 PM jmac...@godaddy.com 
> wrote:
>
>> So, after some additional digging, it appears that Beam does not
>> consistently check for timer expiry before calling process. The result is
>> that it may be the case that the watermark has moved beyond your timer
>> expiry, and if youre counting on the timer callback happening at the time
>> you set it for, that simply may NOT have happened when you are in
>> DoFn.process(). You can “fix” the behavior by simply checking the watermark
>> manually in process() and doing what you would normally do for timestamp
>> exipry before proceeding. See my latest updated code reproducing the issue
>> and showing the fix at  https://github.com/randomsamples/pardo_repro.
>>
>>
>>
>> I would argue that users of this API will naturally expect that timer
>> callback semantics will guarantee that when they are in process(), if the
>> current watermark is past a timers expiry that the timer callback in
>> question will have been called. Is there any reason why this isn’t
>> happening? Am I misunderstanding something?
>>
>
> Timers do not expire synchronously with the watermark advancing. So if you
> have a timer set for 12pm and the watermark advances past 12pm, that timer
> is now eligible to fire, but might not fire immediately. Some other
> elements may process before that timer fires.
>
> There are multiple reasons for this, but one is that Beam does not
> guarantee that watermark advancement is synchronous with element
> processing. The watermark might advance suddenly while in the middle
> processing an element, or at any other time. This makes it impossible (or
> at least, exceedingly difficult) to really provide the guarantee you
> expected.
>
> Reuven
>
>>
>>
>> *From: *"jmac...@godaddy.com" 
>> *Reply-To: *"dev@beam.apache.org" 
>> *Date: *Monday, August 3, 2020 at 10:51 AM
>> *To: *"dev@beam.apache.org" 
>> *Subject: *Re: Stateful Pardo Question
>>
>>
>>
>> Notice: This email is from an external sender.
>>
>>
>>
>> Yeah, unless I am misunderstanding something. The output from my repro
>> code shows event timestamp and the context timestamp every time we process
>> an event.
>>
>> Receiving event at: 2000-01-01T00:00:00.000Z
>>
>> Resetting timer to : 2000-01-01T00:15:00.000Z
>>
>> Receiving event at: 2000-01-01T00:05:00.000Z
>>
>> Resetting timer to : 2000-01-01T00:20:00.000Z ß Shouldn’t the timer have
>> fired before we processed the next event?
>>
>> Receiving event at: 2000-01-01T00:40:00.000Z
>>
>> Why didnt the timer fire?
>>
>> Resetting timer to : 2000-01-01T00:55:00.000Z
>>
>> Receiving event at: 2000-01-01T00:45:00.000Z
>>
>> Resetting timer to : 2000-01-01T01:00:00.000Z
>>
>> Receiving event at: 2000-01-01T00:50:00.000Z
>>
>> Resetting timer to : 2000-01-01T01:05:00.000Z
>>
>> Timer firing at: 2000-01-01T01:05:00.000Z
>>
>>
>>
>> *From: *Reuven Lax 
>> *Reply-To: *"dev@beam.apache.org" 
>> *Date: *Monday, August 3, 2020 at 10:02 AM
>> *To: *dev 
>> *Subject: *Re: Stateful Pardo Question
>>
>>
>>
>> Notice: This email is from an external sender.
>>
>>
>>
>> Are you sure that there is a 15 minute gap in your data?
>>
>>
>>
>> On Mon, Aug 3, 2020 at 6:20 AM jmac...@godaddy.com 
>> wrote:
>>
>> I am confused about the behavior of timers on a simple stateful pardo. I
>> have put together a little repro here:
>> https://github.com/randomsamples/pardo_repro
>>
>>
>>
>> I basically want to build something like a session window, accumulating
>> events until quiescence of the stream for a given key and gap time, then
>> output results. But it appears that the timer is not firing when the
>> watermark is passed it expiration time, so the event stream is not being
>> split as I would have expected. Would love some help getting this work, the
>> behavior is for a project I’m working on.
>>
>>

Re: Failing Python builds & AppEngine application

[Maximilian Michels]

Thanks for letting us know, Tyson!

I think in terms of data, everything is being reported to the new 
InfluxDb instance now. Historic data is missing though, which would be 
nice to have. Also, the graphs are still not 100% on par with the old 
application.


Cheers,
Max

On 06.08.20 16:34, Tyson Hamilton wrote:
It was me! I disabled the App in the premise that it only hosted the old 
perf graphs that were replaced with Grafana. Thanks for fixing the issue.


Is there anything else on the app, or is there more migration to Grafana 
required, or cleanup unfinished?



On Thu, Aug 6, 2020, 4:30 AM Damian Gadomski 
mailto:damian.gadom...@polidea.com>> wrote:


Hey,

A strange thing happened a few hours ago. All python builds (e.g.
[1]) started failing because of:

google.api_core.exceptions.NotFound: 404 The project
apache-beam-testing does not exist or it does not contain an active
Cloud Datastore or Cloud Firestore database. Please visit
http://console.cloud.google.com to create a project or
https://console.cloud.google.com/datastore/setup?project=apache-beam-testing
to add a Cloud Datastore or Cloud Firestore database. Note that
Cloud Datastore or Cloud Firestore always have an associated App
Engine app and this app must not be disabled.

I've checked that manually and the same error appeared while
accessing [2]. Seems that we are using Cloud Datastore and indeed
there was a default AppEngine application [3] that was disabled and
therefore Datastore was inactive. I've just enabled back this app
and the Datastore became active again. Hopefully, that will fix the
builds. Basing on the app statistics it seems that someone disabled
it around Aug 5, 21:00 UTC.

I saw the discussion on the devlist recently about the performance
monitoring. The app [3] is also serving the metrics on [4].
CC +Maximilian Michels , +Kamil Wasilewski
 - as you were involved in the
discussion there regarding [4]. Perhaps you know something more
about this app or at least who may know? :)

[1] https://ci-beam.apache.org/job/beam_PostCommit_Python37/2681/console
[2]
https://console.cloud.google.com/datastore?project=apache-beam-testing
[3]

https://console.cloud.google.com/appengine?project=apache-beam-testing&serviceId=default
[4] https://apache-beam-testing.appspot.com

Regards,
Damian

Re: Broken links in code velocity dashboard

[Damian Gadomski]

Unfortunately, I'm not aware of any recent changes.

On Thu, Aug 6, 2020 at 10:00 PM Ahmet Altay  wrote:

> Damian, or anyone else, do you know if there were other changes to the
> dashboard?
>
> I started to see closed PRs in the currently open PRs list (e.g.
> https://github.com/apache/beam/pull/12349,
> https://github.com/apache/beam/pull/12374). Not sure what is causing it,
> but it seems like a new issue.
>
> On Fri, Jul 31, 2020 at 10:13 AM Ahmet Altay  wrote:
>
>> Looks fixed. Thank you for the quick response!
>>
>> On Fri, Jul 31, 2020 at 7:05 AM Damian Gadomski <
>> damian.gadom...@polidea.com> wrote:
>>
>>> Oops. Sorry about that. Everything should be fixed now. URLs are correct
>>> and I've also removed wrong entries from the DB.
>>>
>>> If you're curious, you were right, it was mistakenly deployed from my
>>> fork. Actually, my private Jenkins instance did it. Will be more cautious
>>> with the jobs.
>>>
>>> Regards,
>>> Damian
>>>
>>> On Fri, Jul 31, 2020 at 1:32 AM Ahmet Altay  wrote:
>>>
 Currently the open PRs section of the dashboard [1] seems to be broken.
 PRs are linking to https://github.com/damgadbot/beam/issues/>>> NUMBER> instead of https://github.com/apache/beam/pull/.
 And the PR list showing PRs from the (https://github.com/damgadbot/beam)
 fork in addition to the main repo.

 Dashboard was working normally last week, so probably something changed
 recently. I do not see a code change in the dashboard [2]. I am not sure
 what happened, maybe the dashboard was deployed from a fork?

 +Damian Gadomski  - based on the url
 changes :)

 Thank you,
 Ahmet

 [1]
 http://metrics.beam.apache.org/d/code_velocity/code-velocity?orgId=1
 [2]
 https://github.com/apache/beam/blob/master/.test-infra/metrics/grafana/dashboards/code_velocity.json

>>>

Re: Java Jenkins tests not running

[Damian Gadomski]

Hey,

We knew about the issue. All Jenkins builds were not being triggered from
the pull requests. There's a ticket for that if you're curious about
details: INF RA-20649

Seems that we've just fixed it. I've also retriggered the tests from your
PR.

Regards,
Damian


On Fri, Aug 7, 2020 at 9:11 AM Reuven Lax  wrote:

> Does anyone know why Jenkins is not triggering any Java tests for
> pr/12474? It is only triggering python tasks, which is odd considering that
> this PR doesn't touch any python files.
>
> Reuven
>

Java Jenkins tests not running

[Reuven Lax]

Does anyone know why Jenkins is not triggering any Java tests for pr/12474?
It is only triggering python tasks, which is odd considering that this PR
doesn't touch any python files.

Reuven