[jira] [Closed] (BEAM-3839) Get Java Mobile-Gaming Running on Spark, Flink, Apex
[ https://issues.apache.org/jira/browse/BEAM-3839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yifan zou closed BEAM-3839. --- Resolution: Won't Fix Fix Version/s: Not applicable > Get Java Mobile-Gaming Running on Spark, Flink, Apex > > > Key: BEAM-3839 > URL: https://issues.apache.org/jira/browse/BEAM-3839 > Project: Beam > Issue Type: Sub-task > Components: examples-java >Affects Versions: 2.5.0 >Reporter: yifan zou >Assignee: yifan zou >Priority: Major > Fix For: Not applicable > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-3838) Beam Release Automation
[ https://issues.apache.org/jira/browse/BEAM-3838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yifan zou resolved BEAM-3838. - Resolution: Fixed Fix Version/s: Not applicable > Beam Release Automation > --- > > Key: BEAM-3838 > URL: https://issues.apache.org/jira/browse/BEAM-3838 > Project: Beam > Issue Type: Task > Components: examples-java, examples-python, testing >Reporter: yifan zou >Assignee: yifan zou >Priority: Major > Fix For: Not applicable > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (BEAM-6358) Python PostCommit WordCountIT failed on assertion error
[ https://issues.apache.org/jira/browse/BEAM-6358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yifan zou closed BEAM-6358. --- Resolution: Cannot Reproduce Fix Version/s: Not applicable > Python PostCommit WordCountIT failed on assertion error > --- > > Key: BEAM-6358 > URL: https://issues.apache.org/jira/browse/BEAM-6358 > Project: Beam > Issue Type: Test > Components: test-failures >Reporter: Boyuan Zhang >Assignee: yifan zou >Priority: Major > Fix For: Not applicable > > > [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6975/] > The pipeline succeeded but verifier failed to find the output file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6358) Python PostCommit WordCountIT failed on assertion error
[ https://issues.apache.org/jira/browse/BEAM-6358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743711#comment-16743711 ] yifan zou commented on BEAM-6358: - Didn't happen again. Closing for now. Reopen it if needed. > Python PostCommit WordCountIT failed on assertion error > --- > > Key: BEAM-6358 > URL: https://issues.apache.org/jira/browse/BEAM-6358 > Project: Beam > Issue Type: Test > Components: test-failures >Reporter: Boyuan Zhang >Assignee: yifan zou >Priority: Major > > [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6975/] > The pipeline succeeded but verifier failed to find the output file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas
[ https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=185613=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185613 ] ASF GitHub Bot logged work on BEAM-4461: Author: ASF GitHub Bot Created on: 16/Jan/19 06:35 Start Date: 16/Jan/19 06:35 Worklog Time Spent: 10m Work Description: reuvenlax commented on issue #7353: [BEAM-4461] Support inner and outer style joins in CoGroup. URL: https://github.com/apache/beam/pull/7353#issuecomment-454668666 @akedin any comments on this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185613) Time Spent: 19h 10m (was: 19h) > Create a library of useful transforms that use schemas > -- > > Key: BEAM-4461 > URL: https://issues.apache.org/jira/browse/BEAM-4461 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Reuven Lax >Assignee: Reuven Lax >Priority: Major > Time Spent: 19h 10m > Remaining Estimate: 0h > > e.g. JoinBy(fields). Project, Filter, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6184) PortableRunner dependency missed in wordcount example maven artifact
[ https://issues.apache.org/jira/browse/BEAM-6184?focusedWorklogId=185600=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185600 ] ASF GitHub Bot logged work on BEAM-6184: Author: ASF GitHub Bot Created on: 16/Jan/19 05:36 Start Date: 16/Jan/19 05:36 Worklog Time Spent: 10m Work Description: HuangLED commented on issue #7532: [BEAM-6184]Turn on javadocmethod checkstyle to report error. URL: https://github.com/apache/beam/pull/7532#issuecomment-454658261 Run Java_Examples_Dataflow PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185600) Time Spent: 4h 20m (was: 4h 10m) > PortableRunner dependency missed in wordcount example maven artifact > > > Key: BEAM-6184 > URL: https://issues.apache.org/jira/browse/BEAM-6184 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Ruoyun Huang >Assignee: Ruoyun Huang >Priority: Minor > Fix For: Not applicable > > Time Spent: 4h 20m > Remaining Estimate: 0h > > > > more context: > https://lists.apache.org/thread.html/8dd60395424425f7502d62888c49014430d1d3b06c026606f3db28ab@%3Cuser.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[
https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yifan zou reassigned BEAM-5811:
---
Assignee: (was: Ahmet Altay)
> Timeout in Python datastore_write_it_test.DatastoreWriteIT
> --
>
> Key: BEAM-5811
> URL: https://issues.apache.org/jira/browse/BEAM-5811
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Kenneth Knowles
>Priority: Critical
> Labels: flake
> Fix For: 2.11.0
>
>
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/]
> [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests]
> {code:java}
> TimedOutException: 'test_datastore_write_limit
> (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6447) Spotless paddedCell appears to cause unpredictable behavior
[ https://issues.apache.org/jira/browse/BEAM-6447?focusedWorklogId=185596=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185596 ] ASF GitHub Bot logged work on BEAM-6447: Author: ASF GitHub Bot Created on: 16/Jan/19 05:15 Start Date: 16/Jan/19 05:15 Worklog Time Spent: 10m Work Description: reuvenlax commented on issue #7531: [BEAM-6447] Turn off spotless paddedCell URL: https://github.com/apache/beam/pull/7531#issuecomment-454654623 Run Java_Examples_Dataflow PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185596) Time Spent: 1h 10m (was: 1h) > Spotless paddedCell appears to cause unpredictable behavior > --- > > Key: BEAM-6447 > URL: https://issues.apache.org/jira/browse/BEAM-6447 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > In https://github.com/apache/beam/pull/7505 spotlessCheck passed even though > spotlessApply was not a no-op. > In https://github.com/apache/beam/pull/7523 and > https://github.com/apache/beam/pull/7527 spotlessApply was run and the result > _also_ passed spotlessCheck. > Confirmed that spotlessCheck fails on > https://github.com/apache/beam/pull/7505 if paddedCell is turned off. The use > of paddedCell is to workaround bugs in the underlying formatter (in our case > google-java-format). Currently, there is no known bug affecting us, so we > should not be using paddedCell. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[
https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743619#comment-16743619
]
yifan zou commented on BEAM-5811:
-
released the bug with unassigned.
> Timeout in Python datastore_write_it_test.DatastoreWriteIT
> --
>
> Key: BEAM-5811
> URL: https://issues.apache.org/jira/browse/BEAM-5811
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Kenneth Knowles
>Priority: Critical
> Labels: flake
> Fix For: 2.11.0
>
>
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/]
> [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests]
> {code:java}
> TimedOutException: 'test_datastore_write_limit
> (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[
https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yifan zou reassigned BEAM-5811:
---
Assignee: Ahmet Altay (was: yifan zou)
> Timeout in Python datastore_write_it_test.DatastoreWriteIT
> --
>
> Key: BEAM-5811
> URL: https://issues.apache.org/jira/browse/BEAM-5811
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Kenneth Knowles
>Assignee: Ahmet Altay
>Priority: Critical
> Labels: flake
> Fix For: 2.11.0
>
>
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/]
> [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests]
> {code:java}
> TimedOutException: 'test_datastore_write_limit
> (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6447) Spotless paddedCell appears to cause unpredictable behavior
[ https://issues.apache.org/jira/browse/BEAM-6447?focusedWorklogId=185582=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185582 ] ASF GitHub Bot logged work on BEAM-6447: Author: ASF GitHub Bot Created on: 16/Jan/19 05:00 Start Date: 16/Jan/19 05:00 Worklog Time Spent: 10m Work Description: HuangLED commented on issue #7531: [BEAM-6447] Turn off spotless paddedCell URL: https://github.com/apache/beam/pull/7531#issuecomment-454652568 LGTM. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185582) Time Spent: 0.5h (was: 20m) > Spotless paddedCell appears to cause unpredictable behavior > --- > > Key: BEAM-6447 > URL: https://issues.apache.org/jira/browse/BEAM-6447 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > In https://github.com/apache/beam/pull/7505 spotlessCheck passed even though > spotlessApply was not a no-op. > In https://github.com/apache/beam/pull/7523 and > https://github.com/apache/beam/pull/7527 spotlessApply was run and the result > _also_ passed spotlessCheck. > Confirmed that spotlessCheck fails on > https://github.com/apache/beam/pull/7505 if paddedCell is turned off. The use > of paddedCell is to workaround bugs in the underlying formatter (in our case > google-java-format). Currently, there is no known bug affecting us, so we > should not be using paddedCell. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6184) PortableRunner dependency missed in wordcount example maven artifact
[ https://issues.apache.org/jira/browse/BEAM-6184?focusedWorklogId=185590=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185590 ] ASF GitHub Bot logged work on BEAM-6184: Author: ASF GitHub Bot Created on: 16/Jan/19 05:04 Start Date: 16/Jan/19 05:04 Worklog Time Spent: 10m Work Description: HuangLED commented on issue #7532: [BEAM-6184]Turn on javadocmethod checkstyle to report error. URL: https://github.com/apache/beam/pull/7532#issuecomment-454653125 Run Java_Examples_Dataflow PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185590) Time Spent: 4h 10m (was: 4h) > PortableRunner dependency missed in wordcount example maven artifact > > > Key: BEAM-6184 > URL: https://issues.apache.org/jira/browse/BEAM-6184 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Ruoyun Huang >Assignee: Ruoyun Huang >Priority: Minor > Fix For: Not applicable > > Time Spent: 4h 10m > Remaining Estimate: 0h > > > > more context: > https://lists.apache.org/thread.html/8dd60395424425f7502d62888c49014430d1d3b06c026606f3db28ab@%3Cuser.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6447) Spotless paddedCell appears to cause unpredictable behavior
[ https://issues.apache.org/jira/browse/BEAM-6447?focusedWorklogId=185589=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185589 ] ASF GitHub Bot logged work on BEAM-6447: Author: ASF GitHub Bot Created on: 16/Jan/19 05:03 Start Date: 16/Jan/19 05:03 Worklog Time Spent: 10m Work Description: reuvenlax commented on issue #7531: [BEAM-6447] Turn off spotless paddedCell URL: https://github.com/apache/beam/pull/7531#issuecomment-454652892 Run Java_Examples_Dataflow PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185589) Time Spent: 1h (was: 50m) > Spotless paddedCell appears to cause unpredictable behavior > --- > > Key: BEAM-6447 > URL: https://issues.apache.org/jira/browse/BEAM-6447 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > In https://github.com/apache/beam/pull/7505 spotlessCheck passed even though > spotlessApply was not a no-op. > In https://github.com/apache/beam/pull/7523 and > https://github.com/apache/beam/pull/7527 spotlessApply was run and the result > _also_ passed spotlessCheck. > Confirmed that spotlessCheck fails on > https://github.com/apache/beam/pull/7505 if paddedCell is turned off. The use > of paddedCell is to workaround bugs in the underlying formatter (in our case > google-java-format). Currently, there is no known bug affecting us, so we > should not be using paddedCell. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4184) S3ResourceIdTest has had a masked failure
[ https://issues.apache.org/jira/browse/BEAM-4184?focusedWorklogId=185579=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185579 ] ASF GitHub Bot logged work on BEAM-4184: Author: ASF GitHub Bot Created on: 16/Jan/19 04:54 Start Date: 16/Jan/19 04:54 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #7414: [BEAM-4184] s3 resource id test has had a masked failure URL: https://github.com/apache/beam/pull/7414#issuecomment-454651513 Wait again - per https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html as of March 2018 underscores are not allowed. So keep the test and reject it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185579) Time Spent: 50m (was: 40m) > S3ResourceIdTest has had a masked failure > - > > Key: BEAM-4184 > URL: https://issues.apache.org/jira/browse/BEAM-4184 > Project: Beam > Issue Type: Bug > Components: io-java-aws >Reporter: Kenneth Knowles >Assignee: Michael Luckey >Priority: Major > Labels: sickbay > Fix For: 2.11.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Sickbayed in https://github.com/apache/beam/pull/5161, the test should be > fixed and no longer ignored. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6447) Spotless paddedCell appears to cause unpredictable behavior
[ https://issues.apache.org/jira/browse/BEAM-6447?focusedWorklogId=185585=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185585 ] ASF GitHub Bot logged work on BEAM-6447: Author: ASF GitHub Bot Created on: 16/Jan/19 05:02 Start Date: 16/Jan/19 05:02 Worklog Time Spent: 10m Work Description: reuvenlax commented on issue #7531: [BEAM-6447] Turn off spotless paddedCell URL: https://github.com/apache/beam/pull/7531#issuecomment-454652788 Run Java_Examples_Dataflow PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185585) Time Spent: 40m (was: 0.5h) > Spotless paddedCell appears to cause unpredictable behavior > --- > > Key: BEAM-6447 > URL: https://issues.apache.org/jira/browse/BEAM-6447 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > In https://github.com/apache/beam/pull/7505 spotlessCheck passed even though > spotlessApply was not a no-op. > In https://github.com/apache/beam/pull/7523 and > https://github.com/apache/beam/pull/7527 spotlessApply was run and the result > _also_ passed spotlessCheck. > Confirmed that spotlessCheck fails on > https://github.com/apache/beam/pull/7505 if paddedCell is turned off. The use > of paddedCell is to workaround bugs in the underlying formatter (in our case > google-java-format). Currently, there is no known bug affecting us, so we > should not be using paddedCell. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6339) In certain cases, spotlessJava fails to work
[
https://issues.apache.org/jira/browse/BEAM-6339?focusedWorklogId=185586=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185586
]
ASF GitHub Bot logged work on BEAM-6339:
Author: ASF GitHub Bot
Created on: 16/Jan/19 05:02
Start Date: 16/Jan/19 05:02
Worklog Time Spent: 10m
Work Description: HuangLED commented on issue #7390: [BEAM-6339] Add
paddedcell fix to spotlessJava rules.
URL: https://github.com/apache/beam/pull/7390#issuecomment-454652796
Saw you comment in the other thread. Got it. Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 185586)
Time Spent: 1h (was: 50m)
> In certain cases, spotlessJava fails to work
>
>
> Key: BEAM-6339
> URL: https://issues.apache.org/jira/browse/BEAM-6339
> Project: Beam
> Issue Type: Improvement
> Components: build-system
>Reporter: Ruoyun Huang
>Assignee: Ruoyun Huang
>Priority: Trivial
> Fix For: Not applicable
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Encounter following error when certain criteria exists in new code:
>
> > Task :beam-runners-google-cloud-dataflow-java:spotlessJava FAILED
>
> FAILURE: Build failed with an exception.
> * What went wrong:
> Execution failed for task
> ':beam-runners-google-cloud-dataflow-java:spotlessJava'.
> > You have a misbehaving rule which can't make up its mind.
> This means that spotlessCheck will fail even after spotlessApply has run.
>
> This is a bug in a formatting rule, not Spotless itself, but Spotless can
> work around this bug and generate helpful bug reports for the broken rule
> if you add 'paddedCell()' to your build.gradle as such:
>
> spotless {
> format 'someFormat', {
> ...
> paddedCell()
> }
> }
>
> The next time you run spotlessCheck, it will put helpful bug reports into
> 'runners/google-cloud-dataflow-java/build/spotless-diagnose-java', and
> spotlessApply
> and spotlessCheck will be self-consistent from here on out.
>
> For details see
> [https://github.com/diffplug/spotless/blob/master/PADDEDCELL.md]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6447) Spotless paddedCell appears to cause unpredictable behavior
[ https://issues.apache.org/jira/browse/BEAM-6447?focusedWorklogId=185587=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185587 ] ASF GitHub Bot logged work on BEAM-6447: Author: ASF GitHub Bot Created on: 16/Jan/19 05:02 Start Date: 16/Jan/19 05:02 Worklog Time Spent: 10m Work Description: reuvenlax commented on issue #7531: [BEAM-6447] Turn off spotless paddedCell URL: https://github.com/apache/beam/pull/7531#issuecomment-454652838 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185587) Time Spent: 50m (was: 40m) > Spotless paddedCell appears to cause unpredictable behavior > --- > > Key: BEAM-6447 > URL: https://issues.apache.org/jira/browse/BEAM-6447 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > In https://github.com/apache/beam/pull/7505 spotlessCheck passed even though > spotlessApply was not a no-op. > In https://github.com/apache/beam/pull/7523 and > https://github.com/apache/beam/pull/7527 spotlessApply was run and the result > _also_ passed spotlessCheck. > Confirmed that spotlessCheck fails on > https://github.com/apache/beam/pull/7505 if paddedCell is turned off. The use > of paddedCell is to workaround bugs in the underlying formatter (in our case > google-java-format). Currently, there is no known bug affecting us, so we > should not be using paddedCell. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6339) In certain cases, spotlessJava fails to work
[
https://issues.apache.org/jira/browse/BEAM-6339?focusedWorklogId=185581=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185581
]
ASF GitHub Bot logged work on BEAM-6339:
Author: ASF GitHub Bot
Created on: 16/Jan/19 04:59
Start Date: 16/Jan/19 04:59
Worklog Time Spent: 10m
Work Description: HuangLED commented on issue #7390: [BEAM-6339] Add
paddedcell fix to spotlessJava rules.
URL: https://github.com/apache/beam/pull/7390#issuecomment-454652391
> What was the actual issue? It looks like paddedCell has bugs, itself. It
seems that we are OK at current head.
oops... The story was: I was testing whether our tools were able to catch
code style violations as they claim. I thus intentionally created different
violations, for verification purpose.
The issue happened with those intentional changes. That is, instead of
reporting those actual, although intentional, violations, the build system was
failing. I didn't keep my changes, thus will need to check which exactly
violation (something related to class level javadoc).
Back then, I thought it would be an improvement, which is even though minor,
but good to have, thus requested this PR since I was already on it.
Is this causing issue? Do you prefer us to revert this PR?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 185581)
Time Spent: 50m (was: 40m)
> In certain cases, spotlessJava fails to work
>
>
> Key: BEAM-6339
> URL: https://issues.apache.org/jira/browse/BEAM-6339
> Project: Beam
> Issue Type: Improvement
> Components: build-system
>Reporter: Ruoyun Huang
>Assignee: Ruoyun Huang
>Priority: Trivial
> Fix For: Not applicable
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Encounter following error when certain criteria exists in new code:
>
> > Task :beam-runners-google-cloud-dataflow-java:spotlessJava FAILED
>
> FAILURE: Build failed with an exception.
> * What went wrong:
> Execution failed for task
> ':beam-runners-google-cloud-dataflow-java:spotlessJava'.
> > You have a misbehaving rule which can't make up its mind.
> This means that spotlessCheck will fail even after spotlessApply has run.
>
> This is a bug in a formatting rule, not Spotless itself, but Spotless can
> work around this bug and generate helpful bug reports for the broken rule
> if you add 'paddedCell()' to your build.gradle as such:
>
> spotless {
> format 'someFormat', {
> ...
> paddedCell()
> }
> }
>
> The next time you run spotlessCheck, it will put helpful bug reports into
> 'runners/google-cloud-dataflow-java/build/spotless-diagnose-java', and
> spotlessApply
> and spotlessCheck will be self-consistent from here on out.
>
> For details see
> [https://github.com/diffplug/spotless/blob/master/PADDEDCELL.md]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4184) S3ResourceIdTest has had a masked failure
[
https://issues.apache.org/jira/browse/BEAM-4184?focusedWorklogId=185577=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185577
]
ASF GitHub Bot logged work on BEAM-4184:
Author: ASF GitHub Bot
Created on: 16/Jan/19 04:51
Start Date: 16/Jan/19 04:51
Worklog Time Spent: 10m
Work Description: kennknowles commented on issue #7414: [BEAM-4184] s3
resource id test has had a masked failure
URL: https://github.com/apache/beam/pull/7414#issuecomment-454651090
I think the `checkArgument` for a slash in the component name makes sense,
if that is something that is really expected to fail. But can they not be
escaped? To me if you have `fromComponents("invalid/", "something")` that
should resolve to `"invalid\//something"`. This is what we get for reinventing
URLs...
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 185577)
Time Spent: 0.5h (was: 20m)
> S3ResourceIdTest has had a masked failure
> -
>
> Key: BEAM-4184
> URL: https://issues.apache.org/jira/browse/BEAM-4184
> Project: Beam
> Issue Type: Bug
> Components: io-java-aws
>Reporter: Kenneth Knowles
>Assignee: Michael Luckey
>Priority: Major
> Labels: sickbay
> Fix For: 2.11.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Sickbayed in https://github.com/apache/beam/pull/5161, the test should be
> fixed and no longer ignored.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6447) Spotless paddedCell appears to cause unpredictable behavior
[ https://issues.apache.org/jira/browse/BEAM-6447?focusedWorklogId=185580=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185580 ] ASF GitHub Bot logged work on BEAM-6447: Author: ASF GitHub Bot Created on: 16/Jan/19 04:56 Start Date: 16/Jan/19 04:56 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #7531: [BEAM-6447] Turn off spotless paddedCell URL: https://github.com/apache/beam/pull/7531#issuecomment-454651839 R: @HuangLED @swegner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185580) Time Spent: 20m (was: 10m) > Spotless paddedCell appears to cause unpredictable behavior > --- > > Key: BEAM-6447 > URL: https://issues.apache.org/jira/browse/BEAM-6447 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > In https://github.com/apache/beam/pull/7505 spotlessCheck passed even though > spotlessApply was not a no-op. > In https://github.com/apache/beam/pull/7523 and > https://github.com/apache/beam/pull/7527 spotlessApply was run and the result > _also_ passed spotlessCheck. > Confirmed that spotlessCheck fails on > https://github.com/apache/beam/pull/7505 if paddedCell is turned off. The use > of paddedCell is to workaround bugs in the underlying formatter (in our case > google-java-format). Currently, there is no known bug affecting us, so we > should not be using paddedCell. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4184) S3ResourceIdTest has had a masked failure
[ https://issues.apache.org/jira/browse/BEAM-4184?focusedWorklogId=185578=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185578 ] ASF GitHub Bot logged work on BEAM-4184: Author: ASF GitHub Bot Created on: 16/Jan/19 04:53 Start Date: 16/Jan/19 04:53 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #7414: [BEAM-4184] s3 resource id test has had a masked failure URL: https://github.com/apache/beam/pull/7414#issuecomment-454651396 Ah, but I forget that this is not actually a hierarchically structured URL, but simply a schema:scheme-specific-part URL. So yes I think just adding the logic that rejects slash is fine and then deleting the spurious test about underscore. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185578) Time Spent: 40m (was: 0.5h) > S3ResourceIdTest has had a masked failure > - > > Key: BEAM-4184 > URL: https://issues.apache.org/jira/browse/BEAM-4184 > Project: Beam > Issue Type: Bug > Components: io-java-aws >Reporter: Kenneth Knowles >Assignee: Michael Luckey >Priority: Major > Labels: sickbay > Fix For: 2.11.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Sickbayed in https://github.com/apache/beam/pull/5161, the test should be > fixed and no longer ignored. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4184) S3ResourceIdTest has had a masked failure
[ https://issues.apache.org/jira/browse/BEAM-4184?focusedWorklogId=185576=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185576 ] ASF GitHub Bot logged work on BEAM-4184: Author: ASF GitHub Bot Created on: 16/Jan/19 04:48 Start Date: 16/Jan/19 04:48 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #7414: [BEAM-4184] s3 resource id test has had a masked failure URL: https://github.com/apache/beam/pull/7414#issuecomment-454650731 Ah, sorry I missed this! Reviewing now. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185576) Time Spent: 20m (was: 10m) > S3ResourceIdTest has had a masked failure > - > > Key: BEAM-4184 > URL: https://issues.apache.org/jira/browse/BEAM-4184 > Project: Beam > Issue Type: Bug > Components: io-java-aws >Reporter: Kenneth Knowles >Assignee: Michael Luckey >Priority: Major > Labels: sickbay > Fix For: 2.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Sickbayed in https://github.com/apache/beam/pull/5161, the test should be > fixed and no longer ignored. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6184) PortableRunner dependency missed in wordcount example maven artifact
[ https://issues.apache.org/jira/browse/BEAM-6184?focusedWorklogId=185574=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185574 ] ASF GitHub Bot logged work on BEAM-6184: Author: ASF GitHub Bot Created on: 16/Jan/19 04:39 Start Date: 16/Jan/19 04:39 Worklog Time Spent: 10m Work Description: HuangLED commented on pull request #7532: [BEAM-6184]Turn on javadocmethod checkstyle to report error. URL: https://github.com/apache/beam/pull/7532 1. Turn on javadocmethod checkstyle (reports error) 2. Added suppressions to all the violations. Created another beam-6446 to track the future progress of removing suppressions. Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking ---
[jira] [Commented] (BEAM-6339) In certain cases, spotlessJava fails to work
[
https://issues.apache.org/jira/browse/BEAM-6339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743606#comment-16743606
]
Kenneth Knowles commented on BEAM-6339:
---
Do you have a commit hash or example of the code that causes the problem?
> In certain cases, spotlessJava fails to work
>
>
> Key: BEAM-6339
> URL: https://issues.apache.org/jira/browse/BEAM-6339
> Project: Beam
> Issue Type: Improvement
> Components: build-system
>Reporter: Ruoyun Huang
>Assignee: Ruoyun Huang
>Priority: Trivial
> Fix For: Not applicable
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Encounter following error when certain criteria exists in new code:
>
> > Task :beam-runners-google-cloud-dataflow-java:spotlessJava FAILED
>
> FAILURE: Build failed with an exception.
> * What went wrong:
> Execution failed for task
> ':beam-runners-google-cloud-dataflow-java:spotlessJava'.
> > You have a misbehaving rule which can't make up its mind.
> This means that spotlessCheck will fail even after spotlessApply has run.
>
> This is a bug in a formatting rule, not Spotless itself, but Spotless can
> work around this bug and generate helpful bug reports for the broken rule
> if you add 'paddedCell()' to your build.gradle as such:
>
> spotless {
> format 'someFormat', {
> ...
> paddedCell()
> }
> }
>
> The next time you run spotlessCheck, it will put helpful bug reports into
> 'runners/google-cloud-dataflow-java/build/spotless-diagnose-java', and
> spotlessApply
> and spotlessCheck will be self-consistent from here on out.
>
> For details see
> [https://github.com/diffplug/spotless/blob/master/PADDEDCELL.md]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-6447) Spotless paddedCell appears to cause unpredictable behavior
Kenneth Knowles created BEAM-6447: - Summary: Spotless paddedCell appears to cause unpredictable behavior Key: BEAM-6447 URL: https://issues.apache.org/jira/browse/BEAM-6447 Project: Beam Issue Type: Bug Components: build-system Reporter: Kenneth Knowles Assignee: Kenneth Knowles In https://github.com/apache/beam/pull/7505 spotlessCheck passed even though spotlessApply was not a no-op. In https://github.com/apache/beam/pull/7523 and https://github.com/apache/beam/pull/7527 spotlessApply was run and the result _also_ passed spotlessCheck. Confirmed that spotlessCheck fails on https://github.com/apache/beam/pull/7505 if paddedCell is turned off. The use of paddedCell is to workaround bugs in the underlying formatter (in our case google-java-format). Currently, there is no known bug affecting us, so we should not be using paddedCell. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6447) Spotless paddedCell appears to cause unpredictable behavior
[ https://issues.apache.org/jira/browse/BEAM-6447?focusedWorklogId=185563=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185563 ] ASF GitHub Bot logged work on BEAM-6447: Author: ASF GitHub Bot Created on: 16/Jan/19 04:31 Start Date: 16/Jan/19 04:31 Worklog Time Spent: 10m Work Description: kennknowles commented on pull request #7531: [BEAM-6447] Turn off spotless paddedCell URL: https://github.com/apache/beam/pull/7531 Our use of paddedCell seems to have triggered some strange spotlessBehavior that is probably a bug in spotless. But we don't need it, so turn it off. It is a workaround only if `google-java-format` has a bug. Follow this checklist to help us incorporate your contribution quickly and easily: - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking
[jira] [Created] (BEAM-6446) Clean up suppression rules in checkstyle suppressions.xml
Ruoyun Huang created BEAM-6446: -- Summary: Clean up suppression rules in checkstyle suppressions.xml Key: BEAM-6446 URL: https://issues.apache.org/jira/browse/BEAM-6446 Project: Beam Issue Type: Improvement Components: build-system Reporter: Ruoyun Huang Assignee: Ruoyun Huang When violations are addressed, clean up suppression rules in checkstyle suppressions.xml -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-6440) FlinkTimerInternals memory leak
[ https://issues.apache.org/jira/browse/BEAM-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-6440: --- Issue Type: Bug (was: Improvement) > FlinkTimerInternals memory leak > --- > > Key: BEAM-6440 > URL: https://issues.apache.org/jira/browse/BEAM-6440 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.9.0 >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > Attachments: image-2019-01-15-10-31-43-618.png, > image-2019-01-15-10-41-08-200.png > > Time Spent: 1h 40m > Remaining Estimate: 0h > > After running a portable streaming app for 2-3 days, we see heap space > exhausted. Memory analysis shows large number of TimerData objects referenced > by heap state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6440) FlinkTimerInternals memory leak
[ https://issues.apache.org/jira/browse/BEAM-6440?focusedWorklogId=185559=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185559 ] ASF GitHub Bot logged work on BEAM-6440: Author: ASF GitHub Bot Created on: 16/Jan/19 03:47 Start Date: 16/Jan/19 03:47 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #7530: [BEAM-6440] Fix leakage of timer de-duplication map URL: https://github.com/apache/beam/pull/7530#issuecomment-454641836 Looks like trouble in the precommits This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185559) Time Spent: 1h 40m (was: 1.5h) > FlinkTimerInternals memory leak > --- > > Key: BEAM-6440 > URL: https://issues.apache.org/jira/browse/BEAM-6440 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Affects Versions: 2.9.0 >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > Attachments: image-2019-01-15-10-31-43-618.png, > image-2019-01-15-10-41-08-200.png > > Time Spent: 1h 40m > Remaining Estimate: 0h > > After running a portable streaming app for 2-3 days, we see heap space > exhausted. Memory analysis shows large number of TimerData objects referenced > by heap state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6439) Move Python Flink VR tests to PreCommit
[
https://issues.apache.org/jira/browse/BEAM-6439?focusedWorklogId=185558=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185558
]
ASF GitHub Bot logged work on BEAM-6439:
Author: ASF GitHub Bot
Created on: 16/Jan/19 03:45
Start Date: 16/Jan/19 03:45
Worklog Time Spent: 10m
Work Description: kennknowles commented on issue #7514: [BEAM-6439] Move
Python Validates Runner Flink test to PreCommit.
URL: https://github.com/apache/beam/pull/7514#issuecomment-454641436
I can clean up old disabled jobs but it would be better to have
`previousNames('beam_PostCommit_Python_VR_Flink')`. This only applies to the
`PreCommit_Cron` variant.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 185558)
Time Spent: 40m (was: 0.5h)
> Move Python Flink VR tests to PreCommit
> ---
>
> Key: BEAM-6439
> URL: https://issues.apache.org/jira/browse/BEAM-6439
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink, testing
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Fix For: Not applicable
>
> Attachments: png.png
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Now that they're fast and stable, it would be good to catch changes that
> break this earlier.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-6428) Allow textual selection syntax for schema fields
[ https://issues.apache.org/jira/browse/BEAM-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-6428: - Assignee: Reuven Lax (was: Kenneth Knowles) > Allow textual selection syntax for schema fields > > > Key: BEAM-6428 > URL: https://issues.apache.org/jira/browse/BEAM-6428 > Project: Beam > Issue Type: Sub-task > Components: beam-model >Affects Versions: 2.9.0 >Reporter: Reuven Lax >Assignee: Reuven Lax >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4076) Schema followups
[ https://issues.apache.org/jira/browse/BEAM-4076?focusedWorklogId=185552=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185552 ] ASF GitHub Bot logged work on BEAM-4076: Author: ASF GitHub Bot Created on: 16/Jan/19 02:53 Start Date: 16/Jan/19 02:53 Worklog Time Spent: 10m Work Description: reuvenlax commented on pull request #7500: [BEAM-4076] Add antlr4 to beam URL: https://github.com/apache/beam/pull/7500 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185552) Time Spent: 18h (was: 17h 50m) > Schema followups > > > Key: BEAM-4076 > URL: https://issues.apache.org/jira/browse/BEAM-4076 > Project: Beam > Issue Type: Improvement > Components: beam-model, dsl-sql, sdk-java-core >Reporter: Kenneth Knowles >Priority: Major > Time Spent: 18h > Remaining Estimate: 0h > > This umbrella bug contains subtasks with followups for Beam schemas, which > were moved from SQL to the core Java SDK and made to be type-name-based > rather than coder based. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6439) Move Python Flink VR tests to PreCommit
[ https://issues.apache.org/jira/browse/BEAM-6439?focusedWorklogId=185551=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185551 ] ASF GitHub Bot logged work on BEAM-6439: Author: ASF GitHub Bot Created on: 16/Jan/19 02:40 Start Date: 16/Jan/19 02:40 Worklog Time Spent: 10m Work Description: tweise commented on issue #7514: [BEAM-6439] Move Python Validates Runner Flink test to PreCommit. URL: https://github.com/apache/beam/pull/7514#issuecomment-454630173 The pre-commit coverage was already provided via https://builds.apache.org/job/beam_PreCommit_Portable_Python_Commit/ As we expect to expand test coverage, is it really necessary to jam all of this into pre-commit? Also, this job was part of the post-commit display and now shows as disabled.. https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185551) Time Spent: 0.5h (was: 20m) > Move Python Flink VR tests to PreCommit > --- > > Key: BEAM-6439 > URL: https://issues.apache.org/jira/browse/BEAM-6439 > Project: Beam > Issue Type: New Feature > Components: runner-flink, testing >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: Not applicable > > Attachments: png.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > Now that they're fast and stable, it would be good to catch changes that > break this earlier. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6440) FlinkTimerInternals memory leak
[ https://issues.apache.org/jira/browse/BEAM-6440?focusedWorklogId=185550=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185550 ] ASF GitHub Bot logged work on BEAM-6440: Author: ASF GitHub Bot Created on: 16/Jan/19 02:28 Start Date: 16/Jan/19 02:28 Worklog Time Spent: 10m Work Description: tweise commented on pull request #7530: [BEAM-6440] Fix leakage of timer de-duplication map URL: https://github.com/apache/beam/pull/7530 The FlinkStateInternals use a keyed map of pending timers to make up for Flink's limitation to only be able to delete timers with their original timestamp, not via timer id. The Map leaked memory because subclasses of DoFnOperator overwrote `fireTimer` which was responsible for performing cleanup in the map upon firing a timer. **Please** add a meaningful description for your change here Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the
[jira] [Commented] (BEAM-6440) FlinkTimerInternals memory leak
[ https://issues.apache.org/jira/browse/BEAM-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743562#comment-16743562 ] Thomas Weise commented on BEAM-6440: Verified changes with our application and heap utilization is stable now. Thanks for the fix! > FlinkTimerInternals memory leak > --- > > Key: BEAM-6440 > URL: https://issues.apache.org/jira/browse/BEAM-6440 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Affects Versions: 2.9.0 >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > Attachments: image-2019-01-15-10-31-43-618.png, > image-2019-01-15-10-41-08-200.png > > Time Spent: 1h 20m > Remaining Estimate: 0h > > After running a portable streaming app for 2-3 days, we see heap space > exhausted. Memory analysis shows large number of TimerData objects referenced > by heap state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6440) FlinkTimerInternals memory leak
[ https://issues.apache.org/jira/browse/BEAM-6440?focusedWorklogId=185549=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185549 ] ASF GitHub Bot logged work on BEAM-6440: Author: ASF GitHub Bot Created on: 16/Jan/19 02:21 Start Date: 16/Jan/19 02:21 Worklog Time Spent: 10m Work Description: tweise commented on pull request #7517: [BEAM-6440] Fix leakage of timer de-duplication map URL: https://github.com/apache/beam/pull/7517 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185549) Time Spent: 1h 20m (was: 1h 10m) > FlinkTimerInternals memory leak > --- > > Key: BEAM-6440 > URL: https://issues.apache.org/jira/browse/BEAM-6440 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Affects Versions: 2.9.0 >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > Attachments: image-2019-01-15-10-31-43-618.png, > image-2019-01-15-10-41-08-200.png > > Time Spent: 1h 20m > Remaining Estimate: 0h > > After running a portable streaming app for 2-3 days, we see heap space > exhausted. Memory analysis shows large number of TimerData objects referenced > by heap state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-6440) FlinkTimerInternals memory leak
[ https://issues.apache.org/jira/browse/BEAM-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-6440: --- Fix Version/s: 2.10.0 > FlinkTimerInternals memory leak > --- > > Key: BEAM-6440 > URL: https://issues.apache.org/jira/browse/BEAM-6440 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Affects Versions: 2.9.0 >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > Attachments: image-2019-01-15-10-31-43-618.png, > image-2019-01-15-10-41-08-200.png > > Time Spent: 1h 20m > Remaining Estimate: 0h > > After running a portable streaming app for 2-3 days, we see heap space > exhausted. Memory analysis shows large number of TimerData objects referenced > by heap state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6440) FlinkTimerInternals memory leak
[ https://issues.apache.org/jira/browse/BEAM-6440?focusedWorklogId=185548=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185548 ] ASF GitHub Bot logged work on BEAM-6440: Author: ASF GitHub Bot Created on: 16/Jan/19 02:20 Start Date: 16/Jan/19 02:20 Worklog Time Spent: 10m Work Description: tweise commented on issue #7517: [BEAM-6440] Fix leakage of timer de-duplication map URL: https://github.com/apache/beam/pull/7517#issuecomment-454626552 Verified changes with our application - stable heap utilization now. Thanks for the quick turnaround! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185548) Time Spent: 1h 10m (was: 1h) > FlinkTimerInternals memory leak > --- > > Key: BEAM-6440 > URL: https://issues.apache.org/jira/browse/BEAM-6440 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Affects Versions: 2.9.0 >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Attachments: image-2019-01-15-10-31-43-618.png, > image-2019-01-15-10-41-08-200.png > > Time Spent: 1h 10m > Remaining Estimate: 0h > > After running a portable streaming app for 2-3 days, we see heap space > exhausted. Memory analysis shows large number of TimerData objects referenced > by heap state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6445) Improve Release Process
Sam Rohde created BEAM-6445: --- Summary: Improve Release Process Key: BEAM-6445 URL: https://issues.apache.org/jira/browse/BEAM-6445 Project: Beam Issue Type: Improvement Components: project-management Reporter: Sam Rohde Assignee: Sam Rohde This JIRA tracks the improvement of the Beam release process as [discussed in the dev list|[https://lists.apache.org/thread.html/d52ffbfca21eee953a230100520bd56d947a359c0029d5c291b736a7@%3Cdev.beam.apache.org%3E].] In summary, this change will hopefully increase the greenness of the build by: increasing coverage, adding pre and post commits to release validation, and adding a regular cadence to look at flaky and backlogged tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-6445) Improve Release Process
[ https://issues.apache.org/jira/browse/BEAM-6445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Rohde updated BEAM-6445: Description: This JIRA tracks the improvement of the Beam release process as [discussed in the dev list|https://lists.apache.org/thread.html/d52ffbfca21eee953a230100520bd56d947a359c0029d5c291b736a7@%3Cdev.beam.apache.org%3E]. In summary, this change will hopefully increase the greenness of the build by: increasing coverage, adding pre and post commits to release validation, and adding a regular cadence to look at flaky and backlogged tests. (was: This JIRA tracks the improvement of the Beam release process as [discussed in the dev list|[https://lists.apache.org/thread.html/d52ffbfca21eee953a230100520bd56d947a359c0029d5c291b736a7@%3Cdev.beam.apache.org%3E]]. In summary, this change will hopefully increase the greenness of the build by: increasing coverage, adding pre and post commits to release validation, and adding a regular cadence to look at flaky and backlogged tests.) > Improve Release Process > --- > > Key: BEAM-6445 > URL: https://issues.apache.org/jira/browse/BEAM-6445 > Project: Beam > Issue Type: Improvement > Components: project-management >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Major > > This JIRA tracks the improvement of the Beam release process as [discussed in > the dev > list|https://lists.apache.org/thread.html/d52ffbfca21eee953a230100520bd56d947a359c0029d5c291b736a7@%3Cdev.beam.apache.org%3E]. > In summary, this change will hopefully increase the greenness of the build > by: increasing coverage, adding pre and post commits to release validation, > and adding a regular cadence to look at flaky and backlogged tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6445) Improve Release Process
[ https://issues.apache.org/jira/browse/BEAM-6445?focusedWorklogId=185543=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185543 ] ASF GitHub Bot logged work on BEAM-6445: Author: ASF GitHub Bot Created on: 16/Jan/19 01:59 Start Date: 16/Jan/19 01:59 Worklog Time Spent: 10m Work Description: rohdesamuel commented on issue #7529: [BEAM-6445]: Release Guide changes for release process improvement URL: https://github.com/apache/beam/pull/7529#issuecomment-454622356 Adding reviewers based from dev list thread: @kennknowles @aaltay @swegner @Ardagan @boyuanzz This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185543) Time Spent: 20m (was: 10m) > Improve Release Process > --- > > Key: BEAM-6445 > URL: https://issues.apache.org/jira/browse/BEAM-6445 > Project: Beam > Issue Type: Improvement > Components: project-management >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > This JIRA tracks the improvement of the Beam release process as [discussed in > the dev > list|https://lists.apache.org/thread.html/d52ffbfca21eee953a230100520bd56d947a359c0029d5c291b736a7@%3Cdev.beam.apache.org%3E]. > In summary, this change will hopefully increase the greenness of the build > by: increasing coverage, adding pre and post commits to release validation, > and adding a regular cadence to look at flaky and backlogged tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6445) Improve Release Process
[ https://issues.apache.org/jira/browse/BEAM-6445?focusedWorklogId=185542=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185542 ] ASF GitHub Bot logged work on BEAM-6445: Author: ASF GitHub Bot Created on: 16/Jan/19 01:58 Start Date: 16/Jan/19 01:58 Worklog Time Spent: 10m Work Description: rohdesamuel commented on pull request #7529: [BEAM-6445]: Release Guide changes for release process improvement URL: https://github.com/apache/beam/pull/7529 This change will hopefully increase the greenness of the build by: increasing coverage, adding pre and post commits to release validation, and adding a regular cadence to look at flaky and backlogged tests. Dev list thread: https://lists.apache.org/thread.html/d52ffbfca21eee953a230100520bd56d947a359c0029d5c291b736a7@%3Cdev.beam.apache.org%3E Follow this checklist to help us incorporate your contribution quickly and easily: - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the
[jira] [Updated] (BEAM-6445) Improve Release Process
[ https://issues.apache.org/jira/browse/BEAM-6445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Rohde updated BEAM-6445: Description: This JIRA tracks the improvement of the Beam release process as [discussed in the dev list|[https://lists.apache.org/thread.html/d52ffbfca21eee953a230100520bd56d947a359c0029d5c291b736a7@%3Cdev.beam.apache.org%3E]]. In summary, this change will hopefully increase the greenness of the build by: increasing coverage, adding pre and post commits to release validation, and adding a regular cadence to look at flaky and backlogged tests. (was: This JIRA tracks the improvement of the Beam release process as [discussed in the dev list|[https://lists.apache.org/thread.html/d52ffbfca21eee953a230100520bd56d947a359c0029d5c291b736a7@%3Cdev.beam.apache.org%3E].] In summary, this change will hopefully increase the greenness of the build by: increasing coverage, adding pre and post commits to release validation, and adding a regular cadence to look at flaky and backlogged tests.) > Improve Release Process > --- > > Key: BEAM-6445 > URL: https://issues.apache.org/jira/browse/BEAM-6445 > Project: Beam > Issue Type: Improvement > Components: project-management >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Major > > This JIRA tracks the improvement of the Beam release process as [discussed in > the dev > list|[https://lists.apache.org/thread.html/d52ffbfca21eee953a230100520bd56d947a359c0029d5c291b736a7@%3Cdev.beam.apache.org%3E]]. > In summary, this change will hopefully increase the greenness of the build > by: increasing coverage, adding pre and post commits to release validation, > and adding a regular cadence to look at flaky and backlogged tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4184) S3ResourceIdTest has had a masked failure
[ https://issues.apache.org/jira/browse/BEAM-4184?focusedWorklogId=185541=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185541 ] ASF GitHub Bot logged work on BEAM-4184: Author: ASF GitHub Bot Created on: 16/Jan/19 01:53 Start Date: 16/Jan/19 01:53 Worklog Time Spent: 10m Work Description: adude3141 commented on issue #7414: [BEAM-4184] s3 resource id test has had a masked failure URL: https://github.com/apache/beam/pull/7414#issuecomment-454621215 @kennknowles PTAL I d suggest merging this change first and decide afterwards how to handle the two other sickbayed tests. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185541) Time Spent: 10m Remaining Estimate: 0h > S3ResourceIdTest has had a masked failure > - > > Key: BEAM-4184 > URL: https://issues.apache.org/jira/browse/BEAM-4184 > Project: Beam > Issue Type: Bug > Components: io-java-aws >Reporter: Kenneth Knowles >Assignee: Michael Luckey >Priority: Major > Labels: sickbay > Fix For: 2.11.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Sickbayed in https://github.com/apache/beam/pull/5161, the test should be > fixed and no longer ignored. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[
https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=185540=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185540
]
ASF GitHub Bot logged work on BEAM-6024:
Author: ASF GitHub Bot
Created on: 16/Jan/19 01:49
Start Date: 16/Jan/19 01:49
Worklog Time Spent: 10m
Work Description: markflyhigh commented on pull request #7423:
[BEAM-6024] Build Python 3 container image with Gradle
URL: https://github.com/apache/beam/pull/7423#discussion_r248127767
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1523,7 +1523,14 @@ artifactId=${project.name}
project.task('setupVirtualenv') {
doLast {
- project.exec { commandLine 'virtualenv', "${project.ext.envdir}" }
+ def virtualenvCmd = [
+'virtualenv',
+"${project.ext.envdir}",
+ ]
+ if (project.hasProperty('python3')) {
+virtualenvCmd += '--python=python3'
Review comment:
You are right. We should consider other version of python 3 in the future.
Currently I don't have a good answer yet. So let's use `python3` in general and
specify `--python=python35` (I think this is the right format) in virtualenv
cmd. A constant with comments sounds good.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 185540)
Time Spent: 1h 10m (was: 1h)
> Gradle setupVirtualenv supports Python 3
>
>
> Key: BEAM-6024
> URL: https://issues.apache.org/jira/browse/BEAM-6024
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-harness
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Need to depend on Python 3 virtualenv in few places:
> - Build Dataflow worker container in Python 3
> - Run ValidatesRunner and integration tests on Jenkins
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-4142) HadoopResourceIdTest has had a masked failure
[ https://issues.apache.org/jira/browse/BEAM-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743526#comment-16743526 ] Kenneth Knowles commented on BEAM-4142: --- [~JozoVilcek] take a look at the similar issues for other ResourceIdTest. Maybe they have a hint. Half of them are now fixed. I thought I would have time for this but I do not. > HadoopResourceIdTest has had a masked failure > - > > Key: BEAM-4142 > URL: https://issues.apache.org/jira/browse/BEAM-4142 > Project: Beam > Issue Type: Bug > Components: io-java-hadoop >Reporter: Kenneth Knowles >Priority: Major > Labels: sickbay > Fix For: 2.11.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Sickbayed in https://github.com/apache/beam/pull/5161, the test should be > fixed and no longer ignored. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-2902) Add user state support for ParDo.Multi for the Dataflow runner
[ https://issues.apache.org/jira/browse/BEAM-2902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-2902: -- Fix Version/s: 2.11.0 > Add user state support for ParDo.Multi for the Dataflow runner > -- > > Key: BEAM-2902 > URL: https://issues.apache.org/jira/browse/BEAM-2902 > Project: Beam > Issue Type: Sub-task > Components: runner-dataflow >Reporter: Luke Cwik >Priority: Minor > Labels: portability, sickbay > Fix For: 2.11.0 > > > Add support for user state in a ParDo.Multi by: > * adding expansion/conversion logic within Dataflow service to add any needed > GBKs to allow expansion. > *OR* > * Add support for expansion inside the Beam SDK once the PTransformMatcher > exposes a way to know when the replacement is not required by checking that > the preceding ParDos to a GBK are key preserving. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[
https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=185536=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185536
]
ASF GitHub Bot logged work on BEAM-6024:
Author: ASF GitHub Bot
Created on: 16/Jan/19 01:33
Start Date: 16/Jan/19 01:33
Worklog Time Spent: 10m
Work Description: markflyhigh commented on pull request #7423:
[BEAM-6024] Build Python 3 container image with Gradle
URL: https://github.com/apache/beam/pull/7423#discussion_r248125099
##
File path:
.test-infra/jenkins/job_PostCommit_Python_ValidatesContainer_Dataflow.groovy
##
@@ -31,5 +31,6 @@
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Py_ValCont',
// Execute shell command to test Python SDK.
steps {
shell('cd ' + commonJobProperties.checkoutDir + ' && bash
sdks/python/container/run_validatescontainer.sh')
+shell('cd ' + commonJobProperties.checkoutDir + ' && bash
sdks/python/container/run_validatescontainer_py3.sh')
Review comment:
Jenkins DSL may have a way to run multiple steps in parallel:
https://stackoverflow.com/a/36893393
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 185536)
Time Spent: 50m (was: 40m)
> Gradle setupVirtualenv supports Python 3
>
>
> Key: BEAM-6024
> URL: https://issues.apache.org/jira/browse/BEAM-6024
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-harness
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Need to depend on Python 3 virtualenv in few places:
> - Build Dataflow worker container in Python 3
> - Run ValidatesRunner and integration tests on Jenkins
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[
https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=185537=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185537
]
ASF GitHub Bot logged work on BEAM-6024:
Author: ASF GitHub Bot
Created on: 16/Jan/19 01:33
Start Date: 16/Jan/19 01:33
Worklog Time Spent: 10m
Work Description: markflyhigh commented on pull request #7423:
[BEAM-6024] Build Python 3 container image with Gradle
URL: https://github.com/apache/beam/pull/7423#discussion_r248125099
##
File path:
.test-infra/jenkins/job_PostCommit_Python_ValidatesContainer_Dataflow.groovy
##
@@ -31,5 +31,6 @@
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Py_ValCont',
// Execute shell command to test Python SDK.
steps {
shell('cd ' + commonJobProperties.checkoutDir + ' && bash
sdks/python/container/run_validatescontainer.sh')
+shell('cd ' + commonJobProperties.checkoutDir + ' && bash
sdks/python/container/run_validatescontainer_py3.sh')
Review comment:
Jenkins DSL may have a way to run multiple steps in parallel:
https://stackoverflow.com/a/36893393. But I never tried before.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 185537)
Time Spent: 1h (was: 50m)
> Gradle setupVirtualenv supports Python 3
>
>
> Key: BEAM-6024
> URL: https://issues.apache.org/jira/browse/BEAM-6024
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-harness
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Need to depend on Python 3 virtualenv in few places:
> - Build Dataflow worker container in Python 3
> - Run ValidatesRunner and integration tests on Jenkins
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-3272) ParDoTranslatorTest: Error creating local cluster while creating checkpoint file
[
https://issues.apache.org/jira/browse/BEAM-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743520#comment-16743520
]
Thomas Weise commented on BEAM-3272:
FWIW the unique directory issue was fixed.
> ParDoTranslatorTest: Error creating local cluster while creating checkpoint
> file
>
>
> Key: BEAM-3272
> URL: https://issues.apache.org/jira/browse/BEAM-3272
> Project: Beam
> Issue Type: Bug
> Components: runner-apex
>Reporter: Eugene Kirpichov
>Priority: Critical
> Labels: flake, sickbay
> Fix For: 2.11.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Failed build:
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-apex/5330/console
> Key output:
> {code}
> 2017-11-29T01:21:26.956 [ERROR]
> testAssertionFailure(org.apache.beam.runners.apex.translation.ParDoTranslatorTest)
> Time elapsed: 2.007 s <<< ERROR!
> java.lang.RuntimeException: Error creating local cluster
> at
> org.apache.apex.engine.EmbeddedAppLauncherImpl.getController(EmbeddedAppLauncherImpl.java:122)
> at
> org.apache.apex.engine.EmbeddedAppLauncherImpl.launchApp(EmbeddedAppLauncherImpl.java:71)
> at
> org.apache.apex.engine.EmbeddedAppLauncherImpl.launchApp(EmbeddedAppLauncherImpl.java:46)
> at org.apache.beam.runners.apex.ApexRunner.run(ApexRunner.java:197)
> at
> org.apache.beam.runners.apex.TestApexRunner.run(TestApexRunner.java:57)
> at
> org.apache.beam.runners.apex.TestApexRunner.run(TestApexRunner.java:31)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:304)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:290)
> at
> org.apache.beam.runners.apex.translation.ParDoTranslatorTest.runExpectingAssertionFailure(ParDoTranslatorTest.java:156)
> {code}
> ...
> {code}
> Caused by: ExitCodeException exitCode=1: chmod: cannot access
> ‘/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/runners/apex/target/com.datatorrent.stram.StramLocalCluster/checkpoints/2/_tmp’:
> No such file or directory
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
> at org.apache.hadoop.util.Shell.run(Shell.java:479)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:225)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:209)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
> at org.apache.hadoop.fs.FileSystem.primitiveCreate(FileSystem.java:1017)
> at
> org.apache.hadoop.fs.DelegateToFileSystem.createInternal(DelegateToFileSystem.java:99)
> at
> org.apache.hadoop.fs.ChecksumFs$ChecksumFSOutputSummer.(ChecksumFs.java:352)
> at org.apache.hadoop.fs.ChecksumFs.createInternal(ChecksumFs.java:399)
> at
> org.apache.hadoop.fs.AbstractFileSystem.create(AbstractFileSystem.java:584)
> at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:686)
> at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:682)
> at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
> at org.apache.hadoop.fs.FileContext.create(FileContext.java:688)
> at
> com.datatorrent.common.util.AsyncFSStorageAgent.copyToHDFS(AsyncFSStorageAgent.java:119)
> ... 50 more
> {code}
> By inspecting code at the stack frames, seems it's trying to copy an
> operator's checkpoint "to HDFS" (which in this case is the local disk), but
> fails while creating the target file of the copy - creation creates the file
> (successfully) and chmods it writable (unsuccessfully). Barring something
> subtle (e.g. chmod being not allowed to call immediately after creating a
> FileOutputStream), this looks like the whole directory was possibly deleted
> from under the process. I don't know why this would be the case though, or
> how to debug it.
> Either way, the path being accessed is funky:
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/runners/apex/target/...
> - I think it'd be better if this test used a "@Rule TemporaryFolder" to
> store Apex checkpoints. I don't know whether the Apex runner allows
[jira] [Updated] (BEAM-688) Failure of beam-sdks-java-maven-archetypes-starter with undeclared dependency error
[
https://issues.apache.org/jira/browse/BEAM-688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-688:
-
Priority: Critical (was: Major)
> Failure of beam-sdks-java-maven-archetypes-starter with undeclared dependency
> error
> ---
>
> Key: BEAM-688
> URL: https://issues.apache.org/jira/browse/BEAM-688
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Scott Wegner
>Priority: Critical
> Labels: flake
>
> The starter archetype has flaky dependencies. It is reported to fail reliably
> on repeatedly install.
> {noformat}
> [INFO] --- maven-dependency-plugin:2.10:analyze-only (default) @
> beam-sdks-java-maven-archetypes-starter ---
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.slf4j:slf4j-api:jar:1.7.14:runtime
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-4142) HadoopResourceIdTest has had a masked failure
[ https://issues.apache.org/jira/browse/BEAM-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-4142: - Assignee: (was: Kenneth Knowles) > HadoopResourceIdTest has had a masked failure > - > > Key: BEAM-4142 > URL: https://issues.apache.org/jira/browse/BEAM-4142 > Project: Beam > Issue Type: Bug > Components: io-java-hadoop >Reporter: Kenneth Knowles >Priority: Major > Labels: sickbay > Fix For: 2.11.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Sickbayed in https://github.com/apache/beam/pull/5161, the test should be > fixed and no longer ignored. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-4527) JMSIOTest testCheckpointMark flake
[
https://issues.apache.org/jira/browse/BEAM-4527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-4527:
--
Fix Version/s: 2.11.0
> JMSIOTest testCheckpointMark flake
> --
>
> Key: BEAM-4527
> URL: https://issues.apache.org/jira/browse/BEAM-4527
> Project: Beam
> Issue Type: Bug
> Components: io-java-jms
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Critical
> Labels: flake
> Fix For: 2.11.0
>
>
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/6301/testReport/junit/org.apache.beam.sdk.io.jms/JmsIOTest/testCheckpointMark/
> {code}
> java.lang.AssertionError: expected:<6> but was:<9>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.beam.sdk.io.jms.JmsIOTest.testCheckpointMark(JmsIOTest.java:318)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-4191) WatermarkManagerTest has had a masked failure (both copies)
[ https://issues.apache.org/jira/browse/BEAM-4191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-4191: -- Fix Version/s: 2.11.0 > WatermarkManagerTest has had a masked failure (both copies) > --- > > Key: BEAM-4191 > URL: https://issues.apache.org/jira/browse/BEAM-4191 > Project: Beam > Issue Type: Bug > Components: runner-direct >Reporter: Kenneth Knowles >Priority: Major > Labels: sickbay > Fix For: 2.11.0 > > > Discovered in https://github.com/apache/beam/pull/5161, the test should be > sickbayed while the code is made correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-1582) ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.
[ https://issues.apache.org/jira/browse/BEAM-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743523#comment-16743523 ] Kenneth Knowles commented on BEAM-1582: --- [~echauchot] this is a pretty old issue, but I'm triaging sickbay/flake to see what needs attention or closing. Any idea if this is still relevant? > ResumeFromCheckpointStreamingTest flakes with what appears as a second firing. > -- > > Key: BEAM-1582 > URL: https://issues.apache.org/jira/browse/BEAM-1582 > Project: Beam > Issue Type: Bug > Components: runner-spark >Reporter: Amit Sela >Priority: Minor > Labels: flake > Fix For: 2.11.0 > > > See: > https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-spark/2788/testReport/junit/org.apache.beam.runners.spark.translation.streaming/ResumeFromCheckpointStreamingTest/testWithResume/ > After some digging in it appears that a second firing occurs (though only one > is expected) but it doesn't come from a stale state (state is empty before it > fires). > Might be a retry happening for some reason, which is OK in terms of > fault-tolerance guarantees (at-least-once), but not so much in terms of flaky > tests. > I'm looking into this hoping to fix this ASAP. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-1582) ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.
[ https://issues.apache.org/jira/browse/BEAM-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-1582: -- Fix Version/s: 2.11.0 > ResumeFromCheckpointStreamingTest flakes with what appears as a second firing. > -- > > Key: BEAM-1582 > URL: https://issues.apache.org/jira/browse/BEAM-1582 > Project: Beam > Issue Type: Bug > Components: runner-spark >Reporter: Amit Sela >Priority: Minor > Labels: flake > Fix For: 2.11.0 > > > See: > https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-spark/2788/testReport/junit/org.apache.beam.runners.spark.translation.streaming/ResumeFromCheckpointStreamingTest/testWithResume/ > After some digging in it appears that a second firing occurs (though only one > is expected) but it doesn't come from a stale state (state is empty before it > fires). > Might be a retry happening for some reason, which is OK in terms of > fault-tolerance guarantees (at-least-once), but not so much in terms of flaky > tests. > I'm looking into this hoping to fix this ASAP. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-688) Failure of beam-sdks-java-maven-archetypes-starter with undeclared dependency error
[
https://issues.apache.org/jira/browse/BEAM-688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-688:
-
Fix Version/s: (was: Not applicable)
2.11.0
> Failure of beam-sdks-java-maven-archetypes-starter with undeclared dependency
> error
> ---
>
> Key: BEAM-688
> URL: https://issues.apache.org/jira/browse/BEAM-688
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Scott Wegner
>Priority: Critical
> Labels: flake
> Fix For: 2.11.0
>
>
> The starter archetype has flaky dependencies. It is reported to fail reliably
> on repeatedly install.
> {noformat}
> [INFO] --- maven-dependency-plugin:2.10:analyze-only (default) @
> beam-sdks-java-maven-archetypes-starter ---
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.slf4j:slf4j-api:jar:1.7.14:runtime
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (BEAM-688) Failure of beam-sdks-java-maven-archetypes-starter with undeclared dependency error
[
https://issues.apache.org/jira/browse/BEAM-688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles resolved BEAM-688.
--
Resolution: Cannot Reproduce
Fix Version/s: Not applicable
This is now too old. Flakes are critical but Maven is gone.
> Failure of beam-sdks-java-maven-archetypes-starter with undeclared dependency
> error
> ---
>
> Key: BEAM-688
> URL: https://issues.apache.org/jira/browse/BEAM-688
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Scott Wegner
>Priority: Critical
> Labels: flake
> Fix For: Not applicable
>
>
> The starter archetype has flaky dependencies. It is reported to fail reliably
> on repeatedly install.
> {noformat}
> [INFO] --- maven-dependency-plugin:2.10:analyze-only (default) @
> beam-sdks-java-maven-archetypes-starter ---
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.slf4j:slf4j-api:jar:1.7.14:runtime
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (BEAM-688) Failure of beam-sdks-java-maven-archetypes-starter with undeclared dependency error
[
https://issues.apache.org/jira/browse/BEAM-688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles reopened BEAM-688:
--
False. This is of the archetype.
> Failure of beam-sdks-java-maven-archetypes-starter with undeclared dependency
> error
> ---
>
> Key: BEAM-688
> URL: https://issues.apache.org/jira/browse/BEAM-688
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Scott Wegner
>Priority: Critical
> Labels: flake
> Fix For: Not applicable
>
>
> The starter archetype has flaky dependencies. It is reported to fail reliably
> on repeatedly install.
> {noformat}
> [INFO] --- maven-dependency-plugin:2.10:analyze-only (default) @
> beam-sdks-java-maven-archetypes-starter ---
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.slf4j:slf4j-api:jar:1.7.14:runtime
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-688) Failure of beam-sdks-java-maven-archetypes-starter with undeclared dependency error
[
https://issues.apache.org/jira/browse/BEAM-688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743519#comment-16743519
]
Kenneth Knowles commented on BEAM-688:
--
This should be a hard fail. TODO is confirm.
> Failure of beam-sdks-java-maven-archetypes-starter with undeclared dependency
> error
> ---
>
> Key: BEAM-688
> URL: https://issues.apache.org/jira/browse/BEAM-688
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Scott Wegner
>Priority: Critical
> Labels: flake
> Fix For: 2.11.0
>
>
> The starter archetype has flaky dependencies. It is reported to fail reliably
> on repeatedly install.
> {noformat}
> [INFO] --- maven-dependency-plugin:2.10:analyze-only (default) @
> beam-sdks-java-maven-archetypes-starter ---
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.slf4j:slf4j-api:jar:1.7.14:runtime
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-2902) Add user state support for ParDo.Multi for the Dataflow runner
[ https://issues.apache.org/jira/browse/BEAM-2902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743515#comment-16743515 ] Kenneth Knowles commented on BEAM-2902: --- Putting on 2.11.0 release since it is marked as sickbay, so it will be checked again to see if the sickbayed tests are critical or known. > Add user state support for ParDo.Multi for the Dataflow runner > -- > > Key: BEAM-2902 > URL: https://issues.apache.org/jira/browse/BEAM-2902 > Project: Beam > Issue Type: Sub-task > Components: runner-dataflow >Reporter: Luke Cwik >Priority: Minor > Labels: portability, sickbay > Fix For: 2.11.0 > > > Add support for user state in a ParDo.Multi by: > * adding expansion/conversion logic within Dataflow service to add any needed > GBKs to allow expansion. > *OR* > * Add support for expansion inside the Beam SDK once the PTransformMatcher > exposes a way to know when the replacement is not required by checking that > the preceding ParDos to a GBK are key preserving. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-3272) ParDoTranslatorTest: Error creating local cluster while creating checkpoint file
[
https://issues.apache.org/jira/browse/BEAM-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-3272:
--
Fix Version/s: 2.11.0
> ParDoTranslatorTest: Error creating local cluster while creating checkpoint
> file
>
>
> Key: BEAM-3272
> URL: https://issues.apache.org/jira/browse/BEAM-3272
> Project: Beam
> Issue Type: Bug
> Components: runner-apex
>Reporter: Eugene Kirpichov
>Priority: Critical
> Labels: flake, sickbay
> Fix For: 2.11.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Failed build:
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-apex/5330/console
> Key output:
> {code}
> 2017-11-29T01:21:26.956 [ERROR]
> testAssertionFailure(org.apache.beam.runners.apex.translation.ParDoTranslatorTest)
> Time elapsed: 2.007 s <<< ERROR!
> java.lang.RuntimeException: Error creating local cluster
> at
> org.apache.apex.engine.EmbeddedAppLauncherImpl.getController(EmbeddedAppLauncherImpl.java:122)
> at
> org.apache.apex.engine.EmbeddedAppLauncherImpl.launchApp(EmbeddedAppLauncherImpl.java:71)
> at
> org.apache.apex.engine.EmbeddedAppLauncherImpl.launchApp(EmbeddedAppLauncherImpl.java:46)
> at org.apache.beam.runners.apex.ApexRunner.run(ApexRunner.java:197)
> at
> org.apache.beam.runners.apex.TestApexRunner.run(TestApexRunner.java:57)
> at
> org.apache.beam.runners.apex.TestApexRunner.run(TestApexRunner.java:31)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:304)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:290)
> at
> org.apache.beam.runners.apex.translation.ParDoTranslatorTest.runExpectingAssertionFailure(ParDoTranslatorTest.java:156)
> {code}
> ...
> {code}
> Caused by: ExitCodeException exitCode=1: chmod: cannot access
> ‘/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/runners/apex/target/com.datatorrent.stram.StramLocalCluster/checkpoints/2/_tmp’:
> No such file or directory
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
> at org.apache.hadoop.util.Shell.run(Shell.java:479)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:225)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:209)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
> at org.apache.hadoop.fs.FileSystem.primitiveCreate(FileSystem.java:1017)
> at
> org.apache.hadoop.fs.DelegateToFileSystem.createInternal(DelegateToFileSystem.java:99)
> at
> org.apache.hadoop.fs.ChecksumFs$ChecksumFSOutputSummer.(ChecksumFs.java:352)
> at org.apache.hadoop.fs.ChecksumFs.createInternal(ChecksumFs.java:399)
> at
> org.apache.hadoop.fs.AbstractFileSystem.create(AbstractFileSystem.java:584)
> at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:686)
> at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:682)
> at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
> at org.apache.hadoop.fs.FileContext.create(FileContext.java:688)
> at
> com.datatorrent.common.util.AsyncFSStorageAgent.copyToHDFS(AsyncFSStorageAgent.java:119)
> ... 50 more
> {code}
> By inspecting code at the stack frames, seems it's trying to copy an
> operator's checkpoint "to HDFS" (which in this case is the local disk), but
> fails while creating the target file of the copy - creation creates the file
> (successfully) and chmods it writable (unsuccessfully). Barring something
> subtle (e.g. chmod being not allowed to call immediately after creating a
> FileOutputStream), this looks like the whole directory was possibly deleted
> from under the process. I don't know why this would be the case though, or
> how to debug it.
> Either way, the path being accessed is funky:
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/runners/apex/target/...
> - I think it'd be better if this test used a "@Rule TemporaryFolder" to
> store Apex checkpoints. I don't know whether the Apex runner allows that, but
> I can see how it could help reduce
[jira] [Updated] (BEAM-3583) Apex Local Cluster Creation fails in TestE2EWordCount
[
https://issues.apache.org/jira/browse/BEAM-3583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-3583:
--
Fix Version/s: 2.11.0
> Apex Local Cluster Creation fails in TestE2EWordCount
> -
>
> Key: BEAM-3583
> URL: https://issues.apache.org/jira/browse/BEAM-3583
> Project: Beam
> Issue Type: Bug
> Components: runner-apex
>Reporter: Thomas Groh
>Priority: Major
> Labels: sickbay
> Fix For: 2.11.0
>
>
> Specifically, in gradle builds creating the local cluster fails with
> {{Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File
> file:/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_GradleBuild/src/examples/java/target/com.datatorrent.stram.StramLocalCluster
> does not exist}}
> This seems to happen reliably on almost all gradle builds:
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1597/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (BEAM-4638) Post-commit tests fail job, but report success on detailed view
[ https://issues.apache.org/jira/browse/BEAM-4638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-4638. --- Resolution: Not A Problem Fix Version/s: Not applicable > Post-commit tests fail job, but report success on detailed view > --- > > Key: BEAM-4638 > URL: https://issues.apache.org/jira/browse/BEAM-4638 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Mikhail Gryzykhin >Assignee: Alan Myrvold >Priority: Major > Labels: flake > Fix For: Not applicable > > > Some of failed jenkins jobs report tests as failed in job overview, but when > you click on test, detailed view shows test as passed with some console > output. > [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/923/] > [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/923/testReport/junit/org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest/testWrite/] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-2791) DirectRunner shuts down when there are pending event time timers
[ https://issues.apache.org/jira/browse/BEAM-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-2791: -- Fix Version/s: 2.11.0 > DirectRunner shuts down when there are pending event time timers > > > Key: BEAM-2791 > URL: https://issues.apache.org/jira/browse/BEAM-2791 > Project: Beam > Issue Type: Bug > Components: runner-direct >Reporter: Kenneth Knowles >Priority: Major > Labels: sickbay > Fix For: 2.11.0 > > > When there are pending event time timers for a window by definition they are > within the window so it should not be possible to GC the window nor to shut > down the pipeline. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-4527) JMSIOTest testCheckpointMark flake
[
https://issues.apache.org/jira/browse/BEAM-4527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles reassigned BEAM-4527:
-
Assignee: (was: Jean-Baptiste Onofré)
> JMSIOTest testCheckpointMark flake
> --
>
> Key: BEAM-4527
> URL: https://issues.apache.org/jira/browse/BEAM-4527
> Project: Beam
> Issue Type: Bug
> Components: io-java-jms
>Reporter: Kenneth Knowles
>Priority: Critical
> Labels: flake
> Fix For: 2.11.0
>
>
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/6301/testReport/junit/org.apache.beam.sdk.io.jms/JmsIOTest/testCheckpointMark/
> {code}
> java.lang.AssertionError: expected:<6> but was:<9>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.beam.sdk.io.jms.JmsIOTest.testCheckpointMark(JmsIOTest.java:318)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-4638) Post-commit tests fail job, but report success on detailed view
[ https://issues.apache.org/jira/browse/BEAM-4638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743513#comment-16743513 ] Kenneth Knowles commented on BEAM-4638: --- I think the Gradle Scan is what you have to use. The Jenkins UI is keyed only on the class name, which is not unique across test runs (and doesn't have to be). > Post-commit tests fail job, but report success on detailed view > --- > > Key: BEAM-4638 > URL: https://issues.apache.org/jira/browse/BEAM-4638 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Mikhail Gryzykhin >Assignee: Alan Myrvold >Priority: Major > Labels: flake > > Some of failed jenkins jobs report tests as failed in job overview, but when > you click on test, detailed view shows test as passed with some console > output. > [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/923/] > [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/923/testReport/junit/org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest/testWrite/] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-4628) Flaky post-commit test org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq
[ https://issues.apache.org/jira/browse/BEAM-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-4628: -- Fix Version/s: 2.11.0 > Flaky post-commit test > org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq > --- > > Key: BEAM-4628 > URL: https://issues.apache.org/jira/browse/BEAM-4628 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Mikhail Gryzykhin >Assignee: Anton Kedin >Priority: Critical > Labels: flake > Fix For: 2.11.0 > > > As part of code lime we want to fix existing flaky tests. This test changed > state in 4 jobs with no code changes during ~2 days. > Link to the test history: > [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/844/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/history/|https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/844/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/history/?start=25] > Links to the jobs where test changed state: > ||Url|| > |[https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/827/] > [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/854/] > [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/867/] > https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/844/| > Job fails with timeout of 5 minutes, while usual runtime is around 70 > seconds. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-3604) MqttIOTest testReadNoClientId failure timeout
[
https://issues.apache.org/jira/browse/BEAM-3604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles reassigned BEAM-3604:
-
Assignee: (was: Jean-Baptiste Onofré)
> MqttIOTest testReadNoClientId failure timeout
> -
>
> Key: BEAM-3604
> URL: https://issues.apache.org/jira/browse/BEAM-3604
> Project: Beam
> Issue Type: Bug
> Components: io-java-mqtt
>Reporter: Kenneth Knowles
>Priority: Critical
> Labels: flake
> Fix For: 2.11.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> I've seen failures a bit today. Here is one:
> [https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1758/testReport/junit/org.apache.beam.sdk.io.mqtt/MqttIOTest/testReadNoClientId/]
> Filing all flakes as "Critical" priority so we can sickbay or fix.
> Since that build will get GC'd, here is the Standard Error. It looks like
> from that perspective everything went as planned, but perhaps the test has a
> race condition or something?
> {code}
> Feb 01, 2018 11:28:01 PM org.apache.beam.sdk.io.mqtt.MqttIOTest startBroker
> INFO: Finding free network port
> Feb 01, 2018 11:28:01 PM org.apache.beam.sdk.io.mqtt.MqttIOTest startBroker
> INFO: Starting ActiveMQ brokerService on 57986
> Feb 01, 2018 11:28:03 PM org.apache.activemq.broker.BrokerService
> doStartPersistenceAdapter
> INFO: Using Persistence Adapter: MemoryPersistenceAdapter
> Feb 01, 2018 11:28:04 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) is
> starting
> Feb 01, 2018 11:28:04 PM
> org.apache.activemq.transport.TransportServerThreadSupport doStart
> INFO: Listening for connections at: mqtt://localhost:57986
> Feb 01, 2018 11:28:04 PM org.apache.activemq.broker.TransportConnector start
> INFO: Connector mqtt://localhost:57986 started
> Feb 01, 2018 11:28:04 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) started
> Feb 01, 2018 11:28:04 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: For help or more information please see: http://activemq.apache.org
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) is
> shutting down
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.TransportConnector stop
> INFO: Connector mqtt://localhost:57986 stopped
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) uptime
> 24.039 seconds
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) is
> shutdown
> Feb 01, 2018 11:28:26 PM org.apache.beam.sdk.io.mqtt.MqttIOTest startBroker
> INFO: Finding free network port
> Feb 01, 2018 11:28:26 PM org.apache.beam.sdk.io.mqtt.MqttIOTest startBroker
> INFO: Starting ActiveMQ brokerService on 46799
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService
> doStartPersistenceAdapter
> INFO: Using Persistence Adapter: MemoryPersistenceAdapter
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:2) is
> starting
> Feb 01, 2018 11:28:26 PM
> org.apache.activemq.transport.TransportServerThreadSupport doStart
> INFO: Listening for connections at: mqtt://localhost:46799
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.TransportConnector start
> INFO: Connector mqtt://localhost:46799 started
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:2) started
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: For help or more information please see: http://activemq.apache.org
> Feb 01, 2018 11:28:28 PM org.apache.beam.sdk.io.mqtt.MqttIOTest
> lambda$testRead$1
> INFO: Waiting pipeline connected to the MQTT broker before sending messages
> ...
> Feb 01, 2018 11:28:35 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:2) is
> shutting down
> Feb 01, 2018 11:28:35 PM org.apache.activemq.broker.TransportConnector stop
> INFO: Connector mqtt://localhost:46799 stopped
> Feb 01, 2018
[jira] [Updated] (BEAM-3604) MqttIOTest testReadNoClientId failure timeout
[
https://issues.apache.org/jira/browse/BEAM-3604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-3604:
--
Fix Version/s: 2.11.0
> MqttIOTest testReadNoClientId failure timeout
> -
>
> Key: BEAM-3604
> URL: https://issues.apache.org/jira/browse/BEAM-3604
> Project: Beam
> Issue Type: Bug
> Components: io-java-mqtt
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Critical
> Labels: flake
> Fix For: 2.11.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> I've seen failures a bit today. Here is one:
> [https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1758/testReport/junit/org.apache.beam.sdk.io.mqtt/MqttIOTest/testReadNoClientId/]
> Filing all flakes as "Critical" priority so we can sickbay or fix.
> Since that build will get GC'd, here is the Standard Error. It looks like
> from that perspective everything went as planned, but perhaps the test has a
> race condition or something?
> {code}
> Feb 01, 2018 11:28:01 PM org.apache.beam.sdk.io.mqtt.MqttIOTest startBroker
> INFO: Finding free network port
> Feb 01, 2018 11:28:01 PM org.apache.beam.sdk.io.mqtt.MqttIOTest startBroker
> INFO: Starting ActiveMQ brokerService on 57986
> Feb 01, 2018 11:28:03 PM org.apache.activemq.broker.BrokerService
> doStartPersistenceAdapter
> INFO: Using Persistence Adapter: MemoryPersistenceAdapter
> Feb 01, 2018 11:28:04 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) is
> starting
> Feb 01, 2018 11:28:04 PM
> org.apache.activemq.transport.TransportServerThreadSupport doStart
> INFO: Listening for connections at: mqtt://localhost:57986
> Feb 01, 2018 11:28:04 PM org.apache.activemq.broker.TransportConnector start
> INFO: Connector mqtt://localhost:57986 started
> Feb 01, 2018 11:28:04 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) started
> Feb 01, 2018 11:28:04 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: For help or more information please see: http://activemq.apache.org
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) is
> shutting down
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.TransportConnector stop
> INFO: Connector mqtt://localhost:57986 stopped
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) uptime
> 24.039 seconds
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) is
> shutdown
> Feb 01, 2018 11:28:26 PM org.apache.beam.sdk.io.mqtt.MqttIOTest startBroker
> INFO: Finding free network port
> Feb 01, 2018 11:28:26 PM org.apache.beam.sdk.io.mqtt.MqttIOTest startBroker
> INFO: Starting ActiveMQ brokerService on 46799
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService
> doStartPersistenceAdapter
> INFO: Using Persistence Adapter: MemoryPersistenceAdapter
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:2) is
> starting
> Feb 01, 2018 11:28:26 PM
> org.apache.activemq.transport.TransportServerThreadSupport doStart
> INFO: Listening for connections at: mqtt://localhost:46799
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.TransportConnector start
> INFO: Connector mqtt://localhost:46799 started
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:2) started
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: For help or more information please see: http://activemq.apache.org
> Feb 01, 2018 11:28:28 PM org.apache.beam.sdk.io.mqtt.MqttIOTest
> lambda$testRead$1
> INFO: Waiting pipeline connected to the MQTT broker before sending messages
> ...
> Feb 01, 2018 11:28:35 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:2) is
> shutting down
> Feb 01, 2018 11:28:35 PM org.apache.activemq.broker.TransportConnector stop
> INFO: Connector mqtt://localhost:46799 stopped
>
[jira] [Commented] (BEAM-5814) Failure in RegisterAndProcessBundleOperationTest.testProcessingBundleHandlesMultimapSideInputRequests
[
https://issues.apache.org/jira/browse/BEAM-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743507#comment-16743507
]
Kenneth Knowles commented on BEAM-5814:
---
How is this going? Are you actively working on it? You might release it if not.
> Failure in
> RegisterAndProcessBundleOperationTest.testProcessingBundleHandlesMultimapSideInputRequests
> -
>
> Key: BEAM-5814
> URL: https://issues.apache.org/jira/browse/BEAM-5814
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Ankur Goenka
>Priority: Critical
> Labels: flake
>
> https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1724/
> https://scans.gradle.com/s/xpsswtkdf62so/tests/jpaz6qjpaw4ma-czu63fxt273w6
> {code}
> java.util.concurrent.ExecutionException:
> org.mockito.exceptions.base.MockitoException:
> No argument value was captured!
> You might have forgotten to use argument.capture() in verify()...
> ...or you used capture() in stubbing but stubbed method was not called.
> Be aware that it is recommended to use capture() only with verify()
> ...
> {code}
> Assigning to someone I think has some context as we have no code history or
> blame for the Dataflow Worker. Do with this bug as you see fit. (filing all
> flakes as critical for project health reasons)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-5814) Failure in RegisterAndProcessBundleOperationTest.testProcessingBundleHandlesMultimapSideInputRequests
[
https://issues.apache.org/jira/browse/BEAM-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-5814:
--
Fix Version/s: 2.11.0
> Failure in
> RegisterAndProcessBundleOperationTest.testProcessingBundleHandlesMultimapSideInputRequests
> -
>
> Key: BEAM-5814
> URL: https://issues.apache.org/jira/browse/BEAM-5814
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Ankur Goenka
>Priority: Critical
> Labels: flake
> Fix For: 2.11.0
>
>
> https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1724/
> https://scans.gradle.com/s/xpsswtkdf62so/tests/jpaz6qjpaw4ma-czu63fxt273w6
> {code}
> java.util.concurrent.ExecutionException:
> org.mockito.exceptions.base.MockitoException:
> No argument value was captured!
> You might have forgotten to use argument.capture() in verify()...
> ...or you used capture() in stubbing but stubbed method was not called.
> Be aware that it is recommended to use capture() only with verify()
> ...
> {code}
> Assigning to someone I think has some context as we have no code history or
> blame for the Dataflow Worker. Do with this bug as you see fit. (filing all
> flakes as critical for project health reasons)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-3604) MqttIOTest testReadNoClientId failure timeout
[
https://issues.apache.org/jira/browse/BEAM-3604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743509#comment-16743509
]
Kenneth Knowles commented on BEAM-3604:
---
Can you confirm it is already fixed? I will unassign so that anyone can come
along and try to confirm.
> MqttIOTest testReadNoClientId failure timeout
> -
>
> Key: BEAM-3604
> URL: https://issues.apache.org/jira/browse/BEAM-3604
> Project: Beam
> Issue Type: Bug
> Components: io-java-mqtt
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Critical
> Labels: flake
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> I've seen failures a bit today. Here is one:
> [https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1758/testReport/junit/org.apache.beam.sdk.io.mqtt/MqttIOTest/testReadNoClientId/]
> Filing all flakes as "Critical" priority so we can sickbay or fix.
> Since that build will get GC'd, here is the Standard Error. It looks like
> from that perspective everything went as planned, but perhaps the test has a
> race condition or something?
> {code}
> Feb 01, 2018 11:28:01 PM org.apache.beam.sdk.io.mqtt.MqttIOTest startBroker
> INFO: Finding free network port
> Feb 01, 2018 11:28:01 PM org.apache.beam.sdk.io.mqtt.MqttIOTest startBroker
> INFO: Starting ActiveMQ brokerService on 57986
> Feb 01, 2018 11:28:03 PM org.apache.activemq.broker.BrokerService
> doStartPersistenceAdapter
> INFO: Using Persistence Adapter: MemoryPersistenceAdapter
> Feb 01, 2018 11:28:04 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) is
> starting
> Feb 01, 2018 11:28:04 PM
> org.apache.activemq.transport.TransportServerThreadSupport doStart
> INFO: Listening for connections at: mqtt://localhost:57986
> Feb 01, 2018 11:28:04 PM org.apache.activemq.broker.TransportConnector start
> INFO: Connector mqtt://localhost:57986 started
> Feb 01, 2018 11:28:04 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) started
> Feb 01, 2018 11:28:04 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: For help or more information please see: http://activemq.apache.org
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) is
> shutting down
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.TransportConnector stop
> INFO: Connector mqtt://localhost:57986 stopped
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) uptime
> 24.039 seconds
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) is
> shutdown
> Feb 01, 2018 11:28:26 PM org.apache.beam.sdk.io.mqtt.MqttIOTest startBroker
> INFO: Finding free network port
> Feb 01, 2018 11:28:26 PM org.apache.beam.sdk.io.mqtt.MqttIOTest startBroker
> INFO: Starting ActiveMQ brokerService on 46799
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService
> doStartPersistenceAdapter
> INFO: Using Persistence Adapter: MemoryPersistenceAdapter
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:2) is
> starting
> Feb 01, 2018 11:28:26 PM
> org.apache.activemq.transport.TransportServerThreadSupport doStart
> INFO: Listening for connections at: mqtt://localhost:46799
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.TransportConnector start
> INFO: Connector mqtt://localhost:46799 started
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:2) started
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: For help or more information please see: http://activemq.apache.org
> Feb 01, 2018 11:28:28 PM org.apache.beam.sdk.io.mqtt.MqttIOTest
> lambda$testRead$1
> INFO: Waiting pipeline connected to the MQTT broker before sending messages
> ...
> Feb 01, 2018 11:28:35 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:2) is
> shutting down
> Feb 01, 2018 11:28:35 PM
[jira] [Commented] (BEAM-6279) Failures in ElasticSearchIOTest: testWriteRetryValidRequest (versions 2 & 5) testDefaultRetryPredicate (version 5)
[
https://issues.apache.org/jira/browse/BEAM-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743506#comment-16743506
]
Kenneth Knowles commented on BEAM-6279:
---
How is this going now? If you aren't actively working on it, perhaps release it
for someone else to see about deflaking.
> Failures in ElasticSearchIOTest: testWriteRetryValidRequest (versions 2 & 5)
> testDefaultRetryPredicate (version 5)
> --
>
> Key: BEAM-6279
> URL: https://issues.apache.org/jira/browse/BEAM-6279
> Project: Beam
> Issue Type: Bug
> Components: io-java-elasticsearch, test-failures
>Reporter: Kenneth Knowles
>Assignee: Tim Robertson
>Priority: Critical
> Labels: flake
> Fix For: 2.11.0
>
>
> [https://builds.apache.org/job/beam_PreCommit_Java_Cron/730/]
> [https://scans.gradle.com/s/6rpbvgwx2nk7c]
> I don't see recent changes, so thinking this is a flake.
> testDefaultRetryPredicate:
> {code:java}
> java.lang.AssertionError
> :
> All incoming requests on node [node_s0] should have finished. Expected 0 but
> got 2540 Expected: <0L> but: was <2540L>
> at __randomizedtesting.SeedInfo.seed([4CF46FDC92DC07B8:B39A5DDFBC85274E]:0)
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.junit.Assert.assertThat(Assert.java:956)
> at
> org.elasticsearch.test.InternalTestCluster.lambda$assertRequestsFinished$17(InternalTestCluster.java:2086)
> at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:705)
> at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:679)
> at
> org.elasticsearch.test.InternalTestCluster.assertRequestsFinished(InternalTestCluster.java:2083)
> at
> org.elasticsearch.test.InternalTestCluster.assertAfterTest(InternalTestCluster.java:2062)
> at
> org.elasticsearch.test.ESIntegTestCase.afterInternal(ESIntegTestCase.java:581)
> at
> org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2054)
> ...
> {code}
> This looks like an internal assertion (aka {{checkState}}) failed and the
> cluster became unhealthy, maybe?
> testWriteRetryValidRequest:
> {code:java}
> org.apache.beam.sdk.Pipeline$PipelineExecutionException
> :
> java.io.IOException: listener timeout after waiting for [9] ms
> Open stacktrace
> Caused by:
> java.io.IOException
> :
> listener timeout after waiting for [9] ms
> Close stacktrace
> at
> org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:899)
> at org.elasticsearch.client.RestClient.performRequest(RestClient.java:227)
> at org.elasticsearch.client.RestClient.performRequest(RestClient.java:321)
> at
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.flushBatch(ElasticsearchIO.java:1285)
> at
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.finishBundle(ElasticsearchIO.java:1260){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[
https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743505#comment-16743505
]
Kenneth Knowles commented on BEAM-5811:
---
Are you actively working on this? Maybe release the bug so someone going around
deflaking will try it and see.
> Timeout in Python datastore_write_it_test.DatastoreWriteIT
> --
>
> Key: BEAM-5811
> URL: https://issues.apache.org/jira/browse/BEAM-5811
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Kenneth Knowles
>Assignee: yifan zou
>Priority: Critical
> Labels: flake
> Fix For: 2.11.0
>
>
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/]
> [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests]
> {code:java}
> TimedOutException: 'test_datastore_write_limit
> (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-6279) Failures in ElasticSearchIOTest: testWriteRetryValidRequest (versions 2 & 5) testDefaultRetryPredicate (version 5)
[
https://issues.apache.org/jira/browse/BEAM-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-6279:
--
Fix Version/s: 2.11.0
> Failures in ElasticSearchIOTest: testWriteRetryValidRequest (versions 2 & 5)
> testDefaultRetryPredicate (version 5)
> --
>
> Key: BEAM-6279
> URL: https://issues.apache.org/jira/browse/BEAM-6279
> Project: Beam
> Issue Type: Bug
> Components: io-java-elasticsearch, test-failures
>Reporter: Kenneth Knowles
>Assignee: Tim Robertson
>Priority: Critical
> Labels: flake
> Fix For: 2.11.0
>
>
> [https://builds.apache.org/job/beam_PreCommit_Java_Cron/730/]
> [https://scans.gradle.com/s/6rpbvgwx2nk7c]
> I don't see recent changes, so thinking this is a flake.
> testDefaultRetryPredicate:
> {code:java}
> java.lang.AssertionError
> :
> All incoming requests on node [node_s0] should have finished. Expected 0 but
> got 2540 Expected: <0L> but: was <2540L>
> at __randomizedtesting.SeedInfo.seed([4CF46FDC92DC07B8:B39A5DDFBC85274E]:0)
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.junit.Assert.assertThat(Assert.java:956)
> at
> org.elasticsearch.test.InternalTestCluster.lambda$assertRequestsFinished$17(InternalTestCluster.java:2086)
> at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:705)
> at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:679)
> at
> org.elasticsearch.test.InternalTestCluster.assertRequestsFinished(InternalTestCluster.java:2083)
> at
> org.elasticsearch.test.InternalTestCluster.assertAfterTest(InternalTestCluster.java:2062)
> at
> org.elasticsearch.test.ESIntegTestCase.afterInternal(ESIntegTestCase.java:581)
> at
> org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2054)
> ...
> {code}
> This looks like an internal assertion (aka {{checkState}}) failed and the
> cluster became unhealthy, maybe?
> testWriteRetryValidRequest:
> {code:java}
> org.apache.beam.sdk.Pipeline$PipelineExecutionException
> :
> java.io.IOException: listener timeout after waiting for [9] ms
> Open stacktrace
> Caused by:
> java.io.IOException
> :
> listener timeout after waiting for [9] ms
> Close stacktrace
> at
> org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:899)
> at org.elasticsearch.client.RestClient.performRequest(RestClient.java:227)
> at org.elasticsearch.client.RestClient.performRequest(RestClient.java:321)
> at
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.flushBatch(ElasticsearchIO.java:1285)
> at
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.finishBundle(ElasticsearchIO.java:1260){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[
https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-5811:
--
Fix Version/s: 2.11.0
> Timeout in Python datastore_write_it_test.DatastoreWriteIT
> --
>
> Key: BEAM-5811
> URL: https://issues.apache.org/jira/browse/BEAM-5811
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Kenneth Knowles
>Assignee: yifan zou
>Priority: Critical
> Labels: flake
> Fix For: 2.11.0
>
>
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/]
> [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests]
> {code:java}
> TimedOutException: 'test_datastore_write_limit
> (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-5642) test_pardo_state_only flaky (times out)
[ https://issues.apache.org/jira/browse/BEAM-5642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-5642: -- Fix Version/s: 2.11.0 > test_pardo_state_only flaky (times out) > --- > > Key: BEAM-5642 > URL: https://issues.apache.org/jira/browse/BEAM-5642 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Ahmet Altay >Assignee: Robert Bradshaw >Priority: Critical > Labels: flake > Fix For: 2.11.0 > > > [https://builds.apache.org/job/beam_PreCommit_Python_Commit/1577/consoleFull] > > *16:43:20* > ==*16:43:20* > ERROR: test_pardo_state_only > (apache_beam.runners.portability.portable_runner_test.PortableRunnerTest)*16:43:20* > > --*16:43:20* > Traceback (most recent call last):*16:43:20* File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py", > line 265, in test_pardo_state_only*16:43:20* > equal_to(expected))*16:43:20* File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/pipeline.py", > line 423, in __exit__*16:43:20* self.run().wait_until_finish()*16:43:20* > File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/runners/portability/portable_runner.py", > line 242, in wait_until_finish*16:43:20* > beam_job_api_pb2.GetJobStateRequest(job_id=self._job_id)):*16:43:20* File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/target/.tox/py3/lib/python3.5/site-packages/grpc/_channel.py", > line 363, in __next__*16:43:20* return self._next()*16:43:20* File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/target/.tox/py3/lib/python3.5/site-packages/grpc/_channel.py", > line 348, in _next*16:43:20* self._state.condition.wait()*16:43:20* > File "/usr/lib/python3.5/threading.py", line 293, in wait*16:43:20* > waiter.acquire()*16:43:20* File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/runners/portability/portable_runner_test.py", > line 68, in handler*16:43:20* raise BaseException(msg)*16:43:20* > BaseException: Timed out after 30 seconds.*16:43:20* >> > begin captured stdout << - -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-5642) test_pardo_state_only flaky (times out)
[ https://issues.apache.org/jira/browse/BEAM-5642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-5642: - Assignee: (was: Robert Bradshaw) > test_pardo_state_only flaky (times out) > --- > > Key: BEAM-5642 > URL: https://issues.apache.org/jira/browse/BEAM-5642 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Ahmet Altay >Priority: Critical > Labels: flake > Fix For: 2.11.0 > > > [https://builds.apache.org/job/beam_PreCommit_Python_Commit/1577/consoleFull] > > *16:43:20* > ==*16:43:20* > ERROR: test_pardo_state_only > (apache_beam.runners.portability.portable_runner_test.PortableRunnerTest)*16:43:20* > > --*16:43:20* > Traceback (most recent call last):*16:43:20* File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py", > line 265, in test_pardo_state_only*16:43:20* > equal_to(expected))*16:43:20* File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/pipeline.py", > line 423, in __exit__*16:43:20* self.run().wait_until_finish()*16:43:20* > File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/runners/portability/portable_runner.py", > line 242, in wait_until_finish*16:43:20* > beam_job_api_pb2.GetJobStateRequest(job_id=self._job_id)):*16:43:20* File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/target/.tox/py3/lib/python3.5/site-packages/grpc/_channel.py", > line 363, in __next__*16:43:20* return self._next()*16:43:20* File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/target/.tox/py3/lib/python3.5/site-packages/grpc/_channel.py", > line 348, in _next*16:43:20* self._state.condition.wait()*16:43:20* > File "/usr/lib/python3.5/threading.py", line 293, in wait*16:43:20* > waiter.acquire()*16:43:20* File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/runners/portability/portable_runner_test.py", > line 68, in handler*16:43:20* raise BaseException(msg)*16:43:20* > BaseException: Timed out after 30 seconds.*16:43:20* >> > begin captured stdout << - -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-4184) S3ResourceIdTest has had a masked failure
[ https://issues.apache.org/jira/browse/BEAM-4184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-4184: -- Fix Version/s: 2.11.0 > S3ResourceIdTest has had a masked failure > - > > Key: BEAM-4184 > URL: https://issues.apache.org/jira/browse/BEAM-4184 > Project: Beam > Issue Type: Bug > Components: io-java-aws >Reporter: Kenneth Knowles >Assignee: Michael Luckey >Priority: Major > Labels: sickbay > Fix For: 2.11.0 > > > Sickbayed in https://github.com/apache/beam/pull/5161, the test should be > fixed and no longer ignored. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-4142) HadoopResourceIdTest has had a masked failure
[ https://issues.apache.org/jira/browse/BEAM-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-4142: -- Fix Version/s: (was: 2.10.0) 2.11.0 > HadoopResourceIdTest has had a masked failure > - > > Key: BEAM-4142 > URL: https://issues.apache.org/jira/browse/BEAM-4142 > Project: Beam > Issue Type: Bug > Components: io-java-hadoop >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Labels: sickbay > Fix For: 2.11.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Sickbayed in https://github.com/apache/beam/pull/5161, the test should be > fixed and no longer ignored. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-4142) HadoopResourceIdTest has had a masked failure
[ https://issues.apache.org/jira/browse/BEAM-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-4142: -- Fix Version/s: 2.10.0 > HadoopResourceIdTest has had a masked failure > - > > Key: BEAM-4142 > URL: https://issues.apache.org/jira/browse/BEAM-4142 > Project: Beam > Issue Type: Bug > Components: io-java-hadoop >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Labels: sickbay > Fix For: 2.10.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Sickbayed in https://github.com/apache/beam/pull/5161, the test should be > fixed and no longer ignored. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-4142) HadoopResourceIdTest has had a masked failure
[ https://issues.apache.org/jira/browse/BEAM-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-4142: - Assignee: Kenneth Knowles > HadoopResourceIdTest has had a masked failure > - > > Key: BEAM-4142 > URL: https://issues.apache.org/jira/browse/BEAM-4142 > Project: Beam > Issue Type: Bug > Components: io-java-hadoop >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Labels: sickbay > Time Spent: 50m > Remaining Estimate: 0h > > Sickbayed in https://github.com/apache/beam/pull/5161, the test should be > fixed and no longer ignored. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4518) Errors when running Python Game stats with a low fixed_window_duration in the DirectRunner
[
https://issues.apache.org/jira/browse/BEAM-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743503#comment-16743503
]
Kenneth Knowles commented on BEAM-4518:
---
I guess it is a flaky bug rather than a flaky test? I put it on 2.11.0 to make
sure it gets another round of attention.
> Errors when running Python Game stats with a low fixed_window_duration in the
> DirectRunner
> --
>
> Key: BEAM-4518
> URL: https://issues.apache.org/jira/browse/BEAM-4518
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Ahmet Altay
>Assignee: Charles Chen
>Priority: Critical
> Labels: flake
> Fix For: 2.11.0
>
>
> Using the injector and the following command to start the DirectRunner
> pipeline:
> python -m apache_beam.examples.complete.game.game_stats \
> --project=google.com:clouddfe \
> --topic projects/google.com:clouddfe/topics/leader_board-$USER-topic-1 \
> --dataset ${USER}_test --fixed_window_duration 1
> Fails with:
> ValueError: PCollection of size 2 with more than one element accessed as a
> singleton view. First two elements encountered are "13.93", "11.78".
> [while running 'CalculateSpammyUsers/ProcessAndFilter']
> Offending code is here:
> global_mean_score = (
> sum_scores
> | beam.Values()
> | beam.CombineGlobally(beam.combiners.MeanCombineFn())\
> .as_singleton_view())
> # Filter the user sums using the global mean.
> filtered = (
> sum_scores
> # Use the derived mean total score (global_mean_score) as a side input.
> | 'ProcessAndFilter' >> beam.Filter(
> lambda key_score, global_mean:\
> key_score[1] > global_mean * self.SCORE_WEIGHT,
> global_mean_score))
> Since global_mean_score is the result of CombineGlobally, this is either an
> issue with CombineGlobally or side inputs implementation in DirectRunner. The
> latter is more likely since it works on DataflowRunner.
> cc: [~mariagh]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-6307) Nexmark Spark failed due to FileNotFoundException
[
https://issues.apache.org/jira/browse/BEAM-6307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-6307:
--
Fix Version/s: 2.11.0
> Nexmark Spark failed due to FileNotFoundException
> -
>
> Key: BEAM-6307
> URL: https://issues.apache.org/jira/browse/BEAM-6307
> Project: Beam
> Issue Type: Bug
> Components: examples-nexmark
>Reporter: Andrew Pilloud
>Priority: Major
> Labels: flake
> Fix For: 2.11.0
>
>
> Nexmark Spark failed due to FileNotFoundException.
> https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Spark/1417/consoleFull
> {code:java}
> 09:28:16 18/12/22 17:27:58 ERROR
> org.apache.spark.storage.DiskBlockObjectWriter: Uncaught exception while
> reverting partial writes to file
> /tmp/blockmgr-d4a681cf-2c36-4c8c-a238-cf16e4a89e73/12/temp_shuffle_c59ca414-109d-4c7b-b1d1-3514aa936662
> 09:28:16 java.io.FileNotFoundException:
> /tmp/blockmgr-d4a681cf-2c36-4c8c-a238-cf16e4a89e73/12/temp_shuffle_c59ca414-109d-4c7b-b1d1-3514aa936662
> (No such file or directory)
> 09:28:16 at java.io.FileOutputStream.open0(Native Method)
> 09:28:16 at java.io.FileOutputStream.open(FileOutputStream.java:270)
> 09:28:16 at java.io.FileOutputStream.(FileOutputStream.java:213)
> 09:28:16 at
> org.apache.spark.storage.DiskBlockObjectWriter$$anonfun$revertPartialWritesAndClose$2.apply$mcV$sp(DiskBlockObjectWriter.scala:217)
> 09:28:16 at
> org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1390)
> 09:28:16 at
> org.apache.spark.storage.DiskBlockObjectWriter.revertPartialWritesAndClose(DiskBlockObjectWriter.scala:214)
> 09:28:16 at
> org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.stop(BypassMergeSortShuffleWriter.java:237)
> 09:28:16 at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:102)
> 09:28:16 at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
> 09:28:16 at org.apache.spark.scheduler.Task.run(Task.scala:109)
> 09:28:16 at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
> 09:28:16 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 09:28:16 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 09:28:16 at java.lang.Thread.run(Thread.java:748)
> 09:28:16 18/12/22 17:27:58 ERROR
> org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter: Error while
> deleting file
> /tmp/blockmgr-d4a681cf-2c36-4c8c-a238-cf16e4a89e73/12/temp_shuffle_c59ca414-109d-4c7b-b1d1-3514aa936662
> 09:28:16 18/12/22 17:27:58 ERROR
> org.apache.spark.storage.DiskBlockObjectWriter: Uncaught exception while
> reverting partial writes to file
> /tmp/blockmgr-d4a681cf-2c36-4c8c-a238-cf16e4a89e73/12/temp_shuffle_0f0fa603-712e-4e36-8ce8-e9770dbab93e
> 09:28:16 java.io.FileNotFoundException:
> /tmp/blockmgr-d4a681cf-2c36-4c8c-a238-cf16e4a89e73/12/temp_shuffle_0f0fa603-712e-4e36-8ce8-e9770dbab93e
> (No such file or directory)
> 09:28:16 at java.io.FileOutputStream.open0(Native Method)
> 09:28:16 at java.io.FileOutputStream.open(FileOutputStream.java:270)
> 09:28:16 at java.io.FileOutputStream.(FileOutputStream.java:213)
> 09:28:16 at
> org.apache.spark.storage.DiskBlockObjectWriter$$anonfun$revertPartialWritesAndClose$2.apply$mcV$sp(DiskBlockObjectWriter.scala:217)
> 09:28:16 at
> org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1390)
> 09:28:16 at
> org.apache.spark.storage.DiskBlockObjectWriter.revertPartialWritesAndClose(DiskBlockObjectWriter.scala:214)
> 09:28:16 at
> org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.stop(BypassMergeSortShuffleWriter.java:237)
> 09:28:16 at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:102)
> 09:28:16 at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
> 09:28:16 at org.apache.spark.scheduler.Task.run(Task.scala:109)
> 09:28:16 at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
> 09:28:16 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 09:28:16 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 09:28:16 at java.lang.Thread.run(Thread.java:748){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-4518) Errors when running Python Game stats with a low fixed_window_duration in the DirectRunner
[
https://issues.apache.org/jira/browse/BEAM-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-4518:
--
Fix Version/s: 2.11.0
> Errors when running Python Game stats with a low fixed_window_duration in the
> DirectRunner
> --
>
> Key: BEAM-4518
> URL: https://issues.apache.org/jira/browse/BEAM-4518
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Ahmet Altay
>Assignee: Charles Chen
>Priority: Critical
> Labels: flake
> Fix For: 2.11.0
>
>
> Using the injector and the following command to start the DirectRunner
> pipeline:
> python -m apache_beam.examples.complete.game.game_stats \
> --project=google.com:clouddfe \
> --topic projects/google.com:clouddfe/topics/leader_board-$USER-topic-1 \
> --dataset ${USER}_test --fixed_window_duration 1
> Fails with:
> ValueError: PCollection of size 2 with more than one element accessed as a
> singleton view. First two elements encountered are "13.93", "11.78".
> [while running 'CalculateSpammyUsers/ProcessAndFilter']
> Offending code is here:
> global_mean_score = (
> sum_scores
> | beam.Values()
> | beam.CombineGlobally(beam.combiners.MeanCombineFn())\
> .as_singleton_view())
> # Filter the user sums using the global mean.
> filtered = (
> sum_scores
> # Use the derived mean total score (global_mean_score) as a side input.
> | 'ProcessAndFilter' >> beam.Filter(
> lambda key_score, global_mean:\
> key_score[1] > global_mean * self.SCORE_WEIGHT,
> global_mean_score))
> Since global_mean_score is the result of CombineGlobally, this is either an
> issue with CombineGlobally or side inputs implementation in DirectRunner. The
> latter is more likely since it works on DataflowRunner.
> cc: [~mariagh]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-6318) beam-sdks-python:setupVirtualenv sometimes fails due to a pip flake "No matching distribution found"
[
https://issues.apache.org/jira/browse/BEAM-6318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743487#comment-16743487
]
Valentyn Tymofieiev commented on BEAM-6318:
---
[https://github.com/apache/beam/pull/7519] increases number of retries pip
should attempt.
If the nature of the failure is such that pip cannot access pypi, this PR
might help, because now pip will retry 10 times over the course of 4 min.
If the nature of the failure is such that pypi sends a valid but incorrect
response, the PR might not help, because pip won't consider that a retry is
necessary. In this case we may have to add a retry call to pip itself.
I'm going to close the issue until we see the next instance of this flake.
> beam-sdks-python:setupVirtualenv sometimes fails due to a pip flake "No
> matching distribution found"
>
>
> Key: BEAM-6318
> URL: https://issues.apache.org/jira/browse/BEAM-6318
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Andrew Pilloud
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: 2.11.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> https://builds.apache.org/job/beam_PreCommit_Python_Cron/752/
> {code}
> 16:00:43 Collecting pluggy<1.0,>=0.3.0 (from tox==3.0.0)
> 16:00:43 Could not find a version that satisfies the requirement
> pluggy<1.0,>=0.3.0 (from tox==3.0.0) (from versions: )
> 16:00:43 No matching distribution found for pluggy<1.0,>=0.3.0 (from
> tox==3.0.0)
> 16:00:46
> 16:00:46 > Task :beam-sdks-python:setupVirtualenv FAILED
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (BEAM-6318) beam-sdks-python:setupVirtualenv sometimes fails due to a pip flake "No matching distribution found"
[
https://issues.apache.org/jira/browse/BEAM-6318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Valentyn Tymofieiev closed BEAM-6318.
-
Resolution: Fixed
> beam-sdks-python:setupVirtualenv sometimes fails due to a pip flake "No
> matching distribution found"
>
>
> Key: BEAM-6318
> URL: https://issues.apache.org/jira/browse/BEAM-6318
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Andrew Pilloud
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: 2.11.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> https://builds.apache.org/job/beam_PreCommit_Python_Cron/752/
> {code}
> 16:00:43 Collecting pluggy<1.0,>=0.3.0 (from tox==3.0.0)
> 16:00:43 Could not find a version that satisfies the requirement
> pluggy<1.0,>=0.3.0 (from tox==3.0.0) (from versions: )
> 16:00:43 No matching distribution found for pluggy<1.0,>=0.3.0 (from
> tox==3.0.0)
> 16:00:46
> 16:00:46 > Task :beam-sdks-python:setupVirtualenv FAILED
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-6294) Use Flink's redistribute for reshuffle.
[ https://issues.apache.org/jira/browse/BEAM-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743484#comment-16743484 ] Thomas Weise commented on BEAM-6294: [~robertwb] [~mxm] we found that the non-native reshuffle expansion seems to have an issue. We saw records being dropped with Flink portable streaming. The issue went away with the switch to Flink rebalance. > Use Flink's redistribute for reshuffle. > --- > > Key: BEAM-6294 > URL: https://issues.apache.org/jira/browse/BEAM-6294 > Project: Beam > Issue Type: New Feature > Components: runner-flink, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.10.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Python needs to publish the URN over the FnAPI which is pretty easy, but > Flink also needs to ensure that the composite structure does not get fused. > Unlike with GBK, we can't assume all runners implement this as a primitive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743486#comment-16743486 ] Maximilian Michels commented on BEAM-4176: -- Awesome, thanks for the info. > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Assignee: Ankur Goenka >Priority: Major > Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 > PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png > > Time Spent: 36h 40m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-6318) beam-sdks-python:setupVirtualenv sometimes fails due to a pip flake "No matching distribution found"
[
https://issues.apache.org/jira/browse/BEAM-6318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Valentyn Tymofieiev updated BEAM-6318:
--
Summary: beam-sdks-python:setupVirtualenv sometimes fails due to a pip
flake "No matching distribution found" (was: beam-sdks-python:setupVirtualenv
pluggy flaky)
> beam-sdks-python:setupVirtualenv sometimes fails due to a pip flake "No
> matching distribution found"
>
>
> Key: BEAM-6318
> URL: https://issues.apache.org/jira/browse/BEAM-6318
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Andrew Pilloud
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: 2.11.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> https://builds.apache.org/job/beam_PreCommit_Python_Cron/752/
> {code}
> 16:00:43 Collecting pluggy<1.0,>=0.3.0 (from tox==3.0.0)
> 16:00:43 Could not find a version that satisfies the requirement
> pluggy<1.0,>=0.3.0 (from tox==3.0.0) (from versions: )
> 16:00:43 No matching distribution found for pluggy<1.0,>=0.3.0 (from
> tox==3.0.0)
> 16:00:46
> 16:00:46 > Task :beam-sdks-python:setupVirtualenv FAILED
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6318) beam-sdks-python:setupVirtualenv pluggy flaky
[
https://issues.apache.org/jira/browse/BEAM-6318?focusedWorklogId=185513=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185513
]
ASF GitHub Bot logged work on BEAM-6318:
Author: ASF GitHub Bot
Created on: 16/Jan/19 00:17
Start Date: 16/Jan/19 00:17
Worklog Time Spent: 10m
Work Description: aaltay commented on pull request #7519: [BEAM-6318]
Increase the amount of pip download retries to 10 from a default 5
URL: https://github.com/apache/beam/pull/7519
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 185513)
Time Spent: 0.5h (was: 20m)
> beam-sdks-python:setupVirtualenv pluggy flaky
> -
>
> Key: BEAM-6318
> URL: https://issues.apache.org/jira/browse/BEAM-6318
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Andrew Pilloud
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: 2.11.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> https://builds.apache.org/job/beam_PreCommit_Python_Cron/752/
> {code}
> 16:00:43 Collecting pluggy<1.0,>=0.3.0 (from tox==3.0.0)
> 16:00:43 Could not find a version that satisfies the requirement
> pluggy<1.0,>=0.3.0 (from tox==3.0.0) (from versions: )
> 16:00:43 No matching distribution found for pluggy<1.0,>=0.3.0 (from
> tox==3.0.0)
> 16:00:46
> 16:00:46 > Task :beam-sdks-python:setupVirtualenv FAILED
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6444) Equality tests in coders.py are broken
[ https://issues.apache.org/jira/browse/BEAM-6444?focusedWorklogId=185511=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185511 ] ASF GitHub Bot logged work on BEAM-6444: Author: ASF GitHub Bot Created on: 16/Jan/19 00:09 Start Date: 16/Jan/19 00:09 Worklog Time Spent: 10m Work Description: takidau commented on issue #7522: [BEAM-6444] Fix equality comparison to be against method invocation, not method object URL: https://github.com/apache/beam/pull/7522#issuecomment-454600314 @aaltay This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185511) Time Spent: 20m (was: 10m) > Equality tests in coders.py are broken > -- > > Key: BEAM-6444 > URL: https://issues.apache.org/jira/browse/BEAM-6444 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Tyler Akidau >Assignee: Tyler Akidau >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Equality checks for TupleSequenceCoder and IterableCoder were broken recently > with > [https://github.com/apache/beam/commit/ffec4716b6db6802a5ff54ee6d6d7fd4e764e8b6.] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6444) Equality tests in coders.py are broken
[ https://issues.apache.org/jira/browse/BEAM-6444?focusedWorklogId=185510=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185510 ] ASF GitHub Bot logged work on BEAM-6444: Author: ASF GitHub Bot Created on: 16/Jan/19 00:08 Start Date: 16/Jan/19 00:08 Worklog Time Spent: 10m Work Description: takidau commented on pull request #7522: [BEAM-6444] Fix equality comparison to be against method invocation, not method object URL: https://github.com/apache/beam/pull/7522 Fixes broken equality comparison introduced in https://github.com/apache/beam/commit/ffec4716b6db6802a5ff54ee6d6d7fd4e764e8b6. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185510) Time Spent: 10m Remaining Estimate: 0h > Equality tests in coders.py are broken > -- > > Key: BEAM-6444 > URL: https://issues.apache.org/jira/browse/BEAM-6444 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Tyler Akidau >Assignee: Tyler Akidau >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Equality checks for TupleSequenceCoder and IterableCoder were broken recently > with >
[jira] [Created] (BEAM-6444) Equality tests in coders.py are broken
Tyler Akidau created BEAM-6444: -- Summary: Equality tests in coders.py are broken Key: BEAM-6444 URL: https://issues.apache.org/jira/browse/BEAM-6444 Project: Beam Issue Type: Bug Components: sdk-py-core Reporter: Tyler Akidau Assignee: Tyler Akidau Equality checks for TupleSequenceCoder and IterableCoder were broken recently with [https://github.com/apache/beam/commit/ffec4716b6db6802a5ff54ee6d6d7fd4e764e8b6.] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5953) Support DataflowRunner on Python 3
[ https://issues.apache.org/jira/browse/BEAM-5953?focusedWorklogId=185502=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185502 ] ASF GitHub Bot logged work on BEAM-5953: Author: ASF GitHub Bot Created on: 16/Jan/19 00:07 Start Date: 16/Jan/19 00:07 Worklog Time Spent: 10m Work Description: markflyhigh commented on pull request #7521: [BEAM-5953] Fix py3 type error in bundle_processor URL: https://github.com/apache/beam/pull/7521 This is one step to enable Python 3 pipeline running on DataflowRunner. This change is to fix error in [BEAM-5953#comment](https://issues.apache.org/jira/browse/BEAM-5953?focusedCommentId=16702212=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16702212). Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please
[jira] [Work logged] (BEAM-6258) Data channel failing after some time for 1G data input
[ https://issues.apache.org/jira/browse/BEAM-6258?focusedWorklogId=185500=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185500 ] ASF GitHub Bot logged work on BEAM-6258: Author: ASF GitHub Bot Created on: 16/Jan/19 00:06 Start Date: 16/Jan/19 00:06 Worklog Time Spent: 10m Work Description: angoenka commented on issue #7415: [BEAM-6258] Set grpc keep alive on server creation URL: https://github.com/apache/beam/pull/7415#issuecomment-454599766 Thanks mxm! Merging it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185500) Time Spent: 2h (was: 1h 50m) > Data channel failing after some time for 1G data input > -- > > Key: BEAM-6258 > URL: https://issues.apache.org/jira/browse/BEAM-6258 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Attachments: d44b7eda9e4c_java_server_logs.logs.gz, > d44b7eda9e4c_python_client_logs.log.bz2 > > Time Spent: 2h > Remaining Estimate: 0h > > Data channel and logging channel are failing after some time with 1GB input > data for chicago taxi. > > E1218 02:44:02.837680206 72 chttp2_transport.cc:1148] Received a GOAWAY with > error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings" > Exception in thread read_grpc_client_inputs: > Traceback (most recent call last): > File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner > self.run() > File "/usr/local/lib/python2.7/threading.py", line 754, in run > self.__target(*self.__args, **self.__kwargs) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 273, in > target=lambda: self._read_inputs(elements_iterator), > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File >
[jira] [Work logged] (BEAM-6258) Data channel failing after some time for 1G data input
[ https://issues.apache.org/jira/browse/BEAM-6258?focusedWorklogId=185501=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185501 ] ASF GitHub Bot logged work on BEAM-6258: Author: ASF GitHub Bot Created on: 16/Jan/19 00:06 Start Date: 16/Jan/19 00:06 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #7415: [BEAM-6258] Set grpc keep alive on server creation URL: https://github.com/apache/beam/pull/7415 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185501) Time Spent: 2h 10m (was: 2h) > Data channel failing after some time for 1G data input > -- > > Key: BEAM-6258 > URL: https://issues.apache.org/jira/browse/BEAM-6258 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Attachments: d44b7eda9e4c_java_server_logs.logs.gz, > d44b7eda9e4c_python_client_logs.log.bz2 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Data channel and logging channel are failing after some time with 1GB input > data for chicago taxi. > > E1218 02:44:02.837680206 72 chttp2_transport.cc:1148] Received a GOAWAY with > error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings" > Exception in thread read_grpc_client_inputs: > Traceback (most recent call last): > File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner > self.run() > File "/usr/local/lib/python2.7/threading.py", line 754, in run > self.__target(*self.__args, **self.__kwargs) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 273, in > target=lambda: self._read_inputs(elements_iterator), > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File >
[jira] [Assigned] (BEAM-6158) Worker failed with save_main_session enabled on Dataflow in Python 3
[
https://issues.apache.org/jira/browse/BEAM-6158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Liu reassigned BEAM-6158:
--
Assignee: Robert Bradshaw
> Worker failed with save_main_session enabled on Dataflow in Python 3
>
>
> Key: BEAM-6158
> URL: https://issues.apache.org/jira/browse/BEAM-6158
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-harness
>Reporter: Mark Liu
>Assignee: Robert Bradshaw
>Priority: Major
>
> This happened when I run wordcount example with portable Dataflow runner in
> Python 3.5. The failure shows in worker log (unfortunately unformatted) of
> [this
> job|https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-29_11_47_38-6731484595556255542?project=google.com:clouddfe]:
> {code}
> Could not load main session: Traceback (most recent call last): File
> "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker_main.py",
> line 125, in main _load_main_session(semi_persistent_directory) File
> "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker_main.py",
> line 201, in _load_main_session pickler.load_session(session_file) File
> "/usr/local/lib/python3.5/site-packages/apache_beam/internal/pickler.py",
> line 269, in load_session return dill.load_session(file_path) File
> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 402, in
> load_session module = unpickler.load() File
> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 465, in
> find_class return StockUnpickler.find_class(self, module, name)
> AttributeError: Can't get attribute 'WordExtractingDoFn' on 'apache_beam.runners.worker.sdk_worker_main' from
> '/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker_main.py'>
> Traceback (most recent call last): File
> "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker_main.py",
> line 125, in main _load_main_session(semi_persistent_directory) File
> "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker_main.py",
> line 201, in _load_main_session pickler.load_session(session_file) File
> "/usr/local/lib/python3.5/site-packages/apache_beam/internal/pickler.py",
> line 269, in load_session return dill.load_session(file_path) File
> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 402, in
> load_session module = unpickler.load() File
> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 465, in
> find_class return StockUnpickler.find_class(self, module, name)
> AttributeError: Can't get attribute 'WordExtractingDoFn' on 'apache_beam.runners.worker.sdk_worker_main' from
> '/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker_main.py'>
> {code}
> In order to get this error, I need to patch few type error fixes first
> ([PR-7051|https://github.com/apache/beam/pull/7051] and
> [PR-4|https://github.com/markflyhigh/incubator-beam/pull/4]).
> Looks like saved main session didn't work properly in Python 3. A recent
> [PR-7104|https://github.com/apache/beam/pull/7104] mentioned [an
> error|https://github.com/uqfoundation/dill/pull/294] in dill probably related
> to this problem but I didn't verify yet.
> +cc: [~tvalentyn] [~robertwb] [~altay]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-6443) decrease the number of thread for BigQuery streaming insertAll
Heejong Lee created BEAM-6443: - Summary: decrease the number of thread for BigQuery streaming insertAll Key: BEAM-6443 URL: https://issues.apache.org/jira/browse/BEAM-6443 Project: Beam Issue Type: Improvement Components: io-java-gcp Reporter: Heejong Lee Assignee: Heejong Lee When inserting (a large number of ) very small elements into BigQuery via streaming insertAll, BigQueryIO causes lots of quota exceeded errors. This implies that 1) BigQueryIO puts unnecessary overheads on BigQuery API layer by sending requests too fast 2) log file becomes very big because of repeated same error messages. Currently we use 50 shards for writing data into BigQuery and in each bundle 20-30 futures are executed simultaneously with unlimited thread pool. It would be worth investigating whether just single thread pool is sufficient for running concurrent insertAll. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
