[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[ https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16729932#comment-16729932 ] yifan zou commented on BEAM-5811: - _Workflow failed. Causes: The Dataflow job appears to be stuck because no worker activity has been seen in the last 1h. You can get help with Cloud Dataflow at [https://cloud.google.com/dataflow/support.]_ *There are errors repeat in the worker log, which cause the worker failed to start:* _Error syncing pod 30a7faf11edde2d98e5bf5992f52eb63 ("dataflow-beamapp-jenkins-122719345-12271141-5f05-harness-41l1_default(30a7faf11edde2d98e5bf5992f52eb63)"), skipping: failed to "StartContainer" for "python" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=python pod=dataflow-beamapp-jenkins-122719345-12271141-5f05-harness-41l1_default(30a7faf11edde2d98e5bf5992f52eb63)"_ Stack driver: https://pantheon.corp.google.com/logs/viewer?resource=dataflow_step%2Fjob_id%2F2018-12-27_11_41_05-5516523713456997908&interval=NO_LIMIT&project=apache-beam-testing&minLogLevel=0&expandAll=false×tamp=2018-12-27T23:08:47.76700Z&customFacets=&limitCustomFacetWidth=true&scrollTimestamp=2018-12-27T20:38:56.483664000Z&dateRangeUnbound=backwardInTime > Timeout in Python datastore_write_it_test.DatastoreWriteIT > -- > > Key: BEAM-5811 > URL: https://issues.apache.org/jira/browse/BEAM-5811 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Kenneth Knowles >Assignee: yifan zou >Priority: Critical > Labels: flake > > [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/] > [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests] > {code:java} > TimedOutException: 'test_datastore_write_limit > (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[ https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16729979#comment-16729979 ] yifan zou commented on BEAM-5811: - Errors occurs when reading metadata of some google-dockercfg package: while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url [http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url] "Failed to install packages: failed to install SDK: exit status 1" > Timeout in Python datastore_write_it_test.DatastoreWriteIT > -- > > Key: BEAM-5811 > URL: https://issues.apache.org/jira/browse/BEAM-5811 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Kenneth Knowles >Assignee: yifan zou >Priority: Critical > Labels: flake > > [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/] > [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests] > {code:java} > TimedOutException: 'test_datastore_write_limit > (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[ https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743505#comment-16743505 ] Kenneth Knowles commented on BEAM-5811: --- Are you actively working on this? Maybe release the bug so someone going around deflaking will try it and see. > Timeout in Python datastore_write_it_test.DatastoreWriteIT > -- > > Key: BEAM-5811 > URL: https://issues.apache.org/jira/browse/BEAM-5811 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Kenneth Knowles >Assignee: yifan zou >Priority: Critical > Labels: flake > Fix For: 2.11.0 > > > [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/] > [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests] > {code:java} > TimedOutException: 'test_datastore_write_limit > (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[ https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743619#comment-16743619 ] yifan zou commented on BEAM-5811: - released the bug with unassigned. > Timeout in Python datastore_write_it_test.DatastoreWriteIT > -- > > Key: BEAM-5811 > URL: https://issues.apache.org/jira/browse/BEAM-5811 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Kenneth Knowles >Priority: Critical > Labels: flake > Fix For: 2.11.0 > > > [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/] > [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests] > {code:java} > TimedOutException: 'test_datastore_write_limit > (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[ https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16729926#comment-16729926 ] Andrew Pilloud commented on BEAM-5811: -- Looks like this is now hitting a 3600s timeout on dataflow. https://builds.apache.org/job/beam_PostCommit_Python_Verify/6933/consoleFull > Timeout in Python datastore_write_it_test.DatastoreWriteIT > -- > > Key: BEAM-5811 > URL: https://issues.apache.org/jira/browse/BEAM-5811 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Kenneth Knowles >Assignee: yifan zou >Priority: Critical > Labels: flake > > [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/] > [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests] > {code:java} > TimedOutException: 'test_datastore_write_limit > (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[ https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16768867#comment-16768867 ] Ahmet Altay commented on BEAM-5811: --- cc: [~chamikara] > Timeout in Python datastore_write_it_test.DatastoreWriteIT > -- > > Key: BEAM-5811 > URL: https://issues.apache.org/jira/browse/BEAM-5811 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Kenneth Knowles >Priority: Critical > Labels: flake > > [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/] > [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests] > {code:java} > TimedOutException: 'test_datastore_write_limit > (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[ https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659032#comment-16659032 ] Kenneth Knowles commented on BEAM-5811: --- Assignee just because code history looks like you might have most recent memory of the test. What do you think? > Timeout in Python datastore_write_it_test.DatastoreWriteIT > -- > > Key: BEAM-5811 > URL: https://issues.apache.org/jira/browse/BEAM-5811 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Kenneth Knowles >Assignee: yifan zou >Priority: Critical > Labels: flake > > [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/] > [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests] > {code:java} > TimedOutException: 'test_datastore_write_limit > (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[ https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659101#comment-16659101 ] yifan zou commented on BEAM-5811: - Looks like a flaky. This test includes 5 pipelines run on the Dataflow. It takes lots of time to start up and tear down the worker, and it exceeds the duration threshold this time. If it continuously happens, we should consider to disable this test from the PostCommitITTests. > Timeout in Python datastore_write_it_test.DatastoreWriteIT > -- > > Key: BEAM-5811 > URL: https://issues.apache.org/jira/browse/BEAM-5811 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Kenneth Knowles >Assignee: yifan zou >Priority: Critical > Labels: flake > > [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/] > [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests] > {code:java} > TimedOutException: 'test_datastore_write_limit > (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[ https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665626#comment-16665626 ] Henning Rohde commented on BEAM-5811: - Happened again: https://scans.gradle.com/s/whyjrsgtrhme4/console-log?task=:beam-sdks-python:postCommitITTests#L219 > Timeout in Python datastore_write_it_test.DatastoreWriteIT > -- > > Key: BEAM-5811 > URL: https://issues.apache.org/jira/browse/BEAM-5811 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Kenneth Knowles >Assignee: yifan zou >Priority: Critical > Labels: flake > > [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/] > [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests] > {code:java} > TimedOutException: 'test_datastore_write_limit > (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[ https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667470#comment-16667470 ] yifan zou commented on BEAM-5811: - Increasing python time limit from 30 minutes to 45 minutes. It would prevent the timeout issue in datastore tests. [https://github.com/apache/beam/pull/6833/files#diff-5924c115647bc4008928b5855e17d635R218] > Timeout in Python datastore_write_it_test.DatastoreWriteIT > -- > > Key: BEAM-5811 > URL: https://issues.apache.org/jira/browse/BEAM-5811 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Kenneth Knowles >Assignee: yifan zou >Priority: Critical > Labels: flake > > [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/] > [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests] > {code:java} > TimedOutException: 'test_datastore_write_limit > (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[ https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17311927#comment-17311927 ] Brian Hulette commented on BEAM-5811: - What suite is exercising this now? Is it beamPostCommitPython{36,37,38}? I don't see any recent flakes in those, e.g.: https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Python36/lastCompletedBuild/testReport/junit/apache_beam.io.gcp.datastore.v1new.datastore_write_it_test/DatastoreWriteIT/history/ Given that and the fact no one has reported a flake here recently I think this is safe to close. > Timeout in Python datastore_write_it_test.DatastoreWriteIT > -- > > Key: BEAM-5811 > URL: https://issues.apache.org/jira/browse/BEAM-5811 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Kenneth Knowles >Priority: P1 > Labels: beam-fixit, flake > > [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/] > [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests] > {code:java} > TimedOutException: 'test_datastore_write_limit > (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code} -- This message was sent by Atlassian Jira (v8.3.4#803005)