[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[
https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16729932#comment-16729932
]
yifan zou commented on BEAM-5811:
-
_Workflow failed. Causes: The Dataflow job appears to be stuck because no
worker activity has been seen in the last 1h. You can get help with Cloud
Dataflow at [https://cloud.google.com/dataflow/support.]_
*There are errors repeat in the worker log, which cause the worker failed to
start:*
_Error syncing pod 30a7faf11edde2d98e5bf5992f52eb63
("dataflow-beamapp-jenkins-122719345-12271141-5f05-harness-41l1_default(30a7faf11edde2d98e5bf5992f52eb63)"),
skipping: failed to "StartContainer" for "python" with CrashLoopBackOff:
"Back-off 5m0s restarting failed container=python
pod=dataflow-beamapp-jenkins-122719345-12271141-5f05-harness-41l1_default(30a7faf11edde2d98e5bf5992f52eb63)"_
Stack driver:
https://pantheon.corp.google.com/logs/viewer?resource=dataflow_step%2Fjob_id%2F2018-12-27_11_41_05-5516523713456997908&interval=NO_LIMIT&project=apache-beam-testing&minLogLevel=0&expandAll=false×tamp=2018-12-27T23:08:47.76700Z&customFacets=&limitCustomFacetWidth=true&scrollTimestamp=2018-12-27T20:38:56.483664000Z&dateRangeUnbound=backwardInTime
> Timeout in Python datastore_write_it_test.DatastoreWriteIT
> --
>
> Key: BEAM-5811
> URL: https://issues.apache.org/jira/browse/BEAM-5811
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Kenneth Knowles
>Assignee: yifan zou
>Priority: Critical
> Labels: flake
>
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/]
> [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests]
> {code:java}
> TimedOutException: 'test_datastore_write_limit
> (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[
https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16729979#comment-16729979
]
yifan zou commented on BEAM-5811:
-
Errors occurs when reading metadata of some google-dockercfg package:
while reading 'google-dockercfg-url' metadata: http status code: 404 while
fetching url
[http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url]
"Failed to install packages: failed to install SDK: exit status 1"
> Timeout in Python datastore_write_it_test.DatastoreWriteIT
> --
>
> Key: BEAM-5811
> URL: https://issues.apache.org/jira/browse/BEAM-5811
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Kenneth Knowles
>Assignee: yifan zou
>Priority: Critical
> Labels: flake
>
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/]
> [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests]
> {code:java}
> TimedOutException: 'test_datastore_write_limit
> (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[
https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743505#comment-16743505
]
Kenneth Knowles commented on BEAM-5811:
---
Are you actively working on this? Maybe release the bug so someone going around
deflaking will try it and see.
> Timeout in Python datastore_write_it_test.DatastoreWriteIT
> --
>
> Key: BEAM-5811
> URL: https://issues.apache.org/jira/browse/BEAM-5811
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Kenneth Knowles
>Assignee: yifan zou
>Priority: Critical
> Labels: flake
> Fix For: 2.11.0
>
>
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/]
> [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests]
> {code:java}
> TimedOutException: 'test_datastore_write_limit
> (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[
https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743619#comment-16743619
]
yifan zou commented on BEAM-5811:
-
released the bug with unassigned.
> Timeout in Python datastore_write_it_test.DatastoreWriteIT
> --
>
> Key: BEAM-5811
> URL: https://issues.apache.org/jira/browse/BEAM-5811
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Kenneth Knowles
>Priority: Critical
> Labels: flake
> Fix For: 2.11.0
>
>
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/]
> [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests]
> {code:java}
> TimedOutException: 'test_datastore_write_limit
> (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[
https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16729926#comment-16729926
]
Andrew Pilloud commented on BEAM-5811:
--
Looks like this is now hitting a 3600s timeout on dataflow.
https://builds.apache.org/job/beam_PostCommit_Python_Verify/6933/consoleFull
> Timeout in Python datastore_write_it_test.DatastoreWriteIT
> --
>
> Key: BEAM-5811
> URL: https://issues.apache.org/jira/browse/BEAM-5811
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Kenneth Knowles
>Assignee: yifan zou
>Priority: Critical
> Labels: flake
>
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/]
> [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests]
> {code:java}
> TimedOutException: 'test_datastore_write_limit
> (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[
https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16768867#comment-16768867
]
Ahmet Altay commented on BEAM-5811:
---
cc: [~chamikara]
> Timeout in Python datastore_write_it_test.DatastoreWriteIT
> --
>
> Key: BEAM-5811
> URL: https://issues.apache.org/jira/browse/BEAM-5811
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Kenneth Knowles
>Priority: Critical
> Labels: flake
>
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/]
> [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests]
> {code:java}
> TimedOutException: 'test_datastore_write_limit
> (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[
https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659032#comment-16659032
]
Kenneth Knowles commented on BEAM-5811:
---
Assignee just because code history looks like you might have most recent memory
of the test. What do you think?
> Timeout in Python datastore_write_it_test.DatastoreWriteIT
> --
>
> Key: BEAM-5811
> URL: https://issues.apache.org/jira/browse/BEAM-5811
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Kenneth Knowles
>Assignee: yifan zou
>Priority: Critical
> Labels: flake
>
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/]
> [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests]
> {code:java}
> TimedOutException: 'test_datastore_write_limit
> (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[
https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659101#comment-16659101
]
yifan zou commented on BEAM-5811:
-
Looks like a flaky. This test includes 5 pipelines run on the Dataflow. It
takes lots of time to start up and tear down the worker, and it exceeds the
duration threshold this time. If it continuously happens, we should consider to
disable this test from the PostCommitITTests.
> Timeout in Python datastore_write_it_test.DatastoreWriteIT
> --
>
> Key: BEAM-5811
> URL: https://issues.apache.org/jira/browse/BEAM-5811
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Kenneth Knowles
>Assignee: yifan zou
>Priority: Critical
> Labels: flake
>
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/]
> [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests]
> {code:java}
> TimedOutException: 'test_datastore_write_limit
> (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[
https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665626#comment-16665626
]
Henning Rohde commented on BEAM-5811:
-
Happened again:
https://scans.gradle.com/s/whyjrsgtrhme4/console-log?task=:beam-sdks-python:postCommitITTests#L219
> Timeout in Python datastore_write_it_test.DatastoreWriteIT
> --
>
> Key: BEAM-5811
> URL: https://issues.apache.org/jira/browse/BEAM-5811
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Kenneth Knowles
>Assignee: yifan zou
>Priority: Critical
> Labels: flake
>
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/]
> [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests]
> {code:java}
> TimedOutException: 'test_datastore_write_limit
> (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[
https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667470#comment-16667470
]
yifan zou commented on BEAM-5811:
-
Increasing python time limit from 30 minutes to 45 minutes. It would prevent
the timeout issue in datastore tests.
[https://github.com/apache/beam/pull/6833/files#diff-5924c115647bc4008928b5855e17d635R218]
> Timeout in Python datastore_write_it_test.DatastoreWriteIT
> --
>
> Key: BEAM-5811
> URL: https://issues.apache.org/jira/browse/BEAM-5811
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Kenneth Knowles
>Assignee: yifan zou
>Priority: Critical
> Labels: flake
>
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/]
> [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests]
> {code:java}
> TimedOutException: 'test_datastore_write_limit
> (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[
https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17311927#comment-17311927
]
Brian Hulette commented on BEAM-5811:
-
What suite is exercising this now? Is it beamPostCommitPython{36,37,38}?
I don't see any recent flakes in those, e.g.:
https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Python36/lastCompletedBuild/testReport/junit/apache_beam.io.gcp.datastore.v1new.datastore_write_it_test/DatastoreWriteIT/history/
Given that and the fact no one has reported a flake here recently I think this
is safe to close.
> Timeout in Python datastore_write_it_test.DatastoreWriteIT
> --
>
> Key: BEAM-5811
> URL: https://issues.apache.org/jira/browse/BEAM-5811
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Kenneth Knowles
>Priority: P1
> Labels: beam-fixit, flake
>
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/]
> [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests]
> {code:java}
> TimedOutException: 'test_datastore_write_limit
> (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
