[jira] [Updated] (BEAM-14548) Jenkins job beam_SeedJob keeps failing
[
https://issues.apache.org/jira/browse/BEAM-14548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmet Altay updated BEAM-14548:
---
Fix Version/s: Not applicable
Resolution: Fixed
Status: Resolved (was: Open)
> Jenkins job beam_SeedJob keeps failing
> --
>
> Key: BEAM-14548
> URL: https://issues.apache.org/jira/browse/BEAM-14548
> Project: Beam
> Issue Type: Bug
> Components: infrastructure, testing
>Reporter: Alexey Romanenko
>Assignee: Ryan Thompson
>Priority: P0
> Labels: jenkins
> Fix For: Not applicable
>
>
> Jenkins job [beam_SeedJob|https://ci-beam.apache.org/job/beam_SeedJob/] keeps
> failing starting from May 19th, the [last successful
> build|https://ci-beam.apache.org/job/beam_SeedJob/lastSuccessfulBuild/].
> The first failed job is [https://ci-beam.apache.org/job/beam_SeedJob/9696/]
> It fails with this error (that says not so much):
>
> {code}
> Processing DSL script .test-infra/jenkins/job_00_seed.groovy
> Processing DSL script .test-infra/jenkins/job_CancelStaleDataflowJobs.groovy
> Processing DSL script .test-infra/jenkins/job_CleanUpPrebuiltSDKImages.groovy
> Processing DSL script .test-infra/jenkins/job_Dependency_Check.groovy
> ERROR: java.io.IOException: Failed to persist config.xml
> {code}
> It blocs a testing and development process to add the new Jenkins jobs with
> DSL
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (BEAM-14548) Jenkins job beam_SeedJob keeps failing
[
https://issues.apache.org/jira/browse/BEAM-14548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17545730#comment-17545730
]
Ahmet Altay commented on BEAM-14548:
Seed job looks healthy now: https://ci-beam.apache.org/job/beam_SeedJob/9762/
> Jenkins job beam_SeedJob keeps failing
> --
>
> Key: BEAM-14548
> URL: https://issues.apache.org/jira/browse/BEAM-14548
> Project: Beam
> Issue Type: Bug
> Components: infrastructure, testing
>Reporter: Alexey Romanenko
>Assignee: Ryan Thompson
>Priority: P0
> Labels: jenkins
>
> Jenkins job [beam_SeedJob|https://ci-beam.apache.org/job/beam_SeedJob/] keeps
> failing starting from May 19th, the [last successful
> build|https://ci-beam.apache.org/job/beam_SeedJob/lastSuccessfulBuild/].
> The first failed job is [https://ci-beam.apache.org/job/beam_SeedJob/9696/]
> It fails with this error (that says not so much):
>
> {code}
> Processing DSL script .test-infra/jenkins/job_00_seed.groovy
> Processing DSL script .test-infra/jenkins/job_CancelStaleDataflowJobs.groovy
> Processing DSL script .test-infra/jenkins/job_CleanUpPrebuiltSDKImages.groovy
> Processing DSL script .test-infra/jenkins/job_Dependency_Check.groovy
> ERROR: java.io.IOException: Failed to persist config.xml
> {code}
> It blocs a testing and development process to add the new Jenkins jobs with
> DSL
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Assigned] (BEAM-14548) Jenkins job beam_SeedJob keeps failing
[
https://issues.apache.org/jira/browse/BEAM-14548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmet Altay reassigned BEAM-14548:
--
Assignee: Ryan Thompson
> Jenkins job beam_SeedJob keeps failing
> --
>
> Key: BEAM-14548
> URL: https://issues.apache.org/jira/browse/BEAM-14548
> Project: Beam
> Issue Type: Bug
> Components: infrastructure, testing
>Reporter: Alexey Romanenko
>Assignee: Ryan Thompson
>Priority: P1
> Labels: jenkins
>
> Jenkins job [beam_SeedJob|https://ci-beam.apache.org/job/beam_SeedJob/] keeps
> failing starting from May 19th, the [last successful
> build|https://ci-beam.apache.org/job/beam_SeedJob/lastSuccessfulBuild/].
> The first failed job is [https://ci-beam.apache.org/job/beam_SeedJob/9696/]
> It fails with this error (that says not so much):
>
> {code}
> Processing DSL script .test-infra/jenkins/job_00_seed.groovy
> Processing DSL script .test-infra/jenkins/job_CancelStaleDataflowJobs.groovy
> Processing DSL script .test-infra/jenkins/job_CleanUpPrebuiltSDKImages.groovy
> Processing DSL script .test-infra/jenkins/job_Dependency_Check.groovy
> ERROR: java.io.IOException: Failed to persist config.xml
> {code}
> It blocs a testing and development process to add the new Jenkins jobs with
> DSL
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (BEAM-14548) Jenkins job beam_SeedJob keeps failing
[
https://issues.apache.org/jira/browse/BEAM-14548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmet Altay updated BEAM-14548:
---
Priority: P0 (was: P1)
> Jenkins job beam_SeedJob keeps failing
> --
>
> Key: BEAM-14548
> URL: https://issues.apache.org/jira/browse/BEAM-14548
> Project: Beam
> Issue Type: Bug
> Components: infrastructure, testing
>Reporter: Alexey Romanenko
>Assignee: Ryan Thompson
>Priority: P0
> Labels: jenkins
>
> Jenkins job [beam_SeedJob|https://ci-beam.apache.org/job/beam_SeedJob/] keeps
> failing starting from May 19th, the [last successful
> build|https://ci-beam.apache.org/job/beam_SeedJob/lastSuccessfulBuild/].
> The first failed job is [https://ci-beam.apache.org/job/beam_SeedJob/9696/]
> It fails with this error (that says not so much):
>
> {code}
> Processing DSL script .test-infra/jenkins/job_00_seed.groovy
> Processing DSL script .test-infra/jenkins/job_CancelStaleDataflowJobs.groovy
> Processing DSL script .test-infra/jenkins/job_CleanUpPrebuiltSDKImages.groovy
> Processing DSL script .test-infra/jenkins/job_Dependency_Check.groovy
> ERROR: java.io.IOException: Failed to persist config.xml
> {code}
> It blocs a testing and development process to add the new Jenkins jobs with
> DSL
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (BEAM-14456) Use Go 1.18.2 to build 2.39 Container Bootloaders
[ https://issues.apache.org/jira/browse/BEAM-14456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534585#comment-17534585 ] Ahmet Altay commented on BEAM-14456: Added this to the release blockers list (set the fix version) based on the description. If this was not the intention, please unset the fix version field. > Use Go 1.18.2 to build 2.39 Container Bootloaders > -- > > Key: BEAM-14456 > URL: https://issues.apache.org/jira/browse/BEAM-14456 > Project: Beam > Issue Type: Bug > Components: sdk-go, sdk-java-core, sdk-py-core >Affects Versions: 2.39.0 >Reporter: Robert Burke >Assignee: Robert Burke >Priority: P2 > Fix For: 2.39.0 > > Time Spent: 20m > Remaining Estimate: 0h > > It's been noted that by using older Go releases to compile Go containers we > run the risk of the bootloaders using vulnerable versions. > This issue is to close the gap for 2.39, while a separate one is to document > the policy of keeping the release artifacts built with the latest Go version. > While it's unlikely to be an attack vector, it's prudent that we keep these > gaps as closed as we're able. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (BEAM-14456) Use Go 1.18.2 to build 2.39 Container Bootloaders
[ https://issues.apache.org/jira/browse/BEAM-14456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay updated BEAM-14456: --- Fix Version/s: 2.39.0 > Use Go 1.18.2 to build 2.39 Container Bootloaders > -- > > Key: BEAM-14456 > URL: https://issues.apache.org/jira/browse/BEAM-14456 > Project: Beam > Issue Type: Bug > Components: sdk-go, sdk-java-core, sdk-py-core >Affects Versions: 2.39.0 >Reporter: Robert Burke >Assignee: Robert Burke >Priority: P2 > Fix For: 2.39.0 > > Time Spent: 20m > Remaining Estimate: 0h > > It's been noted that by using older Go releases to compile Go containers we > run the risk of the bootloaders using vulnerable versions. > This issue is to close the gap for 2.39, while a separate one is to document > the policy of keeping the release artifacts built with the latest Go version. > While it's unlikely to be an attack vector, it's prudent that we keep these > gaps as closed as we're able. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (BEAM-14412) Block release on impersonation FR
[ https://issues.apache.org/jira/browse/BEAM-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay reassigned BEAM-14412: -- Assignee: Kenneth Knowles > Block release on impersonation FR > - > > Key: BEAM-14412 > URL: https://issues.apache.org/jira/browse/BEAM-14412 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Affects Versions: 2.39.0 >Reporter: Ryan Thompson >Assignee: Kenneth Knowles >Priority: P1 > Fix For: 2.39.0 > > > Two PR's are high value PRs that we would like to block the release on. > > The FR is to add impersonation credentials. > The blocking PRs are: > [https://github.com/apache/beam/pull/17394] > [https://github.com/apache/beam/pull/17244] > > See FR at BEAM-14014 > h1. BEAM-14014 > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (BEAM-14403) Allow Prime to be used with Legacy workers
[ https://issues.apache.org/jira/browse/BEAM-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17531986#comment-17531986 ] Ahmet Altay commented on BEAM-14403: [~kenn][~robertwb] Is there something we could do to prevent an issue like this in the future? In an ideal world SDK could leave these types of choices to the service. And if SDK has to make a choice it could distinguish between options SDK set vs the options a user explicitly set. We could check for such things in code review, but beyond that is there a way for us to check if SDK is setting options that is more than necessary in a more automated way. (I might be overthinking this and this might be a single example.) > Allow Prime to be used with Legacy workers > -- > > Key: BEAM-14403 > URL: https://issues.apache.org/jira/browse/BEAM-14403 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Affects Versions: 2.35.0, 2.36.0, 2.37.0, 2.38.0 >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: P1 > Fix For: 2.39.0 > > Time Spent: 50m > Remaining Estimate: 0h > > [https://github.com/apache/beam/blob/v2.37.0/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java#L2315] > assumes that enable_prime forces UW. This decision should be left up to the > service. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (BEAM-5440) Add option to mount a directory inside SDK harness containers
[
https://issues.apache.org/jira/browse/BEAM-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17531472#comment-17531472
]
Ahmet Altay commented on BEAM-5440:
---
Got it. Sounds good. Thank you for the clarification.
[~robertcrowe] - let us know how you would like to proceed.
> Add option to mount a directory inside SDK harness containers
> -
>
> Key: BEAM-5440
> URL: https://issues.apache.org/jira/browse/BEAM-5440
> Project: Beam
> Issue Type: New Feature
> Components: java-fn-execution, sdk-java-core
>Reporter: Maximilian Michels
>Priority: P3
> Labels: Clarified, portability, portability-flink
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> While experimenting with the Python SDK locally, I found it inconvenient that
> I can't mount a host directory to the Docker containers, i.e. the input must
> already be in the container and the results of a Write remain inside the
> container. For local testing, users may want to mount a host directory.
> Since BEAM-5288 the {{Environment}} carries explicit environment information,
> we could a) add volume args to the {{DockerPayload}}, or b) provide a general
> Docker arguments field.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (BEAM-5440) Add option to mount a directory inside SDK harness containers
[
https://issues.apache.org/jira/browse/BEAM-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17527832#comment-17527832
]
Ahmet Altay commented on BEAM-5440:
---
That sounds reasonable. We probably need start from a new PR. Some of those
code paths were most replaced with runner v2 for running on dataflow now.
[~robertcrowe] would you like to start a PR? Otherwise, currently this does not
have an owner.
> Add option to mount a directory inside SDK harness containers
> -
>
> Key: BEAM-5440
> URL: https://issues.apache.org/jira/browse/BEAM-5440
> Project: Beam
> Issue Type: New Feature
> Components: java-fn-execution, sdk-java-core
>Reporter: Maximilian Michels
>Priority: P3
> Labels: Clarified, portability, portability-flink
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> While experimenting with the Python SDK locally, I found it inconvenient that
> I can't mount a host directory to the Docker containers, i.e. the input must
> already be in the container and the results of a Write remain inside the
> container. For local testing, users may want to mount a host directory.
> Since BEAM-5288 the {{Environment}} carries explicit environment information,
> we could a) add volume args to the {{DockerPayload}}, or b) provide a general
> Docker arguments field.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Assigned] (BEAM-14329) Enable exponential backoff retries in FhirIO Execute bundle requests.
[ https://issues.apache.org/jira/browse/BEAM-14329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay reassigned BEAM-14329: -- Assignee: Fathima Mohammed > Enable exponential backoff retries in FhirIO Execute bundle requests. > -- > > Key: BEAM-14329 > URL: https://issues.apache.org/jira/browse/BEAM-14329 > Project: Beam > Issue Type: Improvement > Components: io-java-healthcare >Reporter: Fathima Mohammed >Assignee: Fathima Mohammed >Priority: P2 > > Use the Cloud Healthcare Client in executeFhirBundle method instead of using > raw http requests. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (BEAM-14222) Test failure org.apache.beam.sdk.io.gcp.spanner.SpannerReadIT.testReadAllRecordsInDb
[ https://issues.apache.org/jira/browse/BEAM-14222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1751#comment-1751 ] Ahmet Altay commented on BEAM-14222: Could this be closed? https://github.com/apache/beam/pull/17234 is merged. > Test failure > org.apache.beam.sdk.io.gcp.spanner.SpannerReadIT.testReadAllRecordsInDb > > > Key: BEAM-14222 > URL: https://issues.apache.org/jira/browse/BEAM-14222 > Project: Beam > Issue Type: Sub-task > Components: test-failures >Reporter: Kiley Sok >Assignee: Bingye Li >Priority: P2 > Time Spent: 1h 50m > Remaining Estimate: 0h > > java.lang.AssertionError: Count PG rows/Flatten.PCollections.out: > Expected: <5L> > but: was <0L> > https://ci-beam.apache.org/job/beam_PostCommit_Java/8806/testReport/junit/org.apache.beam.sdk.io.gcp.spanner/SpannerReadIT/testReadAllRecordsInDb/ -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (BEAM-14181) BQ: Fix cache behavior
Ahmet Altay created BEAM-14181: -- Summary: BQ: Fix cache behavior Key: BEAM-14181 URL: https://issues.apache.org/jira/browse/BEAM-14181 Project: Beam Issue Type: Bug Components: io-java-gcp Reporter: Ahmet Altay Assignee: Reuven Lax Fix For: 2.38.0 Creating a jira so that it can be considered whether it is release blocking or not. Related change: https://github.com/apache/beam/pull/17187 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (BEAM-14172) beam_PreCommit_PythonDocs failing (jinja2)
[ https://issues.apache.org/jira/browse/BEAM-14172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay reassigned BEAM-14172: -- Assignee: Kerry Donny-Clark > beam_PreCommit_PythonDocs failing (jinja2) > -- > > Key: BEAM-14172 > URL: https://issues.apache.org/jira/browse/BEAM-14172 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Kyle Weaver >Assignee: Kerry Donny-Clark >Priority: P2 > Labels: currently-failing > Time Spent: 1h 10m > Remaining Estimate: 0h > > I'm guessing a new jinja2 release broke us > (https://pypi.org/project/Jinja2/#history) > 12:16:08 Traceback (most recent call last): > 12:16:08 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_PythonDocs_Commit/src/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/bin/sphinx-apidoc", > line 7, in > 12:16:08 from sphinx.ext.apidoc import main > 12:16:08 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_PythonDocs_Commit/src/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/sphinx/ext/apidoc.py", > line 32, in > 12:16:08 from sphinx.cmd.quickstart import EXTENSIONS > 12:16:08 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_PythonDocs_Commit/src/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/sphinx/cmd/quickstart.py", > line 49, in > 12:16:08 from sphinx.util.template import SphinxRenderer > 12:16:08 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_PythonDocs_Commit/src/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/sphinx/util/template.py", > line 17, in > 12:16:08 from sphinx.jinja2glue import SphinxFileSystemLoader > 12:16:08 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_PythonDocs_Commit/src/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/sphinx/jinja2glue.py", > line 16, in > 12:16:08 from jinja2 import FileSystemLoader, BaseLoader, > TemplateNotFound, \ > 12:16:08 ImportError: cannot import name 'contextfunction' from 'jinja2' > (/home/jenkins/jenkins-slave/workspace/beam_PreCommit_PythonDocs_Commit/src/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/jinja2/__init__.py) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (BEAM-14172) beam_PreCommit_PythonDocs failing (jinja2)
[ https://issues.apache.org/jira/browse/BEAM-14172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay updated BEAM-14172: --- Priority: P2 (was: P1) > beam_PreCommit_PythonDocs failing (jinja2) > -- > > Key: BEAM-14172 > URL: https://issues.apache.org/jira/browse/BEAM-14172 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Kyle Weaver >Priority: P2 > Labels: currently-failing > Time Spent: 1h 10m > Remaining Estimate: 0h > > I'm guessing a new jinja2 release broke us > (https://pypi.org/project/Jinja2/#history) > 12:16:08 Traceback (most recent call last): > 12:16:08 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_PythonDocs_Commit/src/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/bin/sphinx-apidoc", > line 7, in > 12:16:08 from sphinx.ext.apidoc import main > 12:16:08 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_PythonDocs_Commit/src/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/sphinx/ext/apidoc.py", > line 32, in > 12:16:08 from sphinx.cmd.quickstart import EXTENSIONS > 12:16:08 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_PythonDocs_Commit/src/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/sphinx/cmd/quickstart.py", > line 49, in > 12:16:08 from sphinx.util.template import SphinxRenderer > 12:16:08 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_PythonDocs_Commit/src/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/sphinx/util/template.py", > line 17, in > 12:16:08 from sphinx.jinja2glue import SphinxFileSystemLoader > 12:16:08 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_PythonDocs_Commit/src/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/sphinx/jinja2glue.py", > line 16, in > 12:16:08 from jinja2 import FileSystemLoader, BaseLoader, > TemplateNotFound, \ > 12:16:08 ImportError: cannot import name 'contextfunction' from 'jinja2' > (/home/jenkins/jenkins-slave/workspace/beam_PreCommit_PythonDocs_Commit/src/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/jinja2/__init__.py) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (BEAM-14172) beam_PreCommit_PythonDocs failing (jinja2)
[ https://issues.apache.org/jira/browse/BEAM-14172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17512138#comment-17512138 ] Ahmet Altay commented on BEAM-14172: Pre commit should be fixed with https://github.com/apache/beam/pull/17176 I believe we need to update the Sphinx version, otherwise this would happen again. Our sphinx version is quite old. Downgrading priority and assigning to [~kerrydc] for triage. > beam_PreCommit_PythonDocs failing (jinja2) > -- > > Key: BEAM-14172 > URL: https://issues.apache.org/jira/browse/BEAM-14172 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Kyle Weaver >Priority: P1 > Labels: currently-failing > Time Spent: 1h 10m > Remaining Estimate: 0h > > I'm guessing a new jinja2 release broke us > (https://pypi.org/project/Jinja2/#history) > 12:16:08 Traceback (most recent call last): > 12:16:08 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_PythonDocs_Commit/src/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/bin/sphinx-apidoc", > line 7, in > 12:16:08 from sphinx.ext.apidoc import main > 12:16:08 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_PythonDocs_Commit/src/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/sphinx/ext/apidoc.py", > line 32, in > 12:16:08 from sphinx.cmd.quickstart import EXTENSIONS > 12:16:08 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_PythonDocs_Commit/src/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/sphinx/cmd/quickstart.py", > line 49, in > 12:16:08 from sphinx.util.template import SphinxRenderer > 12:16:08 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_PythonDocs_Commit/src/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/sphinx/util/template.py", > line 17, in > 12:16:08 from sphinx.jinja2glue import SphinxFileSystemLoader > 12:16:08 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_PythonDocs_Commit/src/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/sphinx/jinja2glue.py", > line 16, in > 12:16:08 from jinja2 import FileSystemLoader, BaseLoader, > TemplateNotFound, \ > 12:16:08 ImportError: cannot import name 'contextfunction' from 'jinja2' > (/home/jenkins/jenkins-slave/workspace/beam_PreCommit_PythonDocs_Commit/src/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/jinja2/__init__.py) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (BEAM-14122) Python portable precommit broken: 'get_installed_distributions'
[
https://issues.apache.org/jira/browse/BEAM-14122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmet Altay updated BEAM-14122:
---
Fix Version/s: 2.38.0
Resolution: Fixed
Status: Resolved (was: Open)
> Python portable precommit broken: 'get_installed_distributions'
> ---
>
> Key: BEAM-14122
> URL: https://issues.apache.org/jira/browse/BEAM-14122
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Kyle Weaver
>Priority: P1
> Labels: currently-failing
> Fix For: 2.38.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Successfully installed PTable-0.9.2 pip-licenses-2.3.0
> WARNING: Running pip as the 'root' user can result in broken permissions and
> conflicting behaviour with the system package manager. It is recommended to
> use a virtual environment instead: https://pip.pypa.io/warnings/venv
> Traceback (most recent call last):
> File "/usr/local/lib/python3.9/site-packages/piplicenses.py", line 40, in
>
> from pip._internal.utils.misc import get_installed_distributions
> ImportError: cannot import name 'get_installed_distributions' from
> 'pip._internal.utils.misc'
> (/usr/local/lib/python3.9/site-packages/pip/_internal/utils/misc.py)
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "/usr/local/bin/pip-licenses", line 5, in
> from piplicenses import main
> File "/usr/local/lib/python3.9/site-packages/piplicenses.py", line 42, in
>
> from pip import get_installed_distributions
> ImportError: cannot import name 'get_installed_distributions' from 'pip'
> (/usr/local/lib/python3.9/site-packages/pip/__init__.py)
> Traceback (most recent call last):
> File "/tmp/license_scripts/pull_licenses_py.py", line 166, in
> dependencies = run_pip_licenses()
> File "/tmp/license_scripts/pull_licenses_py.py", line 49, in
> run_pip_licenses
> dependencies = run_bash_command(command)
> File "/tmp/license_scripts/pull_licenses_py.py", line 44, in
> run_bash_command
> return subprocess.check_output(command.split()).decode('utf-8')
> File "/usr/local/lib/python3.9/subprocess.py", line 424, in check_output
> return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
> File "/usr/local/lib/python3.9/subprocess.py", line 528, in run
> raise CalledProcessError(retcode, process.args,
> subprocess.CalledProcessError: Command '['pip-licenses',
> '--with-license-file', '--with-urls', '--from=mixed', '--ignore',
> 'apache-beam', '--format=json']' returned non-zero exit status 1.
> The command '/bin/sh -c if [ "$pull_licenses" = "true" ] ; then pip
> install 'pip-licenses<3.0.0' pyyaml tenacity && python
> /tmp/license_scripts/pull_licenses_py.py ; fi' returned a non-zero code: 1
> > Task :sdks:python:container:py39:docker FAILED
> https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/4748
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (BEAM-14122) Python portable precommit broken: 'get_installed_distributions'
[
https://issues.apache.org/jira/browse/BEAM-14122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17509360#comment-17509360
]
Ahmet Altay commented on BEAM-14122:
Pr is merged, latest cron run is passing
(https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/), closing.
> Python portable precommit broken: 'get_installed_distributions'
> ---
>
> Key: BEAM-14122
> URL: https://issues.apache.org/jira/browse/BEAM-14122
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Kyle Weaver
>Priority: P1
> Labels: currently-failing
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Successfully installed PTable-0.9.2 pip-licenses-2.3.0
> WARNING: Running pip as the 'root' user can result in broken permissions and
> conflicting behaviour with the system package manager. It is recommended to
> use a virtual environment instead: https://pip.pypa.io/warnings/venv
> Traceback (most recent call last):
> File "/usr/local/lib/python3.9/site-packages/piplicenses.py", line 40, in
>
> from pip._internal.utils.misc import get_installed_distributions
> ImportError: cannot import name 'get_installed_distributions' from
> 'pip._internal.utils.misc'
> (/usr/local/lib/python3.9/site-packages/pip/_internal/utils/misc.py)
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "/usr/local/bin/pip-licenses", line 5, in
> from piplicenses import main
> File "/usr/local/lib/python3.9/site-packages/piplicenses.py", line 42, in
>
> from pip import get_installed_distributions
> ImportError: cannot import name 'get_installed_distributions' from 'pip'
> (/usr/local/lib/python3.9/site-packages/pip/__init__.py)
> Traceback (most recent call last):
> File "/tmp/license_scripts/pull_licenses_py.py", line 166, in
> dependencies = run_pip_licenses()
> File "/tmp/license_scripts/pull_licenses_py.py", line 49, in
> run_pip_licenses
> dependencies = run_bash_command(command)
> File "/tmp/license_scripts/pull_licenses_py.py", line 44, in
> run_bash_command
> return subprocess.check_output(command.split()).decode('utf-8')
> File "/usr/local/lib/python3.9/subprocess.py", line 424, in check_output
> return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
> File "/usr/local/lib/python3.9/subprocess.py", line 528, in run
> raise CalledProcessError(retcode, process.args,
> subprocess.CalledProcessError: Command '['pip-licenses',
> '--with-license-file', '--with-urls', '--from=mixed', '--ignore',
> 'apache-beam', '--format=json']' returned non-zero exit status 1.
> The command '/bin/sh -c if [ "$pull_licenses" = "true" ] ; then pip
> install 'pip-licenses<3.0.0' pyyaml tenacity && python
> /tmp/license_scripts/pull_licenses_py.py ; fi' returned a non-zero code: 1
> > Task :sdks:python:container:py39:docker FAILED
> https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/4748
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (BEAM-14122) Python portable precommit broken: 'get_installed_distributions'
[
https://issues.apache.org/jira/browse/BEAM-14122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17509150#comment-17509150
]
Ahmet Altay commented on BEAM-14122:
I have a PR to fix this. It is a dependency issue. The failing test is passing
with the change: https://github.com/apache/beam/pull/17132
Feel free to review (and merge if it looks good).
> Python portable precommit broken: 'get_installed_distributions'
> ---
>
> Key: BEAM-14122
> URL: https://issues.apache.org/jira/browse/BEAM-14122
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Kyle Weaver
>Priority: P1
> Labels: currently-failing
>
> Successfully installed PTable-0.9.2 pip-licenses-2.3.0
> WARNING: Running pip as the 'root' user can result in broken permissions and
> conflicting behaviour with the system package manager. It is recommended to
> use a virtual environment instead: https://pip.pypa.io/warnings/venv
> Traceback (most recent call last):
> File "/usr/local/lib/python3.9/site-packages/piplicenses.py", line 40, in
>
> from pip._internal.utils.misc import get_installed_distributions
> ImportError: cannot import name 'get_installed_distributions' from
> 'pip._internal.utils.misc'
> (/usr/local/lib/python3.9/site-packages/pip/_internal/utils/misc.py)
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "/usr/local/bin/pip-licenses", line 5, in
> from piplicenses import main
> File "/usr/local/lib/python3.9/site-packages/piplicenses.py", line 42, in
>
> from pip import get_installed_distributions
> ImportError: cannot import name 'get_installed_distributions' from 'pip'
> (/usr/local/lib/python3.9/site-packages/pip/__init__.py)
> Traceback (most recent call last):
> File "/tmp/license_scripts/pull_licenses_py.py", line 166, in
> dependencies = run_pip_licenses()
> File "/tmp/license_scripts/pull_licenses_py.py", line 49, in
> run_pip_licenses
> dependencies = run_bash_command(command)
> File "/tmp/license_scripts/pull_licenses_py.py", line 44, in
> run_bash_command
> return subprocess.check_output(command.split()).decode('utf-8')
> File "/usr/local/lib/python3.9/subprocess.py", line 424, in check_output
> return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
> File "/usr/local/lib/python3.9/subprocess.py", line 528, in run
> raise CalledProcessError(retcode, process.args,
> subprocess.CalledProcessError: Command '['pip-licenses',
> '--with-license-file', '--with-urls', '--from=mixed', '--ignore',
> 'apache-beam', '--format=json']' returned non-zero exit status 1.
> The command '/bin/sh -c if [ "$pull_licenses" = "true" ] ; then pip
> install 'pip-licenses<3.0.0' pyyaml tenacity && python
> /tmp/license_scripts/pull_licenses_py.py ; fi' returned a non-zero code: 1
> > Task :sdks:python:container:py39:docker FAILED
> https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/4748
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (BEAM-6975) Merge portability status into capability matrix
[ https://issues.apache.org/jira/browse/BEAM-6975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507060#comment-17507060 ] Ahmet Altay commented on BEAM-6975: --- [~lcwik] - Should we update the portability support matrix ? > Merge portability status into capability matrix > --- > > Key: BEAM-6975 > URL: https://issues.apache.org/jira/browse/BEAM-6975 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Ahmet Altay >Priority: P3 > Fix For: Not applicable > > > Should the portability status: > https://s.apache.org/apache-beam-portability-support-table > be merged into capability matrix > https://beam.apache.org/documentation/runners/capability-matrix/ ? > (That is add portable runners to the list of runners as columns in the > capability matrix.) > cc: [~kenn] [~lcwik] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (BEAM-14017) beam_PreCommit_CommunityMetrics_Cron is failing.
[ https://issues.apache.org/jira/browse/BEAM-14017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17504639#comment-17504639 ] Ahmet Altay commented on BEAM-14017: NB: Community metrics are still up and running. Only the tests are failing. > beam_PreCommit_CommunityMetrics_Cron is failing. > > > Key: BEAM-14017 > URL: https://issues.apache.org/jira/browse/BEAM-14017 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: Valentyn Tymofieiev >Assignee: Daniel Oliveira >Priority: P1 > > https://ci-beam.apache.org/job/beam_PreCommit_CommunityMetrics_Cron/4805/console > 10:14:48 > Task :beam-test-infra-metrics:validateConfiguration > 10:14:48 W0228 18:14:48.092605 389274 helpers.go:549] --dry-run=true is > deprecated (boolean value) and can be replaced with --dry-run=client. > 10:15:20 Unable to connect to the server: dial tcp 104.154.102.21:443: i/o > timeout (Client.Timeout exceeded while awaiting headers) > 10:15:20 > 10:15:20 > Task :beam-test-infra-metrics:validateConfiguration FAILED > 10:15:20 > 10:15:20 FAILURE: Build failed with an exception. > 10:15:20 > 10:15:20 * What went wrong: > 10:15:20 Execution failed for task > ':beam-test-infra-metrics:validateConfiguration'. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (BEAM-14011) AWS SDK2 S3FileSystem MultiPart Copy sets incorrect request parameters
[
https://issues.apache.org/jira/browse/BEAM-14011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmet Altay reassigned BEAM-14011:
--
Assignee: Stephen Patel
> AWS SDK2 S3FileSystem MultiPart Copy sets incorrect request parameters
> --
>
> Key: BEAM-14011
> URL: https://issues.apache.org/jira/browse/BEAM-14011
> Project: Beam
> Issue Type: Bug
> Components: io-java-aws
>Affects Versions: 2.29.0, 2.30.0, 2.31.0, 2.32.0, 2.33.0, 2.34.0, 2.35.0,
> 2.36.0
>Reporter: Stephen Patel
>Assignee: Stephen Patel
>Priority: P2
>
> The following code locations use incorrect parameters:
> [Here|https://github.com/apache/beam/blob/v2.36.0/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/s3/S3FileSystem.java#L518-L519]
> and
> [Here|https://github.com/apache/beam/blob/v2.36.0/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/s3/S3FileSystem.java#L539-L540]
> specifies the sourcePath instead of the destinationPath.
> [Here|https://github.com/apache/beam/blob/v2.36.0/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/s3/S3FileSystem.java#L541]
> specifies destinationPath.getBucket() instead of sourcePath.getBucket()
> [Here|https://github.com/apache/beam/blob/v2.36.0/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/s3/S3FileSystem.java#L556]
> specifies a constant part number of 1, instead of using the partNumber
> variable.
> Taken together, these issues cause multipart copies to fail due to:
> {noformat}
> software.amazon.awssdk.services.s3.model.NoSuchUploadException: The specified
> upload does not exist. The upload ID may be invalid, or the upload may have
> been aborted or completed.
> {noformat}
> If the object references are fixes, the part number issue causes multipart
> copies to fail due to:
> {noformat}
> software.amazon.awssdk.services.s3.model.S3Exception: The list of parts was
> not in ascending order. Parts must be ordered by part number.
> {noformat}
> Note: I checked the AWS SDK1 S3FileSystem and did not see the same issues.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (BEAM-12793) beam_PostRelease_NightlySnapshot failed
[
https://issues.apache.org/jira/browse/BEAM-12793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmet Altay reassigned BEAM-12793:
--
Assignee: Kenneth Knowles
> beam_PostRelease_NightlySnapshot failed
> ---
>
> Key: BEAM-12793
> URL: https://issues.apache.org/jira/browse/BEAM-12793
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Kyle Weaver
>Assignee: Kenneth Knowles
>Priority: P1
> Labels: currently-failing, flake
>
> Task :runners:direct-java:runMobileGamingJavaDirect is failing (or severely
> flaking).
> There are a bunch of similar 404 errors for the tables used in the Dataflow
> runner tests, but the Dataflow runner test doesn't appear to be the one
> that's actually failing.
> https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/1470/console
> {code}
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:java
> (default-cli) on project word-count-beam: An exception occured while
> executing the Java class. java.lang.RuntimeException:
> com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not
> Found
> [ERROR] POST
> https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DirectRunner_team/insertAll?prettyPrint=false
> [ERROR] {
> [ERROR] "code" : 404,
> [ERROR] "errors" : [ {
> [ERROR] "domain" : "global",
> [ERROR] "message" : "Not found: table Table is deleted:
> 844138762903:beam_postrelease_mobile_gaming.leaderboard_DirectRunner_team",
> [ERROR] "reason" : "notFound"
> [ERROR] } ],
> [ERROR] "message" : "Not found: table Table is deleted:
> 844138762903:beam_postrelease_mobile_gaming.leaderboard_DirectRunner_team",
> [ERROR] "status" : "NOT_FOUND"
> [ERROR] }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (BEAM-13797) Flakes: Failed to load cache entry
[ https://issues.apache.org/jira/browse/BEAM-13797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17497825#comment-17497825 ] Ahmet Altay commented on BEAM-13797: ping? How bad is this flake? Should we lower the priority? > Flakes: Failed to load cache entry > -- > > Key: BEAM-13797 > URL: https://issues.apache.org/jira/browse/BEAM-13797 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: P1 > Labels: flake > > I've seen this happen with multiple Jenkins jobs. The task will fail when > this happens. > Failed to load cache entry for task ':runners:portability:java:compileJava' > https://ci-beam.apache.org/job/beam_PreCommit_Java_PVR_Flink_Docker_Commit/235/console -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (BEAM-13859) Test flake: test_split_half_sdf
[
https://issues.apache.org/jira/browse/BEAM-13859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17497824#comment-17497824
]
Ahmet Altay commented on BEAM-13859:
[~chamikara] [~kerrydc]- How frequent is this flake? Should we prioritize it?
> Test flake: test_split_half_sdf
> ---
>
> Key: BEAM-13859
> URL: https://issues.apache.org/jira/browse/BEAM-13859
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Kyle Weaver
>Assignee: Chamikara Madhusanka Jayalath
>Priority: P1
> Labels: flake
>
> > raise RuntimeError(result.error)
> E RuntimeError: Traceback (most recent call last):
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/data_plane.py",
> line 487, in input_elements
> E element = received.get(timeout=1)
> E File "/usr/lib/python3.7/queue.py", line 178, in get
> E raise Empty
> E _queue.Empty
> E
> E During handling of the above exception, another exception occurred:
> E
> E Traceback (most recent call last):
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 267, in _execute
> E response = task()
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 340, in
> E lambda: self.create_worker().do_instruction(request), request)
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 581, in do_instruction
> E getattr(request, request_type), request.instruction_id)
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 618, in process_bundle
> E bundle_processor.process_bundle(instruction_id))
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/bundle_processor.py",
> line 985, in process_bundle
> E expected_inputs):
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/data_plane.py",
> line 490, in input_elements
> E raise RuntimeError('Channel closed prematurely.')
> E RuntimeError: Channel closed prematurely.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (BEAM-13998) Build flakes - Timeout waiting to lock daemon addresses registry
[ https://issues.apache.org/jira/browse/BEAM-13998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17497820#comment-17497820 ] Ahmet Altay commented on BEAM-13998: [~kenn] - It is java / gradle issue. Assigning to you. Feel free to re-assign. > Build flakes - Timeout waiting to lock daemon addresses registry > > > Key: BEAM-13998 > URL: https://issues.apache.org/jira/browse/BEAM-13998 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Kyle Weaver >Assignee: Kenneth Knowles >Priority: P1 > Labels: flake > > So far I've seen this on beam_PreCommit_Java_commit, > beam_PreCommit_PythonDocs_Phrase, and > beam_PreCommit_Java_PVR_Flink_Docker_Phrase. > e.g. https://ci-beam.apache.org/job/beam_PreCommit_Java_commit/21195/console > 10:20:47 FAILURE: Build failed with an exception. > 10:20:47 > 10:20:47 * What went wrong: > 10:20:47 Timeout waiting to lock daemon addresses registry. It is currently > in use by another Gradle instance. > 10:20:47 Owner PID: unknown > 10:20:47 Our PID: 2090322 > 10:20:47 Owner Operation: unknown > 10:20:47 Our operation: > 10:20:47 Lock file: /home/jenkins/.gradle/daemon/7.3.2/registry.bin.lock -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (BEAM-13998) Build flakes - Timeout waiting to lock daemon addresses registry
[ https://issues.apache.org/jira/browse/BEAM-13998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay reassigned BEAM-13998: -- Assignee: Kenneth Knowles > Build flakes - Timeout waiting to lock daemon addresses registry > > > Key: BEAM-13998 > URL: https://issues.apache.org/jira/browse/BEAM-13998 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Kyle Weaver >Assignee: Kenneth Knowles >Priority: P1 > Labels: flake > > So far I've seen this on beam_PreCommit_Java_commit, > beam_PreCommit_PythonDocs_Phrase, and > beam_PreCommit_Java_PVR_Flink_Docker_Phrase. > e.g. https://ci-beam.apache.org/job/beam_PreCommit_Java_commit/21195/console > 10:20:47 FAILURE: Build failed with an exception. > 10:20:47 > 10:20:47 * What went wrong: > 10:20:47 Timeout waiting to lock daemon addresses registry. It is currently > in use by another Gradle instance. > 10:20:47 Owner PID: unknown > 10:20:47 Our PID: 2090322 > 10:20:47 Owner Operation: unknown > 10:20:47 Our operation: > 10:20:47 Lock file: /home/jenkins/.gradle/daemon/7.3.2/registry.bin.lock -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (BEAM-12672) Update all Precommits to rerun failed tests K times.
[ https://issues.apache.org/jira/browse/BEAM-12672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17497615#comment-17497615 ] Ahmet Altay commented on BEAM-12672: Relevant infra jira: https://issues.apache.org/jira/browse/INFRA-22878 It might be possible to re-enable this plugin by keeping smaller number of past builds. (probably not a good outcome) > Update all Precommits to rerun failed tests K times. > > > Key: BEAM-12672 > URL: https://issues.apache.org/jira/browse/BEAM-12672 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Alex Amato >Assignee: Benjamin Gonzalez >Priority: P2 > Fix For: Not applicable > > Time Spent: 20h 50m > Remaining Estimate: 0h > > We should configure the precommit test suites to re-run failed tests a few > (K) times. > (A bonus feature would be to tag certain tests as known flakey, and run > multiple attempts in parallel when the test is first ran) > The benefits of this are to: > * Not have flakey tests slow down PR reviews and merges. > * And avoiding the need to repeatedly manually rerun the precommit test with > a PR comment. > * Getting a clear signal that failed tests are due to the code in the PR > * Save computing resources (Rather than needing to run the entire test suite > every time it fails, only the failed tests will rerun). > (Its not realistic to have 0 flakeyness, so this is desirable to improve > productivity, as it can take several attempts to have all the tests pass in a > single run). > Note: This does not mean that we should ignore flakey tests, which are > sometimes indicative of bugs. We should continue to investigate those with a > continuous test suite. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (BEAM-13608) Dynamic Topics management
[
https://issues.apache.org/jira/browse/BEAM-13608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17496361#comment-17496361
]
Ahmet Altay commented on BEAM-13608:
For reference this is the design document:
https://docs.google.com/document/d/1IY4_e5g1g71XvTLL4slHRyVfX7ByiwjD_de3WGsBQXg/edit?usp=sharing
/cc [~johnjcasey]
> Dynamic Topics management
> -
>
> Key: BEAM-13608
> URL: https://issues.apache.org/jira/browse/BEAM-13608
> Project: Beam
> Issue Type: Improvement
> Components: io-java-jms
>Reporter: Vincent BALLADA
>Assignee: Vincent BALLADA
>Priority: P2
> Labels: assigned:
>
> JmsIO write function is able to publish messages to topics with static names:
> company/employee/id/1234567.
> Some AMQP/JMS broker provides the ability to publish to dynamic topics like:
> company/employee/id/\{employeeId}
> If we want to handle that with Apache Beam JmsIO, we must create a branch per
> employeeId, which is not suitable for a company with thousand of employee, or
> other similat use cases.
> The JmsIO write function should provide the ability to handle dynamic topics.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (BEAM-13859) Test flake: test_split_half_sdf
[
https://issues.apache.org/jira/browse/BEAM-13859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmet Altay reassigned BEAM-13859:
--
Assignee: Chamikara Madhusanka Jayalath
> Test flake: test_split_half_sdf
> ---
>
> Key: BEAM-13859
> URL: https://issues.apache.org/jira/browse/BEAM-13859
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Kyle Weaver
>Assignee: Chamikara Madhusanka Jayalath
>Priority: P1
> Labels: flake
>
> > raise RuntimeError(result.error)
> E RuntimeError: Traceback (most recent call last):
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/data_plane.py",
> line 487, in input_elements
> E element = received.get(timeout=1)
> E File "/usr/lib/python3.7/queue.py", line 178, in get
> E raise Empty
> E _queue.Empty
> E
> E During handling of the above exception, another exception occurred:
> E
> E Traceback (most recent call last):
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 267, in _execute
> E response = task()
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 340, in
> E lambda: self.create_worker().do_instruction(request), request)
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 581, in do_instruction
> E getattr(request, request_type), request.instruction_id)
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 618, in process_bundle
> E bundle_processor.process_bundle(instruction_id))
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/bundle_processor.py",
> line 985, in process_bundle
> E expected_inputs):
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/data_plane.py",
> line 490, in input_elements
> E raise RuntimeError('Channel closed prematurely.')
> E RuntimeError: Channel closed prematurely.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (BEAM-13859) Test flake: test_split_half_sdf
[
https://issues.apache.org/jira/browse/BEAM-13859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17494289#comment-17494289
]
Ahmet Altay commented on BEAM-13859:
Python SDF issue. [~chamikara]- could you find an owner?
> Test flake: test_split_half_sdf
> ---
>
> Key: BEAM-13859
> URL: https://issues.apache.org/jira/browse/BEAM-13859
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Kyle Weaver
>Assignee: Chamikara Madhusanka Jayalath
>Priority: P1
> Labels: flake
>
> > raise RuntimeError(result.error)
> E RuntimeError: Traceback (most recent call last):
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/data_plane.py",
> line 487, in input_elements
> E element = received.get(timeout=1)
> E File "/usr/lib/python3.7/queue.py", line 178, in get
> E raise Empty
> E _queue.Empty
> E
> E During handling of the above exception, another exception occurred:
> E
> E Traceback (most recent call last):
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 267, in _execute
> E response = task()
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 340, in
> E lambda: self.create_worker().do_instruction(request), request)
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 581, in do_instruction
> E getattr(request, request_type), request.instruction_id)
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 618, in process_bundle
> E bundle_processor.process_bundle(instruction_id))
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/bundle_processor.py",
> line 985, in process_bundle
> E expected_inputs):
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/data_plane.py",
> line 490, in input_elements
> E raise RuntimeError('Channel closed prematurely.')
> E RuntimeError: Channel closed prematurely.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (BEAM-13810) Flaky tests: Gradle build daemon disappeared unexpectedly
[
https://issues.apache.org/jira/browse/BEAM-13810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17494288#comment-17494288
]
Ahmet Altay commented on BEAM-13810:
How much of a performance penalty? Would it be worse than the cost of
flakiness? (Not a rhetorical question, trying to understand the question.)
> Flaky tests: Gradle build daemon disappeared unexpectedly
> -
>
> Key: BEAM-13810
> URL: https://issues.apache.org/jira/browse/BEAM-13810
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Kyle Weaver
>Assignee: Kenneth Knowles
>Priority: P1
> Labels: flake
>
> This happens to many of our tests. It looks like this is a common issue with
> Gradle and we will have to do more digging to determine the true cause.
> https://stackoverflow.com/questions/37171043/gradle-build-daemon-disappeared-unexpectedly-it-may-have-been-killed-or-may-hav/37171110
> 10:04:08 > Task :sdks:go:test:sparkValidatesRunner
> 10:04:08 Feb 02, 2022 6:04:08 PM
> org.apache.beam.sdk.expansion.service.ExpansionService
> loadRegisteredTransforms
> 10:04:08 INFO: Registering external transforms:
> [beam:transforms:xlang:test:cgbk, beam:transforms:xlang:test:flatten,
> beam:transforms:xlang:test:prefix, beam:transforms:xlang:test:multi,
> beam:transforms:xlang:test:gbk, beam:transforms:xlang:test:comgl,
> beam:transforms:xlang:test:compk, beam:transforms:xlang:count,
> beam:transforms:xlang:filter_less_than_eq,
> beam:transforms:xlang:test:partition,
> beam:transforms:xlang:test:parquet_write, beam:transforms:xlang:parquet_read,
> beam:transforms:xlang:textio_read, beam:external:java:generate_sequence:v1]
> 10:04:09 The message received from the daemon indicates that the daemon has
> disappeared.
> 10:04:09 Build request sent: Build{id=37ea63c1-ec5f-492a-ad6a-f26c92fd7f7e,
> currentDir=/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_VR_Spark/src}
> 10:04:09 Attempting to read last messages from the daemon log...
> 10:04:09 Daemon pid: 3299301
> 10:04:09 log file: /home/jenkins/.gradle/daemon/7.3.2/daemon-3299301.out.log
> 10:04:09 - Last 20 lines from daemon log file - daemon-3299301.out.log
> -
> 10:04:09 2022-02-02T18:04:08.251+ [DEBUG]
> [org.gradle.launcher.daemon.registry.PersistentDaemonRegistry] Marking busy
> by address: [2bd8c084-af4c-4924-90f5-8b8a89f85fc5 port:32889,
> addresses:[localhost/127.0.0.1]]
> 10:04:09 2022-02-02T18:04:08.251+ [DEBUG]
> [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire
> exclusive lock on daemon addresses registry.
> 10:04:09 2022-02-02T18:04:08.251+ [DEBUG]
> [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon
> addresses registry.
> 10:04:09 2022-02-02T18:04:08.252+ [DEBUG]
> [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon
> addresses registry.
> 10:04:09 2022-02-02T18:04:08.252+ [DEBUG]
> [org.gradle.launcher.daemon.server.DaemonStateCoordinator] resetting idle
> timer
> 10:04:09 2022-02-02T18:04:08.252+ [DEBUG]
> [org.gradle.launcher.daemon.server.DaemonStateCoordinator] daemon is running.
> Sleeping until state changes.
> 10:04:09 2022-02-02T18:04:08.253+ [INFO]
> [org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy] Daemon
> is about to start building Build{id=37ea63c1-ec5f-492a-ad6a-f26c92fd7f7e,
> currentDir=/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_VR_Spark/src}.
> Dispatching build started information...
> 10:04:09 2022-02-02T18:04:08.253+ [DEBUG]
> [org.gradle.launcher.daemon.server.SynchronizedDispatchConnection] thread
> 266: dispatching org.gradle.launcher.daemon.protocol.BuildStarted@42569c28
> 10:04:09 2022-02-02T18:04:08.255+ [DEBUG]
> [org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment]
> Configuring env variables: [PATH, RUN_DISPLAY_URL, HUDSON_HOME,
> RUN_CHANGES_DISPLAY_URL, JOB_URL, HUDSON_COOKIE, MOTD_SHOWN,
> DBUS_SESSION_BUS_ADDRESS, JENKINS_SERVER_COOKIE, LOGNAME, PWD,
> RUN_TESTS_DISPLAY_URL, JENKINS_URL, SHELL, BUILD_TAG, ROOT_BUILD_CAUSE,
> BUILD_CAUSE_TIMERTRIGGER, OLDPWD, GIT_CHECKOUT_DIR, JENKINS_HOME, sha1,
> CODECOV_TOKEN, NODE_NAME, BUILD_DISPLAY_NAME, JOB_DISPLAY_URL, GIT_BRANCH,
> SETUPTOOLS_USE_DISTUTILS, SHLVL, WORKSPACE_TMP, GIT_PREVIOUS_COMMIT,
> JAVA_HOME, BUILD_ID, LANG, XDG_SESSION_ID, XDG_SESSION_TYPE, JOB_NAME,
> SPARK_LOCAL_IP, BUILD_CAUSE, GIT_PREVIOUS_SUCCESSFUL_COMMIT, NODE_LABELS,
> HUDSON_URL, WORKSPACE, ROOT_BUILD_CAUSE_TIMERTRIGGER, XDG_SESSION_CLASS, _,
> GIT_COMMIT, COVERALLS_REPO_TOKEN, CI, EXECUTOR_NUMBER, HUDSON_SERVER_COOKIE,
> SSH_CLIENT, JOB_BASE_NAME, USER, SSH_CONNECTION, BUILD_NUMBER, BUILD_URL,
> RUN_ARTIFACTS_DISPLAY_URL, GIT_URL, XDG_RUNTIME_DIR,
[jira] [Commented] (BEAM-13797) Flakes: Failed to load cache entry
[ https://issues.apache.org/jira/browse/BEAM-13797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17494287#comment-17494287 ] Ahmet Altay commented on BEAM-13797: Could we fix it? Disable it? > Flakes: Failed to load cache entry > -- > > Key: BEAM-13797 > URL: https://issues.apache.org/jira/browse/BEAM-13797 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: P1 > Labels: flake > > I've seen this happen with multiple Jenkins jobs. The task will fail when > this happens. > Failed to load cache entry for task ':runners:portability:java:compileJava' > https://ci-beam.apache.org/job/beam_PreCommit_Java_PVR_Flink_Docker_Commit/235/console -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (BEAM-11971) Direct Runner State is null while active timers exist
[
https://issues.apache.org/jira/browse/BEAM-11971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmet Altay reassigned BEAM-11971:
--
Assignee: Reuven Lax
> Direct Runner State is null while active timers exist
> -
>
> Key: BEAM-11971
> URL: https://issues.apache.org/jira/browse/BEAM-11971
> Project: Beam
> Issue Type: Bug
> Components: runner-direct
>Reporter: Reza ardeshir rokni
>Assignee: Reuven Lax
>Priority: P3
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> State is set to {{null}} while active timer is present, this issue does not
> show in other runners.
> The following example will reach the IllegalStateException within 10-20 times
> of it being run. {{LOOP_COUNT}} does not seem to be a factor as it reproduces
> with 100 or 10 {{LOOP_COUNT}}. The number of keys is a factor as it did
> not reproduce with only one key, have not tried with more than 3 keys to see
> if it's easier to reproduce.
>
> {code}
> package test;
> import org.apache.beam.sdk.Pipeline;
> import org.apache.beam.sdk.coders.BigEndianIntegerCoder;
> import org.apache.beam.sdk.coders.KvCoder;
> import org.apache.beam.sdk.state.StateSpec;
> import org.apache.beam.sdk.state.StateSpecs;
> import org.apache.beam.sdk.state.TimeDomain;
> import org.apache.beam.sdk.state.Timer;
> import org.apache.beam.sdk.state.TimerSpec;
> import org.apache.beam.sdk.state.TimerSpecs;
> import org.apache.beam.sdk.state.ValueState;
> import org.apache.beam.sdk.testing.TestStream;
> import org.apache.beam.sdk.transforms.DoFn;
> import org.apache.beam.sdk.transforms.PTransform;
> import org.apache.beam.sdk.transforms.ParDo;
> import org.apache.beam.sdk.transforms.WithKeys;
> import org.apache.beam.sdk.values.KV;
> import org.apache.beam.sdk.values.PCollection;
> import org.joda.time.Duration;
> import org.joda.time.Instant;
> import java.util.Optional;
>
> public class Test {
> public static void main (String [] args) throws Exception{
> Test.testToFailure();
> }
> public static void testToFailure() throws Exception {
> int count = 0;
> while (true) {
> failingTest();
> System.out.println(
> String.format("Got to Count %s", String.valueOf(count++)));
> }
> }
> public static void failingTest() throws Exception {
> Pipeline p = Pipeline.create();
> Instant now = Instant.now();
> TestStream stream =
> TestStream.create(BigEndianIntegerCoder.of())
> .addElements(1)
>
> .advanceWatermarkTo(now.plus(Duration.standardSeconds(1)))
> .addElements(2)
>
> .advanceWatermarkTo(now.plus(Duration.standardSeconds(1)))
> .addElements(3)
> .advanceWatermarkToInfinity();
> p.apply(stream)
> .apply(WithKeys.of(x -> x))
> .setCoder(KvCoder.of(BigEndianIntegerCoder.of(),
> BigEndianIntegerCoder.of()))
> .apply(new TestToFail());
> p.run();
> }
> public static class TestToFail
> extends PTransform>,
> PCollection> {
> @Override
> public PCollection expand(PCollection>
> input) {
> return input.apply(ParDo.of(new LoopingRead()));
> }
> }
> public static class LoopingRead extends DoFn,
> Integer> {
> static int LOOP_COUNT = 100;
> @StateId("value")
> private final StateSpec> value =
> StateSpecs.value(BigEndianIntegerCoder.of());
> @StateId("count")
> private final StateSpec> count =
> StateSpecs.value(BigEndianIntegerCoder.of());
> @TimerId("actionTimers")
> private final TimerSpec timer =
> TimerSpecs.timer(TimeDomain.EVENT_TIME);
> @ProcessElement
> public void processElement(
> ProcessContext c,
> @StateId("value") ValueState value,
> @TimerId("actionTimers") Timer timers) {
> value.write(c.element().getValue());
> timers.set(c.timestamp().plus(Duration.millis(1000)));
> }
> /** */
> @OnTimer("actionTimers")
> public void onTimer(
> OnTimerContext c,
> @StateId("value") ValueState value,
> @StateId("count") ValueState count,
> @TimerId("actionTimers") Timer timers) {
> if (value.read() == null) {
> throw new IllegalStateException("BINGO!");
> }
> Integer counter = Optional.ofNullable(count.read()).orElse(0) + 1;
> count.write(counter);
> value.write(value.read() + counter);
> if (counter < LOOP_COUNT) {
> timers.set(c.timestamp()
[jira] [Commented] (BEAM-13931) BigQueryIO is sending rows that are too large to Deadletter Queue even on RETRY_ALWAYS
[ https://issues.apache.org/jira/browse/BEAM-13931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17492127#comment-17492127 ] Ahmet Altay commented on BEAM-13931: [~pabloem] - Should we revert the change instead of fixing it? Does the fix preserve the previous behavior? - Could you please include this in the known issues list for 2.35.0 and 2.36.0 release notes? > BigQueryIO is sending rows that are too large to Deadletter Queue even on > RETRY_ALWAYS > -- > > Key: BEAM-13931 > URL: https://issues.apache.org/jira/browse/BEAM-13931 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Affects Versions: 2.35.0, 2.36.0 >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: P0 > Fix For: 2.37.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Note that BQ does not support requests over a certain size, and rows that go > past the size may be output into a dead-letter queue that they can get back > with > [BigQueryIO.Write.Result.getFailedInsertsWithErr|https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/gcp/bigquery/WriteResult.html#getFailedInsertsWithErr--] > A change went into Beam that outputs rows into the BQIO DLQ even if they're > meant to be retried indefinitely. > [https://github.com/apache/beam/commit/1f08d1f3ddc2e7bc7341be4b29bdafaec18de9cc#diff-26dbe8f625f702ae3edacdbc02b12acc6e423542fe16835229e22ef8eb4e109cR979-R989] > > > A workaround is to set this pipeline option to a larger amount: > [https://github.com/apache/beam/blob/1f08d1f3ddc2e7bc7341be4b29bdafaec18de9cc/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryOptions.java#L70] > > Currently it's 64KB, which is relatively small. Setting it to 1MB or 5MB or > so should work around this issue (it should be larger than the maximum row > size) - gRPC should support up to 10MB request sizes. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (BEAM-13376) Missing error for nonexistent column family BigTable
[ https://issues.apache.org/jira/browse/BEAM-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17491197#comment-17491197 ] Ahmet Altay commented on BEAM-13376: Assigned this to Igor. If someone else should be the owner please re-assign. Given that it is a P1 and the potential implications it would be good to have an owner. /cc [~chamikara] [~kerrydc] > Missing error for nonexistent column family BigTable > > > Key: BEAM-13376 > URL: https://issues.apache.org/jira/browse/BEAM-13376 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Reporter: PierreOberholzer >Assignee: Igor Bernstein >Priority: P1 > > Currently, there is no error thrown by BigTable in case the Column Families > are not defined at writing time. That is a misleading behavior as the user > believes the job has completed, though with empty table. > A bug was raised on BigTable: > [https://issuetracker.google.com/issues/186053077?pli=1] > But it should be made sure that the Beam IO will log this error appropriately. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (BEAM-13376) Missing error for nonexistent column family BigTable
[ https://issues.apache.org/jira/browse/BEAM-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay reassigned BEAM-13376: -- Assignee: Igor Bernstein > Missing error for nonexistent column family BigTable > > > Key: BEAM-13376 > URL: https://issues.apache.org/jira/browse/BEAM-13376 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Reporter: PierreOberholzer >Assignee: Igor Bernstein >Priority: P1 > > Currently, there is no error thrown by BigTable in case the Column Families > are not defined at writing time. That is a misleading behavior as the user > believes the job has completed, though with empty table. > A bug was raised on BigTable: > [https://issuetracker.google.com/issues/186053077?pli=1] > But it should be made sure that the Beam IO will log this error appropriately. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (BEAM-13810) Flaky tests: Gradle build daemon disappeared unexpectedly
[
https://issues.apache.org/jira/browse/BEAM-13810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17487389#comment-17487389
]
Ahmet Altay commented on BEAM-13810:
[~kenn] [~kileys] - Who could look into this?
> Flaky tests: Gradle build daemon disappeared unexpectedly
> -
>
> Key: BEAM-13810
> URL: https://issues.apache.org/jira/browse/BEAM-13810
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Kyle Weaver
>Assignee: Kenneth Knowles
>Priority: P1
> Labels: flake
>
> This happens to many of our tests. It looks like this is a common issue with
> Gradle and we will have to do more digging to determine the true cause.
> https://stackoverflow.com/questions/37171043/gradle-build-daemon-disappeared-unexpectedly-it-may-have-been-killed-or-may-hav/37171110
> 10:04:08 > Task :sdks:go:test:sparkValidatesRunner
> 10:04:08 Feb 02, 2022 6:04:08 PM
> org.apache.beam.sdk.expansion.service.ExpansionService
> loadRegisteredTransforms
> 10:04:08 INFO: Registering external transforms:
> [beam:transforms:xlang:test:cgbk, beam:transforms:xlang:test:flatten,
> beam:transforms:xlang:test:prefix, beam:transforms:xlang:test:multi,
> beam:transforms:xlang:test:gbk, beam:transforms:xlang:test:comgl,
> beam:transforms:xlang:test:compk, beam:transforms:xlang:count,
> beam:transforms:xlang:filter_less_than_eq,
> beam:transforms:xlang:test:partition,
> beam:transforms:xlang:test:parquet_write, beam:transforms:xlang:parquet_read,
> beam:transforms:xlang:textio_read, beam:external:java:generate_sequence:v1]
> 10:04:09 The message received from the daemon indicates that the daemon has
> disappeared.
> 10:04:09 Build request sent: Build{id=37ea63c1-ec5f-492a-ad6a-f26c92fd7f7e,
> currentDir=/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_VR_Spark/src}
> 10:04:09 Attempting to read last messages from the daemon log...
> 10:04:09 Daemon pid: 3299301
> 10:04:09 log file: /home/jenkins/.gradle/daemon/7.3.2/daemon-3299301.out.log
> 10:04:09 - Last 20 lines from daemon log file - daemon-3299301.out.log
> -
> 10:04:09 2022-02-02T18:04:08.251+ [DEBUG]
> [org.gradle.launcher.daemon.registry.PersistentDaemonRegistry] Marking busy
> by address: [2bd8c084-af4c-4924-90f5-8b8a89f85fc5 port:32889,
> addresses:[localhost/127.0.0.1]]
> 10:04:09 2022-02-02T18:04:08.251+ [DEBUG]
> [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire
> exclusive lock on daemon addresses registry.
> 10:04:09 2022-02-02T18:04:08.251+ [DEBUG]
> [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon
> addresses registry.
> 10:04:09 2022-02-02T18:04:08.252+ [DEBUG]
> [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon
> addresses registry.
> 10:04:09 2022-02-02T18:04:08.252+ [DEBUG]
> [org.gradle.launcher.daemon.server.DaemonStateCoordinator] resetting idle
> timer
> 10:04:09 2022-02-02T18:04:08.252+ [DEBUG]
> [org.gradle.launcher.daemon.server.DaemonStateCoordinator] daemon is running.
> Sleeping until state changes.
> 10:04:09 2022-02-02T18:04:08.253+ [INFO]
> [org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy] Daemon
> is about to start building Build{id=37ea63c1-ec5f-492a-ad6a-f26c92fd7f7e,
> currentDir=/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_VR_Spark/src}.
> Dispatching build started information...
> 10:04:09 2022-02-02T18:04:08.253+ [DEBUG]
> [org.gradle.launcher.daemon.server.SynchronizedDispatchConnection] thread
> 266: dispatching org.gradle.launcher.daemon.protocol.BuildStarted@42569c28
> 10:04:09 2022-02-02T18:04:08.255+ [DEBUG]
> [org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment]
> Configuring env variables: [PATH, RUN_DISPLAY_URL, HUDSON_HOME,
> RUN_CHANGES_DISPLAY_URL, JOB_URL, HUDSON_COOKIE, MOTD_SHOWN,
> DBUS_SESSION_BUS_ADDRESS, JENKINS_SERVER_COOKIE, LOGNAME, PWD,
> RUN_TESTS_DISPLAY_URL, JENKINS_URL, SHELL, BUILD_TAG, ROOT_BUILD_CAUSE,
> BUILD_CAUSE_TIMERTRIGGER, OLDPWD, GIT_CHECKOUT_DIR, JENKINS_HOME, sha1,
> CODECOV_TOKEN, NODE_NAME, BUILD_DISPLAY_NAME, JOB_DISPLAY_URL, GIT_BRANCH,
> SETUPTOOLS_USE_DISTUTILS, SHLVL, WORKSPACE_TMP, GIT_PREVIOUS_COMMIT,
> JAVA_HOME, BUILD_ID, LANG, XDG_SESSION_ID, XDG_SESSION_TYPE, JOB_NAME,
> SPARK_LOCAL_IP, BUILD_CAUSE, GIT_PREVIOUS_SUCCESSFUL_COMMIT, NODE_LABELS,
> HUDSON_URL, WORKSPACE, ROOT_BUILD_CAUSE_TIMERTRIGGER, XDG_SESSION_CLASS, _,
> GIT_COMMIT, COVERALLS_REPO_TOKEN, CI, EXECUTOR_NUMBER, HUDSON_SERVER_COOKIE,
> SSH_CLIENT, JOB_BASE_NAME, USER, SSH_CONNECTION, BUILD_NUMBER, BUILD_URL,
> RUN_ARTIFACTS_DISPLAY_URL, GIT_URL, XDG_RUNTIME_DIR, HOME]
> 10:04:09 2022-02-02T18:04:08.256+ [DEBUG]
> [org.gradle.launcher.daemon.server.exec.Lo
[jira] [Assigned] (BEAM-13797) Flakes: Failed to load cache entry
[ https://issues.apache.org/jira/browse/BEAM-13797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay reassigned BEAM-13797: -- Assignee: Kyle Weaver > Flakes: Failed to load cache entry > -- > > Key: BEAM-13797 > URL: https://issues.apache.org/jira/browse/BEAM-13797 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: P1 > Labels: flake > > I've seen this happen with multiple Jenkins jobs. The task will fail when > this happens. > Failed to load cache entry for task ':runners:portability:java:compileJava' > https://ci-beam.apache.org/job/beam_PreCommit_Java_PVR_Flink_Docker_Commit/235/console -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (BEAM-13810) Flaky tests: Gradle build daemon disappeared unexpectedly
[
https://issues.apache.org/jira/browse/BEAM-13810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmet Altay reassigned BEAM-13810:
--
Assignee: Kenneth Knowles
> Flaky tests: Gradle build daemon disappeared unexpectedly
> -
>
> Key: BEAM-13810
> URL: https://issues.apache.org/jira/browse/BEAM-13810
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Kyle Weaver
>Assignee: Kenneth Knowles
>Priority: P1
> Labels: flake
>
> This happens to many of our tests. It looks like this is a common issue with
> Gradle and we will have to do more digging to determine the true cause.
> https://stackoverflow.com/questions/37171043/gradle-build-daemon-disappeared-unexpectedly-it-may-have-been-killed-or-may-hav/37171110
> 10:04:08 > Task :sdks:go:test:sparkValidatesRunner
> 10:04:08 Feb 02, 2022 6:04:08 PM
> org.apache.beam.sdk.expansion.service.ExpansionService
> loadRegisteredTransforms
> 10:04:08 INFO: Registering external transforms:
> [beam:transforms:xlang:test:cgbk, beam:transforms:xlang:test:flatten,
> beam:transforms:xlang:test:prefix, beam:transforms:xlang:test:multi,
> beam:transforms:xlang:test:gbk, beam:transforms:xlang:test:comgl,
> beam:transforms:xlang:test:compk, beam:transforms:xlang:count,
> beam:transforms:xlang:filter_less_than_eq,
> beam:transforms:xlang:test:partition,
> beam:transforms:xlang:test:parquet_write, beam:transforms:xlang:parquet_read,
> beam:transforms:xlang:textio_read, beam:external:java:generate_sequence:v1]
> 10:04:09 The message received from the daemon indicates that the daemon has
> disappeared.
> 10:04:09 Build request sent: Build{id=37ea63c1-ec5f-492a-ad6a-f26c92fd7f7e,
> currentDir=/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_VR_Spark/src}
> 10:04:09 Attempting to read last messages from the daemon log...
> 10:04:09 Daemon pid: 3299301
> 10:04:09 log file: /home/jenkins/.gradle/daemon/7.3.2/daemon-3299301.out.log
> 10:04:09 - Last 20 lines from daemon log file - daemon-3299301.out.log
> -
> 10:04:09 2022-02-02T18:04:08.251+ [DEBUG]
> [org.gradle.launcher.daemon.registry.PersistentDaemonRegistry] Marking busy
> by address: [2bd8c084-af4c-4924-90f5-8b8a89f85fc5 port:32889,
> addresses:[localhost/127.0.0.1]]
> 10:04:09 2022-02-02T18:04:08.251+ [DEBUG]
> [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire
> exclusive lock on daemon addresses registry.
> 10:04:09 2022-02-02T18:04:08.251+ [DEBUG]
> [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon
> addresses registry.
> 10:04:09 2022-02-02T18:04:08.252+ [DEBUG]
> [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon
> addresses registry.
> 10:04:09 2022-02-02T18:04:08.252+ [DEBUG]
> [org.gradle.launcher.daemon.server.DaemonStateCoordinator] resetting idle
> timer
> 10:04:09 2022-02-02T18:04:08.252+ [DEBUG]
> [org.gradle.launcher.daemon.server.DaemonStateCoordinator] daemon is running.
> Sleeping until state changes.
> 10:04:09 2022-02-02T18:04:08.253+ [INFO]
> [org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy] Daemon
> is about to start building Build{id=37ea63c1-ec5f-492a-ad6a-f26c92fd7f7e,
> currentDir=/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_VR_Spark/src}.
> Dispatching build started information...
> 10:04:09 2022-02-02T18:04:08.253+ [DEBUG]
> [org.gradle.launcher.daemon.server.SynchronizedDispatchConnection] thread
> 266: dispatching org.gradle.launcher.daemon.protocol.BuildStarted@42569c28
> 10:04:09 2022-02-02T18:04:08.255+ [DEBUG]
> [org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment]
> Configuring env variables: [PATH, RUN_DISPLAY_URL, HUDSON_HOME,
> RUN_CHANGES_DISPLAY_URL, JOB_URL, HUDSON_COOKIE, MOTD_SHOWN,
> DBUS_SESSION_BUS_ADDRESS, JENKINS_SERVER_COOKIE, LOGNAME, PWD,
> RUN_TESTS_DISPLAY_URL, JENKINS_URL, SHELL, BUILD_TAG, ROOT_BUILD_CAUSE,
> BUILD_CAUSE_TIMERTRIGGER, OLDPWD, GIT_CHECKOUT_DIR, JENKINS_HOME, sha1,
> CODECOV_TOKEN, NODE_NAME, BUILD_DISPLAY_NAME, JOB_DISPLAY_URL, GIT_BRANCH,
> SETUPTOOLS_USE_DISTUTILS, SHLVL, WORKSPACE_TMP, GIT_PREVIOUS_COMMIT,
> JAVA_HOME, BUILD_ID, LANG, XDG_SESSION_ID, XDG_SESSION_TYPE, JOB_NAME,
> SPARK_LOCAL_IP, BUILD_CAUSE, GIT_PREVIOUS_SUCCESSFUL_COMMIT, NODE_LABELS,
> HUDSON_URL, WORKSPACE, ROOT_BUILD_CAUSE_TIMERTRIGGER, XDG_SESSION_CLASS, _,
> GIT_COMMIT, COVERALLS_REPO_TOKEN, CI, EXECUTOR_NUMBER, HUDSON_SERVER_COOKIE,
> SSH_CLIENT, JOB_BASE_NAME, USER, SSH_CONNECTION, BUILD_NUMBER, BUILD_URL,
> RUN_ARTIFACTS_DISPLAY_URL, GIT_URL, XDG_RUNTIME_DIR, HOME]
> 10:04:09 2022-02-02T18:04:08.256+ [DEBUG]
> [org.gradle.launcher.daemon.server.exec.LogToClient] About to start relaying
> all logs to the client via th
[jira] [Commented] (BEAM-13797) Flakes: Failed to load cache entry
[ https://issues.apache.org/jira/browse/BEAM-13797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17487388#comment-17487388 ] Ahmet Altay commented on BEAM-13797: Is this a Java generic issue? or a Flink runner issue? If it is a Flink runner issue, Kyle you might be the person to investigate. :) > Flakes: Failed to load cache entry > -- > > Key: BEAM-13797 > URL: https://issues.apache.org/jira/browse/BEAM-13797 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Kyle Weaver >Priority: P1 > Labels: flake > > I've seen this happen with multiple Jenkins jobs. The task will fail when > this happens. > Failed to load cache entry for task ':runners:portability:java:compileJava' > https://ci-beam.apache.org/job/beam_PreCommit_Java_PVR_Flink_Docker_Commit/235/console -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (BEAM-13798) Upgrade Kubernetes Clusters
Ahmet Altay created BEAM-13798: -- Summary: Upgrade Kubernetes Clusters Key: BEAM-13798 URL: https://issues.apache.org/jira/browse/BEAM-13798 Project: Beam Issue Type: Bug Components: io-common, community-metrics Reporter: Ahmet Altay Assignee: Kiley Sok """ Kubernetes version 1.18 will not be supported starting Mar. 31, 2022. Upgrade your nodes to mitigate instability and security risks. ... What do I need to do? You can mitigate instability and security risks by manually upgrading your nodes from Kubernetes minor versions 1.18 and older to a supported version before March 31, 2022, when support for 1.18 ends. The following clusters in your project are currently running an unsupported Kubernetes minor version. To resume support for these clusters, an upgrade to a supported version is required. Apache Beam Testing: (apache-beam-testing) io-datastores metrics metrics-staging """ -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (BEAM-13763) Rotate credentials for 'io-datastores' Kubernetes cluster
[ https://issues.apache.org/jira/browse/BEAM-13763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay reassigned BEAM-13763: -- Assignee: Chamikara Madhusanka Jayalath > Rotate credentials for 'io-datastores' Kubernetes cluster > - > > Key: BEAM-13763 > URL: https://issues.apache.org/jira/browse/BEAM-13763 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P1 > > As per following notice from GCP sent to some owners of the > apache-beam-testing project, credentials if the Kubernetes cluster used for > testing will expire on 03/02/2022 if no action is taken and some test suites > might start failing. > - > Our records show that the CA for the following cluster(s) in your project(s) > will expire soon: > Apache Beam Testing (apache-beam-testing) > Cluster: io-datastores CA will expire on Wed Mar 2 11:48:44 AM PST 2022. > To prevent normal cluster operations from being interrupted, please take > these steps before the CA’s expiration: > Perform a credential rotation on the above clusters. > Update all API clients outside of the clusters (such as kubectl on developer > machines) to use the new credentials once credential rotation has been > initiated. > - -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (BEAM-13709) PipelineOptions() and from_dictionary parsing use_public_ips and no_use_public_ips differently
[
https://issues.apache.org/jira/browse/BEAM-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmet Altay reassigned BEAM-13709:
--
Assignee: Valentyn Tymofieiev
> PipelineOptions() and from_dictionary parsing use_public_ips and
> no_use_public_ips differently
> --
>
> Key: BEAM-13709
> URL: https://issues.apache.org/jira/browse/BEAM-13709
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Minbo Bae
>Assignee: Valentyn Tymofieiev
>Priority: P2
> Labels: starter
>
> {{PipelineOptions}} in Python has two methods to pass a param dict: using in
> constructor {{PipelineOptions(**params)}} or
> {{{}PipelineOptions.from_dictionary(params){}}}.
> But, they work slightly differently:
> *
> [PipelineOptions(**params)|https://github.com/apache/beam/blob/v2.35.0/sdks/python/apache_beam/options/pipeline_options.py#L313-L324]
> discards an option if it is not defined as a dest of {{argparse}} in an
> Option class. For example, {{no_use_public_ips=True}} is ignored and the
> Dataflow job will run with public IPs. To disable public IPs, the option
> dictionary must use {{{}use_public_ips{}}}.
> *
> [PipelineOptions.from_dictionary()|https://github.com/apache/beam/blob/v2.35.0/sdks/python/apache_beam/options/pipeline_options.py#L229]
> skips an option if the option value is {{{}False{}}}. For example,
> {{use_public_ips=False}} is ignored and the Dataflow job will run with public
> IPs. To disable public IPs, the option dictionary must use
> {{no_use_public_ips.}}
> This makes the user very confused, and sometimes the pipeline works in an
> unexpected way.
> We must have the consistent behavior between the two methods, or at least a
> warning about invalid ignored options.
> BEAM-9093 dealt with a similar issue for {{PipelineOptions()}}. Like the
> issue, I guess adding a warning in `PipelineOptions.from_dictionary()` for
> ignored options can help reducing the confusion, if we cannot have two
> methods have exactly the same behavior.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (BEAM-13742) beam_PostRelease_NightSnapshot failing to stage jars
[
https://issues.apache.org/jira/browse/BEAM-13742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmet Altay reassigned BEAM-13742:
--
Assignee: Kenneth Knowles
> beam_PostRelease_NightSnapshot failing to stage jars
>
>
> Key: BEAM-13742
> URL: https://issues.apache.org/jira/browse/BEAM-13742
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Brian Hulette
>Assignee: Kenneth Knowles
>Priority: P1
> Labels: currently-failing
>
> Started in
> https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/1631 (after
> Dataflow container update resolved another issue)
> Dataflow job is failing to start with message:
> {code}
> 05:47:55 Jan 25, 2022 1:47:55 PM
> org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
> 05:47:55 SEVERE: 2022-01-25T13:47:53.288Z: Staged package
> beam-vendor-bytebuddy-1_11_0-0.1-SneHjzQgCpxxLkvyN8dugqO_sTzwXJ2995kqFu4l2i4.jar
> at location
> 'gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-vendor-bytebuddy-1_11_0-0.1-SneHjzQgCpxxLkvyN8dugqO_sTzwXJ2995kqFu4l2i4.jar'
> is inaccessible.
> 05:47:55 Jan 25, 2022 1:47:55 PM
> org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
> 05:47:55 SEVERE: 2022-01-25T13:47:53.479Z: Staged package
> checker-compat-qual-2.5.3-12ua_qYcfAgpCAI_DLwUJ_q5q9LfkVyLij56UJvMvG0.jar at
> location
> 'gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/checker-compat-qual-2.5.3-12ua_qYcfAgpCAI_DLwUJ_q5q9LfkVyLij56UJvMvG0.jar'
> is inaccessible.
> 05:47:59 Jan 25, 2022 1:47:59 PM
> org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
> 05:47:59 WARNING: 2022-01-25T13:47:58.697Z: One or more access checks for
> temp location or staged files failed. Please refer to other error messages
> for details. For more information on security and permissions, please see
> https://cloud.google.com/dataflow/security-and-permissions.
> 05:51:00 Jan 25, 2022 1:51:00 PM
> org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
> 05:51:00 WARNING: 2022-01-25T13:50:57.953Z: One or more access checks for
> temp location or staged files failed. Please refer to other error messages
> for details. For more information on security and permissions, please see
> https://cloud.google.com/dataflow/security-and-permissions.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (BEAM-12847) Error running Top transform in Dataflow
[
https://issues.apache.org/jira/browse/BEAM-12847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmet Altay reassigned BEAM-12847:
--
Assignee: Valentyn Tymofieiev
> Error running Top transform in Dataflow
> ---
>
> Key: BEAM-12847
> URL: https://issues.apache.org/jira/browse/BEAM-12847
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
>Affects Versions: 2.32.0
>Reporter: Miren Esnaola
>Assignee: Valentyn Tymofieiev
>Priority: P3
>
> When running the following transform in Dataflow (the problem does not happen
> with the direct running)... I is a streaming pipeline where I am using a
> SlidingWindow.
> {code:java}
> beam.combiners.Top.Of(n=10, key=lambda item: item[1]).without_defaults()
> {code}
>
> I am getting an error
>
> {code:java}
> Error message from worker: generic::unknown: Traceback (most recent call
> last): File
> "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 284, in _execute response = task() File
> "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 357, in lambda: self.create_worker().do_instruction(request),
> request) File
> "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 602, in do_instruction getattr(request, request_type),
> request.instruction_id) File
> "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 639, in process_bundle bundle_processor.process_bundle(instruction_id))
> File
> "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 994, in process_bundle element.data) File
> "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 222, in process_encoded self.output(decoded_value) File
> "apache_beam/runners/worker/operations.py", line 351, in
> apache_beam.runners.worker.operations.Operation.output File
> "apache_beam/runners/worker/operations.py", line 353, in
> apache_beam.runners.worker.operations.Operation.output File
> "apache_beam/runners/worker/operations.py", line 215, in
> apache_beam.runners.worker.operations.SingletonConsumerSet.receive File
> "apache_beam/runners/worker/operations.py", line 921, in
> apache_beam.runners.worker.operations.CombineOperation.process File
> "apache_beam/runners/worker/operations.py", line 925, in
> apache_beam.runners.worker.operations.CombineOperation.process File
> "/usr/local/lib/python3.7/site-packages/apache_beam/transforms/combiners.py",
> line 835, in extract_only return self.combine_fn.extract_output(accumulator)
> File
> "/usr/local/lib/python3.7/site-packages/apache_beam/transforms/combiners.py",
> line 502, in extract_output heap.sort(reverse=True) File
> "apache_beam/transforms/cy_combiners.py", line 389, in
> apache_beam.transforms.cy_combiners.ComparableValue.__lt__ AssertionError
> passed through: ==>
> dist_proc/dax/workflow/worker/fnapi_service_impl.cc:644{code}
>
> {{What might be the cause of this issue?}}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (BEAM-13665) Spanner IO request metrics requires projectId within the config when it didn't in the past
[ https://issues.apache.org/jira/browse/BEAM-13665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476309#comment-17476309 ] Ahmet Altay commented on BEAM-13665: [~chamikara] [~kerrydc] - This is a backward incompatibility. Should this be a higher priority and block the next release? > Spanner IO request metrics requires projectId within the config when it > didn't in the past > -- > > Key: BEAM-13665 > URL: https://issues.apache.org/jira/browse/BEAM-13665 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Reporter: Luke Cwik >Priority: P2 > Fix For: 2.36.0 > > > https://github.com/apache/beam/pull/15493 makes the GCP projectID a required > parameter - which it was not before, as it could be inferred from the > environment - and thus breaks backward compatibility. > Specifically: BatchSpannerRead.java:175 -- the toString() is not required on > the valueProvider, it should just be a get(), and createServiceCallMetric() > in line 195 should handle the situation where projectID could be null -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (BEAM-13504) Remove provided/compileOnly deps not intended for external use
[ https://issues.apache.org/jira/browse/BEAM-13504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay reassigned BEAM-13504: -- Assignee: Emily Ye > Remove provided/compileOnly deps not intended for external use > -- > > Key: BEAM-13504 > URL: https://issues.apache.org/jira/browse/BEAM-13504 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Daniel Collins >Assignee: Emily Ye >Priority: P1 > Fix For: 2.36.0 > > Time Spent: 20m > Remaining Estimate: 0h > > There are a lot of compileOnly deps in place to make certain libraries > lighter. In many cases, these don't actually reduce the size of the resulting > binary, and are a result of an overriding strategy where the types of the > implementation are needed in the loading library. > In the short term, these can all be replaced by `implementation`, with the > downside that jars will have extra explicit dependencies, and the upside that > users will not experience runtime errors due to missing dependencies. In the > longer term, we should restructure these libraries (sql being the worst > offender) to use serviceLoaders or other strategies that do not require > naming the types of the overriders within the same package that the interface > is declared. > The one exception to this is libraries that we intend to use whatever version > the user puts on the classpath at runtime without expressing a version > preference, such as hadoop or kafka. We should still consider if this has > value though, as users are still free to explicitly override the dependency > version we choose, even if we declare an `implementation` dependency on it. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (BEAM-13575) Flink testParDoRequiresStableInput flaky
[ https://issues.apache.org/jira/browse/BEAM-13575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay reassigned BEAM-13575: -- Assignee: Kyle Weaver > Flink testParDoRequiresStableInput flaky > > > Key: BEAM-13575 > URL: https://issues.apache.org/jira/browse/BEAM-13575 > Project: Beam > Issue Type: Bug > Components: runner-flink, test-failures >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: P2 > Labels: flake > Time Spent: 40m > Remaining Estimate: 0h > > https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/20253/ > org.apache.beam.runners.flink.FlinkRequiresStableInputTest.testParDoRequiresStableInput > java.util.concurrent.ExecutionException: > java.util.concurrent.CompletionException: > org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint > triggering task Source: Impulse -> > CreatePCollectionOfOneValue/Read(CreateSource)/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) > -> > CreatePCollectionOfOneValue/Read(CreateSource)/ParDo(BoundedSourceAsSDFWrapper)/ParMultiDo(BoundedSourceAsSDFWrapper)/Pair > with initial restriction/ParMultiDo(PairWithRestriction) -> > CreatePCollectionOfOneValue/Read(CreateSource)/ParDo(BoundedSourceAsSDFWrapper)/ParMultiDo(BoundedSourceAsSDFWrapper)/Split > restriction/ParMultiDo(SplitRestriction) -> > CreatePCollectionOfOneValue/Read(CreateSource)/ParDo(BoundedSourceAsSDFWrapper)/ParMultiDo(BoundedSourceAsSDFWrapper)/Explode > windows/ParMultiDo(ExplodeWindows) -> > CreatePCollectionOfOneValue/Read(CreateSource)/ParDo(BoundedSourceAsSDFWrapper)/ParMultiDo(BoundedSourceAsSDFWrapper)/Assign > unique key/AddKeys/Map/ParMultiDo(Anonymous) -> ToKeyedWorkItem (1/1) of job > 7bbb425ba325dbc1dc4d3cdf1c8b88f9 is not being executed at the moment. > Aborting checkpoint. Failure reason: Not all required tasks are currently > running. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (BEAM-13630) Improve documentation for the Dataflow Service options
[
https://issues.apache.org/jira/browse/BEAM-13630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17474764#comment-17474764
]
Ahmet Altay commented on BEAM-13630:
[~pcoet] - Is there are way to link to `dataflowServiceOptions` under
https://cloud.google.com/dataflow/docs/reference/pipeline-options#basic_options
Also the documentation there could also be a bit expanded. Currently it says:
{code:java}
Specifies additional job modes and configurations. Also provides forward
compatibility for SDK versions that don’t have explicit pipeline options for
later Dataflow features. Requires Apache Beam SDK 2.29.0 or later. For example,
to use Dataflow Prime, set: --dataflowServiceOptions=enable_prime.
{code}
After that we can update the code comments.
> Improve documentation for the Dataflow Service options
> ---
>
> Key: BEAM-13630
> URL: https://issues.apache.org/jira/browse/BEAM-13630
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow, website
>Affects Versions: 2.35.0
>Reporter: Matt Casters
>Assignee: David Huntsperger
>Priority: P2
> Labels: dataflow, documentation, options
>
> Class DataflowPipelineOptions has the ability to set "Dataflow service
> options".
> The description in the code is:
> {code:java}
> Service options are set by the user and configure the service. This decouples
> service side feature availability from the Apache Beam release cycle.{code}
> It would be great to have a link here to the Google cloud documentation where
> we can find a list of options or perhaps an example on how such an option
> looks like.
> See also: https://github.com/apache/beam/pull/14159#discussion_r589727367
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (BEAM-13630) Improve documentation for the Dataflow Service options
[
https://issues.apache.org/jira/browse/BEAM-13630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmet Altay reassigned BEAM-13630:
--
Assignee: David Huntsperger
> Improve documentation for the Dataflow Service options
> ---
>
> Key: BEAM-13630
> URL: https://issues.apache.org/jira/browse/BEAM-13630
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow, website
>Affects Versions: 2.35.0
>Reporter: Matt Casters
>Assignee: David Huntsperger
>Priority: P2
> Labels: dataflow, documentation, options
>
> Class DataflowPipelineOptions has the ability to set "Dataflow service
> options".
> The description in the code is:
> {code:java}
> Service options are set by the user and configure the service. This decouples
> service side feature availability from the Apache Beam release cycle.{code}
> It would be great to have a link here to the Google cloud documentation where
> we can find a list of options or perhaps an example on how such an option
> looks like.
> See also: https://github.com/apache/beam/pull/14159#discussion_r589727367
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (BEAM-13630) Improve documentation for the Dataflow Service options
[
https://issues.apache.org/jira/browse/BEAM-13630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmet Altay updated BEAM-13630:
---
Component/s: website
> Improve documentation for the Dataflow Service options
> ---
>
> Key: BEAM-13630
> URL: https://issues.apache.org/jira/browse/BEAM-13630
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow, website
>Affects Versions: 2.35.0
>Reporter: Matt Casters
>Priority: P2
> Labels: dataflow, documentation, options
>
> Class DataflowPipelineOptions has the ability to set "Dataflow service
> options".
> The description in the code is:
> {code:java}
> Service options are set by the user and configure the service. This decouples
> service side feature availability from the Apache Beam release cycle.{code}
> It would be great to have a link here to the Google cloud documentation where
> we can find a list of options or perhaps an example on how such an option
> looks like.
> See also: https://github.com/apache/beam/pull/14159#discussion_r589727367
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (BEAM-13504) Remove provided/compileOnly deps not intended for external use
[ https://issues.apache.org/jira/browse/BEAM-13504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay updated BEAM-13504: --- Priority: P1 (was: P2) > Remove provided/compileOnly deps not intended for external use > -- > > Key: BEAM-13504 > URL: https://issues.apache.org/jira/browse/BEAM-13504 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Daniel Collins >Priority: P1 > Fix For: 2.36.0 > > > There are a lot of compileOnly deps in place to make certain libraries > lighter. In many cases, these don't actually reduce the size of the resulting > binary, and are a result of an overriding strategy where the types of the > implementation are needed in the loading library. > In the short term, these can all be replaced by `implementation`, with the > downside that jars will have extra explicit dependencies, and the upside that > users will not experience runtime errors due to missing dependencies. In the > longer term, we should restructure these libraries (sql being the worst > offender) to use serviceLoaders or other strategies that do not require > naming the types of the overriders within the same package that the interface > is declared. > The one exception to this is libraries that we intend to use whatever version > the user puts on the classpath at runtime without expressing a version > preference, such as hadoop or kafka. We should still consider if this has > value though, as users are still free to explicitly override the dependency > version we choose, even if we declare an `implementation` dependency on it. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (BEAM-13606) bigtable io doesn't handle non-ok row mutations
[ https://issues.apache.org/jira/browse/BEAM-13606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17470324#comment-17470324 ] Ahmet Altay commented on BEAM-13606: [~chamikara] [~kerrydc] - Would it be possible to verify if there is a risk of data loss in this scenario? If that is the case, we need to fix it sooner than later. Do we have an owner for bigtable IO from the bigtable people? > bigtable io doesn't handle non-ok row mutations > --- > > Key: BEAM-13606 > URL: https://issues.apache.org/jira/browse/BEAM-13606 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Reporter: Ning >Assignee: Chamikara Madhusanka Jayalath >Priority: P1 > > bigtable io has no logic to retry row mutations for rows with non-ok return > status (this includes None return value when bigtable suppresses retryable > errors, details see BEAM-13602). > > To avoid data loss, the solution should be: > # Retry for those retryable-failed row mutations; > # Tagged output for those non-retryable-failed row mutations. > Or clarify that the I/O doesn't handle failed row mutations in docstrings. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (BEAM-13606) bigtable io doesn't handle non-ok row mutations
[ https://issues.apache.org/jira/browse/BEAM-13606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay reassigned BEAM-13606: -- Assignee: Chamikara Madhusanka Jayalath > bigtable io doesn't handle non-ok row mutations > --- > > Key: BEAM-13606 > URL: https://issues.apache.org/jira/browse/BEAM-13606 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Reporter: Ning >Assignee: Chamikara Madhusanka Jayalath >Priority: P2 > > bigtable io has no logic to retry row mutations for rows with non-ok return > status (this includes None return value when bigtable suppresses retryable > errors, details see BEAM-13602). > > To avoid data loss, the solution should be: > # Retry for those retryable-failed row mutations; > # Tagged output for those non-retryable-failed row mutations. > Or clarify that the I/O doesn't handle failed row mutations in docstrings. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (BEAM-13606) bigtable io doesn't handle non-ok row mutations
[ https://issues.apache.org/jira/browse/BEAM-13606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay updated BEAM-13606: --- Priority: P1 (was: P2) > bigtable io doesn't handle non-ok row mutations > --- > > Key: BEAM-13606 > URL: https://issues.apache.org/jira/browse/BEAM-13606 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Reporter: Ning >Assignee: Chamikara Madhusanka Jayalath >Priority: P1 > > bigtable io has no logic to retry row mutations for rows with non-ok return > status (this includes None return value when bigtable suppresses retryable > errors, details see BEAM-13602). > > To avoid data loss, the solution should be: > # Retry for those retryable-failed row mutations; > # Tagged output for those non-retryable-failed row mutations. > Or clarify that the I/O doesn't handle failed row mutations in docstrings. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (BEAM-13522) Spark tests failing PerKeyOrderingTest
[ https://issues.apache.org/jira/browse/BEAM-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay reassigned BEAM-13522: -- Assignee: Pablo Estrada > Spark tests failing PerKeyOrderingTest > -- > > Key: BEAM-13522 > URL: https://issues.apache.org/jira/browse/BEAM-13522 > Project: Beam > Issue Type: Bug > Components: runner-spark, test-failures >Reporter: Kyle Weaver >Assignee: Pablo Estrada >Priority: P1 > > Looks like new tests were added missing annotations. > java.lang.IllegalStateException: Not expected to directly translate > splittable DoFn, should have been overridden: > org.apache.beam.sdk.transforms.PeriodicSequence$PeriodicSequenceFn@3fddc80d > https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (BEAM-12830) Remove GoGradle plugin
[ https://issues.apache.org/jira/browse/BEAM-12830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay updated BEAM-12830: --- Parent: BEAM-13430 Issue Type: Sub-task (was: Improvement) > Remove GoGradle plugin > -- > > Key: BEAM-12830 > URL: https://issues.apache.org/jira/browse/BEAM-12830 > Project: Beam > Issue Type: Sub-task > Components: sdk-go >Reporter: Robert Burke >Assignee: Robert Burke >Priority: P2 > Time Spent: 8h > Remaining Estimate: 0h > > GoGradle has never been updated to handle Go Modules, and it turns out it > leaves .gogradle folders everywhere, which contain it's local gopath, and > it's non-writable mod caches. This was the root cause of > https://issues.apache.org/jira/browse/BEAM-12829. > The solution is to stop using it, as it's never been ideal for our use, and > replace it with direct shellouts to the go command. > The main functionality that needs replacing is getting and using a set Go > version so we can be sure what we're testing/validating against and building > with, and using a common GoPath/ModuleCache throughout the repo. Otherwise, > most instances can be replaced with simple tasks that call the Go function. > A single module cache, if able to be reused across jenkins builds will also > reduce build times. Note that these are for external versioned dependencies > anyway, and not for Beam repo code. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (BEAM-13597) Java GHA Test (Wordcount on Dataflow) failing with ClassNotFoundException
[
https://issues.apache.org/jira/browse/BEAM-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmet Altay reassigned BEAM-13597:
--
Assignee: Kyle Weaver
> Java GHA Test (Wordcount on Dataflow) failing with ClassNotFoundException
> -
>
> Key: BEAM-13597
> URL: https://issues.apache.org/jira/browse/BEAM-13597
> Project: Beam
> Issue Type: Sub-task
> Components: test-failures
>Reporter: Brian Hulette
>Assignee: Kyle Weaver
>Priority: P1
>
> Java GHA tests that run wordount on Dataflow started failing a couple weeks
> ago:
> https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule.
> Last success: https://github.com/apache/beam/actions/runs/1604907786, first
> failure: https://github.com/apache/beam/actions/runs/1609549754
> {code}
> org.apache.beam.examples.WordCountIT > testE2EWordCount FAILED
> java.lang.IllegalArgumentException at WordCountIT.java:58
> Caused by: java.lang.ClassNotFoundException at WordCountIT.java:58
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (BEAM-13010) Delete orphaned files
[ https://issues.apache.org/jira/browse/BEAM-13010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17468817#comment-17468817 ] Ahmet Altay commented on BEAM-13010: Should this move to 2.37.0 or is it a hard blocker for 2.36.0? > Delete orphaned files > - > > Key: BEAM-13010 > URL: https://issues.apache.org/jira/browse/BEAM-13010 > Project: Beam > Issue Type: Bug > Components: io-py-files >Affects Versions: 2.34.0 >Reporter: David >Assignee: Pablo Estrada >Priority: P1 > Fix For: 2.36.0 > > > Until version 2.33.0 of Apache Beam, (tested with a Python streaming pipeline > consuming events from PubSub and writing them into GCS), some files were > being deleted from the temporary folder before being moved to the > destination. This was the original issue: > https://issues.apache.org/jira/browse/BEAM-12950 > In version 2.34.0 we applied a temporary workaround to be sure that no data > is dropped. Instead of deleting the orphaned files, we just log them: > [https://github.com/apache/beam/pull/15576] > Most probably the root cause of the missing event was that we were removing > files at an erroneous time. We need to delete orphaned files in a subsequent > step (after we're sure that there won't be retries). > Once the original issue is fixed and the orphaned files are deleted at the > correct time, we should remove the decorator of the unit test skipped in the > Pull Request above. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (BEAM-13010) Delete orphaned files
[ https://issues.apache.org/jira/browse/BEAM-13010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay reassigned BEAM-13010: -- Assignee: Pablo Estrada > Delete orphaned files > - > > Key: BEAM-13010 > URL: https://issues.apache.org/jira/browse/BEAM-13010 > Project: Beam > Issue Type: Bug > Components: io-py-files >Affects Versions: 2.34.0 >Reporter: David >Assignee: Pablo Estrada >Priority: P1 > Fix For: 2.36.0 > > > Until version 2.33.0 of Apache Beam, (tested with a Python streaming pipeline > consuming events from PubSub and writing them into GCS), some files were > being deleted from the temporary folder before being moved to the > destination. This was the original issue: > https://issues.apache.org/jira/browse/BEAM-12950 > In version 2.34.0 we applied a temporary workaround to be sure that no data > is dropped. Instead of deleting the orphaned files, we just log them: > [https://github.com/apache/beam/pull/15576] > Most probably the root cause of the missing event was that we were removing > files at an erroneous time. We need to delete orphaned files in a subsequent > step (after we're sure that there won't be retries). > Once the original issue is fixed and the orphaned files are deleted at the > correct time, we should remove the decorator of the unit test skipped in the > Pull Request above. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (BEAM-13591) Bump log4j2 version to 2.17.1 due to the vulnerability
[ https://issues.apache.org/jira/browse/BEAM-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17468814#comment-17468814 ] Ahmet Altay commented on BEAM-13591: https://github.com/apache/beam/pull/16415 is merged. Closing this. > Bump log4j2 version to 2.17.1 due to the vulnerability > -- > > Key: BEAM-13591 > URL: https://issues.apache.org/jira/browse/BEAM-13591 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Affects Versions: 2.35.0 >Reporter: Ada Wong >Priority: P2 > Fix For: 2.36.0 > > Time Spent: 20m > Remaining Estimate: 0h > > https://logging.apache.org/log4j/2.x/security.html#CVE-2021-44832 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (BEAM-13591) Bump log4j2 version to 2.17.1 due to the vulnerability
[ https://issues.apache.org/jira/browse/BEAM-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay updated BEAM-13591: --- Resolution: Fixed Status: Resolved (was: Triage Needed) > Bump log4j2 version to 2.17.1 due to the vulnerability > -- > > Key: BEAM-13591 > URL: https://issues.apache.org/jira/browse/BEAM-13591 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Affects Versions: 2.35.0 >Reporter: Ada Wong >Priority: P2 > Fix For: 2.36.0 > > Time Spent: 20m > Remaining Estimate: 0h > > https://logging.apache.org/log4j/2.x/security.html#CVE-2021-44832 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (BEAM-13579) Cannot run python_xlang_kafka_taxi_dataflow validation script on 2.35.0
[
https://issues.apache.org/jira/browse/BEAM-13579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmet Altay updated BEAM-13579:
---
Fix Version/s: 2.36.0
> Cannot run python_xlang_kafka_taxi_dataflow validation script on 2.35.0
> ---
>
> Key: BEAM-13579
> URL: https://issues.apache.org/jira/browse/BEAM-13579
> Project: Beam
> Issue Type: Bug
> Components: release
>Reporter: Valentyn Tymofieiev
>Assignee: Chamikara Madhusanka Jayalath
>Priority: P1
> Fix For: 2.36.0
>
>
> Dataflow job gets errors like:
> {noformat}
> Error message from worker: generic::unknown:
> org.apache.beam.sdk.util.UserCodeException: java.io.IOException: KafkaWriter
> : failed to send 1 records (since last report)
>
> org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
>
> org.apache.beam.sdk.io.kafka.KafkaWriter$DoFnInvoker.invokeProcessElement(Unknown
> Source)
>
> org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo(FnApiDoFnRunner.java:758)
>
> org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:257)
>
> org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:209)
>
> org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo(FnApiDoFnRunner.java:1745)
>
> org.apache.beam.fn.harness.FnApiDoFnRunner.access$2700(FnApiDoFnRunner.java:142)
>
> org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.outputWithTimestamp(FnApiDoFnRunner.java:2263)
>
> org.apache.beam.fn.harness.FnApiDoFnRunner$ProcessBundleContextBase.output(FnApiDoFnRunner.java:2432)
>
> org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output(DoFnOutputReceivers.java:78)
>
> org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:142)
>
> org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(Unknown
> Source)
>
> org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo(FnApiDoFnRunner.java:758)
>
> org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:257)
>
> org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:209)
>
> org.apache.beam.fn.harness.BeamFnDataReadRunner.forwardElementToConsumer(BeamFnDataReadRunner.java:172)
>
> org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver2.awaitCompletion(BeamFnDataInboundObserver2.java:126)
>
> org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:467)
>
> org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:151)
>
> org.apache.beam.fn.harness.control.BeamFnControlClient$InboundObserver.lambda$onNext$0(BeamFnControlClient.java:116)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: KafkaWriter : failed to send 1 records (since
> last report)
>
> org.apache.beam.sdk.io.kafka.KafkaWriter.checkForFailures(KafkaWriter.java:133)
>
> org.apache.beam.sdk.io.kafka.KafkaWriter.processElement(KafkaWriter.java:58)
> Caused by: org.apache.kafka.common.errors.TimeoutException: Topic
> beam-runnerv2 not present in metadata after 6 ms.
> {noformat}
> run_rc_validation.sh fails with errors like:
> {noformat}
> * Please wait for at least 10 mins to let Dataflow job be launched and
> results get populated.
> * Sleeping for 10 mins
> * How to verify results:
> * 1. Goto your Dataflow job console and check whether there is any error.
> * 2. Check whether valentyn_python_validations_1229_10742.xlang_kafka_taxi
> has data, retrieving BigQuery data as below:
> BigQuery error in head operation: Not found: Table
> google.com:clouddfe:valentyn_python_validations_1229_10742.xlang_kafka_taxi
> Couldn't find expected output. Please confirm the output by visiting the
> console manually.
> {noformat}
> I suspect something is misconfigured but couldn't figure out what.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (BEAM-12879) Downloading GCS objects suddenly require storage.buckets.get permission
[
https://issues.apache.org/jira/browse/BEAM-12879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17466200#comment-17466200
]
Ahmet Altay commented on BEAM-12879:
[~tvalentyn] - Assigned this to find an owner. I do not think it is very
urgent, but is definitely a getting started and ease of use issue. It is also
possible a simple fix by ignoring the errors for unauthenticated users.
> Downloading GCS objects suddenly require storage.buckets.get permission
> ---
>
> Key: BEAM-12879
> URL: https://issues.apache.org/jira/browse/BEAM-12879
> Project: Beam
> Issue Type: Bug
> Components: io-py-gcp
>Affects Versions: 2.32.0
>Reporter: Robert Jany
>Assignee: Valentyn Tymofieiev
>Priority: P2
>
> With PR [https://github.com/apache/beam/pull/14770] downloading GCS objects
> requires an additional IAM role `storage.objects.get` to get the
> project_number based on the bucket name.
> If the service account or user does not have said role the following error
> will show:
> {code:python}
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.7/site-packages/dataflow_worker/batchworker.py", line
> 651, in do_work
> work_executor.execute()
> File "/usr/local/lib/python3.7/site-packages/dataflow_worker/executor.py",
> line 179, in execute
> op.start()
> File "dataflow_worker/native_operations.py", line 38, in
> dataflow_worker.native_operations.NativeReadOperation.start
> File "dataflow_worker/native_operations.py", line 39, in
> dataflow_worker.native_operations.NativeReadOperation.start
> File "dataflow_worker/native_operations.py", line 44, in
> dataflow_worker.native_operations.NativeReadOperation.start
> File "dataflow_worker/native_operations.py", line 54, in
> dataflow_worker.native_operations.NativeReadOperation.start
> File "apache_beam/runners/worker/operations.py", line 353, in
> apache_beam.runners.worker.operations.Operation.output
> File "apache_beam/runners/worker/operations.py", line 215, in
> apache_beam.runners.worker.operations.SingletonConsumerSet.receive
> File "apache_beam/runners/worker/operations.py", line 712, in
> apache_beam.runners.worker.operations.DoOperation.process
> File "apache_beam/runners/worker/operations.py", line 713, in
> apache_beam.runners.worker.operations.DoOperation.process
> File "apache_beam/runners/common.py", line 1234, in
> apache_beam.runners.common.DoFnRunner.process
> File "apache_beam/runners/common.py", line 1315, in
> apache_beam.runners.common.DoFnRunner._reraise_augmented
> File "apache_beam/runners/common.py", line 1232, in
> apache_beam.runners.common.DoFnRunner.process
> File "apache_beam/runners/common.py", line 571, in
> apache_beam.runners.common.SimpleInvoker.invoke_process
> File "apache_beam/runners/common.py", line 1368, in
> apache_beam.runners.common._OutputProcessor.process_outputs
> File "/usr/local/lib/python3.7/site-packages/xyz/package/file.py", line
> 112, in process
> with FileSystems.open(element["gcs_uri"]) as file:
> File
> "/usr/local/lib/python3.7/site-packages/apache_beam/io/filesystems.py", line
> 244, in open
> return filesystem.open(path, mime_type, compression_type)
> File
> "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/gcsfilesystem.py",
> line 177, in open
> return self._path_open(path, 'rb', mime_type, compression_type)
> File
> "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/gcsfilesystem.py",
> line 138, in _path_open
> raw_file = gcsio.GcsIO().open(path, mode, mime_type=mime_type)
> File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/gcsio.py",
> line 227, in open
> get_project_number=self.get_project_number)
> File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/gcsio.py",
> line 585, in __init__
> project_number = self._get_project_number(self._bucket)
> File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/gcsio.py",
> line 166, in get_project_number
> self.bucket_to_project_number[bucket] = bucket_metadata.projectNumber
> AttributeError: 'NoneType' object has no attribute 'projectNumber' [while
> running 'read from GCS']
> {code}
>
> The error message does not hint what goes wrong exactly but after some
> digging my assumption is that when trying to get the `bucket_metadata ` in
> [get_project_number|https://github.com/roger-mike/beam/blob/f0d0dd561a0955afb73cf595a3015a7ca839d5b7/sdks/python/apache_beam/io/gcp/gcsio.py#L161]
> we get a a HTTP Error and thus a None (since when catching this error a None
> is returned) due to the lack of permissions leading to `bucket_metadata`
> being None.
> The problem is, that the required permission (`storage.buckets.get`) is only
> covered in the
[jira] [Assigned] (BEAM-12879) Downloading GCS objects suddenly require storage.buckets.get permission
[
https://issues.apache.org/jira/browse/BEAM-12879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmet Altay reassigned BEAM-12879:
--
Assignee: Valentyn Tymofieiev
> Downloading GCS objects suddenly require storage.buckets.get permission
> ---
>
> Key: BEAM-12879
> URL: https://issues.apache.org/jira/browse/BEAM-12879
> Project: Beam
> Issue Type: Bug
> Components: io-py-gcp
>Affects Versions: 2.32.0
>Reporter: Robert Jany
>Assignee: Valentyn Tymofieiev
>Priority: P2
>
> With PR [https://github.com/apache/beam/pull/14770] downloading GCS objects
> requires an additional IAM role `storage.objects.get` to get the
> project_number based on the bucket name.
> If the service account or user does not have said role the following error
> will show:
> {code:python}
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.7/site-packages/dataflow_worker/batchworker.py", line
> 651, in do_work
> work_executor.execute()
> File "/usr/local/lib/python3.7/site-packages/dataflow_worker/executor.py",
> line 179, in execute
> op.start()
> File "dataflow_worker/native_operations.py", line 38, in
> dataflow_worker.native_operations.NativeReadOperation.start
> File "dataflow_worker/native_operations.py", line 39, in
> dataflow_worker.native_operations.NativeReadOperation.start
> File "dataflow_worker/native_operations.py", line 44, in
> dataflow_worker.native_operations.NativeReadOperation.start
> File "dataflow_worker/native_operations.py", line 54, in
> dataflow_worker.native_operations.NativeReadOperation.start
> File "apache_beam/runners/worker/operations.py", line 353, in
> apache_beam.runners.worker.operations.Operation.output
> File "apache_beam/runners/worker/operations.py", line 215, in
> apache_beam.runners.worker.operations.SingletonConsumerSet.receive
> File "apache_beam/runners/worker/operations.py", line 712, in
> apache_beam.runners.worker.operations.DoOperation.process
> File "apache_beam/runners/worker/operations.py", line 713, in
> apache_beam.runners.worker.operations.DoOperation.process
> File "apache_beam/runners/common.py", line 1234, in
> apache_beam.runners.common.DoFnRunner.process
> File "apache_beam/runners/common.py", line 1315, in
> apache_beam.runners.common.DoFnRunner._reraise_augmented
> File "apache_beam/runners/common.py", line 1232, in
> apache_beam.runners.common.DoFnRunner.process
> File "apache_beam/runners/common.py", line 571, in
> apache_beam.runners.common.SimpleInvoker.invoke_process
> File "apache_beam/runners/common.py", line 1368, in
> apache_beam.runners.common._OutputProcessor.process_outputs
> File "/usr/local/lib/python3.7/site-packages/xyz/package/file.py", line
> 112, in process
> with FileSystems.open(element["gcs_uri"]) as file:
> File
> "/usr/local/lib/python3.7/site-packages/apache_beam/io/filesystems.py", line
> 244, in open
> return filesystem.open(path, mime_type, compression_type)
> File
> "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/gcsfilesystem.py",
> line 177, in open
> return self._path_open(path, 'rb', mime_type, compression_type)
> File
> "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/gcsfilesystem.py",
> line 138, in _path_open
> raw_file = gcsio.GcsIO().open(path, mode, mime_type=mime_type)
> File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/gcsio.py",
> line 227, in open
> get_project_number=self.get_project_number)
> File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/gcsio.py",
> line 585, in __init__
> project_number = self._get_project_number(self._bucket)
> File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/gcsio.py",
> line 166, in get_project_number
> self.bucket_to_project_number[bucket] = bucket_metadata.projectNumber
> AttributeError: 'NoneType' object has no attribute 'projectNumber' [while
> running 'read from GCS']
> {code}
>
> The error message does not hint what goes wrong exactly but after some
> digging my assumption is that when trying to get the `bucket_metadata ` in
> [get_project_number|https://github.com/roger-mike/beam/blob/f0d0dd561a0955afb73cf595a3015a7ca839d5b7/sdks/python/apache_beam/io/gcp/gcsio.py#L161]
> we get a a HTTP Error and thus a None (since when catching this error a None
> is returned) due to the lack of permissions leading to `bucket_metadata`
> being None.
> The problem is, that the required permission (`storage.buckets.get`) is only
> covered in the predefined role `Storage Admin (roles/storage.admin)` which I
> believe shouldn't be necessary in order to access objects from GCS.
> Not sure what the solution would look like: We want the metadata incl. the
> project number but on the ot
[jira] [Assigned] (BEAM-13524) ParDo comment is grammatically incorrect and ambiguous.
[ https://issues.apache.org/jira/browse/BEAM-13524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay reassigned BEAM-13524: -- Assignee: David > ParDo comment is grammatically incorrect and ambiguous. > --- > > Key: BEAM-13524 > URL: https://issues.apache.org/jira/browse/BEAM-13524 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: David >Assignee: David >Priority: P3 > Original Estimate: 24h > Time Spent: 50m > Remaining Estimate: 23h 10m > > See: > [https://github.com/apache/beam/blob/af2f8ee6cf39a1d63818dbefef322740d5ad794c/sdks/go/pkg/beam/pardo.go#L106] > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (BEAM-13569) Python postcommit failing portableWordCountSparkRunnerBatch
[ https://issues.apache.org/jira/browse/BEAM-13569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay updated BEAM-13569: --- Parent: BEAM-13430 Issue Type: Sub-task (was: Bug) > Python postcommit failing portableWordCountSparkRunnerBatch > --- > > Key: BEAM-13569 > URL: https://issues.apache.org/jira/browse/BEAM-13569 > Project: Beam > Issue Type: Sub-task > Components: runner-spark, test-failures >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: P1 > > Guessing this was due to recent changes such as for the gradle upgrade > (BEAM-13430). > Execution failed for task > ':sdks:python:test-suites:portable:py37:portableWordCountSparkRunnerBatch'. > 'Exception in thread "grpc-default-executor-0" > java.lang.NoClassDefFoundError: > org/apache/spark/streaming/api/java/JavaStreamingListener ' > '\tat > org.apache.beam.runners.spark.SparkJobInvoker.createJobInvocation(SparkJobInvoker.java:101)' > '\tat > org.apache.beam.runners.spark.SparkJobInvoker.invokeWithExecutor(SparkJobInvoker.java:82)' > '\tat > org.apache.beam.runners.jobsubmission.JobInvoker.invoke(JobInvoker.java:48)' > '\tat > org.apache.beam.runners.jobsubmission.InMemoryJobService.run(InMemoryJobService.java:246)' > '\tat > org.apache.beam.model.jobmanagement.v1.JobServiceGrpc$MethodHandlers.invoke(JobServiceGrpc.java:948)' > '\tat > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCall > s.java:182)' > '\tat > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.ja > va:35)' > '\tat > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)' > '\tat > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(Forwarding > ServerCallListener.java:40)' > '\tat > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.Contexts$ContextualizedServerCallListener.onHalfClose(Contexts.java:86)' > '\tat > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331)' > '\tat > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInConte > xt(ServerImpl.java:797)' > '\tat > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)' > '\tat > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)' > '\tat > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)' > '\tat > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)' > '\tat java.lang.Thread.run(Thread.java:748)' > 'Caused by: java.lang.ClassNotFoundException: > org.apache.spark.streaming.api.java.JavaStreamingListener' > '\tat java.net.URLClassLoader.findClass(URLClassLoader.java:382)' -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (BEAM-13540) beam_Release_NightlySnapshot build failing to find golang.org/dl/go1.16.12@latest
[ https://issues.apache.org/jira/browse/BEAM-13540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay updated BEAM-13540: --- Parent: BEAM-13430 Issue Type: Sub-task (was: Bug) > beam_Release_NightlySnapshot build failing to find > golang.org/dl/go1.16.12@latest > - > > Key: BEAM-13540 > URL: https://issues.apache.org/jira/browse/BEAM-13540 > Project: Beam > Issue Type: Sub-task > Components: test-failures >Reporter: Emily Ye >Assignee: Robert Burke >Priority: P1 > Labels: currently-failing > > > * Console: > [https://ci-|https://ci-/]beam.apache.org/job/beam_Release_NightlySnapshot/1400/console > Seems related to [https://github.com/apache/beam/pull/16330] or > [https://github.com/apache/beam/pull/16333,] maybe jenkins specific? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (BEAM-13569) Python postcommit failing portableWordCountSparkRunnerBatch
[ https://issues.apache.org/jira/browse/BEAM-13569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay reassigned BEAM-13569: -- Assignee: Kyle Weaver > Python postcommit failing portableWordCountSparkRunnerBatch > --- > > Key: BEAM-13569 > URL: https://issues.apache.org/jira/browse/BEAM-13569 > Project: Beam > Issue Type: Bug > Components: runner-spark, test-failures >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: P1 > > Guessing this was due to recent changes such as for the gradle upgrade > (BEAM-13430). > Execution failed for task > ':sdks:python:test-suites:portable:py37:portableWordCountSparkRunnerBatch'. > 'Exception in thread "grpc-default-executor-0" > java.lang.NoClassDefFoundError: > org/apache/spark/streaming/api/java/JavaStreamingListener ' > '\tat > org.apache.beam.runners.spark.SparkJobInvoker.createJobInvocation(SparkJobInvoker.java:101)' > '\tat > org.apache.beam.runners.spark.SparkJobInvoker.invokeWithExecutor(SparkJobInvoker.java:82)' > '\tat > org.apache.beam.runners.jobsubmission.JobInvoker.invoke(JobInvoker.java:48)' > '\tat > org.apache.beam.runners.jobsubmission.InMemoryJobService.run(InMemoryJobService.java:246)' > '\tat > org.apache.beam.model.jobmanagement.v1.JobServiceGrpc$MethodHandlers.invoke(JobServiceGrpc.java:948)' > '\tat > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCall > s.java:182)' > '\tat > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.ja > va:35)' > '\tat > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)' > '\tat > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(Forwarding > ServerCallListener.java:40)' > '\tat > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.Contexts$ContextualizedServerCallListener.onHalfClose(Contexts.java:86)' > '\tat > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331)' > '\tat > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInConte > xt(ServerImpl.java:797)' > '\tat > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)' > '\tat > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)' > '\tat > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)' > '\tat > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)' > '\tat java.lang.Thread.run(Thread.java:748)' > 'Caused by: java.lang.ClassNotFoundException: > org.apache.spark.streaming.api.java.JavaStreamingListener' > '\tat java.net.URLClassLoader.findClass(URLClassLoader.java:382)' -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (BEAM-13541) Use runtime information to improve CoGroupByKey caching
[ https://issues.apache.org/jira/browse/BEAM-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17466169#comment-17466169 ] Ahmet Altay commented on BEAM-13541: In addition to this change, would it make sense to have a flag to configure DEFAULT_IN_MEMORY_ELEMENT_COUNT (https://github.com/apache/beam/blob/release-2.34.0/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/join/CoGbkResult.java#L60) similar to the setWorkerCacheMb flag? > Use runtime information to improve CoGroupByKey caching > --- > > Key: BEAM-13541 > URL: https://issues.apache.org/jira/browse/BEAM-13541 > Project: Beam > Issue Type: Improvement > Components: sdk-ideas >Reporter: Sunil Pedapudi >Assignee: Robert Bradshaw >Priority: P2 > Time Spent: 40m > Remaining Estimate: 0h > > Currently, CoGroupByKey creates UnionTables that are Flattened. The Flattened > output is processed by a GroupByKey to produce a CoGbkResult (via > ConstructCoGbkResultFn). > > Given the performance of CoGBK is greatly impacted based on the which > elements are cached in the (finitely sized) in-memory results, it would be > useful if CoGbkResult can use runtime information to prioritize which > elements are stored in-memory. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (BEAM-13508) Java PostCommits (Examples, PerformanceTests, XVR] Failing with ClassNotFound
[ https://issues.apache.org/jira/browse/BEAM-13508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay updated BEAM-13508: --- Parent: BEAM-13430 Issue Type: Sub-task (was: Bug) > Java PostCommits (Examples, PerformanceTests, XVR] Failing with ClassNotFound > - > > Key: BEAM-13508 > URL: https://issues.apache.org/jira/browse/BEAM-13508 > Project: Beam > Issue Type: Sub-task > Components: test-failures >Reporter: Emily Ye >Assignee: Brian Hulette >Priority: P1 > Labels: currently-failing > > _Jenkin Jobs:_ > * [https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/4832/] > * [https://ci-beam.apache.org/job/beam_PostCommit_Java_DataflowV1/1053/] > * > [https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/1589] > Several tests are failing with ClassNotFound exceptions, falling into two > categories: > * java.lang.NoClassDefFoundError in org.apache.beam.examples.* > * java.lang.IllegalArgumentException: Unknown 'runner' specified > 'PortableRunner', supported pipeline runners [RegisteredTestRunner] (caused > by Caused by: java.lang.ClassNotFoundException: (also seen > DataflowRunner for > [https://ci-beam.apache.org/job/beam_PerformanceTests_AvroIOIT/5727/console)] > > My guess is that these are due to changes for BEAM-13430, lining up with one > of: > * [https://github.com/apache/beam/pull/16307] > * [https://github.com/apache/beam/pull/16292] > Assigning to Daniel, who might have more insight into these changes, but will > continue to investigate > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (BEAM-13507) Python PostCommit Failures -
[ https://issues.apache.org/jira/browse/BEAM-13507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay updated BEAM-13507: --- Parent: BEAM-13430 Issue Type: Sub-task (was: Bug) > Python PostCommit Failures - > - > > Key: BEAM-13507 > URL: https://issues.apache.org/jira/browse/BEAM-13507 > Project: Beam > Issue Type: Sub-task > Components: test-failures >Reporter: Emily Ye >Assignee: Daniel Collins >Priority: P1 > Labels: currently-failing, test-failures > > Links: > * [Jenkins Job > (Python38)|https://ci-beam.apache.org/job/beam_PostCommit_Python38/2074/] > * [Gradle Build Scan|https://gradle.com/s/vjcpd66o2k6ka] > Initial investigation: > Seeing error: > FAILURE: Build failed with an exception. > What went wrong: Could not determine the dependencies of task > ':runners:google-cloud-dataflow-java:worker:shadowJar'. > > Cannot change dependencies of dependency configuration > > ':runners:spark:2:compileOnly' after it has been included in dependency > > resolution. > which seems related to [https://github.com/apache/beam/pull/16308] > Python postcommits were failing before with actual test failure (related to > [https://github.com/apache/beam/pull/16315)] before this, as a note. This is > the first run from this issue. > (Add any investigation notes so far) > > _After you've filled out the above details, please [assign the issue to an > individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist]. > Assignee should [treat test failures as > high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test], > helping to fix the issue or find a more appropriate owner. See [Apache Beam > Post-Commit > Policies|https://beam.apache.org/contribute/postcommits-policies]._ -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (BEAM-13519) Java precommit flaky (timing out)
[ https://issues.apache.org/jira/browse/BEAM-13519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay reassigned BEAM-13519: -- Assignee: Kiley Sok > Java precommit flaky (timing out) > - > > Key: BEAM-13519 > URL: https://issues.apache.org/jira/browse/BEAM-13519 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Kyle Weaver >Assignee: Kiley Sok >Priority: P1 > Labels: flake > > Java precommits are sometimes timing out with no clear cause. Gradle will log > a bunch of routine build tasks, and then Jenkins will abort the job much > later. There are no logs to indicate what happened. It is not even clear > which task or tasks, if any, was the culprit, since many tasks are run in > parallel. > 01:53:28 > Task :sdks:java:testing:nexmark:build > 01:53:28 > Task :sdks:java:testing:nexmark:buildDependents > 01:53:28 > Task :sdks:java:extensions:sql:zetasql:buildDependents > 01:53:28 > Task :sdks:java:io:google-cloud-platform:buildDependents > 01:53:28 > Task :sdks:java:extensions:sql:buildDependents > 01:53:28 > Task :sdks:java:io:kafka:buildDependents > 01:53:28 > Task :sdks:java:extensions:join-library:buildDependents > 01:53:28 > Task :sdks:java:io:synthetic:buildDependents > 01:53:28 > Task :sdks:java:io:mongodb:buildDependents > 01:53:28 > Task :sdks:java:io:thrift:buildDependents > 01:53:28 > Task :sdks:java:testing:test-utils:buildDependents > 01:53:28 > Task :sdks:java:expansion-service:buildDependents > 01:53:28 > Task :sdks:java:extensions:arrow:buildDependents > 01:53:28 > Task :sdks:java:extensions:protobuf:buildDependents > 01:53:28 > Task :sdks:java:io:common:buildDependents > 01:53:28 > Task :runners:direct-java:buildDependents > 01:53:28 > Task :runners:local-java:buildDependents > 01:53:28 Build timed out (after 120 minutes). Marking the build as aborted. > https://ci-beam.apache.org/job/beam_PreCommit_Java_cron/4874/ -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (BEAM-12898) Flink Load Tests failure- UncheckedExecutionException - leaking vms
[ https://issues.apache.org/jira/browse/BEAM-12898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463465#comment-17463465 ] Ahmet Altay commented on BEAM-12898: A changed to use a supported dataproc clusters (https://github.com/apache/beam/pull/16310) makes this problem worse and now the load tests fail to create flink clusters. We went with the change since it was vulnerability related, and tests were disabled anyway. Work to re-enable the tests, first need to start with fixing the creation of flink clusters with the update dataproc image version (1.5 instead of 1.2) > Flink Load Tests failure- UncheckedExecutionException - leaking vms > --- > > Key: BEAM-12898 > URL: https://issues.apache.org/jira/browse/BEAM-12898 > Project: Beam > Issue Type: Test > Components: test-failures >Reporter: Alex Amato >Assignee: Kyle Weaver >Priority: P2 > Attachments: 6L8weM2p7mDLMJV.png, BmJoKx8T8pZT2Ls.png > > Time Spent: 6h > Remaining Estimate: 0h > > Same failure from different tests: > [https://ci-beam.apache.org/job/beam_LoadTests_Go_CoGBK_Flink_Batch/277/console] > [https://ci-beam.apache.org/job/beam_LoadTests_Go_Combine_Flink_Batch/289/console] > [https://ci-beam.apache.org/job/beam_LoadTests_Go_GBK_Flink_Batch/290/console] > [https://ci-beam.apache.org/job/beam_LoadTests_Go_ParDo_Flink_Batch/295/console] > I think that this test may also be responsible for leaking some gce vms on > apache-beam-testing. As this morning we discovered several vms that were not > torn down. I suspect this is the cause of the leaked vms. > The vms have names like this: > vm names: > beam-loadtests-python*flink* > beam-loadtests-go*flink* > i.e. > beam-loadtests-go-cogbk-flink-batch-277-m > beam-loadtests-go-gbk-flink-batch-290-w-2 > beam-loadtests-go-pardo-flink-batch-295-m > beam-loadtests-go-sideinput-flink-batch-269-w-2 > beam-loadtests-python-combine-flink-batch-766-m > beam-loadtests-python-combine-flink-streaming-368-w-0 > beam-loadtests-python-pardo-flink-batch-716-m > > It seems like this tests are spinning up a dataproc cluster. The gce metadata > on the vms refers to a lot of dataproc stuff (attached). Likely the tests are > crashing and not running their code to clean up/shutdown the dataproc cluster. > Logs > > [https://ci-beam.apache.org/job/beam_LoadTests_Go_Combine_Flink_Batch/lastBuild/console] > 01:43:59 2021/09/14 08:43:59 (): > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.IllegalArgumentException: Encountered unsupported logical type > URN: int01:43:59 2021/09/14 08:43:59 (): > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.IllegalArgumentException: Encountered unsupported logical type > URN: int01:43:59 at > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050)01:43:59 > at > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.get(LocalCache.java:3952)01:43:59 > at > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)01:43:59 > at > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)01:43:59 > at > org.apache.beam.runners.core.construction.RehydratedComponents.getCoder(RehydratedComponents.java:168)01:43:59 > at > org.apache.beam.runners.fnexecution.wire.WireCoders.instantiateRunnerWireCoder(WireCoders.java:94)01:43:59 > at > org.apache.beam.runners.fnexecution.wire.WireCoders.instantiateRunnerWireCoder(WireCoders.java:75)01:43:59 > at > org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator.translateExecutableStage(FlinkBatchPortablePipelineTranslator.java:311)01:43:59 > at > org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator.translate(FlinkBatchPortablePipelineTranslator.java:272)01:43:59 > at > org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator.translate(FlinkBatchPortablePipelineTranslator
[jira] [Updated] (BEAM-13504) Remove provided/compileOnly deps not intended for external use
[ https://issues.apache.org/jira/browse/BEAM-13504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay updated BEAM-13504: --- Parent: BEAM-13430 Issue Type: Sub-task (was: Task) > Remove provided/compileOnly deps not intended for external use > -- > > Key: BEAM-13504 > URL: https://issues.apache.org/jira/browse/BEAM-13504 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Daniel Collins >Priority: P2 > > There are a lot of compileOnly deps in place to make certain libraries > lighter. In many cases, these don't actually reduce the size of the resulting > binary, and are a result of an overriding strategy where the types of the > implementation are needed in the loading library. > In the short term, these can all be replaced by `implementation`, with the > downside that jars will have extra explicit dependencies, and the upside that > users will not experience runtime errors due to missing dependencies. In the > longer term, we should restructure these libraries (sql being the worst > offender) to use serviceLoaders or other strategies that do not require > naming the types of the overriders within the same package that the interface > is declared. > The one exception to this is libraries that we intend to use whatever version > the user puts on the classpath at runtime without expressing a version > preference, such as hadoop or kafka. We should still consider if this has > value though, as users are still free to explicitly override the dependency > version we choose, even if we declare an `implementation` dependency on it. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (BEAM-13430) Upgrade Gradle version to 7.3
[ https://issues.apache.org/jira/browse/BEAM-13430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17462982#comment-17462982 ] Ahmet Altay commented on BEAM-13430: /cc [~dpcollins-google] [~lostluck] [~lukec...@gmail.com] - FYI for using this as the jira for PRs where it makes sense. > Upgrade Gradle version to 7.3 > - > > Key: BEAM-13430 > URL: https://issues.apache.org/jira/browse/BEAM-13430 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Kiley Sok >Assignee: Kiley Sok >Priority: P1 > Time Spent: 1h 50m > Remaining Estimate: 0h > > This is the first version to support Java 17 > > https://docs.gradle.org/current/userguide/compatibility.html -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (BEAM-13430) Upgrade Gradle version to 7.3
[ https://issues.apache.org/jira/browse/BEAM-13430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay updated BEAM-13430: --- Priority: P1 (was: P2) > Upgrade Gradle version to 7.3 > - > > Key: BEAM-13430 > URL: https://issues.apache.org/jira/browse/BEAM-13430 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Kiley Sok >Assignee: Kiley Sok >Priority: P1 > Time Spent: 1h 50m > Remaining Estimate: 0h > > This is the first version to support Java 17 > > https://docs.gradle.org/current/userguide/compatibility.html -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (BEAM-13433) beam_PostCommit_Python37 failing, potentially due to apache_beam.ml.gcp.cloud_dlp_it_test.CloudDLPIT
[
https://issues.apache.org/jira/browse/BEAM-13433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17460343#comment-17460343
]
Ahmet Altay commented on BEAM-13433:
Is this a duplicate of https://issues.apache.org/jira/browse/BEAM-13388
> beam_PostCommit_Python37 failing, potentially due to
> apache_beam.ml.gcp.cloud_dlp_it_test.CloudDLPIT
>
>
> Key: BEAM-13433
> URL: https://issues.apache.org/jira/browse/BEAM-13433
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Affects Versions: 2.36.0
>Reporter: Daniel Oliveira
>Assignee: Pablo Estrada
>Priority: P2
>
> It's difficult for me to test for sure, because each run seems to show
> slightly different errors, and sometimes the errors don't even show at all.
> To track this down, you need to check the gradle build scan for the test,
> because the raw logs are too long to find the appropriate error.
> This is one that shows an error:
> [https://ci-beam.apache.org/job/beam_PostCommit_Python37/4617/]
> As far as I can tell, this is the error, apparently happening due to
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/ml/gcp/cloud_dlp_it_test.py]
> {noformat}
> Traceback (most recent call last):
> File "apache_beam/runners/common.py", line 1198, in
> apache_beam.runners.common.DoFnRunner.process
> File "apache_beam/runners/common.py", line 536, in
> apache_beam.runners.common.SimpleInvoker.invoke_process
> File "apache_beam/runners/common.py", line 1334, in
> apache_beam.runners.common._OutputProcessor.process_outputs
> File
> "/usr/local/lib/python3.7/site-packages/apache_beam/ml/gcp/cloud_dlp.py",
> line 199, in process
> item={"value": element}, **self.params)
> TypeError: deidentify_content() got an unexpected keyword argument 'item'
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "/usr/local/lib/python3.7/site-packages/dataflow_worker/batchworker.py",
> line 644, in do_work
> work_executor.execute()
> File "/usr/local/lib/python3.7/site-packages/dataflow_worker/executor.py",
> line 208, in execute
> op.start()
> File "dataflow_worker/native_operations.py", line 38, in
> dataflow_worker.native_operations.NativeReadOperation.start
> File "dataflow_worker/native_operations.py", line 39, in
> dataflow_worker.native_operations.NativeReadOperation.start
> File "dataflow_worker/native_operations.py", line 44, in
> dataflow_worker.native_operations.NativeReadOperation.start
> File "dataflow_worker/native_operations.py", line 54, in
> dataflow_worker.native_operations.NativeReadOperation.start
> File "apache_beam/runners/worker/operations.py", line 348, in
> apache_beam.runners.worker.operations.Operation.output
> File "apache_beam/runners/worker/operations.py", line 215, in
> apache_beam.runners.worker.operations.SingletonConsumerSet.receive
> File "apache_beam/runners/worker/operations.py", line 707, in
> apache_beam.runners.worker.operations.DoOperation.process
> File "apache_beam/runners/worker/operations.py", line 708, in
> apache_beam.runners.worker.operations.DoOperation.process
> File "apache_beam/runners/common.py", line 1200, in
> apache_beam.runners.common.DoFnRunner.process
> File "apache_beam/runners/common.py", line 1281, in
> apache_beam.runners.common.DoFnRunner._reraise_augmented
> File "apache_beam/runners/common.py", line 1198, in
> apache_beam.runners.common.DoFnRunner.process
> File "apache_beam/runners/common.py", line 536, in
> apache_beam.runners.common.SimpleInvoker.invoke_process
> File "apache_beam/runners/common.py", line 1334, in
> apache_beam.runners.common._OutputProcessor.process_outputs
> File
> "/usr/local/lib/python3.7/site-packages/apache_beam/ml/gcp/cloud_dlp.py",
> line 199, in process
> item={"value": element}, **self.params)
> TypeError: deidentify_content() got an unexpected keyword argument 'item'
> [while running 'MaskDetectedDetails/ParDo(_DeidentifyFn)']{noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (BEAM-13314) Revise recommendations to manage Python pipeline dependencies.
[ https://issues.apache.org/jira/browse/BEAM-13314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay reassigned BEAM-13314: -- Assignee: David Huntsperger > Revise recommendations to manage Python pipeline dependencies. > --- > > Key: BEAM-13314 > URL: https://issues.apache.org/jira/browse/BEAM-13314 > Project: Beam > Issue Type: Bug > Components: sdk-py-core, website >Reporter: Valentyn Tymofieiev >Assignee: David Huntsperger >Priority: P2 > Labels: usability > > The page > https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/ > recommends managing Python dependencies via requirements files. > This approach is currently inefficient in light of introduction and adoption > of PEP-517 by some packages, see: > https://lists.apache.org/thread/trljnxo39c0cmff790yff3h8n5okqt3q and the > rest of the thread, and does not mention Custom Containers or SDK prebuilding > workflows. > > We should revise it and document best practices. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (BEAM-13389) Enable JdbcIO expansion service to more easily receive JDBC Driver as a dependency
[ https://issues.apache.org/jira/browse/BEAM-13389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay updated BEAM-13389: --- Labels: starter (was: ) > Enable JdbcIO expansion service to more easily receive JDBC Driver as a > dependency > -- > > Key: BEAM-13389 > URL: https://issues.apache.org/jira/browse/BEAM-13389 > Project: Beam > Issue Type: New Feature > Components: cross-language >Reporter: Pablo Estrada >Priority: P2 > Labels: starter > > Currently, to use the JDBCIO transform in Python, a user needs to clone the > Beam repo, and rebuild the expansion service with their specific dependency. > A nice feature would be to add a pipeline option to specify extra JDBC > drivers for the Expansion Service (these would be needed at construction > time, so they would need to be added to the expansion service's CLASSPATH at > runtime). -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (BEAM-13188) Java precommit timing out
[ https://issues.apache.org/jira/browse/BEAM-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17452667#comment-17452667 ] Ahmet Altay commented on BEAM-13188: Is this still an issue? Do we know the root cause? /cc [~kenn] > Java precommit timing out > - > > Key: BEAM-13188 > URL: https://issues.apache.org/jira/browse/BEAM-13188 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Kyle Weaver >Priority: P1 > Labels: flake > > Java precommit is timing out frequently. Looks like it's just idling for an > hour or more. Often the last thing that shows up in the logs is > buildDependents tasks, but that could just be a coincidence. > https://ci-beam.apache.org/job/beam_PreCommit_Java_cron -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (BEAM-13318) Python version of schema docs missing examples
Ahmet Altay created BEAM-13318: -- Summary: Python version of schema docs missing examples Key: BEAM-13318 URL: https://issues.apache.org/jira/browse/BEAM-13318 Project: Beam Issue Type: Improvement Components: website Reporter: Ahmet Altay User reported issue: "The python version of https://beam.apache.org/documentation/programming-guide/#using-schemas has a bunch of missing examples..." https://lists.apache.org/thread/2dpf31r4t0c56ln03ym48dmx04h14yk9 /cc [~bhulette] [~pcoet] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (BEAM-13204) Missing code snippets in programming guide
[ https://issues.apache.org/jira/browse/BEAM-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447773#comment-17447773 ] Ahmet Altay commented on BEAM-13204: Comment from rebo on the email thread: """ Specifically, if the reader has selected an SDK that doesn't have a populate code highlight block, that block doesn't appear at all, even if the other languages have them populated. I filled all appropriate blocks (with jira links for missing features) in for the Go SDK a few months ago, so it's likely Python examples that are missing. """ > Missing code snippets in programming guide > -- > > Key: BEAM-13204 > URL: https://issues.apache.org/jira/browse/BEAM-13204 > Project: Beam > Issue Type: Bug > Components: website >Reporter: Kyle Weaver >Assignee: David Huntsperger >Priority: P2 > > There are a number of code snippets missing in the programming guide, e.g. > "10.4 counter metric" -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (BEAM-13204) Missing code snippets in programming guide
[ https://issues.apache.org/jira/browse/BEAM-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay reassigned BEAM-13204: -- Assignee: David Huntsperger > Missing code snippets in programming guide > -- > > Key: BEAM-13204 > URL: https://issues.apache.org/jira/browse/BEAM-13204 > Project: Beam > Issue Type: Bug > Components: website >Reporter: Kyle Weaver >Assignee: David Huntsperger >Priority: P2 > > There are a number of code snippets missing in the programming guide, e.g. > "10.4 counter metric" -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (BEAM-12928) beam_PostCommit_Python36 - CrossLanguageSpannerIOTest - flakey failing
[
https://issues.apache.org/jira/browse/BEAM-12928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432578#comment-17432578
]
Ahmet Altay commented on BEAM-12928:
How flakey is this test? Would it be worth sikcbaying it?
> beam_PostCommit_Python36 - CrossLanguageSpannerIOTest - flakey failing
> --
>
> Key: BEAM-12928
> URL: https://issues.apache.org/jira/browse/BEAM-12928
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Alex Amato
>Assignee: Piotr Szuberski
>Priority: P1
> Labels: currently-failing, flake
>
> _Use this form to file an issue for test failure_
> _https://ci-beam.apache.org/job/beam_PostCommit_Python36/4385/_
>
> *
> [apache_beam.io.gcp.tests.xlang_spannerio_it_test.CrossLanguageSpannerIOTest.test_spanner_read_query|https://ci-beam.apache.org/job/beam_PostCommit_Python36/4385/testReport/junit/apache_beam.io.gcp.tests.xlang_spannerio_it_test/CrossLanguageSpannerIOTest/test_spanner_read_query/]
> *
> [apache_beam.io.gcp.tests.xlang_spannerio_it_test.CrossLanguageSpannerIOTest.test_spanner_read_table|https://ci-beam.apache.org/job/beam_PostCommit_Python36/4385/testReport/junit/apache_beam.io.gcp.tests.xlang_spannerio_it_test/CrossLanguageSpannerIOTest/test_spanner_read_table/]
>
> Initial investigation:
>
> Error Message
> RuntimeError: Pipeline
> BeamApp-jenkins-0921123257-8b921bc2_f20c9b92-9261-4f0b-9d6c-31732a12287b
> failed in state FAILED: akka.pattern.AskTimeoutException: Ask timed out on
> [Actor[akka://flink/user/rpc/taskmanager_0#31887928]] after [1 ms].
> Message of type [org.apache.flink.runtime.rpc.messages.LocalRpcInvocation]. A
> typical reason for `AskTimeoutException` is that the recipient actor didn't
> send a reply.
> Stacktrace
> self =
> testMethod=test_spanner_read_query>
> def test_spanner_read_query(self):
> self.insert_read_values('query_read')
> > self.run_read_pipeline('query_read', query=f'SELECT * FROM \{self.table}')
> apache_beam/io/gcp/tests/xlang_spannerio_it_test.py:194:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> apache_beam/io/gcp/tests/xlang_spannerio_it_test.py:220: in run_read_pipeline
> SpannerTestRow(f_int64=2, f_string=f'\{prefix}2', f_boolean=False),
> apache_beam/pipeline.py:590: in __exit__
> self.result = self.run()
> apache_beam/testing/test_pipeline.py:116: in run
> state = result.wait_until_finish()
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> self = at 0x7fa74c495860>
> duration = None
> def wait_until_finish(self, duration=None):
> """
> :param duration: The maximum time in milliseconds to wait for the result of
> the execution. If None or zero, will wait until the pipeline finishes.
> :return: The result of the pipeline, i.e. PipelineResult.
> """
> def read_messages():
> # type: () -> None
> previous_state = -1
> for message in self._message_stream:
> if message.HasField('message_response'):
> logging.log(
> MESSAGE_LOG_LEVELS[message.message_response.importance],
> "%s",
> message.message_response.message_text)
> else:
> current_state = message.state_response.state
> if current_state != previous_state:
> _LOGGER.info(
> "Job state changed to %s",
> self._runner_api_state_to_pipeline_state(current_state))
> previous_state = current_state
> self._messages.append(message)
>
> message_thread = threading.Thread(
> target=read_messages, name='wait_until_finish_read')
> message_thread.daemon = True
> message_thread.start()
>
> if duration:
> state_thread = threading.Thread(
> target=functools.partial(self._observe_state, message_thread),
> name='wait_until_finish_state_observer')
> state_thread.daemon = True
> state_thread.start()
> start_time = time.time()
> duration_secs = duration / 1000
> while (time.time() - start_time < duration_secs and
> state_thread.is_alive()):
> time.sleep(1)
> else:
> self._observe_state(message_thread)
>
> if self._runtime_exception:
> > raise self._runtime_exception
> E RuntimeError: Pipeline
> BeamApp-jenkins-0921123257-8b921bc2_f20c9b92-9261-4f0b-9d6c-31732a12287b
> failed in state FAILED: akka.pattern.AskTimeoutException: Ask timed out on
> [Actor[akka://flink/user/rpc/taskmanager_0#31887928]] after [1 ms].
> Message of type [org.apache.flink.runtime.rpc.messages.LocalRpcInvocation]. A
> typical reason for `AskTimeoutException` is that the recipient actor didn't
> send a reply.
> apache_beam/runners/portability/portable_runner.py:600: RuntimeError
>
>
> _After you've filled out the above details, please [assign the issue to an
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
> Assignee should [treat test fail
[jira] [Assigned] (BEAM-13086) Failing to prebuild python sdk container using cloud build
[ https://issues.apache.org/jira/browse/BEAM-13086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay reassigned BEAM-13086: -- Assignee: Yichi Zhang > Failing to prebuild python sdk container using cloud build > -- > > Key: BEAM-13086 > URL: https://issues.apache.org/jira/browse/BEAM-13086 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Ludovic Post >Assignee: Yichi Zhang >Priority: P2 > > Today, cloud build jobs started to fail with the following error: "error > checking push permissions". > I was able to fix the issue using the following workaround: Update the > hardcoded image url from "gcr.io/kaniko-project/executor:latest" to > "gcr.io/kaniko-project/executor:v1.6.0" in > https://github.com/apache/beam/blob/070ea22d507a2c26d838c872bb5790b8dad7f9b8/sdks/python/apache_beam/runners/portability/sdk_container_builder.py#L248 > Kanito executor related bug: [Error on pushing image to GCR · Issue #1791 · > GoogleContainerTools/kaniko · > GitHub|https://github.com/GoogleContainerTools/kaniko/issues/1791] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-13086) Failing to prebuild python sdk container using cloud build
[ https://issues.apache.org/jira/browse/BEAM-13086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay updated BEAM-13086: --- Priority: P1 (was: P2) > Failing to prebuild python sdk container using cloud build > -- > > Key: BEAM-13086 > URL: https://issues.apache.org/jira/browse/BEAM-13086 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Ludovic Post >Assignee: Yichi Zhang >Priority: P1 > > Today, cloud build jobs started to fail with the following error: "error > checking push permissions". > I was able to fix the issue using the following workaround: Update the > hardcoded image url from "gcr.io/kaniko-project/executor:latest" to > "gcr.io/kaniko-project/executor:v1.6.0" in > https://github.com/apache/beam/blob/070ea22d507a2c26d838c872bb5790b8dad7f9b8/sdks/python/apache_beam/runners/portability/sdk_container_builder.py#L248 > Kanito executor related bug: [Error on pushing image to GCR · Issue #1791 · > GoogleContainerTools/kaniko · > GitHub|https://github.com/GoogleContainerTools/kaniko/issues/1791] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-13067) GroupIntoBatches should be considered a key-preserving operation by Dataflow
[ https://issues.apache.org/jira/browse/BEAM-13067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17430134#comment-17430134 ] Ahmet Altay edited comment on BEAM-13067 at 10/18/21, 6:25 PM: --- Not a blocker, quoting Reuven from an offline conversation: "a nice to have. not a blocker this affects performance only" (Removed the fix version field.) was (Author: altay): Not a blocker, quoting Reuven from an offline conversation: "a nice to have. not a blocker this affects performance only" > GroupIntoBatches should be considered a key-preserving operation by Dataflow > > > Key: BEAM-13067 > URL: https://issues.apache.org/jira/browse/BEAM-13067 > Project: Beam > Issue Type: New Feature > Components: runner-dataflow >Affects Versions: 2.33.0 >Reporter: Reuven Lax >Priority: P2 > > Dataflow today only marks GroupIntoBatches as key preserving if auto sharding > is enabled. It should mark it as key preserving regardless. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-13067) GroupIntoBatches should be considered a key-preserving operation by Dataflow
[ https://issues.apache.org/jira/browse/BEAM-13067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay updated BEAM-13067: --- Fix Version/s: (was: 2.34.0) > GroupIntoBatches should be considered a key-preserving operation by Dataflow > > > Key: BEAM-13067 > URL: https://issues.apache.org/jira/browse/BEAM-13067 > Project: Beam > Issue Type: New Feature > Components: runner-dataflow >Affects Versions: 2.33.0 >Reporter: Reuven Lax >Priority: P2 > > Dataflow today only marks GroupIntoBatches as key preserving if auto sharding > is enabled. It should mark it as key preserving regardless. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-13067) GroupIntoBatches should be considered a key-preserving operation by Dataflow
[ https://issues.apache.org/jira/browse/BEAM-13067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17430134#comment-17430134 ] Ahmet Altay commented on BEAM-13067: Not a blocker, quoting Reuven from an offline conversation: "a nice to have. not a blocker this affects performance only" > GroupIntoBatches should be considered a key-preserving operation by Dataflow > > > Key: BEAM-13067 > URL: https://issues.apache.org/jira/browse/BEAM-13067 > Project: Beam > Issue Type: New Feature > Components: runner-dataflow >Affects Versions: 2.33.0 >Reporter: Reuven Lax >Priority: P2 > Fix For: 2.34.0 > > > Dataflow today only marks GroupIntoBatches as key preserving if auto sharding > is enabled. It should mark it as key preserving regardless. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-13073) Unexpected GC when using Java 11
[ https://issues.apache.org/jira/browse/BEAM-13073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay reassigned BEAM-13073: -- Assignee: Kenneth Knowles > Unexpected GC when using Java 11 > > > Key: BEAM-13073 > URL: https://issues.apache.org/jira/browse/BEAM-13073 > Project: Beam > Issue Type: Bug > Components: sdk-java-harness >Reporter: Luis >Assignee: Kenneth Knowles >Priority: P2 > Labels: java11, java9, performance > Time Spent: 10m > Remaining Estimate: 0h > > Beam SDK has been supporting Java 11 for a while (I guess the support was > introduced here https://issues.apache.org/jira/browse/BEAM-2530). > Unfortunately, in Spotify we are still experiencing performance issues when > using Beam SDK 2.32, Google Dataflow and Java 11. > Thanks to [~emilyye] and [~iht], they confirmed JVM 11 is using SerialGC, > while Java 8 uses ParallelGC. It sound like SerialGC a good option for high > throughput / low latency jobs. For Java11 we'd expect to use G1GC or > ParallelGC. > This SO question [1] clarifies that JVM choses SerialGC when it treats the > machine as a "client". It looks like the Java SDK container could benefit > from using `-XX:+AlwaysActAsServerClassMachine`. Is that correct? > Let me know if the ticket needs further context or adjustment. (It is my > first time creating a ticket here). > [1] > https://stackoverflow.com/questions/52474162/why-is-serialgc-chosen-over-g1gc -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-12898) Flink Load Tests failure- UncheckedExecutionException - leaking vms
[ https://issues.apache.org/jira/browse/BEAM-12898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17429072#comment-17429072 ] Ahmet Altay commented on BEAM-12898: It looks like tests are running again. Thank you all. Closing this. > Flink Load Tests failure- UncheckedExecutionException - leaking vms > --- > > Key: BEAM-12898 > URL: https://issues.apache.org/jira/browse/BEAM-12898 > Project: Beam > Issue Type: Test > Components: test-failures >Reporter: Alex Amato >Assignee: Kyle Weaver >Priority: P2 > Attachments: 6L8weM2p7mDLMJV.png, BmJoKx8T8pZT2Ls.png > > Time Spent: 6h > Remaining Estimate: 0h > > Same failure from different tests: > [https://ci-beam.apache.org/job/beam_LoadTests_Go_CoGBK_Flink_Batch/277/console] > [https://ci-beam.apache.org/job/beam_LoadTests_Go_Combine_Flink_Batch/289/console] > [https://ci-beam.apache.org/job/beam_LoadTests_Go_GBK_Flink_Batch/290/console] > [https://ci-beam.apache.org/job/beam_LoadTests_Go_ParDo_Flink_Batch/295/console] > I think that this test may also be responsible for leaking some gce vms on > apache-beam-testing. As this morning we discovered several vms that were not > torn down. I suspect this is the cause of the leaked vms. > The vms have names like this: > vm names: > beam-loadtests-python*flink* > beam-loadtests-go*flink* > i.e. > beam-loadtests-go-cogbk-flink-batch-277-m > beam-loadtests-go-gbk-flink-batch-290-w-2 > beam-loadtests-go-pardo-flink-batch-295-m > beam-loadtests-go-sideinput-flink-batch-269-w-2 > beam-loadtests-python-combine-flink-batch-766-m > beam-loadtests-python-combine-flink-streaming-368-w-0 > beam-loadtests-python-pardo-flink-batch-716-m > > It seems like this tests are spinning up a dataproc cluster. The gce metadata > on the vms refers to a lot of dataproc stuff (attached). Likely the tests are > crashing and not running their code to clean up/shutdown the dataproc cluster. > Logs > > [https://ci-beam.apache.org/job/beam_LoadTests_Go_Combine_Flink_Batch/lastBuild/console] > 01:43:59 2021/09/14 08:43:59 (): > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.IllegalArgumentException: Encountered unsupported logical type > URN: int01:43:59 2021/09/14 08:43:59 (): > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.IllegalArgumentException: Encountered unsupported logical type > URN: int01:43:59 at > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050)01:43:59 > at > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.get(LocalCache.java:3952)01:43:59 > at > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)01:43:59 > at > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)01:43:59 > at > org.apache.beam.runners.core.construction.RehydratedComponents.getCoder(RehydratedComponents.java:168)01:43:59 > at > org.apache.beam.runners.fnexecution.wire.WireCoders.instantiateRunnerWireCoder(WireCoders.java:94)01:43:59 > at > org.apache.beam.runners.fnexecution.wire.WireCoders.instantiateRunnerWireCoder(WireCoders.java:75)01:43:59 > at > org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator.translateExecutableStage(FlinkBatchPortablePipelineTranslator.java:311)01:43:59 > at > org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator.translate(FlinkBatchPortablePipelineTranslator.java:272)01:43:59 > at > org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator.translate(FlinkBatchPortablePipelineTranslator.java:118)01:43:59 > at > org.apache.beam.runners.flink.FlinkPipelineRunner.runPipelineWithTranslator(FlinkPipelineRunner.java:115)01:43:59 > at > org.apache.beam.runners.flink.FlinkPipelineRunner.run(FlinkPipelineRunner.java:85)01:43:59 > at > org.apache.beam.runners.jobsubmission.JobInvocation.runPipeline(JobInvocation.java:86)01:43:59 > at
[jira] [Updated] (BEAM-12898) Flink Load Tests failure- UncheckedExecutionException - leaking vms
[ https://issues.apache.org/jira/browse/BEAM-12898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay updated BEAM-12898: --- Resolution: Fixed Status: Resolved (was: Open) > Flink Load Tests failure- UncheckedExecutionException - leaking vms > --- > > Key: BEAM-12898 > URL: https://issues.apache.org/jira/browse/BEAM-12898 > Project: Beam > Issue Type: Test > Components: test-failures >Reporter: Alex Amato >Assignee: Kyle Weaver >Priority: P2 > Attachments: 6L8weM2p7mDLMJV.png, BmJoKx8T8pZT2Ls.png > > Time Spent: 6h > Remaining Estimate: 0h > > Same failure from different tests: > [https://ci-beam.apache.org/job/beam_LoadTests_Go_CoGBK_Flink_Batch/277/console] > [https://ci-beam.apache.org/job/beam_LoadTests_Go_Combine_Flink_Batch/289/console] > [https://ci-beam.apache.org/job/beam_LoadTests_Go_GBK_Flink_Batch/290/console] > [https://ci-beam.apache.org/job/beam_LoadTests_Go_ParDo_Flink_Batch/295/console] > I think that this test may also be responsible for leaking some gce vms on > apache-beam-testing. As this morning we discovered several vms that were not > torn down. I suspect this is the cause of the leaked vms. > The vms have names like this: > vm names: > beam-loadtests-python*flink* > beam-loadtests-go*flink* > i.e. > beam-loadtests-go-cogbk-flink-batch-277-m > beam-loadtests-go-gbk-flink-batch-290-w-2 > beam-loadtests-go-pardo-flink-batch-295-m > beam-loadtests-go-sideinput-flink-batch-269-w-2 > beam-loadtests-python-combine-flink-batch-766-m > beam-loadtests-python-combine-flink-streaming-368-w-0 > beam-loadtests-python-pardo-flink-batch-716-m > > It seems like this tests are spinning up a dataproc cluster. The gce metadata > on the vms refers to a lot of dataproc stuff (attached). Likely the tests are > crashing and not running their code to clean up/shutdown the dataproc cluster. > Logs > > [https://ci-beam.apache.org/job/beam_LoadTests_Go_Combine_Flink_Batch/lastBuild/console] > 01:43:59 2021/09/14 08:43:59 (): > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.IllegalArgumentException: Encountered unsupported logical type > URN: int01:43:59 2021/09/14 08:43:59 (): > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.IllegalArgumentException: Encountered unsupported logical type > URN: int01:43:59 at > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050)01:43:59 > at > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.get(LocalCache.java:3952)01:43:59 > at > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)01:43:59 > at > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)01:43:59 > at > org.apache.beam.runners.core.construction.RehydratedComponents.getCoder(RehydratedComponents.java:168)01:43:59 > at > org.apache.beam.runners.fnexecution.wire.WireCoders.instantiateRunnerWireCoder(WireCoders.java:94)01:43:59 > at > org.apache.beam.runners.fnexecution.wire.WireCoders.instantiateRunnerWireCoder(WireCoders.java:75)01:43:59 > at > org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator.translateExecutableStage(FlinkBatchPortablePipelineTranslator.java:311)01:43:59 > at > org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator.translate(FlinkBatchPortablePipelineTranslator.java:272)01:43:59 > at > org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator.translate(FlinkBatchPortablePipelineTranslator.java:118)01:43:59 > at > org.apache.beam.runners.flink.FlinkPipelineRunner.runPipelineWithTranslator(FlinkPipelineRunner.java:115)01:43:59 > at > org.apache.beam.runners.flink.FlinkPipelineRunner.run(FlinkPipelineRunner.java:85)01:43:59 > at > org.apache.beam.runners.jobsubmission.JobInvocation.runPipeline(JobInvocation.java:86)01:43:59 > at > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.
[jira] [Commented] (BEAM-12859) org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingServiceTest.testMultipleClientsFailingIsHandledGracefullyByServer is flaky
[
https://issues.apache.org/jira/browse/BEAM-12859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17429036#comment-17429036
]
Ahmet Altay commented on BEAM-12859:
Is this still flaky? ( /cc [~kenn] )
> org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingServiceTest.testMultipleClientsFailingIsHandledGracefullyByServer
> is flaky
> --
>
> Key: BEAM-12859
> URL: https://issues.apache.org/jira/browse/BEAM-12859
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Valentyn Tymofieiev
>Priority: P2
> Labels: flake
>
> As per [prior discussion on
> dev|https://lists.apache.org/thread.html/ra75bec744b6184d6576244b6d5e6ac7ca321c74b7c6df4d4d7bdddf1%40%3Cdev.beam.apache.org%3E],
> this may have started after upgrading vendored gRPC from 1.26.0 to 1.36.0.
> Sample error from
> https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/4469/testReport/junit/org.apache.beam.runners.dataflow.worker.fn.logging/BeamFnLoggingServiceTest/testMultipleClientsFailingIsHandledGracefullyByServer/
> {noformat}
> Error Message
> org.junit.runners.model.TestTimedOutException: test timed out after 5000
> milliseconds
> Stacktrace
> org.junit.runners.model.TestTimedOutException: test timed out after 5000
> milliseconds
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at
> org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingServiceTest.testMultipleClientsFailingIsHandledGracefullyByServer(BeamFnLoggingServiceTest.java:156)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> Standard Error
> Sep 07, 2021 12:12:20 AM
> org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingService
> INFO: Launched Beam Fn Logging service url: "127.0.0.1:34883"
> Sep 07, 2021 12:12:20 AM
> org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingService
> logging
> INFO: Beam Fn Logging client connected for client
> Sep 07, 2021 12:12:20 AM
> org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingService
> logging
> INFO: Beam Fn Logging client connected for client
> Sep 07, 2021 12:12:20
> ...[truncated 5654 chars]...
> nnected for client
> Sep 07, 2021 12:12:20 AM
> org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingService
> logging
> INFO: Beam Fn Logging client connected for client
> Sep 07, 2021 12:12:20 AM
> org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingService
> logging
> INFO: Beam Fn Logging client connected for client
> Sep 07, 2021 12:12:20 AM
> org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingService close
> INFO: 3 Beam Fn Logging clients still connected during shutdown.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-13036) Failure in beam_Prober_CommunityMetrics
[
https://issues.apache.org/jira/browse/BEAM-13036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmet Altay updated BEAM-13036:
---
Resolution: Fixed
Status: Resolved (was: Triage Needed)
> Failure in beam_Prober_CommunityMetrics
> ---
>
> Key: BEAM-13036
> URL: https://issues.apache.org/jira/browse/BEAM-13036
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Heejong Lee
>Assignee: Fernando Morales
>Priority: P1
>
> Failure in beam_Prober_CommunityMetrics:
> {noformat}
> org.apache.beam.testinfra.metrics.ProberTests > CheckGrafanaStalenessAlerts
> FAILED
> 11:08:56 java.lang.AssertionError at ProberTests.groovy:47
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-13036) Failure in beam_Prober_CommunityMetrics
[
https://issues.apache.org/jira/browse/BEAM-13036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17428588#comment-17428588
]
Ahmet Altay commented on BEAM-13036:
Looks like it is fixed after https://github.com/apache/beam/pull/15705
Thank you [~ibzib]
> Failure in beam_Prober_CommunityMetrics
> ---
>
> Key: BEAM-13036
> URL: https://issues.apache.org/jira/browse/BEAM-13036
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Heejong Lee
>Assignee: Fernando Morales
>Priority: P1
>
> Failure in beam_Prober_CommunityMetrics:
> {noformat}
> org.apache.beam.testinfra.metrics.ProberTests > CheckGrafanaStalenessAlerts
> FAILED
> 11:08:56 java.lang.AssertionError at ProberTests.groovy:47
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-9482) beam_PerformanceTests_Kafka_IO failing due to " provided port is already allocated"
[ https://issues.apache.org/jira/browse/BEAM-9482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay reassigned BEAM-9482: - Assignee: (was: Ankur Goenka) > beam_PerformanceTests_Kafka_IO failing due to " provided port is already > allocated" > --- > > Key: BEAM-9482 > URL: https://issues.apache.org/jira/browse/BEAM-9482 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Chamikara Madhusanka Jayalath >Priority: P1 > Labels: sickbay, stale-assigned > Time Spent: 3.5h > Remaining Estimate: 0h > > For example, > [https://builds.apache.org/view/A-D/view/Beam/view/PerformanceTests/job/beam_PerformanceTests_Kafka_IO/514/console] > > 18:55:33 Error from server (Invalid): error when creating > "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Kafka_IO/src/.test-infra/kubernetes/kafka-cluster/04-outside-services/outside-0.yml": > Service "outside-0" is invalid: spec.ports[0].nodePort: Invalid value: > 32400: provided port is already allocated > 18:55:33 Error from server (Invalid): error when creating > "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Kafka_IO/src/.test-infra/kubernetes/kafka-cluster/04-outside-services/outside-1.yml": > Service "outside-1" is invalid: spec.ports[0].nodePort: Invalid value: > 32401: provided port is already allocated > 18:55:33 Error from server (Invalid): error when creating > "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Kafka_IO/src/.test-infra/kubernetes/kafka-cluster/04-outside-services/outside-2.yml": > Service "outside-2" is invalid: spec.ports[0].nodePort: Invalid value: > 32402: provided port is already allocated > 1 > > Seems like we tried three ports but they were being used. Probably we should > update code to find an unused port dynamically. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-8123) Support CloudPickle as pickler for Apache Beam.
[ https://issues.apache.org/jira/browse/BEAM-8123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay reassigned BEAM-8123: - Assignee: Ryan Thompson > Support CloudPickle as pickler for Apache Beam. > --- > > Key: BEAM-8123 > URL: https://issues.apache.org/jira/browse/BEAM-8123 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: Ryan Thompson >Priority: P3 > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-13036) Failure in beam_Prober_CommunityMetrics
[
https://issues.apache.org/jira/browse/BEAM-13036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427380#comment-17427380
]
Ahmet Altay commented on BEAM-13036:
/cc [~kileys]
> Failure in beam_Prober_CommunityMetrics
> ---
>
> Key: BEAM-13036
> URL: https://issues.apache.org/jira/browse/BEAM-13036
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Heejong Lee
>Assignee: Fernando Morales
>Priority: P2
>
> Failure in beam_Prober_CommunityMetrics:
> {noformat}
> org.apache.beam.testinfra.metrics.ProberTests > CheckGrafanaStalenessAlerts
> FAILED
> 11:08:56 java.lang.AssertionError at ProberTests.groovy:47
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
