[jira] [Work logged] (BEAM-3608) Vendor Guava
[
https://issues.apache.org/jira/browse/BEAM-3608?focusedWorklogId=184483=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184483
]
ASF GitHub Bot logged work on BEAM-3608:
Author: ASF GitHub Bot
Created on: 12/Jan/19 04:42
Start Date: 12/Jan/19 04:42
Worklog Time Spent: 10m
Work Description: kennknowles commented on issue #7494: [BEAM-3608] Port
shaded Guava to vendored Guava
URL: https://github.com/apache/beam/pull/7494#issuecomment-453719276
R: @iemejia
I didn't actually remove the shadow plugin to really speed up the build, but
I confirmed that there's no Guava in the direct runner jar.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 184483)
Time Spent: 7h 50m (was: 7h 40m)
> Vendor Guava
>
>
> Key: BEAM-3608
> URL: https://issues.apache.org/jira/browse/BEAM-3608
> Project: Beam
> Issue Type: Sub-task
> Components: runner-core, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 7h 50m
> Remaining Estimate: 0h
>
> Instead of shading as part of our build, we can shade before build so that it
> is apparent when reading code, and in IDEs, that a particular class resides
> in a hidden namespace.
> {{import com.google.common.reflect.TypeToken}}
> becomes something like
> {{import org.apache.beam.private.guava21.com.google.common.reflect.TypeToken}}
> So we can very trivially ban `org.apache.beam.private` from public APIs
> unless they are annotated {{@Internal}}, and it makes sharing between our own
> modules never get broken by shading again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3608) Vendor Guava
[
https://issues.apache.org/jira/browse/BEAM-3608?focusedWorklogId=184485=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184485
]
ASF GitHub Bot logged work on BEAM-3608:
Author: ASF GitHub Bot
Created on: 12/Jan/19 04:43
Start Date: 12/Jan/19 04:43
Worklog Time Spent: 10m
Work Description: kennknowles commented on issue #7494: [BEAM-3608] Port
shaded Guava to vendored Guava
URL: https://github.com/apache/beam/pull/7494#issuecomment-453719320
I stopped work after I saw most of the `needsRunnerTests` were on their way
to passing. There's going to be some more to do. I didn't compile or test
anything that wasn't part of that dependency chain.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 184485)
Time Spent: 8h (was: 7h 50m)
> Vendor Guava
>
>
> Key: BEAM-3608
> URL: https://issues.apache.org/jira/browse/BEAM-3608
> Project: Beam
> Issue Type: Sub-task
> Components: runner-core, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 8h
> Remaining Estimate: 0h
>
> Instead of shading as part of our build, we can shade before build so that it
> is apparent when reading code, and in IDEs, that a particular class resides
> in a hidden namespace.
> {{import com.google.common.reflect.TypeToken}}
> becomes something like
> {{import org.apache.beam.private.guava21.com.google.common.reflect.TypeToken}}
> So we can very trivially ban `org.apache.beam.private` from public APIs
> unless they are annotated {{@Internal}}, and it makes sharing between our own
> modules never get broken by shading again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-6420) Vendored Guava hits some errorprone issues
Kenneth Knowles created BEAM-6420:
-
Summary: Vendored Guava hits some errorprone issues
Key: BEAM-6420
URL: https://issues.apache.org/jira/browse/BEAM-6420
Project: Beam
Issue Type: Bug
Components: sdk-java-core
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles
{code}
sdks/java/core/src/main/java/org/apache/beam/sdk/io/Compression.java:145:
warning: [ImmutableEnumChecker] enums should be immutable: 'Compression' has
field 'detectedSuffixes' of type
'org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList',
the declaration of type
'org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList'
is not annotated with @com.google.errorprone.annotations.Immutable or
@javax.annotation.concurrent.Immutable
private final ImmutableList detectedSuffixes;
^
(see http://errorprone.info/bugpattern/ImmutableEnumChecker)
error: warnings found and -Werror specified
{code}
The annotations should be preserved, so it may be that errorprone has a
whitelist of classes and the relocation fools it. Probably should be fixed in
errorprone if that is the case. If there is something we need to preserve about
the rewritten class files, we should do that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3608) Vendor Guava
[ https://issues.apache.org/jira/browse/BEAM-3608?focusedWorklogId=184482=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184482 ] ASF GitHub Bot logged work on BEAM-3608: Author: ASF GitHub Bot Created on: 12/Jan/19 04:23 Start Date: 12/Jan/19 04:23 Worklog Time Spent: 10m Work Description: kennknowles commented on pull request #7494: [BEAM-3608] Port shaded Guava to vendored Guava URL: https://github.com/apache/beam/pull/7494 This is a global search and replace of `compile library.java.guava` for `shadow library.java.vendored_guava_20_0` followed by replacing the imports throughout. It is expect that a couple of places where modules intend to use public Guava (notably CassandraIO) will need to be patched up. Opening this to get async feedback from Jenkins. There is something strange about our classpaths or the Gradle cache that compile errors were not forthcoming when I thought they should be. Follow this checklist to help us incorporate your contribution quickly and easily: - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an
[jira] [Assigned] (BEAM-3608) Vendor Guava
[
https://issues.apache.org/jira/browse/BEAM-3608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles reassigned BEAM-3608:
-
Assignee: Kenneth Knowles
> Vendor Guava
>
>
> Key: BEAM-3608
> URL: https://issues.apache.org/jira/browse/BEAM-3608
> Project: Beam
> Issue Type: Sub-task
> Components: runner-core, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> Instead of shading as part of our build, we can shade before build so that it
> is apparent when reading code, and in IDEs, that a particular class resides
> in a hidden namespace.
> {{import com.google.common.reflect.TypeToken}}
> becomes something like
> {{import org.apache.beam.private.guava21.com.google.common.reflect.TypeToken}}
> So we can very trivially ban `org.apache.beam.private` from public APIs
> unless they are annotated {{@Internal}}, and it makes sharing between our own
> modules never get broken by shading again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-6419) [SQL] Jacoco error: Classes in bundle 'beam-sdks-java-extensions-sql' do no match with execution data.
[
https://issues.apache.org/jira/browse/BEAM-6419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740985#comment-16740985
]
Kenneth Knowles commented on BEAM-6419:
---
Tried it on sdks/java/testing/nexmark and got actual numbers, so it isn't just
shading.
> [SQL] Jacoco error: Classes in bundle 'beam-sdks-java-extensions-sql' do no
> match with execution data.
> --
>
> Key: BEAM-6419
> URL: https://issues.apache.org/jira/browse/BEAM-6419
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>
> {code}
> [ant:jacocoReport] Classes in bundle 'beam-sdks-java-extensions-sql' do no
> match with execution data. For report generation the same class files must be
> used as at runtime.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTable does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/udf/IsNan does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnnestRel$Transform does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamSetOperatorRelBase$OpType
> does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProvider$RowToCsv
> does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamSortRel does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/transform/agg/CovarianceFn does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/parser/impl/ParseException does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rule/BeamMinusRule does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations$IntegerAvg
> does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProvider
> does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rule/BeamUncollectRule does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel$1 does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel$CalcFn does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/ParseException does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTable does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/UdfUdafProvider does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaCSVTable$CsvRecorderDecoder
> does not match.
> {code}
> ... and so on.
> There's some discussion of similar-sounding issues at
> https://stackoverflow.com/questions/31720139/jacoco-code-coverage-report-generator-showing-error-classes-in-bundle-code-c
> If JaCoCo is looking at the class files, but tests run against the shaded
> jar, this would be expected because only byte-for-byte identical class files
> will match.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (BEAM-6403) Improve checkstyle rules on javadoc comments
[ https://issues.apache.org/jira/browse/BEAM-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-6403 started by Ruoyun Huang. -- > Improve checkstyle rules on javadoc comments > > > Key: BEAM-6403 > URL: https://issues.apache.org/jira/browse/BEAM-6403 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Ruoyun Huang >Assignee: Ruoyun Huang >Priority: Minor > > Make checkstyle check comments on non-trivial public methods. > > discussions: > https://lists.apache.org/thread.html/819a68f69940e60cb820370df90ce15cecd289493b28149e1df1719e@%3Cdev.beam.apache.org%3E > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6184) PortableRunner dependency missed in wordcount example maven artifact
[ https://issues.apache.org/jira/browse/BEAM-6184?focusedWorklogId=184471=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184471 ] ASF GitHub Bot logged work on BEAM-6184: Author: ASF GitHub Bot Created on: 12/Jan/19 03:06 Start Date: 12/Jan/19 03:06 Worklog Time Spent: 10m Work Description: HuangLED commented on issue #7454: [BEAM-6184] Enforce javadoc check on non-trivial public methods URL: https://github.com/apache/beam/pull/7454#issuecomment-453714516 Was trying to include a few examples on how to define a suppressio. Now the example is moved into xml file as comment. PR Updated. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184471) Time Spent: 3.5h (was: 3h 20m) > PortableRunner dependency missed in wordcount example maven artifact > > > Key: BEAM-6184 > URL: https://issues.apache.org/jira/browse/BEAM-6184 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Ruoyun Huang >Assignee: Ruoyun Huang >Priority: Minor > Fix For: Not applicable > > Time Spent: 3.5h > Remaining Estimate: 0h > > > > more context: > https://lists.apache.org/thread.html/8dd60395424425f7502d62888c49014430d1d3b06c026606f3db28ab@%3Cuser.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6231) Triage test failures introduced by use_executable_stage_bundle_execution
[ https://issues.apache.org/jira/browse/BEAM-6231?focusedWorklogId=184468=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184468 ] ASF GitHub Bot logged work on BEAM-6231: Author: ASF GitHub Bot Created on: 12/Jan/19 02:45 Start Date: 12/Jan/19 02:45 Worklog Time Spent: 10m Work Description: boyuanzz commented on issue #7356: [BEAM-6231] Make Dataflow runner harness work with FixedWindow URL: https://github.com/apache/beam/pull/7356#issuecomment-453713302 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184468) Time Spent: 6h 20m (was: 6h 10m) > Triage test failures introduced by use_executable_stage_bundle_execution > > > Key: BEAM-6231 > URL: https://issues.apache.org/jira/browse/BEAM-6231 > Project: Beam > Issue Type: Test > Components: runner-dataflow >Reporter: Boyuan Zhang >Assignee: Boyuan Zhang >Priority: Major > Time Spent: 6h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5873) Python test failure: "ImportError: No module named pip._internal"
[ https://issues.apache.org/jira/browse/BEAM-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740973#comment-16740973 ] Valentyn Tymofieiev commented on BEAM-5873: --- We believe the rootcause of the issue is caused by GCE VM restart around the start of the Dataflow pipeline, which causes corruption of filesystem in the Docker container. We continue investigation. This issue would be hard to fix in Beam plane, and it needs to be fixed in Dataflow and/or downstream dependencies. > Python test failure: "ImportError: No module named pip._internal" > - > > Key: BEAM-5873 > URL: https://issues.apache.org/jira/browse/BEAM-5873 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Henning Rohde >Assignee: Valentyn Tymofieiev >Priority: Major > > https://scans.gradle.com/s/r55ln7mdibu2w/console-log?task=:beam-sdks-python:postCommitITTests#L163 > Logs: > https://pantheon.corp.google.com/logs/viewer?resource=dataflow_step%2Fjob_id%2F2018-10-26_06_46_26-13501822612780835073=projects%2Fapache-beam-testing%2Flogs%2Fdataflow.googleapis.com%252Fworker-startup=NO_LIMIT=apache-beam-testing=0=false=2018-10-26T20:01:54.77300Z==true=2018-10-26T13:49:18.405228000Z > Executing: /usr/local/bin/pip install > /var/opt/google/dataflow/dataflow_python_sdk.tar[gcp] > Debug: delayed tasks complete > Debug: download complete > Traceback (most recent call last): > File "/usr/local/bin/pip", line 7, in > from pip._internal import main > ImportError > : > No module named pip._internal > /usr/local/bin/pip failed with exit status 1 > Maybe a flake? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6419) [SQL] Jacoco error: Classes in bundle 'beam-sdks-java-extensions-sql' do no match with execution data.
[
https://issues.apache.org/jira/browse/BEAM-6419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740975#comment-16740975
]
Kenneth Knowles commented on BEAM-6419:
---
Seems a common issue. Another at
https://stackoverflow.com/questions/44776226/execution-data-for-class-does-not-match-jacoco
> [SQL] Jacoco error: Classes in bundle 'beam-sdks-java-extensions-sql' do no
> match with execution data.
> --
>
> Key: BEAM-6419
> URL: https://issues.apache.org/jira/browse/BEAM-6419
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>
> {code}
> [ant:jacocoReport] Classes in bundle 'beam-sdks-java-extensions-sql' do no
> match with execution data. For report generation the same class files must be
> used as at runtime.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTable does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/udf/IsNan does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnnestRel$Transform does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamSetOperatorRelBase$OpType
> does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProvider$RowToCsv
> does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamSortRel does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/transform/agg/CovarianceFn does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/parser/impl/ParseException does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rule/BeamMinusRule does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations$IntegerAvg
> does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProvider
> does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rule/BeamUncollectRule does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel$1 does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel$CalcFn does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/ParseException does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTable does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/UdfUdafProvider does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaCSVTable$CsvRecorderDecoder
> does not match.
> {code}
> ... and so on.
> There's some discussion of similar-sounding issues at
> https://stackoverflow.com/questions/31720139/jacoco-code-coverage-report-generator-showing-error-classes-in-bundle-code-c
> If JaCoCo is looking at the class files, but tests run against the shaded
> jar, this would be expected because only byte-for-byte identical class files
> will match.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6248) Add Flink 1.7.x build target to Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-6248?focusedWorklogId=184464=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184464 ] ASF GitHub Bot logged work on BEAM-6248: Author: ASF GitHub Bot Created on: 12/Jan/19 02:26 Start Date: 12/Jan/19 02:26 Worklog Time Spent: 10m Work Description: mxm commented on issue #7489: [BEAM-6248] Disable tests for Flink 1.7 build target URL: https://github.com/apache/beam/pull/7489#issuecomment-453711918 Thanks for merging. @aaltay Yes, this should fix the issue. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184464) Time Spent: 7h (was: 6h 50m) > Add Flink 1.7.x build target to Flink Runner > > > Key: BEAM-6248 > URL: https://issues.apache.org/jira/browse/BEAM-6248 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0, 2.11.0 > > Time Spent: 7h > Remaining Estimate: 0h > > With BEAM-5419 we can add a Flink 1.7.x build target. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6419) [SQL] Jacoco error: Classes in bundle 'beam-sdks-java-extensions-sql' do no match with execution data.
Kenneth Knowles created BEAM-6419:
-
Summary: [SQL] Jacoco error: Classes in bundle
'beam-sdks-java-extensions-sql' do no match with execution data.
Key: BEAM-6419
URL: https://issues.apache.org/jira/browse/BEAM-6419
Project: Beam
Issue Type: Bug
Components: dsl-sql
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles
{code}
[ant:jacocoReport] Classes in bundle 'beam-sdks-java-extensions-sql' do no
match with execution data. For report generation the same class files must be
used as at runtime.
[ant:jacocoReport] Execution data for class
org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTable does not match.
[ant:jacocoReport] Execution data for class
org/apache/beam/sdk/extensions/sql/impl/udf/IsNan does not match.
[ant:jacocoReport] Execution data for class
org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnnestRel$Transform does not
match.
[ant:jacocoReport] Execution data for class
org/apache/beam/sdk/extensions/sql/impl/rel/BeamSetOperatorRelBase$OpType does
not match.
[ant:jacocoReport] Execution data for class
org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProvider$RowToCsv
does not match.
[ant:jacocoReport] Execution data for class
org/apache/beam/sdk/extensions/sql/impl/rel/BeamSortRel does not match.
[ant:jacocoReport] Execution data for class
org/apache/beam/sdk/extensions/sql/impl/transform/agg/CovarianceFn does not
match.
[ant:jacocoReport] Execution data for class
org/apache/beam/sdk/extensions/sql/impl/parser/impl/ParseException does not
match.
[ant:jacocoReport] Execution data for class
org/apache/beam/sdk/extensions/sql/impl/rule/BeamMinusRule does not match.
[ant:jacocoReport] Execution data for class
org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations$IntegerAvg
does not match.
[ant:jacocoReport] Execution data for class
org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProvider does
not match.
[ant:jacocoReport] Execution data for class
org/apache/beam/sdk/extensions/sql/impl/rule/BeamUncollectRule does not match.
[ant:jacocoReport] Execution data for class
org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel$1 does not match.
[ant:jacocoReport] Execution data for class
org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel$CalcFn does not match.
[ant:jacocoReport] Execution data for class
org/apache/beam/sdk/extensions/sql/impl/ParseException does not match.
[ant:jacocoReport] Execution data for class
org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTable does not match.
[ant:jacocoReport] Execution data for class
org/apache/beam/sdk/extensions/sql/meta/provider/UdfUdafProvider does not match.
[ant:jacocoReport] Execution data for class
org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaCSVTable$CsvRecorderDecoder
does not match.
{code}
... and so on.
There's some discussion of similar-sounding issues at
https://stackoverflow.com/questions/31720139/jacoco-code-coverage-report-generator-showing-error-classes-in-bundle-code-c
If JaCoCo is looking at the class files, but tests run against the shaded jar,
this would be expected because only byte-for-byte identical class files will
match.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6184) PortableRunner dependency missed in wordcount example maven artifact
[ https://issues.apache.org/jira/browse/BEAM-6184?focusedWorklogId=184465=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184465 ] ASF GitHub Bot logged work on BEAM-6184: Author: ASF GitHub Bot Created on: 12/Jan/19 02:31 Start Date: 12/Jan/19 02:31 Worklog Time Spent: 10m Work Description: HuangLED commented on issue #7454: [BEAM-6184] Enforce javadoc check on non-trivial public methods URL: https://github.com/apache/beam/pull/7454#issuecomment-453705069 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184465) Time Spent: 3h 10m (was: 3h) > PortableRunner dependency missed in wordcount example maven artifact > > > Key: BEAM-6184 > URL: https://issues.apache.org/jira/browse/BEAM-6184 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Ruoyun Huang >Assignee: Ruoyun Huang >Priority: Minor > Fix For: Not applicable > > Time Spent: 3h 10m > Remaining Estimate: 0h > > > > more context: > https://lists.apache.org/thread.html/8dd60395424425f7502d62888c49014430d1d3b06c026606f3db28ab@%3Cuser.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6184) PortableRunner dependency missed in wordcount example maven artifact
[ https://issues.apache.org/jira/browse/BEAM-6184?focusedWorklogId=184466=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184466 ] ASF GitHub Bot logged work on BEAM-6184: Author: ASF GitHub Bot Created on: 12/Jan/19 02:31 Start Date: 12/Jan/19 02:31 Worklog Time Spent: 10m Work Description: HuangLED commented on issue #7454: [BEAM-6184] Enforce javadoc check on non-trivial public methods URL: https://github.com/apache/beam/pull/7454#issuecomment-453712318 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184466) Time Spent: 3h 20m (was: 3h 10m) > PortableRunner dependency missed in wordcount example maven artifact > > > Key: BEAM-6184 > URL: https://issues.apache.org/jira/browse/BEAM-6184 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Ruoyun Huang >Assignee: Ruoyun Huang >Priority: Minor > Fix For: Not applicable > > Time Spent: 3h 20m > Remaining Estimate: 0h > > > > more context: > https://lists.apache.org/thread.html/8dd60395424425f7502d62888c49014430d1d3b06c026606f3db28ab@%3Cuser.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=184462=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184462 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 12/Jan/19 02:23 Start Date: 12/Jan/19 02:23 Worklog Time Spent: 10m Work Description: tweise commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-453711696 Run Python Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184462) Time Spent: 5h 20m (was: 5h 10m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 5h 20m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6405) Improve PortableValidatesRunner test reliability on Jenkins
[ https://issues.apache.org/jira/browse/BEAM-6405?focusedWorklogId=184461=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184461 ] ASF GitHub Bot logged work on BEAM-6405: Author: ASF GitHub Bot Created on: 12/Jan/19 02:22 Start Date: 12/Jan/19 02:22 Worklog Time Spent: 10m Work Description: tweise commented on issue #7461: [BEAM-6405] Let PortableValidatesRunner tests run in EMBEDDED environment URL: https://github.com/apache/beam/pull/7461#issuecomment-453711637 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184461) Time Spent: 4h 10m (was: 4h) > Improve PortableValidatesRunner test reliability on Jenkins > --- > > Key: BEAM-6405 > URL: https://issues.apache.org/jira/browse/BEAM-6405 > Project: Beam > Issue Type: Test > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Time Spent: 4h 10m > Remaining Estimate: 0h > > The PVR tests seem to be passing fine and then failing consecutively for no > reason: https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/ > It looks like the outrageous parallelism, i.e. number of available cores, is > responsible for the flakiness if there is additional load on the build > slaves. We should lower the parallelism. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=184460=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184460 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 12/Jan/19 02:02 Start Date: 12/Jan/19 02:02 Worklog Time Spent: 10m Work Description: tweise commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-453710271 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184460) Time Spent: 5h 10m (was: 5h) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 5h 10m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6248) Add Flink 1.7.x build target to Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-6248?focusedWorklogId=184459=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184459 ] ASF GitHub Bot logged work on BEAM-6248: Author: ASF GitHub Bot Created on: 12/Jan/19 01:58 Start Date: 12/Jan/19 01:58 Worklog Time Spent: 10m Work Description: tweise commented on issue #7489: [BEAM-6248] Disable tests for Flink 1.7 build target URL: https://github.com/apache/beam/pull/7489#issuecomment-453709965 Merging after 3 build runs that did not trigger the memory issue. @aaltay yes, I believe this is the same issue. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184459) Time Spent: 6h 50m (was: 6h 40m) > Add Flink 1.7.x build target to Flink Runner > > > Key: BEAM-6248 > URL: https://issues.apache.org/jira/browse/BEAM-6248 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0, 2.11.0 > > Time Spent: 6h 50m > Remaining Estimate: 0h > > With BEAM-5419 we can add a Flink 1.7.x build target. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6248) Add Flink 1.7.x build target to Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-6248?focusedWorklogId=184458=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184458 ] ASF GitHub Bot logged work on BEAM-6248: Author: ASF GitHub Bot Created on: 12/Jan/19 01:56 Start Date: 12/Jan/19 01:56 Worklog Time Spent: 10m Work Description: tweise commented on pull request #7489: [BEAM-6248] Disable tests for Flink 1.7 build target URL: https://github.com/apache/beam/pull/7489 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184458) Time Spent: 6h 40m (was: 6.5h) > Add Flink 1.7.x build target to Flink Runner > > > Key: BEAM-6248 > URL: https://issues.apache.org/jira/browse/BEAM-6248 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0, 2.11.0 > > Time Spent: 6h 40m > Remaining Estimate: 0h > > With BEAM-5419 we can add a Flink 1.7.x build target. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6248) Add Flink 1.7.x build target to Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-6248?focusedWorklogId=184456=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184456 ] ASF GitHub Bot logged work on BEAM-6248: Author: ASF GitHub Bot Created on: 12/Jan/19 01:32 Start Date: 12/Jan/19 01:32 Worklog Time Spent: 10m Work Description: tweise commented on issue #7489: [BEAM-6248] Disable tests for Flink 1.7 build target URL: https://github.com/apache/beam/pull/7489#issuecomment-453707749 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184456) Time Spent: 6.5h (was: 6h 20m) > Add Flink 1.7.x build target to Flink Runner > > > Key: BEAM-6248 > URL: https://issues.apache.org/jira/browse/BEAM-6248 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0, 2.11.0 > > Time Spent: 6.5h > Remaining Estimate: 0h > > With BEAM-5419 we can add a Flink 1.7.x build target. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6248) Add Flink 1.7.x build target to Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-6248?focusedWorklogId=184454=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184454 ] ASF GitHub Bot logged work on BEAM-6248: Author: ASF GitHub Bot Created on: 12/Jan/19 01:28 Start Date: 12/Jan/19 01:28 Worklog Time Spent: 10m Work Description: aaltay commented on issue #7489: [BEAM-6248] Disable tests for Flink 1.7 build target URL: https://github.com/apache/beam/pull/7489#issuecomment-453707382 Is this related to https://issues.apache.org/jira/browse/BEAM-6418 ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184454) Time Spent: 6h 20m (was: 6h 10m) > Add Flink 1.7.x build target to Flink Runner > > > Key: BEAM-6248 > URL: https://issues.apache.org/jira/browse/BEAM-6248 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0, 2.11.0 > > Time Spent: 6h 20m > Remaining Estimate: 0h > > With BEAM-5419 we can add a Flink 1.7.x build target. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6284) [FLAKE][beam_PostCommit_Java_ValidatesRunner_Dataflow] TestRunner fails with result UNKNOWN on succeeded job and checks passed
[ https://issues.apache.org/jira/browse/BEAM-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740936#comment-16740936 ] Daniel Oliveira commented on BEAM-6284: --- This issue is being actively investigated internally in Dataflow right now, which is why there aren't any visible updates here. It seems valuable to keep this bug up and keep recording instances of this happening, so I'll try to keep this bug updated with progress so it doesn't look like it's being ignored. > [FLAKE][beam_PostCommit_Java_ValidatesRunner_Dataflow] TestRunner fails with > result UNKNOWN on succeeded job and checks passed > -- > > Key: BEAM-6284 > URL: https://issues.apache.org/jira/browse/BEAM-6284 > Project: Beam > Issue Type: Bug > Components: test-failures, testing >Reporter: Mikhail Gryzykhin >Assignee: Daniel Oliveira >Priority: Major > Labels: currently-failing > > _Use this form to file an issue for test failure:_ > * > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/testReport/junit/org.apache.beam.sdk.transforms/ViewTest/testWindowedSideInputFixedToGlobal/ > Initial investigation: > According to logs all test-relevant checks have passed and it seem to be > testing framework failure. > > _After you've filled out the above details, please [assign the issue to an > individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist]. > Assignee should [treat test failures as > high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test], > helping to fix the issue or find a more appropriate owner. See [Apache Beam > Post-Commit > Policies|https://beam.apache.org/contribute/postcommits-policies]._ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6184) PortableRunner dependency missed in wordcount example maven artifact
[ https://issues.apache.org/jira/browse/BEAM-6184?focusedWorklogId=184453=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184453 ] ASF GitHub Bot logged work on BEAM-6184: Author: ASF GitHub Bot Created on: 12/Jan/19 01:05 Start Date: 12/Jan/19 01:05 Worklog Time Spent: 10m Work Description: HuangLED commented on issue #7454: [BEAM-6184] Enforce javadoc check on non-trivial public methods URL: https://github.com/apache/beam/pull/7454#issuecomment-453705069 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184453) Time Spent: 3h (was: 2h 50m) > PortableRunner dependency missed in wordcount example maven artifact > > > Key: BEAM-6184 > URL: https://issues.apache.org/jira/browse/BEAM-6184 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Ruoyun Huang >Assignee: Ruoyun Huang >Priority: Minor > Fix For: Not applicable > > Time Spent: 3h > Remaining Estimate: 0h > > > > more context: > https://lists.apache.org/thread.html/8dd60395424425f7502d62888c49014430d1d3b06c026606f3db28ab@%3Cuser.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-6294) Use Flink's redistribute for reshuffle.
[ https://issues.apache.org/jira/browse/BEAM-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved BEAM-6294. Resolution: Fixed > Use Flink's redistribute for reshuffle. > --- > > Key: BEAM-6294 > URL: https://issues.apache.org/jira/browse/BEAM-6294 > Project: Beam > Issue Type: New Feature > Components: runner-flink, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.10.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Python needs to publish the URN over the FnAPI which is pretty easy, but > Flink also needs to ensure that the composite structure does not get fused. > Unlike with GBK, we can't assume all runners implement this as a primitive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6294) Use Flink's redistribute for reshuffle.
[ https://issues.apache.org/jira/browse/BEAM-6294?focusedWorklogId=184452=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184452 ] ASF GitHub Bot logged work on BEAM-6294: Author: ASF GitHub Bot Created on: 12/Jan/19 00:59 Start Date: 12/Jan/19 00:59 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #7490: [BEAM-6294] Ensure input and output coders are equal for reshuffle transforms URL: https://github.com/apache/beam/pull/7490#issuecomment-453704397 LGTM (was waiting for green) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184452) Time Spent: 2h 20m (was: 2h 10m) > Use Flink's redistribute for reshuffle. > --- > > Key: BEAM-6294 > URL: https://issues.apache.org/jira/browse/BEAM-6294 > Project: Beam > Issue Type: New Feature > Components: runner-flink, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.10.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Python needs to publish the URN over the FnAPI which is pretty easy, but > Flink also needs to ensure that the composite structure does not get fused. > Unlike with GBK, we can't assume all runners implement this as a primitive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6248) Add Flink 1.7.x build target to Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-6248?focusedWorklogId=184451=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184451 ] ASF GitHub Bot logged work on BEAM-6248: Author: ASF GitHub Bot Created on: 12/Jan/19 00:58 Start Date: 12/Jan/19 00:58 Worklog Time Spent: 10m Work Description: tweise commented on issue #7489: [BEAM-6248] Disable tests for Flink 1.7 build target URL: https://github.com/apache/beam/pull/7489#issuecomment-453704246 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184451) Time Spent: 6h 10m (was: 6h) > Add Flink 1.7.x build target to Flink Runner > > > Key: BEAM-6248 > URL: https://issues.apache.org/jira/browse/BEAM-6248 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0, 2.11.0 > > Time Spent: 6h 10m > Remaining Estimate: 0h > > With BEAM-5419 we can add a Flink 1.7.x build target. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6418) beam_PreCommit_Java_Cron failing
Ahmet Altay created BEAM-6418: - Summary: beam_PreCommit_Java_Cron failing Key: BEAM-6418 URL: https://issues.apache.org/jira/browse/BEAM-6418 Project: Beam Issue Type: Bug Components: runner-flink, test-failures Reporter: Ahmet Altay Assignee: Maximilian Michels [https://builds.apache.org/job/beam_PreCommit_Java_Cron/814/console] *16:22:16* 1: Task failed with an exception.*16:22:16* ---*16:22:16* * What went wrong:*16:22:16* Execution failed for task ':beam-runners-flink-1.6:test'.*16:22:16* > Process 'Gradle Test Executor 108' finished with non-zero exit value 137*16:22:16* This problem might be caused by incorrect test process configuration.*16:22:16* Please refer to the test execution section in the user guide at [https://docs.gradle.org/4.10.3/userguide/java_plugin.html#sec:test_execution] *16:22:16* *16:22:16* * Try:*16:22:16* Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.*16:22:16* ==*16:22:16* *16:22:16* 2: Task failed with an exception.*16:22:16* ---*16:22:16* * What went wrong:*16:22:16* Execution failed for task ':beam-runners-flink_2.11:test'.*16:22:16* > Process 'Gradle Test Executor 110' finished with non-zero exit value 1*16:22:16* This problem might be caused by incorrect test process configuration.*16:22:16* Please refer to the test execution section in the user guide at [https://docs.gradle.org/4.10.3/userguide/java_plugin.html#sec:test_execution] *16:22:16* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-6284) [FLAKE][beam_PostCommit_Java_ValidatesRunner_Dataflow] TestRunner fails with result UNKNOWN on succeeded job and checks passed
[ https://issues.apache.org/jira/browse/BEAM-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Rohde reassigned BEAM-6284: --- Assignee: Daniel Oliveira (was: yifan zou) > [FLAKE][beam_PostCommit_Java_ValidatesRunner_Dataflow] TestRunner fails with > result UNKNOWN on succeeded job and checks passed > -- > > Key: BEAM-6284 > URL: https://issues.apache.org/jira/browse/BEAM-6284 > Project: Beam > Issue Type: Bug > Components: test-failures, testing >Reporter: Mikhail Gryzykhin >Assignee: Daniel Oliveira >Priority: Major > Labels: currently-failing > > _Use this form to file an issue for test failure:_ > * > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/testReport/junit/org.apache.beam.sdk.transforms/ViewTest/testWindowedSideInputFixedToGlobal/ > Initial investigation: > According to logs all test-relevant checks have passed and it seem to be > testing framework failure. > > _After you've filled out the above details, please [assign the issue to an > individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist]. > Assignee should [treat test failures as > high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test], > helping to fix the issue or find a more appropriate owner. See [Apache Beam > Post-Commit > Policies|https://beam.apache.org/contribute/postcommits-policies]._ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6294) Use Flink's redistribute for reshuffle.
[ https://issues.apache.org/jira/browse/BEAM-6294?focusedWorklogId=184450=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184450 ] ASF GitHub Bot logged work on BEAM-6294: Author: ASF GitHub Bot Created on: 12/Jan/19 00:54 Start Date: 12/Jan/19 00:54 Worklog Time Spent: 10m Work Description: tweise commented on pull request #7490: [BEAM-6294] Ensure input and output coders are equal for reshuffle transforms URL: https://github.com/apache/beam/pull/7490 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184450) Time Spent: 2h 10m (was: 2h) > Use Flink's redistribute for reshuffle. > --- > > Key: BEAM-6294 > URL: https://issues.apache.org/jira/browse/BEAM-6294 > Project: Beam > Issue Type: New Feature > Components: runner-flink, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.10.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Python needs to publish the URN over the FnAPI which is pretty easy, but > Flink also needs to ensure that the composite structure does not get fused. > Unlike with GBK, we can't assume all runners implement this as a primitive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-6341) Python VR failure: test_flatten_multiple_pcollections_having_multiple_consumers
[ https://issues.apache.org/jira/browse/BEAM-6341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira resolved BEAM-6341. --- Resolution: Duplicate Fix Version/s: Not applicable > Python VR failure: > test_flatten_multiple_pcollections_having_multiple_consumers > --- > > Key: BEAM-6341 > URL: https://issues.apache.org/jira/browse/BEAM-6341 > Project: Beam > Issue Type: Test > Components: test-failures >Reporter: Boyuan Zhang >Assignee: Daniel Oliveira >Priority: Major > Fix For: Not applicable > > > [https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/2154/] > Workflow failed. Causes: The Dataflow job appears to be stuck because no > worker activity has been seen in the last 1h. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (BEAM-6341) Python VR failure: test_flatten_multiple_pcollections_having_multiple_consumers
[ https://issues.apache.org/jira/browse/BEAM-6341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740919#comment-16740919 ] Daniel Oliveira edited comment on BEAM-6341 at 1/12/19 12:42 AM: - Looks like this issue is a duplicate of BEAM-5873 so I'll close this for now. was (Author: danoliveira): Looks like this issue is a duplicate of **BEAM-5873 so I'll close this for now. > Python VR failure: > test_flatten_multiple_pcollections_having_multiple_consumers > --- > > Key: BEAM-6341 > URL: https://issues.apache.org/jira/browse/BEAM-6341 > Project: Beam > Issue Type: Test > Components: test-failures >Reporter: Boyuan Zhang >Assignee: Daniel Oliveira >Priority: Major > > [https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/2154/] > Workflow failed. Causes: The Dataflow job appears to be stuck because no > worker activity has been seen in the last 1h. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6341) Python VR failure: test_flatten_multiple_pcollections_having_multiple_consumers
[ https://issues.apache.org/jira/browse/BEAM-6341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740919#comment-16740919 ] Daniel Oliveira commented on BEAM-6341: --- Looks like this issue is a duplicate of **BEAM-5873 so I'll close this for now. > Python VR failure: > test_flatten_multiple_pcollections_having_multiple_consumers > --- > > Key: BEAM-6341 > URL: https://issues.apache.org/jira/browse/BEAM-6341 > Project: Beam > Issue Type: Test > Components: test-failures >Reporter: Boyuan Zhang >Assignee: Daniel Oliveira >Priority: Major > > [https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/2154/] > Workflow failed. Causes: The Dataflow job appears to be stuck because no > worker activity has been seen in the last 1h. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6248) Add Flink 1.7.x build target to Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-6248?focusedWorklogId=18=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-18 ] ASF GitHub Bot logged work on BEAM-6248: Author: ASF GitHub Bot Created on: 12/Jan/19 00:26 Start Date: 12/Jan/19 00:26 Worklog Time Spent: 10m Work Description: akedin commented on issue #7489: [BEAM-6248] Disable tests for Flink 1.7 build target URL: https://github.com/apache/beam/pull/7489#issuecomment-453700189 Ah yes, seeing it :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 18) Time Spent: 6h (was: 5h 50m) > Add Flink 1.7.x build target to Flink Runner > > > Key: BEAM-6248 > URL: https://issues.apache.org/jira/browse/BEAM-6248 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0, 2.11.0 > > Time Spent: 6h > Remaining Estimate: 0h > > With BEAM-5419 we can add a Flink 1.7.x build target. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6248) Add Flink 1.7.x build target to Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-6248?focusedWorklogId=184441=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184441 ] ASF GitHub Bot logged work on BEAM-6248: Author: ASF GitHub Bot Created on: 12/Jan/19 00:25 Start Date: 12/Jan/19 00:25 Worklog Time Spent: 10m Work Description: akedin commented on issue #7489: [BEAM-6248] Disable tests for Flink 1.7 build target URL: https://github.com/apache/beam/pull/7489#issuecomment-45370 Is the 1.7 change causing these failures as well? https://builds.apache.org/job/beam_PreCommit_Java_Commit/3484/console This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184441) Time Spent: 5h 50m (was: 5h 40m) > Add Flink 1.7.x build target to Flink Runner > > > Key: BEAM-6248 > URL: https://issues.apache.org/jira/browse/BEAM-6248 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0, 2.11.0 > > Time Spent: 5h 50m > Remaining Estimate: 0h > > With BEAM-5419 we can add a Flink 1.7.x build target. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5873) Python test failure: "ImportError: No module named pip._internal"
[ https://issues.apache.org/jira/browse/BEAM-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740904#comment-16740904 ] Daniel Oliveira commented on BEAM-5873: --- Valentyn, this looks like it's the root cause of the flakiness reported here: BEAM-6341. I have more details in the comments there. Can you take a look and confirm if this is the same issue? If so, I could collaborate with you if it would help. > Python test failure: "ImportError: No module named pip._internal" > - > > Key: BEAM-5873 > URL: https://issues.apache.org/jira/browse/BEAM-5873 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Henning Rohde >Assignee: Valentyn Tymofieiev >Priority: Major > > https://scans.gradle.com/s/r55ln7mdibu2w/console-log?task=:beam-sdks-python:postCommitITTests#L163 > Logs: > https://pantheon.corp.google.com/logs/viewer?resource=dataflow_step%2Fjob_id%2F2018-10-26_06_46_26-13501822612780835073=projects%2Fapache-beam-testing%2Flogs%2Fdataflow.googleapis.com%252Fworker-startup=NO_LIMIT=apache-beam-testing=0=false=2018-10-26T20:01:54.77300Z==true=2018-10-26T13:49:18.405228000Z > Executing: /usr/local/bin/pip install > /var/opt/google/dataflow/dataflow_python_sdk.tar[gcp] > Debug: delayed tasks complete > Debug: download complete > Traceback (most recent call last): > File "/usr/local/bin/pip", line 7, in > from pip._internal import main > ImportError > : > No module named pip._internal > /usr/local/bin/pip failed with exit status 1 > Maybe a flake? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6341) Python VR failure: test_flatten_multiple_pcollections_having_multiple_consumers
[
https://issues.apache.org/jira/browse/BEAM-6341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740900#comment-16740900
]
Daniel Oliveira commented on BEAM-6341:
---
I see some errors in the logs that seem relevant. There's this error:
{noformat}
I Traceback (most recent call last):
IFile "/usr/local/bin/pip", line 7, in
I
I from pip._internal import main
I ImportError
I :
I No module named pip._internal
I /usr/local/bin/pip failed with exit status 1
I Dataflow base path override: https://dataflow.googleapis.com/
I Failed to report setup error to service: could not lease work item to report
failure (no work items returned)
F Failed to install packages: failed to install SDK: exit status 1
{noformat}
The actual failure seems to be happening in boot.go:
{noformat}
jsonPayload: {
line: "boot.go:145"
message: "Failed to install packages: failed to install SDK: exit status 1"
}{noformat}
Timestamp lines up to when the test was running (2018-12-30 14:23:55.933 PST,
test started at 2:17 PST) and the errors stop when the test ended. So it's
almost definitely boot.go being unable to install the SDK, now I just have to
find out why.
> Python VR failure:
> test_flatten_multiple_pcollections_having_multiple_consumers
> ---
>
> Key: BEAM-6341
> URL: https://issues.apache.org/jira/browse/BEAM-6341
> Project: Beam
> Issue Type: Test
> Components: test-failures
>Reporter: Boyuan Zhang
>Assignee: Daniel Oliveira
>Priority: Major
>
> [https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/2154/]
> Workflow failed. Causes: The Dataflow job appears to be stuck because no
> worker activity has been seen in the last 1h.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6280) Failure in PortableRunnerTest.test_error_traceback_includes_user_code
[
https://issues.apache.org/jira/browse/BEAM-6280?focusedWorklogId=184433=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184433
]
ASF GitHub Bot logged work on BEAM-6280:
Author: ASF GitHub Bot
Created on: 12/Jan/19 00:01
Start Date: 12/Jan/19 00:01
Worklog Time Spent: 10m
Work Description: rohdesamuel commented on issue #7433: [BEAM-6280]
Refactors Python portability tests to be multi-threaded aware
URL: https://github.com/apache/beam/pull/7433#issuecomment-453695836
Run Portable_Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 184433)
Time Spent: 4h 20m (was: 4h 10m)
> Failure in PortableRunnerTest.test_error_traceback_includes_user_code
> -
>
> Key: BEAM-6280
> URL: https://issues.apache.org/jira/browse/BEAM-6280
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Kenneth Knowles
>Assignee: Sam Rohde
>Priority: Critical
> Labels: flaky-test
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PreCommit_Python_Cron/732/]
> [https://builds.apache.org/job/beam_PreCommit_Python_Cron/732/testReport/apache_beam.runners.portability.portable_runner_test/PortableRunnerTest/test_error_traceback_includes_user_code/]
> [https://scans.gradle.com/s/do3hjulee3gaa/console-log?task=:beam-sdks-python:testPython3]
> {code:java}
> 'second' not found in 'Traceback (most recent call last):\n File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py",
> line 466, in test_error_traceback_includes_user_code\np |
> beam.Create([0]) | beam.Map(first) # pylint:
> disable=expression-not-assigned\n File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/apache_beam/pipeline.py",
> line 425, in __exit__\nself.run().wait_until_finish()\n File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/apache_beam/runners/portability/portable_runner.py",
> line 314, in wait_until_finish\nself._job_id, self._state,
> self._last_error_message()))\nRuntimeError: Pipeline
> job-cdcefe6d-1caa-4487-9e63-e971f67ec68c failed in state FAILED: start
> coder=WindowedValueCoder[BytesCoder], len(consumers)=1]]>\n'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6248) Add Flink 1.7.x build target to Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-6248?focusedWorklogId=184426=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184426 ] ASF GitHub Bot logged work on BEAM-6248: Author: ASF GitHub Bot Created on: 11/Jan/19 23:52 Start Date: 11/Jan/19 23:52 Worklog Time Spent: 10m Work Description: mxm commented on issue #7489: [BEAM-6248] Disable tests for Flink 1.7 build target URL: https://github.com/apache/beam/pull/7489#issuecomment-453694196 No 1 successful: https://builds.apache.org/job/beam_PreCommit_Java_Commit/3488/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184426) Time Spent: 5h 40m (was: 5.5h) > Add Flink 1.7.x build target to Flink Runner > > > Key: BEAM-6248 > URL: https://issues.apache.org/jira/browse/BEAM-6248 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0, 2.11.0 > > Time Spent: 5h 40m > Remaining Estimate: 0h > > With BEAM-5419 we can add a Flink 1.7.x build target. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6294) Use Flink's redistribute for reshuffle.
[ https://issues.apache.org/jira/browse/BEAM-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740879#comment-16740879 ] Thomas Weise commented on BEAM-6294: Yes, this is fixed: [https://github.com/apache/beam/pull/7490] > Use Flink's redistribute for reshuffle. > --- > > Key: BEAM-6294 > URL: https://issues.apache.org/jira/browse/BEAM-6294 > Project: Beam > Issue Type: New Feature > Components: runner-flink, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.10.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Python needs to publish the URN over the FnAPI which is pretty easy, but > Flink also needs to ensure that the composite structure does not get fused. > Unlike with GBK, we can't assume all runners implement this as a primitive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6294) Use Flink's redistribute for reshuffle.
[ https://issues.apache.org/jira/browse/BEAM-6294?focusedWorklogId=184404=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184404 ] ASF GitHub Bot logged work on BEAM-6294: Author: ASF GitHub Bot Created on: 11/Jan/19 23:31 Start Date: 11/Jan/19 23:31 Worklog Time Spent: 10m Work Description: tweise commented on pull request #7490: [BEAM-6294] Ensure input and output coders are equal for reshuffle transforms URL: https://github.com/apache/beam/pull/7490 The type declarations were there, but not getting applied due to a longstanding TODO. This doesn't resolve that TODO completely, but fixes a large number of cases, including this one. **Please** add a meaningful description for your change here Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please
[jira] [Commented] (BEAM-6182) Use of conscrypt SSL results in stuck workflows in Dataflow
[ https://issues.apache.org/jira/browse/BEAM-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740873#comment-16740873 ] Kenneth Knowles commented on BEAM-6182: --- Just making sure of this: I do not need this to be resolved and in the 2.10.0 release notes. If it should be listed in the release notes, then I will resolve it and open a follow-up to re-enable. > Use of conscrypt SSL results in stuck workflows in Dataflow > --- > > Key: BEAM-6182 > URL: https://issues.apache.org/jira/browse/BEAM-6182 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Ahmet Altay >Assignee: Tyler Akidau >Priority: Blocker > Fix For: 2.10.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > An experimental flag is being added to disable it for now with an option to > enable it per-workflow. > Also related: > https://issues.apache.org/jira/browse/BEAM-5747 - Upgrade conscrypt to its > latest version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6182) Use of conscrypt SSL results in stuck workflows in Dataflow
[ https://issues.apache.org/jira/browse/BEAM-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740872#comment-16740872 ] Kenneth Knowles commented on BEAM-6182: --- OK, I think since the validation is where it needs to be, I will remove this from the list of release blockers. > Use of conscrypt SSL results in stuck workflows in Dataflow > --- > > Key: BEAM-6182 > URL: https://issues.apache.org/jira/browse/BEAM-6182 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Ahmet Altay >Assignee: Tyler Akidau >Priority: Blocker > Fix For: 2.10.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > An experimental flag is being added to disable it for now with an option to > enable it per-workflow. > Also related: > https://issues.apache.org/jira/browse/BEAM-5747 - Upgrade conscrypt to its > latest version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-6182) Use of conscrypt SSL results in stuck workflows in Dataflow
[ https://issues.apache.org/jira/browse/BEAM-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-6182: -- Fix Version/s: (was: 2.10.0) > Use of conscrypt SSL results in stuck workflows in Dataflow > --- > > Key: BEAM-6182 > URL: https://issues.apache.org/jira/browse/BEAM-6182 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Ahmet Altay >Assignee: Tyler Akidau >Priority: Blocker > Time Spent: 1.5h > Remaining Estimate: 0h > > An experimental flag is being added to disable it for now with an option to > enable it per-workflow. > Also related: > https://issues.apache.org/jira/browse/BEAM-5747 - Upgrade conscrypt to its > latest version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6248) Add Flink 1.7.x build target to Flink Runner
[
https://issues.apache.org/jira/browse/BEAM-6248?focusedWorklogId=184400=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184400
]
ASF GitHub Bot logged work on BEAM-6248:
Author: ASF GitHub Bot
Created on: 11/Jan/19 23:24
Start Date: 11/Jan/19 23:24
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #7489: [BEAM-6248]
Disable tests for Flink 1.7 build target
URL: https://github.com/apache/beam/pull/7489#discussion_r247282953
##
File path: runners/flink/flink_runner.gradle
##
@@ -68,6 +68,10 @@ test {
if (System.getProperty("beamSurefireArgline")) {
jvmArgs System.getProperty("beamSurefireArgline")
}
+}.onlyIf {
+ // TODO Running tests of all Flink versions in parallel can be too harsh on
Jenkins memory
+ // Skip 1.7 tests for now, to avoid OOM on Jenkins
Review comment:
Should add that its not a Java OOM but the OS kills the Gradle testing VM.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 184400)
Time Spent: 5.5h (was: 5h 20m)
> Add Flink 1.7.x build target to Flink Runner
>
>
> Key: BEAM-6248
> URL: https://issues.apache.org/jira/browse/BEAM-6248
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
> Fix For: 2.10.0, 2.11.0
>
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> With BEAM-5419 we can add a Flink 1.7.x build target.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6248) Add Flink 1.7.x build target to Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-6248?focusedWorklogId=184399=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184399 ] ASF GitHub Bot logged work on BEAM-6248: Author: ASF GitHub Bot Created on: 11/Jan/19 23:22 Start Date: 11/Jan/19 23:22 Worklog Time Spent: 10m Work Description: tweise commented on issue #7489: [BEAM-6248] Disable tests for Flink 1.7 build target URL: https://github.com/apache/beam/pull/7489#issuecomment-453688387 Let's run Java precommit a few times for confirmation. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184399) Time Spent: 5h 20m (was: 5h 10m) > Add Flink 1.7.x build target to Flink Runner > > > Key: BEAM-6248 > URL: https://issues.apache.org/jira/browse/BEAM-6248 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0, 2.11.0 > > Time Spent: 5h 20m > Remaining Estimate: 0h > > With BEAM-5419 we can add a Flink 1.7.x build target. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6248) Add Flink 1.7.x build target to Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-6248?focusedWorklogId=184398=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184398 ] ASF GitHub Bot logged work on BEAM-6248: Author: ASF GitHub Bot Created on: 11/Jan/19 23:19 Start Date: 11/Jan/19 23:19 Worklog Time Spent: 10m Work Description: mxm commented on pull request #7489: [BEAM-6248] Disable tests for Flink 1.7 build target URL: https://github.com/apache/beam/pull/7489 Running 1.5, 1.6, and 1.7 in parallel can cause OOM on Jenkins. Disabling 1.7 tests for now to restore PreCommit stability. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184398) Time Spent: 5h 10m (was: 5h) > Add Flink 1.7.x build target to Flink Runner > > > Key: BEAM-6248 > URL: https://issues.apache.org/jira/browse/BEAM-6248 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0, 2.11.0 > > Time Spent: 5h 10m > Remaining Estimate: 0h > > With BEAM-5419 we can add a Flink 1.7.x build target. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6405) Improve PortableValidatesRunner test reliability on Jenkins
[ https://issues.apache.org/jira/browse/BEAM-6405?focusedWorklogId=184393=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184393 ] ASF GitHub Bot logged work on BEAM-6405: Author: ASF GitHub Bot Created on: 11/Jan/19 23:01 Start Date: 11/Jan/19 23:01 Worklog Time Spent: 10m Work Description: mxm commented on issue #7461: [BEAM-6405] Let PortableValidatesRunner tests run in EMBEDDED environment URL: https://github.com/apache/beam/pull/7461#issuecomment-453684302 Run Java Flink PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184393) Time Spent: 3h 40m (was: 3.5h) > Improve PortableValidatesRunner test reliability on Jenkins > --- > > Key: BEAM-6405 > URL: https://issues.apache.org/jira/browse/BEAM-6405 > Project: Beam > Issue Type: Test > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > The PVR tests seem to be passing fine and then failing consecutively for no > reason: https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/ > It looks like the outrageous parallelism, i.e. number of available cores, is > responsible for the flakiness if there is additional load on the build > slaves. We should lower the parallelism. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6405) Improve PortableValidatesRunner test reliability on Jenkins
[ https://issues.apache.org/jira/browse/BEAM-6405?focusedWorklogId=184397=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184397 ] ASF GitHub Bot logged work on BEAM-6405: Author: ASF GitHub Bot Created on: 11/Jan/19 23:15 Start Date: 11/Jan/19 23:15 Worklog Time Spent: 10m Work Description: tweise commented on issue #7461: [BEAM-6405] Let PortableValidatesRunner tests run in EMBEDDED environment URL: https://github.com/apache/beam/pull/7461#issuecomment-453687098 Looks like this cuts the build time by ~30% - nice! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184397) Time Spent: 3h 50m (was: 3h 40m) > Improve PortableValidatesRunner test reliability on Jenkins > --- > > Key: BEAM-6405 > URL: https://issues.apache.org/jira/browse/BEAM-6405 > Project: Beam > Issue Type: Test > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > > The PVR tests seem to be passing fine and then failing consecutively for no > reason: https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/ > It looks like the outrageous parallelism, i.e. number of available cores, is > responsible for the flakiness if there is additional load on the build > slaves. We should lower the parallelism. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6184) PortableRunner dependency missed in wordcount example maven artifact
[
https://issues.apache.org/jira/browse/BEAM-6184?focusedWorklogId=184396=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184396
]
ASF GitHub Bot logged work on BEAM-6184:
Author: ASF GitHub Bot
Created on: 11/Jan/19 23:06
Start Date: 11/Jan/19 23:06
Worklog Time Spent: 10m
Work Description: kennknowles commented on pull request #7454:
[BEAM-6184] Enforce javadoc check on non-trivial public methods
URL: https://github.com/apache/beam/pull/7454#discussion_r247280005
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java
##
@@ -331,10 +331,8 @@ private void fireTimer(
public void close() throws Exception {
// close may be called multiple times when an exception is thrown
if (stageContext != null) {
- try (@SuppressWarnings("unused")
Review comment:
Sounds good to me. If we care about unused variables, we can turn it on
again. Removed the not-needed annotation is an improvement.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 184396)
Time Spent: 2h 50m (was: 2h 40m)
> PortableRunner dependency missed in wordcount example maven artifact
>
>
> Key: BEAM-6184
> URL: https://issues.apache.org/jira/browse/BEAM-6184
> Project: Beam
> Issue Type: Improvement
> Components: build-system
>Reporter: Ruoyun Huang
>Assignee: Ruoyun Huang
>Priority: Minor
> Fix For: Not applicable
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
>
>
> more context:
> https://lists.apache.org/thread.html/8dd60395424425f7502d62888c49014430d1d3b06c026606f3db28ab@%3Cuser.beam.apache.org%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6184) PortableRunner dependency missed in wordcount example maven artifact
[
https://issues.apache.org/jira/browse/BEAM-6184?focusedWorklogId=184394=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184394
]
ASF GitHub Bot logged work on BEAM-6184:
Author: ASF GitHub Bot
Created on: 11/Jan/19 23:04
Start Date: 11/Jan/19 23:04
Worklog Time Spent: 10m
Work Description: kennknowles commented on pull request #7454:
[BEAM-6184] Enforce javadoc check on non-trivial public methods
URL: https://github.com/apache/beam/pull/7454#discussion_r247245595
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamTableUtils.java
##
@@ -90,6 +90,7 @@ public static String beamRow2CsvLine(Row row, CSVFormat
csvFormat) {
return writer.toString();
}
+ @SuppressWarnings("checkstyle:JavadocMethod")
Review comment:
Seems like it should have documentation. I suggest `Attempt to cast an
object to a specified Schema.Field.Type @throws IllegalArgumentException if the
value cannot be cast to that type`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 184394)
Time Spent: 2h 40m (was: 2.5h)
> PortableRunner dependency missed in wordcount example maven artifact
>
>
> Key: BEAM-6184
> URL: https://issues.apache.org/jira/browse/BEAM-6184
> Project: Beam
> Issue Type: Improvement
> Components: build-system
>Reporter: Ruoyun Huang
>Assignee: Ruoyun Huang
>Priority: Minor
> Fix For: Not applicable
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
>
>
> more context:
> https://lists.apache.org/thread.html/8dd60395424425f7502d62888c49014430d1d3b06c026606f3db28ab@%3Cuser.beam.apache.org%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6184) PortableRunner dependency missed in wordcount example maven artifact
[ https://issues.apache.org/jira/browse/BEAM-6184?focusedWorklogId=184390=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184390 ] ASF GitHub Bot logged work on BEAM-6184: Author: ASF GitHub Bot Created on: 11/Jan/19 22:56 Start Date: 11/Jan/19 22:56 Worklog Time Spent: 10m Work Description: HuangLED commented on pull request #7454: [BEAM-6184] Enforce javadoc check on non-trivial public methods URL: https://github.com/apache/beam/pull/7454#discussion_r247278467 ## File path: sdks/java/build-tools/src/main/resources/beam/suppressions.xml ## @@ -21,6 +21,11 @@ + + + Review comment: Done. Removed. (apology after pulling from master and push again, the file status gets out of sync). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184390) Time Spent: 2.5h (was: 2h 20m) > PortableRunner dependency missed in wordcount example maven artifact > > > Key: BEAM-6184 > URL: https://issues.apache.org/jira/browse/BEAM-6184 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Ruoyun Huang >Assignee: Ruoyun Huang >Priority: Minor > Fix For: Not applicable > > Time Spent: 2.5h > Remaining Estimate: 0h > > > > more context: > https://lists.apache.org/thread.html/8dd60395424425f7502d62888c49014430d1d3b06c026606f3db28ab@%3Cuser.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-3772) BigQueryIO - Can't use DynamicDestination with CREATE_IF_NEEDED for unbounded PCollection and FILE_LOADS
[
https://issues.apache.org/jira/browse/BEAM-3772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Kirpichov reassigned BEAM-3772:
--
Assignee: Chamikara Jayalath (was: Eugene Kirpichov)
> BigQueryIO - Can't use DynamicDestination with CREATE_IF_NEEDED for unbounded
> PCollection and FILE_LOADS
>
>
> Key: BEAM-3772
> URL: https://issues.apache.org/jira/browse/BEAM-3772
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
>Affects Versions: 2.2.0, 2.3.0
> Environment: Dataflow streaming pipeline
>Reporter: Benjamin BENOIST
>Assignee: Chamikara Jayalath
>Priority: Major
>
> My workflow : KAFKA -> Dataflow streaming -> BigQuery
> Given that having low-latency isn't important in my case, I use FILE_LOADS to
> reduce the costs. I'm using _BigQueryIO.Write_ with a _DynamicDestination_,
> which is a table with the current hour as a suffix.
> This _BigQueryIO.Write_ is configured like this :
> {code:java}
> .withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED)
> .withMethod(Method.FILE_LOADS)
> .withTriggeringFrequency(triggeringFrequency)
> .withNumFileShards(100)
> {code}
> The first table is successfully created and is written to. But then the
> following tables are never created and I get these exceptions:
> {code:java}
> (99e5cd8c66414e7a): java.lang.RuntimeException: Failed to create load job
> with id prefix
> 5047f71312a94bf3a42ee5d67feede75_5295fbf25e1a7534f85e25dcaa9f4986_1_00023,
> reached max retries: 3, last failed load job: {
> "configuration" : {
> "load" : {
> "createDisposition" : "CREATE_NEVER",
> "destinationTable" : {
> "datasetId" : "dev_mydataset",
> "projectId" : "myproject-id",
> "tableId" : "mytable_20180302_16"
> },
> {code}
> The _CreateDisposition_ used is _CREATE_NEVER_, contrary as
> _CREATE_IF_NEEDED_ as specified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6280) Failure in PortableRunnerTest.test_error_traceback_includes_user_code
[
https://issues.apache.org/jira/browse/BEAM-6280?focusedWorklogId=184389=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184389
]
ASF GitHub Bot logged work on BEAM-6280:
Author: ASF GitHub Bot
Created on: 11/Jan/19 22:47
Start Date: 11/Jan/19 22:47
Worklog Time Spent: 10m
Work Description: rohdesamuel commented on pull request #7433:
[BEAM-6280] Refactors Python portability tests to be multi-threaded aware
URL: https://github.com/apache/beam/pull/7433#discussion_r247276818
##
File path: sdks/python/apache_beam/runners/portability/local_job_service.py
##
@@ -185,55 +152,129 @@ class BeamJob(threading.Thread):
The current state of the pipeline is available as self.state.
"""
- def __init__(self,
- job_id,
- pipeline_options,
- pipeline_proto):
+ def __init__(self, pipeline_proto):
super(BeamJob, self).__init__()
-self._job_id = job_id
-self._pipeline_options = pipeline_options
self._pipeline_proto = pipeline_proto
self._state = None
-self._state_change_callbacks = []
-self._last_log_message = None
-self._log_callbacks = [lambda msg: setattr(self, '_last_log_message', msg)]
+self._logs = []
+self._final_log_count = -1
+self._state_changes = []
+self._final_state_count = -1
+self._state_notifier = threading.Condition()
+self._log_notifier = threading.Condition()
self.state = beam_job_api_pb2.JobState.STARTING
-self.daemon = True
-
- def add_state_change_callback(self, f):
-self._state_change_callbacks.append(f)
-f(self.state)
-
- def add_log_callback(self, f):
-self._log_callbacks.append(f)
@property
def state(self):
return self._state
@state.setter
def state(self, new_state):
-for state_change_callback in self._state_change_callbacks:
- state_change_callback(new_state)
+"""Sets the job state.
+
+This will inform GetStateStream and GetMessageStream of the new state.
+"""
+
+self._state_notifier.acquire()
+self._log_notifier.acquire()
+
+self._state_changes.append(new_state)
self._state = new_state
+self._log_notifier.notify_all()
+self._state_notifier.notify_all()
+
+self._log_notifier.release()
+self._state_notifier.release()
+
+ def write_log(self, log):
+self._log_notifier.acquire()
+self._logs.append(log)
+self._log_notifier.notify_all()
+self._log_notifier.release()
+
+ def _cleanup(self):
+self._state_notifier.acquire()
+self._log_notifier.acquire()
+
+self._final_log_count = len(self._logs)
+self._final_state_count = len(self._state_changes)
+
+self._state_notifier.notify_all()
+self._log_notifier.notify_all()
+
+self._log_notifier.release()
+self._state_notifier.release()
+
def run(self):
-with JobLogHandler(self._log_callbacks):
+with JobLogHandler(self):
try:
fn_api_runner.FnApiRunner().run_via_runner_api(self._pipeline_proto)
logging.info('Successfully completed job.')
self.state = beam_job_api_pb2.JobState.DONE
except: # pylint: disable=bare-except
logging.exception('Error running pipeline.')
-traceback.print_exc()
+logging.exception(traceback.format_exc())
self.state = beam_job_api_pb2.JobState.FAILED
raise
+ finally:
+# In order for consumers to read all messages, this must be the final
+# instruction after a terminal state.
+self._cleanup()
def cancel(self):
if self.state not in TERMINAL_STATES:
self.state = beam_job_api_pb2.JobState.CANCELLING
# TODO(robertwb): Actually cancel...
self.state = beam_job_api_pb2.JobState.CANCELLED
+ self._cleanup()
+
+ def _stream_array(self, array, notifier, final_count):
+"""Yields all elements in array until array length reaches final_count.
+
+This method streams all elements in array. It uses the notifier to wait
+until new messages are received. The streams ends when this method emits
+up to final_count number of elements.
+"""
+index = 0
+
+# Pull all changes until the job finishes.
+while index != self.__dict__[final_count]:
+ notifier.acquire()
+ notifier.wait(5)
+ while index < len(array):
+yield array[index]
+index += 1
+ notifier.release()
+
+ def GetStateStream(self):
Review comment:
Sounds good, changed to PEP 8
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 184389)
[jira] [Work logged] (BEAM-6280) Failure in PortableRunnerTest.test_error_traceback_includes_user_code
[
https://issues.apache.org/jira/browse/BEAM-6280?focusedWorklogId=184384=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184384
]
ASF GitHub Bot logged work on BEAM-6280:
Author: ASF GitHub Bot
Created on: 11/Jan/19 22:46
Start Date: 11/Jan/19 22:46
Worklog Time Spent: 10m
Work Description: rohdesamuel commented on pull request #7433:
[BEAM-6280] Refactors Python portability tests to be multi-threaded aware
URL: https://github.com/apache/beam/pull/7433#discussion_r247276611
##
File path: sdks/python/apache_beam/runners/portability/local_job_service.py
##
@@ -185,55 +152,89 @@ class BeamJob(threading.Thread):
The current state of the pipeline is available as self.state.
"""
- def __init__(self,
- job_id,
- pipeline_options,
- pipeline_proto):
+ def __init__(self, pipeline_proto):
super(BeamJob, self).__init__()
-self._job_id = job_id
-self._pipeline_options = pipeline_options
self._pipeline_proto = pipeline_proto
self._state = None
-self._state_change_callbacks = []
-self._last_log_message = None
-self._log_callbacks = [lambda msg: setattr(self, '_last_log_message', msg)]
+self._logs = []
Review comment:
Gotcha, I was under the impression that this was a testing only class. I
understand how troublesome long running jobs can be on memory usage. I changed
it the semantics back to the queue usage and moved the Get(State|Message)Stream
to before the job is run.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 184384)
Time Spent: 3h 40m (was: 3.5h)
> Failure in PortableRunnerTest.test_error_traceback_includes_user_code
> -
>
> Key: BEAM-6280
> URL: https://issues.apache.org/jira/browse/BEAM-6280
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Kenneth Knowles
>Assignee: Sam Rohde
>Priority: Critical
> Labels: flaky-test
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PreCommit_Python_Cron/732/]
> [https://builds.apache.org/job/beam_PreCommit_Python_Cron/732/testReport/apache_beam.runners.portability.portable_runner_test/PortableRunnerTest/test_error_traceback_includes_user_code/]
> [https://scans.gradle.com/s/do3hjulee3gaa/console-log?task=:beam-sdks-python:testPython3]
> {code:java}
> 'second' not found in 'Traceback (most recent call last):\n File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py",
> line 466, in test_error_traceback_includes_user_code\np |
> beam.Create([0]) | beam.Map(first) # pylint:
> disable=expression-not-assigned\n File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/apache_beam/pipeline.py",
> line 425, in __exit__\nself.run().wait_until_finish()\n File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/apache_beam/runners/portability/portable_runner.py",
> line 314, in wait_until_finish\nself._job_id, self._state,
> self._last_error_message()))\nRuntimeError: Pipeline
> job-cdcefe6d-1caa-4487-9e63-e971f67ec68c failed in state FAILED: start
> coder=WindowedValueCoder[BytesCoder], len(consumers)=1]]>\n'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6280) Failure in PortableRunnerTest.test_error_traceback_includes_user_code
[
https://issues.apache.org/jira/browse/BEAM-6280?focusedWorklogId=184388=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184388
]
ASF GitHub Bot logged work on BEAM-6280:
Author: ASF GitHub Bot
Created on: 11/Jan/19 22:46
Start Date: 11/Jan/19 22:46
Worklog Time Spent: 10m
Work Description: rohdesamuel commented on pull request #7433:
[BEAM-6280] Refactors Python portability tests to be multi-threaded aware
URL: https://github.com/apache/beam/pull/7433#discussion_r247276707
##
File path: sdks/python/apache_beam/runners/portability/local_job_service.py
##
@@ -185,55 +152,129 @@ class BeamJob(threading.Thread):
The current state of the pipeline is available as self.state.
"""
- def __init__(self,
- job_id,
- pipeline_options,
- pipeline_proto):
+ def __init__(self, pipeline_proto):
super(BeamJob, self).__init__()
-self._job_id = job_id
-self._pipeline_options = pipeline_options
self._pipeline_proto = pipeline_proto
self._state = None
-self._state_change_callbacks = []
-self._last_log_message = None
-self._log_callbacks = [lambda msg: setattr(self, '_last_log_message', msg)]
+self._logs = []
+self._final_log_count = -1
+self._state_changes = []
+self._final_state_count = -1
+self._state_notifier = threading.Condition()
+self._log_notifier = threading.Condition()
self.state = beam_job_api_pb2.JobState.STARTING
-self.daemon = True
-
- def add_state_change_callback(self, f):
-self._state_change_callbacks.append(f)
-f(self.state)
-
- def add_log_callback(self, f):
-self._log_callbacks.append(f)
@property
def state(self):
return self._state
@state.setter
def state(self, new_state):
-for state_change_callback in self._state_change_callbacks:
- state_change_callback(new_state)
+"""Sets the job state.
+
+This will inform GetStateStream and GetMessageStream of the new state.
+"""
+
+self._state_notifier.acquire()
+self._log_notifier.acquire()
+
+self._state_changes.append(new_state)
self._state = new_state
+self._log_notifier.notify_all()
+self._state_notifier.notify_all()
+
+self._log_notifier.release()
+self._state_notifier.release()
+
+ def write_log(self, log):
+self._log_notifier.acquire()
+self._logs.append(log)
+self._log_notifier.notify_all()
+self._log_notifier.release()
+
+ def _cleanup(self):
+self._state_notifier.acquire()
+self._log_notifier.acquire()
+
+self._final_log_count = len(self._logs)
+self._final_state_count = len(self._state_changes)
+
+self._state_notifier.notify_all()
+self._log_notifier.notify_all()
+
+self._log_notifier.release()
+self._state_notifier.release()
+
def run(self):
-with JobLogHandler(self._log_callbacks):
+with JobLogHandler(self):
try:
fn_api_runner.FnApiRunner().run_via_runner_api(self._pipeline_proto)
logging.info('Successfully completed job.')
self.state = beam_job_api_pb2.JobState.DONE
except: # pylint: disable=bare-except
logging.exception('Error running pipeline.')
-traceback.print_exc()
+logging.exception(traceback.format_exc())
self.state = beam_job_api_pb2.JobState.FAILED
raise
+ finally:
+# In order for consumers to read all messages, this must be the final
+# instruction after a terminal state.
+self._cleanup()
def cancel(self):
if self.state not in TERMINAL_STATES:
self.state = beam_job_api_pb2.JobState.CANCELLING
# TODO(robertwb): Actually cancel...
self.state = beam_job_api_pb2.JobState.CANCELLED
+ self._cleanup()
+
+ def _stream_array(self, array, notifier, final_count):
+"""Yields all elements in array until array length reaches final_count.
+
+This method streams all elements in array. It uses the notifier to wait
+until new messages are received. The streams ends when this method emits
+up to final_count number of elements.
+"""
+index = 0
+
+# Pull all changes until the job finishes.
+while index != self.__dict__[final_count]:
Review comment:
Agreed, changed impl to queue
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 184388)
Time Spent: 4h (was: 3h 50m)
> Failure in PortableRunnerTest.test_error_traceback_includes_user_code
> -
>
>
[jira] [Work logged] (BEAM-6280) Failure in PortableRunnerTest.test_error_traceback_includes_user_code
[
https://issues.apache.org/jira/browse/BEAM-6280?focusedWorklogId=184385=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184385
]
ASF GitHub Bot logged work on BEAM-6280:
Author: ASF GitHub Bot
Created on: 11/Jan/19 22:46
Start Date: 11/Jan/19 22:46
Worklog Time Spent: 10m
Work Description: rohdesamuel commented on pull request #7433:
[BEAM-6280] Refactors Python portability tests to be multi-threaded aware
URL: https://github.com/apache/beam/pull/7433#discussion_r247276652
##
File path: sdks/python/apache_beam/runners/portability/local_job_service.py
##
@@ -185,55 +152,129 @@ class BeamJob(threading.Thread):
The current state of the pipeline is available as self.state.
"""
- def __init__(self,
- job_id,
- pipeline_options,
- pipeline_proto):
+ def __init__(self, pipeline_proto):
super(BeamJob, self).__init__()
-self._job_id = job_id
-self._pipeline_options = pipeline_options
self._pipeline_proto = pipeline_proto
self._state = None
-self._state_change_callbacks = []
-self._last_log_message = None
-self._log_callbacks = [lambda msg: setattr(self, '_last_log_message', msg)]
+self._logs = []
+self._final_log_count = -1
+self._state_changes = []
+self._final_state_count = -1
+self._state_notifier = threading.Condition()
+self._log_notifier = threading.Condition()
self.state = beam_job_api_pb2.JobState.STARTING
-self.daemon = True
-
- def add_state_change_callback(self, f):
-self._state_change_callbacks.append(f)
-f(self.state)
-
- def add_log_callback(self, f):
-self._log_callbacks.append(f)
@property
def state(self):
return self._state
@state.setter
def state(self, new_state):
-for state_change_callback in self._state_change_callbacks:
- state_change_callback(new_state)
+"""Sets the job state.
+
+This will inform GetStateStream and GetMessageStream of the new state.
+"""
+
+self._state_notifier.acquire()
Review comment:
Changed impl to queue
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 184385)
Time Spent: 3h 50m (was: 3h 40m)
> Failure in PortableRunnerTest.test_error_traceback_includes_user_code
> -
>
> Key: BEAM-6280
> URL: https://issues.apache.org/jira/browse/BEAM-6280
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Kenneth Knowles
>Assignee: Sam Rohde
>Priority: Critical
> Labels: flaky-test
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PreCommit_Python_Cron/732/]
> [https://builds.apache.org/job/beam_PreCommit_Python_Cron/732/testReport/apache_beam.runners.portability.portable_runner_test/PortableRunnerTest/test_error_traceback_includes_user_code/]
> [https://scans.gradle.com/s/do3hjulee3gaa/console-log?task=:beam-sdks-python:testPython3]
> {code:java}
> 'second' not found in 'Traceback (most recent call last):\n File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py",
> line 466, in test_error_traceback_includes_user_code\np |
> beam.Create([0]) | beam.Map(first) # pylint:
> disable=expression-not-assigned\n File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/apache_beam/pipeline.py",
> line 425, in __exit__\nself.run().wait_until_finish()\n File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/apache_beam/runners/portability/portable_runner.py",
> line 314, in wait_until_finish\nself._job_id, self._state,
> self._last_error_message()))\nRuntimeError: Pipeline
> job-cdcefe6d-1caa-4487-9e63-e971f67ec68c failed in state FAILED: start
> coder=WindowedValueCoder[BytesCoder], len(consumers)=1]]>\n'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6405) Improve PortableValidatesRunner test reliability on Jenkins
[ https://issues.apache.org/jira/browse/BEAM-6405?focusedWorklogId=184379=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184379 ] ASF GitHub Bot logged work on BEAM-6405: Author: ASF GitHub Bot Created on: 11/Jan/19 22:33 Start Date: 11/Jan/19 22:33 Worklog Time Spent: 10m Work Description: mxm commented on issue #7461: [BEAM-6405] Let PortableValidatesRunner tests run in EMBEDDED environment URL: https://github.com/apache/beam/pull/7461#issuecomment-453678425 Run Java Flink PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184379) Time Spent: 3.5h (was: 3h 20m) > Improve PortableValidatesRunner test reliability on Jenkins > --- > > Key: BEAM-6405 > URL: https://issues.apache.org/jira/browse/BEAM-6405 > Project: Beam > Issue Type: Test > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > The PVR tests seem to be passing fine and then failing consecutively for no > reason: https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/ > It looks like the outrageous parallelism, i.e. number of available cores, is > responsible for the flakiness if there is additional load on the build > slaves. We should lower the parallelism. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6405) Improve PortableValidatesRunner test reliability on Jenkins
[ https://issues.apache.org/jira/browse/BEAM-6405?focusedWorklogId=184378=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184378 ] ASF GitHub Bot logged work on BEAM-6405: Author: ASF GitHub Bot Created on: 11/Jan/19 22:33 Start Date: 11/Jan/19 22:33 Worklog Time Spent: 10m Work Description: mxm commented on issue #7461: [BEAM-6405] Let PortableValidatesRunner tests run in EMBEDDED environment URL: https://github.com/apache/beam/pull/7461#issuecomment-453678396 Run Java Flink PortableValidatesRunner Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184378) Time Spent: 3h 20m (was: 3h 10m) > Improve PortableValidatesRunner test reliability on Jenkins > --- > > Key: BEAM-6405 > URL: https://issues.apache.org/jira/browse/BEAM-6405 > Project: Beam > Issue Type: Test > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > The PVR tests seem to be passing fine and then failing consecutively for no > reason: https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/ > It looks like the outrageous parallelism, i.e. number of available cores, is > responsible for the flakiness if there is additional load on the build > slaves. We should lower the parallelism. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6231) Triage test failures introduced by use_executable_stage_bundle_execution
[ https://issues.apache.org/jira/browse/BEAM-6231?focusedWorklogId=184374=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184374 ] ASF GitHub Bot logged work on BEAM-6231: Author: ASF GitHub Bot Created on: 11/Jan/19 22:25 Start Date: 11/Jan/19 22:25 Worklog Time Spent: 10m Work Description: boyuanzz commented on issue #7356: [BEAM-6231] Make Dataflow runner harness work with FixedWindow URL: https://github.com/apache/beam/pull/7356#issuecomment-453676603 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184374) Time Spent: 6h 10m (was: 6h) > Triage test failures introduced by use_executable_stage_bundle_execution > > > Key: BEAM-6231 > URL: https://issues.apache.org/jira/browse/BEAM-6231 > Project: Beam > Issue Type: Test > Components: runner-dataflow >Reporter: Boyuan Zhang >Assignee: Boyuan Zhang >Priority: Major > Time Spent: 6h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4606) Upper bound for pytz dependency
[ https://issues.apache.org/jira/browse/BEAM-4606?focusedWorklogId=184373=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184373 ] ASF GitHub Bot logged work on BEAM-4606: Author: ASF GitHub Bot Created on: 11/Jan/19 22:15 Start Date: 11/Jan/19 22:15 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #7487: [BEAM-4606] Remove the upper bound for pytz URL: https://github.com/apache/beam/pull/7487 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184373) Time Spent: 2h 10m (was: 2h) > Upper bound for pytz dependency > --- > > Key: BEAM-4606 > URL: https://issues.apache.org/jira/browse/BEAM-4606 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Ahmet Altay >Assignee: Udi Meiri >Priority: Major > Fix For: Not applicable > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Do we need an upper bound for the pytz dependency? > ([https://github.com/apache/beam/blob/release-2.5.0/sdks/python/setup.py#L108)] > We typically have upper bounds, in order to avoid future breakages due to a > possibility of breaking/backward incompatible change of that depepdency. > Good practice is to upper bound either at known version, or next major > version. Do we need an exception for pytz because it does not seem to be > following semantic versioning? > cc: [~yifanzou] Is this something dependency notifier can warn on? Dependency > without upper version bounds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5650) Timeout exceptions while reading a lot of files from a bounded source like S3 with Flink runner
[
https://issues.apache.org/jira/browse/BEAM-5650?focusedWorklogId=184372=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184372
]
ASF GitHub Bot logged work on BEAM-5650:
Author: ASF GitHub Bot
Created on: 11/Jan/19 22:12
Start Date: 11/Jan/19 22:12
Worklog Time Spent: 10m
Work Description: jhalaria commented on issue #6952: [BEAM-5650]: Modify
BoundedToUnboundedSourceAdapter to configure its reader to read more than 1
bounded source
URL: https://github.com/apache/beam/pull/6952#issuecomment-453673695
@robertwb - I was out on vacation. Haven't had a chance to resume work on it
yet. Will let you know once I have something.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 184372)
Time Spent: 2h 20m (was: 2h 10m)
> Timeout exceptions while reading a lot of files from a bounded source like S3
> with Flink runner
> ---
>
> Key: BEAM-5650
> URL: https://issues.apache.org/jira/browse/BEAM-5650
> Project: Beam
> Issue Type: Bug
> Components: runner-core, runner-flink
>Reporter: Ankit Jhalaria
>Assignee: Ankit Jhalaria
>Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> * Using TextIO, I was trying to read around 850 files.
> * Getting this exception while using FlinkRunner
>
> {code:java}
> //Caused by: org.apache.flink.runtime.client.JobExecutionException:
> java.io.IOException: com.amazonaws.SdkClientException: Unable to execute HTTP
> request: Timeout waiting for connection from pool at
> org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:625)
> at
> org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:123)
> at
> org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.executePipeline(FlinkPipelineExecutionEnvironment.java:175)
> at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:115) ...
> 28 more Caused by: java.io.IOException: com.amazonaws.SdkClientException:
> Unable to execute HTTP request: Timeout waiting for connection from pool at
> org.apache.beam.sdk.io.aws.s3.S3ReadableSeekableByteChannel.read(S3ReadableSeekableByteChannel.java:91)
> at
> org.apache.beam.sdk.io.CompressedSource$CompressedReader$CountingChannel.read(CompressedSource.java:382)
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65) at
> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) at
> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) at
> java.io.FilterInputStream.read(FilterInputStream.java:133) at
> java.io.PushbackInputStream.read(PushbackInputStream.java:186) at
> org.apache.beam.repackaged.beam_sdks_java_core.com.google.common.io.ByteStreams.read(ByteStreams.java:859)
> at
> org.apache.beam.sdk.io.Compression$3.readDecompressed(Compression.java:81) at
> org.apache.beam.sdk.io.CompressedSource$CompressionMode.createDecompressingChannel(CompressedSource.java:110)
> at
> org.apache.beam.sdk.io.CompressedSource$CompressedReader.startReading(CompressedSource.java:417)
> at
> org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.startImpl(FileBasedSource.java:476)
> at
> org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.start(OffsetBasedSource.java:249)
> at
> org.apache.beam.runners.core.construction.UnboundedReadFromBoundedSource$BoundedToUnboundedSourceAdapter$ResidualSource.advance(UnboundedReadFromBoundedSource.java:456)
> at
> org.apache.beam.runners.core.construction.UnboundedReadFromBoundedSource$BoundedToUnboundedSourceAdapter$ResidualSource.access$300(UnboundedReadFromBoundedSource.java:434)
> at
> org.apache.beam.runners.core.construction.UnboundedReadFromBoundedSource$BoundedToUnboundedSourceAdapter$Reader.advance(UnboundedReadFromBoundedSource.java:286)
> at
> org.apache.beam.runners.core.construction.UnboundedReadFromBoundedSource$BoundedToUnboundedSourceAdapter$Reader.start(UnboundedReadFromBoundedSource.java:279)
> at
> org.apache.beam.runners.flink.metrics.ReaderInvocationUtil.invokeStart(ReaderInvocationUtil.java:51)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:250)
> at
> org.apache.beam.runners.flink.FlinkStreamingTransformTranslators$UnboundedSourceWrapperNoValueWithRecordId.run(FlinkStreamingTransformTranslators.java:1299)
> at
>
[jira] [Created] (BEAM-6417) Option to not fork DirectRunner
Thomas Deegan created BEAM-6417: --- Summary: Option to not fork DirectRunner Key: BEAM-6417 URL: https://issues.apache.org/jira/browse/BEAM-6417 Project: Beam Issue Type: Improvement Components: beam-model Reporter: Thomas Deegan Assignee: Kenneth Knowles Hi. When using a python debugger, it would be nice to be able to debug ops as they are run locally by --runner DirectRunner however it seems that Beam forks the process to actually run the ops. Unfortuatnely most python debuggers (mine is pydevd) don't support following forks. It it possible to add a flag to DirectRunner to keep the DIrectRunner in-process? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6416) Enable local filesystem IO in ULR
Daniel Oliveira created BEAM-6416: - Summary: Enable local filesystem IO in ULR Key: BEAM-6416 URL: https://issues.apache.org/jira/browse/BEAM-6416 Project: Beam Issue Type: Improvement Components: runner-direct Reporter: Daniel Oliveira Assignee: Daniel Oliveira Figure out how to enable local filesystem IO with the ULR. The ULR currently can't do local filesystem IO due to using portability with a dockerized SDK. Having the SDK harness in Docker means that local files are unable to be accessed, even if the pipeline is running locally. This bug is to figure out a way to enable local files to be used in the ULR. (Meta note: This bug is only for the ULR, I'll probably create another bug for this as a general portability feature later) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-6354) Hanging BoundedReadFromUnboundedSourceTest#testTimeBound and SplittableDoFnTest#testLateData
[ https://issues.apache.org/jira/browse/BEAM-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-6354: -- Fix Version/s: 2.10.0 > Hanging BoundedReadFromUnboundedSourceTest#testTimeBound and > SplittableDoFnTest#testLateData > > > Key: BEAM-6354 > URL: https://issues.apache.org/jira/browse/BEAM-6354 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Gleb Kanterov >Priority: Major > Fix For: 2.10.0 > > > It seems that they have a similar root cause because both of them use > unbounded streams. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6405) Improve PortableValidatesRunner test reliability on Jenkins
[ https://issues.apache.org/jira/browse/BEAM-6405?focusedWorklogId=184363=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184363 ] ASF GitHub Bot logged work on BEAM-6405: Author: ASF GitHub Bot Created on: 11/Jan/19 21:47 Start Date: 11/Jan/19 21:47 Worklog Time Spent: 10m Work Description: mxm commented on issue #7461: [BEAM-6405] Let PortableValidatesRunner tests run in EMBEDDED environment URL: https://github.com/apache/beam/pull/7461#issuecomment-453667482 Had to fix the `PipelineOptions` not being passed on correctly for the `EmbeddedEnvironmentFactory`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184363) Time Spent: 2h 50m (was: 2h 40m) > Improve PortableValidatesRunner test reliability on Jenkins > --- > > Key: BEAM-6405 > URL: https://issues.apache.org/jira/browse/BEAM-6405 > Project: Beam > Issue Type: Test > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > The PVR tests seem to be passing fine and then failing consecutively for no > reason: https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/ > It looks like the outrageous parallelism, i.e. number of available cores, is > responsible for the flakiness if there is additional load on the build > slaves. We should lower the parallelism. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6231) Triage test failures introduced by use_executable_stage_bundle_execution
[ https://issues.apache.org/jira/browse/BEAM-6231?focusedWorklogId=184366=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184366 ] ASF GitHub Bot logged work on BEAM-6231: Author: ASF GitHub Bot Created on: 11/Jan/19 21:52 Start Date: 11/Jan/19 21:52 Worklog Time Spent: 10m Work Description: boyuanzz commented on issue #7356: [BEAM-6231] Make Dataflow runner harness work with FixedWindow URL: https://github.com/apache/beam/pull/7356#issuecomment-453668839 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184366) Time Spent: 6h (was: 5h 50m) > Triage test failures introduced by use_executable_stage_bundle_execution > > > Key: BEAM-6231 > URL: https://issues.apache.org/jira/browse/BEAM-6231 > Project: Beam > Issue Type: Test > Components: runner-dataflow >Reporter: Boyuan Zhang >Assignee: Boyuan Zhang >Priority: Major > Time Spent: 6h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6405) Improve PortableValidatesRunner test reliability on Jenkins
[ https://issues.apache.org/jira/browse/BEAM-6405?focusedWorklogId=184365=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184365 ] ASF GitHub Bot logged work on BEAM-6405: Author: ASF GitHub Bot Created on: 11/Jan/19 21:48 Start Date: 11/Jan/19 21:48 Worklog Time Spent: 10m Work Description: mxm commented on issue #7461: [BEAM-6405] Let PortableValidatesRunner tests run in EMBEDDED environment URL: https://github.com/apache/beam/pull/7461#issuecomment-453667692 Run Java Flink PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184365) Time Spent: 3h 10m (was: 3h) > Improve PortableValidatesRunner test reliability on Jenkins > --- > > Key: BEAM-6405 > URL: https://issues.apache.org/jira/browse/BEAM-6405 > Project: Beam > Issue Type: Test > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > The PVR tests seem to be passing fine and then failing consecutively for no > reason: https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/ > It looks like the outrageous parallelism, i.e. number of available cores, is > responsible for the flakiness if there is additional load on the build > slaves. We should lower the parallelism. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6405) Improve PortableValidatesRunner test reliability on Jenkins
[ https://issues.apache.org/jira/browse/BEAM-6405?focusedWorklogId=184364=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184364 ] ASF GitHub Bot logged work on BEAM-6405: Author: ASF GitHub Bot Created on: 11/Jan/19 21:48 Start Date: 11/Jan/19 21:48 Worklog Time Spent: 10m Work Description: mxm commented on issue #7461: [BEAM-6405] Let PortableValidatesRunner tests run in EMBEDDED environment URL: https://github.com/apache/beam/pull/7461#issuecomment-453667661 Run Java Flink PortableValidatesRunner Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184364) Time Spent: 3h (was: 2h 50m) > Improve PortableValidatesRunner test reliability on Jenkins > --- > > Key: BEAM-6405 > URL: https://issues.apache.org/jira/browse/BEAM-6405 > Project: Beam > Issue Type: Test > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > The PVR tests seem to be passing fine and then failing consecutively for no > reason: https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/ > It looks like the outrageous parallelism, i.e. number of available cores, is > responsible for the flakiness if there is additional load on the build > slaves. We should lower the parallelism. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6354) Hanging BoundedReadFromUnboundedSourceTest#testTimeBound and SplittableDoFnTest#testLateData
[ https://issues.apache.org/jira/browse/BEAM-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740820#comment-16740820 ] Kenneth Knowles commented on BEAM-6354: --- OK. You discovered by un-ignoring them? I will put this on release blocking and take a look before I move ahead with the RC. > Hanging BoundedReadFromUnboundedSourceTest#testTimeBound and > SplittableDoFnTest#testLateData > > > Key: BEAM-6354 > URL: https://issues.apache.org/jira/browse/BEAM-6354 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Gleb Kanterov >Priority: Major > > It seems that they have a similar root cause because both of them use > unbounded streams. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6280) Failure in PortableRunnerTest.test_error_traceback_includes_user_code
[
https://issues.apache.org/jira/browse/BEAM-6280?focusedWorklogId=184355=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184355
]
ASF GitHub Bot logged work on BEAM-6280:
Author: ASF GitHub Bot
Created on: 11/Jan/19 21:29
Start Date: 11/Jan/19 21:29
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #7433: [BEAM-6280]
Refactors Python portability tests to be multi-threaded aware
URL: https://github.com/apache/beam/pull/7433#discussion_r247259140
##
File path: sdks/python/apache_beam/runners/portability/local_job_service.py
##
@@ -185,55 +152,129 @@ class BeamJob(threading.Thread):
The current state of the pipeline is available as self.state.
"""
- def __init__(self,
- job_id,
- pipeline_options,
- pipeline_proto):
+ def __init__(self, pipeline_proto):
super(BeamJob, self).__init__()
-self._job_id = job_id
-self._pipeline_options = pipeline_options
self._pipeline_proto = pipeline_proto
self._state = None
-self._state_change_callbacks = []
-self._last_log_message = None
-self._log_callbacks = [lambda msg: setattr(self, '_last_log_message', msg)]
+self._logs = []
+self._final_log_count = -1
+self._state_changes = []
+self._final_state_count = -1
+self._state_notifier = threading.Condition()
+self._log_notifier = threading.Condition()
self.state = beam_job_api_pb2.JobState.STARTING
-self.daemon = True
-
- def add_state_change_callback(self, f):
-self._state_change_callbacks.append(f)
-f(self.state)
-
- def add_log_callback(self, f):
-self._log_callbacks.append(f)
@property
def state(self):
return self._state
@state.setter
def state(self, new_state):
-for state_change_callback in self._state_change_callbacks:
- state_change_callback(new_state)
+"""Sets the job state.
+
+This will inform GetStateStream and GetMessageStream of the new state.
+"""
+
+self._state_notifier.acquire()
Review comment:
I think blocking queue will better fit here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 184355)
Time Spent: 3.5h (was: 3h 20m)
> Failure in PortableRunnerTest.test_error_traceback_includes_user_code
> -
>
> Key: BEAM-6280
> URL: https://issues.apache.org/jira/browse/BEAM-6280
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Kenneth Knowles
>Assignee: Sam Rohde
>Priority: Critical
> Labels: flaky-test
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PreCommit_Python_Cron/732/]
> [https://builds.apache.org/job/beam_PreCommit_Python_Cron/732/testReport/apache_beam.runners.portability.portable_runner_test/PortableRunnerTest/test_error_traceback_includes_user_code/]
> [https://scans.gradle.com/s/do3hjulee3gaa/console-log?task=:beam-sdks-python:testPython3]
> {code:java}
> 'second' not found in 'Traceback (most recent call last):\n File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py",
> line 466, in test_error_traceback_includes_user_code\np |
> beam.Create([0]) | beam.Map(first) # pylint:
> disable=expression-not-assigned\n File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/apache_beam/pipeline.py",
> line 425, in __exit__\nself.run().wait_until_finish()\n File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/apache_beam/runners/portability/portable_runner.py",
> line 314, in wait_until_finish\nself._job_id, self._state,
> self._last_error_message()))\nRuntimeError: Pipeline
> job-cdcefe6d-1caa-4487-9e63-e971f67ec68c failed in state FAILED: start
> coder=WindowedValueCoder[BytesCoder], len(consumers)=1]]>\n'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6184) PortableRunner dependency missed in wordcount example maven artifact
[ https://issues.apache.org/jira/browse/BEAM-6184?focusedWorklogId=184350=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184350 ] ASF GitHub Bot logged work on BEAM-6184: Author: ASF GitHub Bot Created on: 11/Jan/19 21:05 Start Date: 11/Jan/19 21:05 Worklog Time Spent: 10m Work Description: HuangLED commented on pull request #7454: [BEAM-6184] Enforce javadoc check on non-trivial public methods URL: https://github.com/apache/beam/pull/7454#discussion_r247253184 ## File path: sdks/java/build-tools/src/main/resources/beam/suppressions.xml ## @@ -21,6 +21,11 @@ + + + Review comment: Good point. sending an update This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184350) Time Spent: 2h 20m (was: 2h 10m) > PortableRunner dependency missed in wordcount example maven artifact > > > Key: BEAM-6184 > URL: https://issues.apache.org/jira/browse/BEAM-6184 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Ruoyun Huang >Assignee: Ruoyun Huang >Priority: Minor > Fix For: Not applicable > > Time Spent: 2h 20m > Remaining Estimate: 0h > > > > more context: > https://lists.apache.org/thread.html/8dd60395424425f7502d62888c49014430d1d3b06c026606f3db28ab@%3Cuser.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6184) PortableRunner dependency missed in wordcount example maven artifact
[
https://issues.apache.org/jira/browse/BEAM-6184?focusedWorklogId=184347=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184347
]
ASF GitHub Bot logged work on BEAM-6184:
Author: ASF GitHub Bot
Created on: 11/Jan/19 21:04
Start Date: 11/Jan/19 21:04
Worklog Time Spent: 10m
Work Description: HuangLED commented on pull request #7454: [BEAM-6184]
Enforce javadoc check on non-trivial public methods
URL: https://github.com/apache/beam/pull/7454#discussion_r247253024
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/annotations/Experimental.java
##
@@ -39,6 +39,7 @@
ElementType.TYPE
})
@Documented
+@SuppressWarnings("checkstyle:JavadocMethod")
Review comment:
Ok. let me remove this one. I was misled by the @Documented annotation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 184347)
Time Spent: 2h (was: 1h 50m)
> PortableRunner dependency missed in wordcount example maven artifact
>
>
> Key: BEAM-6184
> URL: https://issues.apache.org/jira/browse/BEAM-6184
> Project: Beam
> Issue Type: Improvement
> Components: build-system
>Reporter: Ruoyun Huang
>Assignee: Ruoyun Huang
>Priority: Minor
> Fix For: Not applicable
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
>
>
> more context:
> https://lists.apache.org/thread.html/8dd60395424425f7502d62888c49014430d1d3b06c026606f3db28ab@%3Cuser.beam.apache.org%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6184) PortableRunner dependency missed in wordcount example maven artifact
[
https://issues.apache.org/jira/browse/BEAM-6184?focusedWorklogId=184349=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184349
]
ASF GitHub Bot logged work on BEAM-6184:
Author: ASF GitHub Bot
Created on: 11/Jan/19 21:05
Start Date: 11/Jan/19 21:05
Worklog Time Spent: 10m
Work Description: HuangLED commented on pull request #7454: [BEAM-6184]
Enforce javadoc check on non-trivial public methods
URL: https://github.com/apache/beam/pull/7454#discussion_r247253024
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/annotations/Experimental.java
##
@@ -39,6 +39,7 @@
ElementType.TYPE
})
@Documented
+@SuppressWarnings("checkstyle:JavadocMethod")
Review comment:
Ok. let me remove this one. :-) I was misled by the @Documented annotation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 184349)
Time Spent: 2h 10m (was: 2h)
> PortableRunner dependency missed in wordcount example maven artifact
>
>
> Key: BEAM-6184
> URL: https://issues.apache.org/jira/browse/BEAM-6184
> Project: Beam
> Issue Type: Improvement
> Components: build-system
>Reporter: Ruoyun Huang
>Assignee: Ruoyun Huang
>Priority: Minor
> Fix For: Not applicable
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
>
>
> more context:
> https://lists.apache.org/thread.html/8dd60395424425f7502d62888c49014430d1d3b06c026606f3db28ab@%3Cuser.beam.apache.org%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6184) PortableRunner dependency missed in wordcount example maven artifact
[
https://issues.apache.org/jira/browse/BEAM-6184?focusedWorklogId=184344=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184344
]
ASF GitHub Bot logged work on BEAM-6184:
Author: ASF GitHub Bot
Created on: 11/Jan/19 21:03
Start Date: 11/Jan/19 21:03
Worklog Time Spent: 10m
Work Description: HuangLED commented on pull request #7454: [BEAM-6184]
Enforce javadoc check on non-trivial public methods
URL: https://github.com/apache/beam/pull/7454#discussion_r247252770
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java
##
@@ -331,10 +331,8 @@ private void fireTimer(
public void close() throws Exception {
// close may be called multiple times when an exception is thrown
if (stageContext != null) {
- try (@SuppressWarnings("unused")
Review comment:
In this particular place, these @SuppressWarnings were removed because no
longer
needed (that is, without them, none of our tool gives complains).
SuppressWarnings is something that less is always better, thus I took the
liberty removing them. :-)
One side note, and also to @kennknowles 's question: Yes, it should be
either by javac or findbugs. checkstyle does not cover unused. The fact is ,
check/warning on unused-local-variable is completely turned off everywhere in
our code base.
I tested and verified by intentionally adding ones but no tool complains on
that, thus I assume it is intended behavior. If it is not expected (is that
so?), that deserves further investigation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 184344)
Time Spent: 1h 50m (was: 1h 40m)
> PortableRunner dependency missed in wordcount example maven artifact
>
>
> Key: BEAM-6184
> URL: https://issues.apache.org/jira/browse/BEAM-6184
> Project: Beam
> Issue Type: Improvement
> Components: build-system
>Reporter: Ruoyun Huang
>Assignee: Ruoyun Huang
>Priority: Minor
> Fix For: Not applicable
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
>
>
> more context:
> https://lists.apache.org/thread.html/8dd60395424425f7502d62888c49014430d1d3b06c026606f3db28ab@%3Cuser.beam.apache.org%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-6413) Document multiple committers per component on the Wiki
Kenneth Knowles created BEAM-6413: - Summary: Document multiple committers per component on the Wiki Key: BEAM-6413 URL: https://issues.apache.org/jira/browse/BEAM-6413 Project: Beam Issue Type: Improvement Components: project-management Reporter: Kenneth Knowles Assignee: Kenneth Knowles -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6414) Automatically unassign bugs that aren't getting attention
Kenneth Knowles created BEAM-6414: - Summary: Automatically unassign bugs that aren't getting attention Key: BEAM-6414 URL: https://issues.apache.org/jira/browse/BEAM-6414 Project: Beam Issue Type: Improvement Components: project-management Reporter: Kenneth Knowles Assignee: Kenneth Knowles -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6415) Add SLOs to bug priorities, reports, and pings
Kenneth Knowles created BEAM-6415: - Summary: Add SLOs to bug priorities, reports, and pings Key: BEAM-6415 URL: https://issues.apache.org/jira/browse/BEAM-6415 Project: Beam Issue Type: Improvement Components: project-management Reporter: Kenneth Knowles Assignee: Kenneth Knowles -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6410) Add Needs Review status
Kenneth Knowles created BEAM-6410: - Summary: Add Needs Review status Key: BEAM-6410 URL: https://issues.apache.org/jira/browse/BEAM-6410 Project: Beam Issue Type: Improvement Components: project-management Reporter: Kenneth Knowles Assignee: Kenneth Knowles -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6412) Unassign bugs from folks with >30
Kenneth Knowles created BEAM-6412: - Summary: Unassign bugs from folks with >30 Key: BEAM-6412 URL: https://issues.apache.org/jira/browse/BEAM-6412 Project: Beam Issue Type: Improvement Components: project-management Reporter: Kenneth Knowles Assignee: Kenneth Knowles -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6411) Start tickets in Needs Review, unassigned
Kenneth Knowles created BEAM-6411: - Summary: Start tickets in Needs Review, unassigned Key: BEAM-6411 URL: https://issues.apache.org/jira/browse/BEAM-6411 Project: Beam Issue Type: Improvement Components: project-management Reporter: Kenneth Knowles Assignee: Kenneth Knowles -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6409) beam_PostCommit_Java_PortabilityApi fails
Ahmet Altay created BEAM-6409: - Summary: beam_PostCommit_Java_PortabilityApi fails Key: BEAM-6409 URL: https://issues.apache.org/jira/browse/BEAM-6409 Project: Beam Issue Type: Bug Components: test-failures Reporter: Ahmet Altay Assignee: Scott Wegner [https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/660/console#gradle-task-153] *10:14:29* > *Task :beam-runners-google-cloud-dataflow-java:googleCloudPlatformFnApiWorkerIntegrationTest* *10:26:18* *10:26:18* org.apache.beam.sdk.io.gcp.bigquery.BigQueryToTableIT > testLegacyQueryWithoutReshuffle FAILED*10:26:18* java.lang.RuntimeException at BigQueryToTableIT.java:111*10:43:22* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6184) PortableRunner dependency missed in wordcount example maven artifact
[
https://issues.apache.org/jira/browse/BEAM-6184?focusedWorklogId=184335=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184335
]
ASF GitHub Bot logged work on BEAM-6184:
Author: ASF GitHub Bot
Created on: 11/Jan/19 20:30
Start Date: 11/Jan/19 20:30
Worklog Time Spent: 10m
Work Description: kennknowles commented on pull request #7454:
[BEAM-6184] Enforce javadoc check on non-trivial public methods
URL: https://github.com/apache/beam/pull/7454#discussion_r247245058
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/annotations/Experimental.java
##
@@ -39,6 +39,7 @@
ElementType.TYPE
})
@Documented
+@SuppressWarnings("checkstyle:JavadocMethod")
Review comment:
Seems like it should have documentation. I suggest `You can indicate a
category for the experimental feature. This is unused and serves only as a hint
to the reader.`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 184335)
Time Spent: 1.5h (was: 1h 20m)
> PortableRunner dependency missed in wordcount example maven artifact
>
>
> Key: BEAM-6184
> URL: https://issues.apache.org/jira/browse/BEAM-6184
> Project: Beam
> Issue Type: Improvement
> Components: build-system
>Reporter: Ruoyun Huang
>Assignee: Ruoyun Huang
>Priority: Minor
> Fix For: Not applicable
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
>
>
> more context:
> https://lists.apache.org/thread.html/8dd60395424425f7502d62888c49014430d1d3b06c026606f3db28ab@%3Cuser.beam.apache.org%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-6289) Running a join on two Cassandra tables using FlinkRunner fails
[
https://issues.apache.org/jira/browse/BEAM-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740768#comment-16740768
]
Shahar Frank commented on BEAM-6289:
[~mxm] Thanks. I saw it. Put that in my todo list. Will eventually get to it.
I'm working on another issue regarding adding a Wher clause to the CasandraIO
Reader first.
> Running a join on two Cassandra tables using FlinkRunner fails
> --
>
> Key: BEAM-6289
> URL: https://issues.apache.org/jira/browse/BEAM-6289
> Project: Beam
> Issue Type: Bug
> Components: io-java-cassandra, runner-flink
>Affects Versions: 2.8.0, 2.9.0
> Environment: Tested on Ubuntu 18
> Beam 2.8
> Tested with Flink:
> 1) [local]
> 2) Cluster inside a K8S cluster on minikube
> 3) Cluster inside a K8S cluster on GCP
> Tested using Cassandra [cqlsh 5.0.1 | Cassandra 3.11.3 | CQL spec 3.4.4 |
> Native protocol v4]:
> 1) In a local container
> 2) Cluster inside a K8S cluster on minikube
> 3) Cluster inside a K8S cluster on GCP
>Reporter: Shahar Frank
>Assignee: Maximilian Michels
>Priority: Critical
> Labels: FlinkRunner, beam, bug, cassandra, flink, join
> Fix For: Not applicable
>
> Attachments: direct_runner_build.log, flink_runner_build.log
>
>
> Can't make a simple join on two Cassandra tables when using FlinkRunner.
> The same code works with a DirectRunner fails when used with FlinkRunner
> giving these (as well as many other) errors:
> {code:java}
> Caused by: akka.pattern.AskTimeoutException: Ask timed out on
> [Actor[akka://flink/user/dispatchere1f5abe7-6299-43ea-9182-24a2193e078f#-1757043920]]
> after [1 ms]. Sender[null] sent message of type
> "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
> at
> akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
> at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
> at
> scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
> at
> scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
> at
> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
> at
> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
> The code can be found [here|https://github.com/srfrnk/beam-playground]
> Steps to reproduce:
> # Clone the repo to a linux (I;m on Ubuntu 18 but any *nix system would
> probably work - i.e. repl.it)
> # Follow the README to set up a Cassandra container + schema
> # Run with
> {code}
> gradle --console=plain join-from-cassandra -Drunner=flink > output/build.log
> 2>&1{code}
> to use FlinkRunner. See error in log at ./output/build.log
> # Run with
> {code}
> gradle --console=plain join-from-cassandra -Drunner=direct > output/build.log
> 2>&1{code}
> to use DirectRunner. See error in log at ./output/build.log
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5918) Add Cast transform for Rows
[ https://issues.apache.org/jira/browse/BEAM-5918?focusedWorklogId=184339=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184339 ] ASF GitHub Bot logged work on BEAM-5918: Author: ASF GitHub Bot Created on: 11/Jan/19 20:33 Start Date: 11/Jan/19 20:33 Worklog Time Spent: 10m Work Description: kennknowles commented on pull request #7373: [BEAM-5918] Fix casting of non-numeric types URL: https://github.com/apache/beam/pull/7373 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184339) Time Spent: 9h 40m (was: 9.5h) > Add Cast transform for Rows > --- > > Key: BEAM-5918 > URL: https://issues.apache.org/jira/browse/BEAM-5918 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Gleb Kanterov >Assignee: Gleb Kanterov >Priority: Major > Time Spent: 9h 40m > Remaining Estimate: 0h > > There is a need for a generic transform that given two Row schemas will > convert rows between them. There must be a possibility to opt-out from > certain kind of conversions, for instance, converting ints to shorts can > cause overflow. Another example, a schema could have a nullable field, but > never have NULL value in practice, because it was filtered out. > What is needed: > - widening values (e.g., int -> long) > - narrowwing (e.g., int -> short) > - runtime check for overflow while narrowing > - ignoring nullability (nullable=true -> nullable=false) > - weakening nullability (nullable=false -> nullable=true) > - projection (Schema(a: Int32, b: Int32) -> Schema(a: Int32)) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6407) regression: FileIO.writeDynamic() with side inputs fails in DirectRunner
[
https://issues.apache.org/jira/browse/BEAM-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740752#comment-16740752
]
Kenneth Knowles commented on BEAM-6407:
---
Seems like a good candidate for bisection, if the offending PR is not obvious.
> regression: FileIO.writeDynamic() with side inputs fails in DirectRunner
>
>
> Key: BEAM-6407
> URL: https://issues.apache.org/jira/browse/BEAM-6407
> Project: Beam
> Issue Type: Bug
> Components: beam-model
>Affects Versions: 2.9.0
>Reporter: Niel Markwick
>Assignee: Kenneth Knowles
>Priority: Major
> Labels: regression
> Fix For: 2.10.0
>
> Attachments: beam-filewriter-demo.tgz
>
>
> When FileIO.writeDynamic is used with automatic sharding and a Contextful.Fn
> that uses side inputs for the file naming, DirectRunner (and TestPipeline)
> fail with:
> {{java.lang.IllegalStateException: All PCollectionViews that are consumed
> must be written by some WriteView PTransform: Missing [
> [RunnerPCollectionView]]}}
>
> Example code:
> {code:java}
> PCollectionView outputFileName =
> pipeline.apply(
> "outputDir",
> Create.of("/tmp/testout")).apply(View.asSingleton());
> Contextful.Fn manifestNaming =
> (element, c) ->
> (window, pane, numShards, shardIndex, compression) ->
> c.sideInput(outputFileName)+shardIndex;
> pipeline.apply(FileIO.writeDynamic()
> .by(SerializableFunctions.constant(""))
> .withDestinationCoder(StringUtf8Coder.of())
> .via(TextIO.sink())
> .withTempDirectory("/tmp")
> .withNaming(Contextful.of(
> manifestNaming,
> Requirements.requiresSideInputs(outputFileName;
> {code}
>
> This does not occur in Dataflow-runner
> It does not occur if the ContextFul.Fn is not given side inputs.
> It does not occur if withNumShards(1) is set.
> It did not occur in 2.8.0, and does in 2.9.0 and 2.10.0-SNAPSHOT (as of today)
>
> The cause appears to be due to the DirectRunner using TransformOverrides
> re-writing FileIO sinks to use runner-determined-sharding
> ( see [DirectRunner.java line
> 226|https://github.com/apache/beam/blob/master/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java#L226]
> )
> but I do not know why this started occuring in 2.9.0...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-6407) regression: FileIO.writeDynamic() with side inputs fails in DirectRunner
[
https://issues.apache.org/jira/browse/BEAM-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-6407:
--
Fix Version/s: 2.10.0
> regression: FileIO.writeDynamic() with side inputs fails in DirectRunner
>
>
> Key: BEAM-6407
> URL: https://issues.apache.org/jira/browse/BEAM-6407
> Project: Beam
> Issue Type: Bug
> Components: beam-model
>Affects Versions: 2.9.0
>Reporter: Niel Markwick
>Assignee: Kenneth Knowles
>Priority: Major
> Labels: regression
> Fix For: 2.10.0
>
> Attachments: beam-filewriter-demo.tgz
>
>
> When FileIO.writeDynamic is used with automatic sharding and a Contextful.Fn
> that uses side inputs for the file naming, DirectRunner (and TestPipeline)
> fail with:
> {{java.lang.IllegalStateException: All PCollectionViews that are consumed
> must be written by some WriteView PTransform: Missing [
> [RunnerPCollectionView]]}}
>
> Example code:
> {code:java}
> PCollectionView outputFileName =
> pipeline.apply(
> "outputDir",
> Create.of("/tmp/testout")).apply(View.asSingleton());
> Contextful.Fn manifestNaming =
> (element, c) ->
> (window, pane, numShards, shardIndex, compression) ->
> c.sideInput(outputFileName)+shardIndex;
> pipeline.apply(FileIO.writeDynamic()
> .by(SerializableFunctions.constant(""))
> .withDestinationCoder(StringUtf8Coder.of())
> .via(TextIO.sink())
> .withTempDirectory("/tmp")
> .withNaming(Contextful.of(
> manifestNaming,
> Requirements.requiresSideInputs(outputFileName;
> {code}
>
> This does not occur in Dataflow-runner
> It does not occur if the ContextFul.Fn is not given side inputs.
> It does not occur if withNumShards(1) is set.
> It did not occur in 2.8.0, and does in 2.9.0 and 2.10.0-SNAPSHOT (as of today)
>
> The cause appears to be due to the DirectRunner using TransformOverrides
> re-writing FileIO sinks to use runner-determined-sharding
> ( see [DirectRunner.java line
> 226|https://github.com/apache/beam/blob/master/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java#L226]
> )
> but I do not know why this started occuring in 2.9.0...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6184) PortableRunner dependency missed in wordcount example maven artifact
[
https://issues.apache.org/jira/browse/BEAM-6184?focusedWorklogId=184334=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184334
]
ASF GitHub Bot logged work on BEAM-6184:
Author: ASF GitHub Bot
Created on: 11/Jan/19 20:30
Start Date: 11/Jan/19 20:30
Worklog Time Spent: 10m
Work Description: kennknowles commented on pull request #7454:
[BEAM-6184] Enforce javadoc check on non-trivial public methods
URL: https://github.com/apache/beam/pull/7454#discussion_r247244427
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java
##
@@ -331,10 +331,8 @@ private void fireTimer(
public void close() throws Exception {
// close may be called multiple times when an exception is thrown
if (stageContext != null) {
- try (@SuppressWarnings("unused")
Review comment:
This would be a javac or findbugs issue, right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 184334)
Time Spent: 1.5h (was: 1h 20m)
> PortableRunner dependency missed in wordcount example maven artifact
>
>
> Key: BEAM-6184
> URL: https://issues.apache.org/jira/browse/BEAM-6184
> Project: Beam
> Issue Type: Improvement
> Components: build-system
>Reporter: Ruoyun Huang
>Assignee: Ruoyun Huang
>Priority: Minor
> Fix For: Not applicable
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
>
>
> more context:
> https://lists.apache.org/thread.html/8dd60395424425f7502d62888c49014430d1d3b06c026606f3db28ab@%3Cuser.beam.apache.org%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6184) PortableRunner dependency missed in wordcount example maven artifact
[ https://issues.apache.org/jira/browse/BEAM-6184?focusedWorklogId=184336=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184336 ] ASF GitHub Bot logged work on BEAM-6184: Author: ASF GitHub Bot Created on: 11/Jan/19 20:30 Start Date: 11/Jan/19 20:30 Worklog Time Spent: 10m Work Description: kennknowles commented on pull request #7454: [BEAM-6184] Enforce javadoc check on non-trivial public methods URL: https://github.com/apache/beam/pull/7454#discussion_r247244945 ## File path: sdks/java/build-tools/src/main/resources/beam/suppressions.xml ## @@ -21,6 +21,11 @@ + + + Review comment: I'm not sure about this one. If it is not in a `src/test` directory then it is not tests, but utility code that needs documentation. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184336) Time Spent: 1h 40m (was: 1.5h) > PortableRunner dependency missed in wordcount example maven artifact > > > Key: BEAM-6184 > URL: https://issues.apache.org/jira/browse/BEAM-6184 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Ruoyun Huang >Assignee: Ruoyun Huang >Priority: Minor > Fix For: Not applicable > > Time Spent: 1h 40m > Remaining Estimate: 0h > > > > more context: > https://lists.apache.org/thread.html/8dd60395424425f7502d62888c49014430d1d3b06c026606f3db28ab@%3Cuser.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-6407) regression: FileIO.writeDynamic() with side inputs fails in DirectRunner
[
https://issues.apache.org/jira/browse/BEAM-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-6407:
--
Affects Version/s: (was: 2.10.0)
> regression: FileIO.writeDynamic() with side inputs fails in DirectRunner
>
>
> Key: BEAM-6407
> URL: https://issues.apache.org/jira/browse/BEAM-6407
> Project: Beam
> Issue Type: Bug
> Components: beam-model
>Affects Versions: 2.9.0
>Reporter: Niel Markwick
>Assignee: Kenneth Knowles
>Priority: Major
> Labels: regression
> Attachments: beam-filewriter-demo.tgz
>
>
> When FileIO.writeDynamic is used with automatic sharding and a Contextful.Fn
> that uses side inputs for the file naming, DirectRunner (and TestPipeline)
> fail with:
> {{java.lang.IllegalStateException: All PCollectionViews that are consumed
> must be written by some WriteView PTransform: Missing [
> [RunnerPCollectionView]]}}
>
> Example code:
> {code:java}
> PCollectionView outputFileName =
> pipeline.apply(
> "outputDir",
> Create.of("/tmp/testout")).apply(View.asSingleton());
> Contextful.Fn manifestNaming =
> (element, c) ->
> (window, pane, numShards, shardIndex, compression) ->
> c.sideInput(outputFileName)+shardIndex;
> pipeline.apply(FileIO.writeDynamic()
> .by(SerializableFunctions.constant(""))
> .withDestinationCoder(StringUtf8Coder.of())
> .via(TextIO.sink())
> .withTempDirectory("/tmp")
> .withNaming(Contextful.of(
> manifestNaming,
> Requirements.requiresSideInputs(outputFileName;
> {code}
>
> This does not occur in Dataflow-runner
> It does not occur if the ContextFul.Fn is not given side inputs.
> It does not occur if withNumShards(1) is set.
> It did not occur in 2.8.0, and does in 2.9.0 and 2.10.0-SNAPSHOT (as of today)
>
> The cause appears to be due to the DirectRunner using TransformOverrides
> re-writing FileIO sinks to use runner-determined-sharding
> ( see [DirectRunner.java line
> 226|https://github.com/apache/beam/blob/master/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java#L226]
> )
> but I do not know why this started occuring in 2.9.0...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6365) Add ZStandard compression support for Java SDK
[ https://issues.apache.org/jira/browse/BEAM-6365?focusedWorklogId=184331=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184331 ] ASF GitHub Bot logged work on BEAM-6365: Author: ASF GitHub Bot Created on: 11/Jan/19 20:21 Start Date: 11/Jan/19 20:21 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #7416: [BEAM-6365] Add ZStandard compression support for Java SDK URL: https://github.com/apache/beam/pull/7416#issuecomment-453645833 Is it marked experimental? I think most new features should be, for a while. It might be hard to mark the right pieces of code, given how this fits in, but still seems a good idea. With that, I think as-is is fine. The implementation and tests are consistent with what we already have. I do think moving to an extension model would be a good improvement, and we can probably do it to all the pieces of the enum at the same time and this will not make it much harder. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184331) Time Spent: 1h 50m (was: 1h 40m) > Add ZStandard compression support for Java SDK > -- > > Key: BEAM-6365 > URL: https://issues.apache.org/jira/browse/BEAM-6365 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Jeff Klukas >Assignee: Jeff Klukas >Priority: Minor > Labels: has-pr > Time Spent: 1h 50m > Remaining Estimate: 0h > > ZStandard (zstd) compression has been gaining popularity since its > introduction a few years ago. It is supported in commons-compress, the > library we already use for handling file compression and decompression in > various formats, so should be straightforward to add support in the Java SDK > by adding a ZSTD enum value in Compression.java. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4606) Upper bound for pytz dependency
[ https://issues.apache.org/jira/browse/BEAM-4606?focusedWorklogId=184326=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184326 ] ASF GitHub Bot logged work on BEAM-4606: Author: ASF GitHub Bot Created on: 11/Jan/19 20:03 Start Date: 11/Jan/19 20:03 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #7487: [BEAM-4606] Remove the upper bound for pytz URL: https://github.com/apache/beam/pull/7487 **Please** add a meaningful description for your change here Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184326) Time Spent: 1h 50m (was: 1h 40m) > Upper bound for pytz dependency > --- > >
[jira] [Work logged] (BEAM-4606) Upper bound for pytz dependency
[ https://issues.apache.org/jira/browse/BEAM-4606?focusedWorklogId=184327=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184327 ] ASF GitHub Bot logged work on BEAM-4606: Author: ASF GitHub Bot Created on: 11/Jan/19 20:04 Start Date: 11/Jan/19 20:04 Worklog Time Spent: 10m Work Description: aaltay commented on issue #5751: [BEAM-4606] Pin pytz version URL: https://github.com/apache/beam/pull/5751#issuecomment-453639724 Agreed, let's remove the upper bound. Initially I was trying to be defensive that it does not seem to be necessary. I will send a revert PR shortly. Sent: https://github.com/apache/beam/pull/7487 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184327) Time Spent: 2h (was: 1h 50m) > Upper bound for pytz dependency > --- > > Key: BEAM-4606 > URL: https://issues.apache.org/jira/browse/BEAM-4606 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Ahmet Altay >Assignee: Udi Meiri >Priority: Major > Fix For: Not applicable > > Time Spent: 2h > Remaining Estimate: 0h > > Do we need an upper bound for the pytz dependency? > ([https://github.com/apache/beam/blob/release-2.5.0/sdks/python/setup.py#L108)] > We typically have upper bounds, in order to avoid future breakages due to a > possibility of breaking/backward incompatible change of that depepdency. > Good practice is to upper bound either at known version, or next major > version. Do we need an exception for pytz because it does not seem to be > following semantic versioning? > cc: [~yifanzou] Is this something dependency notifier can warn on? Dependency > without upper version bounds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4606) Upper bound for pytz dependency
[ https://issues.apache.org/jira/browse/BEAM-4606?focusedWorklogId=184324=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184324 ] ASF GitHub Bot logged work on BEAM-4606: Author: ASF GitHub Bot Created on: 11/Jan/19 20:01 Start Date: 11/Jan/19 20:01 Worklog Time Spent: 10m Work Description: aaltay commented on issue #5751: [BEAM-4606] Pin pytz version URL: https://github.com/apache/beam/pull/5751#issuecomment-453639724 Agreed, let's remove the upper bound. Initially I was trying to be defensive that it does not seem to be necessary. I will send a revert PR shortly. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184324) Time Spent: 1h 40m (was: 1.5h) > Upper bound for pytz dependency > --- > > Key: BEAM-4606 > URL: https://issues.apache.org/jira/browse/BEAM-4606 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Ahmet Altay >Assignee: Udi Meiri >Priority: Major > Fix For: Not applicable > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Do we need an upper bound for the pytz dependency? > ([https://github.com/apache/beam/blob/release-2.5.0/sdks/python/setup.py#L108)] > We typically have upper bounds, in order to avoid future breakages due to a > possibility of breaking/backward incompatible change of that depepdency. > Good practice is to upper bound either at known version, or next major > version. Do we need an exception for pytz because it does not seem to be > following semantic versioning? > cc: [~yifanzou] Is this something dependency notifier can warn on? Dependency > without upper version bounds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6231) Triage test failures introduced by use_executable_stage_bundle_execution
[ https://issues.apache.org/jira/browse/BEAM-6231?focusedWorklogId=184319=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184319 ] ASF GitHub Bot logged work on BEAM-6231: Author: ASF GitHub Bot Created on: 11/Jan/19 19:57 Start Date: 11/Jan/19 19:57 Worklog Time Spent: 10m Work Description: asfgit commented on issue #7356: [BEAM-6231] Make Dataflow runner harness work with FixedWindow URL: https://github.com/apache/beam/pull/7356#issuecomment-453637098 SUCCESS --none-- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184319) Time Spent: 5h 50m (was: 5h 40m) > Triage test failures introduced by use_executable_stage_bundle_execution > > > Key: BEAM-6231 > URL: https://issues.apache.org/jira/browse/BEAM-6231 > Project: Beam > Issue Type: Test > Components: runner-dataflow >Reporter: Boyuan Zhang >Assignee: Boyuan Zhang >Priority: Major > Time Spent: 5h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6231) Triage test failures introduced by use_executable_stage_bundle_execution
[ https://issues.apache.org/jira/browse/BEAM-6231?focusedWorklogId=184313=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184313 ] ASF GitHub Bot logged work on BEAM-6231: Author: ASF GitHub Bot Created on: 11/Jan/19 19:38 Start Date: 11/Jan/19 19:38 Worklog Time Spent: 10m Work Description: boyuanzz commented on issue #7356: [BEAM-6231] Make Dataflow runner harness work with FixedWindow URL: https://github.com/apache/beam/pull/7356#issuecomment-453633518 Please take another look at commit https://github.com/apache/beam/pull/7356/commits/7857ab0886275d6170f7e189048406baaf2f10a4, @robertwb . Thanks so much for your help! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184313) Time Spent: 5h 40m (was: 5.5h) > Triage test failures introduced by use_executable_stage_bundle_execution > > > Key: BEAM-6231 > URL: https://issues.apache.org/jira/browse/BEAM-6231 > Project: Beam > Issue Type: Test > Components: runner-dataflow >Reporter: Boyuan Zhang >Assignee: Boyuan Zhang >Priority: Major > Time Spent: 5h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6231) Triage test failures introduced by use_executable_stage_bundle_execution
[
https://issues.apache.org/jira/browse/BEAM-6231?focusedWorklogId=184312=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184312
]
ASF GitHub Bot logged work on BEAM-6231:
Author: ASF GitHub Bot
Created on: 11/Jan/19 19:38
Start Date: 11/Jan/19 19:38
Worklog Time Spent: 10m
Work Description: boyuanzz commented on pull request #7356: [BEAM-6231]
Make Dataflow runner harness work with FixedWindow
URL: https://github.com/apache/beam/pull/7356#discussion_r247232400
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/CreateExecutableStageNodeFunction.java
##
@@ -195,38 +200,8 @@ public Node apply(MutableNetwork input) {
Iterables.filter(input.nodes(), InstructionOutputNode.class)) {
InstructionOutput instructionOutput = node.getInstructionOutput();
- // If this is the input PCollection or the output PCollection for an
ExecutableStage, it's
- // necessary to check whether the window coder is not a GlobalWindow
coder.
- if (isExecutableStageInputPCollection(input, node)
- || isExecutableStageOutputPCollection(input, node)) {
-Coder javaCoder =
-
CloudObjects.coderFromCloudObject(CloudObject.fromSpec(instructionOutput.getCodec()));
-// For now, Dataflow runner harness only deal with FixedWindow.
-if (FullWindowedValueCoder.class.isInstance(javaCoder)) {
- FullWindowedValueCoder windowedValueCoder =
(FullWindowedValueCoder) javaCoder;
- Coder windowCoder = windowedValueCoder.getWindowCoder();
- if (IntervalWindowCoder.class.isInstance(windowCoder)) {
-fakeWindowingStrategyId = "generatedFixedWindowingStrategy" +
idGenerator.getId();
-try {
- // Since the coder is the only needed from a WindowingStrategy,
the size field of one
- // FixedWindows is meaningless here.
- RunnerApi.MessageWithComponents windowingStrategyProto =
- WindowingStrategyTranslation.toMessageProto(
-
WindowingStrategy.of(FixedWindows.of(Duration.standardSeconds(1))),
- sdkComponents);
- componentsBuilder.putWindowingStrategies(
- fakeWindowingStrategyId,
windowingStrategyProto.getWindowingStrategy());
-
componentsBuilder.putAllCoders(windowingStrategyProto.getComponents().getCodersMap());
- componentsBuilder.putAllEnvironments(
- windowingStrategyProto.getComponents().getEnvironmentsMap());
-} catch (IOException exc) {
- throw new RuntimeException("Could not convert FixedWindow
stratey to proto", exc);
-}
- }
-}
- }
-
String coderId = "generatedCoder" + idGenerator.getId();
+ String windowingStrategyId = fakeGlobalWindowingStrategyId;
Review comment:
I don't think we should have an exception clause if this is not a java
coder. In my latest changes, I set it to use GlobalWindow if it's not a java
coder.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 184312)
Time Spent: 5.5h (was: 5h 20m)
> Triage test failures introduced by use_executable_stage_bundle_execution
>
>
> Key: BEAM-6231
> URL: https://issues.apache.org/jira/browse/BEAM-6231
> Project: Beam
> Issue Type: Test
> Components: runner-dataflow
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-3772) BigQueryIO - Can't use DynamicDestination with CREATE_IF_NEEDED for unbounded PCollection and FILE_LOADS
[
https://issues.apache.org/jira/browse/BEAM-3772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740698#comment-16740698
]
Reuven Lax commented on BEAM-3772:
--
There was a similar bug in the past, but I believe it was fixed. Surprising
if this is happening in 2.9.
The text of the error message you posted does not seem to match the current
code in Beam 2.9. Can you verify that you are actually using Beam 2.9?
On Fri, Jan 11, 2019 at 1:22 AM Marco Veluscek (JIRA)
> BigQueryIO - Can't use DynamicDestination with CREATE_IF_NEEDED for unbounded
> PCollection and FILE_LOADS
>
>
> Key: BEAM-3772
> URL: https://issues.apache.org/jira/browse/BEAM-3772
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
>Affects Versions: 2.2.0, 2.3.0
> Environment: Dataflow streaming pipeline
>Reporter: Benjamin BENOIST
>Assignee: Eugene Kirpichov
>Priority: Major
>
> My workflow : KAFKA -> Dataflow streaming -> BigQuery
> Given that having low-latency isn't important in my case, I use FILE_LOADS to
> reduce the costs. I'm using _BigQueryIO.Write_ with a _DynamicDestination_,
> which is a table with the current hour as a suffix.
> This _BigQueryIO.Write_ is configured like this :
> {code:java}
> .withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED)
> .withMethod(Method.FILE_LOADS)
> .withTriggeringFrequency(triggeringFrequency)
> .withNumFileShards(100)
> {code}
> The first table is successfully created and is written to. But then the
> following tables are never created and I get these exceptions:
> {code:java}
> (99e5cd8c66414e7a): java.lang.RuntimeException: Failed to create load job
> with id prefix
> 5047f71312a94bf3a42ee5d67feede75_5295fbf25e1a7534f85e25dcaa9f4986_1_00023,
> reached max retries: 3, last failed load job: {
> "configuration" : {
> "load" : {
> "createDisposition" : "CREATE_NEVER",
> "destinationTable" : {
> "datasetId" : "dev_mydataset",
> "projectId" : "myproject-id",
> "tableId" : "mytable_20180302_16"
> },
> {code}
> The _CreateDisposition_ used is _CREATE_NEVER_, contrary as
> _CREATE_IF_NEEDED_ as specified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-6289) Running a join on two Cassandra tables using FlinkRunner fails
[
https://issues.apache.org/jira/browse/BEAM-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740646#comment-16740646
]
Maximilian Michels commented on BEAM-6289:
--
[~srfrnk] Just mentioning you in case you didn't see the above comment.
> Running a join on two Cassandra tables using FlinkRunner fails
> --
>
> Key: BEAM-6289
> URL: https://issues.apache.org/jira/browse/BEAM-6289
> Project: Beam
> Issue Type: Bug
> Components: io-java-cassandra, runner-flink
>Affects Versions: 2.8.0, 2.9.0
> Environment: Tested on Ubuntu 18
> Beam 2.8
> Tested with Flink:
> 1) [local]
> 2) Cluster inside a K8S cluster on minikube
> 3) Cluster inside a K8S cluster on GCP
> Tested using Cassandra [cqlsh 5.0.1 | Cassandra 3.11.3 | CQL spec 3.4.4 |
> Native protocol v4]:
> 1) In a local container
> 2) Cluster inside a K8S cluster on minikube
> 3) Cluster inside a K8S cluster on GCP
>Reporter: Shahar Frank
>Assignee: Maximilian Michels
>Priority: Critical
> Labels: FlinkRunner, beam, bug, cassandra, flink, join
> Fix For: Not applicable
>
> Attachments: direct_runner_build.log, flink_runner_build.log
>
>
> Can't make a simple join on two Cassandra tables when using FlinkRunner.
> The same code works with a DirectRunner fails when used with FlinkRunner
> giving these (as well as many other) errors:
> {code:java}
> Caused by: akka.pattern.AskTimeoutException: Ask timed out on
> [Actor[akka://flink/user/dispatchere1f5abe7-6299-43ea-9182-24a2193e078f#-1757043920]]
> after [1 ms]. Sender[null] sent message of type
> "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
> at
> akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
> at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
> at
> scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
> at
> scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
> at
> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
> at
> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
> The code can be found [here|https://github.com/srfrnk/beam-playground]
> Steps to reproduce:
> # Clone the repo to a linux (I;m on Ubuntu 18 but any *nix system would
> probably work - i.e. repl.it)
> # Follow the README to set up a Cassandra container + schema
> # Run with
> {code}
> gradle --console=plain join-from-cassandra -Drunner=flink > output/build.log
> 2>&1{code}
> to use FlinkRunner. See error in log at ./output/build.log
> # Run with
> {code}
> gradle --console=plain join-from-cassandra -Drunner=direct > output/build.log
> 2>&1{code}
> to use DirectRunner. See error in log at ./output/build.log
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[
https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=184293=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184293
]
ASF GitHub Bot logged work on BEAM-6024:
Author: ASF GitHub Bot
Created on: 11/Jan/19 18:37
Start Date: 11/Jan/19 18:37
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #7423: [BEAM-6024]
Build Python 3 container image with Gradle
URL: https://github.com/apache/beam/pull/7423#discussion_r247209615
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1523,7 +1523,14 @@ artifactId=${project.name}
project.task('setupVirtualenv') {
doLast {
- project.exec { commandLine 'virtualenv', "${project.ext.envdir}" }
+ def virtualenvCmd = [
+'virtualenv',
+"${project.ext.envdir}",
+ ]
+ if (project.hasProperty('python3')) {
+virtualenvCmd += '--python=python3'
Review comment:
In the future, I'd like us to be able to customize which Python 3 version to
use (3.5, 3.6, etc). Dockerfile can take an argument for the FROM clause as
well. This can happen in the future PR, but for the scope of this PR I'd like
to make sure that we use the same Python 3 version to build the SDK as we use
in the Dockerfile. So we I think should pass `--python=python3.5`, but we could
make this a constant somewhere up in the file and make a note that the version
should match the version we [inside the Docker
container](https://github.com/apache/beam/blob/3a531323322b3f5bbd695b64744a0603a1c97607/sdks/python/container/py3/Dockerfile#L19).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 184293)
Time Spent: 40m (was: 0.5h)
> Gradle setupVirtualenv supports Python 3
>
>
> Key: BEAM-6024
> URL: https://issues.apache.org/jira/browse/BEAM-6024
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-harness
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Need to depend on Python 3 virtualenv in few places:
> - Build Dataflow worker container in Python 3
> - Run ValidatesRunner and integration tests on Jenkins
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
