[jira] [Work logged] (BEAM-6857) Support dynamic timers
[
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=356866&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356866
]
ASF GitHub Bot logged work on BEAM-6857:
Author: ASF GitHub Bot
Created on: 10/Dec/19 07:47
Start Date: 10/Dec/19 07:47
Worklog Time Spent: 10m
Work Description: rehmanmuradali commented on pull request #10315:
[BEAM-6857] Support Dynamic Timers
URL: https://github.com/apache/beam/pull/10315
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356866)
Time Spent: 20m (was: 10m)
> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified
> in the DoFn. The user must provide a separate callback method per timer. For
> example:
>
> {code:java}
> DoFn()
> {
> @TimerId("timer1")
> private final TimerSpec timer1 = TimerSpecs.timer(...);
> @TimerId("timer2")
> private final TimerSpec timer2 = TimerSpecs.timer(...);
> .. set timers in processElement
> @OnTimer("timer1")
> public void onTimer1() { .}
> @OnTimer("timer2")
> public void onTimer2() {}
> }
> {code}
>
> However there are many cases where the user does not know the set of timers
> statically when writing their code. This happens when the timer tag should be
> based on the data. It also happens when writing a DSL on top of Beam, where
> the DSL author has to create DoFns but does not know statically which timers
> their users will want to set (e.g. Scio).
>
> The goal is to support dynamic timers. Something as follows;
>
> {code:java}
> DoFn()
> {
> @TimerId("timer")
> private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
> @ProcessElement process(@TimerId("timer") DynamicTimer timer)
> {
> timer.set("tag1'", ts);
>timer.set("tag2", ts);
> }
> @OnTimer("timer")
> public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8935) Fail fast if sdk harness startup failed
[ https://issues.apache.org/jira/browse/BEAM-8935?focusedWorklogId=356849&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356849 ] ASF GitHub Bot logged work on BEAM-8935: Author: ASF GitHub Bot Created on: 10/Dec/19 06:06 Start Date: 10/Dec/19 06:06 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10338: [BEAM-8935] Fail fast if sdk harness startup failed. URL: https://github.com/apache/beam/pull/10338#issuecomment-563879682 R: @tgroh This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356849) Time Spent: 20m (was: 10m) > Fail fast if sdk harness startup failed > --- > > Key: BEAM-8935 > URL: https://issues.apache.org/jira/browse/BEAM-8935 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Currently the runner waits for the sdk harness to startup blockingly until > the sdk harness is available or timeout occurs. The timeout is 1 or 2 > minutes. If the sdk harness startup failed for some reason, the runner may be > aware of it after 1 or 2 minutes. This is too long. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8935) Fail fast if sdk harness startup failed
[ https://issues.apache.org/jira/browse/BEAM-8935?focusedWorklogId=356848&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356848 ] ASF GitHub Bot logged work on BEAM-8935: Author: ASF GitHub Bot Created on: 10/Dec/19 06:00 Start Date: 10/Dec/19 06:00 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on pull request #10338: [BEAM-8935] Fail fast if sdk harness startup failed. URL: https://github.com/apache/beam/pull/10338 Currently the runner waits for the sdk harness to startup blockingly until the sdk harness is available or timeout occurs. The timeout is 1 or 2 minutes. If the sdk harness startup failed for some reason, the runner may be aware of it after 1 or 2 minutes. This is too long. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostC
[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests
[
https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=356836&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356836
]
ASF GitHub Bot logged work on BEAM-8575:
Author: ASF GitHub Bot
Created on: 10/Dec/19 05:10
Start Date: 10/Dec/19 05:10
Worklog Time Spent: 10m
Work Description: HuangLED commented on pull request #10050: [BEAM-8575]
Add streaming test case for multi-triggered GBK as side input
URL: https://github.com/apache/beam/pull/10050#discussion_r355846341
##
File path: sdks/python/apache_beam/transforms/sideinputs_test.py
##
@@ -309,6 +316,66 @@ def test_flattened_side_input(self):
assert_that(results, equal_to(['a', 'b']))
pipeline.run()
+ @attr('ValidatesRunner')
+ def test_multi_triggered_gbk_side_input(self):
+"""Test a GBK sideinput, with multiple triggering."""
+options = PipelineOptions()
+options.view_as(StandardOptions).streaming = True
+p = TestPipeline(options=options)
+
+test_stream = (p
+ | 'Mixed TestStream' >> TestStream()
+ .advance_watermark_to(3, tag='main')
+ .add_elements(['a1'], tag='main')
+ .advance_watermark_to(8, tag='main')
+ .add_elements(['a2'], tag='main')
+ .add_elements([window.TimestampedValue(('k', 100), 2)],
+ tag='side')
+ .add_elements([window.TimestampedValue(('k', 400), 7)],
+ tag='side')
+ .advance_watermark_to_infinity(tag='main')
+ .advance_watermark_to_infinity(tag='side'))
+
+main_data = (test_stream['main']
+ | 'Main windowInto' >> beam.WindowInto(
+ window.FixedWindows(5),
+ accumulation_mode=trigger.AccumulationMode.DISCARDING))
+
+side_data = (test_stream['side']
+ | 'Side windowInto' >> beam.WindowInto(
+ window.FixedWindows(5),
+ trigger=trigger.AfterWatermark(
+ early=trigger.AfterCount(1)),
+ accumulation_mode=trigger.AccumulationMode.DISCARDING)
+ | beam.CombinePerKey(sum)
+ | 'Values' >> Map(lambda k_vs: k_vs[1]))
+
+class RecordFn(beam.DoFn):
+ def process(self,
+ elm=beam.DoFn.ElementParam,
+ ts=beam.DoFn.TimestampParam,
+ side=beam.DoFn.SideInputParam):
+yield (elm, ts, side)
+
+records = (main_data
+ | beam.ParDo(RecordFn(), beam.pvalue.AsList(side_data)))
+
+expected_window_to_elements = {
+window.IntervalWindow(0, 5): [
+('a1', Timestamp(3), [100, 0]),
Review comment:
@rohdesamuel I've been using the multi-output testStream feature.
Everything works perfectly well except for this "0" in the side input data. Is
this WAI?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356836)
Time Spent: 30h 50m (was: 30h 40m)
> Add more Python validates runner tests
> --
>
> Key: BEAM-8575
> URL: https://issues.apache.org/jira/browse/BEAM-8575
> Project: Beam
> Issue Type: Test
> Components: sdk-py-core, testing
>Reporter: wendy liu
>Assignee: wendy liu
>Priority: Major
> Time Spent: 30h 50m
> Remaining Estimate: 0h
>
> This is the umbrella issue to track the work of adding more Python tests to
> improve test coverage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests
[ https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=356827&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356827 ] ASF GitHub Bot logged work on BEAM-8575: Author: ASF GitHub Bot Created on: 10/Dec/19 04:26 Start Date: 10/Dec/19 04:26 Worklog Time Spent: 10m Work Description: HuangLED commented on issue #10050: [BEAM-8575] Add streaming test case for multi-triggered GBK as side input URL: https://github.com/apache/beam/pull/10050#issuecomment-563763740 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356827) Time Spent: 30h 40m (was: 30.5h) > Add more Python validates runner tests > -- > > Key: BEAM-8575 > URL: https://issues.apache.org/jira/browse/BEAM-8575 > Project: Beam > Issue Type: Test > Components: sdk-py-core, testing >Reporter: wendy liu >Assignee: wendy liu >Priority: Major > Time Spent: 30h 40m > Remaining Estimate: 0h > > This is the umbrella issue to track the work of adding more Python tests to > improve test coverage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8911) Upgrade Guava to 25
[
https://issues.apache.org/jira/browse/BEAM-8911?focusedWorklogId=356824&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356824
]
ASF GitHub Bot logged work on BEAM-8911:
Author: ASF GitHub Bot
Created on: 10/Dec/19 04:21
Start Date: 10/Dec/19 04:21
Worklog Time Spent: 10m
Work Description: suztomo commented on issue #10318: [BEAM-8911] New
non-vendored Guava version: 25.1-jre
URL: https://github.com/apache/beam/pull/10318#issuecomment-563756989
Run Dataflow ValidatesRunner
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356824)
Time Spent: 5h (was: 4h 50m)
> Upgrade Guava to 25
> ---
>
> Key: BEAM-8911
> URL: https://issues.apache.org/jira/browse/BEAM-8911
> Project: Beam
> Issue Type: Improvement
> Components: build-system
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Now [BEAM-8822] Hadoop client version 2.8.5 from 2.7 is merged. It's time to
> upgrade Guava.
> h1. New Guava version: 25.1-jre
> CharMatcher.DIGIT, used by
> org.apache.cassandra.io.sstable.format.SSTableFormat, is removed since Guava
> 26.0. 25.1 is the latest version:
> https://guava.dev/releases/25.1-jre/api/docs/com/google/common/base/CharMatcher.html
> https://guava.dev/releases/26.0-jre/api/docs/com/google/common/base/CharMatcher.html
> Guava has 2 variations "-jre" for Java8 (stream/lambda support) and
> "-android" for Java7. Gcsio:1.9.16, which Beam has not have in dependency
> tree yet, needs ImmutableSet.toImmutableSet() which is available in Guava
> "-jre".
> https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557680928
> h2. What about upgrading Cassandra-all?
> The latest cassandra-all release is "4.0-alpha2", which is still alpha.
> Non-alpha version is 3.11.5
> https://search.maven.org/artifact/org.apache.cassandra/cassandra-all/3.11.5/jar,
> which still relies on Guava 19 and it still has reference to DIGIT:
> {noformat}
> suztomo@suxtomo24:~/grpc-java$ javap -v -cp
> ~/Downloads/cassandra-all-3.11.5.jar
> 'org.apache.cassandra.io.sstable.format.SSTableFormat$Type' |grep DIGIT
> #9 = Fieldref #79.#80 //
> com/google/common/base/CharMatcher.DIGIT:Lcom/google/common/base/CharMatcher;
>#80 = NameAndType#110:#111 //
> DIGIT:Lcom/google/common/base/CharMatcher;
> #110 = Utf8 DIGIT
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8858) Missing dependencies in SQL module pom
[ https://issues.apache.org/jira/browse/BEAM-8858?focusedWorklogId=356822&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356822 ] ASF GitHub Bot logged work on BEAM-8858: Author: ASF GitHub Bot Created on: 10/Dec/19 04:20 Start Date: 10/Dec/19 04:20 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10332: [BEAM-8858] sdks/java/extensions/sql to declare used-but-undeclared dependencies URL: https://github.com/apache/beam/pull/10332#issuecomment-563755114 Run Java PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356822) Time Spent: 1.5h (was: 1h 20m) > Missing dependencies in SQL module pom > -- > > Key: BEAM-8858 > URL: https://issues.apache.org/jira/browse/BEAM-8858 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Affects Versions: 2.18.0 >Reporter: Ismaël Mejía >Assignee: Tomo Suzuki >Priority: Blocker > Fix For: 2.18.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > I moved an user project that depends on the SQL module to the version > 2.18.0-SNAPSHOT and I discovered that it was not working anymore because some > of the transitive dependencies were not correctly exposed anymore by the SQL > module e.g. commons-codec and Nullable annotations dependency. > There are some missing dependencies in the generated pom file when compared > with previous versions, probably cases of `usedUndeclaredArtifacts` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8917) javax.annotation.Nullable is missing for org.apache.beam.sdk.schemas.FieldValueTypeInformation
[
https://issues.apache.org/jira/browse/BEAM-8917?focusedWorklogId=356823&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356823
]
ASF GitHub Bot logged work on BEAM-8917:
Author: ASF GitHub Bot
Created on: 10/Dec/19 04:20
Start Date: 10/Dec/19 04:20
Worklog Time Spent: 10m
Work Description: suztomo commented on issue #10324: [BEAM-8917] jsr305
dependency declaration for Nullable class
URL: https://github.com/apache/beam/pull/10324#issuecomment-563756186
Run Dataflow ValidatesRunner
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356823)
Time Spent: 3h 10m (was: 3h)
> javax.annotation.Nullable is missing for
> org.apache.beam.sdk.schemas.FieldValueTypeInformation
> --
>
> Key: BEAM-8917
> URL: https://issues.apache.org/jira/browse/BEAM-8917
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> This ticket is from the result of static analysis by Linkage Checker
> ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1045])
> h1. Example Project
> Example project to produce an issue:
> https://github.com/suztomo/beam-java-sdk-missing-nullable .
> I think the Maven artifact {{org.apache.beam:beam-sdks-java-core}}, which
> contains {{org.apache.beam.sdk.schemas.FieldValueTypeInformation}}, should
> declare the dependency to {{com.google.code.findbugs:jsr305}}.
> h1. Why there's no problem in compilation and tests of sdks/java/core?
> The compilation succeeds because the {{Nullable}} annotation is in the
> transitive dependency of compileOnly {{spotbugs-annotations}} dependency:
> {noformat}
> compileOnly - Compile only dependencies for source set 'main'.
> ...
> +--- com.github.spotbugs:spotbugs-annotations:3.1.12
> |\--- com.google.code.findbugs:jsr305:3.0.2
> ...
> {noformat}
> The tests succeed because the {{Nullable}} annotation is in the transitive
> dependency of {{guava-testlib}}.
> {noformat}
> testRuntime - Runtime dependencies for source set 'test' (deprecated, use
> 'testRuntimeOnly' instead).
> ...
> +--- com.google.guava:guava-testlib:20.0
> |+--- com.google.code.findbugs:jsr305:1.3.9
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8858) Missing dependencies in SQL module pom
[ https://issues.apache.org/jira/browse/BEAM-8858?focusedWorklogId=356821&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356821 ] ASF GitHub Bot logged work on BEAM-8858: Author: ASF GitHub Bot Created on: 10/Dec/19 04:19 Start Date: 10/Dec/19 04:19 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10332: [BEAM-8858] sdks/java/extensions/sql to declare used-but-undeclared dependencies URL: https://github.com/apache/beam/pull/10332#issuecomment-563754498 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356821) Time Spent: 1h 20m (was: 1h 10m) > Missing dependencies in SQL module pom > -- > > Key: BEAM-8858 > URL: https://issues.apache.org/jira/browse/BEAM-8858 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Affects Versions: 2.18.0 >Reporter: Ismaël Mejía >Assignee: Tomo Suzuki >Priority: Blocker > Fix For: 2.18.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > I moved an user project that depends on the SQL module to the version > 2.18.0-SNAPSHOT and I discovered that it was not working anymore because some > of the transitive dependencies were not correctly exposed anymore by the SQL > module e.g. commons-codec and Nullable annotations dependency. > There are some missing dependencies in the generated pom file when compared > with previous versions, probably cases of `usedUndeclaredArtifacts` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8936) BigQuery related ITs are failing in PostCommit: quota exceeded
Yueyang Qiu created BEAM-8936: - Summary: BigQuery related ITs are failing in PostCommit: quota exceeded Key: BEAM-8936 URL: https://issues.apache.org/jira/browse/BEAM-8936 Project: Beam Issue Type: Bug Components: io-py-gcp, test-failures Reporter: Yueyang Qiu Assignee: Mark Liu beam_PostCommit_Java: [https://builds.apache.org/job/beam_PostCommit_Java/4852/] beam_PostCommit_Python2: [https://builds.apache.org/job/beam_PostCommit_Python2/1178|https://builds.apache.org/job/beam_PostCommit_Python2/1178/#showFailuresLink] beam_PostCommit_Python35: [https://builds.apache.org/job/beam_PostCommit_Python35/1185] ... This seems to be a GCP quota issue. Mark, could you help take a look or find a owner of this bug? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8935) Fail fast if sdk harness startup failed
sunjincheng created BEAM-8935: - Summary: Fail fast if sdk harness startup failed Key: BEAM-8935 URL: https://issues.apache.org/jira/browse/BEAM-8935 Project: Beam Issue Type: Improvement Components: java-fn-execution Reporter: sunjincheng Assignee: sunjincheng Fix For: 2.19.0 Currently the runner waits for the sdk harness to startup blockingly until the sdk harness is available or timeout occurs. The timeout is 1 or 2 minutes. If the sdk harness startup failed for some reason, the runner may be aware of it after 1 or 2 minutes. This is too long. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8931) [beam_PostCommit_Java_Nexmark_Dataflow] Several queries are failing
[ https://issues.apache.org/jira/browse/BEAM-8931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yueyang Qiu updated BEAM-8931: -- Description: [https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/4484/] This test suite has been failing most of the time recently. *18:33:14* java.lang.RuntimeException: Execution was not successful*18:33:14* at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:170)*18:33:14* at org.apache.beam.sdk.nexmark.Main.main(Main.java:415)*18:33:14* *18:33:14* > *Task :sdks:java:testing:nexmark:run* FAILED*18:33:14* *18:33:14* FAILURE: Build failed with an exception.*18:33:14* *18:33:14* * What went wrong:*18:33:14* Execution failed for task ':sdks:java:testing:nexmark:run'.*18:33:14* > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit was: [https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/4484/] This test suite has been very flaky recently. *18:33:14* java.lang.RuntimeException: Execution was not successful*18:33:14* at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:170)*18:33:14* at org.apache.beam.sdk.nexmark.Main.main(Main.java:415)*18:33:14* *18:33:14* > *Task :sdks:java:testing:nexmark:run* FAILED*18:33:14* *18:33:14* FAILURE: Build failed with an exception.*18:33:14* *18:33:14* * What went wrong:*18:33:14* Execution failed for task ':sdks:java:testing:nexmark:run'.*18:33:14* > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit > [beam_PostCommit_Java_Nexmark_Dataflow] Several queries are failing > --- > > Key: BEAM-8931 > URL: https://issues.apache.org/jira/browse/BEAM-8931 > Project: Beam > Issue Type: Bug > Components: test-failures, testing-nexmark >Reporter: Yueyang Qiu >Assignee: Kenneth Knowles >Priority: Major > Labels: currently-failing > > [https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/4484/] > This test suite has been failing most of the time recently. > > *18:33:14* java.lang.RuntimeException: Execution was not > successful*18:33:14* at > org.apache.beam.sdk.nexmark.Main.runAll(Main.java:170)*18:33:14* at > org.apache.beam.sdk.nexmark.Main.main(Main.java:415)*18:33:14* *18:33:14* > > *Task :sdks:java:testing:nexmark:run* > FAILED*18:33:14* *18:33:14* FAILURE: Build failed with an > exception.*18:33:14* *18:33:14* * What went wrong:*18:33:14* Execution failed > for task ':sdks:java:testing:nexmark:run'.*18:33:14* > Process 'command > '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-8931) [beam_PostCommit_Java_Nexmark_Dataflow] Several queries are failing
[ https://issues.apache.org/jira/browse/BEAM-8931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yueyang Qiu reassigned BEAM-8931: - Assignee: Yueyang Qiu > [beam_PostCommit_Java_Nexmark_Dataflow] Several queries are failing > --- > > Key: BEAM-8931 > URL: https://issues.apache.org/jira/browse/BEAM-8931 > Project: Beam > Issue Type: Bug > Components: test-failures, testing-nexmark >Reporter: Yueyang Qiu >Assignee: Yueyang Qiu >Priority: Major > Labels: currently-failing > > [https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/4484/] > This test suite has been very flaky recently. > > *18:33:14* java.lang.RuntimeException: Execution was not successful*18:33:14* > at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:170)*18:33:14* > at org.apache.beam.sdk.nexmark.Main.main(Main.java:415)*18:33:14* > *18:33:14* > > *Task :sdks:java:testing:nexmark:run* > FAILED*18:33:14* *18:33:14* FAILURE: Build failed with an > exception.*18:33:14* *18:33:14* * What went wrong:*18:33:14* Execution failed > for task ':sdks:java:testing:nexmark:run'.*18:33:14* > Process 'command > '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8931) [beam_PostCommit_Java_Nexmark_Dataflow] Several queries are failing
[ https://issues.apache.org/jira/browse/BEAM-8931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yueyang Qiu updated BEAM-8931: -- Description: [https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/4484/] This test suite has been very flaky recently. *18:33:14* java.lang.RuntimeException: Execution was not successful*18:33:14* at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:170)*18:33:14* at org.apache.beam.sdk.nexmark.Main.main(Main.java:415)*18:33:14* *18:33:14* > *Task :sdks:java:testing:nexmark:run* FAILED*18:33:14* *18:33:14* FAILURE: Build failed with an exception.*18:33:14* *18:33:14* * What went wrong:*18:33:14* Execution failed for task ':sdks:java:testing:nexmark:run'.*18:33:14* > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit was: [https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/4484/console] (Add any investigation notes so far) _After you've filled out the above details, please [assign the issue to an individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist]. Assignee should [treat test failures as high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test], helping to fix the issue or find a more appropriate owner. See [Apache Beam Post-Commit Policies|https://beam.apache.org/contribute/postcommits-policies]._ > [beam_PostCommit_Java_Nexmark_Dataflow] Several queries are failing > --- > > Key: BEAM-8931 > URL: https://issues.apache.org/jira/browse/BEAM-8931 > Project: Beam > Issue Type: Bug > Components: test-failures, testing-nexmark >Reporter: Yueyang Qiu >Priority: Major > Labels: currently-failing > > [https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/4484/] > This test suite has been very flaky recently. > > *18:33:14* java.lang.RuntimeException: Execution was not successful*18:33:14* > at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:170)*18:33:14* > at org.apache.beam.sdk.nexmark.Main.main(Main.java:415)*18:33:14* > *18:33:14* > > *Task :sdks:java:testing:nexmark:run* > FAILED*18:33:14* *18:33:14* FAILURE: Build failed with an > exception.*18:33:14* *18:33:14* * What went wrong:*18:33:14* Execution failed > for task ':sdks:java:testing:nexmark:run'.*18:33:14* > Process 'command > '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-8931) [beam_PostCommit_Java_Nexmark_Dataflow] Several queries are failing
[ https://issues.apache.org/jira/browse/BEAM-8931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yueyang Qiu reassigned BEAM-8931: - Assignee: Kenneth Knowles (was: Yueyang Qiu) > [beam_PostCommit_Java_Nexmark_Dataflow] Several queries are failing > --- > > Key: BEAM-8931 > URL: https://issues.apache.org/jira/browse/BEAM-8931 > Project: Beam > Issue Type: Bug > Components: test-failures, testing-nexmark >Reporter: Yueyang Qiu >Assignee: Kenneth Knowles >Priority: Major > Labels: currently-failing > > [https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/4484/] > This test suite has been very flaky recently. > > *18:33:14* java.lang.RuntimeException: Execution was not successful*18:33:14* > at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:170)*18:33:14* > at org.apache.beam.sdk.nexmark.Main.main(Main.java:415)*18:33:14* > *18:33:14* > > *Task :sdks:java:testing:nexmark:run* > FAILED*18:33:14* *18:33:14* FAILURE: Build failed with an > exception.*18:33:14* *18:33:14* * What went wrong:*18:33:14* Execution failed > for task ':sdks:java:testing:nexmark:run'.*18:33:14* > Process 'command > '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8917) javax.annotation.Nullable is missing for org.apache.beam.sdk.schemas.FieldValueTypeInformation
[
https://issues.apache.org/jira/browse/BEAM-8917?focusedWorklogId=356794&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356794
]
ASF GitHub Bot logged work on BEAM-8917:
Author: ASF GitHub Bot
Created on: 10/Dec/19 03:02
Start Date: 10/Dec/19 03:02
Worklog Time Spent: 10m
Work Description: kennknowles commented on pull request #10324:
[BEAM-8917] jsr305 dependency declaration for Nullable class
URL: https://github.com/apache/beam/pull/10324#discussion_r355823988
##
File path: sdks/java/core/build.gradle
##
@@ -69,6 +69,7 @@ dependencies {
compile library.java.protobuf_java
compile library.java.commons_compress
compile library.java.commons_lang3
+ compile library.java.jsr305
Review comment:
We have a special configuration for annotation-only deps here:
https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L741
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356794)
Time Spent: 3h (was: 2h 50m)
> javax.annotation.Nullable is missing for
> org.apache.beam.sdk.schemas.FieldValueTypeInformation
> --
>
> Key: BEAM-8917
> URL: https://issues.apache.org/jira/browse/BEAM-8917
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Time Spent: 3h
> Remaining Estimate: 0h
>
> This ticket is from the result of static analysis by Linkage Checker
> ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1045])
> h1. Example Project
> Example project to produce an issue:
> https://github.com/suztomo/beam-java-sdk-missing-nullable .
> I think the Maven artifact {{org.apache.beam:beam-sdks-java-core}}, which
> contains {{org.apache.beam.sdk.schemas.FieldValueTypeInformation}}, should
> declare the dependency to {{com.google.code.findbugs:jsr305}}.
> h1. Why there's no problem in compilation and tests of sdks/java/core?
> The compilation succeeds because the {{Nullable}} annotation is in the
> transitive dependency of compileOnly {{spotbugs-annotations}} dependency:
> {noformat}
> compileOnly - Compile only dependencies for source set 'main'.
> ...
> +--- com.github.spotbugs:spotbugs-annotations:3.1.12
> |\--- com.google.code.findbugs:jsr305:3.0.2
> ...
> {noformat}
> The tests succeed because the {{Nullable}} annotation is in the transitive
> dependency of {{guava-testlib}}.
> {noformat}
> testRuntime - Runtime dependencies for source set 'test' (deprecated, use
> 'testRuntimeOnly' instead).
> ...
> +--- com.google.guava:guava-testlib:20.0
> |+--- com.google.code.findbugs:jsr305:1.3.9
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8929) Remove unnecessary exception handling in FnApiControlClientPoolService
[
https://issues.apache.org/jira/browse/BEAM-8929?focusedWorklogId=356789&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356789
]
ASF GitHub Bot logged work on BEAM-8929:
Author: ASF GitHub Bot
Created on: 10/Dec/19 02:49
Start Date: 10/Dec/19 02:49
Worklog Time Spent: 10m
Work Description: sunjincheng121 commented on pull request #10326:
[BEAM-8929] Remove unnecessary exception handling in FnApiControlClie…
URL: https://github.com/apache/beam/pull/10326#discussion_r355821207
##
File path:
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClientPoolService.java
##
@@ -99,9 +99,6 @@ public static FnApiControlClientPoolService
offeringClientsToPool(
// We do not attempt to transactionally add the client to our internal
list and offer it to
// the sink.
clientSink.put(headerAccessor.getSdkWorkerId(), newClient);
-} catch (InterruptedException e) {
- Thread.currentThread().interrupt();
- throw new RuntimeException(e);
Review comment:
This exception catch was added in
[2017](https://github.com/apache/beam/commit/9ed655be780630e1218d185bd0d2ebfea099b988#diff-40daa607c5288b4785dada738d9de8ecR31).
The purpose was to catch InterruptedException from `BlockingQueue # put (E
var1) throws InterruptedException`. At present, the implementation of the code
has changed from BlockingQueue to [ControlClientPool.Sink #
put](https://github.com/apache/beam/blob/59b4b6d5318594c6fd7fae2dd6be2dd8aca972fa/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClientPoolService.java#L40).
There is no chance of throwing an `InterruptedException`. So I think it is
possible to delete the logic of this exception catch. Am i missing something
here？
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356789)
Time Spent: 40m (was: 0.5h)
> Remove unnecessary exception handling in FnApiControlClientPoolService
> --
>
> Key: BEAM-8929
> URL: https://issues.apache.org/jira/browse/BEAM-8929
> Project: Beam
> Issue Type: Improvement
> Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.19.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The exception handling logic in
> [FnApiControlClientPoolService|https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClientPoolService.java#L102]
> is unnecessary and could be removed.(Clean up usless code)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8877) beam_PostCommit_Py_VR_Dataflow is timing out
[ https://issues.apache.org/jira/browse/BEAM-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992135#comment-16992135 ] Valentyn Tymofieiev commented on BEAM-8877: --- The suite continues to fail. Currently due to quota errors in apache-beam-testing but over a weekend over seemed to be timing out[1], likely in some streaming test. Logs are not sufficiently helpful to show which test is problematic. We need to improve debuggability of these tests suites. [[1] https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/5248/|https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/5248/] > beam_PostCommit_Py_VR_Dataflow is timing out > > > Key: BEAM-8877 > URL: https://issues.apache.org/jira/browse/BEAM-8877 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core, test-failures >Reporter: Ahmet Altay >Assignee: Valentyn Tymofieiev >Priority: Critical > > Error: > 06:47:45 Build timed out (after 100 minutes). Marking the build as aborted. > 06:47:45 Build was aborted > Log: > [https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/5214/console] > > Should we increase the timeout here similar to : > [https://github.com/apache/beam/pull/10234] > cc: [~Ardagan] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8858) Missing dependencies in SQL module pom
[ https://issues.apache.org/jira/browse/BEAM-8858?focusedWorklogId=356779&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356779 ] ASF GitHub Bot logged work on BEAM-8858: Author: ASF GitHub Bot Created on: 10/Dec/19 02:30 Start Date: 10/Dec/19 02:30 Worklog Time Spent: 10m Work Description: vectorijk commented on issue #10332: [BEAM-8858] sdks/java/extensions/sql to declare used-but-undeclared dependencies URL: https://github.com/apache/beam/pull/10332#issuecomment-563611182 Run Spark Runner Nexmark Tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356779) Time Spent: 1h 10m (was: 1h) > Missing dependencies in SQL module pom > -- > > Key: BEAM-8858 > URL: https://issues.apache.org/jira/browse/BEAM-8858 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Affects Versions: 2.18.0 >Reporter: Ismaël Mejía >Assignee: Tomo Suzuki >Priority: Blocker > Fix For: 2.18.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > I moved an user project that depends on the SQL module to the version > 2.18.0-SNAPSHOT and I discovered that it was not working anymore because some > of the transitive dependencies were not correctly exposed anymore by the SQL > module e.g. commons-codec and Nullable annotations dependency. > There are some missing dependencies in the generated pom file when compared > with previous versions, probably cases of `usedUndeclaredArtifacts` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8934) Store&Read offset with KafkaIO
jiefeng zheng created BEAM-8934: --- Summary: Store&Read offset with KafkaIO Key: BEAM-8934 URL: https://issues.apache.org/jira/browse/BEAM-8934 Project: Beam Issue Type: New Feature Components: io-java-kafka Reporter: jiefeng zheng [When creating a Pipeline through a KafkaIO object, I want to be able to specify the starting offset of consumption, and when traversing the message later, I can get the offset of the current message for storage in a relational database / NoSQL.|http://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html] This feature is used to implement the exactly-once semantics of spark streaming consumption. In the "Your own data store" section of the following url content, you can find how to achieve exactly-once semantics with spark streaming: [http://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html|http://example.com] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-8917) javax.annotation.Nullable is missing for org.apache.beam.sdk.schemas.FieldValueTypeInformation
[
https://issues.apache.org/jira/browse/BEAM-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992115#comment-16992115
]
Tomo Suzuki edited comment on BEAM-8917 at 12/10/19 2:13 AM:
-
[~iemejia] On the missing Nullable annotation you mentioned in BEAM-8858. I
believe the SQL module 2.16 also has this missing-Nullable problem ([my
experiment|https://github.com/suztomo/beam-java-sdk-missing-nullable/tree/beam-sdks-java-extensions-sql-2.16]).
For the project using SQL module 2.*16*, would you check which Maven artifact
provides {{javax.annotation.Nullable}} and the dependency tree (or path) to the
artifact?
was (Author: suztomo):
[~iemejia] On missing Nullable annotation. I believe 2.16 also has this
missing-Nullable problem. For the project using SQL module 2.*16*, would you
check which Maven artifact provides {{javax.annotation.Nullable}} and the
dependency tree (or path) to the artifact?
> javax.annotation.Nullable is missing for
> org.apache.beam.sdk.schemas.FieldValueTypeInformation
> --
>
> Key: BEAM-8917
> URL: https://issues.apache.org/jira/browse/BEAM-8917
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> This ticket is from the result of static analysis by Linkage Checker
> ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1045])
> h1. Example Project
> Example project to produce an issue:
> https://github.com/suztomo/beam-java-sdk-missing-nullable .
> I think the Maven artifact {{org.apache.beam:beam-sdks-java-core}}, which
> contains {{org.apache.beam.sdk.schemas.FieldValueTypeInformation}}, should
> declare the dependency to {{com.google.code.findbugs:jsr305}}.
> h1. Why there's no problem in compilation and tests of sdks/java/core?
> The compilation succeeds because the {{Nullable}} annotation is in the
> transitive dependency of compileOnly {{spotbugs-annotations}} dependency:
> {noformat}
> compileOnly - Compile only dependencies for source set 'main'.
> ...
> +--- com.github.spotbugs:spotbugs-annotations:3.1.12
> |\--- com.google.code.findbugs:jsr305:3.0.2
> ...
> {noformat}
> The tests succeed because the {{Nullable}} annotation is in the transitive
> dependency of {{guava-testlib}}.
> {noformat}
> testRuntime - Runtime dependencies for source set 'test' (deprecated, use
> 'testRuntimeOnly' instead).
> ...
> +--- com.google.guava:guava-testlib:20.0
> |+--- com.google.code.findbugs:jsr305:1.3.9
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8917) javax.annotation.Nullable is missing for org.apache.beam.sdk.schemas.FieldValueTypeInformation
[
https://issues.apache.org/jira/browse/BEAM-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992115#comment-16992115
]
Tomo Suzuki commented on BEAM-8917:
---
[~iemejia] On missing Nullable annotation. I believe 2.16 also has this
missing-Nullable problem. For the project using SQL module 2.*16*, would you
check which Maven artifact provides {{javax.annotation.Nullable}} and the
dependency tree (or path) to the artifact?
> javax.annotation.Nullable is missing for
> org.apache.beam.sdk.schemas.FieldValueTypeInformation
> --
>
> Key: BEAM-8917
> URL: https://issues.apache.org/jira/browse/BEAM-8917
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> This ticket is from the result of static analysis by Linkage Checker
> ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1045])
> h1. Example Project
> Example project to produce an issue:
> https://github.com/suztomo/beam-java-sdk-missing-nullable .
> I think the Maven artifact {{org.apache.beam:beam-sdks-java-core}}, which
> contains {{org.apache.beam.sdk.schemas.FieldValueTypeInformation}}, should
> declare the dependency to {{com.google.code.findbugs:jsr305}}.
> h1. Why there's no problem in compilation and tests of sdks/java/core?
> The compilation succeeds because the {{Nullable}} annotation is in the
> transitive dependency of compileOnly {{spotbugs-annotations}} dependency:
> {noformat}
> compileOnly - Compile only dependencies for source set 'main'.
> ...
> +--- com.github.spotbugs:spotbugs-annotations:3.1.12
> |\--- com.google.code.findbugs:jsr305:3.0.2
> ...
> {noformat}
> The tests succeed because the {{Nullable}} annotation is in the transitive
> dependency of {{guava-testlib}}.
> {noformat}
> testRuntime - Runtime dependencies for source set 'test' (deprecated, use
> 'testRuntimeOnly' instead).
> ...
> +--- com.google.guava:guava-testlib:20.0
> |+--- com.google.code.findbugs:jsr305:1.3.9
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8858) Missing dependencies in SQL module pom
[
https://issues.apache.org/jira/browse/BEAM-8858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992114#comment-16992114
]
Tomo Suzuki commented on BEAM-8858:
---
[~iemejia] On missing Nullable annotation. For the project using SQL module
2.*16*, would you check which Maven artifact provides
{{javax.annotation.Nullable}} and the dependency tree (or path) to the artifact?
> Missing dependencies in SQL module pom
> --
>
> Key: BEAM-8858
> URL: https://issues.apache.org/jira/browse/BEAM-8858
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.18.0
>Reporter: Ismaël Mejía
>Assignee: Tomo Suzuki
>Priority: Blocker
> Fix For: 2.18.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> I moved an user project that depends on the SQL module to the version
> 2.18.0-SNAPSHOT and I discovered that it was not working anymore because some
> of the transitive dependencies were not correctly exposed anymore by the SQL
> module e.g. commons-codec and Nullable annotations dependency.
> There are some missing dependencies in the generated pom file when compared
> with previous versions, probably cases of `usedUndeclaredArtifacts`
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Issue Comment Deleted] (BEAM-8858) Missing dependencies in SQL module pom
[
https://issues.apache.org/jira/browse/BEAM-8858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomo Suzuki updated BEAM-8858:
--
Comment: was deleted
(was: [~iemejia] On missing Nullable annotation. For the project using SQL
module 2.*16*, would you check which Maven artifact provides
{{javax.annotation.Nullable}} and the dependency tree (or path) to the
artifact?)
> Missing dependencies in SQL module pom
> --
>
> Key: BEAM-8858
> URL: https://issues.apache.org/jira/browse/BEAM-8858
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.18.0
>Reporter: Ismaël Mejía
>Assignee: Tomo Suzuki
>Priority: Blocker
> Fix For: 2.18.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> I moved an user project that depends on the SQL module to the version
> 2.18.0-SNAPSHOT and I discovered that it was not working anymore because some
> of the transitive dependencies were not correctly exposed anymore by the SQL
> module e.g. commons-codec and Nullable annotations dependency.
> There are some missing dependencies in the generated pom file when compared
> with previous versions, probably cases of `usedUndeclaredArtifacts`
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8858) Missing dependencies in SQL module pom
[ https://issues.apache.org/jira/browse/BEAM-8858?focusedWorklogId=356772&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356772 ] ASF GitHub Bot logged work on BEAM-8858: Author: ASF GitHub Bot Created on: 10/Dec/19 02:05 Start Date: 10/Dec/19 02:05 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10332: [BEAM-8858] sdks/java/extensions/sql to declare used-but-undeclared dependencies URL: https://github.com/apache/beam/pull/10332#issuecomment-563577940 Run Java PostComm This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356772) Time Spent: 1h (was: 50m) > Missing dependencies in SQL module pom > -- > > Key: BEAM-8858 > URL: https://issues.apache.org/jira/browse/BEAM-8858 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Affects Versions: 2.18.0 >Reporter: Ismaël Mejía >Assignee: Tomo Suzuki >Priority: Blocker > Fix For: 2.18.0 > > Time Spent: 1h > Remaining Estimate: 0h > > I moved an user project that depends on the SQL module to the version > 2.18.0-SNAPSHOT and I discovered that it was not working anymore because some > of the transitive dependencies were not correctly exposed anymore by the SQL > module e.g. commons-codec and Nullable annotations dependency. > There are some missing dependencies in the generated pom file when compared > with previous versions, probably cases of `usedUndeclaredArtifacts` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8858) Missing dependencies in SQL module pom
[ https://issues.apache.org/jira/browse/BEAM-8858?focusedWorklogId=356771&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356771 ] ASF GitHub Bot logged work on BEAM-8858: Author: ASF GitHub Bot Created on: 10/Dec/19 02:04 Start Date: 10/Dec/19 02:04 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10332: [BEAM-8858] sdks/java/extensions/sql to declare used-but-undeclared dependencies URL: https://github.com/apache/beam/pull/10332#issuecomment-563577528 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356771) Time Spent: 50m (was: 40m) > Missing dependencies in SQL module pom > -- > > Key: BEAM-8858 > URL: https://issues.apache.org/jira/browse/BEAM-8858 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Affects Versions: 2.18.0 >Reporter: Ismaël Mejía >Assignee: Tomo Suzuki >Priority: Blocker > Fix For: 2.18.0 > > Time Spent: 50m > Remaining Estimate: 0h > > I moved an user project that depends on the SQL module to the version > 2.18.0-SNAPSHOT and I discovered that it was not working anymore because some > of the transitive dependencies were not correctly exposed anymore by the SQL > module e.g. commons-codec and Nullable annotations dependency. > There are some missing dependencies in the generated pom file when compared > with previous versions, probably cases of `usedUndeclaredArtifacts` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8870) beam_PostCommit_Python_VR_Spark is permanently failing
[ https://issues.apache.org/jira/browse/BEAM-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Weaver resolved BEAM-8870. --- Fix Version/s: Not applicable Resolution: Fixed > beam_PostCommit_Python_VR_Spark is permanently failing > -- > > Key: BEAM-8870 > URL: https://issues.apache.org/jira/browse/BEAM-8870 > Project: Beam > Issue Type: Bug > Components: runner-spark, test-failures >Reporter: Kenneth Knowles >Assignee: Kyle Weaver >Priority: Critical > Labels: currently-failing > Fix For: Not applicable > > Time Spent: 1h 20m > Remaining Estimate: 0h > > See https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ > Is this a known issue? Should this suite be disabled until it is expected to > pass? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8870) beam_PostCommit_Python_VR_Spark is permanently failing
[ https://issues.apache.org/jira/browse/BEAM-8870?focusedWorklogId=356770&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356770 ] ASF GitHub Bot logged work on BEAM-8870: Author: ASF GitHub Bot Created on: 10/Dec/19 02:04 Start Date: 10/Dec/19 02:04 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #10279: [BEAM-8870] Fix Spark Python VR failures. URL: https://github.com/apache/beam/pull/10279 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356770) Time Spent: 1h 20m (was: 1h 10m) > beam_PostCommit_Python_VR_Spark is permanently failing > -- > > Key: BEAM-8870 > URL: https://issues.apache.org/jira/browse/BEAM-8870 > Project: Beam > Issue Type: Bug > Components: runner-spark, test-failures >Reporter: Kenneth Knowles >Assignee: Kyle Weaver >Priority: Critical > Labels: currently-failing > Fix For: Not applicable > > Time Spent: 1h 20m > Remaining Estimate: 0h > > See https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ > Is this a known issue? Should this suite be disabled until it is expected to > pass? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8512) Add integration tests for Python "flink_runner.py"
[
https://issues.apache.org/jira/browse/BEAM-8512?focusedWorklogId=356767&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356767
]
ASF GitHub Bot logged work on BEAM-8512:
Author: ASF GitHub Bot
Created on: 10/Dec/19 02:03
Start Date: 10/Dec/19 02:03
Worklog Time Spent: 10m
Work Description: ibzib commented on issue #10284: [BEAM-8512] Add
integration tests for flink_runner.py.
URL: https://github.com/apache/beam/pull/10284#issuecomment-563575173
Run PortableJar_Flink PostCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356767)
Time Spent: 3.5h (was: 3h 20m)
> Add integration tests for Python "flink_runner.py"
> --
>
> Key: BEAM-8512
> URL: https://issues.apache.org/jira/browse/BEAM-8512
> Project: Beam
> Issue Type: Test
> Components: runner-flink, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Kyle Weaver
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> There are currently no integration tests for the Python FlinkRunner. We need
> a set of tests similar to {{flink_runner_test.py}} which currently use the
> PortableRunner and not the FlinkRunner.
> CC [~robertwb] [~ibzib] [~thw]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8858) Missing dependencies in SQL module pom
[ https://issues.apache.org/jira/browse/BEAM-8858?focusedWorklogId=356768&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356768 ] ASF GitHub Bot logged work on BEAM-8858: Author: ASF GitHub Bot Created on: 10/Dec/19 02:03 Start Date: 10/Dec/19 02:03 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10332: [BEAM-8858] sdks/java/extensions/sql to declare used-but-undeclared dependencies URL: https://github.com/apache/beam/pull/10332#issuecomment-563576204 Run Java_Examples_Dataflow PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356768) Time Spent: 0.5h (was: 20m) > Missing dependencies in SQL module pom > -- > > Key: BEAM-8858 > URL: https://issues.apache.org/jira/browse/BEAM-8858 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Affects Versions: 2.18.0 >Reporter: Ismaël Mejía >Assignee: Tomo Suzuki >Priority: Blocker > Fix For: 2.18.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > I moved an user project that depends on the SQL module to the version > 2.18.0-SNAPSHOT and I discovered that it was not working anymore because some > of the transitive dependencies were not correctly exposed anymore by the SQL > module e.g. commons-codec and Nullable annotations dependency. > There are some missing dependencies in the generated pom file when compared > with previous versions, probably cases of `usedUndeclaredArtifacts` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8858) Missing dependencies in SQL module pom
[ https://issues.apache.org/jira/browse/BEAM-8858?focusedWorklogId=356769&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356769 ] ASF GitHub Bot logged work on BEAM-8858: Author: ASF GitHub Bot Created on: 10/Dec/19 02:03 Start Date: 10/Dec/19 02:03 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10332: [BEAM-8858] sdks/java/extensions/sql to declare used-but-undeclared dependencies URL: https://github.com/apache/beam/pull/10332#issuecomment-563576445 Run JavaPortabilityApi PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356769) Time Spent: 40m (was: 0.5h) > Missing dependencies in SQL module pom > -- > > Key: BEAM-8858 > URL: https://issues.apache.org/jira/browse/BEAM-8858 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Affects Versions: 2.18.0 >Reporter: Ismaël Mejía >Assignee: Tomo Suzuki >Priority: Blocker > Fix For: 2.18.0 > > Time Spent: 40m > Remaining Estimate: 0h > > I moved an user project that depends on the SQL module to the version > 2.18.0-SNAPSHOT and I discovered that it was not working anymore because some > of the transitive dependencies were not correctly exposed anymore by the SQL > module e.g. commons-codec and Nullable annotations dependency. > There are some missing dependencies in the generated pom file when compared > with previous versions, probably cases of `usedUndeclaredArtifacts` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8911) Upgrade Guava to 25
[
https://issues.apache.org/jira/browse/BEAM-8911?focusedWorklogId=356766&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356766
]
ASF GitHub Bot logged work on BEAM-8911:
Author: ASF GitHub Bot
Created on: 10/Dec/19 02:01
Start Date: 10/Dec/19 02:01
Worklog Time Spent: 10m
Work Description: suztomo commented on issue #10318: [BEAM-8911] New
non-vendored Guava version: 25.1-jre
URL: https://github.com/apache/beam/pull/10318#issuecomment-563572926
Run Dataflow ValidatesRunner
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356766)
Time Spent: 4h 50m (was: 4h 40m)
> Upgrade Guava to 25
> ---
>
> Key: BEAM-8911
> URL: https://issues.apache.org/jira/browse/BEAM-8911
> Project: Beam
> Issue Type: Improvement
> Components: build-system
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> Now [BEAM-8822] Hadoop client version 2.8.5 from 2.7 is merged. It's time to
> upgrade Guava.
> h1. New Guava version: 25.1-jre
> CharMatcher.DIGIT, used by
> org.apache.cassandra.io.sstable.format.SSTableFormat, is removed since Guava
> 26.0. 25.1 is the latest version:
> https://guava.dev/releases/25.1-jre/api/docs/com/google/common/base/CharMatcher.html
> https://guava.dev/releases/26.0-jre/api/docs/com/google/common/base/CharMatcher.html
> Guava has 2 variations "-jre" for Java8 (stream/lambda support) and
> "-android" for Java7. Gcsio:1.9.16, which Beam has not have in dependency
> tree yet, needs ImmutableSet.toImmutableSet() which is available in Guava
> "-jre".
> https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557680928
> h2. What about upgrading Cassandra-all?
> The latest cassandra-all release is "4.0-alpha2", which is still alpha.
> Non-alpha version is 3.11.5
> https://search.maven.org/artifact/org.apache.cassandra/cassandra-all/3.11.5/jar,
> which still relies on Guava 19 and it still has reference to DIGIT:
> {noformat}
> suztomo@suxtomo24:~/grpc-java$ javap -v -cp
> ~/Downloads/cassandra-all-3.11.5.jar
> 'org.apache.cassandra.io.sstable.format.SSTableFormat$Type' |grep DIGIT
> #9 = Fieldref #79.#80 //
> com/google/common/base/CharMatcher.DIGIT:Lcom/google/common/base/CharMatcher;
>#80 = NameAndType#110:#111 //
> DIGIT:Lcom/google/common/base/CharMatcher;
> #110 = Utf8 DIGIT
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8917) javax.annotation.Nullable is missing for org.apache.beam.sdk.schemas.FieldValueTypeInformation
[
https://issues.apache.org/jira/browse/BEAM-8917?focusedWorklogId=356764&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356764
]
ASF GitHub Bot logged work on BEAM-8917:
Author: ASF GitHub Bot
Created on: 10/Dec/19 02:00
Start Date: 10/Dec/19 02:00
Worklog Time Spent: 10m
Work Description: suztomo commented on issue #10324: [BEAM-8917] jsr305
dependency declaration for Nullable class
URL: https://github.com/apache/beam/pull/10324#issuecomment-563571876
Run Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356764)
Time Spent: 2h 40m (was: 2.5h)
> javax.annotation.Nullable is missing for
> org.apache.beam.sdk.schemas.FieldValueTypeInformation
> --
>
> Key: BEAM-8917
> URL: https://issues.apache.org/jira/browse/BEAM-8917
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> This ticket is from the result of static analysis by Linkage Checker
> ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1045])
> h1. Example Project
> Example project to produce an issue:
> https://github.com/suztomo/beam-java-sdk-missing-nullable .
> I think the Maven artifact {{org.apache.beam:beam-sdks-java-core}}, which
> contains {{org.apache.beam.sdk.schemas.FieldValueTypeInformation}}, should
> declare the dependency to {{com.google.code.findbugs:jsr305}}.
> h1. Why there's no problem in compilation and tests of sdks/java/core?
> The compilation succeeds because the {{Nullable}} annotation is in the
> transitive dependency of compileOnly {{spotbugs-annotations}} dependency:
> {noformat}
> compileOnly - Compile only dependencies for source set 'main'.
> ...
> +--- com.github.spotbugs:spotbugs-annotations:3.1.12
> |\--- com.google.code.findbugs:jsr305:3.0.2
> ...
> {noformat}
> The tests succeed because the {{Nullable}} annotation is in the transitive
> dependency of {{guava-testlib}}.
> {noformat}
> testRuntime - Runtime dependencies for source set 'test' (deprecated, use
> 'testRuntimeOnly' instead).
> ...
> +--- com.google.guava:guava-testlib:20.0
> |+--- com.google.code.findbugs:jsr305:1.3.9
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8917) javax.annotation.Nullable is missing for org.apache.beam.sdk.schemas.FieldValueTypeInformation
[
https://issues.apache.org/jira/browse/BEAM-8917?focusedWorklogId=356765&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356765
]
ASF GitHub Bot logged work on BEAM-8917:
Author: ASF GitHub Bot
Created on: 10/Dec/19 02:00
Start Date: 10/Dec/19 02:00
Worklog Time Spent: 10m
Work Description: suztomo commented on issue #10324: [BEAM-8917] jsr305
dependency declaration for Nullable class
URL: https://github.com/apache/beam/pull/10324#issuecomment-563572082
Run Dataflow ValidatesRunner
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356765)
Time Spent: 2h 50m (was: 2h 40m)
> javax.annotation.Nullable is missing for
> org.apache.beam.sdk.schemas.FieldValueTypeInformation
> --
>
> Key: BEAM-8917
> URL: https://issues.apache.org/jira/browse/BEAM-8917
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> This ticket is from the result of static analysis by Linkage Checker
> ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1045])
> h1. Example Project
> Example project to produce an issue:
> https://github.com/suztomo/beam-java-sdk-missing-nullable .
> I think the Maven artifact {{org.apache.beam:beam-sdks-java-core}}, which
> contains {{org.apache.beam.sdk.schemas.FieldValueTypeInformation}}, should
> declare the dependency to {{com.google.code.findbugs:jsr305}}.
> h1. Why there's no problem in compilation and tests of sdks/java/core?
> The compilation succeeds because the {{Nullable}} annotation is in the
> transitive dependency of compileOnly {{spotbugs-annotations}} dependency:
> {noformat}
> compileOnly - Compile only dependencies for source set 'main'.
> ...
> +--- com.github.spotbugs:spotbugs-annotations:3.1.12
> |\--- com.google.code.findbugs:jsr305:3.0.2
> ...
> {noformat}
> The tests succeed because the {{Nullable}} annotation is in the transitive
> dependency of {{guava-testlib}}.
> {noformat}
> testRuntime - Runtime dependencies for source set 'test' (deprecated, use
> 'testRuntimeOnly' instead).
> ...
> +--- com.google.guava:guava-testlib:20.0
> |+--- com.google.code.findbugs:jsr305:1.3.9
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests
[
https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=356762&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356762
]
ASF GitHub Bot logged work on BEAM-8575:
Author: ASF GitHub Bot
Created on: 10/Dec/19 01:56
Start Date: 10/Dec/19 01:56
Worklog Time Spent: 10m
Work Description: HuangLED commented on pull request #10050: [BEAM-8575]
Add streaming test case for multi-triggered GBK as side input
URL: https://github.com/apache/beam/pull/10050#discussion_r355809332
##
File path: sdks/python/apache_beam/transforms/sideinputs_test.py
##
@@ -309,6 +316,65 @@ def test_flattened_side_input(self):
assert_that(results, equal_to(['a', 'b']))
pipeline.run()
+ @attr('ValidatesRunner')
+ def test_multi_triggered_gbk_side_input(self):
+"""Test a GBK sideinput, with multiple triggering."""
+options = PipelineOptions()
+options.view_as(StandardOptions).streaming = True
+p = TestPipeline(options=options)
+
+main_stream = (p
+ | 'main TestStream' >> TestStream()
+ .advance_watermark_to(3)
+ .add_elements(['a1'])
+ .advance_watermark_to(8)
+ .add_elements(['a2'])
+ | 'main windowInto' >> beam.WindowInto(
+ window.FixedWindows(5),
+ trigger=trigger.AfterWatermark(
+ early=trigger.AfterCount(1)),
+ accumulation_mode=trigger.AccumulationMode.DISCARDING))
+
+emit_vals = Map(lambda k_vs: k_vs[1])
+
+side_stream = (p
+ | 'side TestStream' >> TestStream()
+ .add_elements([window.TimestampedValue(('k', 100), 2)])
+ .add_elements([window.TimestampedValue(('k', 400), 7)])
+ | 'side windowInto' >> beam.WindowInto(
+ window.FixedWindows(5),
+ trigger=trigger.AfterWatermark(
+ early=trigger.AfterCount(1)),
+ accumulation_mode=trigger.AccumulationMode.DISCARDING)
+ | 'GBK' >> beam.GroupByKey()
+ | 'Values' >> emit_vals)
+
+class RecordFn(beam.DoFn):
+ def process(self,
+ elm=beam.DoFn.ElementParam,
+ ts=beam.DoFn.TimestampParam,
+ side=beam.DoFn.SideInputParam):
+yield (elm, ts, side)
+
+records = (main_stream
+ | beam.ParDo(RecordFn(), beam.pvalue.AsList(side_stream)))
+
+expected_window_to_elements = {
+window.IntervalWindow(0, 5): [
+('a1', Timestamp(3), [[100]]),
+],
+window.IntervalWindow(5, 10): [
+('a2', Timestamp(8), [[400], []])
Review comment:
issue is gone after using multiple output test stream
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356762)
Time Spent: 30.5h (was: 30h 20m)
> Add more Python validates runner tests
> --
>
> Key: BEAM-8575
> URL: https://issues.apache.org/jira/browse/BEAM-8575
> Project: Beam
> Issue Type: Test
> Components: sdk-py-core, testing
>Reporter: wendy liu
>Assignee: wendy liu
>Priority: Major
> Time Spent: 30.5h
> Remaining Estimate: 0h
>
> This is the umbrella issue to track the work of adding more Python tests to
> improve test coverage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests
[
https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=356759&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356759
]
ASF GitHub Bot logged work on BEAM-8575:
Author: ASF GitHub Bot
Created on: 10/Dec/19 01:52
Start Date: 10/Dec/19 01:52
Worklog Time Spent: 10m
Work Description: HuangLED commented on pull request #10050: [BEAM-8575]
Add streaming test case for multi-triggered GBK as side input
URL: https://github.com/apache/beam/pull/10050#discussion_r355808479
##
File path: sdks/python/apache_beam/transforms/sideinputs_test.py
##
@@ -309,6 +316,65 @@ def test_flattened_side_input(self):
assert_that(results, equal_to(['a', 'b']))
pipeline.run()
+ @attr('ValidatesRunner')
+ def test_multi_triggered_gbk_side_input(self):
+"""Test a GBK sideinput, with multiple triggering."""
+options = PipelineOptions()
+options.view_as(StandardOptions).streaming = True
+p = TestPipeline(options=options)
+
+main_stream = (p
+ | 'main TestStream' >> TestStream()
+ .advance_watermark_to(3)
+ .add_elements(['a1'])
+ .advance_watermark_to(8)
+ .add_elements(['a2'])
+ | 'main windowInto' >> beam.WindowInto(
+ window.FixedWindows(5),
+ trigger=trigger.AfterWatermark(
Review comment:
Without trigger on main stream, how do we expect multiple outputs in the
final result?
The current sequence is:
1) side input triggers
2) main triggers ==> leads to output #1
3) side input triggers (thus different side input)
4) main triggers ==> leads to output #2
Does that make sense? or did I miss anything?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356759)
Time Spent: 30h 20m (was: 30h 10m)
> Add more Python validates runner tests
> --
>
> Key: BEAM-8575
> URL: https://issues.apache.org/jira/browse/BEAM-8575
> Project: Beam
> Issue Type: Test
> Components: sdk-py-core, testing
>Reporter: wendy liu
>Assignee: wendy liu
>Priority: Major
> Time Spent: 30h 20m
> Remaining Estimate: 0h
>
> This is the umbrella issue to track the work of adding more Python tests to
> improve test coverage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests
[
https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=356755&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356755
]
ASF GitHub Bot logged work on BEAM-8575:
Author: ASF GitHub Bot
Created on: 10/Dec/19 01:50
Start Date: 10/Dec/19 01:50
Worklog Time Spent: 10m
Work Description: HuangLED commented on pull request #10050: [BEAM-8575]
Add streaming test case for multi-triggered GBK as side input
URL: https://github.com/apache/beam/pull/10050#discussion_r355808044
##
File path: sdks/python/apache_beam/transforms/sideinputs_test.py
##
@@ -309,6 +316,65 @@ def test_flattened_side_input(self):
assert_that(results, equal_to(['a', 'b']))
pipeline.run()
+ @attr('ValidatesRunner')
+ def test_multi_triggered_gbk_side_input(self):
+"""Test a GBK sideinput, with multiple triggering."""
+options = PipelineOptions()
+options.view_as(StandardOptions).streaming = True
+p = TestPipeline(options=options)
+
+main_stream = (p
+ | 'main TestStream' >> TestStream()
+ .advance_watermark_to(3)
+ .add_elements(['a1'])
+ .advance_watermark_to(8)
+ .add_elements(['a2'])
+ | 'main windowInto' >> beam.WindowInto(
+ window.FixedWindows(5),
+ trigger=trigger.AfterWatermark(
+ early=trigger.AfterCount(1)),
+ accumulation_mode=trigger.AccumulationMode.DISCARDING))
+
+emit_vals = Map(lambda k_vs: k_vs[1])
+
+side_stream = (p
+ | 'side TestStream' >> TestStream()
Review comment:
fixed by using test stream's multiple output.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356755)
Time Spent: 30h 10m (was: 30h)
> Add more Python validates runner tests
> --
>
> Key: BEAM-8575
> URL: https://issues.apache.org/jira/browse/BEAM-8575
> Project: Beam
> Issue Type: Test
> Components: sdk-py-core, testing
>Reporter: wendy liu
>Assignee: wendy liu
>Priority: Major
> Time Spent: 30h 10m
> Remaining Estimate: 0h
>
> This is the umbrella issue to track the work of adding more Python tests to
> improve test coverage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-3713) Consider moving away from nose to nose2 or pytest.
[ https://issues.apache.org/jira/browse/BEAM-3713?focusedWorklogId=356750&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356750 ] ASF GitHub Bot logged work on BEAM-3713: Author: ASF GitHub Bot Created on: 10/Dec/19 01:37 Start Date: 10/Dec/19 01:37 Worklog Time Spent: 10m Work Description: udim commented on pull request #10322: [BEAM-3713] pytest migration: py27-gcp-pytest URL: https://github.com/apache/beam/pull/10322 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356750) Time Spent: 13h 50m (was: 13h 40m) > Consider moving away from nose to nose2 or pytest. > -- > > Key: BEAM-3713 > URL: https://issues.apache.org/jira/browse/BEAM-3713 > Project: Beam > Issue Type: Test > Components: sdk-py-core, testing >Reporter: Robert Bradshaw >Assignee: Udi Meiri >Priority: Minor > Time Spent: 13h 50m > Remaining Estimate: 0h > > Per > [https://nose.readthedocs.io/en/latest/|https://nose.readthedocs.io/en/latest/,] > , nose is in maintenance mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8932) Expose complete Cloud Pub/Sub messages through PubsubIO API
[ https://issues.apache.org/jira/browse/BEAM-8932?focusedWorklogId=356738&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356738 ] ASF GitHub Bot logged work on BEAM-8932: Author: ASF GitHub Bot Created on: 10/Dec/19 01:28 Start Date: 10/Dec/19 01:28 Worklog Time Spent: 10m Work Description: boyuanzz commented on issue #10331: [BEAM-8932] Modify PubsubClient to use the proto message throughout. URL: https://github.com/apache/beam/pull/10331#issuecomment-563529512 Please fix unit test failure: `org.apache.beam.sdk.io.gcp.GcpApiSurfaceTest.testGcpApiSurface`. For more failure details, please refer to: https://builds.apache.org/job/beam_PreCommit_Java_Commit/9162/testReport/junit/org.apache.beam.sdk.io.gcp/GcpApiSurfaceTest/testGcpApiSurface/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356738) Time Spent: 40m (was: 0.5h) > Expose complete Cloud Pub/Sub messages through PubsubIO API > --- > > Key: BEAM-8932 > URL: https://issues.apache.org/jira/browse/BEAM-8932 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Daniel Collins >Assignee: Daniel Collins >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > The PubsubIO API only exposes a subset of the fields in the underlying > PubsubMessage protocol buffer. To accomodate future feature changes as well > as for greater compatability with code using the Cloud Pub/Sub apis, a method > to read and write these protocol messages should be exposed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8932) Expose complete Cloud Pub/Sub messages through PubsubIO API
[ https://issues.apache.org/jira/browse/BEAM-8932?focusedWorklogId=356736&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356736 ] ASF GitHub Bot logged work on BEAM-8932: Author: ASF GitHub Bot Created on: 10/Dec/19 01:25 Start Date: 10/Dec/19 01:25 Worklog Time Spent: 10m Work Description: boyuanzz commented on issue #10331: [BEAM-8932] Modify PubsubClient to use the proto message throughout. URL: https://github.com/apache/beam/pull/10331#issuecomment-563524796 Please run `./gradlew spotlessApply` under your root dir of beam project to address `Spotless` failures. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356736) Time Spent: 0.5h (was: 20m) > Expose complete Cloud Pub/Sub messages through PubsubIO API > --- > > Key: BEAM-8932 > URL: https://issues.apache.org/jira/browse/BEAM-8932 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Daniel Collins >Assignee: Daniel Collins >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > The PubsubIO API only exposes a subset of the fields in the underlying > PubsubMessage protocol buffer. To accomodate future feature changes as well > as for greater compatability with code using the Cloud Pub/Sub apis, a method > to read and write these protocol messages should be exposed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8811) Upgrade Beam pipeline diagrams in docs
[ https://issues.apache.org/jira/browse/BEAM-8811?focusedWorklogId=356712&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356712 ] ASF GitHub Bot logged work on BEAM-8811: Author: ASF GitHub Bot Created on: 10/Dec/19 00:55 Start Date: 10/Dec/19 00:55 Worklog Time Spent: 10m Work Description: rosetn commented on issue #10200: [BEAM-8811] Upgrade Beam pipeline diagrams in docs URL: https://github.com/apache/beam/pull/10200#issuecomment-563509970 Your changes look good! A wrapped flowchart might be a good compromise on web page space and text size readability here. The longer figures can be split up, maintaining the left to right reading pattern e.g. https://wcs.smartdraw.com/flowchart/img/wrap-flowchart.jpg. Or you can do the same thing vertically e.g. https://www.researchgate.net/figure/s-a-flowchart-for-wrapping-process-that-has-been-clearly-done-after-the-host-and-the_fig5_320295347 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356712) Time Spent: 2h 50m (was: 2h 40m) > Upgrade Beam pipeline diagrams in docs > -- > > Key: BEAM-8811 > URL: https://issues.apache.org/jira/browse/BEAM-8811 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Time Spent: 2h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8337) Add Flink job server container images to release process
[ https://issues.apache.org/jira/browse/BEAM-8337?focusedWorklogId=356707&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356707 ] ASF GitHub Bot logged work on BEAM-8337: Author: ASF GitHub Bot Created on: 10/Dec/19 00:49 Start Date: 10/Dec/19 00:49 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #10335: Cherry-pick #10272 [BEAM-8337] publish Flink job server container images URL: https://github.com/apache/beam/pull/10335 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[ Show PCollection with Interactive Beam in a data-centric user flow
[
https://issues.apache.org/jira/browse/BEAM-7926?focusedWorklogId=356706&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356706
]
ASF GitHub Bot logged work on BEAM-7926:
Author: ASF GitHub Bot
Created on: 10/Dec/19 00:46
Start Date: 10/Dec/19 00:46
Worklog Time Spent: 10m
Work Description: aaltay commented on pull request #10276: [BEAM-7926]
Data-centric Interactive Part1
URL: https://github.com/apache/beam/pull/10276
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356706)
Time Spent: 26h 40m (was: 26.5h)
> Show PCollection with Interactive Beam in a data-centric user flow
> --
>
> Key: BEAM-7926
> URL: https://issues.apache.org/jira/browse/BEAM-7926
> Project: Beam
> Issue Type: New Feature
> Components: runner-py-interactive
>Reporter: Ning Kang
>Assignee: Ning Kang
>Priority: Major
> Time Spent: 26h 40m
> Remaining Estimate: 0h
>
> Support auto plotting / charting of materialized data of a given PCollection
> with Interactive Beam.
> Say an Interactive Beam pipeline defined as
>
> {code:java}
> p = beam.Pipeline(InteractiveRunner())
> pcoll = p | 'Transform' >> transform()
> pcoll2 = ...
> pcoll3 = ...{code}
> The use can call a single function and get auto-magical charting of the data.
> e.g.,
> {code:java}
> show(pcoll, pcoll2)
> {code}
> Throughout the process, a pipeline fragment is built to include only
> transforms necessary to produce the desired pcolls (pcoll and pcoll2) and
> execute that fragment.
> This makes the Interactive Beam user flow data-centric.
>
> Detailed
> [design|https://docs.google.com/document/d/1DYWrT6GL_qDCXhRMoxpjinlVAfHeVilK5Mtf8gO6zxQ/edit#heading=h.v6k2o3roarzz].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8274) Remove dependency on experiments=beam_fn_api flag from PortableRunner
[
https://issues.apache.org/jira/browse/BEAM-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992061#comment-16992061
]
Kyle Weaver commented on BEAM-8274:
---
Do we still want to do this? Enabling beam_fn_api by default seems like an
adequate solution, so I'm not sure we want to do a deeper change unless there's
a real benefit.
> Remove dependency on experiments=beam_fn_api flag from PortableRunner
> -
>
> Key: BEAM-8274
> URL: https://issues.apache.org/jira/browse/BEAM-8274
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Maximilian Michels
>Priority: Major
>
> The execution path of Python's {{PortableRunner}} depends on the
> {{experiments=beam_fn_api}} flag. This flag is set automatically but has been
> broken in the past, e.g. when the {{FlinkRunner}} was subclassed from the
> {{PortableRunner}} (BEAM-8037).
> We should get rid of the flag, as the only supported execution path for the
> PortableRunner is the BeamFnApi. Perhaps we can move the flag to the Dataflow
> runners which may still depend on it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8337) Add Flink job server container images to release process
[ https://issues.apache.org/jira/browse/BEAM-8337?focusedWorklogId=356703&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356703 ] ASF GitHub Bot logged work on BEAM-8337: Author: ASF GitHub Bot Created on: 10/Dec/19 00:38 Start Date: 10/Dec/19 00:38 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #10272: [BEAM-8337] publish Flink job server container images URL: https://github.com/apache/beam/pull/10272 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356703) Time Spent: 3.5h (was: 3h 20m) > Add Flink job server container images to release process > > > Key: BEAM-8337 > URL: https://issues.apache.org/jira/browse/BEAM-8337 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > Could be added to the release process similar to how we now publish SDK > worker images. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8536) Migrate usage of DelayedBundleApplication.requested_execution_time to time duration
[ https://issues.apache.org/jira/browse/BEAM-8536?focusedWorklogId=356701&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356701 ] ASF GitHub Bot logged work on BEAM-8536: Author: ASF GitHub Bot Created on: 10/Dec/19 00:35 Start Date: 10/Dec/19 00:35 Worklog Time Spent: 10m Work Description: boyuanzz commented on issue #10333: [BEAM-8536] Migrate request_execution_time to request_delay_time in Java URL: https://github.com/apache/beam/pull/10333#issuecomment-563504915 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356701) Time Spent: 20m (was: 10m) > Migrate usage of DelayedBundleApplication.requested_execution_time to time > duration > > > Key: BEAM-8536 > URL: https://issues.apache.org/jira/browse/BEAM-8536 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow, sdk-java-harness >Reporter: Boyuan Zhang >Assignee: Boyuan Zhang >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > In DelayedBundleApplication, we used to use an absolute time to represent > rescheduling time. We want to switch to use a relative time duration, which > requires a migration in Java SDK and dataflow java runner harness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=356699&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356699 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 10/Dec/19 00:31 Start Date: 10/Dec/19 00:31 Worklog Time Spent: 10m Work Description: chadrik commented on issue #9915: [BEAM-7746] Add python type hints (part 1) URL: https://github.com/apache/beam/pull/9915#issuecomment-563503862 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356699) Time Spent: 34.5h (was: 34h 20m) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 34.5h > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8835) Artifact retrieval fails with FlinkUberJarJobServer
[ https://issues.apache.org/jira/browse/BEAM-8835?focusedWorklogId=356696&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356696 ] ASF GitHub Bot logged work on BEAM-8835: Author: ASF GitHub Bot Created on: 10/Dec/19 00:16 Start Date: 10/Dec/19 00:16 Worklog Time Spent: 10m Work Description: ibzib commented on issue #10320: [BEAM-8835] Cherry-pick #10308 onto 2.18.0 release branch URL: https://github.com/apache/beam/pull/10320#issuecomment-563500299 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356696) Time Spent: 4h 20m (was: 4h 10m) > Artifact retrieval fails with FlinkUberJarJobServer > --- > > Key: BEAM-8835 > URL: https://issues.apache.org/jira/browse/BEAM-8835 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Fix For: 2.18.0 > > Time Spent: 4h 20m > Remaining Estimate: 0h > > We seem to be able to stage artifacts and retrieve the manifest fine, but > retrieving the artifacts doesn't work. This happens on both my k8s Flink > cluster and on my local Flink cluster. At a quick glance the artifact is in > the jar where it should be. cc [~robertwb] > 2019-11-21 18:43:39,336 INFO > org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService > - GetArtifact name: "pickled_main_session" > retrieval_token: "BEAM-PIPELINE/pipeline/artifact-manifest.json" > failed > java.io.IOException: Unable to load > e1d24d848414cecf805a7b5c2b950c6430c20eb32875dac00b40f80f3c73a141/ea0d10d07f4601782ed647e8f6ba4a055be13674ab79fa0c6e2fa44917c5264c > with > org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader@785297ac -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8512) Add integration tests for Python "flink_runner.py"
[
https://issues.apache.org/jira/browse/BEAM-8512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992050#comment-16992050
]
Kyle Weaver commented on BEAM-8512:
---
For the record: 1) was because the classes that needed to do the loading of the
pipeline resources were being loaded by the mini cluster, not the submitted job
jar, so the classloader was wrong.
> Add integration tests for Python "flink_runner.py"
> --
>
> Key: BEAM-8512
> URL: https://issues.apache.org/jira/browse/BEAM-8512
> Project: Beam
> Issue Type: Test
> Components: runner-flink, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Kyle Weaver
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> There are currently no integration tests for the Python FlinkRunner. We need
> a set of tests similar to {{flink_runner_test.py}} which currently use the
> PortableRunner and not the FlinkRunner.
> CC [~robertwb] [~ibzib] [~thw]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8343) Add means for IO APIs to support predicate and/or project push-down when running SQL pipelines
[ https://issues.apache.org/jira/browse/BEAM-8343?focusedWorklogId=356693&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356693 ] ASF GitHub Bot logged work on BEAM-8343: Author: ASF GitHub Bot Created on: 10/Dec/19 00:11 Start Date: 10/Dec/19 00:11 Worklog Time Spent: 10m Work Description: apilloud commented on issue #10060: [BEAM-8343] [SQL] Updated the cost model to favor IO with push-down. URL: https://github.com/apache/beam/pull/10060#issuecomment-563499121 Run SQL PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356693) Time Spent: 7.5h (was: 7h 20m) > Add means for IO APIs to support predicate and/or project push-down when > running SQL pipelines > -- > > Key: BEAM-8343 > URL: https://issues.apache.org/jira/browse/BEAM-8343 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Fix For: 2.18.0 > > Time Spent: 7.5h > Remaining Estimate: 0h > > The objective is to create a universal way for Beam SQL IO APIs to support > predicate/project push-down. > A proposed way to achieve that is by introducing an interface responsible > for identifying what portion(s) of a Calc can be moved down to IO layer. > Also, adding following methods to a BeamSqlTable interface to pass necessary > parameters to IO APIs: > - BeamSqlTableFilter constructFilter(List filter) > - ProjectSupport supportsProjects() > - PCollection buildIOReader(PBegin begin, BeamSqlTableFilter filters, > List fieldNames) > > ProjectSupport is an enum with the following options: > * NONE > * WITHOUT_FIELD_REORDERING > * WITH_FIELD_REORDERING > > Design doc > [link|https://docs.google.com/document/d/1-ysD7U7qF3MAmSfkbXZO_5PLJBevAL9bktlLCerd_jE/edit?usp=sharing]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8343) Add means for IO APIs to support predicate and/or project push-down when running SQL pipelines
[ https://issues.apache.org/jira/browse/BEAM-8343?focusedWorklogId=356692&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356692 ] ASF GitHub Bot logged work on BEAM-8343: Author: ASF GitHub Bot Created on: 10/Dec/19 00:10 Start Date: 10/Dec/19 00:10 Worklog Time Spent: 10m Work Description: apilloud commented on issue #10060: [BEAM-8343] [SQL] Updated the cost model to favor IO with push-down. URL: https://github.com/apache/beam/pull/10060#issuecomment-563498844 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356692) Time Spent: 7h 20m (was: 7h 10m) > Add means for IO APIs to support predicate and/or project push-down when > running SQL pipelines > -- > > Key: BEAM-8343 > URL: https://issues.apache.org/jira/browse/BEAM-8343 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Fix For: 2.18.0 > > Time Spent: 7h 20m > Remaining Estimate: 0h > > The objective is to create a universal way for Beam SQL IO APIs to support > predicate/project push-down. > A proposed way to achieve that is by introducing an interface responsible > for identifying what portion(s) of a Calc can be moved down to IO layer. > Also, adding following methods to a BeamSqlTable interface to pass necessary > parameters to IO APIs: > - BeamSqlTableFilter constructFilter(List filter) > - ProjectSupport supportsProjects() > - PCollection buildIOReader(PBegin begin, BeamSqlTableFilter filters, > List fieldNames) > > ProjectSupport is an enum with the following options: > * NONE > * WITHOUT_FIELD_REORDERING > * WITH_FIELD_REORDERING > > Design doc > [link|https://docs.google.com/document/d/1-ysD7U7qF3MAmSfkbXZO_5PLJBevAL9bktlLCerd_jE/edit?usp=sharing]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8512) Add integration tests for Python "flink_runner.py"
[
https://issues.apache.org/jira/browse/BEAM-8512?focusedWorklogId=356685&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356685
]
ASF GitHub Bot logged work on BEAM-8512:
Author: ASF GitHub Bot
Created on: 09/Dec/19 23:56
Start Date: 09/Dec/19 23:56
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #10284: [BEAM-8512] Add
integration tests for flink_runner.py.
URL: https://github.com/apache/beam/pull/10284#discussion_r355749948
##
File path: runners/flink/job-server/flink_job_server.gradle
##
@@ -90,6 +90,7 @@ dependencies {
runtime project(":sdks:java:io:kafka")
runtime library.java.kafka_clients
runtime project(":sdks:java:io:google-cloud-platform")
+ runtime project(path: flinkRunnerProject, configuration: "miniCluster")
Review comment:
Alright, I've updated this PR to build a separate miniCluster jar that
includes/excludes the correct stuff. PTAL
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356685)
Time Spent: 3h 20m (was: 3h 10m)
> Add integration tests for Python "flink_runner.py"
> --
>
> Key: BEAM-8512
> URL: https://issues.apache.org/jira/browse/BEAM-8512
> Project: Beam
> Issue Type: Test
> Components: runner-flink, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Kyle Weaver
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> There are currently no integration tests for the Python FlinkRunner. We need
> a set of tests similar to {{flink_runner_test.py}} which currently use the
> PortableRunner and not the FlinkRunner.
> CC [~robertwb] [~ibzib] [~thw]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8930) External workers should receive artifact endpoint when started from python
[
https://issues.apache.org/jira/browse/BEAM-8930?focusedWorklogId=356683&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356683
]
ASF GitHub Bot logged work on BEAM-8930:
Author: ASF GitHub Bot
Created on: 09/Dec/19 23:54
Start Date: 09/Dec/19 23:54
Worklog Time Spent: 10m
Work Description: chadrik commented on issue #10328: [BEAM-8930] Pass
artifact and provision endpoints to external workers from python
URL: https://github.com/apache/beam/pull/10328#issuecomment-563494421
Run Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356683)
Time Spent: 40m (was: 0.5h)
> External workers should receive artifact endpoint when started from python
> --
>
> Key: BEAM-8930
> URL: https://issues.apache.org/jira/browse/BEAM-8930
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> {{ExternalWorkerHandler}} does not pass the artifact and provision endpoints,
> making it impossible to provision artifacts when the external worker is
> started from python. The Java code is properly sending this information.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8512) Add integration tests for Python "flink_runner.py"
[
https://issues.apache.org/jira/browse/BEAM-8512?focusedWorklogId=356682&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356682
]
ASF GitHub Bot logged work on BEAM-8512:
Author: ASF GitHub Bot
Created on: 09/Dec/19 23:53
Start Date: 09/Dec/19 23:53
Worklog Time Spent: 10m
Work Description: ibzib commented on issue #10284: [BEAM-8512] Add
integration tests for flink_runner.py.
URL: https://github.com/apache/beam/pull/10284#issuecomment-563494062
Run PortableJar_Flink PostCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356682)
Time Spent: 3h 10m (was: 3h)
> Add integration tests for Python "flink_runner.py"
> --
>
> Key: BEAM-8512
> URL: https://issues.apache.org/jira/browse/BEAM-8512
> Project: Beam
> Issue Type: Test
> Components: runner-flink, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Kyle Weaver
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> There are currently no integration tests for the Python FlinkRunner. We need
> a set of tests similar to {{flink_runner_test.py}} which currently use the
> PortableRunner and not the FlinkRunner.
> CC [~robertwb] [~ibzib] [~thw]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8613) Add environment variable support to Docker environment
[
https://issues.apache.org/jira/browse/BEAM-8613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992035#comment-16992035
]
Chad Dombrova commented on BEAM-8613:
-
{quote}What kind of environment variables are you trying to pass here?
{quote}
We're primarily interested in configuring various libraries and applications
used by our UDFs. These each have their own set of environment variables which
typically need to be configured before modules are imported.
Another use case which we intend to explore soon is passing env vars to control
the behavior of pip in {{boot}}. For example, to point it at our internal pypi
mirror. Do you think this falls into the category of "building too much into
these (unstructured) string fields"?
{quote}Is there not another way to pass this data to the operations being
performed in this container?
{quote}
Let's frame this as a user story:
"As a developer, I want to set library- and application-specific env variables
(usually third-party) in the SDK process before any affected modules are
imported, so that I can bind a particular configuration to a job."
Let's evaluate a few options:
- custom PipelineOptions: by the time we can read the pipeline options, our
UDF and its pcollection element types have been unpickled, thereby importing
many dependent modules.
- custom config file uploaded to artifact service: same problem as above.
- custom docker container: we don't want to create a new docker container for
every permutation that we might need. we want this to be user controlled at job
submission time
- custom docker ARGS: theoretically if we had a custom docker container with a
custom entrypoint script and the ability to configure docker args via the
DOCKER environment we could get this to work. this just seems needlessly
complicated. we already have the ability to set env vars for PROCESS
environment type, so doing the same for DOCKER seems natural.
I'm not sure what other good options there are. Environment variables seem like
the most direct and generally useful approach.
> Add environment variable support to Docker environment
> --
>
> Key: BEAM-8613
> URL: https://issues.apache.org/jira/browse/BEAM-8613
> Project: Beam
> Issue Type: Improvement
> Components: java-fn-execution, runner-core, runner-direct
>Reporter: Nathan Rusch
>Assignee: Nathan Rusch
>Priority: Trivial
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The Process environment allows specifying environment variables via a map
> field on its payload message. The Docker environment should support this same
> pattern, and forward the contents of the map through to the container runtime.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8905) matching Java PCollectionTuple translation naming convention in expansion service with index only
[
https://issues.apache.org/jira/browse/BEAM-8905?focusedWorklogId=356669&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356669
]
ASF GitHub Bot logged work on BEAM-8905:
Author: ASF GitHub Bot
Created on: 09/Dec/19 23:36
Start Date: 09/Dec/19 23:36
Worklog Time Spent: 10m
Work Description: ihji commented on issue #10310: [BEAM-8905] matching
Java PCollectionTuple translation naming convention in expansion service
URL: https://github.com/apache/beam/pull/10310#issuecomment-563489886
Run Java_Examples_Dataflow PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356669)
Time Spent: 1h 20m (was: 1h 10m)
> matching Java PCollectionTuple translation naming convention in expansion
> service with index only
> -
>
> Key: BEAM-8905
> URL: https://issues.apache.org/jira/browse/BEAM-8905
> Project: Beam
> Issue Type: Improvement
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Python PCollectionTuple is translated to an index-keyed map e.g.
> \{0->pcollection1, 1->pcollection2}, however Java PCollectionTuple is
> translated to slightly different formats such as \{output_0->pcollection1,
> output_1->pcollection2}. We need to match these naming conventions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8905) matching Java PCollectionTuple translation naming convention in expansion service with index only
[
https://issues.apache.org/jira/browse/BEAM-8905?focusedWorklogId=356668&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356668
]
ASF GitHub Bot logged work on BEAM-8905:
Author: ASF GitHub Bot
Created on: 09/Dec/19 23:36
Start Date: 09/Dec/19 23:36
Worklog Time Spent: 10m
Work Description: ihji commented on issue #10310: [BEAM-8905] matching
Java PCollectionTuple translation naming convention in expansion service
URL: https://github.com/apache/beam/pull/10310#issuecomment-563489886
Run Java_Examples_Dataflow PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356668)
Time Spent: 1h 10m (was: 1h)
> matching Java PCollectionTuple translation naming convention in expansion
> service with index only
> -
>
> Key: BEAM-8905
> URL: https://issues.apache.org/jira/browse/BEAM-8905
> Project: Beam
> Issue Type: Improvement
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Python PCollectionTuple is translated to an index-keyed map e.g.
> \{0->pcollection1, 1->pcollection2}, however Java PCollectionTuple is
> translated to slightly different formats such as \{output_0->pcollection1,
> output_1->pcollection2}. We need to match these naming conventions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8904) properly update output pcollections from expanded transforms
[ https://issues.apache.org/jira/browse/BEAM-8904?focusedWorklogId=356667&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356667 ] ASF GitHub Bot logged work on BEAM-8904: Author: ASF GitHub Bot Created on: 09/Dec/19 23:35 Start Date: 09/Dec/19 23:35 Worklog Time Spent: 10m Work Description: ihji commented on issue #10309: [BEAM-8904] properly update output pcollections from expanded transforms URL: https://github.com/apache/beam/pull/10309#issuecomment-563489686 The result of `extractOutputs` is ignored if we don't update the output map. I added the test (the single output should have a key `output` from `extractOutputs` but test would fail without this PR). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356667) Time Spent: 50m (was: 40m) > properly update output pcollections from expanded transforms > > > Key: BEAM-8904 > URL: https://issues.apache.org/jira/browse/BEAM-8904 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > currently output pcollections from expanded transforms are ignored. we need > to properly update output pcollections when it's returned to the caller of > expansion service. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8858) Missing dependencies in SQL module pom
[ https://issues.apache.org/jira/browse/BEAM-8858?focusedWorklogId=35&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-35 ] ASF GitHub Bot logged work on BEAM-8858: Author: ASF GitHub Bot Created on: 09/Dec/19 23:33 Start Date: 09/Dec/19 23:33 Worklog Time Spent: 10m Work Description: vectorijk commented on issue #10332: [BEAM-8858] sdks/java/extensions/sql to declare used-but-undeclared dependencies URL: https://github.com/apache/beam/pull/10332#issuecomment-563489123 lgtm This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 35) Time Spent: 20m (was: 10m) > Missing dependencies in SQL module pom > -- > > Key: BEAM-8858 > URL: https://issues.apache.org/jira/browse/BEAM-8858 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Affects Versions: 2.18.0 >Reporter: Ismaël Mejía >Assignee: Tomo Suzuki >Priority: Blocker > Fix For: 2.18.0 > > Time Spent: 20m > Remaining Estimate: 0h > > I moved an user project that depends on the SQL module to the version > 2.18.0-SNAPSHOT and I discovered that it was not working anymore because some > of the transitive dependencies were not correctly exposed anymore by the SQL > module e.g. commons-codec and Nullable annotations dependency. > There are some missing dependencies in the generated pom file when compared > with previous versions, probably cases of `usedUndeclaredArtifacts` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8933) BigQuery IO should support read/write in Arrow format
[
https://issues.apache.org/jira/browse/BEAM-8933?focusedWorklogId=356663&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356663
]
ASF GitHub Bot logged work on BEAM-8933:
Author: ASF GitHub Bot
Created on: 09/Dec/19 23:17
Start Date: 09/Dec/19 23:17
Worklog Time Spent: 10m
Work Description: 11moon11 commented on pull request #10334: [BEAM-8933]
Update BigQuery proto dependency
URL: https://github.com/apache/beam/pull/10334#discussion_r355738177
##
File path:
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageReadTest.java
##
@@ -1493,6 +1399,17 @@ public void testReadFromBigQueryIO() throws Exception {
StreamPosition.newBuilder().setStream(Stream.newBuilder().setName("streamName")))
.build();
+/*
Review comment:
Was using it as a reference when debugging. Should be deleted now.
Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356663)
Time Spent: 0.5h (was: 20m)
> BigQuery IO should support read/write in Arrow format
> -
>
> Key: BEAM-8933
> URL: https://issues.apache.org/jira/browse/BEAM-8933
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
>Reporter: Kirill Kozlov
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> As of right now BigQuery uses Avro format for reading and writing.
> We should add a config to BigQueryIO to specify which format to use (with
> Avro as default).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8933) BigQuery IO should support read/write in Arrow format
[
https://issues.apache.org/jira/browse/BEAM-8933?focusedWorklogId=356662&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356662
]
ASF GitHub Bot logged work on BEAM-8933:
Author: ASF GitHub Bot
Created on: 09/Dec/19 23:15
Start Date: 09/Dec/19 23:15
Worklog Time Spent: 10m
Work Description: TheNeuralBit commented on pull request #10334:
[BEAM-8933] Update BigQuery proto dependency
URL: https://github.com/apache/beam/pull/10334#discussion_r355736979
##
File path:
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageReadTest.java
##
@@ -1493,6 +1399,17 @@ public void testReadFromBigQueryIO() throws Exception {
StreamPosition.newBuilder().setStream(Stream.newBuilder().setName("streamName")))
.build();
+/*
Review comment:
Should this be here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356662)
Time Spent: 20m (was: 10m)
> BigQuery IO should support read/write in Arrow format
> -
>
> Key: BEAM-8933
> URL: https://issues.apache.org/jira/browse/BEAM-8933
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
>Reporter: Kirill Kozlov
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> As of right now BigQuery uses Avro format for reading and writing.
> We should add a config to BigQueryIO to specify which format to use (with
> Avro as default).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8886) Add a python mongodbio integration test that triggers load split
[ https://issues.apache.org/jira/browse/BEAM-8886?focusedWorklogId=356661&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356661 ] ASF GitHub Bot logged work on BEAM-8886: Author: ASF GitHub Bot Created on: 09/Dec/19 23:10 Start Date: 09/Dec/19 23:10 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10289: [BEAM-8886] Add a python mongodb io dataflow integration test URL: https://github.com/apache/beam/pull/10289#issuecomment-563482584 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356661) Time Spent: 7h 10m (was: 7h) > Add a python mongodbio integration test that triggers load split > > > Key: BEAM-8886 > URL: https://issues.apache.org/jira/browse/BEAM-8886 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Minor > Time Spent: 7h 10m > Remaining Estimate: 0h > > Current integration test doesn't seem to trigger liquid sharding at all, we > should change integration test that has more load and potentially use the > mongodb k8s cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8933) BigQuery IO should support read/write in Arrow format
[ https://issues.apache.org/jira/browse/BEAM-8933?focusedWorklogId=356656&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356656 ] ASF GitHub Bot logged work on BEAM-8933: Author: ASF GitHub Bot Created on: 09/Dec/19 23:04 Start Date: 09/Dec/19 23:04 Worklog Time Spent: 10m Work Description: 11moon11 commented on pull request #10334: [BEAM-8933] Update BigQuery proto dependency URL: https://github.com/apache/beam/pull/10334 Before adding a capability to a BigQuery IO connector to read and write in Arrow format we need to update proto dependency. 1. Updated `proto-bigquerystorage-v1beta1` to `0.83.0`. 2. Update tests to use proper proto fields. R: @TheNeuralBit CC: @apilloud CC: @amaliujia CC: @aryann Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
[jira] [Work logged] (BEAM-8837) PCollectionVisualizationTest: possible bug
[
https://issues.apache.org/jira/browse/BEAM-8837?focusedWorklogId=356655&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356655
]
ASF GitHub Bot logged work on BEAM-8837:
Author: ASF GitHub Bot
Created on: 09/Dec/19 23:04
Start Date: 09/Dec/19 23:04
Worklog Time Spent: 10m
Work Description: KevinGG commented on issue #10321: [BEAM-8837] Fix
pcoll_visualization tests
URL: https://github.com/apache/beam/pull/10321#issuecomment-563480861
Run Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356655)
Time Spent: 50m (was: 40m)
> PCollectionVisualizationTest: possible bug
> --
>
> Key: BEAM-8837
> URL: https://issues.apache.org/jira/browse/BEAM-8837
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Ning Kang
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> This seems like a bug, even though the test passes:
> {code}
> test_display_plain_text_when_kernel_has_no_frontend
> (apache_beam.runners.interactive.display.pcoll_visualization_test.PCollectionVisualizationTest)
> ... Exception in thread Thread-4405:
> Traceback (most recent call last):
> File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
> self.run()
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/.eggs/timeloop-1.0.2-py3.7.egg/timeloop/job.py",
> line 19, in run
> self.execute(*self.args, **self.kwargs)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py",
> line 132, in continuous_update_display
> updated_pv.display_facets(updating_pv=pv)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py",
> line 209, in display_facets
> data = self._to_dataframe()
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py",
> line 278, in _to_dataframe
> for el in self._to_element_list():
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py",
> line 266, in _to_element_list
> if ie.current_env().cache_manager().exists('full', self._cache_key):
> AttributeError: 'NoneType' object has no attribute 'exists'
> ok
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-8933) BigQuery IO should support read/write in Arrow format
Kirill Kozlov created BEAM-8933: --- Summary: BigQuery IO should support read/write in Arrow format Key: BEAM-8933 URL: https://issues.apache.org/jira/browse/BEAM-8933 Project: Beam Issue Type: Improvement Components: io-java-gcp Reporter: Kirill Kozlov As of right now BigQuery uses Avro format for reading and writing. We should add a config to BigQueryIO to specify which format to use (with Avro as default). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests
[ https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=356634&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356634 ] ASF GitHub Bot logged work on BEAM-8575: Author: ASF GitHub Bot Created on: 09/Dec/19 22:45 Start Date: 09/Dec/19 22:45 Worklog Time Spent: 10m Work Description: HuangLED commented on issue #10143: [BEAM-8575] To test state backed iterable coder in py sdk. URL: https://github.com/apache/beam/pull/10143#issuecomment-563474930 All suggestions are addressed. Gentle ping for merging PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356634) Time Spent: 30h (was: 29h 50m) > Add more Python validates runner tests > -- > > Key: BEAM-8575 > URL: https://issues.apache.org/jira/browse/BEAM-8575 > Project: Beam > Issue Type: Test > Components: sdk-py-core, testing >Reporter: wendy liu >Assignee: wendy liu >Priority: Major > Time Spent: 30h > Remaining Estimate: 0h > > This is the umbrella issue to track the work of adding more Python tests to > improve test coverage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8693) Beam Dependency Update Request: com.google.cloud.datastore:datastore-v1-proto-client
[ https://issues.apache.org/jira/browse/BEAM-8693?focusedWorklogId=356630&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356630 ] ASF GitHub Bot logged work on BEAM-8693: Author: ASF GitHub Bot Created on: 09/Dec/19 22:35 Start Date: 09/Dec/19 22:35 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10327: [BEAM-8693] com.google.cloud.datastore:datastore-v1-proto-client 1.6.3 URL: https://github.com/apache/beam/pull/10327#issuecomment-563471780 Run Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356630) Time Spent: 2h 20m (was: 2h 10m) > Beam Dependency Update Request: > com.google.cloud.datastore:datastore-v1-proto-client > > > Key: BEAM-8693 > URL: https://issues.apache.org/jira/browse/BEAM-8693 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Assignee: Tomo Suzuki >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > - 2019-11-15 19:39:56.526732 > - > Please consider upgrading the dependency > com.google.cloud.datastore:datastore-v1-proto-client. > The current version is 1.6.0. The latest version is 1.6.3 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:05:51.468284 > - > Please consider upgrading the dependency > com.google.cloud.datastore:datastore-v1-proto-client. > The current version is 1.6.0. The latest version is 1.6.3 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:11:37.877225 > - > Please consider upgrading the dependency > com.google.cloud.datastore:datastore-v1-proto-client. > The current version is 1.6.0. The latest version is 1.6.3 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:10:45.889899 > - > Please consider upgrading the dependency > com.google.cloud.datastore:datastore-v1-proto-client. > The current version is 1.6.0. The latest version is 1.6.3 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=356625&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356625 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Dec/19 22:33 Start Date: 09/Dec/19 22:33 Worklog Time Spent: 10m Work Description: chadrik commented on issue #9915: [BEAM-7746] Add python type hints (part 1) URL: https://github.com/apache/beam/pull/9915#issuecomment-563471088 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356625) Time Spent: 34h (was: 33h 50m) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 34h > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=356626&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356626 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Dec/19 22:33 Start Date: 09/Dec/19 22:33 Worklog Time Spent: 10m Work Description: chadrik commented on pull request #9915: [BEAM-7746] Add python type hints (part 1) URL: https://github.com/apache/beam/pull/9915 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356626) Time Spent: 34h 10m (was: 34h) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 34h 10m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=356627&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356627 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Dec/19 22:33 Start Date: 09/Dec/19 22:33 Worklog Time Spent: 10m Work Description: chadrik commented on pull request #9915: [BEAM-7746] Add python type hints (part 1) URL: https://github.com/apache/beam/pull/9915 This is part 1 of https://github.com/apache/beam/pull/9056 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_P
[jira] [Work logged] (BEAM-8536) Migrate usage of DelayedBundleApplication.requested_execution_time to time duration
[ https://issues.apache.org/jira/browse/BEAM-8536?focusedWorklogId=356619&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356619 ] ASF GitHub Bot logged work on BEAM-8536: Author: ASF GitHub Bot Created on: 09/Dec/19 22:25 Start Date: 09/Dec/19 22:25 Worklog Time Spent: 10m Work Description: boyuanzz commented on pull request #10333: [BEAM-8536] Migrate request_execution_time to request_delay_time in Java URL: https://github.com/apache/beam/pull/10333 +R: @robertwb cc: @lukecwik This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356619) Remaining Estimate: 0h Time Spent: 10m > Migrate usage of DelayedBundleApplication.requested_execution_time to time > duration > > > Key: BEAM-8536 > URL: https://issues.apache.org/jira/browse/BEAM-8536 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow, sdk-java-harness >Reporter: Boyuan Zhang >Assignee: Boyuan Zhang >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > In DelayedBundleApplication, we used to use an absolute time to represent > rescheduling time. We want to switch to use a relative time duration, which > requires a migration in Java SDK and dataflow java runner harness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8905) matching Java PCollectionTuple translation naming convention in expansion service with index only
[
https://issues.apache.org/jira/browse/BEAM-8905?focusedWorklogId=356615&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356615
]
ASF GitHub Bot logged work on BEAM-8905:
Author: ASF GitHub Bot
Created on: 09/Dec/19 22:19
Start Date: 09/Dec/19 22:19
Worklog Time Spent: 10m
Work Description: ihji commented on issue #10310: [BEAM-8905] matching
Java PCollectionTuple translation naming convention in expansion service
URL: https://github.com/apache/beam/pull/10310#issuecomment-563466519
Test added.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356615)
Time Spent: 1h (was: 50m)
> matching Java PCollectionTuple translation naming convention in expansion
> service with index only
> -
>
> Key: BEAM-8905
> URL: https://issues.apache.org/jira/browse/BEAM-8905
> Project: Beam
> Issue Type: Improvement
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Python PCollectionTuple is translated to an index-keyed map e.g.
> \{0->pcollection1, 1->pcollection2}, however Java PCollectionTuple is
> translated to slightly different formats such as \{output_0->pcollection1,
> output_1->pcollection2}. We need to match these naming conventions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=356612&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356612 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 09/Dec/19 22:15 Start Date: 09/Dec/19 22:15 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-563464985 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356612) Time Spent: 5h 10m (was: 5h) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 5h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8824) Add support for allowed lateness in python sdk
[ https://issues.apache.org/jira/browse/BEAM-8824?focusedWorklogId=356613&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356613 ] ASF GitHub Bot logged work on BEAM-8824: Author: ASF GitHub Bot Created on: 09/Dec/19 22:15 Start Date: 09/Dec/19 22:15 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10216: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk URL: https://github.com/apache/beam/pull/10216#issuecomment-562765769 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356613) Time Spent: 5h 20m (was: 5h 10m) > Add support for allowed lateness in python sdk > -- > > Key: BEAM-8824 > URL: https://issues.apache.org/jira/browse/BEAM-8824 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 5h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8362) Don't use ZetaSQL's unimplemented functions
[ https://issues.apache.org/jira/browse/BEAM-8362?focusedWorklogId=356609&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356609 ] ASF GitHub Bot logged work on BEAM-8362: Author: ASF GitHub Bot Created on: 09/Dec/19 22:08 Start Date: 09/Dec/19 22:08 Worklog Time Spent: 10m Work Description: apilloud commented on issue #9739: [BEAM-8362] Don't use toString() for accessing Enum Types URL: https://github.com/apache/beam/pull/9739#issuecomment-563462708 @amaliujia Removed the error messages from this fix. Please take another look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356609) Time Spent: 1.5h (was: 1h 20m) > Don't use ZetaSQL's unimplemented functions > --- > > Key: BEAM-8362 > URL: https://issues.apache.org/jira/browse/BEAM-8362 > Project: Beam > Issue Type: New Feature > Components: dsl-sql-zetasql >Affects Versions: 2.15.0 >Reporter: Andrew Pilloud >Assignee: Andrew Pilloud >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > Unfortunately a bunch of debug functionality is still unimplemented in > ZetaSQL. We should avoid calling those functions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8362) Don't use ZetaSQL's unimplemented functions
[ https://issues.apache.org/jira/browse/BEAM-8362?focusedWorklogId=356607&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356607 ] ASF GitHub Bot logged work on BEAM-8362: Author: ASF GitHub Bot Created on: 09/Dec/19 22:06 Start Date: 09/Dec/19 22:06 Worklog Time Spent: 10m Work Description: stale[bot] commented on issue #9739: [BEAM-8362] Don't use ZetaSQL's unimplemented functions URL: https://github.com/apache/beam/pull/9739#issuecomment-563462205 This pull request is no longer marked as stale. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356607) Time Spent: 1h 20m (was: 1h 10m) > Don't use ZetaSQL's unimplemented functions > --- > > Key: BEAM-8362 > URL: https://issues.apache.org/jira/browse/BEAM-8362 > Project: Beam > Issue Type: New Feature > Components: dsl-sql-zetasql >Affects Versions: 2.15.0 >Reporter: Andrew Pilloud >Assignee: Andrew Pilloud >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > Unfortunately a bunch of debug functionality is still unimplemented in > ZetaSQL. We should avoid calling those functions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8858) Missing dependencies in SQL module pom
[ https://issues.apache.org/jira/browse/BEAM-8858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991974#comment-16991974 ] Tomo Suzuki commented on BEAM-8858: --- Created a PR https://github.com/apache/beam/pull/10332 to solve this problem. > Missing dependencies in SQL module pom > -- > > Key: BEAM-8858 > URL: https://issues.apache.org/jira/browse/BEAM-8858 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Affects Versions: 2.18.0 >Reporter: Ismaël Mejía >Assignee: Tomo Suzuki >Priority: Blocker > Fix For: 2.18.0 > > Time Spent: 10m > Remaining Estimate: 0h > > I moved an user project that depends on the SQL module to the version > 2.18.0-SNAPSHOT and I discovered that it was not working anymore because some > of the transitive dependencies were not correctly exposed anymore by the SQL > module e.g. commons-codec and Nullable annotations dependency. > There are some missing dependencies in the generated pom file when compared > with previous versions, probably cases of `usedUndeclaredArtifacts` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8858) Missing dependencies in SQL module pom
[ https://issues.apache.org/jira/browse/BEAM-8858?focusedWorklogId=356605&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356605 ] ASF GitHub Bot logged work on BEAM-8858: Author: ASF GitHub Bot Created on: 09/Dec/19 22:02 Start Date: 09/Dec/19 22:02 Worklog Time Spent: 10m Work Description: suztomo commented on pull request #10332: [BEAM-8858] sdks/java/extensions/sql to declare used-but-undeclared dependencies URL: https://github.com/apache/beam/pull/10332 https://issues.apache.org/jira/browse/BEAM-8858 As per [Gradle Lint Plugin output ](https://gist.github.com/suztomo/e3fa9e5ce9eda5265082c1db8401d6db), restoring these dependencies that have been removed by [PR #9189](https://github.com/apache/beam/pull/9189/files#r355589635). CC: @iemejia @kennknowles @vectorijk Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_Validat
[jira] [Assigned] (BEAM-8858) Missing dependencies in SQL module pom
[ https://issues.apache.org/jira/browse/BEAM-8858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomo Suzuki reassigned BEAM-8858: - Assignee: Tomo Suzuki (was: Kenneth Knowles) > Missing dependencies in SQL module pom > -- > > Key: BEAM-8858 > URL: https://issues.apache.org/jira/browse/BEAM-8858 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Affects Versions: 2.18.0 >Reporter: Ismaël Mejía >Assignee: Tomo Suzuki >Priority: Blocker > Fix For: 2.18.0 > > > I moved an user project that depends on the SQL module to the version > 2.18.0-SNAPSHOT and I discovered that it was not working anymore because some > of the transitive dependencies were not correctly exposed anymore by the SQL > module e.g. commons-codec and Nullable annotations dependency. > There are some missing dependencies in the generated pom file when compared > with previous versions, probably cases of `usedUndeclaredArtifacts` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7516) Add a watermark manager for the fn_api_runner
[ https://issues.apache.org/jira/browse/BEAM-7516?focusedWorklogId=356601&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356601 ] ASF GitHub Bot logged work on BEAM-7516: Author: ASF GitHub Bot Created on: 09/Dec/19 21:58 Start Date: 09/Dec/19 21:58 Worklog Time Spent: 10m Work Description: pabloem commented on issue #10291: [BEAM-7516][BEAM-8823] FnApiRunner works with work queues, and a primitive watermark manager URL: https://github.com/apache/beam/pull/10291#issuecomment-563459134 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356601) Time Spent: 2h 10m (was: 2h) > Add a watermark manager for the fn_api_runner > - > > Key: BEAM-7516 > URL: https://issues.apache.org/jira/browse/BEAM-7516 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > To track watermarks for each stage -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8932) Expose complete Cloud Pub/Sub messages through PubsubIO API
[ https://issues.apache.org/jira/browse/BEAM-8932?focusedWorklogId=356598&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356598 ] ASF GitHub Bot logged work on BEAM-8932: Author: ASF GitHub Bot Created on: 09/Dec/19 21:55 Start Date: 09/Dec/19 21:55 Worklog Time Spent: 10m Work Description: boyuanzz commented on issue #10331: [BEAM-8932] Modify PubsubClient to use the proto message throughout. URL: https://github.com/apache/beam/pull/10331#issuecomment-563457965 +R: @boyuanzz This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356598) Time Spent: 20m (was: 10m) > Expose complete Cloud Pub/Sub messages through PubsubIO API > --- > > Key: BEAM-8932 > URL: https://issues.apache.org/jira/browse/BEAM-8932 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Daniel Collins >Assignee: Daniel Collins >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > The PubsubIO API only exposes a subset of the fields in the underlying > PubsubMessage protocol buffer. To accomodate future feature changes as well > as for greater compatability with code using the Cloud Pub/Sub apis, a method > to read and write these protocol messages should be exposed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8335) Add streaming support to Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-8335?focusedWorklogId=356597&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356597 ] ASF GitHub Bot logged work on BEAM-8335: Author: ASF GitHub Bot Created on: 09/Dec/19 21:53 Start Date: 09/Dec/19 21:53 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #9953: [BEAM-8335] Adds support for multi-output TestStream URL: https://github.com/apache/beam/pull/9953 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356597) Time Spent: 43h 50m (was: 43h 40m) > Add streaming support to Interactive Beam > - > > Key: BEAM-8335 > URL: https://issues.apache.org/jira/browse/BEAM-8335 > Project: Beam > Issue Type: Improvement > Components: runner-py-interactive >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Major > Time Spent: 43h 50m > Remaining Estimate: 0h > > This issue tracks the work items to introduce streaming support to the > Interactive Beam experience. This will allow users to: > * Write and run a streaming job in IPython > * Automatically cache records from unbounded sources > * Add a replay experience that replays all cached records to simulate the > original pipeline execution > * Add controls to play/pause/stop/step individual elements from the cached > records > * Add ability to inspect/visualize unbounded PCollections -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-8932) Expose complete Cloud Pub/Sub messages through PubsubIO API
[ https://issues.apache.org/jira/browse/BEAM-8932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pablo Estrada reassigned BEAM-8932: --- Assignee: Daniel Collins > Expose complete Cloud Pub/Sub messages through PubsubIO API > --- > > Key: BEAM-8932 > URL: https://issues.apache.org/jira/browse/BEAM-8932 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Daniel Collins >Assignee: Daniel Collins >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > The PubsubIO API only exposes a subset of the fields in the underlying > PubsubMessage protocol buffer. To accomodate future feature changes as well > as for greater compatability with code using the Cloud Pub/Sub apis, a method > to read and write these protocol messages should be exposed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8932) Expose complete Cloud Pub/Sub messages through PubsubIO API
[ https://issues.apache.org/jira/browse/BEAM-8932?focusedWorklogId=356595&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356595 ] ASF GitHub Bot logged work on BEAM-8932: Author: ASF GitHub Bot Created on: 09/Dec/19 21:50 Start Date: 09/Dec/19 21:50 Worklog Time Spent: 10m Work Description: dpcollins-google commented on pull request #10331: [BEAM-8932] Modify PubsubClient to use the proto message throughout. URL: https://github.com/apache/beam/pull/10331 **Please** add a meaningful description for your change here This change will enable creating a PubsubIO transform that reads and writes the full proto, useful if there are proto changes in the future. It also accomodates the orderingKey field in the JSON client library. R: boyuanz - [ x ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ x ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ x ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://
[jira] [Work logged] (BEAM-8886) Add a python mongodbio integration test that triggers load split
[ https://issues.apache.org/jira/browse/BEAM-8886?focusedWorklogId=356594&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356594 ] ASF GitHub Bot logged work on BEAM-8886: Author: ASF GitHub Bot Created on: 09/Dec/19 21:47 Start Date: 09/Dec/19 21:47 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10289: [BEAM-8886] Add a python mongodb io dataflow integration test URL: https://github.com/apache/beam/pull/10289#issuecomment-563454932 Run Python MongoDBIO Load Test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356594) Time Spent: 7h (was: 6h 50m) > Add a python mongodbio integration test that triggers load split > > > Key: BEAM-8886 > URL: https://issues.apache.org/jira/browse/BEAM-8886 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Minor > Time Spent: 7h > Remaining Estimate: 0h > > Current integration test doesn't seem to trigger liquid sharding at all, we > should change integration test that has more load and potentially use the > mongodb k8s cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests
[
https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=356593&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356593
]
ASF GitHub Bot logged work on BEAM-8575:
Author: ASF GitHub Bot
Created on: 09/Dec/19 21:46
Start Date: 09/Dec/19 21:46
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10190: [BEAM-8575]
Added two unit tests to CombineTest class to test that Co…
URL: https://github.com/apache/beam/pull/10190#discussion_r355700749
##
File path: sdks/python/apache_beam/transforms/combiners_test.py
##
@@ -399,6 +432,108 @@ def test_global_fanout(self):
| beam.CombineGlobally(combine.MeanCombineFn()).with_fanout(11))
assert_that(result, equal_to([49.5]))
+ # Test that three different kinds of metrics work with a customized
+ # CounterIncrememtingCombineFn.
+ def test_simple_combine(self):
+p = TestPipeline()
+input = (p
+ | beam.Create([('c', 'b'),
+('c', 'be'),
+('c', 'bea'),
+('d', 'beam'),
+('d', 'apache')]))
+
+# The result of concatenating all values regardless of key.
+global_concat = (input
+ | beam.Values()
+ | beam.CombineGlobally(CounterIncrememtingCombineFn()))
+
+# The (key, concatenated_string) pairs for all keys.
+concat_per_key = (input | beam.CombinePerKey(
+CounterIncrememtingCombineFn()))
+
+result = p.run()
+result.wait_until_finish()
+
+# Verify the concatenated strings are correct.
+expected_concat_per_key = [('c', 'bbebea'), ('d', 'beamapache')]
+assert_that(global_concat, equal_to(['bbebeabeamapache']),
+label='global concat')
+assert_that(concat_per_key, equal_to(expected_concat_per_key),
+label='concat per key')
+
+# Verify the values of metrics are correct.
+word_counter_filter = MetricsFilter().with_name('word_counter')
+query_result = result.metrics().query(word_counter_filter)
+if query_result['counters']:
+ word_counter = query_result['counters'][0]
+ self.assertEqual(word_counter.result, 5)
+
+word_lengths_filter = MetricsFilter().with_name('word_lengths')
+query_result = result.metrics().query(word_lengths_filter)
+if query_result['counters']:
+ word_lengths = query_result['counters'][0]
+ self.assertEqual(word_lengths.result, 16)
+
+word_len_dist_filter = MetricsFilter().with_name('word_len_dist')
+query_result = result.metrics().query(word_len_dist_filter)
+if query_result['distributions']:
+ word_len_dist = query_result['distributions'][0]
+ self.assertEqual(word_len_dist.result.mean, 3.2)
+
+last_word_len_filter = MetricsFilter().with_name('last_word_len')
+query_result = result.metrics().query(last_word_len_filter)
+if query_result['gauges']:
+ last_word_len = query_result['gauges'][0]
+ self.assertEqual(last_word_len.result.value, 6)
+
+ # Test that three different kinds of metrics work with the customized
+ # CounterIncrememtingCombineFn when the PCollection is empty.
+ def test_simple_combine_empty(self):
+p = TestPipeline()
+input = p | beam.Create([])
+
+# The result of concatenating all values regardless of key.
+global_concat = (input
+ | beam.Values()
+ | beam.CombineGlobally(CounterIncrememtingCombineFn()))
+
+# The (key, concatenated_string) pairs for all keys.
+concat_per_key = (input | beam.CombinePerKey(
+CounterIncrememtingCombineFn()))
+
+result = p.run()
+result.wait_until_finish()
+
+# Verify the concatenated strings are correct.
+assert_that(global_concat, equal_to(['']), label='global concat')
+assert_that(concat_per_key, equal_to([]), label='concat per key')
+
+# Verify the values of metrics are correct.
+word_counter_filter = MetricsFilter().with_name('word_counter')
+query_result = result.metrics().query(word_counter_filter)
+if query_result['counters']:
+ word_counter = query_result['counters'][0]
+ self.assertEqual(word_counter.result, 0)
+
+word_lengths_filter = MetricsFilter().with_name('word_lengths')
+query_result = result.metrics().query(word_lengths_filter)
+if query_result['counters']:
+ word_lengths = query_result['counters'][0]
+ self.assertEqual(word_lengths.result, 0)
+
+word_len_dist_filter = MetricsFilter().with_name('word_len_dist')
+query_result = result.metrics().query(word_len_dist_filter)
+if query_result['distributions']:
+ word_len_dist = query_result['distributions'][0]
+ self.assertEqual(word_len_dist.result.mean, 0)
Review comment:
The mean should not be 0, it should be undefined. You could test the cou
[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests
[ https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=356591&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356591 ] ASF GitHub Bot logged work on BEAM-8575: Author: ASF GitHub Bot Created on: 09/Dec/19 21:46 Start Date: 09/Dec/19 21:46 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10190: [BEAM-8575] Added two unit tests to CombineTest class to test that Co… URL: https://github.com/apache/beam/pull/10190#discussion_r355698173 ## File path: sdks/python/apache_beam/transforms/combiners_test.py ## @@ -48,6 +50,37 @@ from apache_beam.utils.timestamp import Timestamp +class CounterIncrememtingCombineFn(beam.CombineFn): + """CombineFn for incrementing three different counters: + counter, distribution, gauge, + at the same time concatenating words.""" + + def __init__(self): +beam.CombineFn.__init__(self) +self.word_counter = Metrics.counter(self.__class__, 'word_counter') +self.word_lengths_counter = Metrics.counter( +self.__class__, 'word_lengths') +self.word_lengths_dist = Metrics.distribution( +self.__class__, 'word_len_dist') +self.last_word_len = Metrics.gauge(self.__class__, 'last_word_len') + + def create_accumulator(self): +return '' + + def add_input(self, acc, element): +self.word_counter.inc(1) +self.word_lengths_counter.inc(len(element)) +self.word_lengths_dist.update(len(element)) +self.last_word_len.set(len(element)) +return acc + element Review comment: This is non-deterministic. Perhaps `sorted(acc + element)`? (Similarly below.) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356591) Time Spent: 29h 40m (was: 29.5h) > Add more Python validates runner tests > -- > > Key: BEAM-8575 > URL: https://issues.apache.org/jira/browse/BEAM-8575 > Project: Beam > Issue Type: Test > Components: sdk-py-core, testing >Reporter: wendy liu >Assignee: wendy liu >Priority: Major > Time Spent: 29h 40m > Remaining Estimate: 0h > > This is the umbrella issue to track the work of adding more Python tests to > improve test coverage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests
[
https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=356592&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356592
]
ASF GitHub Bot logged work on BEAM-8575:
Author: ASF GitHub Bot
Created on: 09/Dec/19 21:46
Start Date: 09/Dec/19 21:46
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10190: [BEAM-8575]
Added two unit tests to CombineTest class to test that Co…
URL: https://github.com/apache/beam/pull/10190#discussion_r355700501
##
File path: sdks/python/apache_beam/transforms/combiners_test.py
##
@@ -399,6 +432,108 @@ def test_global_fanout(self):
| beam.CombineGlobally(combine.MeanCombineFn()).with_fanout(11))
assert_that(result, equal_to([49.5]))
+ # Test that three different kinds of metrics work with a customized
+ # CounterIncrememtingCombineFn.
+ def test_simple_combine(self):
+p = TestPipeline()
+input = (p
+ | beam.Create([('c', 'b'),
+('c', 'be'),
+('c', 'bea'),
+('d', 'beam'),
+('d', 'apache')]))
+
+# The result of concatenating all values regardless of key.
+global_concat = (input
+ | beam.Values()
+ | beam.CombineGlobally(CounterIncrememtingCombineFn()))
+
+# The (key, concatenated_string) pairs for all keys.
+concat_per_key = (input | beam.CombinePerKey(
+CounterIncrememtingCombineFn()))
+
+result = p.run()
+result.wait_until_finish()
+
+# Verify the concatenated strings are correct.
+expected_concat_per_key = [('c', 'bbebea'), ('d', 'beamapache')]
+assert_that(global_concat, equal_to(['bbebeabeamapache']),
+label='global concat')
+assert_that(concat_per_key, equal_to(expected_concat_per_key),
+label='concat per key')
+
+# Verify the values of metrics are correct.
+word_counter_filter = MetricsFilter().with_name('word_counter')
+query_result = result.metrics().query(word_counter_filter)
+if query_result['counters']:
+ word_counter = query_result['counters'][0]
+ self.assertEqual(word_counter.result, 5)
+
+word_lengths_filter = MetricsFilter().with_name('word_lengths')
+query_result = result.metrics().query(word_lengths_filter)
+if query_result['counters']:
+ word_lengths = query_result['counters'][0]
+ self.assertEqual(word_lengths.result, 16)
+
+word_len_dist_filter = MetricsFilter().with_name('word_len_dist')
+query_result = result.metrics().query(word_len_dist_filter)
+if query_result['distributions']:
+ word_len_dist = query_result['distributions'][0]
+ self.assertEqual(word_len_dist.result.mean, 3.2)
+
+last_word_len_filter = MetricsFilter().with_name('last_word_len')
+query_result = result.metrics().query(last_word_len_filter)
+if query_result['gauges']:
+ last_word_len = query_result['gauges'][0]
+ self.assertEqual(last_word_len.result.value, 6)
Review comment:
There is no promise about the order in which words are processed, you can
only assert that the gauge is in the set of possible values.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356592)
Time Spent: 29h 40m (was: 29.5h)
> Add more Python validates runner tests
> --
>
> Key: BEAM-8575
> URL: https://issues.apache.org/jira/browse/BEAM-8575
> Project: Beam
> Issue Type: Test
> Components: sdk-py-core, testing
>Reporter: wendy liu
>Assignee: wendy liu
>Priority: Major
> Time Spent: 29h 40m
> Remaining Estimate: 0h
>
> This is the umbrella issue to track the work of adding more Python tests to
> improve test coverage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests
[
https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=356590&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356590
]
ASF GitHub Bot logged work on BEAM-8575:
Author: ASF GitHub Bot
Created on: 09/Dec/19 21:46
Start Date: 09/Dec/19 21:46
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10190: [BEAM-8575]
Added two unit tests to CombineTest class to test that Co…
URL: https://github.com/apache/beam/pull/10190#discussion_r355699412
##
File path: sdks/python/apache_beam/transforms/combiners_test.py
##
@@ -399,6 +432,108 @@ def test_global_fanout(self):
| beam.CombineGlobally(combine.MeanCombineFn()).with_fanout(11))
assert_that(result, equal_to([49.5]))
+ # Test that three different kinds of metrics work with a customized
+ # CounterIncrememtingCombineFn.
+ def test_simple_combine(self):
+p = TestPipeline()
+input = (p
+ | beam.Create([('c', 'b'),
+('c', 'be'),
+('c', 'bea'),
+('d', 'beam'),
+('d', 'apache')]))
+
+# The result of concatenating all values regardless of key.
+global_concat = (input
+ | beam.Values()
+ | beam.CombineGlobally(CounterIncrememtingCombineFn()))
+
+# The (key, concatenated_string) pairs for all keys.
+concat_per_key = (input | beam.CombinePerKey(
+CounterIncrememtingCombineFn()))
+
+result = p.run()
+result.wait_until_finish()
+
+# Verify the concatenated strings are correct.
+expected_concat_per_key = [('c', 'bbebea'), ('d', 'beamapache')]
+assert_that(global_concat, equal_to(['bbebeabeamapache']),
+label='global concat')
+assert_that(concat_per_key, equal_to(expected_concat_per_key),
+label='concat per key')
+
+# Verify the values of metrics are correct.
+word_counter_filter = MetricsFilter().with_name('word_counter')
+query_result = result.metrics().query(word_counter_filter)
+if query_result['counters']:
+ word_counter = query_result['counters'][0]
+ self.assertEqual(word_counter.result, 5)
+
+word_lengths_filter = MetricsFilter().with_name('word_lengths')
+query_result = result.metrics().query(word_lengths_filter)
+if query_result['counters']:
+ word_lengths = query_result['counters'][0]
+ self.assertEqual(word_lengths.result, 16)
+
+word_len_dist_filter = MetricsFilter().with_name('word_len_dist')
+query_result = result.metrics().query(word_len_dist_filter)
+if query_result['distributions']:
+ word_len_dist = query_result['distributions'][0]
+ self.assertEqual(word_len_dist.result.mean, 3.2)
Review comment:
To avoid spurious failures due to floating point issues, choose data such
that the results can be represented exactly (e.g. as an integer or binary
fraction like 3.5 or 3.25).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356590)
Time Spent: 29.5h (was: 29h 20m)
> Add more Python validates runner tests
> --
>
> Key: BEAM-8575
> URL: https://issues.apache.org/jira/browse/BEAM-8575
> Project: Beam
> Issue Type: Test
> Components: sdk-py-core, testing
>Reporter: wendy liu
>Assignee: wendy liu
>Priority: Major
> Time Spent: 29.5h
> Remaining Estimate: 0h
>
> This is the umbrella issue to track the work of adding more Python tests to
> improve test coverage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8930) External workers should receive artifact endpoint when started from python
[
https://issues.apache.org/jira/browse/BEAM-8930?focusedWorklogId=356588&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356588
]
ASF GitHub Bot logged work on BEAM-8930:
Author: ASF GitHub Bot
Created on: 09/Dec/19 21:37
Start Date: 09/Dec/19 21:37
Worklog Time Spent: 10m
Work Description: tweise commented on issue #10328: [BEAM-8930] Pass
artifact and provision endpoints to external workers from python
URL: https://github.com/apache/beam/pull/10328#issuecomment-563451501
Run Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356588)
Time Spent: 0.5h (was: 20m)
> External workers should receive artifact endpoint when started from python
> --
>
> Key: BEAM-8930
> URL: https://issues.apache.org/jira/browse/BEAM-8930
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {{ExternalWorkerHandler}} does not pass the artifact and provision endpoints,
> making it impossible to provision artifacts when the external worker is
> started from python. The Java code is properly sending this information.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7926) Show PCollection with Interactive Beam in a data-centric user flow
[
https://issues.apache.org/jira/browse/BEAM-7926?focusedWorklogId=356581&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356581
]
ASF GitHub Bot logged work on BEAM-7926:
Author: ASF GitHub Bot
Created on: 09/Dec/19 21:34
Start Date: 09/Dec/19 21:34
Worklog Time Spent: 10m
Work Description: KevinGG commented on issue #10276: [BEAM-7926]
Data-centric Interactive Part1
URL: https://github.com/apache/beam/pull/10276#issuecomment-563450019
R: @aaltay ,
Hi Ahmet, the merge conflicts have been resolved and the PR is ready to be
merged.
Thanks a lot!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356581)
Time Spent: 26.5h (was: 26h 20m)
> Show PCollection with Interactive Beam in a data-centric user flow
> --
>
> Key: BEAM-7926
> URL: https://issues.apache.org/jira/browse/BEAM-7926
> Project: Beam
> Issue Type: New Feature
> Components: runner-py-interactive
>Reporter: Ning Kang
>Assignee: Ning Kang
>Priority: Major
> Time Spent: 26.5h
> Remaining Estimate: 0h
>
> Support auto plotting / charting of materialized data of a given PCollection
> with Interactive Beam.
> Say an Interactive Beam pipeline defined as
>
> {code:java}
> p = beam.Pipeline(InteractiveRunner())
> pcoll = p | 'Transform' >> transform()
> pcoll2 = ...
> pcoll3 = ...{code}
> The use can call a single function and get auto-magical charting of the data.
> e.g.,
> {code:java}
> show(pcoll, pcoll2)
> {code}
> Throughout the process, a pipeline fragment is built to include only
> transforms necessary to produce the desired pcolls (pcoll and pcoll2) and
> execute that fragment.
> This makes the Interactive Beam user flow data-centric.
>
> Detailed
> [design|https://docs.google.com/document/d/1DYWrT6GL_qDCXhRMoxpjinlVAfHeVilK5Mtf8gO6zxQ/edit#heading=h.v6k2o3roarzz].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests
[ https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=356584&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356584 ] ASF GitHub Bot logged work on BEAM-8575: Author: ASF GitHub Bot Created on: 09/Dec/19 21:34 Start Date: 09/Dec/19 21:34 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10159: [BEAM-8575] Added a unit test to CombineTest class to test that Combi… URL: https://github.com/apache/beam/pull/10159#discussion_r355696654 ## File path: sdks/python/apache_beam/transforms/combiners_test.py ## @@ -399,6 +418,43 @@ def test_global_fanout(self): | beam.CombineGlobally(combine.MeanCombineFn()).with_fanout(11)) assert_that(result, equal_to([49.5])) + def test_combining_with_accumulation_mode(self): +# PCollection will contain elements from 1 to 5. +elements = [i for i in range(1, 6)] + +ts = TestStream().advance_watermark_to(0) +for i in elements: + ts.add_elements([i]) +ts.advance_watermark_to_infinity() + +options = PipelineOptions() +options.view_as(StandardOptions).streaming = True +with TestPipeline(options=options) as p: + result = (p +| ts +| beam.WindowInto( +GlobalWindows(), +accumulation_mode=trigger.AccumulationMode.ACCUMULATING, +trigger=AfterWatermark(early=AfterAll(AfterCount(1))) +) +| beam.CombineGlobally(sum).without_defaults().with_fanout(2) +| beam.ParDo(self.record_dofn())) + +# The trigger should fire repeatedly for each newly added element, +# and at least once for advancing the watermark to infinity. +# The firings should accumulate the output. +# First firing: 1 = 1 +# Second firing: 3 = 1 + 2 +# Third firing: 6 = 1 + 2 + 3 +# Fourth firing: 10 = 1 + 2 + 3 + 4 +# Fifth firing: 15 = 1 + 2 + 3 + 4 + 5 +# Next firings: 15 = 15 + 0 (advancing the watermark to infinity) +# The exact number of firings may vary, Review comment: This should be deterministic. That's the point of TestStream. When does the last firing not happen? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356584) Time Spent: 29h 10m (was: 29h) > Add more Python validates runner tests > -- > > Key: BEAM-8575 > URL: https://issues.apache.org/jira/browse/BEAM-8575 > Project: Beam > Issue Type: Test > Components: sdk-py-core, testing >Reporter: wendy liu >Assignee: wendy liu >Priority: Major > Time Spent: 29h 10m > Remaining Estimate: 0h > > This is the umbrella issue to track the work of adding more Python tests to > improve test coverage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests
[ https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=356585&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356585 ] ASF GitHub Bot logged work on BEAM-8575: Author: ASF GitHub Bot Created on: 09/Dec/19 21:34 Start Date: 09/Dec/19 21:34 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10159: [BEAM-8575] Added a unit test to CombineTest class to test that Combi… URL: https://github.com/apache/beam/pull/10159#discussion_r355695696 ## File path: sdks/python/apache_beam/transforms/combiners_test.py ## @@ -31,24 +31,43 @@ import apache_beam as beam import apache_beam.transforms.combiners as combine from apache_beam.options.pipeline_options import PipelineOptions +from apache_beam.options.pipeline_options import StandardOptions from apache_beam.testing.test_pipeline import TestPipeline from apache_beam.testing.test_stream import TestStream from apache_beam.testing.util import assert_that from apache_beam.testing.util import equal_to from apache_beam.testing.util import equal_to_per_window +from apache_beam.transforms import trigger from apache_beam.transforms import window from apache_beam.transforms.core import CombineGlobally from apache_beam.transforms.core import Create from apache_beam.transforms.core import Map from apache_beam.transforms.display import DisplayData from apache_beam.transforms.display_test import DisplayDataItemMatcher from apache_beam.transforms.ptransform import PTransform +from apache_beam.transforms.trigger import AfterAll +from apache_beam.transforms.trigger import AfterCount +from apache_beam.transforms.trigger import AfterWatermark +from apache_beam.transforms.window import GlobalWindows from apache_beam.transforms.window import TimestampCombiner from apache_beam.typehints import TypeCheckError from apache_beam.utils.timestamp import Timestamp class CombineTest(unittest.TestCase): + def setUp(self): +# Use state on the TestCase class, since other references would be pickled +# into a closure and not have the desired side effects. +CombineTest.all_records = [] Review comment: This make it so we can't make any of these validates runner. If you need to record things in order, use state. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356585) Time Spent: 29h 20m (was: 29h 10m) > Add more Python validates runner tests > -- > > Key: BEAM-8575 > URL: https://issues.apache.org/jira/browse/BEAM-8575 > Project: Beam > Issue Type: Test > Components: sdk-py-core, testing >Reporter: wendy liu >Assignee: wendy liu >Priority: Major > Time Spent: 29h 20m > Remaining Estimate: 0h > > This is the umbrella issue to track the work of adding more Python tests to > improve test coverage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests
[ https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=356583&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356583 ] ASF GitHub Bot logged work on BEAM-8575: Author: ASF GitHub Bot Created on: 09/Dec/19 21:34 Start Date: 09/Dec/19 21:34 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10159: [BEAM-8575] Added a unit test to CombineTest class to test that Combi… URL: https://github.com/apache/beam/pull/10159#discussion_r355695873 ## File path: sdks/python/apache_beam/transforms/combiners_test.py ## @@ -31,24 +31,43 @@ import apache_beam as beam import apache_beam.transforms.combiners as combine from apache_beam.options.pipeline_options import PipelineOptions +from apache_beam.options.pipeline_options import StandardOptions from apache_beam.testing.test_pipeline import TestPipeline from apache_beam.testing.test_stream import TestStream from apache_beam.testing.util import assert_that from apache_beam.testing.util import equal_to from apache_beam.testing.util import equal_to_per_window +from apache_beam.transforms import trigger from apache_beam.transforms import window from apache_beam.transforms.core import CombineGlobally from apache_beam.transforms.core import Create from apache_beam.transforms.core import Map from apache_beam.transforms.display import DisplayData from apache_beam.transforms.display_test import DisplayDataItemMatcher from apache_beam.transforms.ptransform import PTransform +from apache_beam.transforms.trigger import AfterAll +from apache_beam.transforms.trigger import AfterCount +from apache_beam.transforms.trigger import AfterWatermark +from apache_beam.transforms.window import GlobalWindows from apache_beam.transforms.window import TimestampCombiner from apache_beam.typehints import TypeCheckError from apache_beam.utils.timestamp import Timestamp class CombineTest(unittest.TestCase): + def setUp(self): +# Use state on the TestCase class, since other references would be pickled +# into a closure and not have the desired side effects. +CombineTest.all_records = [] + + def record_dofn(self): Review comment: As before, don't use a DoFn where a lambda would suffice. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356583) > Add more Python validates runner tests > -- > > Key: BEAM-8575 > URL: https://issues.apache.org/jira/browse/BEAM-8575 > Project: Beam > Issue Type: Test > Components: sdk-py-core, testing >Reporter: wendy liu >Assignee: wendy liu >Priority: Major > Time Spent: 29h > Remaining Estimate: 0h > > This is the umbrella issue to track the work of adding more Python tests to > improve test coverage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests
[ https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=356582&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356582 ] ASF GitHub Bot logged work on BEAM-8575: Author: ASF GitHub Bot Created on: 09/Dec/19 21:34 Start Date: 09/Dec/19 21:34 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10159: [BEAM-8575] Added a unit test to CombineTest class to test that Combi… URL: https://github.com/apache/beam/pull/10159#discussion_r355695267 ## File path: sdks/python/apache_beam/transforms/combiners_test.py ## @@ -399,6 +418,43 @@ def test_global_fanout(self): | beam.CombineGlobally(combine.MeanCombineFn()).with_fanout(11)) assert_that(result, equal_to([49.5])) + def test_combining_with_accumulation_mode(self): +# PCollection will contain elements from 1 to 5. +elements = [i for i in range(1, 6)] + +ts = TestStream().advance_watermark_to(0) +for i in elements: + ts.add_elements([i]) +ts.advance_watermark_to_infinity() + +options = PipelineOptions() +options.view_as(StandardOptions).streaming = True Review comment: Note that due to https://issues.apache.org/jira/browse/BEAM-8601 this makes it impossible to use as a validates runner test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356582) Time Spent: 29h (was: 28h 50m) > Add more Python validates runner tests > -- > > Key: BEAM-8575 > URL: https://issues.apache.org/jira/browse/BEAM-8575 > Project: Beam > Issue Type: Test > Components: sdk-py-core, testing >Reporter: wendy liu >Assignee: wendy liu >Priority: Major > Time Spent: 29h > Remaining Estimate: 0h > > This is the umbrella issue to track the work of adding more Python tests to > improve test coverage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8932) Expose complete Cloud Pub/Sub messages through PubsubIO API
[ https://issues.apache.org/jira/browse/BEAM-8932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Collins updated BEAM-8932: - Status: Open (was: Triage Needed) > Expose complete Cloud Pub/Sub messages through PubsubIO API > --- > > Key: BEAM-8932 > URL: https://issues.apache.org/jira/browse/BEAM-8932 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Daniel Collins >Priority: Major > > The PubsubIO API only exposes a subset of the fields in the underlying > PubsubMessage protocol buffer. To accomodate future feature changes as well > as for greater compatability with code using the Cloud Pub/Sub apis, a method > to read and write these protocol messages should be exposed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8932) Expose complete Cloud Pub/Sub messages through PubsubIO API
Daniel Collins created BEAM-8932: Summary: Expose complete Cloud Pub/Sub messages through PubsubIO API Key: BEAM-8932 URL: https://issues.apache.org/jira/browse/BEAM-8932 Project: Beam Issue Type: Bug Components: beam-model Reporter: Daniel Collins The PubsubIO API only exposes a subset of the fields in the underlying PubsubMessage protocol buffer. To accomodate future feature changes as well as for greater compatability with code using the Cloud Pub/Sub apis, a method to read and write these protocol messages should be exposed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8337) Add Flink job server container images to release process
[
https://issues.apache.org/jira/browse/BEAM-8337?focusedWorklogId=356575&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356575
]
ASF GitHub Bot logged work on BEAM-8337:
Author: ASF GitHub Bot
Created on: 09/Dec/19 21:29
Start Date: 09/Dec/19 21:29
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #10272: [BEAM-8337]
publish Flink job server container images
URL: https://github.com/apache/beam/pull/10272#discussion_r355694293
##
File path: website/src/contribute/release-guide.md
##
@@ -691,6 +691,15 @@ done
./gradlew :sdks:go:container:dockerPush -Pdocker-tag=${RELEASE}_rc{RC_NUM}
```
+* Build Flink job server images and push to DockerHub.
+
+```
+FLINK_VER=("1.7", "1.8", "1.9")
Review comment:
Included your suggestions @mxm
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356575)
Time Spent: 3h 20m (was: 3h 10m)
> Add Flink job server container images to release process
>
>
> Key: BEAM-8337
> URL: https://issues.apache.org/jira/browse/BEAM-8337
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Could be added to the release process similar to how we now publish SDK
> worker images.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8858) Missing dependencies in SQL module pom
[ https://issues.apache.org/jira/browse/BEAM-8858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991954#comment-16991954 ] Tomo Suzuki commented on BEAM-8858: --- Beam Java SQL Extension undeclared dependency https://gist.github.com/suztomo/e3fa9e5ce9eda5265082c1db8401d6db > Missing dependencies in SQL module pom > -- > > Key: BEAM-8858 > URL: https://issues.apache.org/jira/browse/BEAM-8858 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Affects Versions: 2.18.0 >Reporter: Ismaël Mejía >Assignee: Kenneth Knowles >Priority: Blocker > Fix For: 2.18.0 > > > I moved an user project that depends on the SQL module to the version > 2.18.0-SNAPSHOT and I discovered that it was not working anymore because some > of the transitive dependencies were not correctly exposed anymore by the SQL > module e.g. commons-codec and Nullable annotations dependency. > There are some missing dependencies in the generated pom file when compared > with previous versions, probably cases of `usedUndeclaredArtifacts` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests
[ https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=356570&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356570 ] ASF GitHub Bot logged work on BEAM-8575: Author: ASF GitHub Bot Created on: 09/Dec/19 21:27 Start Date: 09/Dec/19 21:27 Worklog Time Spent: 10m Work Description: robertwb commented on issue #10173: [BEAM-8575] Added two unit tests in CombineTest class to test AccumulatingCombine URL: https://github.com/apache/beam/pull/10173#issuecomment-563447339 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356570) Time Spent: 28h 50m (was: 28h 40m) > Add more Python validates runner tests > -- > > Key: BEAM-8575 > URL: https://issues.apache.org/jira/browse/BEAM-8575 > Project: Beam > Issue Type: Test > Components: sdk-py-core, testing >Reporter: wendy liu >Assignee: wendy liu >Priority: Major > Time Spent: 28h 50m > Remaining Estimate: 0h > > This is the umbrella issue to track the work of adding more Python tests to > improve test coverage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
