[jira] [Commented] (BEAM-9710) Got current time instead of timestamp value
[
https://issues.apache.org/jira/browse/BEAM-9710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125583#comment-17125583
]
Robin Qiu commented on BEAM-9710:
-
This will be fixed after the next ZetaSql release is available. I expect that
to come soon.
> Got current time instead of timestamp value
> ---
>
> Key: BEAM-9710
> URL: https://issues.apache.org/jira/browse/BEAM-9710
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Assignee: Robin Qiu
>Priority: P4
> Labels: stale-assigned, zetasql-compliance
>
> one failure in shard 13
> {code}
> Expected: ARRAY>[{2014-12-01 00:00:00+00}]
> Actual: ARRAY>[{2020-04-06
> 00:20:40.052+00}],
> {code}
> {code}
> [prepare_database]
> CREATE TABLE Table1 AS
> SELECT timestamp '2014-12-01' as timestamp_val
> --
> ARRAY>[{2014-12-01 00:00:00+00}]
> ==
> [name=timestamp_type_2]
> SELECT timestamp_val
> FROM Table1
> --
> ARRAY>[{2014-12-01 00:00:00+00}]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9179) Simplify types in Beam ZetaSQL
[ https://issues.apache.org/jira/browse/BEAM-9179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125582#comment-17125582 ] Robin Qiu commented on BEAM-9179: - This is an umbrella issue for the work I have been working on. I would like to keep it open. > Simplify types in Beam ZetaSQL > -- > > Key: BEAM-9179 > URL: https://issues.apache.org/jira/browse/BEAM-9179 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Robin Qiu >Assignee: Robin Qiu >Priority: P2 > Labels: stale-assigned > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently, how types work in Beam ZetaSQL is very confusing: code that does > type conversion between Calcite/Beam/ZetaSQL type systems are everywhere and > is likely to be not consistent. In addition, the recent migration to > BeamZetaSqlCalcRel makes the situation even worse: it adds another layer for > type conversion and makes a lot of code (for making ZetaSQL functions > manually) obsolete. This makes debugging and supporting new features very > hard. A few examples: > * Support new ZetaSQL types (e.g. DATE/TIME/DATETIME) > * Fix the long-standing precision & timezone issue of TIMESTMAP type > * Support BeamSQL query parameters > Therefore, I propose we work on the following items to make the story of Beam > ZetaSQL type clearer: > * Define a clear mapping between Calcite/Beam/ZetaSQL types and values > * Refactor the codebase to move all type conversion code to a single source > of truth > * Add thorough testing for it > * Remove obsolete code that is added to make functions work before > BeamZetaSqlCalcRel -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9641) Support ZetaSQL DATE functions in BeamSQL
[ https://issues.apache.org/jira/browse/BEAM-9641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Qiu resolved BEAM-9641. - Fix Version/s: 2.22.0 Resolution: Fixed > Support ZetaSQL DATE functions in BeamSQL > - > > Key: BEAM-9641 > URL: https://issues.apache.org/jira/browse/BEAM-9641 > Project: Beam > Issue Type: New Feature > Components: dsl-sql-zetasql >Reporter: Robin Qiu >Assignee: Robin Qiu >Priority: P2 > Labels: stale-assigned, zetasql-compliance > Fix For: 2.22.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9709) timezone off by 8 hours
[
https://issues.apache.org/jira/browse/BEAM-9709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125576#comment-17125576
]
Robin Qiu commented on BEAM-9709:
-
This will be fixed after the next ZetaSql release is available. I expect that
to come soon.
> timezone off by 8 hours
> ---
>
> Key: BEAM-9709
> URL: https://issues.apache.org/jira/browse/BEAM-9709
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Assignee: Robin Qiu
>Priority: P4
> Labels: stale-assigned, zetasql-compliance
>
> two failures in shard 13, one failure in shard 19
> {code}
> Expected: ARRAY>[{2014-01-31 00:00:00+00}]
> Actual: ARRAY>[{2014-01-31 08:00:00+00}],
> {code}
> {code}
> select timestamp(date '2014-01-31')
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9712) setting default timezone doesn't work
[
https://issues.apache.org/jira/browse/BEAM-9712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125574#comment-17125574
]
Robin Qiu commented on BEAM-9712:
-
This will be fixed after the next ZetaSql release is available. I expect that
to come soon.
> setting default timezone doesn't work
> -
>
> Key: BEAM-9712
> URL: https://issues.apache.org/jira/browse/BEAM-9712
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Assignee: Robin Qiu
>Priority: P4
> Labels: stale-assigned, zetasql-compliance
>
> several failures in shard 14
> (note: fixing the internal tests requires plumbing through the timezone
> config.)
> {code}
> [name=timestamp_to_string_1]
> select [cast(timestamp "2015-01-28" as string),
> cast(timestamp "2015-01-28 00:00:00" as string),
> cast(timestamp "2015-01-28 00:00:00.0" as string),
> cast(timestamp "2015-01-28 00:00:00.00" as string),
> cast(timestamp "2015-01-28 00:00:00.000" as string),
> cast(timestamp "2015-01-28 00:00:00." as string),
> cast(timestamp "2015-01-28 00:00:00.0" as string),
> cast(timestamp "2015-01-28 00:00:00.00" as string)]
> --
> ARRAY>>[
> {ARRAY[
> "2015-01-28 00:00:00+13:45",
> "2015-01-28 00:00:00+13:45",
> "2015-01-28 00:00:00+13:45",
> "2015-01-28 00:00:00+13:45",
> "2015-01-28 00:00:00+13:45",
> "2015-01-28 00:00:00+13:45",
> "2015-01-28 00:00:00+13:45",
> "2015-01-28 00:00:00+13:45"
>]}
> ]
> {code}
> {code}
> [default_time_zone=Pacific/Chatham]
> [name=timestamp_to_string_1]
> select [cast(timestamp "2015-01-28" as string),
> cast(timestamp "2015-01-28 00:00:00" as string),
> cast(timestamp "2015-01-28 00:00:00.0" as string),
> cast(timestamp "2015-01-28 00:00:00.00" as string),
> cast(timestamp "2015-01-28 00:00:00.000" as string),
> cast(timestamp "2015-01-28 00:00:00." as string),
> cast(timestamp "2015-01-28 00:00:00.0" as string),
> cast(timestamp "2015-01-28 00:00:00.00" as string)]
> --
> ARRAY>>[
> {ARRAY[
> "2015-01-28 00:00:00+13:45",
> "2015-01-28 00:00:00+13:45",
> "2015-01-28 00:00:00+13:45",
> "2015-01-28 00:00:00+13:45",
> "2015-01-28 00:00:00+13:45",
> "2015-01-28 00:00:00+13:45",
> "2015-01-28 00:00:00+13:45",
> "2015-01-28 00:00:00+13:45"
>]}
> ]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-10188) Automate Github release
[ https://issues.apache.org/jira/browse/BEAM-10188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125568#comment-17125568 ] Julien Phalip commented on BEAM-10188: -- I submitted a PR for this here: [https://github.com/apache/beam/pull/11918] > Automate Github release > --- > > Key: BEAM-10188 > URL: https://issues.apache.org/jira/browse/BEAM-10188 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Kyle Weaver >Priority: P2 > > Currently, we push the tag to Github and fill in the release notes in > separate steps. For feeds consuming these updates, it would be better to do > both in the same step using the Github API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9679) Core Transforms | Go SDK Code Katas
[
https://issues.apache.org/jira/browse/BEAM-9679?focusedWorklogId=441104&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441104
]
ASF GitHub Bot logged work on BEAM-9679:
Author: ASF GitHub Bot
Created on: 04/Jun/20 05:19
Start Date: 04/Jun/20 05:19
Worklog Time Spent: 10m
Work Description: henryken commented on a change in pull request #11883:
URL: https://github.com/apache/beam/pull/11883#discussion_r434997866
##
File path: learning/katas/go/Core Transforms/Combine/CombineFn/pkg/task/task.go
##
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package task
+
+import (
+ "github.com/apache/beam/sdks/go/pkg/beam"
+)
+
+func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection {
+ return beam.Combine(s, &averageFn{}, input)
Review comment:
I would suggest this to be part of answer placeholder as well.
##
File path: learning/katas/go/Core Transforms/Combine/CombineFn/pkg/task/task.go
##
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package task
+
+import (
+ "github.com/apache/beam/sdks/go/pkg/beam"
+)
+
+func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection {
+ return beam.Combine(s, &averageFn{}, input)
+}
+
+type averageAccum struct {
+ Count int64
+ Sum float64
+}
+
+type averageFn struct{}
+
+func (c *averageFn) CreateAccumulator() averageAccum {
+ return averageAccum{}
+}
+
+func (c *averageFn) AddInput(accum averageAccum, input int) averageAccum {
+ accum.Count++
+ accum.Sum += float64(input)
Review comment:
Why should the input be casted to float?
I think since the inputs are all int, the Sum can be int?
##
File path: learning/katas/go/Core Transforms/Combine/CombineFn/pkg/task/task.go
##
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package task
+
+import (
+ "github.com/apache/beam/sdks/go/pkg/beam"
+)
+
+func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection {
+ return beam.Combine(s, &averageFn{}, input)
+}
+
+type averageAccum struct {
+ Count int64
+ Sum float64
+}
+
+type averageFn struct{}
+
+func (c *averageFn) CreateAccumulator() averageAccum {
Review comment:
Is there any good documentation that outlines what function names that
need to be implemented?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above
[jira] [Updated] (BEAM-9096) Processing stuck in step Write Valid Event To GCS/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/ReadStream
[ https://issues.apache.org/jira/browse/BEAM-9096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaydeep Vaghasiya updated BEAM-9096: Labels: p2 (was: stale-P2) > Processing stuck in step Write Valid Event To > GCS/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/ReadStream > -- > > Key: BEAM-9096 > URL: https://issues.apache.org/jira/browse/BEAM-9096 > Project: Beam > Issue Type: Bug > Components: io-java-gcp, runner-dataflow >Affects Versions: 2.16.0 > Environment: Dataflow >Reporter: Jaydeep Vaghasiya >Priority: P2 > Labels: p2 > > Using JAVA SDK of apache-beam (2.16) in dataflow I am trying to write > streaming data from Kafka to GCS using dataflow, I am facing an issue with > writing data to bucket in my pipeline. Looking at the error I can see that > the processing gets stuck for a while in step WriteShardedBundlesToTempFiles > and eventually it gets terminated, which results in data loss. > Also, this issue is not persistent. Restarting the pipeline resolves it some > time and it may also fail again. > below is a stack trace > ``` > Processing stuck in step Write Valid Event To > GCS/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/ReadStream for > at least 05m01s without outputting or completing in state process at > org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110) > at > org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73) > at > org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:153) > at > org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143) > at > org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105) > at > org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73) > at > org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60) > at org.apache.beam.sdk.coders.AvroCoder.encode(AvroCoder.java:306) at > org.apache.beam.sdk.coders.Coder.getEncodedElementByteSize(Coder.java:297) at > org.apache.beam.sdk.coders.Coder.registerByteSizeObserver(Coder.java:291) at > org.apache.beam.sdk.coders.IterableLikeCoder.registerByteSizeObserver(IterableLikeCoder.java:203) > at > org.apache.beam.sdk.coders.IterableLikeCoder.registerByteSizeObserver(IterableLikeCoder.java:60) > at > org.apache.beam.sdk.coders.KvCoder.registerByteSizeObserver(KvCoder.java:128) > at > org.apache.beam.sdk.coders.KvCoder.registerByteSizeObserver(KvCoder.java:36) > at > org.apache.beam.sdk.util.WindowedValue$FullWindowedValueCoder.registerByteSizeObserver(WindowedValue.java:613) > at > org.apache.beam.sdk.util.WindowedValue$FullWindowedValueCoder.registerByteSizeObserver(WindowedValue.java:529) > at > org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$ElementByteSizeObservableCoder.registerByteSizeObserver(IntrinsicMapTaskExecutorFactory.java:400) > at > org.apache.beam.runners.dataflow.worker.util.common.worker.OutputObjectAndByteCounter.update(OutputObjectAndByteCounter.java:125) > at > org.apache.beam.runners.dataflow.worker.DataflowOutputCounter.update(DataflowOutputCounter.java:64) > at > org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:43) > at > org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowsParDoFn$1.output(GroupAlsoByWindowsParDoFn.java:182) > at > org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner$1.outputWindowedValue(GroupAlsoByWindowFnRunner.java:102) > at > org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.ReduceFnRunner.lambda$onTrigger$1(ReduceFnRunner.java:1057) > at > org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.ReduceFnRunner$$Lambda$176/1157856648.output(Unknown > Source) at > org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.ReduceFnContextFactory$OnTriggerContextImpl.output(ReduceFnContextFactory.java:438) > at > org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SystemReduceFn.onTrigger(SystemReduceFn.java:125) > at > org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.ReduceFnRunner.onTrigger(ReduceFnRunner.java:1060) > at > org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.ReduceFnRunner.emit(ReduceFnRunner.java:930) > at > org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.ReduceFnRunner.onTimers(ReduceFnRunner.java:790) > at > org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(Stream
[jira] [Commented] (BEAM-3736) Add SetUp() and TearDown() for CombineFns
[ https://issues.apache.org/jira/browse/BEAM-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125520#comment-17125520 ] Pawel Pasterz commented on BEAM-3736: - [~lcwik] [~kenn] Thanks for the input! > Add SetUp() and TearDown() for CombineFns > - > > Key: BEAM-3736 > URL: https://issues.apache.org/jira/browse/BEAM-3736 > Project: Beam > Issue Type: Improvement > Components: beam-model, sdk-py-core >Reporter: Chuan Yu Foo >Priority: P2 > > I have a CombineFn that has a large amount of state that needs to be loaded > once before it can add_input or merge_combiners (for example, the CombineFn > might load up a large lookup table used for combining). > Right now, to initialise this state, for each of the methods, I check if the > state has already been initialised, and if not, I initialise it. It would be > nice if CombineFn provided a SetUp() method that is called once to initialise > this state (and a corresponding TearDown() method to clean up this state if > necessary). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8602) Always use shadow configuration for direct runner dependencies
[ https://issues.apache.org/jira/browse/BEAM-8602?focusedWorklogId=441072&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441072 ] ASF GitHub Bot logged work on BEAM-8602: Author: ASF GitHub Bot Created on: 04/Jun/20 02:26 Start Date: 04/Jun/20 02:26 Worklog Time Spent: 10m Work Description: lukecwik merged pull request #11917: URL: https://github.com/apache/beam/pull/11917 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 441072) Time Spent: 1h 10m (was: 1h) > Always use shadow configuration for direct runner dependencies > -- > > Key: BEAM-8602 > URL: https://issues.apache.org/jira/browse/BEAM-8602 > Project: Beam > Issue Type: Improvement > Components: dsl-sql, sdk-java-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: P2 > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8602) Always use shadow configuration for direct runner dependencies
[ https://issues.apache.org/jira/browse/BEAM-8602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Cwik resolved BEAM-8602. - Fix Version/s: 2.23.0 Resolution: Fixed > Always use shadow configuration for direct runner dependencies > -- > > Key: BEAM-8602 > URL: https://issues.apache.org/jira/browse/BEAM-8602 > Project: Beam > Issue Type: Improvement > Components: dsl-sql, sdk-java-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: P2 > Fix For: 2.23.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7405) Task :sdks:python:hdfsIntegrationTest is failing in Python PostCommits - docker-credential-gcloud not installed
[
https://issues.apache.org/jira/browse/BEAM-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125481#comment-17125481
]
Udi Meiri commented on BEAM-7405:
-
Well there was the workaround I put in place, but I left the bug open for
option no. 5 which required a VM image update.
> Task :sdks:python:hdfsIntegrationTest is failing in Python PostCommits -
> docker-credential-gcloud not installed
> ---
>
> Key: BEAM-7405
> URL: https://issues.apache.org/jira/browse/BEAM-7405
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Udi Meiri
>Priority: P2
> Labels: stale-assigned
> Fix For: 2.14.0
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> This failure happened on apache-beam-jenkins-14.
> {noformat}
> 18:47:03 > Task :sdks:python:hdfsIntegrationTest
> 18:47:03 ++ dirname
> ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
> 18:47:03 + TEST_DIR=./apache_beam/io/hdfs_integration_test
> 18:47:03 + ROOT_DIR=./apache_beam/io/hdfs_integration_test/../../../../..
> 18:47:03 +
> CONTEXT_DIR=./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 + rm -r
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 rm: cannot remove
> './apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration':
> No such file or directory
> 18:47:03 + true
> 18:47:03 + mkdir -p
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/sdks
> 18:47:03 + cp ./apache_beam/io/hdfs_integration_test/docker-compose.yml
> ./apache_beam/io/hdfs_integration_test/Dockerfile
> ./apache_beam/io/hdfs_integration_test/hdfscli.cfg
> ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/
> 18:47:03 + cp -r
> ./apache_beam/io/hdfs_integration_test/../../../../../sdks/python
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/sdks/
> 18:47:03 + cp -r ./apache_beam/io/hdfs_integration_test/../../../../../model
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/
> 18:47:03 ++ echo hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714
> 18:47:03 + PROJECT_NAME=hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714
> 18:47:03 + '[' -z jenkins-beam_PostCommit_Python_Verify_PR-714 ']'
> 18:47:03 + COLOR_OPT=--no-ansi
> 18:47:03 + COMPOSE_OPT='-p
> hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714 --no-ansi'
> 18:47:03 + cd
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 + docker network prune --force
> 18:47:03 + trap finally EXIT
> 18:47:03 + docker-compose -p
> hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714 --no-ansi build
> 18:47:03 namenode uses an image, skipping
> 18:47:03 datanode uses an image, skipping
> 18:47:03 Building test
> 18:47:03 [29234] Failed to execute script docker-compose
> 18:47:03 Traceback (most recent call last):
> 18:47:03 File "bin/docker-compose", line 6, in
> 18:47:03 File "compose/cli/main.py", line 71, in main
> 18:47:03 File "compose/cli/main.py", line 127, in perform_command
> 18:47:03 File "compose/cli/main.py", line 287, in build
> 18:47:03 File "compose/project.py", line 386, in build
> 18:47:03 File "compose/project.py", line 368, in build_service
> 18:47:03 File "compose/service.py", line 1084, in build
> 18:47:03 File "site-packages/docker/api/build.py", line 260, in build
> 18:47:03 File "site-packages/docker/api/build.py", line 307, in
> _set_auth_headers
> 18:47:03 File "site-packages/docker/auth.py", line 310, in
> get_all_credentials
> 18:47:03 File "site-packages/docker/auth.py", line 262, in
> _resolve_authconfig_credstore
> 18:47:03 File "site-packages/docker/auth.py", line 287, in
> _get_store_instance
> 18:47:03 File "site-packages/dockerpycreds/store.py", line 25, in __init__
> 18:47:03 dockerpycreds.errors.InitializationError: docker-credential-gcloud
> not installed or not available in PATH
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9802) Provide a way to customize automatically started services.
[ https://issues.apache.org/jira/browse/BEAM-9802?focusedWorklogId=441068&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441068 ] ASF GitHub Bot logged work on BEAM-9802: Author: ASF GitHub Bot Created on: 04/Jun/20 01:42 Start Date: 04/Jun/20 01:42 Worklog Time Spent: 10m Work Description: chamikaramj commented on a change in pull request #11495: URL: https://github.com/apache/beam/pull/11495#discussion_r434945430 ## File path: sdks/python/apache_beam/options/pipeline_options.py ## @@ -425,6 +425,24 @@ def _add_argparse_args(cls, parser): help='Whether to enable streaming mode.') +class CrossLanguageOptions(PipelineOptions): + @classmethod Review comment: Hmm, I thought I understood this but may be I didn't :). Is this supposed to get picked by expansion services automatically started by IOs (for example Kafka) ? That doesn't seem to be working. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 441068) Time Spent: 1.5h (was: 1h 20m) > Provide a way to customize automatically started services. > -- > > Key: BEAM-9802 > URL: https://issues.apache.org/jira/browse/BEAM-9802 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > This can be useful for testing and alternative production environments. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10178) Error messages for unspecified options should display the command line flag that needs to be specified
[
https://issues.apache.org/jira/browse/BEAM-10178?focusedWorklogId=441064&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441064
]
ASF GitHub Bot logged work on BEAM-10178:
-
Author: ASF GitHub Bot
Created on: 04/Jun/20 01:23
Start Date: 04/Jun/20 01:23
Worklog Time Spent: 10m
Work Description: pabloem commented on pull request #11898:
URL: https://github.com/apache/beam/pull/11898#issuecomment-638545151
Run Java PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 441064)
Time Spent: 2.5h (was: 2h 20m)
> Error messages for unspecified options should display the command line flag
> that needs to be specified
> --
>
> Key: BEAM-10178
> URL: https://issues.apache.org/jira/browse/BEAM-10178
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core, sdk-py-core
>Reporter: Ning Kang
>Assignee: Ning Kang
>Priority: P3
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> An example error trace:
> {code:java}
> java.lang.IllegalArgumentException: BigQueryIO.Read needs a GCS temp location
> to store temp files.
> at
> org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$TypedRead.validate(BigQueryIO.java:762)
> at
> org.apache.beam.sdk.Pipeline$ValidateVisitor.enterCompositeTransform(Pipeline.java:641)
> at
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:653)
> at
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
> at
> org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:317)
> at
> org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:251)
> at
> org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458)
> at org.apache.beam.sdk.Pipeline.validate(Pipeline.java:577)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:312)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
> at
> org.apache.beam.examples.cookbook.BigQueryTornadoes.runBigQueryTornadoes(BigQueryTornadoes.java:199)
> at
> org.apache.beam.examples.cookbook.BigQueryTornadoesIT.runE2EBigQueryTornadoesTest(BigQueryTornadoesIT.java:70)
> at
> org.apache.beam.examples.cookbook.BigQueryTornadoesIT.testE2EBigQueryTornadoesWithExport(BigQueryTornadoesIT.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:349)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:314)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:312)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:396)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.exe
[jira] [Resolved] (BEAM-10003) Need two PR to submit snippets to website
[ https://issues.apache.org/jira/browse/BEAM-10003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aizhamal Nurmamat kyzy resolved BEAM-10003. --- Fix Version/s: 2.21.0 Resolution: Fixed > Need two PR to submit snippets to website > - > > Key: BEAM-10003 > URL: https://issues.apache.org/jira/browse/BEAM-10003 > Project: Beam > Issue Type: New Feature > Components: website >Reporter: Reza ardeshir rokni >Assignee: Aizhamal Nurmamat kyzy >Priority: P2 > Fix For: 2.21.0 > > Time Spent: 3h > Remaining Estimate: 0h > > Looks like build_github_samples.sh uses code already on the repo to build > local serving; > do > fileName=$(echo "$url" | sed -e 's/\//_/g') > curl -o "$DIST_DIR"/"$fileName" > "[https://raw.githubusercontent.com|https://raw.githubusercontent.com/]$url"; > done > So when tying to test locally, the code needs to have already be in Beam. > Ideally the script should make use of local code when building so : > 1- Easier to build & test changes. > 2- No need to raise two PR for what is a single change > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10189) Add ValueState to python sdk
[ https://issues.apache.org/jira/browse/BEAM-10189?focusedWorklogId=441063&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441063 ] ASF GitHub Bot logged work on BEAM-10189: - Author: ASF GitHub Bot Created on: 04/Jun/20 01:13 Start Date: 04/Jun/20 01:13 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11916: URL: https://github.com/apache/beam/pull/11916#issuecomment-638541628 Last time someone started adding this feature, we decided to call it ReadModifyWrite state. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 441063) Time Spent: 0.5h (was: 20m) > Add ValueState to python sdk > > > Key: BEAM-10189 > URL: https://issues.apache.org/jira/browse/BEAM-10189 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: P2 > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8602) Always use shadow configuration for direct runner dependencies
[ https://issues.apache.org/jira/browse/BEAM-8602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125431#comment-17125431 ] Brian Hulette commented on BEAM-8602: - Just put up a PR to close this out > Always use shadow configuration for direct runner dependencies > -- > > Key: BEAM-8602 > URL: https://issues.apache.org/jira/browse/BEAM-8602 > Project: Beam > Issue Type: Improvement > Components: dsl-sql, sdk-java-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: P2 > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8602) Always use shadow configuration for direct runner dependencies
[ https://issues.apache.org/jira/browse/BEAM-8602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Hulette updated BEAM-8602: Labels: (was: stale-assigned) > Always use shadow configuration for direct runner dependencies > -- > > Key: BEAM-8602 > URL: https://issues.apache.org/jira/browse/BEAM-8602 > Project: Beam > Issue Type: Improvement > Components: dsl-sql, sdk-java-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: P2 > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8602) Always use shadow configuration for direct runner dependencies
[ https://issues.apache.org/jira/browse/BEAM-8602?focusedWorklogId=441050&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441050 ] ASF GitHub Bot logged work on BEAM-8602: Author: ASF GitHub Bot Created on: 04/Jun/20 00:33 Start Date: 04/Jun/20 00:33 Worklog Time Spent: 10m Work Description: TheNeuralBit edited a comment on pull request #11917: URL: https://github.com/apache/beam/pull/11917#issuecomment-638530705 I grepped the build.gradle files to look for any other instances of adding the direct runner as a dependency dynamically, and couldn't find anything else: ``` ❯ grep -iIrn "runners:direct-java" **/build.gradle | grep -v shadow build.gradle:154: dependsOn ":runners:direct-java:needsRunnerTests" release/build.gradle:33: dependsOn ":runners:direct-java:runQuickstartJavaDirect" release/build.gradle:38: dependsOn ":runners:direct-java:runMobileGamingJavaDirect" runners/direct-java/build.gradle:154:// Generates :runners:direct-java:runQuickstartJavaDirect runners/direct-java/build.gradle:157:// Generates :runners:direct-java:runMobileGamingJavaDirect sdks/java/testing/load-tests/build.gradle:40:: ":runners:direct-java") sdks/java/testing/load-tests/build.gradle:78: // to not hardcode any runner other than :runners:direct-java sdks/java/testing/nexmark/build.gradle:35:?: ":runners:direct-java" sdks/java/testing/nexmark/build.gradle:85: // to not hardcode any runner other than :runners:direct-java sdks/java/testing/nexmark/build.gradle:107:// Defaults to ":runners:direct-java" ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 441050) Time Spent: 1h (was: 50m) > Always use shadow configuration for direct runner dependencies > -- > > Key: BEAM-8602 > URL: https://issues.apache.org/jira/browse/BEAM-8602 > Project: Beam > Issue Type: Improvement > Components: dsl-sql, sdk-java-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: P2 > Labels: stale-assigned > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8602) Always use shadow configuration for direct runner dependencies
[ https://issues.apache.org/jira/browse/BEAM-8602?focusedWorklogId=441049&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441049 ] ASF GitHub Bot logged work on BEAM-8602: Author: ASF GitHub Bot Created on: 04/Jun/20 00:33 Start Date: 04/Jun/20 00:33 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on pull request #11917: URL: https://github.com/apache/beam/pull/11917#issuecomment-638530705 I grepped the build.gradle files to look for any other instances of adding the direct runner as a dependency dynamically, and couldn't find anything else: ``` ❯ grep -iIrn "runners:direct-java" **/build.gradle | grep -v shadow build.gradle:154: dependsOn ":runners:direct-java:needsRunnerTests" release/build.gradle:33: dependsOn ":runners:direct-java:runQuickstartJavaDirect" release/build.gradle:38: dependsOn ":runners:direct-java:runMobileGamingJavaDirect" runners/direct-java/build.gradle:154:// Generates :runners:direct-java:runQuickstartJavaDirect runners/direct-java/build.gradle:157:// Generates :runners:direct-java:runMobileGamingJavaDirect sdks/java/testing/load-tests/build.gradle:40:: ":runners:direct-java") sdks/java/testing/load-tests/build.gradle:78: // to not hardcode any runner other than :runners:direct-java sdks/java/testing/nexmark/build.gradle:35:?: ":runners:direct-java" sdks/java/testing/nexmark/build.gradle:85: // to not hardcode any runner other than :runners:direct-java sdks/java/testing/nexmark/build.gradle:107:// Defaults to ":runners:direct-java" `` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 441049) Time Spent: 50m (was: 40m) > Always use shadow configuration for direct runner dependencies > -- > > Key: BEAM-8602 > URL: https://issues.apache.org/jira/browse/BEAM-8602 > Project: Beam > Issue Type: Improvement > Components: dsl-sql, sdk-java-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: P2 > Labels: stale-assigned > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8602) Always use shadow configuration for direct runner dependencies
[ https://issues.apache.org/jira/browse/BEAM-8602?focusedWorklogId=441047&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441047 ] ASF GitHub Bot logged work on BEAM-8602: Author: ASF GitHub Bot Created on: 04/Jun/20 00:31 Start Date: 04/Jun/20 00:31 Worklog Time Spent: 10m Work Description: TheNeuralBit opened a new pull request #11917: URL: https://github.com/apache/beam/pull/11917 R: @lukecwik Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [![Bu
[jira] [Work logged] (BEAM-10189) Add ValueState to python sdk
[ https://issues.apache.org/jira/browse/BEAM-10189?focusedWorklogId=441045&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441045 ] ASF GitHub Bot logged work on BEAM-10189: - Author: ASF GitHub Bot Created on: 04/Jun/20 00:25 Start Date: 04/Jun/20 00:25 Worklog Time Spent: 10m Work Description: y1chi commented on pull request #11916: URL: https://github.com/apache/beam/pull/11916#issuecomment-638528628 R: @angoenka @robertwb This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 441045) Time Spent: 20m (was: 10m) > Add ValueState to python sdk > > > Key: BEAM-10189 > URL: https://issues.apache.org/jira/browse/BEAM-10189 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: P2 > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10189) Add ValueState to python sdk
[ https://issues.apache.org/jira/browse/BEAM-10189?focusedWorklogId=441044&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441044 ] ASF GitHub Bot logged work on BEAM-10189: - Author: ASF GitHub Bot Created on: 04/Jun/20 00:25 Start Date: 04/Jun/20 00:25 Worklog Time Spent: 10m Work Description: y1chi opened a new pull request #11916: URL: https://github.com/apache/beam/pull/11916 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[ Samza tests failing [Java 11]
[
https://issues.apache.org/jira/browse/BEAM-10079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125416#comment-17125416
]
Xinyu Liu commented on BEAM-10079:
--
[~kenn]: hmm, this doesn't happen in our local folk at LinkedIn so not sure
whether it has something to do with Java 11. We can take a look on our side.
> Samza tests failing [Java 11]
> -
>
> Key: BEAM-10079
> URL: https://issues.apache.org/jira/browse/BEAM-10079
> Project: Beam
> Issue Type: Sub-task
> Components: runner-samza
>Reporter: Pawel Pasterz
>Priority: P2
>
> Gradle task *_:runners:samza:test_* fails during Java 11 Precommit job
>
> Example stack trace:
> {code:java}
> > Task :runners:samza:test
> May 26, 2020 7:33:55 AM
> org.apache.beam.runners.samza.runtime.SamzaTimerInternalsFactory$SamzaTimerState
> loadEventTimeTimers
> INFO: Loaded 0 event time timers in memory
> May 26, 2020 7:33:55 AM
> org.apache.beam.runners.samza.runtime.SamzaTimerInternalsFactory$SamzaTimerState
> loadProcessingTimeTimers
> INFO: Loaded 0 processing time timers in memory
> May 26, 2020 7:33:56 AM org.apache.samza.util.Logging$class info
> INFO: Registering task instances with consumers.
> May 26, 2020 7:33:56 AM org.apache.samza.util.Logging$class info
> INFO: Starting consumer multiplexer.
> May 26, 2020 7:33:56 AM org.apache.samza.util.Logging$class info
> INFO: Entering run loop.
> May 26, 2020 7:33:56 AM
> org.apache.samza.processor.StreamProcessor$ContainerListener afterStart
> WARNING: Received container start notification for container:
> org.apache.samza.container.SamzaContainer@af148fb in stream processor: 1.
> May 26, 2020 7:33:56 AM org.apache.samza.util.Logging$class info
> INFO: End of stream reached for partition: SystemStreamPartition
> [11-PAssert_0_GroupGlobally_Create_Values_Read_CreateSource__out__PCollection_,
>
> 11-PAssert_0_GroupGlobally_Create_Values_Read_CreateSource__out__PCollection_,
> 0]
> May 26, 2020 7:33:56 AM org.apache.beam.runners.samza.runtime.OpAdapter
> processWatermark
> SEVERE: Op org.apache.beam.runners.samza.runtime.OpAdapter threw an
> exception during processing watermark
> java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException:
> java.lang.IncompatibleClassChangeError: Method
> org.apache.samza.storage.kv.KeyValueStore.deleteAll(Ljava/util/List;)V must
> be InterfaceMethodref constant
> at
> org.apache.beam.runners.samza.metrics.DoFnRunnerWithMetrics.withMetrics(DoFnRunnerWithMetrics.java:98)
> at
> org.apache.beam.runners.samza.metrics.DoFnRunnerWithMetrics.processElement(DoFnRunnerWithMetrics.java:55)
> at
> org.apache.beam.runners.samza.runtime.GroupByKeyOp.fireTimer(GroupByKeyOp.java:225)
> at
> org.apache.beam.runners.samza.runtime.GroupByKeyOp.processWatermark(GroupByKeyOp.java:203)
> at
> org.apache.beam.runners.samza.runtime.OpAdapter.processWatermark(OpAdapter.java:109)
> at
> org.apache.samza.operators.impl.OperatorImpl.onWatermark(OperatorImpl.java:399)
> at
> org.apache.samza.operators.impl.OperatorImpl.lambda$propagateWatermark$26(OperatorImpl.java:433)
> at
> java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
> at
> java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1603)
> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
> at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
> at
> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:550)
> at
> java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
> at
> java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:517)
> at
> org.apache.samza.operators.impl.OperatorImpl.propagateWatermark(OperatorImpl.java:434)
> at
> org.apache.samza.operators.impl.OperatorImpl.lambda$onWatermark$25(OperatorImpl.java:416)
> at
> java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
> at
> java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
> at
> java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:143)
> at
> org.apache.samza.operators.impl.OperatorImpl.onWatermark(OperatorImpl.java:416)
> at
> org.apache.samza.operators.impl.OperatorImpl.lambda$propagateWatermark$26(OperatorImpl.java:433)
> at
> java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
> at
> java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1603)
> at
> java.base/java.util.str
[jira] [Commented] (BEAM-6466) KafkaIO doesn't commit offsets while being used as bounded source
[
https://issues.apache.org/jira/browse/BEAM-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125415#comment-17125415
]
Chai Nadig commented on BEAM-6466:
--
Hi, I'm running into this problem when using KafkaIO with fixed windows. I see
that the offsets don't get committed or advanced. The pipeline reads a few
records on start and then stops consuming any more records after that. We're
unable to scale our pipelines as we're restricted to using KafkaIO without
windows. Can we bump the priority to p1? Thanks.
> KafkaIO doesn't commit offsets while being used as bounded source
> -
>
> Key: BEAM-6466
> URL: https://issues.apache.org/jira/browse/BEAM-6466
> Project: Beam
> Issue Type: Bug
> Components: io-java-kafka
>Affects Versions: 2.9.0
>Reporter: Alexey Romanenko
>Priority: P2
> Labels: stale-P2
>
> While using KafkaIO as bounded source (with {{withMaxReadTime()}} or
> {{withMaxNumRecords()}}) it seems doesn't commit offsets all the time.
> See the details in [the
> discussion|https://lists.apache.org/thread.html/bcec8a1fb166029a4adf3f3491c407d49843406020b20f203ec3c2d2@%3Cuser.beam.apache.org%3E]
> on user@list.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-10063) Run pandas doctests for Beam dataframes API.
[
https://issues.apache.org/jira/browse/BEAM-10063?focusedWorklogId=441039&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441039
]
ASF GitHub Bot logged work on BEAM-10063:
-
Author: ASF GitHub Bot
Created on: 04/Jun/20 00:03
Start Date: 04/Jun/20 00:03
Worklog Time Spent: 10m
Work Description: TheNeuralBit commented on a change in pull request
#11787:
URL: https://github.com/apache/beam/pull/11787#discussion_r434919303
##
File path: sdks/python/apache_beam/dataframe/doctests.py
##
@@ -66,30 +93,19 @@ def __init__(self):
self._all_frames = {}
def fake_pandas_module(self):
-class FakePandas(object):
- """A stand-in for the pandas top-level module.
- """
- # For now, only populated with the frame types (below).
- # TODO(BEAM-9561): We may want to put more here.
- pass
-
-fake_pd = FakePandas()
-for pandas_type, deferred_type in DeferredFrame._pandas_type_map.items():
- setattr(
- fake_pd,
- pandas_type.__name__,
- self._deferred_frame(pandas_type, deferred_type))
-
-return fake_pd
-
- def _deferred_frame(self, pandas_type, deferred_type):
+return FakePandasObject(pd, self)
+
+ def _deferred_frame(self, pandas_callable):
"""Creates a "constructor" that record the actual value as an input and
returns a placeholder frame in its place."""
def wrapper(*args, **kwargs):
- df = pandas_type(*args, **kwargs)
- placeholder = expressions.PlaceholderExpression(df[0:0])
- self._inputs[placeholder] = df
- return deferred_type(placeholder)
+ df = pandas_callable(*args, **kwargs)
+ if type(df) in DeferredFrame._pandas_type_map.keys():
+placeholder = expressions.PlaceholderExpression(df[0:0])
+self._inputs[placeholder] = df
+return DeferredFrame.wrap(placeholder)
+ else:
+return df
Review comment:
It looks like this is inlined in `FakePandasObject.__call__` now, we
should either remove `_deferred_frame` or call it from FakePandasObject. I
think I'm partial to the latter since the method modifies `_inputs`, but either
way is fine.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 441039)
Time Spent: 1h 10m (was: 1h)
> Run pandas doctests for Beam dataframes API.
>
>
> Key: BEAM-10063
> URL: https://issues.apache.org/jira/browse/BEAM-10063
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: P2
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-10190) Reduce cost of toString of MetricKey and MetricName
[ https://issues.apache.org/jira/browse/BEAM-10190?focusedWorklogId=441038&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441038 ] ASF GitHub Bot logged work on BEAM-10190: - Author: ASF GitHub Bot Created on: 04/Jun/20 00:02 Start Date: 04/Jun/20 00:02 Worklog Time Spent: 10m Work Description: xinyuiscool commented on pull request #11915: URL: https://github.com/apache/beam/pull/11915#issuecomment-638522514 Sure, I will take a look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 441038) Time Spent: 0.5h (was: 20m) > Reduce cost of toString of MetricKey and MetricName > --- > > Key: BEAM-10190 > URL: https://issues.apache.org/jira/browse/BEAM-10190 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Yixing Zhang >Assignee: Yixing Zhang >Priority: P2 > Time Spent: 0.5h > Remaining Estimate: 0h > > Samza runner heavily uses MetricKey.toString() and MetricName.toString() to > update Samza metrics. We found that the toString methods have high CPU cost. > And according to this article: > [https://redfin.engineering/java-string-concatenation-which-way-is-best-8f590a7d22a8], > we should use "+" operator instead of String.format for string concatenation > for better performance. > We do see a 10% QPS gain in nexmark queries using Samza runner with the > change of using "+" operator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10036) More flexible dataframes partitioning.
[ https://issues.apache.org/jira/browse/BEAM-10036?focusedWorklogId=441032&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441032 ] ASF GitHub Bot logged work on BEAM-10036: - Author: ASF GitHub Bot Created on: 03/Jun/20 23:48 Start Date: 03/Jun/20 23:48 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on a change in pull request #11766: URL: https://github.com/apache/beam/pull/11766#discussion_r434916844 ## File path: sdks/python/apache_beam/dataframe/expressions.py ## @@ -85,16 +87,10 @@ def evaluate_at(self, session): # type: (Session) -> T """Returns the result of self with the bindings given in session.""" raise NotImplementedError(type(self)) - def requires_partition_by_index(self): # type: () -> bool -"""Whether this expression requires its argument(s) to be partitioned -by index.""" -# TODO: It might be necessary to support partitioning by part of the index, -# for some args, which would require returning more than a boolean here. + def requires_partition_by(self): # type: () -> Partitioning raise NotImplementedError(type(self)) - def preserves_partition_by_index(self): # type: () -> bool -"""Whether the result of this expression will be partitioned by index -whenever all of its inputs are partitioned by index.""" + def preserves_partition_by(self): # type: () -> Partitioning Review comment: Ah makes sense. So perhaps "preserves" could be thought of as an upper bound on the partitioning of the output (similar to how "requires" is a lower bound on the partitioning of the input). It looks like every current expression has preserves set to either Nothing or Singleton. Wouldn't it be simpler to just keep preserves as a boolean? Or maybe you have some other expression in mind where a boolean won't be sufficient? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 441032) Time Spent: 1h (was: 50m) > More flexible dataframes partitioning. > -- > > Key: BEAM-10036 > URL: https://issues.apache.org/jira/browse/BEAM-10036 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: P2 > Time Spent: 1h > Remaining Estimate: 0h > > Currently we only track a boolean of whether a dataframe is partitioned by > the (full) index. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9869) adding self-contained Kafka service jar for testing
[ https://issues.apache.org/jira/browse/BEAM-9869?focusedWorklogId=441027&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441027 ] ASF GitHub Bot logged work on BEAM-9869: Author: ASF GitHub Bot Created on: 03/Jun/20 23:41 Start Date: 03/Jun/20 23:41 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11846: URL: https://github.com/apache/beam/pull/11846#issuecomment-638516729 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 441027) Time Spent: 3h (was: 2h 50m) > adding self-contained Kafka service jar for testing > --- > > Key: BEAM-9869 > URL: https://issues.apache.org/jira/browse/BEAM-9869 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Labels: stale-assigned > Time Spent: 3h > Remaining Estimate: 0h > > adding self-contained Kafka service jar for testing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10192) Duplicate PubSub subscriptions with python direct runner in Jupyter/Colab environment
[ https://issues.apache.org/jira/browse/BEAM-10192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harrison updated BEAM-10192: Description: When running a streaming pipeline on Colab with direct runner, ReadFromPubSub can retain old subscriptions and cause message duplication. For example, manually killing a cell that is running a streaming pubsub pipeline does not delete the pubsub subscription. If the cell is rerun, the ReadFromPubSub component will actually be subscribed twice which results in duplicate messages. Manually deleting old subscriptions (e.g. via the GCP dashboard) temporarily fixes the problem. This Colab notebook: [https://gist.github.com/hgarrereyn/64ce87cbbcbe9c34ccdd13eafe49e3fb] contains a runnable example of the bug. was: When running a streaming pipeline on Colab with direct runner, ReadFromPubSub can retain old subscriptions and cause message duplication. For example, manually killing a cell that is running a streaming pubsub pipeline does not delete the pubsub subscription. If the cell is rerun, the ReadFromPubSub component will actually be subscribed twice which results in duplicate messages. Manually deleting old subscriptions (e.g. via the GCP dashboard) temporarily fixes the problem. This [Colab notebook|[https://gist.github.com/hgarrereyn/64ce87cbbcbe9c34ccdd13eafe49e3fb]] contains a runnable example of the bug. > Duplicate PubSub subscriptions with python direct runner in Jupyter/Colab > environment > - > > Key: BEAM-10192 > URL: https://issues.apache.org/jira/browse/BEAM-10192 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Affects Versions: 2.22.0 > Environment: Ubuntu 18 (Colab notebook), Python SDK >Reporter: Harrison >Priority: P2 > > When running a streaming pipeline on Colab with direct runner, ReadFromPubSub > can retain old subscriptions and cause message duplication. For example, > manually killing a cell that is running a streaming pubsub pipeline does not > delete the pubsub subscription. If the cell is rerun, the ReadFromPubSub > component will actually be subscribed twice which results in duplicate > messages. > Manually deleting old subscriptions (e.g. via the GCP dashboard) temporarily > fixes the problem. > This Colab notebook: > [https://gist.github.com/hgarrereyn/64ce87cbbcbe9c34ccdd13eafe49e3fb] > contains a runnable example of the bug. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10192) Duplicate PubSub subscriptions with python direct runner in Jupyter/Colab environment
Harrison created BEAM-10192: --- Summary: Duplicate PubSub subscriptions with python direct runner in Jupyter/Colab environment Key: BEAM-10192 URL: https://issues.apache.org/jira/browse/BEAM-10192 Project: Beam Issue Type: Bug Components: io-py-gcp Affects Versions: 2.22.0 Environment: Ubuntu 18 (Colab notebook), Python SDK Reporter: Harrison When running a streaming pipeline on Colab with direct runner, ReadFromPubSub can retain old subscriptions and cause message duplication. For example, manually killing a cell that is running a streaming pubsub pipeline does not delete the pubsub subscription. If the cell is rerun, the ReadFromPubSub component will actually be subscribed twice which results in duplicate messages. Manually deleting old subscriptions (e.g. via the GCP dashboard) temporarily fixes the problem. This [Colab notebook|[https://gist.github.com/hgarrereyn/64ce87cbbcbe9c34ccdd13eafe49e3fb]] contains a runnable example of the bug. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10080) Java Core Tests failing [Java 11]
[
https://issues.apache.org/jira/browse/BEAM-10080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-10080:
---
Status: Open (was: Triage Needed)
> Java Core Tests failing [Java 11]
> -
>
> Key: BEAM-10080
> URL: https://issues.apache.org/jira/browse/BEAM-10080
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-core
>Reporter: Pawel Pasterz
>Assignee: Kenneth Knowles
>Priority: P2
>
> {{Gradle task :sdks:java:core:test fails during Java 11 precommit job.}}
>
> Example stacktrace:
> {code:java}
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest >
> testGettersAnnotatedWithInconsistentDefault FAILED
> java.lang.AssertionError:
> Expected: (an instance of java.lang.IllegalArgumentException and
> exception with message a string containing "Property [object] is marked with
> contradictory annotations. Found [[Default.Integer(value=1) on
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithDefault#getObject()],
> [Default.String(value=abc) on
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithInconsistentDefaultType#getObject()]].")
> but: exception with message a string containing "Property [object]
> is marked with contradictory annotations. Found [[Default.Integer(value=1) on
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithDefault#getObject()],
> [Default.String(value=abc) on
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithInconsistentDefaultType#getObject()]]."
> message was "Property [object] is marked with contradictory annotations.
> Found [[Default.Integer(value=1) on
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithDefault#getObject()],
> [Default.String(value="abc") on
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithInconsistentDefaultType#getObject()]]."
> Stacktrace was: java.lang.IllegalArgumentException: Property [object] is
> marked with contradictory annotations. Found [[Default.Integer(value=1) on
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithDefault#getObject()],
> [Default.String(value="abc") on
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithInconsistentDefaultType#getObject()]].
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateGettersHaveConsistentAnnotation(PipelineOptionsFactory.java:1089)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateMethodAnnotations(PipelineOptionsFactory.java:1044)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:973)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.access$2200(PipelineOptionsFactory.java:115)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory$Cache.validateWellFormed(PipelineOptionsFactory.java:1901)
> at
> org.apache.beam.sdk.options.ProxyInvocationHandler.as(ProxyInvocationHandler.java:219)
> at
> org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:140)
> at com.sun.proxy.$Proxy139.as(Unknown Source)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest.testGettersAnnotatedWithInconsistentDefault(PipelineOptionsFactoryTest.java:651)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:266)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.
[jira] [Commented] (BEAM-10080) Java Core Tests failing [Java 11]
[
https://issues.apache.org/jira/browse/BEAM-10080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125401#comment-17125401
]
Kenneth Knowles commented on BEAM-10080:
This is just a bad test - it is checking for exact equality of error message
instead of keywords to check that the basic message is present.
> Java Core Tests failing [Java 11]
> -
>
> Key: BEAM-10080
> URL: https://issues.apache.org/jira/browse/BEAM-10080
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-core
>Reporter: Pawel Pasterz
>Assignee: Kenneth Knowles
>Priority: P2
>
> {{Gradle task :sdks:java:core:test fails during Java 11 precommit job.}}
>
> Example stacktrace:
> {code:java}
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest >
> testGettersAnnotatedWithInconsistentDefault FAILED
> java.lang.AssertionError:
> Expected: (an instance of java.lang.IllegalArgumentException and
> exception with message a string containing "Property [object] is marked with
> contradictory annotations. Found [[Default.Integer(value=1) on
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithDefault#getObject()],
> [Default.String(value=abc) on
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithInconsistentDefaultType#getObject()]].")
> but: exception with message a string containing "Property [object]
> is marked with contradictory annotations. Found [[Default.Integer(value=1) on
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithDefault#getObject()],
> [Default.String(value=abc) on
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithInconsistentDefaultType#getObject()]]."
> message was "Property [object] is marked with contradictory annotations.
> Found [[Default.Integer(value=1) on
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithDefault#getObject()],
> [Default.String(value="abc") on
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithInconsistentDefaultType#getObject()]]."
> Stacktrace was: java.lang.IllegalArgumentException: Property [object] is
> marked with contradictory annotations. Found [[Default.Integer(value=1) on
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithDefault#getObject()],
> [Default.String(value="abc") on
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithInconsistentDefaultType#getObject()]].
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateGettersHaveConsistentAnnotation(PipelineOptionsFactory.java:1089)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateMethodAnnotations(PipelineOptionsFactory.java:1044)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:973)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.access$2200(PipelineOptionsFactory.java:115)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory$Cache.validateWellFormed(PipelineOptionsFactory.java:1901)
> at
> org.apache.beam.sdk.options.ProxyInvocationHandler.as(ProxyInvocationHandler.java:219)
> at
> org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:140)
> at com.sun.proxy.$Proxy139.as(Unknown Source)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest.testGettersAnnotatedWithInconsistentDefault(PipelineOptionsFactoryTest.java:651)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:266)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
> at
> org.junit.runners.BlockJUnit4ClassRunne
[jira] [Assigned] (BEAM-10080) Java Core Tests failing [Java 11]
[
https://issues.apache.org/jira/browse/BEAM-10080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles reassigned BEAM-10080:
--
Assignee: Kenneth Knowles
> Java Core Tests failing [Java 11]
> -
>
> Key: BEAM-10080
> URL: https://issues.apache.org/jira/browse/BEAM-10080
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-core
>Reporter: Pawel Pasterz
>Assignee: Kenneth Knowles
>Priority: P2
>
> {{Gradle task :sdks:java:core:test fails during Java 11 precommit job.}}
>
> Example stacktrace:
> {code:java}
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest >
> testGettersAnnotatedWithInconsistentDefault FAILED
> java.lang.AssertionError:
> Expected: (an instance of java.lang.IllegalArgumentException and
> exception with message a string containing "Property [object] is marked with
> contradictory annotations. Found [[Default.Integer(value=1) on
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithDefault#getObject()],
> [Default.String(value=abc) on
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithInconsistentDefaultType#getObject()]].")
> but: exception with message a string containing "Property [object]
> is marked with contradictory annotations. Found [[Default.Integer(value=1) on
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithDefault#getObject()],
> [Default.String(value=abc) on
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithInconsistentDefaultType#getObject()]]."
> message was "Property [object] is marked with contradictory annotations.
> Found [[Default.Integer(value=1) on
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithDefault#getObject()],
> [Default.String(value="abc") on
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithInconsistentDefaultType#getObject()]]."
> Stacktrace was: java.lang.IllegalArgumentException: Property [object] is
> marked with contradictory annotations. Found [[Default.Integer(value=1) on
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithDefault#getObject()],
> [Default.String(value="abc") on
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithInconsistentDefaultType#getObject()]].
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateGettersHaveConsistentAnnotation(PipelineOptionsFactory.java:1089)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateMethodAnnotations(PipelineOptionsFactory.java:1044)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:973)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.access$2200(PipelineOptionsFactory.java:115)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory$Cache.validateWellFormed(PipelineOptionsFactory.java:1901)
> at
> org.apache.beam.sdk.options.ProxyInvocationHandler.as(ProxyInvocationHandler.java:219)
> at
> org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:140)
> at com.sun.proxy.$Proxy139.as(Unknown Source)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactoryTest.testGettersAnnotatedWithInconsistentDefault(PipelineOptionsFactoryTest.java:651)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:266)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.ru
[jira] [Commented] (BEAM-10081) GCP Core Tests failing [Java 11]
[
https://issues.apache.org/jira/browse/BEAM-10081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125395#comment-17125395
]
Kenneth Knowles commented on BEAM-10081:
My recollection is that the API surface tests are basically just broken anyhow.
They were changed from scanning API surfaces (a textual/semantic property of a
program) to scanning the classpath. The latter is maybe more useful, but is not
the API surface per se, and sensitive to irrelevancies like the test classpath.
It looks like Java 11 breaks them more, by making it so that the traversal of
the classpath does not detect anything.
I expect the solution for this one is to remove the test. I don't think it has
saved us any trouble recently.
> GCP Core Tests failing [Java 11]
>
>
> Key: BEAM-10081
> URL: https://issues.apache.org/jira/browse/BEAM-10081
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-core
>Reporter: Pawel Pasterz
>Priority: P2
>
> Gradle task _:sdks:java:extensions:google-cloud-platform-core:test_ fails
> during Java 11 Precommit job
> Example stack trace
> {code:java}
> > Task :sdks:java:extensions:google-cloud-platform-core:test
> org.apache.beam.sdk.extensions.gcp.GcpCoreApiSurfaceTest >
> testGcpCoreApiSurface FAILED
> java.lang.AssertionError:
> Expected: API surface to include only:
> Classes in package "com.google.api.client.googleapis"
> Classes in package "com.google.api.client.http"
> Classes in package "com.google.api.client.json"
> Classes in package "com.google.api.client.util"
> Classes in package "com.google.api.services.storage"
> Classes in package "com.google.auth"
> Classes in package "com.fasterxml.jackson.annotation"
> Classes in package "java"
> Classes in package "javax"
> Classes in package "org.apache.beam.sdk"
> Classes in package "org.joda.time"
> Classes in package "org.junit"
>
> but: The following white-listed scopes did not have matching classes
> on the API surface:
> No Classes in package "com.fasterxml.jackson.annotation"
> No Classes in package "com.google.api.client.googleapis"
> No Classes in package "com.google.api.client.http"
> No Classes in package "com.google.api.client.json"
> No Classes in package "com.google.api.client.util"
> No Classes in package "com.google.api.services.storage"
> No Classes in package "com.google.auth"
> No Classes in package "java"
> No Classes in package "javax"
> No Classes in package "org.apache.beam.sdk"
> No Classes in package "org.joda.time"
> No Classes in package "org.junit"
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
> at
> org.apache.beam.sdk.extensions.gcp.GcpCoreApiSurfaceTest.testGcpCoreApiSurface(GcpCoreApiSurfaceTest.java:63)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-10079) Samza tests failing [Java 11]
[
https://issues.apache.org/jira/browse/BEAM-10079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125394#comment-17125394
]
Kenneth Knowles commented on BEAM-10079:
[~xinyu] are you familiar with this error? I have never seen it actually.
> Samza tests failing [Java 11]
> -
>
> Key: BEAM-10079
> URL: https://issues.apache.org/jira/browse/BEAM-10079
> Project: Beam
> Issue Type: Sub-task
> Components: runner-samza
>Reporter: Pawel Pasterz
>Priority: P2
>
> Gradle task *_:runners:samza:test_* fails during Java 11 Precommit job
>
> Example stack trace:
> {code:java}
> > Task :runners:samza:test
> May 26, 2020 7:33:55 AM
> org.apache.beam.runners.samza.runtime.SamzaTimerInternalsFactory$SamzaTimerState
> loadEventTimeTimers
> INFO: Loaded 0 event time timers in memory
> May 26, 2020 7:33:55 AM
> org.apache.beam.runners.samza.runtime.SamzaTimerInternalsFactory$SamzaTimerState
> loadProcessingTimeTimers
> INFO: Loaded 0 processing time timers in memory
> May 26, 2020 7:33:56 AM org.apache.samza.util.Logging$class info
> INFO: Registering task instances with consumers.
> May 26, 2020 7:33:56 AM org.apache.samza.util.Logging$class info
> INFO: Starting consumer multiplexer.
> May 26, 2020 7:33:56 AM org.apache.samza.util.Logging$class info
> INFO: Entering run loop.
> May 26, 2020 7:33:56 AM
> org.apache.samza.processor.StreamProcessor$ContainerListener afterStart
> WARNING: Received container start notification for container:
> org.apache.samza.container.SamzaContainer@af148fb in stream processor: 1.
> May 26, 2020 7:33:56 AM org.apache.samza.util.Logging$class info
> INFO: End of stream reached for partition: SystemStreamPartition
> [11-PAssert_0_GroupGlobally_Create_Values_Read_CreateSource__out__PCollection_,
>
> 11-PAssert_0_GroupGlobally_Create_Values_Read_CreateSource__out__PCollection_,
> 0]
> May 26, 2020 7:33:56 AM org.apache.beam.runners.samza.runtime.OpAdapter
> processWatermark
> SEVERE: Op org.apache.beam.runners.samza.runtime.OpAdapter threw an
> exception during processing watermark
> java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException:
> java.lang.IncompatibleClassChangeError: Method
> org.apache.samza.storage.kv.KeyValueStore.deleteAll(Ljava/util/List;)V must
> be InterfaceMethodref constant
> at
> org.apache.beam.runners.samza.metrics.DoFnRunnerWithMetrics.withMetrics(DoFnRunnerWithMetrics.java:98)
> at
> org.apache.beam.runners.samza.metrics.DoFnRunnerWithMetrics.processElement(DoFnRunnerWithMetrics.java:55)
> at
> org.apache.beam.runners.samza.runtime.GroupByKeyOp.fireTimer(GroupByKeyOp.java:225)
> at
> org.apache.beam.runners.samza.runtime.GroupByKeyOp.processWatermark(GroupByKeyOp.java:203)
> at
> org.apache.beam.runners.samza.runtime.OpAdapter.processWatermark(OpAdapter.java:109)
> at
> org.apache.samza.operators.impl.OperatorImpl.onWatermark(OperatorImpl.java:399)
> at
> org.apache.samza.operators.impl.OperatorImpl.lambda$propagateWatermark$26(OperatorImpl.java:433)
> at
> java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
> at
> java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1603)
> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
> at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
> at
> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:550)
> at
> java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
> at
> java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:517)
> at
> org.apache.samza.operators.impl.OperatorImpl.propagateWatermark(OperatorImpl.java:434)
> at
> org.apache.samza.operators.impl.OperatorImpl.lambda$onWatermark$25(OperatorImpl.java:416)
> at
> java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
> at
> java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
> at
> java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:143)
> at
> org.apache.samza.operators.impl.OperatorImpl.onWatermark(OperatorImpl.java:416)
> at
> org.apache.samza.operators.impl.OperatorImpl.lambda$propagateWatermark$26(OperatorImpl.java:433)
> at
> java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
> at
> java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1603)
> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
> at
[jira] [Commented] (BEAM-10088) Examples Spark Runner tests failing [Java 11]
[
https://issues.apache.org/jira/browse/BEAM-10088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125392#comment-17125392
]
Kenneth Knowles commented on BEAM-10088:
[~iemejia] pinged you on another SparkRunner + Java 11 Jira. Is the issue that
it will only work with newer Spark maybe?
> Examples Spark Runner tests failing [Java 11]
> -
>
> Key: BEAM-10088
> URL: https://issues.apache.org/jira/browse/BEAM-10088
> Project: Beam
> Issue Type: Sub-task
> Components: runner-spark
>Reporter: Pawel Pasterz
>Priority: P2
>
> Gradle *_:examples:java:sparkRunnerPreCommit _*fails during Java 11 Precommit
> job
> Example stack trace:
> {code:java}
> > Task :examples:java:sparkRunnerPreCommit
> 20/05/26 07:11:13 INFO
> org.apache.beam.runners.spark.SparkRunner$Evaluator: Entering
> directly-translatable composite transform:
> 'WordCount.CountWords/Count.PerElement/Combine.perKey(Count)'
> 20/05/26 07:11:13 INFO
> org.apache.beam.runners.spark.SparkRunner$Evaluator: Entering
> directly-translatable composite transform:
> 'WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle'
> 20/05/26 07:11:14 INFO
> org.apache.beam.runners.spark.metrics.MetricsAccumulator: Instantiated
> metrics accumulator: {
> "metrics": {
> }
> }
> 20/05/26 07:11:14 INFO
> org.apache.beam.runners.spark.aggregators.AggregatorsAccumulator:
> Instantiated aggregators accumulator:
> 20/05/26 07:11:14 INFO
> org.apache.beam.runners.spark.SparkRunner$Evaluator: Evaluating
> Read(CompressedSource)
> 20/05/26 07:11:14 INFO
> org.apache.beam.runners.spark.SparkRunner$Evaluator: Evaluating
> org.apache.beam.examples.WordCount$ExtractWordsFn@174ba402
> 20/05/26 07:11:15 INFO
> org.apache.beam.runners.spark.SparkRunner$Evaluator: Evaluating
> org.apache.beam.sdk.transforms.MapElements$1@4c4d8e23
> 20/05/26 07:11:15 INFO
> org.apache.beam.runners.spark.SparkRunner$Evaluator: Entering
> directly-translatable composite transform:
> 'WordCount.CountWords/Count.PerElement/Combine.perKey(Count)'
> 20/05/26 07:11:15 INFO
> org.apache.beam.runners.spark.SparkRunner$Evaluator: Evaluating
> Combine.perKey(Count)
> 20/05/26 07:11:19 INFO
> org.apache.beam.runners.spark.SparkRunner$Evaluator: Evaluating
> org.apache.beam.sdk.transforms.MapElements$1@6bfdc26a
> 20/05/26 07:11:19 INFO
> org.apache.beam.runners.spark.SparkRunner$Evaluator: Evaluating Window.Assign
> 20/05/26 07:11:19 INFO
> org.apache.beam.runners.spark.SparkRunner$Evaluator: Evaluating
> org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesFn@57c4b142
> 20/05/26 07:11:19 INFO
> org.apache.beam.runners.spark.SparkRunner$Evaluator: Evaluating GroupByKey
> 20/05/26 07:11:19 WARN
> org.apache.beam.runners.spark.translation.GroupNonMergingWindowsFunctions:
> Either coder ShardedKeyCoder(VarIntCoder) or GlobalWindow$Coder is not
> consistent with equals. That might cause issues on some runners.
> 20/05/26 07:11:19 INFO
> org.apache.beam.runners.spark.SparkRunner$Evaluator: Evaluating
> org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn@3122d00d
> 20/05/26 07:11:19 INFO
> org.apache.beam.runners.spark.SparkRunner$Evaluator: Evaluating
> org.apache.beam.sdk.io.WriteFiles$WriteUnshardedBundlesToTempFiles$1@7639f2af
> 20/05/26 07:11:19 INFO
> org.apache.beam.runners.spark.SparkRunner$Evaluator: Evaluating
> Flatten.PCollections
> 20/05/26 07:11:19 INFO
> org.apache.beam.runners.spark.SparkRunner$Evaluator: Evaluating
> org.apache.beam.sdk.transforms.View$VoidKeyToMultimapMaterialization$VoidKeyToMultimapMaterializationDoFn@bcb712b
> 20/05/26 07:11:19 INFO
> org.apache.beam.runners.spark.SparkRunner$Evaluator: Evaluating
> View.CreatePCollectionView
> org.apache.beam.examples.WordCountIT > testE2EWordCount FAILED
> java.lang.IllegalArgumentException: Unsupported class file major version
> 55
> 1 test completed, 1 failed
> Finished generating test XML results (0.002 secs) into:
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java11_Phrase/src/examples/java/build/test-results/sparkRunnerPreCommit
> Generating HTML test report...
> Finished generating test html results (0.003 secs) into:
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java11_Phrase/src/examples/java/build/reports/tests/sparkRunnerPreCommit
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-10083) Spark Runner Tests failing [Java 11]
[
https://issues.apache.org/jira/browse/BEAM-10083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125391#comment-17125391
]
Kenneth Knowles commented on BEAM-10083:
[~iemejia] does this look familiar? Do I remember that you have used Beam with
Java 11? On SparkRunner?
> Spark Runner Tests failing [Java 11]
>
>
> Key: BEAM-10083
> URL: https://issues.apache.org/jira/browse/BEAM-10083
> Project: Beam
> Issue Type: Sub-task
> Components: runner-spark
>Reporter: Pawel Pasterz
>Priority: P2
>
> Gradle task *_:runners:spark:test_* fails during Java 11 Precommit job
>
> Example stack trace:
> {code:java}
> > Task :runners:spark:test
> 20/05/26 07:26:31 INFO
> org.apache.beam.runners.spark.metrics.MetricsAccumulator: Instantiated
> metrics accumulator: {
> "metrics": {
> }
> }
> org.apache.beam.runners.spark.structuredstreaming.StructuredStreamingPipelineStateTest
> > testBatchPipelineRunningState STANDARD_ERROR
> 20/05/26 07:26:32 INFO
> org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingRunner:
> *** SparkStructuredStreamingRunner is based on spark structured streaming
> framework and is no more
> based on RDD/DStream API. See
>
> https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html
> It is still experimental, its coverage of the Beam model is partial. ***
> org.apache.beam.runners.spark.SparkPortableExecutionTest > testExecution
> STANDARD_ERROR
> 20/05/26 07:26:33 WARN
> org.apache.beam.runners.spark.translation.GroupNonMergingWindowsFunctions:
> Either coder LengthPrefixCoder(ByteArrayCoder) or GlobalWindow$Coder is not
> consistent with equals. That might cause issues on some runners.
> org.apache.beam.runners.spark.structuredstreaming.translation.batch.FlattenTest
> > testFlatten STANDARD_ERROR
> 20/05/26 07:26:34 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to
> load native-hadoop library for your platform... using builtin-java classes
> where applicable
> org.apache.beam.runners.spark.SparkPortableExecutionTest > testExecution
> STANDARD_ERROR
> 20/05/26 07:26:34 ERROR
> org.apache.beam.runners.jobsubmission.JobInvocation: Error during job
> invocation fakeId.
> java.lang.IllegalArgumentException: Unsupported class file major version
> 55
> at org.apache.xbean.asm6.ClassReader.(ClassReader.java:166)
> at org.apache.xbean.asm6.ClassReader.(ClassReader.java:148)
> at org.apache.xbean.asm6.ClassReader.(ClassReader.java:136)
> at org.apache.xbean.asm6.ClassReader.(ClassReader.java:237)
> at
> org.apache.spark.util.ClosureCleaner$.getClassReader(ClosureCleaner.scala:49)
> at
> org.apache.spark.util.FieldAccessFinder$$anon$3$$anonfun$visitMethodInsn$2.apply(ClosureCleaner.scala:517)
> at
> org.apache.spark.util.FieldAccessFinder$$anon$3$$anonfun$visitMethodInsn$2.apply(ClosureCleaner.scala:500)
> at
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
> at
> scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:134)
> at
> scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:134)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap$$anon$1.foreach(HashMap.scala:134)
> at
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
> at
> org.apache.spark.util.FieldAccessFinder$$anon$3.visitMethodInsn(ClosureCleaner.scala:500)
> at org.apache.xbean.asm6.ClassReader.readCode(ClassReader.java:2175)
> at org.apache.xbean.asm6.ClassReader.readMethod(ClassReader.java:1238)
> at org.apache.xbean.asm6.ClassReader.accept(ClassReader.java:631)
> at org.apache.xbean.asm6.ClassReader.accept(ClassReader.java:355)
> at
> org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean$14.apply(ClosureCleaner.scala:307)
> at
> org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean$14.apply(ClosureCleaner.scala:306)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at
> org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:306)
> at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:162)
> at org.apache.spark.SparkContext.clean(SparkContext.scala:2326)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2100)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2126)
> at org.apache.spark.rdd.RDD$$anonfu
[jira] [Commented] (BEAM-10087) Flink 1.10 failing [Java 11]
[
https://issues.apache.org/jira/browse/BEAM-10087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125390#comment-17125390
]
Kenneth Knowles commented on BEAM-10087:
Interesting. [~mxm] do you know anything about this? It seems like it has to do
with the tests being compiled with Java 11 but there's a piece of code
involved in injecting data to the website that is built with Java 8 (or
presumably whatever "main" sourcesets are compiled with)
> Flink 1.10 failing [Java 11]
>
>
> Key: BEAM-10087
> URL: https://issues.apache.org/jira/browse/BEAM-10087
> Project: Beam
> Issue Type: Sub-task
> Components: runner-flink
>Reporter: Pawel Pasterz
>Priority: P2
>
> Gradle task _*:runners:flink:1.10:generatePipelineOptionsTableJava*_ fails
> during Java 11 Precommit job
> Example stack trace
> {code:java}
> > Task :runners:flink:1.10:generatePipelineOptionsTableJava FAILED
> ...
> Error: A JNI error has occurred, please check your installation and try again
> Exception in thread "main" java.lang.UnsupportedClassVersionError:
> org/apache/beam/runners/flink/website/PipelineOptionsTableGenerator has been
> compiled by a more recent version of the Java Runtime (class file version
> 55.0), this version of the Java Runtime only recognizes class file versions
> up to 52.0
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:495)
> :runners:flink:1.10:generatePipelineOptionsTableJava (Thread[Execution worker
> for ':' Thread 11,5,main]) completed. Took 1.744 secs.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-10089) Apex Runner tests failing [Java 11]
[
https://issues.apache.org/jira/browse/BEAM-10089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125389#comment-17125389
]
Kenneth Knowles commented on BEAM-10089:
This is a pretty good reason to delete it. We shouldn't even bother triaging
this. [~thw] already said it makes sense to remove. I feel like maybe good to
move such a big change through a dev thread, then a user thread. Once approved,
maybe it is a good starter task actually!
> Apex Runner tests failing [Java 11]
> ---
>
> Key: BEAM-10089
> URL: https://issues.apache.org/jira/browse/BEAM-10089
> Project: Beam
> Issue Type: Sub-task
> Components: runner-apex
>Reporter: Pawel Pasterz
>Priority: P2
> Labels: beam-fixit
>
> Gradle task _*:runners:apex:test*_ fails during Java 11 Precommit job
> Example stack trace:
> {code:java}
> org.apache.beam.runners.apex.ApexYarnLauncherTest >
> testGetYarnDeployDependencies FAILED
> java.lang.ClassCastException: class
> jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
> java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and
> java.net.URLClassLoader are in module java.base of loader 'bootstrap')
> at
> org.apache.beam.runners.apex.ApexYarnLauncher.getYarnDeployDependencies(ApexYarnLauncher.java:222)
> at
> org.apache.beam.runners.apex.ApexYarnLauncherTest.testGetYarnDeployDependencies(ApexYarnLauncherTest.java:56)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-10191) _find_protoc_gen_mypy: Windows support
Udi Meiri created BEAM-10191: Summary: _find_protoc_gen_mypy: Windows support Key: BEAM-10191 URL: https://issues.apache.org/jira/browse/BEAM-10191 Project: Beam Issue Type: Bug Components: sdk-py-core Reporter: Udi Meiri Assignee: Chad Dombrova "pip install -e .[gcp, test]" fails. The script is looking for protoc-gen-mypy when it should be looking for protoc_gen_mypy.bat. https://github.com/dropbox/mypy-protobuf#python-implementation -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-10084) XML IO java tests failing [Java 11]
[
https://issues.apache.org/jira/browse/BEAM-10084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125388#comment-17125388
]
Kenneth Knowles commented on BEAM-10084:
This looks like the chosen underlying XML libraries use private APIs. I think
we will need to upgrade or switch. I wonder how much usage XmlIO has. It is a
bit esoteric. [~chamikara]
> XML IO java tests failing [Java 11]
> ---
>
> Key: BEAM-10084
> URL: https://issues.apache.org/jira/browse/BEAM-10084
> Project: Beam
> Issue Type: Sub-task
> Components: io-java-xml
>Reporter: Pawel Pasterz
>Priority: P2
>
> Gradle task :sdks:java:io:xml:test fails during Java 11 Precommit job
> Example stack trace:
> {code:java}
> > Task :sdks:java:io:xml:test
> org.apache.beam.sdk.io.xml.XmlIOTest > testWriteThenReadLarger STANDARD_ERROR
> May 26, 2020 7:29:37 AM
> org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement
> INFO: Opening writer 4c054a13-7ff7-4b54-9398-f6bd6e531aff for window
> org.apache.beam.sdk.transforms.windowing.GlobalWindow@5967d09d pane
> PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0}
> destination null
> May 26, 2020 7:29:37 AM org.apache.beam.sdk.io.FileBasedSink$Writer open
> SEVERE: Beginning write to failed, closing channel.
> java.io.IOException: javax.xml.bind.JAXBException
> - with linked exception:
> [java.lang.ClassNotFoundException:
> com.sun.xml.internal.bind.v2.ContextFactory]
> at org.apache.beam.sdk.io.xml.XmlIO$Sink.open(XmlIO.java:664)
> at
> org.apache.beam.sdk.io.FileIO$Write$ViaFileBasedSink$1$1.prepareWrite(FileIO.java:1390)
> at
> org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:952)
> at
> org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn.processElement(WriteFiles.java:769)
> at
> org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn$DoFnInvoker.invokeProcessElement(Unknown
> Source)
> at
> org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:227)
> at
> org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:183)
> at
> org.apache.beam.repackaged.direct_java.runners.core.SimplePushbackSideInputDoFnRunner.processElementInReadyWindows(SimplePushbackSideInputDoFnRunner.java:78)
> at
> org.apache.beam.runners.direct.ParDoEvaluator.processElement(ParDoEvaluator.java:240)
> at
> org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.processElement(DoFnLifecycleManagerRemovingTransformEvaluator.java:54)
> at
> org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:160)
> at
> org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:124)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: javax.xml.bind.JAXBException
> - with linked exception:
> [java.lang.ClassNotFoundException:
> com.sun.xml.internal.bind.v2.ContextFactory]
> at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:241)
> at javax.xml.bind.ContextFinder.find(ContextFinder.java:477)
> at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:656)
> at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:599)
> at org.apache.beam.sdk.io.xml.XmlIO$Sink.open(XmlIO.java:659)
> ... 16 more
> Caused by: java.lang.ClassNotFoundException:
> com.sun.xml.internal.bind.v2.ContextFactory
> at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
> at javax.xml.bind.ContextFinder.safeLoadClass(ContextFinder.java:594)
> at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:239)
> ... 20 more
> org.apache.beam.sdk.io.xml.XmlIOTest > testWriteThenReadLarger FAILED
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.io.IOException: javax.xml.bind.JAXBException
> - with linked exception:
> [java.lang.ClassNotFoundException:
> com.sun.xml.internal.bind.v2.ContextFactory]
> at
> org.apache.beam.runners.direct.DirectRun
[jira] [Resolved] (BEAM-10187) build_release_candidate.sh does not push tag to Github
[ https://issues.apache.org/jira/browse/BEAM-10187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Hulette resolved BEAM-10187. -- Fix Version/s: Not applicable Resolution: Fixed > build_release_candidate.sh does not push tag to Github > -- > > Key: BEAM-10187 > URL: https://issues.apache.org/jira/browse/BEAM-10187 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: P2 > Fix For: Not applicable > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-10086) GCP IO tests failing [Java 11]
[
https://issues.apache.org/jira/browse/BEAM-10086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125386#comment-17125386
]
Kenneth Knowles commented on BEAM-10086:
This one looks a lot like a flake. It seems to think the BQ job failed to
start? [~chamikara] any ideas?
> GCP IO tests failing [Java 11]
> --
>
> Key: BEAM-10086
> URL: https://issues.apache.org/jira/browse/BEAM-10086
> Project: Beam
> Issue Type: Sub-task
> Components: io-java-gcp
>Reporter: Pawel Pasterz
>Priority: P2
>
> Gradle task *_:sdks:java:io:google-cloud-platform:test_* fails during Java 11
> Precommit job
> Example stack trace
> {code:java}
> > Task :sdks:java:io:google-cloud-platform:test
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOReadTest >
> testReadFromTableWithTemplateCompatibility STANDARD_ERROR
> May 26, 2020 7:19:04 AM
> org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase executeExtract
> INFO: Starting BigQuery extract job:
> beam_job_f96ecf5ccd324021938a3c3545e1279c_bigqueryioreadtest0testreadfromtablewithtemplatecompatibilityjenkins052607190433b8fa8c-extract
> May 26, 2020 7:19:04 AM
> org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase executeExtract
> INFO: BigQuery extract job completed:
> beam_job_f96ecf5ccd324021938a3c3545e1279c_bigqueryioreadtest0testreadfromtablewithtemplatecompatibilityjenkins052607190433b8fa8c-extract
> May 26, 2020 7:19:04 AM
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$TypedRead$2 processElement
> INFO: Extract job produced 1 files
> May 26, 2020 7:19:04 AM org.apache.beam.sdk.io.FileBasedSource
> createReader
> INFO: Matched 1 files for pattern
> /tmp/junit818678511668450193/BigQueryExtractTemp/f96ecf5ccd324021938a3c3545e1279c/.avro
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpersTest >
> testPendingJobManager STANDARD_ERROR
> May 26, 2020 7:19:05 AM
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
> waitForDone
> INFO: Waiting for jobs to complete.
> May 26, 2020 7:19:05 AM
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
> waitForDone
> INFO: Job JOB_0-0 failed. retrying.
> May 26, 2020 7:19:05 AM
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
> waitForDone
> INFO: Job JOB_1-0 failed. retrying.
> May 26, 2020 7:19:05 AM
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
> waitForDone
> INFO: Job JOB_2-0 failed. retrying.
> May 26, 2020 7:19:05 AM
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
> waitForDone
> INFO: Job JOB_3-0 failed. retrying.
> May 26, 2020 7:19:05 AM
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
> waitForDone
> INFO: Job JOB_4-0 failed. retrying.
> May 26, 2020 7:19:05 AM
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob runJob
> INFO: job JOB_0-0 started
> May 26, 2020 7:19:05 AM
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob runJob
> WARNING: Job JOB_1-0 failed with {}
> java.lang.RuntimeException: Failing to start.
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpersTest.lambda$testPendingJobManager$c871dfe6$1(BigQueryHelpersTest.java:170)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob.runJob(BigQueryHelpers.java:209)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager.waitForDone(BigQueryHelpers.java:152)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpersTest.testPendingJobManager(BigQueryHelpersTest.java:204)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:266)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
> at
> org.
[jira] [Work logged] (BEAM-10187) build_release_candidate.sh does not push tag to Github
[ https://issues.apache.org/jira/browse/BEAM-10187?focusedWorklogId=441021&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441021 ] ASF GitHub Bot logged work on BEAM-10187: - Author: ASF GitHub Bot Created on: 03/Jun/20 23:09 Start Date: 03/Jun/20 23:09 Worklog Time Spent: 10m Work Description: ibzib merged pull request #11914: URL: https://github.com/apache/beam/pull/11914 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 441021) Time Spent: 1h (was: 50m) > build_release_candidate.sh does not push tag to Github > -- > > Key: BEAM-10187 > URL: https://issues.apache.org/jira/browse/BEAM-10187 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: P2 > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7405) Task :sdks:python:hdfsIntegrationTest is failing in Python PostCommits - docker-credential-gcloud not installed
[
https://issues.apache.org/jira/browse/BEAM-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125382#comment-17125382
]
Brian Hulette commented on BEAM-7405:
-
[~udim] what is the action on this now?
> Task :sdks:python:hdfsIntegrationTest is failing in Python PostCommits -
> docker-credential-gcloud not installed
> ---
>
> Key: BEAM-7405
> URL: https://issues.apache.org/jira/browse/BEAM-7405
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Udi Meiri
>Priority: P2
> Labels: stale-assigned
> Fix For: 2.14.0
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> This failure happened on apache-beam-jenkins-14.
> {noformat}
> 18:47:03 > Task :sdks:python:hdfsIntegrationTest
> 18:47:03 ++ dirname
> ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
> 18:47:03 + TEST_DIR=./apache_beam/io/hdfs_integration_test
> 18:47:03 + ROOT_DIR=./apache_beam/io/hdfs_integration_test/../../../../..
> 18:47:03 +
> CONTEXT_DIR=./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 + rm -r
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 rm: cannot remove
> './apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration':
> No such file or directory
> 18:47:03 + true
> 18:47:03 + mkdir -p
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/sdks
> 18:47:03 + cp ./apache_beam/io/hdfs_integration_test/docker-compose.yml
> ./apache_beam/io/hdfs_integration_test/Dockerfile
> ./apache_beam/io/hdfs_integration_test/hdfscli.cfg
> ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/
> 18:47:03 + cp -r
> ./apache_beam/io/hdfs_integration_test/../../../../../sdks/python
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/sdks/
> 18:47:03 + cp -r ./apache_beam/io/hdfs_integration_test/../../../../../model
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/
> 18:47:03 ++ echo hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714
> 18:47:03 + PROJECT_NAME=hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714
> 18:47:03 + '[' -z jenkins-beam_PostCommit_Python_Verify_PR-714 ']'
> 18:47:03 + COLOR_OPT=--no-ansi
> 18:47:03 + COMPOSE_OPT='-p
> hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714 --no-ansi'
> 18:47:03 + cd
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 + docker network prune --force
> 18:47:03 + trap finally EXIT
> 18:47:03 + docker-compose -p
> hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714 --no-ansi build
> 18:47:03 namenode uses an image, skipping
> 18:47:03 datanode uses an image, skipping
> 18:47:03 Building test
> 18:47:03 [29234] Failed to execute script docker-compose
> 18:47:03 Traceback (most recent call last):
> 18:47:03 File "bin/docker-compose", line 6, in
> 18:47:03 File "compose/cli/main.py", line 71, in main
> 18:47:03 File "compose/cli/main.py", line 127, in perform_command
> 18:47:03 File "compose/cli/main.py", line 287, in build
> 18:47:03 File "compose/project.py", line 386, in build
> 18:47:03 File "compose/project.py", line 368, in build_service
> 18:47:03 File "compose/service.py", line 1084, in build
> 18:47:03 File "site-packages/docker/api/build.py", line 260, in build
> 18:47:03 File "site-packages/docker/api/build.py", line 307, in
> _set_auth_headers
> 18:47:03 File "site-packages/docker/auth.py", line 310, in
> get_all_credentials
> 18:47:03 File "site-packages/docker/auth.py", line 262, in
> _resolve_authconfig_credstore
> 18:47:03 File "site-packages/docker/auth.py", line 287, in
> _get_store_instance
> 18:47:03 File "site-packages/dockerpycreds/store.py", line 25, in __init__
> 18:47:03 dockerpycreds.errors.InitializationError: docker-credential-gcloud
> not installed or not available in PATH
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-5173) org.apache.beam.runners.fnexecution.control.RemoteExecutionTest.testExecutionWithMultipleStages is flaky
[ https://issues.apache.org/jira/browse/BEAM-5173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Hulette resolved BEAM-5173. - Fix Version/s: Not applicable Resolution: Cannot Reproduce > org.apache.beam.runners.fnexecution.control.RemoteExecutionTest.testExecutionWithMultipleStages > is flaky > > > Key: BEAM-5173 > URL: https://issues.apache.org/jira/browse/BEAM-5173 > Project: Beam > Issue Type: Bug > Components: runner-core >Reporter: Valentyn Tymofieiev >Assignee: Brian Hulette >Priority: P2 > Labels: stale-assigned > Fix For: Not applicable > > > Hi [~lcwik], this test failed in a [recent postcommit > build|https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1285/testReport/junit/org.apache.beam.runners.fnexecution.control/RemoteExecutionTest/testExecutionWithMultipleStages]. > Could you please take a look or help triage to the right owner? Thank you. > Stack trace: > ava.util.concurrent.ExecutionException: java.lang.RuntimeException: > java.util.concurrent.ExecutionException: > org.apache.beam.vendor.grpc.v1.io.grpc.StatusRuntimeException: CANCELLED: > Runner closed connection > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.beam.runners.fnexecution.control.RemoteExecutionTest.tearDown(RemoteExecutionTest.java:198) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) > at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38) > at > org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66) > at > org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) > at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) > at > org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32) > at > org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93) > at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) > at > org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at
[jira] [Commented] (BEAM-5173) org.apache.beam.runners.fnexecution.control.RemoteExecutionTest.testExecutionWithMultipleStages is flaky
[ https://issues.apache.org/jira/browse/BEAM-5173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125377#comment-17125377 ] Brian Hulette commented on BEAM-5173: - This still hasn't flaked on PreCommit since the one I linked in Nov 2018: https://builds.apache.org/job/beam_PreCommit_Java_Cron/2831/testReport/junit/org.apache.beam.runners.fnexecution.control/RemoteExecutionTest/testExecutionWithMultipleStages/history/ I'm closing this > org.apache.beam.runners.fnexecution.control.RemoteExecutionTest.testExecutionWithMultipleStages > is flaky > > > Key: BEAM-5173 > URL: https://issues.apache.org/jira/browse/BEAM-5173 > Project: Beam > Issue Type: Bug > Components: runner-core >Reporter: Valentyn Tymofieiev >Assignee: Brian Hulette >Priority: P2 > Labels: stale-assigned > > Hi [~lcwik], this test failed in a [recent postcommit > build|https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1285/testReport/junit/org.apache.beam.runners.fnexecution.control/RemoteExecutionTest/testExecutionWithMultipleStages]. > Could you please take a look or help triage to the right owner? Thank you. > Stack trace: > ava.util.concurrent.ExecutionException: java.lang.RuntimeException: > java.util.concurrent.ExecutionException: > org.apache.beam.vendor.grpc.v1.io.grpc.StatusRuntimeException: CANCELLED: > Runner closed connection > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.beam.runners.fnexecution.control.RemoteExecutionTest.tearDown(RemoteExecutionTest.java:198) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) > at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38) > at > org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66) > at > org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) > at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) > at > org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32) > at > org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93) > at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) > at > org.gradle.api.internal.tasks.testing.worker.Te
[jira] [Commented] (BEAM-9208) Add support for mapping columns to pubsub message attributes in flat schemas DDL
[
https://issues.apache.org/jira/browse/BEAM-9208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125374#comment-17125374
]
Brian Hulette commented on BEAM-9208:
-
I think this should remain P2
> Add support for mapping columns to pubsub message attributes in flat schemas
> DDL
>
>
> Key: BEAM-9208
> URL: https://issues.apache.org/jira/browse/BEAM-9208
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Brian Hulette
>Priority: P2
>
> Context:
> https://lists.apache.org/thread.html/bf4c37f21bda194d7f8c40f6e7b9a776262415755cc1658412af3c76%40%3Cdev.beam.apache.org%3E
> The syntax should look something like this (proposed by [~alexvanboxel]):
> {code}
> CREATE TABLE people (
> my_timestamp TIMESTAMP *OPTION(ref="pubsub:event_timestamp)*,
> my_id VARCHAR *OPTION(ref="pubsub:attributes['id_name']")*,
> name VARCHAR,
> age INTEGER
> )
> TYPE 'pubsub'
> LOCATION 'projects/my-project/topics/my-topic'
> {code}
> This jira pertains specifically to the my_id field in this example.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9208) Add support for mapping columns to pubsub message attributes in flat schemas DDL
[
https://issues.apache.org/jira/browse/BEAM-9208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brian Hulette updated BEAM-9208:
Labels: (was: stale-P2)
> Add support for mapping columns to pubsub message attributes in flat schemas
> DDL
>
>
> Key: BEAM-9208
> URL: https://issues.apache.org/jira/browse/BEAM-9208
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Brian Hulette
>Priority: P2
>
> Context:
> https://lists.apache.org/thread.html/bf4c37f21bda194d7f8c40f6e7b9a776262415755cc1658412af3c76%40%3Cdev.beam.apache.org%3E
> The syntax should look something like this (proposed by [~alexvanboxel]):
> {code}
> CREATE TABLE people (
> my_timestamp TIMESTAMP *OPTION(ref="pubsub:event_timestamp)*,
> my_id VARCHAR *OPTION(ref="pubsub:attributes['id_name']")*,
> name VARCHAR,
> age INTEGER
> )
> TYPE 'pubsub'
> LOCATION 'projects/my-project/topics/my-topic'
> {code}
> This jira pertains specifically to the my_id field in this example.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-8330) PubSubIO.readAvros should produce a schema'd PCollection if clazz has a schema
[
https://issues.apache.org/jira/browse/BEAM-8330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brian Hulette updated BEAM-8330:
Labels: (was: stale-P2)
> PubSubIO.readAvros should produce a schema'd PCollection if clazz has a schema
> --
>
> Key: BEAM-8330
> URL: https://issues.apache.org/jira/browse/BEAM-8330
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
>Reporter: Brian Hulette
>Priority: P3
>
> Currently {{PubsubIO.readAvros(clazz)}} *always* yields a PCollection with an
> AvroCoder. This should only be a fallback in the event that no coder can be
> inferred. That way if we can infer a schema for `clazz` we will produce a
> PCollection with a schema.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-8330) PubSubIO.readAvros should produce a schema'd PCollection if clazz has a schema
[
https://issues.apache.org/jira/browse/BEAM-8330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brian Hulette updated BEAM-8330:
Priority: P3 (was: P2)
> PubSubIO.readAvros should produce a schema'd PCollection if clazz has a schema
> --
>
> Key: BEAM-8330
> URL: https://issues.apache.org/jira/browse/BEAM-8330
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
>Reporter: Brian Hulette
>Priority: P3
> Labels: stale-P2
>
> Currently {{PubsubIO.readAvros(clazz)}} *always* yields a PCollection with an
> AvroCoder. This should only be a fallback in the event that no coder can be
> inferred. That way if we can infer a schema for `clazz` we will produce a
> PCollection with a schema.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-10190) Reduce cost of toString of MetricKey and MetricName
[ https://issues.apache.org/jira/browse/BEAM-10190?focusedWorklogId=441012&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441012 ] ASF GitHub Bot logged work on BEAM-10190: - Author: ASF GitHub Bot Created on: 03/Jun/20 22:36 Start Date: 03/Jun/20 22:36 Worklog Time Spent: 10m Work Description: Zhangyx39 commented on pull request #11915: URL: https://github.com/apache/beam/pull/11915#issuecomment-638497048 @xinyuiscool would you like to take a look at this pr? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 441012) Time Spent: 20m (was: 10m) > Reduce cost of toString of MetricKey and MetricName > --- > > Key: BEAM-10190 > URL: https://issues.apache.org/jira/browse/BEAM-10190 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Yixing Zhang >Assignee: Yixing Zhang >Priority: P2 > Time Spent: 20m > Remaining Estimate: 0h > > Samza runner heavily uses MetricKey.toString() and MetricName.toString() to > update Samza metrics. We found that the toString methods have high CPU cost. > And according to this article: > [https://redfin.engineering/java-string-concatenation-which-way-is-best-8f590a7d22a8], > we should use "+" operator instead of String.format for string concatenation > for better performance. > We do see a 10% QPS gain in nexmark queries using Samza runner with the > change of using "+" operator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10190) Reduce cost of toString of MetricKey and MetricName
[ https://issues.apache.org/jira/browse/BEAM-10190?focusedWorklogId=441011&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441011 ] ASF GitHub Bot logged work on BEAM-10190: - Author: ASF GitHub Bot Created on: 03/Jun/20 22:35 Start Date: 03/Jun/20 22:35 Worklog Time Spent: 10m Work Description: Zhangyx39 opened a new pull request #11915: URL: https://github.com/apache/beam/pull/11915 Samza runner heavily uses MetricKey.toString() and MetricName.toString() to update Samza metrics. We found that the toString methods have high CPU cost. And according to this article: https://redfin.engineering/java-string-concatenation-which-way-is-best-8f590a7d22a8, we should use "+" operator instead of String.format for string concatenation for better performance. We do see about 10% QPS gain in nexmark queries using Samza runner with the change of using "+" operator. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [ Reduce cost of toString of MetricKey and MetricName
[ https://issues.apache.org/jira/browse/BEAM-10190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-10190 started by Yixing Zhang. --- > Reduce cost of toString of MetricKey and MetricName > --- > > Key: BEAM-10190 > URL: https://issues.apache.org/jira/browse/BEAM-10190 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Yixing Zhang >Assignee: Yixing Zhang >Priority: P2 > > Samza runner heavily uses MetricKey.toString() and MetricName.toString() to > update Samza metrics. We found that the toString methods have high CPU cost. > And according to this article: > [https://redfin.engineering/java-string-concatenation-which-way-is-best-8f590a7d22a8], > we should use "+" operator instead of String.format for string concatenation > for better performance. > We do see a 10% QPS gain in nexmark queries using Samza runner with the > change of using "+" operator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10190) Reduce cost of toString of MetricKey and MetricName
[ https://issues.apache.org/jira/browse/BEAM-10190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yixing Zhang updated BEAM-10190: Description: Samza runner heavily uses MetricKey.toString() and MetricName.toString() to update Samza metrics. We found that the toString methods have high CPU cost. And according to this article: [https://redfin.engineering/java-string-concatenation-which-way-is-best-8f590a7d22a8], we should use "+" operator instead of String.format for string concatenation for better performance. We do see a 10% QPS gain in nexmark queries using Samza runner with the change of using "+" operator. > Reduce cost of toString of MetricKey and MetricName > --- > > Key: BEAM-10190 > URL: https://issues.apache.org/jira/browse/BEAM-10190 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Yixing Zhang >Assignee: Yixing Zhang >Priority: P2 > > Samza runner heavily uses MetricKey.toString() and MetricName.toString() to > update Samza metrics. We found that the toString methods have high CPU cost. > And according to this article: > [https://redfin.engineering/java-string-concatenation-which-way-is-best-8f590a7d22a8], > we should use "+" operator instead of String.format for string concatenation > for better performance. > We do see a 10% QPS gain in nexmark queries using Samza runner with the > change of using "+" operator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9977) Build Kafka Read on top of Java SplittableDoFn
[
https://issues.apache.org/jira/browse/BEAM-9977?focusedWorklogId=441005&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441005
]
ASF GitHub Bot logged work on BEAM-9977:
Author: ASF GitHub Bot
Created on: 03/Jun/20 22:23
Start Date: 03/Jun/20 22:23
Worklog Time Spent: 10m
Work Description: boyuanzz commented on a change in pull request #11749:
URL: https://github.com/apache/beam/pull/11749#discussion_r434890835
##
File path:
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
##
@@ -817,6 +847,24 @@ public void setValueDeserializer(String valueDeserializer)
{
return toBuilder().setCommitOffsetsInFinalizeEnabled(true).build();
}
+/**
+ * The {@link Read} transform will be expanded with {@link
ReadFromKafkaViaSDF} transform. While
+ * working with {@link #useSDFTransformInRead()} and {@link
+ * #withTimestampPolicyFactory(TimestampPolicyFactory)} together, only
{@link
+ * TimestampPolicyFactory#withCreateTime(Duration)}, {@link
+ * TimestampPolicyFactory#withLogAppendTime()} and {@link
+ * TimestampPolicyFactory#withProcessingTime()} will be populated
correctly. For other custom
+ * {@link TimestampPolicy}, the transform will use {@link
+ * TimestampPolicyFactory#withProcessingTime()} by default. It's
recommended to use {@link
+ * ReadFromKafkaViaSDF} directly in that case.
+ *
+ * Note that the expansion only happens when tbe pipeline has
"beam_fn_api" experiment and
+ * meanwhile "beam_fn_api_use_deprecated_read" is not set.
+ */
+public Read useSDFTransformInRead() {
Review comment:
Discussed with Luke offline. We think it would be better to make
`KafkaIO.Read()` expand with SDF transform bu default when `beam_fn_api` is
enabled( before introducing this SDF transform, we expand the `Read` with
SDFUnboundedWrapper with beam_fn_api).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 441005)
Time Spent: 8h 50m (was: 8h 40m)
> Build Kafka Read on top of Java SplittableDoFn
> --
>
> Key: BEAM-9977
> URL: https://issues.apache.org/jira/browse/BEAM-9977
> Project: Beam
> Issue Type: New Feature
> Components: io-java-kafka
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: P2
> Time Spent: 8h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-10190) Reduce cost of toString of MetricKey and MetricName
Yixing Zhang created BEAM-10190: --- Summary: Reduce cost of toString of MetricKey and MetricName Key: BEAM-10190 URL: https://issues.apache.org/jira/browse/BEAM-10190 Project: Beam Issue Type: Improvement Components: sdk-java-core Reporter: Yixing Zhang Assignee: Yixing Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10190) Reduce cost of toString of MetricKey and MetricName
[ https://issues.apache.org/jira/browse/BEAM-10190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-10190: --- Status: Open (was: Triage Needed) > Reduce cost of toString of MetricKey and MetricName > --- > > Key: BEAM-10190 > URL: https://issues.apache.org/jira/browse/BEAM-10190 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Yixing Zhang >Assignee: Yixing Zhang >Priority: P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9679) Core Transforms | Go SDK Code Katas
[ https://issues.apache.org/jira/browse/BEAM-9679?focusedWorklogId=441003&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441003 ] ASF GitHub Bot logged work on BEAM-9679: Author: ASF GitHub Bot Created on: 03/Jun/20 22:12 Start Date: 03/Jun/20 22:12 Worklog Time Spent: 10m Work Description: damondouglas commented on pull request #11883: URL: https://github.com/apache/beam/pull/11883#issuecomment-638488229 @lostluck I was waiting on @henryken approval. Should I just go ahead and update stepik and commit the `*-remote.yaml` to this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 441003) Time Spent: 7h 10m (was: 7h) > Core Transforms | Go SDK Code Katas > --- > > Key: BEAM-9679 > URL: https://issues.apache.org/jira/browse/BEAM-9679 > Project: Beam > Issue Type: Sub-task > Components: katas, sdk-go >Reporter: Damon Douglas >Assignee: Damon Douglas >Priority: P2 > Time Spent: 7h 10m > Remaining Estimate: 0h > > A kata devoted to core beam transforms patterns after > [https://github.com/apache/beam/tree/master/learning/katas/java/Core%20Transforms] > where the take away is an individual's ability to master the following using > an Apache Beam pipeline using the Golang SDK. > > ||Transform||Pull Request||Status|| > |Map|[11564|https://github.com/apache/beam/pull/11564]|Closed| > |GroupByKey|[11734|https://github.com/apache/beam/pull/11734]|Closed| > |CoGroupByKey|[11803|https://github.com/apache/beam/pull/11803]|Closed| > |Combine Simple > Function|[11866|https://github.com/apache/beam/pull/11866]|Closed| > |CombineFn|[11883|https://github.com/apache/beam/pull/11883]|Open| > |Flatten|[11806|https://github.com/apache/beam/pull/11806]|Closed| > |Partition| | | > |Side Input| | | > |Side Output| | | > |Branching| | | > |Composite Transform| | | > |DoFn Additional Parameters| | | -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9008) Add readAll() method to CassandraIO
[
https://issues.apache.org/jira/browse/BEAM-9008?focusedWorklogId=440999&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440999
]
ASF GitHub Bot logged work on BEAM-9008:
Author: ASF GitHub Bot
Created on: 03/Jun/20 21:33
Start Date: 03/Jun/20 21:33
Worklog Time Spent: 10m
Work Description: vmarquez commented on a change in pull request #10546:
URL: https://github.com/apache/beam/pull/10546#discussion_r434870508
##
File path:
sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/CassandraIO.java
##
@@ -1170,4 +898,44 @@ private void waitForFuturesToFinish() throws
ExecutionException, InterruptedExce
}
}
}
+
+ /**
+ * A {@link PTransform} to read data from Apache Cassandra. See {@link
CassandraIO} for more
+ * information on usage and configuration.
+ */
+ @AutoValue
+ public abstract static class ReadAll extends
PTransform>, PCollection> {
+
+@Nullable
+abstract Coder coder();
+
+abstract ReadAll.Builder builder();
+
+/** Specify the {@link Coder} used to serialize the entity in the {@link
PCollection}. */
+public ReadAll withCoder(Coder coder) {
Review comment:
I think we do, but am not sure. You have to call `setCoder` on the
PCollection itself, so we don't have access to the `Read` at a point when we
also have access to a single `Read` (they are only supplied in the `ReadFn`
which can't call `setCoder` on the returned PTransform). Is my thinking
correct? I could be unaware of a different way to do this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440999)
Time Spent: 13h 20m (was: 13h 10m)
> Add readAll() method to CassandraIO
> ---
>
> Key: BEAM-9008
> URL: https://issues.apache.org/jira/browse/BEAM-9008
> Project: Beam
> Issue Type: New Feature
> Components: io-java-cassandra
>Affects Versions: 2.16.0
>Reporter: vincent marquez
>Assignee: vincent marquez
>Priority: P3
> Time Spent: 13h 20m
> Remaining Estimate: 0h
>
> When querying a large cassandra database, it's often *much* more useful to
> programatically generate the queries needed to to be run rather than reading
> all partitions and attempting some filtering.
> As an example:
> {code:java}
> public class Event {
>@PartitionKey(0) public UUID accountId;
>@PartitionKey(1)public String yearMonthDay;
>@ClusteringKey public UUID eventId;
>//other data...
> }{code}
> If there is ten years worth of data, you may want to only query one year's
> worth. Here each token range would represent one 'token' but all events for
> the day.
> {code:java}
> Set accounts = getRelevantAccounts();
> Set dateRange = generateDateRange("2018-01-01", "2019-01-01");
> PCollection tokens = generateTokens(accounts, dateRange);
> {code}
>
> I propose an additional _readAll()_ PTransform that can take a PCollection
> of token ranges and can return a PCollection of what the query would
> return.
> *Question: How much code should be in common between both methods?*
> Currently the read connector already groups all partitions into a List of
> Token Ranges, so it would be simple to refactor the current read() based
> method to a 'ParDo' based one and have them both share the same function.
> Reasons against sharing code between read and readAll
> * Not having the read based method return a BoundedSource connector would
> mean losing the ability to know the size of the data returned
> * Currently the CassandraReader executes all the grouped TokenRange queries
> *asynchronously* which is (maybe?) fine when all that's happening is
> splitting up all the partition ranges but terrible for executing potentially
> millions of queries.
> Reasons _for_ sharing code would be simplified code base and that both of
> the above issues would most likely have a negligable performance impact.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-10024) Spark runner failing testOutputTimestampDefault
[
https://issues.apache.org/jira/browse/BEAM-10024?focusedWorklogId=440998&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440998
]
ASF GitHub Bot logged work on BEAM-10024:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 21:32
Start Date: 03/Jun/20 21:32
Worklog Time Spent: 10m
Work Description: ibzib commented on a change in pull request #11739:
URL: https://github.com/apache/beam/pull/11739#discussion_r434869939
##
File path:
sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
##
@@ -3539,6 +3539,7 @@ public void onTimer() {}
@Category({
ValidatesRunner.class,
UsesTimersInParDo.class,
+ UsesUnboundedPCollections.class,
Review comment:
ping @lukecwik @reuvenlax
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440998)
Time Spent: 1.5h (was: 1h 20m)
> Spark runner failing testOutputTimestampDefault
> ---
>
> Key: BEAM-10024
> URL: https://issues.apache.org/jira/browse/BEAM-10024
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: P2
> Labels: currently-failing
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> This is causing postcommit to fail
> java.lang.UnsupportedOperationException: Found TimerId annotations on
> org.apache.beam.sdk.transforms.ParDoTest$TimerTests$12, but DoFn cannot yet
> be used with timers in the SparkRunner.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9679) Core Transforms | Go SDK Code Katas
[ https://issues.apache.org/jira/browse/BEAM-9679?focusedWorklogId=440995&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440995 ] ASF GitHub Bot logged work on BEAM-9679: Author: ASF GitHub Bot Created on: 03/Jun/20 21:28 Start Date: 03/Jun/20 21:28 Worklog Time Spent: 10m Work Description: lostluck commented on pull request #11883: URL: https://github.com/apache/beam/pull/11883#issuecomment-638471588 @damondouglas Has stepik been updated? Is this clear to merge? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440995) Time Spent: 7h (was: 6h 50m) > Core Transforms | Go SDK Code Katas > --- > > Key: BEAM-9679 > URL: https://issues.apache.org/jira/browse/BEAM-9679 > Project: Beam > Issue Type: Sub-task > Components: katas, sdk-go >Reporter: Damon Douglas >Assignee: Damon Douglas >Priority: P2 > Time Spent: 7h > Remaining Estimate: 0h > > A kata devoted to core beam transforms patterns after > [https://github.com/apache/beam/tree/master/learning/katas/java/Core%20Transforms] > where the take away is an individual's ability to master the following using > an Apache Beam pipeline using the Golang SDK. > > ||Transform||Pull Request||Status|| > |Map|[11564|https://github.com/apache/beam/pull/11564]|Closed| > |GroupByKey|[11734|https://github.com/apache/beam/pull/11734]|Closed| > |CoGroupByKey|[11803|https://github.com/apache/beam/pull/11803]|Closed| > |Combine Simple > Function|[11866|https://github.com/apache/beam/pull/11866]|Closed| > |CombineFn|[11883|https://github.com/apache/beam/pull/11883]|Open| > |Flatten|[11806|https://github.com/apache/beam/pull/11806]|Closed| > |Partition| | | > |Side Input| | | > |Side Output| | | > |Branching| | | > |Composite Transform| | | > |DoFn Additional Parameters| | | -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10166) Improve execution time errors
[
https://issues.apache.org/jira/browse/BEAM-10166?focusedWorklogId=440994&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440994
]
ASF GitHub Bot logged work on BEAM-10166:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 21:27
Start Date: 03/Jun/20 21:27
Worklog Time Spent: 10m
Work Description: lostluck merged pull request #11881:
URL: https://github.com/apache/beam/pull/11881
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440994)
Time Spent: 40m (was: 0.5h)
> Improve execution time errors
> -
>
> Key: BEAM-10166
> URL: https://issues.apache.org/jira/browse/BEAM-10166
> Project: Beam
> Issue Type: Improvement
> Components: sdk-go
>Reporter: Robert Burke
>Priority: P3
> Labels: beginner, n00b, starter
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The Go SDK uses errors returned by DoFns to signal failures to process
> bundles, and terminate bundle processing. However, if the preceding DoFn uses
> emitters, rather than error returns, the code has no choice to panic to avoid
> user code handling or ignoring the cross DoFn error (which could cause
> dataloss or other correctness problems).
> All bundle executions are wrapped in `callNoPanic` to prevent worker
> termination on such panics, and orderly terminate just the affected bundle
> instead.`callNoPanic` uses Go's built in recover mechanism to get the error
> and provide a stack trace.
> We can do better.
> The value returned by recover is just an interface{} which means we could
> detect the specific type of error it is. In particular, we could have the
> exec package have an error that we can detect. If the recovered value is that
> error, then we could use that to provide a clearer error message than a
> panic stack trace.
> Such an error wrapper would contain: the error in question, the user DoFn
> that caused it, the debug id of the DoFn node (To be related back to the
> plan.)
> Then in `callNoPanic` we could detect this error wrapper and produce a
> clearer error message based on the existing plan. If not, we can maintain the
> current behavior. This latter part is necessary to handle panics originating
> in user code.
> To avoid mistaken user use which would breach this protocol, we're best off
> keeping the wrapper unexported from the exec package.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-10189) Add ValueState to python sdk
[ https://issues.apache.org/jira/browse/BEAM-10189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-10189: --- Status: Open (was: Triage Needed) > Add ValueState to python sdk > > > Key: BEAM-10189 > URL: https://issues.apache.org/jira/browse/BEAM-10189 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10189) Add ValueState to python sdk
Yichi Zhang created BEAM-10189: -- Summary: Add ValueState to python sdk Key: BEAM-10189 URL: https://issues.apache.org/jira/browse/BEAM-10189 Project: Beam Issue Type: Improvement Components: sdk-py-core Reporter: Yichi Zhang Assignee: Yichi Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10188) Automate Github release
Kyle Weaver created BEAM-10188: -- Summary: Automate Github release Key: BEAM-10188 URL: https://issues.apache.org/jira/browse/BEAM-10188 Project: Beam Issue Type: Improvement Components: build-system Reporter: Kyle Weaver Currently, we push the tag to Github and fill in the release notes in separate steps. For feeds consuming these updates, it would be better to do both in the same step using the Github API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10187) build_release_candidate.sh does not push tag to Github
[
https://issues.apache.org/jira/browse/BEAM-10187?focusedWorklogId=440985&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440985
]
ASF GitHub Bot logged work on BEAM-10187:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 21:00
Start Date: 03/Jun/20 21:00
Worklog Time Spent: 10m
Work Description: ibzib commented on a change in pull request #11914:
URL: https://github.com/apache/beam/pull/11914#discussion_r434850943
##
File path: release/src/main/scripts/build_release_candidate.sh
##
@@ -106,9 +106,7 @@ if [[ $confirmation = "y" ]]; then
-Prelease.releaseVersion=${RELEASE}-RC${RC_NUM} \
-Prelease.useAutomaticVersion=true --info --no-daemon
- echo "Please make sure gradle release succeed: "
- echo "1. release code has been pushed to github repo."
- echo "2. new rc tag has created in github."
+ git push "v${RELEASE}-RC${RC_NUM}"
Review comment:
Ah you're right about both. Fixed, PTAL
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440985)
Time Spent: 50m (was: 40m)
> build_release_candidate.sh does not push tag to Github
> --
>
> Key: BEAM-10187
> URL: https://issues.apache.org/jira/browse/BEAM-10187
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: P2
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-10178) Error messages for unspecified options should display the command line flag that needs to be specified
[
https://issues.apache.org/jira/browse/BEAM-10178?focusedWorklogId=440984&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440984
]
ASF GitHub Bot logged work on BEAM-10178:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 21:00
Start Date: 03/Jun/20 21:00
Worklog Time Spent: 10m
Work Description: KevinGG commented on pull request #11898:
URL: https://github.com/apache/beam/pull/11898#issuecomment-638458752
> Run Java PreCommit
Thanks Pablo!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440984)
Time Spent: 2h 20m (was: 2h 10m)
> Error messages for unspecified options should display the command line flag
> that needs to be specified
> --
>
> Key: BEAM-10178
> URL: https://issues.apache.org/jira/browse/BEAM-10178
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core, sdk-py-core
>Reporter: Ning Kang
>Assignee: Ning Kang
>Priority: P3
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> An example error trace:
> {code:java}
> java.lang.IllegalArgumentException: BigQueryIO.Read needs a GCS temp location
> to store temp files.
> at
> org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$TypedRead.validate(BigQueryIO.java:762)
> at
> org.apache.beam.sdk.Pipeline$ValidateVisitor.enterCompositeTransform(Pipeline.java:641)
> at
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:653)
> at
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
> at
> org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:317)
> at
> org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:251)
> at
> org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458)
> at org.apache.beam.sdk.Pipeline.validate(Pipeline.java:577)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:312)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
> at
> org.apache.beam.examples.cookbook.BigQueryTornadoes.runBigQueryTornadoes(BigQueryTornadoes.java:199)
> at
> org.apache.beam.examples.cookbook.BigQueryTornadoesIT.runE2EBigQueryTornadoesTest(BigQueryTornadoesIT.java:70)
> at
> org.apache.beam.examples.cookbook.BigQueryTornadoesIT.testE2EBigQueryTornadoesWithExport(BigQueryTornadoesIT.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:349)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:314)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:312)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:396)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
> at
> org.gradle.api.internal.tasks.testing.junit
[jira] [Work logged] (BEAM-10175) FhirIO execute bundle uses deprecated auth uri param
[ https://issues.apache.org/jira/browse/BEAM-10175?focusedWorklogId=440983&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440983 ] ASF GitHub Bot logged work on BEAM-10175: - Author: ASF GitHub Bot Created on: 03/Jun/20 20:59 Start Date: 03/Jun/20 20:59 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #11893: URL: https://github.com/apache/beam/pull/11893#issuecomment-638458169 I'm ready to merge. I jsut want to confirm whether the Fhir import test is flaky, or if you've fixed it in your last commit? @jaketf This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440983) Time Spent: 1h 40m (was: 1.5h) > FhirIO execute bundle uses deprecated auth uri param > > > Key: BEAM-10175 > URL: https://issues.apache.org/jira/browse/BEAM-10175 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Affects Versions: 2.22.0 >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: P2 > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10178) Error messages for unspecified options should display the command line flag that needs to be specified
[
https://issues.apache.org/jira/browse/BEAM-10178?focusedWorklogId=440982&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440982
]
ASF GitHub Bot logged work on BEAM-10178:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 20:58
Start Date: 03/Jun/20 20:58
Worklog Time Spent: 10m
Work Description: pabloem commented on pull request #11898:
URL: https://github.com/apache/beam/pull/11898#issuecomment-638457890
Run Java PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440982)
Time Spent: 2h 10m (was: 2h)
> Error messages for unspecified options should display the command line flag
> that needs to be specified
> --
>
> Key: BEAM-10178
> URL: https://issues.apache.org/jira/browse/BEAM-10178
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core, sdk-py-core
>Reporter: Ning Kang
>Assignee: Ning Kang
>Priority: P3
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> An example error trace:
> {code:java}
> java.lang.IllegalArgumentException: BigQueryIO.Read needs a GCS temp location
> to store temp files.
> at
> org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$TypedRead.validate(BigQueryIO.java:762)
> at
> org.apache.beam.sdk.Pipeline$ValidateVisitor.enterCompositeTransform(Pipeline.java:641)
> at
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:653)
> at
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
> at
> org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:317)
> at
> org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:251)
> at
> org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458)
> at org.apache.beam.sdk.Pipeline.validate(Pipeline.java:577)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:312)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
> at
> org.apache.beam.examples.cookbook.BigQueryTornadoes.runBigQueryTornadoes(BigQueryTornadoes.java:199)
> at
> org.apache.beam.examples.cookbook.BigQueryTornadoesIT.runE2EBigQueryTornadoesTest(BigQueryTornadoesIT.java:70)
> at
> org.apache.beam.examples.cookbook.BigQueryTornadoesIT.testE2EBigQueryTornadoesWithExport(BigQueryTornadoesIT.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:349)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:314)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:312)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:396)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.exe
[jira] [Work logged] (BEAM-8841) Add ability to perform BigQuery file loads using avro
[
https://issues.apache.org/jira/browse/BEAM-8841?focusedWorklogId=440981&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440981
]
ASF GitHub Bot logged work on BEAM-8841:
Author: ASF GitHub Bot
Created on: 03/Jun/20 20:57
Start Date: 03/Jun/20 20:57
Worklog Time Spent: 10m
Work Description: pabloem merged pull request #11896:
URL: https://github.com/apache/beam/pull/11896
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440981)
Time Spent: 9h 40m (was: 9.5h)
> Add ability to perform BigQuery file loads using avro
> -
>
> Key: BEAM-8841
> URL: https://issues.apache.org/jira/browse/BEAM-8841
> Project: Beam
> Issue Type: Improvement
> Components: io-py-gcp
>Reporter: Chun Yang
>Assignee: Chun Yang
>Priority: P3
> Fix For: 2.21.0
>
> Time Spent: 9h 40m
> Remaining Estimate: 0h
>
> Currently, JSON format is used for file loads into BigQuery in the Python
> SDK. JSON has some disadvantages including size of serialized data and
> inability to represent NaN and infinity float values.
> BigQuery supports loading files in avro format, which can overcome these
> disadvantages. The Java SDK already supports loading files using avro format
> (BEAM-2879) so it makes sense to support it in the Python SDK as well.
> The change will be somewhere around
> [{{BigQueryBatchFileLoads}}|https://github.com/apache/beam/blob/3e7865ee6c6a56e51199515ec5b4b16de1ddd166/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py#L554].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-10187) build_release_candidate.sh does not push tag to Github
[
https://issues.apache.org/jira/browse/BEAM-10187?focusedWorklogId=440978&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440978
]
ASF GitHub Bot logged work on BEAM-10187:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 20:48
Start Date: 03/Jun/20 20:48
Worklog Time Spent: 10m
Work Description: ibzib commented on a change in pull request #11914:
URL: https://github.com/apache/beam/pull/11914#discussion_r434844066
##
File path: release/src/main/scripts/build_release_candidate.sh
##
@@ -106,9 +106,7 @@ if [[ $confirmation = "y" ]]; then
-Prelease.releaseVersion=${RELEASE}-RC${RC_NUM} \
-Prelease.useAutomaticVersion=true --info --no-daemon
- echo "Please make sure gradle release succeed: "
- echo "1. release code has been pushed to github repo."
- echo "2. new rc tag has created in github."
+ git push "v${RELEASE}-RC${RC_NUM}"
Review comment:
> When I did this locally there was one more commit on the
release-2.22.0 branch changing the version to 2.22.0-SNAPSHOT, so I had to run
git push origin release-2.22.0 as well.
I see two commits, one changing it to RC1 and next changing it back to
SNAPSHOT. Isn't that just a no-op?
https://github.com/apache/beam/commits/release-2.22.0
> Also is it ok to omit the origin?
Origin is the default IIUC.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440978)
Time Spent: 0.5h (was: 20m)
> build_release_candidate.sh does not push tag to Github
> --
>
> Key: BEAM-10187
> URL: https://issues.apache.org/jira/browse/BEAM-10187
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: P2
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-10187) build_release_candidate.sh does not push tag to Github
[
https://issues.apache.org/jira/browse/BEAM-10187?focusedWorklogId=440980&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440980
]
ASF GitHub Bot logged work on BEAM-10187:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 20:55
Start Date: 03/Jun/20 20:55
Worklog Time Spent: 10m
Work Description: TheNeuralBit commented on a change in pull request
#11914:
URL: https://github.com/apache/beam/pull/11914#discussion_r434847753
##
File path: release/src/main/scripts/build_release_candidate.sh
##
@@ -106,9 +106,7 @@ if [[ $confirmation = "y" ]]; then
-Prelease.releaseVersion=${RELEASE}-RC${RC_NUM} \
-Prelease.useAutomaticVersion=true --info --no-daemon
- echo "Please make sure gradle release succeed: "
- echo "1. release code has been pushed to github repo."
- echo "2. new rc tag has created in github."
+ git push "v${RELEASE}-RC${RC_NUM}"
Review comment:
> I see two commits, one changing it to RC1 and next changing it back to
SNAPSHOT. Isn't that just a no-op?
https://github.com/apache/beam/commits/release-2.22.0
Yes but the first one is tagged with v2.22.0-RC1
https://github.com/apache/beam/commits/v2.22.0-RC1. So if you only push the tag
it won't actually be a noop.
> Origin is the default IIUC.
`git help push` says `git push ... [ [...]]` which I
take to mean if you only pass one arg it's interpreted as the `repository`.
That does seem to be what happens when I try it:
```
❯ git push release-2.22.0
fatal: 'release-2.22.0' does not appear to be a git repository
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440980)
Time Spent: 40m (was: 0.5h)
> build_release_candidate.sh does not push tag to Github
> --
>
> Key: BEAM-10187
> URL: https://issues.apache.org/jira/browse/BEAM-10187
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: P2
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9869) adding self-contained Kafka service jar for testing
[ https://issues.apache.org/jira/browse/BEAM-9869?focusedWorklogId=440977&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440977 ] ASF GitHub Bot logged work on BEAM-9869: Author: ASF GitHub Bot Created on: 03/Jun/20 20:43 Start Date: 03/Jun/20 20:43 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11846: URL: https://github.com/apache/beam/pull/11846#issuecomment-638451041 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440977) Time Spent: 2h 50m (was: 2h 40m) > adding self-contained Kafka service jar for testing > --- > > Key: BEAM-9869 > URL: https://issues.apache.org/jira/browse/BEAM-9869 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Labels: stale-assigned > Time Spent: 2h 50m > Remaining Estimate: 0h > > adding self-contained Kafka service jar for testing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10187) build_release_candidate.sh does not push tag to Github
[
https://issues.apache.org/jira/browse/BEAM-10187?focusedWorklogId=440976&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440976
]
ASF GitHub Bot logged work on BEAM-10187:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 20:41
Start Date: 03/Jun/20 20:41
Worklog Time Spent: 10m
Work Description: TheNeuralBit commented on a change in pull request
#11914:
URL: https://github.com/apache/beam/pull/11914#discussion_r434840793
##
File path: release/src/main/scripts/build_release_candidate.sh
##
@@ -106,9 +106,7 @@ if [[ $confirmation = "y" ]]; then
-Prelease.releaseVersion=${RELEASE}-RC${RC_NUM} \
-Prelease.useAutomaticVersion=true --info --no-daemon
- echo "Please make sure gradle release succeed: "
- echo "1. release code has been pushed to github repo."
- echo "2. new rc tag has created in github."
+ git push "v${RELEASE}-RC${RC_NUM}"
Review comment:
When I did this locally there was one more commit on the
`release-2.22.0` branch changing the version to 2.22.0-SNAPSHOT, so I had to
run `git push origin release-2.22.0` as well.
Also is it ok to omit the `origin`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440976)
Time Spent: 20m (was: 10m)
> build_release_candidate.sh does not push tag to Github
> --
>
> Key: BEAM-10187
> URL: https://issues.apache.org/jira/browse/BEAM-10187
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: P2
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-10187) build_release_candidate.sh does not push tag to Github
[ https://issues.apache.org/jira/browse/BEAM-10187?focusedWorklogId=440971&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440971 ] ASF GitHub Bot logged work on BEAM-10187: - Author: ASF GitHub Bot Created on: 03/Jun/20 20:30 Start Date: 03/Jun/20 20:30 Worklog Time Spent: 10m Work Description: ibzib opened a new pull request #11914: URL: https://github.com/apache/beam/pull/11914 R: @TheNeuralBit @amaliujia Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[ build_release_candidate.sh does not push tag to Github
[ https://issues.apache.org/jira/browse/BEAM-10187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-10187: --- Status: Open (was: Triage Needed) > build_release_candidate.sh does not push tag to Github > -- > > Key: BEAM-10187 > URL: https://issues.apache.org/jira/browse/BEAM-10187 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10187) build_release_candidate.sh does not push tag to Github
Kyle Weaver created BEAM-10187: -- Summary: build_release_candidate.sh does not push tag to Github Key: BEAM-10187 URL: https://issues.apache.org/jira/browse/BEAM-10187 Project: Beam Issue Type: Bug Components: build-system Reporter: Kyle Weaver Assignee: Kyle Weaver -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10165) Return error messages after job finishes for portable runner.
[ https://issues.apache.org/jira/browse/BEAM-10165?focusedWorklogId=440967&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440967 ] ASF GitHub Bot logged work on BEAM-10165: - Author: ASF GitHub Bot Created on: 03/Jun/20 20:18 Start Date: 03/Jun/20 20:18 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11912: URL: https://github.com/apache/beam/pull/11912#issuecomment-638439054 R: @angoenka This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440967) Time Spent: 20m (was: 10m) > Return error messages after job finishes for portable runner. > - > > Key: BEAM-10165 > URL: https://issues.apache.org/jira/browse/BEAM-10165 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Robert Bradshaw >Priority: P2 > Time Spent: 20m > Remaining Estimate: 0h > > The Java PortableRunner stub requests the message stream after the job has > finished on failure; we must cache the error messages and return them to get > good error reporting. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9977) Build Kafka Read on top of Java SplittableDoFn
[ https://issues.apache.org/jira/browse/BEAM-9977?focusedWorklogId=440966&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440966 ] ASF GitHub Bot logged work on BEAM-9977: Author: ASF GitHub Bot Created on: 03/Jun/20 20:18 Start Date: 03/Jun/20 20:18 Worklog Time Spent: 10m Work Description: boyuanzz commented on pull request #11749: URL: https://github.com/apache/beam/pull/11749#issuecomment-638439090 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440966) Time Spent: 8h 40m (was: 8.5h) > Build Kafka Read on top of Java SplittableDoFn > -- > > Key: BEAM-9977 > URL: https://issues.apache.org/jira/browse/BEAM-9977 > Project: Beam > Issue Type: New Feature > Components: io-java-kafka >Reporter: Boyuan Zhang >Assignee: Boyuan Zhang >Priority: P2 > Time Spent: 8h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10165) Return error messages after job finishes for portable runner.
[ https://issues.apache.org/jira/browse/BEAM-10165?focusedWorklogId=440964&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440964 ] ASF GitHub Bot logged work on BEAM-10165: - Author: ASF GitHub Bot Created on: 03/Jun/20 20:16 Start Date: 03/Jun/20 20:16 Worklog Time Spent: 10m Work Description: robertwb opened a new pull request #11912: URL: https://github.com/apache/beam/pull/11912 The Java portable runner only fetches messages and looks for errors after the pipeline returns failure. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/last
[jira] [Work logged] (BEAM-10186) Send error response to runner when split or progress requests are received out of order
[ https://issues.apache.org/jira/browse/BEAM-10186?focusedWorklogId=440949&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440949 ] ASF GitHub Bot logged work on BEAM-10186: - Author: ASF GitHub Bot Created on: 03/Jun/20 19:54 Start Date: 03/Jun/20 19:54 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11911: URL: https://github.com/apache/beam/pull/11911#issuecomment-638427096 R: @lukecwik This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440949) Time Spent: 20m (was: 10m) > Send error response to runner when split or progress requests are received > out of order > --- > > Key: BEAM-10186 > URL: https://issues.apache.org/jira/browse/BEAM-10186 > Project: Beam > Issue Type: Improvement > Components: sdk-java-harness >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P2 > Time Spent: 20m > Remaining Estimate: 0h > > We need to send an error to the runner when split requests or progress > updates are received by the SDK harness before receiving the corresponding > bundle. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10186) Send error response to runner when split or progress requests are received out of order
[ https://issues.apache.org/jira/browse/BEAM-10186?focusedWorklogId=440951&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440951 ] ASF GitHub Bot logged work on BEAM-10186: - Author: ASF GitHub Bot Created on: 03/Jun/20 19:54 Start Date: 03/Jun/20 19:54 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11911: URL: https://github.com/apache/beam/pull/11911#issuecomment-638427486 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440951) Time Spent: 0.5h (was: 20m) > Send error response to runner when split or progress requests are received > out of order > --- > > Key: BEAM-10186 > URL: https://issues.apache.org/jira/browse/BEAM-10186 > Project: Beam > Issue Type: Improvement > Components: sdk-java-harness >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P2 > Time Spent: 0.5h > Remaining Estimate: 0h > > We need to send an error to the runner when split requests or progress > updates are received by the SDK harness before receiving the corresponding > bundle. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10186) Send error response to runner when split or progress requests are received out of order
[ https://issues.apache.org/jira/browse/BEAM-10186?focusedWorklogId=440946&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440946 ] ASF GitHub Bot logged work on BEAM-10186: - Author: ASF GitHub Bot Created on: 03/Jun/20 19:49 Start Date: 03/Jun/20 19:49 Worklog Time Spent: 10m Work Description: chamikaramj opened a new pull request #11911: URL: https://github.com/apache/beam/pull/11911 Send an error to the runner for certain split and progress request failures. This is needed to prevent runner from waiting unnecessarily and timing out for progress and split requests that are received by the runner before receiving the corresponding bundle descriptor. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build St
[jira] [Work logged] (BEAM-8841) Add ability to perform BigQuery file loads using avro
[
https://issues.apache.org/jira/browse/BEAM-8841?focusedWorklogId=440943&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440943
]
ASF GitHub Bot logged work on BEAM-8841:
Author: ASF GitHub Bot
Created on: 03/Jun/20 19:35
Start Date: 03/Jun/20 19:35
Worklog Time Spent: 10m
Work Description: pabloem commented on pull request #11896:
URL: https://github.com/apache/beam/pull/11896#issuecomment-638416678
Run PythonLint PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440943)
Time Spent: 9.5h (was: 9h 20m)
> Add ability to perform BigQuery file loads using avro
> -
>
> Key: BEAM-8841
> URL: https://issues.apache.org/jira/browse/BEAM-8841
> Project: Beam
> Issue Type: Improvement
> Components: io-py-gcp
>Reporter: Chun Yang
>Assignee: Chun Yang
>Priority: P3
> Fix For: 2.21.0
>
> Time Spent: 9.5h
> Remaining Estimate: 0h
>
> Currently, JSON format is used for file loads into BigQuery in the Python
> SDK. JSON has some disadvantages including size of serialized data and
> inability to represent NaN and infinity float values.
> BigQuery supports loading files in avro format, which can overcome these
> disadvantages. The Java SDK already supports loading files using avro format
> (BEAM-2879) so it makes sense to support it in the Python SDK as well.
> The change will be somewhere around
> [{{BigQueryBatchFileLoads}}|https://github.com/apache/beam/blob/3e7865ee6c6a56e51199515ec5b4b16de1ddd166/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py#L554].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-10186) Send error response to runner when split or progress requests are received out of order
[ https://issues.apache.org/jira/browse/BEAM-10186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath reassigned BEAM-10186: Assignee: Chamikara Madhusanka Jayalath > Send error response to runner when split or progress requests are received > out of order > --- > > Key: BEAM-10186 > URL: https://issues.apache.org/jira/browse/BEAM-10186 > Project: Beam > Issue Type: Improvement > Components: sdk-java-harness >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P2 > > We need to send an error to the runner when split requests or progress > updates are received by the SDK harness before receiving the corresponding > bundle. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10186) Send error response to runner when split or progress requests are received out of order
Chamikara Madhusanka Jayalath created BEAM-10186: Summary: Send error response to runner when split or progress requests are received out of order Key: BEAM-10186 URL: https://issues.apache.org/jira/browse/BEAM-10186 Project: Beam Issue Type: Improvement Components: sdk-java-harness Reporter: Chamikara Madhusanka Jayalath We need to send an error to the runner when split requests or progress updates are received by the SDK harness before receiving the corresponding bundle. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8841) Add ability to perform BigQuery file loads using avro
[
https://issues.apache.org/jira/browse/BEAM-8841?focusedWorklogId=440931&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440931
]
ASF GitHub Bot logged work on BEAM-8841:
Author: ASF GitHub Bot
Created on: 03/Jun/20 19:11
Start Date: 03/Jun/20 19:11
Worklog Time Spent: 10m
Work Description: pabloem commented on pull request #11896:
URL: https://github.com/apache/beam/pull/11896#issuecomment-638403708
thanks @chunyang ! this LGTM. I'll merge after lint passes
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440931)
Time Spent: 9h 20m (was: 9h 10m)
> Add ability to perform BigQuery file loads using avro
> -
>
> Key: BEAM-8841
> URL: https://issues.apache.org/jira/browse/BEAM-8841
> Project: Beam
> Issue Type: Improvement
> Components: io-py-gcp
>Reporter: Chun Yang
>Assignee: Chun Yang
>Priority: P3
> Fix For: 2.21.0
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> Currently, JSON format is used for file loads into BigQuery in the Python
> SDK. JSON has some disadvantages including size of serialized data and
> inability to represent NaN and infinity float values.
> BigQuery supports loading files in avro format, which can overcome these
> disadvantages. The Java SDK already supports loading files using avro format
> (BEAM-2879) so it makes sense to support it in the Python SDK as well.
> The change will be somewhere around
> [{{BigQueryBatchFileLoads}}|https://github.com/apache/beam/blob/3e7865ee6c6a56e51199515ec5b4b16de1ddd166/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py#L554].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8841) Add ability to perform BigQuery file loads using avro
[
https://issues.apache.org/jira/browse/BEAM-8841?focusedWorklogId=440930&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440930
]
ASF GitHub Bot logged work on BEAM-8841:
Author: ASF GitHub Bot
Created on: 03/Jun/20 19:10
Start Date: 03/Jun/20 19:10
Worklog Time Spent: 10m
Work Description: pabloem commented on pull request #11896:
URL: https://github.com/apache/beam/pull/11896#issuecomment-638403555
retest this please
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440930)
Time Spent: 9h 10m (was: 9h)
> Add ability to perform BigQuery file loads using avro
> -
>
> Key: BEAM-8841
> URL: https://issues.apache.org/jira/browse/BEAM-8841
> Project: Beam
> Issue Type: Improvement
> Components: io-py-gcp
>Reporter: Chun Yang
>Assignee: Chun Yang
>Priority: P3
> Fix For: 2.21.0
>
> Time Spent: 9h 10m
> Remaining Estimate: 0h
>
> Currently, JSON format is used for file loads into BigQuery in the Python
> SDK. JSON has some disadvantages including size of serialized data and
> inability to represent NaN and infinity float values.
> BigQuery supports loading files in avro format, which can overcome these
> disadvantages. The Java SDK already supports loading files using avro format
> (BEAM-2879) so it makes sense to support it in the Python SDK as well.
> The change will be somewhere around
> [{{BigQueryBatchFileLoads}}|https://github.com/apache/beam/blob/3e7865ee6c6a56e51199515ec5b4b16de1ddd166/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py#L554].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-10181) pull_licenses script should create python3 virtualenv
[ https://issues.apache.org/jira/browse/BEAM-10181?focusedWorklogId=440929&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440929 ] ASF GitHub Bot logged work on BEAM-10181: - Author: ASF GitHub Bot Created on: 03/Jun/20 19:00 Start Date: 03/Jun/20 19:00 Worklog Time Spent: 10m Work Description: TheNeuralBit merged pull request #11900: URL: https://github.com/apache/beam/pull/11900 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440929) Time Spent: 20m (was: 10m) > pull_licenses script should create python3 virtualenv > - > > Key: BEAM-10181 > URL: https://issues.apache.org/jira/browse/BEAM-10181 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Setting fix version as 2.22.0 since this is preventing me from building > release containers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-10181) pull_licenses script should create python3 virtualenv
[ https://issues.apache.org/jira/browse/BEAM-10181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Hulette resolved BEAM-10181. -- Resolution: Fixed > pull_licenses script should create python3 virtualenv > - > > Key: BEAM-10181 > URL: https://issues.apache.org/jira/browse/BEAM-10181 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: P2 > Fix For: 2.23.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Setting fix version as 2.22.0 since this is preventing me from building > release containers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10181) pull_licenses script should create python3 virtualenv
[ https://issues.apache.org/jira/browse/BEAM-10181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Hulette updated BEAM-10181: - Fix Version/s: (was: 2.22.0) 2.23.0 > pull_licenses script should create python3 virtualenv > - > > Key: BEAM-10181 > URL: https://issues.apache.org/jira/browse/BEAM-10181 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: P2 > Fix For: 2.23.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Setting fix version as 2.22.0 since this is preventing me from building > release containers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10184) Build python wheels on GitHub Actions for Linux/MacOS
[ https://issues.apache.org/jira/browse/BEAM-10184?focusedWorklogId=440927&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440927 ] ASF GitHub Bot logged work on BEAM-10184: - Author: ASF GitHub Bot Created on: 03/Jun/20 18:56 Start Date: 03/Jun/20 18:56 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #11877: URL: https://github.com/apache/beam/pull/11877#issuecomment-638395828 Could we make it such that: - we have a cron job that builds nightly (like a snapshot release) - and a way to manually trigger this from the release branch, so that the release manager can build wheel files on demand. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440927) Time Spent: 50m (was: 40m) > Build python wheels on GitHub Actions for Linux/MacOS > - > > Key: BEAM-10184 > URL: https://issues.apache.org/jira/browse/BEAM-10184 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Tobiasz Kedzierski >Priority: P2 > Labels: build, python, python-packages, python-wheel > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10184) Build python wheels on GitHub Actions for Linux/MacOS
[
https://issues.apache.org/jira/browse/BEAM-10184?focusedWorklogId=440923&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440923
]
ASF GitHub Bot logged work on BEAM-10184:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 18:45
Start Date: 03/Jun/20 18:45
Worklog Time Spent: 10m
Work Description: tvalentyn commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r434779681
##
File path: .github/workflows/build_wheels.yml
##
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+branches:
+ - master
+ - release-*
+tags:
+ - v*
+
+jobs:
+
+ build_source:
+runs-on: ubuntu-18.04
+steps:
+ - name: Checkout code
+uses: actions/checkout@v2
+ - name: Install python
+uses: actions/setup-python@v2
+with:
+ python-version: 3.7
+ - name: Get build dependencies
+working-directory: ./sdks/python
+run: python3 -m pip install cython && python3 -m pip install -r
build-requirements.txt
+ - name: Install wheels
+run: python3 -m pip install wheel
+ - name: Buld source
+working-directory: ./sdks/python
+run: python3 setup.py sdist --formats=gztar,zip
+ - name: Unzip source
+working-directory: ./sdks/python
+run: unzip dist/$(ls dist | grep .zip | head -n 1)
+ - name: Rename source directory
+working-directory: ./sdks/python
+run: mv $(ls | grep apache-beam) apache-beam-source
+ - name: Upload source
+uses: actions/upload-artifact@v2
+with:
+ name: source
+ path: sdks/python/apache-beam-source
+ - name: Upload compressed sources
+uses: actions/upload-artifact@v2
+with:
+ name: source_gztar_zip
+ path: sdks/python/dist
+
+ prepare_gcs:
+name: Prepare GCS
+needs: build_source
+runs-on: ubuntu-18.04
+steps:
+ - name: Authenticate on GCP
+uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Remove existing files on GCS bucket
+run: gsutil rm -r "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/"
|| true
+
+ upload_source_to_gcs:
+name: Upload source to GCS bucket
+needs: prepare_gcs
+runs-on: ubuntu-18.04
+steps:
+ - name: Download wheels
+uses: actions/download-artifact@v2
+with:
+ name: source_gztar_zip
+ path: source/
+ - name: Authenticate on GCP
+uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Copy sources to GCS bucket
+run: gsutil cp -r -a public-read source/* gs://${{ secrets.CCP_BUCKET
}}/${GITHUB_REF##*/}/
+ - name: List sources on GCS bucket
+run: |
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.tar.gz"
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.zip"
+
+ build_wheels:
+name: Build wheels on ${{ matrix.os }}
+needs: prepare_gcs
+runs-on: ${{ matrix.os }}
+strategy:
+ matrix:
+os : [ubuntu-18.04, macos-10.15]
+steps:
+- name: Download source
+ uses: actions/download-artifact@v2
+ with:
+name: source
+path: apache-beam-source
+- name: Install Python
+ uses: actions/setup-python@v2
+ with:
+python-version: 3.7
+- name: Install packages on Mac
+ if: startsWith(matrix.os, 'macos')
+ run: |
+brew update
+brew install pkg-config
+- name: Install cibuildwheel
+ run: pip install cibuildwheel==1.4.2
+- name: Build wheel
+ working-directory: apache-beam-source
+ env:
+CIBW_BUILD: cp27-* cp35-* cp36-* cp37-*
Review comment:
Yes, let's add py3.8. Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440923)
Time Spent: 40m (was: 0.5h)
> Build python wheels on GitHub Actions for Linux/MacOS
> -
>
> Key: BEAM-10184
> URL: https://issues.apache.org/jira/browse/BEAM-10184
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Tobiasz Kedzierski
>
[jira] [Commented] (BEAM-9676) Go SDK Code Katas
[ https://issues.apache.org/jira/browse/BEAM-9676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125215#comment-17125215 ] Damon Douglas commented on BEAM-9676: - Thank you for the clarification. For better house-keeping, I removed my self-assignment from sub-tasks not actively being pursued to communicate to others that they can jump in and help. > Go SDK Code Katas > - > > Key: BEAM-9676 > URL: https://issues.apache.org/jira/browse/BEAM-9676 > Project: Beam > Issue Type: Improvement > Components: katas, sdk-go >Reporter: Robert Burke >Assignee: Damon Douglas >Priority: P2 > > There should be code katas for the Go SDK similar to the Java and Python SDKs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3489) Expose the message id of received messages within PubsubMessage
[ https://issues.apache.org/jira/browse/BEAM-3489?focusedWorklogId=440914&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440914 ] ASF GitHub Bot logged work on BEAM-3489: Author: ASF GitHub Bot Created on: 03/Jun/20 18:28 Start Date: 03/Jun/20 18:28 Worklog Time Spent: 10m Work Description: thinhha commented on pull request #11873: URL: https://github.com/apache/beam/pull/11873#issuecomment-638380598 Thanks @lukecwik. I've updated the change. I thought I had changed it back to your suggestion before! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440914) Time Spent: 9h 20m (was: 9h 10m) > Expose the message id of received messages within PubsubMessage > --- > > Key: BEAM-3489 > URL: https://issues.apache.org/jira/browse/BEAM-3489 > Project: Beam > Issue Type: New Feature > Components: io-java-gcp >Reporter: Luke Cwik >Assignee: Thinh Ha >Priority: P3 > Labels: newbie, starter > Time Spent: 9h 20m > Remaining Estimate: 0h > > This task is about passing forward the message id from the pubsub proto to > the java PubsubMessage. > Add a message id field to PubsubMessage. > Update the coder for PubsubMessage to encode the message id. > Update the translation from the Pubsub proto message to the Dataflow message: > https://github.com/apache/beam/blob/2e275264b21db45787833502e5e42907b05e28b8/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java#L976 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-10096) Spark runners are numbered 1,2,2
[ https://issues.apache.org/jira/browse/BEAM-10096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Weaver resolved BEAM-10096. Fix Version/s: Not applicable Resolution: Fixed > Spark runners are numbered 1,2,2 > > > Key: BEAM-10096 > URL: https://issues.apache.org/jira/browse/BEAM-10096 > Project: Beam > Issue Type: Bug > Components: website >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: P4 > Fix For: Not applicable > > Time Spent: 20m > Remaining Estimate: 0h > > https://beam.apache.org/documentation/runners/spark/ > 1. A legacy Runner... > 2. An Structured Streaming Spark Runner... > 2. A portable Runner... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9977) Build Kafka Read on top of Java SplittableDoFn
[
https://issues.apache.org/jira/browse/BEAM-9977?focusedWorklogId=440912&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440912
]
ASF GitHub Bot logged work on BEAM-9977:
Author: ASF GitHub Bot
Created on: 03/Jun/20 18:21
Start Date: 03/Jun/20 18:21
Worklog Time Spent: 10m
Work Description: boyuanzz commented on a change in pull request #11749:
URL: https://github.com/apache/beam/pull/11749#discussion_r434766297
##
File path:
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ReadFromKafkaViaSDF.java
##
@@ -0,0 +1,697 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.kafka;
+
+import static
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderRegistry;
+import org.apache.beam.sdk.io.range.OffsetRange;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.DoFn.Element;
+import org.apache.beam.sdk.transforms.DoFn.GetRestrictionCoder;
+import org.apache.beam.sdk.transforms.DoFn.OutputReceiver;
+import org.apache.beam.sdk.transforms.DoFn.ProcessElement;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import
org.apache.beam.sdk.transforms.splittabledofn.GrowableOffsetRangeTracker;
+import org.apache.beam.sdk.transforms.splittabledofn.OffsetRangeTracker;
+import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
+import org.apache.beam.sdk.transforms.splittabledofn.WatermarkEstimator;
+import
org.apache.beam.sdk.transforms.splittabledofn.WatermarkEstimators.MonotonicallyIncreasing;
+import org.apache.beam.sdk.values.PCollection;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Closeables;
+import org.apache.kafka.clients.consumer.Consumer;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.utils.AppInfoParser;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link PTransform} that takes a PCollection of {@link
KafkaSourceDescription} as input and
+ * outputs a PCollection of {@link KafkaRecord}. The core implementation is
based on {@code
+ * SplittableDoFn}. For more details about the concept of {@code
SplittableDoFn}, please refer to
+ * the beam blog post: https://beam.apache.org/blog/splittable-do-fn/ and
design
+ * doc:https://s.apache.org/beam-fn-api. The major difference from {@link
KafkaIO.Read} is, {@link
+ * ReadFromKafkaViaSDF} doesn't require source descriptions(e.g., {@link
+ * KafkaIO.Read#getTopicPartitions()}, {@link KafkaIO.Read#getTopics()}, {@link
+ * KafkaIO.Read#getStartReadTime()}, etc.) during the pipeline construction
time. Instead, the
+ * pipeline can populate these source descriptions during runtime. For
example, the pipeline can
+ * query Kafka topics from BigQuery table and read these topics via {@link
ReadFromKafkaViaSDF}.
+ *
+ * Common Kafka Consumer Configurations
+ *
+ * Most Kafka consumer configurations are similar to {@link KafkaIO.Read}:
+ *
+ *
+ * {@link ReadFromKafkaViaSDF#getConsumerConfig()} is the same as {@link
+ * KafkaIO.Read#getConsumerConfig()}.
+ * {@link ReadFromKafkaViaSDF#getConsumerFactoryFn()} is the same as
{@link
+ * KafkaIO.Read#getConsumerFactoryFn()}.
+ * {@link ReadFromKafkaViaSDF#getOffsetConsumerConfig()}
[jira] [Commented] (BEAM-10027) Support for Kotlin-based Beam Katas
[ https://issues.apache.org/jira/browse/BEAM-10027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125203#comment-17125203 ] Pablo Estrada commented on BEAM-10027: -- It's technically not a part of the release, so it's not that necessary to specify, but changes merged now will be in Beam 2.23.0 - so I've added it. Thanks! > Support for Kotlin-based Beam Katas > --- > > Key: BEAM-10027 > URL: https://issues.apache.org/jira/browse/BEAM-10027 > Project: Beam > Issue Type: Improvement > Components: katas >Reporter: Rion Williams >Assignee: Rion Williams >Priority: P2 > Fix For: 2.23.0 > > Original Estimate: 8h > Time Spent: 15h 40m > Remaining Estimate: 0h > > Currently, there are a series of examples available demonstrating the use of > Apache Beam with Kotlin. It would be nice to have support for the same Beam > Katas that exist for Python, Go, and Java to also support Kotlin. > The port itself shouldn't be that involved since it can still target the JVM, > so it would likely just require the inclusion for Kotlin dependencies and a > conversion for all of the existing Java examples. -- This message was sent by Atlassian Jira (v8.3.4#803005)
