[jira] [Commented] (BEAM-9144) Beam's own Avro TimeConversion class in beam-sdk-java-core
[
https://issues.apache.org/jira/browse/BEAM-9144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018480#comment-17018480
]
Aaron Dixon commented on BEAM-9144:
---
Perfect. Thanks [~suztomo], will report back
> Beam's own Avro TimeConversion class in beam-sdk-java-core
> ---
>
> Key: BEAM-9144
> URL: https://issues.apache.org/jira/browse/BEAM-9144
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Fix For: 2.19.0
>
> Attachments: avro-beam-dependency-graph.png
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> From Aaron's comment in
> https://issues.apache.org/jira/browse/BEAM-8388?focusedCommentId=17016476=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17016476
> .
> {quote}My org must use Avro 1.9.x (due to some Avro schema resolution issues
> resolved in 1.9.x) so downgrading Avro is not possible for us.
> Beam 2.16.0 is compatible with our usage of Avro 1.9.x – but upgrading to
> 2.17.0 we are broken as 2.17.0 links to Java classes in Avro 1.8.x that are
> not available in 1.9.x.
> {quote}
> The Java class is
> {{org.apache.avro.data.TimeConversions.TimestampConversion}} in Avro 1.8.
> It's renamed to {{org.apache.avro.data.JodaTimeConversions}} in Avro 1.9.
> h1. Beam Java SDK cannot upgrade Avro to 1.9
> Beam has Spark runners and Spark has not yet upgraded to Avro 1.9.
> Illustration of the dependency
> !avro-beam-dependency-graph.png|width=799,height=385!
> h1. Short-term Solution
> As illustrated above, as long as Beam Java SDK uses only the intersection of
> Avro classes, method, and fields between Avro 1.8 and 1.9, it will provide
> flexibility in runtime Avro versions (as it did until Beam 2.16).
> h2. Difference of the TimeConversion Classes
> Avro 1.9's TimestampConversion overrides {{getRecommendedSchema}} method.
> Details below:
> Avro 1.8's TimeConversions.TimestampConversion:
> {code:java}
> public static class TimestampConversion extends Conversion {
> @Override
> public Class getConvertedType() {
> return DateTime.class;
> }
> @Override
> public String getLogicalTypeName() {
> return "timestamp-millis";
> }
> @Override
> public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType
> type) {
> return new DateTime(millisFromEpoch, DateTimeZone.UTC);
> }
> @Override
> public Long toLong(DateTime timestamp, Schema schema, LogicalType type) {
> return timestamp.getMillis();
> }
> }
> {code}
> Avro 1.9's JodaTimeConversions.TimestampConversion:
> {code:java}
> public static class TimestampConversion extends Conversion {
> @Override
> public Class getConvertedType() {
> return DateTime.class;
> }
> @Override
> public String getLogicalTypeName() {
> return "timestamp-millis";
> }
> @Override
> public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType
> type) {
> return new DateTime(millisFromEpoch, DateTimeZone.UTC);
> }
> @Override
> public Long toLong(DateTime timestamp, Schema schema, LogicalType type) {
> return timestamp.getMillis();
> }
> @Override
> public Schema getRecommendedSchema() {
> return
> LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG));
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9144) Beam's own Avro TimeConversion class in beam-sdk-java-core
[
https://issues.apache.org/jira/browse/BEAM-9144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018471#comment-17018471
]
Tomo Suzuki commented on BEAM-9144:
---
Yes, new artifact will be at
https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-core/2.20.0-SNAPSHOT/
.
> Beam's own Avro TimeConversion class in beam-sdk-java-core
> ---
>
> Key: BEAM-9144
> URL: https://issues.apache.org/jira/browse/BEAM-9144
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Fix For: 2.19.0
>
> Attachments: avro-beam-dependency-graph.png
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> From Aaron's comment in
> https://issues.apache.org/jira/browse/BEAM-8388?focusedCommentId=17016476=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17016476
> .
> {quote}My org must use Avro 1.9.x (due to some Avro schema resolution issues
> resolved in 1.9.x) so downgrading Avro is not possible for us.
> Beam 2.16.0 is compatible with our usage of Avro 1.9.x – but upgrading to
> 2.17.0 we are broken as 2.17.0 links to Java classes in Avro 1.8.x that are
> not available in 1.9.x.
> {quote}
> The Java class is
> {{org.apache.avro.data.TimeConversions.TimestampConversion}} in Avro 1.8.
> It's renamed to {{org.apache.avro.data.JodaTimeConversions}} in Avro 1.9.
> h1. Beam Java SDK cannot upgrade Avro to 1.9
> Beam has Spark runners and Spark has not yet upgraded to Avro 1.9.
> Illustration of the dependency
> !avro-beam-dependency-graph.png|width=799,height=385!
> h1. Short-term Solution
> As illustrated above, as long as Beam Java SDK uses only the intersection of
> Avro classes, method, and fields between Avro 1.8 and 1.9, it will provide
> flexibility in runtime Avro versions (as it did until Beam 2.16).
> h2. Difference of the TimeConversion Classes
> Avro 1.9's TimestampConversion overrides {{getRecommendedSchema}} method.
> Details below:
> Avro 1.8's TimeConversions.TimestampConversion:
> {code:java}
> public static class TimestampConversion extends Conversion {
> @Override
> public Class getConvertedType() {
> return DateTime.class;
> }
> @Override
> public String getLogicalTypeName() {
> return "timestamp-millis";
> }
> @Override
> public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType
> type) {
> return new DateTime(millisFromEpoch, DateTimeZone.UTC);
> }
> @Override
> public Long toLong(DateTime timestamp, Schema schema, LogicalType type) {
> return timestamp.getMillis();
> }
> }
> {code}
> Avro 1.9's JodaTimeConversions.TimestampConversion:
> {code:java}
> public static class TimestampConversion extends Conversion {
> @Override
> public Class getConvertedType() {
> return DateTime.class;
> }
> @Override
> public String getLogicalTypeName() {
> return "timestamp-millis";
> }
> @Override
> public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType
> type) {
> return new DateTime(millisFromEpoch, DateTimeZone.UTC);
> }
> @Override
> public Long toLong(DateTime timestamp, Schema schema, LogicalType type) {
> return timestamp.getMillis();
> }
> @Override
> public Schema getRecommendedSchema() {
> return
> LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG));
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7861) Make it easy to change between multi-process and multi-thread mode for Python Direct runners
[ https://issues.apache.org/jira/browse/BEAM-7861?focusedWorklogId=373993=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373993 ] ASF GitHub Bot logged work on BEAM-7861: Author: ASF GitHub Bot Created on: 18/Jan/20 02:04 Start Date: 18/Jan/20 02:04 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #10616: [BEAM-7861] update documentation about --direct_running_mode option with direct runner. URL: https://github.com/apache/beam/pull/10616#issuecomment-575855799 Run Website_Stage_GCS PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373993) Time Spent: 4h 20m (was: 4h 10m) > Make it easy to change between multi-process and multi-thread mode for Python > Direct runners > > > Key: BEAM-7861 > URL: https://issues.apache.org/jira/browse/BEAM-7861 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.19.0 > > Time Spent: 4h 20m > Remaining Estimate: 0h > > BEAM-3645 makes it possible to run a map task parallel. > However, users need to change runner when switch between multithreading and > multiprocessing mode. > We want to add a flag (ex: --use-multiprocess) to make the switch easy > without changing the runner each time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7861) Make it easy to change between multi-process and multi-thread mode for Python Direct runners
[ https://issues.apache.org/jira/browse/BEAM-7861?focusedWorklogId=373992=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373992 ] ASF GitHub Bot logged work on BEAM-7861: Author: ASF GitHub Bot Created on: 18/Jan/20 02:04 Start Date: 18/Jan/20 02:04 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #10616: [BEAM-7861] update documentation about --direct_running_mode option with direct runner. URL: https://github.com/apache/beam/pull/10616#issuecomment-575855782 Run Website_Stage_GCS PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373992) Time Spent: 4h 10m (was: 4h) > Make it easy to change between multi-process and multi-thread mode for Python > Direct runners > > > Key: BEAM-7861 > URL: https://issues.apache.org/jira/browse/BEAM-7861 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.19.0 > > Time Spent: 4h 10m > Remaining Estimate: 0h > > BEAM-3645 makes it possible to run a map task parallel. > However, users need to change runner when switch between multithreading and > multiprocessing mode. > We want to add a flag (ex: --use-multiprocess) to make the switch easy > without changing the runner each time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7861) Make it easy to change between multi-process and multi-thread mode for Python Direct runners
[ https://issues.apache.org/jira/browse/BEAM-7861?focusedWorklogId=373994=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373994 ] ASF GitHub Bot logged work on BEAM-7861: Author: ASF GitHub Bot Created on: 18/Jan/20 02:04 Start Date: 18/Jan/20 02:04 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #10616: [BEAM-7861] update documentation about --direct_running_mode option with direct runner. URL: https://github.com/apache/beam/pull/10616#issuecomment-575855842 Run Website PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373994) Time Spent: 4.5h (was: 4h 20m) > Make it easy to change between multi-process and multi-thread mode for Python > Direct runners > > > Key: BEAM-7861 > URL: https://issues.apache.org/jira/browse/BEAM-7861 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.19.0 > > Time Spent: 4.5h > Remaining Estimate: 0h > > BEAM-3645 makes it possible to run a map task parallel. > However, users need to change runner when switch between multithreading and > multiprocessing mode. > We want to add a flag (ex: --use-multiprocess) to make the switch easy > without changing the runner each time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9137) beam_PostCommit_Py_ValCont should run with dataflow_worker_jar
[ https://issues.apache.org/jira/browse/BEAM-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay updated BEAM-9137: -- Parent: BEAM-8193 Issue Type: Sub-task (was: Bug) > beam_PostCommit_Py_ValCont should run with dataflow_worker_jar > -- > > Key: BEAM-9137 > URL: https://issues.apache.org/jira/browse/BEAM-9137 > Project: Beam > Issue Type: Sub-task > Components: testing >Reporter: Boyuan Zhang >Assignee: Valentyn Tymofieiev >Priority: Major > > For the first failure, please refer to > https://builds.apache.org/job/beam_PostCommit_Py_ValCont/5172/#showFailuresLink -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9137) beam_PostCommit_Py_ValCont should run with dataflow_worker_jar
[ https://issues.apache.org/jira/browse/BEAM-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay reassigned BEAM-9137: - Assignee: Valentyn Tymofieiev > beam_PostCommit_Py_ValCont should run with dataflow_worker_jar > -- > > Key: BEAM-9137 > URL: https://issues.apache.org/jira/browse/BEAM-9137 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: Boyuan Zhang >Assignee: Valentyn Tymofieiev >Priority: Major > > For the first failure, please refer to > https://builds.apache.org/job/beam_PostCommit_Py_ValCont/5172/#showFailuresLink -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8626) Implement status api handler in python sdk harness
[ https://issues.apache.org/jira/browse/BEAM-8626?focusedWorklogId=373989=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373989 ] ASF GitHub Bot logged work on BEAM-8626: Author: ASF GitHub Bot Created on: 18/Jan/20 01:08 Start Date: 18/Jan/20 01:08 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #10598: [BEAM-8626] Implement status fn api handler in python sdk URL: https://github.com/apache/beam/pull/10598#discussion_r368193392 ## File path: sdks/python/apache_beam/runners/worker/worker_status.py ## @@ -0,0 +1,139 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +"""Worker status api handler for reporting SDK harness debug info.""" + +from __future__ import absolute_import +from __future__ import division + +import queue +import sys +import threading +import traceback +from collections import defaultdict + +import grpc + +from apache_beam.portability.api import beam_fn_api_pb2 +from apache_beam.portability.api import beam_fn_api_pb2_grpc +from apache_beam.runners.worker.channel_factory import GRPCChannelFactory +from apache_beam.runners.worker.worker_id_interceptor import WorkerIdInterceptor + + +def thread_dump(): Review comment: I agree, we can get rid of status http server. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373989) Time Spent: 2h (was: 1h 50m) > Implement status api handler in python sdk harness > -- > > Key: BEAM-8626 > URL: https://issues.apache.org/jira/browse/BEAM-8626 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8626) Implement status api handler in python sdk harness
[
https://issues.apache.org/jira/browse/BEAM-8626?focusedWorklogId=373990=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373990
]
ASF GitHub Bot logged work on BEAM-8626:
Author: ASF GitHub Bot
Created on: 18/Jan/20 01:08
Start Date: 18/Jan/20 01:08
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #10598: [BEAM-8626]
Implement status fn api handler in python sdk
URL: https://github.com/apache/beam/pull/10598#discussion_r368193443
##
File path: sdks/python/apache_beam/runners/worker/worker_status.py
##
@@ -0,0 +1,139 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Worker status api handler for reporting SDK harness debug info."""
+
+from __future__ import absolute_import
+from __future__ import division
+
+import queue
+import sys
+import threading
+import traceback
+from collections import defaultdict
+
+import grpc
+
+from apache_beam.portability.api import beam_fn_api_pb2
+from apache_beam.portability.api import beam_fn_api_pb2_grpc
+from apache_beam.runners.worker.channel_factory import GRPCChannelFactory
+from apache_beam.runners.worker.worker_id_interceptor import
WorkerIdInterceptor
+
+
+def thread_dump():
+ # deduplicate threads with same stack trace
+ stack_traces = defaultdict(list)
+ frames = sys._current_frames() # pylint: disable=protected-access
+
+ for t in threading.enumerate():
+stack_trace = ''.join(traceback.format_stack(frames[t.ident]))
+thread_ident_name = (t.ident, t.name)
+stack_traces[stack_trace].append(thread_ident_name)
+
+ all_traces = ['=' * 10 + 'THREAD DUMP' + '=' * 10]
+ for stack, identity in stack_traces.items():
+ident, name = identity[0]
+trace = '--- Thread #%s name: %s %s---\n' % (
+ident, name, 'and other %d threads' %
+(len(identity) - 1) if len(identity) > 1 else '')
+if len(identity) > 1:
+ trace += 'threads: %s\n' % identity
+trace += stack
+all_traces.append(trace)
+ all_traces.append('=' * 30)
+ return '\n'.join(x.encode('utf-8') for x in all_traces)
+
+
+def active_processing_bundles_state(bundle_process_cache):
+ active_bundles = ['=' * 10 + 'ACTIVE PROCESSING BUNDLES' + '=' * 10]
+ if not bundle_process_cache.active_bundle_processors:
+active_bundles.append("No active processing bundles.")
+ else:
+cache = []
+for instruction in list(
+bundle_process_cache.active_bundle_processors.keys()):
+ processor = bundle_process_cache.lookup(instruction)
+ if processor:
+info = processor.state_sampler.get_info()
+cache.append((instruction,
+ processor.process_bundle_descriptor.id,
+ info.tracked_thread, info.time_since_transition))
+# reverse sort active bundle by time since last transition, keep top 10.
+cache.sort(key=lambda x: x[-1], reverse=True)
+for s in cache[:10]:
+ state = '--- instruction %s ---\n' % s[0]
+ state += 'ProcessBundleDescriptorId: %s\n' % s[1]
+ state += "tracked thread: %s\n" % s[2]
+ state += "time since transition: %.2f seconds\n" % (s[3] / 1e9)
+ active_bundles.append(state)
+
+ active_bundles.append('=' * 30)
+ return '\n'.join(x.encode('utf-8') for x in active_bundles)
+
+
+DONE = object()
+
+
+class FnApiWorkerStatusHandler(object):
+ def __init__(self, status_address, bundle_process_cache=None):
+self._alive = True
+self._bundle_process_cache = bundle_process_cache
+ch = GRPCChannelFactory.insecure_channel(status_address)
+grpc.channel_ready_future(ch).result(timeout=60)
+self._status_channel = grpc.intercept_channel(ch, WorkerIdInterceptor())
+self._status_stub = beam_fn_api_pb2_grpc.BeamFnWorkerStatusStub(
+self._status_channel)
+self._responses = queue.Queue()
+self._server = threading.Thread(target=lambda: self._serve(),
+name='fn_api_status_handler')
+self._server.daemon = True
+self._server.start()
+
+ def _get_responses(self):
+while True:
+ response = self._responses.get()
+ if response is DONE:
+self._alive =
[jira] [Updated] (BEAM-9137) beam_PostCommit_Py_ValCont should run with dataflow_worker_jar
[ https://issues.apache.org/jira/browse/BEAM-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang updated BEAM-9137: --- Component/s: (was: test-failures) testing > beam_PostCommit_Py_ValCont should run with dataflow_worker_jar > -- > > Key: BEAM-9137 > URL: https://issues.apache.org/jira/browse/BEAM-9137 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: Boyuan Zhang >Priority: Major > > For the first failure, please refer to > https://builds.apache.org/job/beam_PostCommit_Py_ValCont/5172/#showFailuresLink -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9137) beam_PostCommit_Py_ValCont should run with dataflow_worker_jar
[ https://issues.apache.org/jira/browse/BEAM-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang updated BEAM-9137: --- Affects Version/s: (was: 2.19.0) > beam_PostCommit_Py_ValCont should run with dataflow_worker_jar > -- > > Key: BEAM-9137 > URL: https://issues.apache.org/jira/browse/BEAM-9137 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Boyuan Zhang >Priority: Major > > For the first failure, please refer to > https://builds.apache.org/job/beam_PostCommit_Py_ValCont/5172/#showFailuresLink -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9137) beam_PostCommit_Py_ValCont should run with dataflow_worker_jar
[ https://issues.apache.org/jira/browse/BEAM-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang updated BEAM-9137: --- Priority: Major (was: Blocker) > beam_PostCommit_Py_ValCont should run with dataflow_worker_jar > -- > > Key: BEAM-9137 > URL: https://issues.apache.org/jira/browse/BEAM-9137 > Project: Beam > Issue Type: Bug > Components: test-failures >Affects Versions: 2.19.0 >Reporter: Boyuan Zhang >Priority: Major > Fix For: 2.19.0 > > > For the first failure, please refer to > https://builds.apache.org/job/beam_PostCommit_Py_ValCont/5172/#showFailuresLink -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9137) beam_PostCommit_Py_ValCont should run with dataflow_worker_jar
[ https://issues.apache.org/jira/browse/BEAM-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang updated BEAM-9137: --- Fix Version/s: (was: 2.19.0) > beam_PostCommit_Py_ValCont should run with dataflow_worker_jar > -- > > Key: BEAM-9137 > URL: https://issues.apache.org/jira/browse/BEAM-9137 > Project: Beam > Issue Type: Bug > Components: test-failures >Affects Versions: 2.19.0 >Reporter: Boyuan Zhang >Priority: Major > > For the first failure, please refer to > https://builds.apache.org/job/beam_PostCommit_Py_ValCont/5172/#showFailuresLink -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8684) Beam Dependency Update Request: com.google.apis:google-api-services-bigquery
[ https://issues.apache.org/jira/browse/BEAM-8684?focusedWorklogId=373987=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373987 ] ASF GitHub Bot logged work on BEAM-8684: Author: ASF GitHub Bot Created on: 18/Jan/20 01:04 Start Date: 18/Jan/20 01:04 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10631: [BEAM-8684] Google proto and google-services library upgrades URL: https://github.com/apache/beam/pull/10631#issuecomment-575849736 Java HadoopFormatIO Performance Test failed. This is critical. I missed to check linkage errors of sdks:java:io:hadoop-format: ``` 19:58:37 > Task :sdks:java:io:hadoop-format:integrationTest FAILED 19:58:37 19:58:37 org.apache.beam.sdk.io.hadoop.format.HadoopFormatIOIT > writeAndReadUsingHadoopFormat FAILED 19:58:37 java.lang.NoSuchMethodError: com.google.cloud.ServiceOptions.getQuotaProjectId()Ljava/lang/String; ``` I will investigate further next week. —— Run JavaPortabilityApi PreCommit failed: ``` 19:45:37 > Task :runners:google-cloud-dataflow-java:buildAndPushDockerContainer FAILED 19:45:37 Error response from daemon: No such image: apachebeam/java_sdk:2.20.0-SNAPSHOT 19:45:37 19:45:37 > Task :runners:google-cloud-dataflow-java:cleanUpDockerImages FAILED 19:45:37 Error: No such image: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20200118004259 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373987) Time Spent: 2.5h (was: 2h 20m) > Beam Dependency Update Request: com.google.apis:google-api-services-bigquery > > > Key: BEAM-8684 > URL: https://issues.apache.org/jira/browse/BEAM-8684 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > > - 2019-11-15 19:39:07.113511 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:04:31.460554 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:09:35.656962 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:08:45.501496 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:08:46.385699 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:04:15.902178 > - > Please consider upgrading the dependency >
[jira] [Commented] (BEAM-9137) PR10338 breaks beam_PostCommit_Py_ValCont
[ https://issues.apache.org/jira/browse/BEAM-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018452#comment-17018452 ] Boyuan Zhang commented on BEAM-9137: I'll drop it from release blocker since it doesn't make sense for now. > PR10338 breaks beam_PostCommit_Py_ValCont > - > > Key: BEAM-9137 > URL: https://issues.apache.org/jira/browse/BEAM-9137 > Project: Beam > Issue Type: Bug > Components: test-failures >Affects Versions: 2.19.0 >Reporter: Boyuan Zhang >Priority: Blocker > Fix For: 2.19.0 > > > For the first failure, please refer to > https://builds.apache.org/job/beam_PostCommit_Py_ValCont/5172/#showFailuresLink -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9137) beam_PostCommit_Py_ValCont should run with dataflow_worker_jar
[ https://issues.apache.org/jira/browse/BEAM-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang updated BEAM-9137: --- Summary: beam_PostCommit_Py_ValCont should run with dataflow_worker_jar (was: PR10338 breaks beam_PostCommit_Py_ValCont) > beam_PostCommit_Py_ValCont should run with dataflow_worker_jar > -- > > Key: BEAM-9137 > URL: https://issues.apache.org/jira/browse/BEAM-9137 > Project: Beam > Issue Type: Bug > Components: test-failures >Affects Versions: 2.19.0 >Reporter: Boyuan Zhang >Priority: Blocker > Fix For: 2.19.0 > > > For the first failure, please refer to > https://builds.apache.org/job/beam_PostCommit_Py_ValCont/5172/#showFailuresLink -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9137) PR10338 breaks beam_PostCommit_Py_ValCont
[ https://issues.apache.org/jira/browse/BEAM-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018451#comment-17018451 ] Boyuan Zhang commented on BEAM-9137: Turns out the root cause should be the python ValidationContainer tests are kind of mis-configured. The test is running with dataflow runner, but not provides a dataflow_worker_jar option, which forces dataflow service pulls a pre-built runner image. So the mismatch between sdk(built with latest code) and runner(maybe last release ver) will break this test. > PR10338 breaks beam_PostCommit_Py_ValCont > - > > Key: BEAM-9137 > URL: https://issues.apache.org/jira/browse/BEAM-9137 > Project: Beam > Issue Type: Bug > Components: test-failures >Affects Versions: 2.19.0 >Reporter: Boyuan Zhang >Priority: Blocker > Fix For: 2.19.0 > > > For the first failure, please refer to > https://builds.apache.org/job/beam_PostCommit_Py_ValCont/5172/#showFailuresLink -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7273) :beam-sdks-java-io-hadoop-format:test failing on release-2.13.0
[ https://issues.apache.org/jira/browse/BEAM-7273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018438#comment-17018438 ] Udi Meiri commented on BEAM-7273: - I got this as well. Could you explain why this was resolved as not a problem? > :beam-sdks-java-io-hadoop-format:test failing on release-2.13.0 > --- > > Key: BEAM-7273 > URL: https://issues.apache.org/jira/browse/BEAM-7273 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Ankur Goenka >Priority: Major > Fix For: Not applicable > > > :beam-sdks-java-io-hadoop-format:test is crashing JVM on release-2.13.0 and > master. > > Build scan [https://scans.gradle.com/s/5uhhirrcpgtjw/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8684) Beam Dependency Update Request: com.google.apis:google-api-services-bigquery
[ https://issues.apache.org/jira/browse/BEAM-8684?focusedWorklogId=373984=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373984 ] ASF GitHub Bot logged work on BEAM-8684: Author: ASF GitHub Bot Created on: 18/Jan/20 00:42 Start Date: 18/Jan/20 00:42 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10631: [BEAM-8684] Google proto and google-services library upgrades URL: https://github.com/apache/beam/pull/10631#issuecomment-575846752 Run Spark ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373984) Time Spent: 2h 10m (was: 2h) > Beam Dependency Update Request: com.google.apis:google-api-services-bigquery > > > Key: BEAM-8684 > URL: https://issues.apache.org/jira/browse/BEAM-8684 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > - 2019-11-15 19:39:07.113511 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:04:31.460554 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:09:35.656962 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:08:45.501496 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:08:46.385699 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:04:15.902178 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:07:51.708019 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:07:45.384750 > - > Please consider upgrading
[jira] [Work logged] (BEAM-8684) Beam Dependency Update Request: com.google.apis:google-api-services-bigquery
[ https://issues.apache.org/jira/browse/BEAM-8684?focusedWorklogId=373985=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373985 ] ASF GitHub Bot logged work on BEAM-8684: Author: ASF GitHub Bot Created on: 18/Jan/20 00:42 Start Date: 18/Jan/20 00:42 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10631: [BEAM-8684] Google proto and google-services library upgrades URL: https://github.com/apache/beam/pull/10631#issuecomment-575846775 Run SQL Postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373985) Time Spent: 2h 20m (was: 2h 10m) > Beam Dependency Update Request: com.google.apis:google-api-services-bigquery > > > Key: BEAM-8684 > URL: https://issues.apache.org/jira/browse/BEAM-8684 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > - 2019-11-15 19:39:07.113511 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:04:31.460554 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:09:35.656962 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:08:45.501496 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:08:46.385699 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:04:15.902178 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:07:51.708019 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:07:45.384750 > - > Please consider upgrading the
[jira] [Work logged] (BEAM-7861) Make it easy to change between multi-process and multi-thread mode for Python Direct runners
[ https://issues.apache.org/jira/browse/BEAM-7861?focusedWorklogId=373978=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373978 ] ASF GitHub Bot logged work on BEAM-7861: Author: ASF GitHub Bot Created on: 18/Jan/20 00:41 Start Date: 18/Jan/20 00:41 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #10616: [BEAM-7861] update documentation about --direct_running_mode option with direct runner. URL: https://github.com/apache/beam/pull/10616#issuecomment-575846603 In addition, it's better to have a kind of `switch-version` class which is similar to `switch-language` class, so users can navigate different versions easily. Are there something like this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373978) Time Spent: 3h 50m (was: 3h 40m) > Make it easy to change between multi-process and multi-thread mode for Python > Direct runners > > > Key: BEAM-7861 > URL: https://issues.apache.org/jira/browse/BEAM-7861 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.19.0 > > Time Spent: 3h 50m > Remaining Estimate: 0h > > BEAM-3645 makes it possible to run a map task parallel. > However, users need to change runner when switch between multithreading and > multiprocessing mode. > We want to add a flag (ex: --use-multiprocess) to make the switch easy > without changing the runner each time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7861) Make it easy to change between multi-process and multi-thread mode for Python Direct runners
[ https://issues.apache.org/jira/browse/BEAM-7861?focusedWorklogId=373980=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373980 ] ASF GitHub Bot logged work on BEAM-7861: Author: ASF GitHub Bot Created on: 18/Jan/20 00:41 Start Date: 18/Jan/20 00:41 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #10616: [BEAM-7861] update documentation about --direct_running_mode option with direct runner. URL: https://github.com/apache/beam/pull/10616#issuecomment-575846663 A fix was uploaded, PTAL. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373980) Time Spent: 4h (was: 3h 50m) > Make it easy to change between multi-process and multi-thread mode for Python > Direct runners > > > Key: BEAM-7861 > URL: https://issues.apache.org/jira/browse/BEAM-7861 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.19.0 > > Time Spent: 4h > Remaining Estimate: 0h > > BEAM-3645 makes it possible to run a map task parallel. > However, users need to change runner when switch between multithreading and > multiprocessing mode. > We want to add a flag (ex: --use-multiprocess) to make the switch easy > without changing the runner each time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8684) Beam Dependency Update Request: com.google.apis:google-api-services-bigquery
[ https://issues.apache.org/jira/browse/BEAM-8684?focusedWorklogId=373981=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373981 ] ASF GitHub Bot logged work on BEAM-8684: Author: ASF GitHub Bot Created on: 18/Jan/20 00:41 Start Date: 18/Jan/20 00:41 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10631: [BEAM-8684] Google proto and google-services library upgrades URL: https://github.com/apache/beam/pull/10631#issuecomment-575846676 Run Java HadoopFormatIO Performance Test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373981) Time Spent: 1h 40m (was: 1.5h) > Beam Dependency Update Request: com.google.apis:google-api-services-bigquery > > > Key: BEAM-8684 > URL: https://issues.apache.org/jira/browse/BEAM-8684 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > - 2019-11-15 19:39:07.113511 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:04:31.460554 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:09:35.656962 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:08:45.501496 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:08:46.385699 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:04:15.902178 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:07:51.708019 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:07:45.384750 > - > Please
[jira] [Work logged] (BEAM-8684) Beam Dependency Update Request: com.google.apis:google-api-services-bigquery
[ https://issues.apache.org/jira/browse/BEAM-8684?focusedWorklogId=373983=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373983 ] ASF GitHub Bot logged work on BEAM-8684: Author: ASF GitHub Bot Created on: 18/Jan/20 00:41 Start Date: 18/Jan/20 00:41 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10631: [BEAM-8684] Google proto and google-services library upgrades URL: https://github.com/apache/beam/pull/10631#issuecomment-575846737 Run Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373983) Time Spent: 2h (was: 1h 50m) > Beam Dependency Update Request: com.google.apis:google-api-services-bigquery > > > Key: BEAM-8684 > URL: https://issues.apache.org/jira/browse/BEAM-8684 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > - 2019-11-15 19:39:07.113511 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:04:31.460554 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:09:35.656962 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:08:45.501496 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:08:46.385699 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:04:15.902178 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:07:51.708019 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:07:45.384750 > - > Please consider upgrading
[jira] [Work logged] (BEAM-8684) Beam Dependency Update Request: com.google.apis:google-api-services-bigquery
[ https://issues.apache.org/jira/browse/BEAM-8684?focusedWorklogId=373979=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373979 ] ASF GitHub Bot logged work on BEAM-8684: Author: ASF GitHub Bot Created on: 18/Jan/20 00:41 Start Date: 18/Jan/20 00:41 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10631: [BEAM-8684] Google proto and google-services library upgrades URL: https://github.com/apache/beam/pull/10631#issuecomment-575846664 Run Java PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373979) Time Spent: 1.5h (was: 1h 20m) > Beam Dependency Update Request: com.google.apis:google-api-services-bigquery > > > Key: BEAM-8684 > URL: https://issues.apache.org/jira/browse/BEAM-8684 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > - 2019-11-15 19:39:07.113511 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:04:31.460554 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:09:35.656962 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:08:45.501496 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:08:46.385699 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:04:15.902178 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:07:51.708019 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:07:45.384750 > - > Please consider upgrading the
[jira] [Work logged] (BEAM-8684) Beam Dependency Update Request: com.google.apis:google-api-services-bigquery
[ https://issues.apache.org/jira/browse/BEAM-8684?focusedWorklogId=373982=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373982 ] ASF GitHub Bot logged work on BEAM-8684: Author: ASF GitHub Bot Created on: 18/Jan/20 00:41 Start Date: 18/Jan/20 00:41 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10631: [BEAM-8684] Google proto and google-services library upgrades URL: https://github.com/apache/beam/pull/10631#issuecomment-575846711 Run BigQueryIO Streaming Performance Test Java This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373982) Time Spent: 1h 50m (was: 1h 40m) > Beam Dependency Update Request: com.google.apis:google-api-services-bigquery > > > Key: BEAM-8684 > URL: https://issues.apache.org/jira/browse/BEAM-8684 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > - 2019-11-15 19:39:07.113511 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:04:31.460554 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:09:35.656962 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:08:45.501496 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:08:46.385699 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:04:15.902178 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:07:51.708019 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:07:45.384750 > - >
[jira] [Work logged] (BEAM-9151) Dataflow legacy worker tests are mis-configured
[ https://issues.apache.org/jira/browse/BEAM-9151?focusedWorklogId=373977=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373977 ] ASF GitHub Bot logged work on BEAM-9151: Author: ASF GitHub Bot Created on: 18/Jan/20 00:39 Start Date: 18/Jan/20 00:39 Worklog Time Spent: 10m Work Description: boyuanzz commented on issue #10635: [BEAM-9151] Fix misconfigured legacy dataflow tests. URL: https://github.com/apache/beam/pull/10635#issuecomment-575846344 Run Java_Examples_Dataflow PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373977) Time Spent: 20m (was: 10m) > Dataflow legacy worker tests are mis-configured > --- > > Key: BEAM-9151 > URL: https://issues.apache.org/jira/browse/BEAM-9151 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: Boyuan Zhang >Assignee: Boyuan Zhang >Priority: Major > Fix For: 2.19.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Please refer to the last comment of https://github.com/apache/beam/pull/8183 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7861) Make it easy to change between multi-process and multi-thread mode for Python Direct runners
[ https://issues.apache.org/jira/browse/BEAM-7861?focusedWorklogId=373976=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373976 ] ASF GitHub Bot logged work on BEAM-7861: Author: ASF GitHub Bot Created on: 18/Jan/20 00:38 Start Date: 18/Jan/20 00:38 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #10616: [BEAM-7861] update documentation about --direct_running_mode option with direct runner. URL: https://github.com/apache/beam/pull/10616#issuecomment-575846174 The `Execution Mode` section should be visible only for Python SDK. I tried to hide it with Java SDK, but didn't find a good way to do it while keeping HTML styles. Are there any recommended way? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373976) Time Spent: 3h 40m (was: 3.5h) > Make it easy to change between multi-process and multi-thread mode for Python > Direct runners > > > Key: BEAM-7861 > URL: https://issues.apache.org/jira/browse/BEAM-7861 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.19.0 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > BEAM-3645 makes it possible to run a map task parallel. > However, users need to change runner when switch between multithreading and > multiprocessing mode. > We want to add a flag (ex: --use-multiprocess) to make the switch easy > without changing the runner each time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8933) BigQuery IO should support read/write in Arrow format
[ https://issues.apache.org/jira/browse/BEAM-8933?focusedWorklogId=373975=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373975 ] ASF GitHub Bot logged work on BEAM-8933: Author: ASF GitHub Bot Created on: 18/Jan/20 00:34 Start Date: 18/Jan/20 00:34 Worklog Time Spent: 10m Work Description: 11moon11 commented on issue #10369: [BEAM-8933] BigQueryIO Arrow for read URL: https://github.com/apache/beam/pull/10369#issuecomment-57683 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373975) Time Spent: 8.5h (was: 8h 20m) > BigQuery IO should support read/write in Arrow format > - > > Key: BEAM-8933 > URL: https://issues.apache.org/jira/browse/BEAM-8933 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Time Spent: 8.5h > Remaining Estimate: 0h > > As of right now BigQuery uses Avro format for reading and writing. > We should add a config to BigQueryIO to specify which format to use: Arrow or > Avro (with Avro as default). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8684) Beam Dependency Update Request: com.google.apis:google-api-services-bigquery
[ https://issues.apache.org/jira/browse/BEAM-8684?focusedWorklogId=373974=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373974 ] ASF GitHub Bot logged work on BEAM-8684: Author: ASF GitHub Bot Created on: 18/Jan/20 00:34 Start Date: 18/Jan/20 00:34 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10631: [BEAM-8684] Google proto and google-services library upgrades URL: https://github.com/apache/beam/pull/10631#issuecomment-575845625 @aaltay Your email client might hide the following 6 commands in my email (my Gmail did). Would you run them? Run Java PostCommit Run Java HadoopFormatIO Performance Test Run BigQueryIO Streaming Performance Test Java Run Dataflow ValidatesRunner Run Spark ValidatesRunner Run SQL Postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373974) Time Spent: 1h 20m (was: 1h 10m) > Beam Dependency Update Request: com.google.apis:google-api-services-bigquery > > > Key: BEAM-8684 > URL: https://issues.apache.org/jira/browse/BEAM-8684 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > - 2019-11-15 19:39:07.113511 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:04:31.460554 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:09:35.656962 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:08:45.501496 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:08:46.385699 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:04:15.902178 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:07:51.708019 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc:
[jira] [Work logged] (BEAM-8684) Beam Dependency Update Request: com.google.apis:google-api-services-bigquery
[ https://issues.apache.org/jira/browse/BEAM-8684?focusedWorklogId=373973=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373973 ] ASF GitHub Bot logged work on BEAM-8684: Author: ASF GitHub Bot Created on: 18/Jan/20 00:31 Start Date: 18/Jan/20 00:31 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10631: [BEAM-8684] Google proto and google-services library upgrades URL: https://github.com/apache/beam/pull/10631#issuecomment-575845062 Run JavaPortabilityApi PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373973) Time Spent: 1h 10m (was: 1h) > Beam Dependency Update Request: com.google.apis:google-api-services-bigquery > > > Key: BEAM-8684 > URL: https://issues.apache.org/jira/browse/BEAM-8684 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > - 2019-11-15 19:39:07.113511 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:04:31.460554 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:09:35.656962 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:08:45.501496 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:08:46.385699 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:04:15.902178 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:07:51.708019 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:07:45.384750 > - > Please consider
[jira] [Work logged] (BEAM-6936) Add a Jenkins job running Java examples on Java 11 Dataflow
[
https://issues.apache.org/jira/browse/BEAM-6936?focusedWorklogId=373966=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373966
]
ASF GitHub Bot logged work on BEAM-6936:
Author: ASF GitHub Bot
Created on: 18/Jan/20 00:15
Start Date: 18/Jan/20 00:15
Worklog Time Spent: 10m
Work Description: boyuanzz commented on pull request #8183: [BEAM-6936]
Added Jenkins jobs running Java examples on Dataflow with Java 11
URL: https://github.com/apache/beam/pull/8183#discussion_r368185902
##
File path: runners/google-cloud-dataflow-java/examples/build.gradle
##
@@ -42,57 +42,59 @@ def dockerImageName =
project(':beam-runners-google-cloud-dataflow-java').ext.do
// If -PuseExecutableStage is set, the use_executable_stage_bundle_execution
wil be enabled.
def fnapiExperiments = project.hasProperty('useExecutableStage') ?
'beam_fn_api,use_executable_stage_bundle_execution' : "beam_fn_api"
+def commonConfig = { dataflowWorkerJar, workerHarnessContainerImage = '',
additionalOptions = [] ->
+ // return the preevaluated configuration closure
+ return {
+ testClassesDirs =
files(project(":beam-examples-java").sourceSets.test.output.classesDirs)
+ include "**/WordCountIT.class"
+ include "**/WindowedWordCountIT.class"
+ forkEvery 1
+ maxParallelForks 4
+
+ def preCommitBeamTestPipelineOptions = [
+ "--project=${gcpProject}",
+ "--tempRoot=${gcsTempRoot}",
+ "--runner=TestDataflowRunner",
+ "--dataflowWorkerJar=${dataflowWorkerJar}",
+ workerHarnessContainerImage.isEmpty()
?'':"--workerHarnessContainerImage=${workerHarnessContainerImage}"
Review comment:
Fix here: https://github.com/apache/beam/pull/10635
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373966)
Time Spent: 7h 20m (was: 7h 10m)
> Add a Jenkins job running Java examples on Java 11 Dataflow
> ---
>
> Key: BEAM-6936
> URL: https://issues.apache.org/jira/browse/BEAM-6936
> Project: Beam
> Issue Type: Sub-task
> Components: examples-java, testing
>Reporter: Michal Walenia
>Assignee: Michal Walenia
>Priority: Minor
> Fix For: 2.14.0
>
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=373965=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373965 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 18/Jan/20 00:15 Start Date: 18/Jan/20 00:15 Worklog Time Spent: 10m Work Description: udim commented on pull request #10594: [BEAM-7746] Stop automatically creating staticmethods in register_urn URL: https://github.com/apache/beam/pull/10594#discussion_r368185763 ## File path: sdks/python/apache_beam/coders/coders.py ## @@ -1014,6 +1015,7 @@ def to_runner_api_parameter(self, context): else: return super(TupleCoder, self).to_runner_api_parameter(context) + @staticmethod @Coder.register_urn(common_urns.coders.KV.urn, None) def from_runner_api_parameter(unused_payload, components, unused_context): return TupleCoder(components) Review comment: Should `ParamWindowedValueCoder.from_runner_api_parameter` also get the `@staticmethod` decorator? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373965) Time Spent: 53h 50m (was: 53h 40m) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 53h 50m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9072) [SQL] Add support for Datastore source
[ https://issues.apache.org/jira/browse/BEAM-9072?focusedWorklogId=373964=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373964 ] ASF GitHub Bot logged work on BEAM-9072: Author: ASF GitHub Bot Created on: 18/Jan/20 00:11 Start Date: 18/Jan/20 00:11 Worklog Time Spent: 10m Work Description: apilloud commented on issue #10440: [BEAM-9072] [SQL] DataStoreV1 IO connector URL: https://github.com/apache/beam/pull/10440#issuecomment-575842055 Run JavaPortabilityApi PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373964) Time Spent: 4h 50m (was: 4h 40m) > [SQL] Add support for Datastore source > -- > > Key: BEAM-9072 > URL: https://issues.apache.org/jira/browse/BEAM-9072 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Time Spent: 4h 50m > Remaining Estimate: 0h > > * Create a Datastore table and table provider > * Conversion between Datastore and Beam data types > * Implement buildIOReader > * Implement buildIOWrite > * Implement getTableStatistics > Doc: > [https://docs.google.com/document/d/1FxuEGewJ3GPDl0IKglfOYf1edwa2m_wryFZYRMpRNbA/edit?pli=1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9061) New release of pyhamcrest==1.10.0 breaks portable Python precommits.
[
https://issues.apache.org/jira/browse/BEAM-9061?focusedWorklogId=373963=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373963
]
ASF GitHub Bot logged work on BEAM-9061:
Author: ASF GitHub Bot
Created on: 18/Jan/20 00:11
Start Date: 18/Jan/20 00:11
Worklog Time Spent: 10m
Work Description: tvalentyn commented on issue #10568: [BEAM-9061] Add
version guards to requirements file for integration tests.
URL: https://github.com/apache/beam/pull/10568#issuecomment-575842037
1.10.1 does, 2.0.0 does not.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373963)
Time Spent: 3h 10m (was: 3h)
> New release of pyhamcrest==1.10.0 breaks portable Python precommits.
>
>
> Key: BEAM-9061
> URL: https://issues.apache.org/jira/browse/BEAM-9061
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> {noformat}
> 08:35:26 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 38, in
> 08:35:26 import hamcrest # pylint: disable=ungrouped-imports
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/__init__.py",
> line 2, in
> 08:35:26 from hamcrest.library import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/__init__.py",
> line 7, in
> 08:35:26 from hamcrest.library.object import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/__init__.py",
> line 4, in
> 08:35:26 from .hasproperty import has_properties, has_property
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/hasproperty.py",
> line 174
> 08:35:26 ),
> 08:35:26 ^
> {noformat}
> rootcause: https://github.com/hamcrest/PyHamcrest/issues/131
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9061) New release of pyhamcrest==1.10.0 breaks portable Python precommits.
[
https://issues.apache.org/jira/browse/BEAM-9061?focusedWorklogId=373961=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373961
]
ASF GitHub Bot logged work on BEAM-9061:
Author: ASF GitHub Bot
Created on: 18/Jan/20 00:10
Start Date: 18/Jan/20 00:10
Worklog Time Spent: 10m
Work Description: aaltay commented on issue #10568: [BEAM-9061] Add
version guards to requirements file for integration tests.
URL: https://github.com/apache/beam/pull/10568#issuecomment-575841800
Does pyhamcrest > 1.10.0 work with python 2?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373961)
Time Spent: 3h (was: 2h 50m)
> New release of pyhamcrest==1.10.0 breaks portable Python precommits.
>
>
> Key: BEAM-9061
> URL: https://issues.apache.org/jira/browse/BEAM-9061
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> {noformat}
> 08:35:26 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 38, in
> 08:35:26 import hamcrest # pylint: disable=ungrouped-imports
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/__init__.py",
> line 2, in
> 08:35:26 from hamcrest.library import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/__init__.py",
> line 7, in
> 08:35:26 from hamcrest.library.object import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/__init__.py",
> line 4, in
> 08:35:26 from .hasproperty import has_properties, has_property
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/hasproperty.py",
> line 174
> 08:35:26 ),
> 08:35:26 ^
> {noformat}
> rootcause: https://github.com/hamcrest/PyHamcrest/issues/131
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9151) Dataflow legacy worker tests are mis-configured
[ https://issues.apache.org/jira/browse/BEAM-9151?focusedWorklogId=373962=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373962 ] ASF GitHub Bot logged work on BEAM-9151: Author: ASF GitHub Bot Created on: 18/Jan/20 00:10 Start Date: 18/Jan/20 00:10 Worklog Time Spent: 10m Work Description: boyuanzz commented on pull request #10635: [BEAM-9151] Fix misconfigured legacy dataflow tests. URL: https://github.com/apache/beam/pull/10635 R: @lukecwik When `dataflowWorkerJar` is provided, the `workerHarnessContainerImage` should be set to empty explicitly, otherwise, the dataflow service will ignore worker jar and still pick harness container This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373962) Remaining Estimate: 0h Time Spent: 10m > Dataflow legacy worker tests are mis-configured > --- > > Key: BEAM-9151 > URL: https://issues.apache.org/jira/browse/BEAM-9151 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: Boyuan Zhang >Assignee: Boyuan Zhang >Priority: Major > Fix For: 2.19.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Please refer to the last comment of https://github.com/apache/beam/pull/8183 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9151) Dataflow legacy worker tests are mis-configured
Boyuan Zhang created BEAM-9151: -- Summary: Dataflow legacy worker tests are mis-configured Key: BEAM-9151 URL: https://issues.apache.org/jira/browse/BEAM-9151 Project: Beam Issue Type: Bug Components: testing Reporter: Boyuan Zhang Assignee: Boyuan Zhang Fix For: 2.19.0 Please refer to the last comment of https://github.com/apache/beam/pull/8183 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9061) New release of pyhamcrest==1.10.0 breaks portable Python precommits.
[
https://issues.apache.org/jira/browse/BEAM-9061?focusedWorklogId=373957=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373957
]
ASF GitHub Bot logged work on BEAM-9061:
Author: ASF GitHub Bot
Created on: 17/Jan/20 23:56
Start Date: 17/Jan/20 23:56
Worklog Time Spent: 10m
Work Description: tvalentyn commented on issue #10619: [BEAM-9061] Add
version guards to requirements file for integration tests.
URL: https://github.com/apache/beam/pull/10619#issuecomment-575839089
Precommits are failing because containers for DF workers are not created
yet, so a couple of integration tests that we have, fail.
Given that this commit is a cherry-pick from master and only affects test
dependencies, this should be safe to merge.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373957)
Time Spent: 2h 50m (was: 2h 40m)
> New release of pyhamcrest==1.10.0 breaks portable Python precommits.
>
>
> Key: BEAM-9061
> URL: https://issues.apache.org/jira/browse/BEAM-9061
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> {noformat}
> 08:35:26 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 38, in
> 08:35:26 import hamcrest # pylint: disable=ungrouped-imports
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/__init__.py",
> line 2, in
> 08:35:26 from hamcrest.library import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/__init__.py",
> line 7, in
> 08:35:26 from hamcrest.library.object import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/__init__.py",
> line 4, in
> 08:35:26 from .hasproperty import has_properties, has_property
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/hasproperty.py",
> line 174
> 08:35:26 ),
> 08:35:26 ^
> {noformat}
> rootcause: https://github.com/hamcrest/PyHamcrest/issues/131
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9072) [SQL] Add support for Datastore source
[ https://issues.apache.org/jira/browse/BEAM-9072?focusedWorklogId=373954=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373954 ] ASF GitHub Bot logged work on BEAM-9072: Author: ASF GitHub Bot Created on: 17/Jan/20 23:54 Start Date: 17/Jan/20 23:54 Worklog Time Spent: 10m Work Description: apilloud commented on issue #10440: [BEAM-9072] [SQL] DataStoreV1 IO connector URL: https://github.com/apache/beam/pull/10440#issuecomment-575838697 Run JavaPortabilityApi PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373954) Time Spent: 4h 40m (was: 4.5h) > [SQL] Add support for Datastore source > -- > > Key: BEAM-9072 > URL: https://issues.apache.org/jira/browse/BEAM-9072 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Time Spent: 4h 40m > Remaining Estimate: 0h > > * Create a Datastore table and table provider > * Conversion between Datastore and Beam data types > * Implement buildIOReader > * Implement buildIOWrite > * Implement getTableStatistics > Doc: > [https://docs.google.com/document/d/1FxuEGewJ3GPDl0IKglfOYf1edwa2m_wryFZYRMpRNbA/edit?pli=1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=373953=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373953 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 17/Jan/20 23:53 Start Date: 17/Jan/20 23:53 Worklog Time Spent: 10m Work Description: udim commented on pull request #10590: [BEAM-7746] Fix a typing issue where SourceBase was assumed to have a coder attribute URL: https://github.com/apache/beam/pull/10590 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373953) Time Spent: 53h 40m (was: 53.5h) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 53h 40m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9072) [SQL] Add support for Datastore source
[ https://issues.apache.org/jira/browse/BEAM-9072?focusedWorklogId=373952=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373952 ] ASF GitHub Bot logged work on BEAM-9072: Author: ASF GitHub Bot Created on: 17/Jan/20 23:49 Start Date: 17/Jan/20 23:49 Worklog Time Spent: 10m Work Description: 11moon11 commented on issue #10440: [BEAM-9072] [SQL] DataStoreV1 IO connector URL: https://github.com/apache/beam/pull/10440#issuecomment-575837709 JavaPortabilityApi failed with: ``` Execution failed for task ':runners:google-cloud-dataflow-java:buildAndPushDockerContainer'. Process 'command 'docker'' finished with non-zero exit value 1 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373952) Time Spent: 4.5h (was: 4h 20m) > [SQL] Add support for Datastore source > -- > > Key: BEAM-9072 > URL: https://issues.apache.org/jira/browse/BEAM-9072 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Time Spent: 4.5h > Remaining Estimate: 0h > > * Create a Datastore table and table provider > * Conversion between Datastore and Beam data types > * Implement buildIOReader > * Implement buildIOWrite > * Implement getTableStatistics > Doc: > [https://docs.google.com/document/d/1FxuEGewJ3GPDl0IKglfOYf1edwa2m_wryFZYRMpRNbA/edit?pli=1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8684) Beam Dependency Update Request: com.google.apis:google-api-services-bigquery
[ https://issues.apache.org/jira/browse/BEAM-8684?focusedWorklogId=373946=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373946 ] ASF GitHub Bot logged work on BEAM-8684: Author: ASF GitHub Bot Created on: 17/Jan/20 23:32 Start Date: 17/Jan/20 23:32 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10631: [BEAM-8684] Google proto and google-services library upgrades URL: https://github.com/apache/beam/pull/10631#issuecomment-575834545 Java Portability API precommit failed: ``` 17:51:37 Execution failed for task ':runners:google-cloud-dataflow-java:buildAndPushDockerContainer'. 17:51:37 > Process 'command 'docker'' finished with non-zero exit value 1 ``` This should be a transient error. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373946) Time Spent: 1h (was: 50m) > Beam Dependency Update Request: com.google.apis:google-api-services-bigquery > > > Key: BEAM-8684 > URL: https://issues.apache.org/jira/browse/BEAM-8684 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > - 2019-11-15 19:39:07.113511 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:04:31.460554 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:09:35.656962 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:08:45.501496 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:08:46.385699 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:04:15.902178 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:07:51.708019 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency
[jira] [Commented] (BEAM-9144) Beam's own Avro TimeConversion class in beam-sdk-java-core
[
https://issues.apache.org/jira/browse/BEAM-9144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018400#comment-17018400
]
Aaron Dixon commented on BEAM-9144:
---
[~iemejia] Thanks, I'd like to help test. I'm relatively new to Beam and very
new to its dev/build etc. Will it be sufficient for me to pull 2.20.0-SNAPSHOTS
from apache snapshots repo
([https://repository.apache.org/content/repositories/snapshots/])? And I assume
I'll need to wait til tomorrow so that this work is incorporated in a nightly?
If so, this weekend I can run my Dataflow pipeline against 2.20.0-SNAPHOTS from
apache/snapshots repo and report back that it is happy. FYI/fwiw, my specific
Beam dependencies are these, so I'll be testing against these 2.20.0-SNAPSHOTS:
{code:java}
[org.apache.beam/beam-sdks-java-core]
[org.apache.beam/beam-sdks-java-io-kafka]
[org.apache.beam/beam-runners-google-cloud-dataflow-java]{code}
> Beam's own Avro TimeConversion class in beam-sdk-java-core
> ---
>
> Key: BEAM-9144
> URL: https://issues.apache.org/jira/browse/BEAM-9144
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Fix For: 2.19.0
>
> Attachments: avro-beam-dependency-graph.png
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> From Aaron's comment in
> https://issues.apache.org/jira/browse/BEAM-8388?focusedCommentId=17016476=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17016476
> .
> {quote}My org must use Avro 1.9.x (due to some Avro schema resolution issues
> resolved in 1.9.x) so downgrading Avro is not possible for us.
> Beam 2.16.0 is compatible with our usage of Avro 1.9.x – but upgrading to
> 2.17.0 we are broken as 2.17.0 links to Java classes in Avro 1.8.x that are
> not available in 1.9.x.
> {quote}
> The Java class is
> {{org.apache.avro.data.TimeConversions.TimestampConversion}} in Avro 1.8.
> It's renamed to {{org.apache.avro.data.JodaTimeConversions}} in Avro 1.9.
> h1. Beam Java SDK cannot upgrade Avro to 1.9
> Beam has Spark runners and Spark has not yet upgraded to Avro 1.9.
> Illustration of the dependency
> !avro-beam-dependency-graph.png|width=799,height=385!
> h1. Short-term Solution
> As illustrated above, as long as Beam Java SDK uses only the intersection of
> Avro classes, method, and fields between Avro 1.8 and 1.9, it will provide
> flexibility in runtime Avro versions (as it did until Beam 2.16).
> h2. Difference of the TimeConversion Classes
> Avro 1.9's TimestampConversion overrides {{getRecommendedSchema}} method.
> Details below:
> Avro 1.8's TimeConversions.TimestampConversion:
> {code:java}
> public static class TimestampConversion extends Conversion {
> @Override
> public Class getConvertedType() {
> return DateTime.class;
> }
> @Override
> public String getLogicalTypeName() {
> return "timestamp-millis";
> }
> @Override
> public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType
> type) {
> return new DateTime(millisFromEpoch, DateTimeZone.UTC);
> }
> @Override
> public Long toLong(DateTime timestamp, Schema schema, LogicalType type) {
> return timestamp.getMillis();
> }
> }
> {code}
> Avro 1.9's JodaTimeConversions.TimestampConversion:
> {code:java}
> public static class TimestampConversion extends Conversion {
> @Override
> public Class getConvertedType() {
> return DateTime.class;
> }
> @Override
> public String getLogicalTypeName() {
> return "timestamp-millis";
> }
> @Override
> public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType
> type) {
> return new DateTime(millisFromEpoch, DateTimeZone.UTC);
> }
> @Override
> public Long toLong(DateTime timestamp, Schema schema, LogicalType type) {
> return timestamp.getMillis();
> }
> @Override
> public Schema getRecommendedSchema() {
> return
> LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG));
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9072) [SQL] Add support for Datastore source
[ https://issues.apache.org/jira/browse/BEAM-9072?focusedWorklogId=373942=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373942 ] ASF GitHub Bot logged work on BEAM-9072: Author: ASF GitHub Bot Created on: 17/Jan/20 23:26 Start Date: 17/Jan/20 23:26 Worklog Time Spent: 10m Work Description: apilloud commented on issue #10440: [BEAM-9072] [SQL] DataStoreV1 IO connector URL: https://github.com/apache/beam/pull/10440#issuecomment-575833263 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373942) Time Spent: 4h 20m (was: 4h 10m) > [SQL] Add support for Datastore source > -- > > Key: BEAM-9072 > URL: https://issues.apache.org/jira/browse/BEAM-9072 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Time Spent: 4h 20m > Remaining Estimate: 0h > > * Create a Datastore table and table provider > * Conversion between Datastore and Beam data types > * Implement buildIOReader > * Implement buildIOWrite > * Implement getTableStatistics > Doc: > [https://docs.google.com/document/d/1FxuEGewJ3GPDl0IKglfOYf1edwa2m_wryFZYRMpRNbA/edit?pli=1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite
[
https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=373939=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373939
]
ASF GitHub Bot logged work on BEAM-7961:
Author: ASF GitHub Bot
Created on: 17/Jan/20 23:21
Start Date: 17/Jan/20 23:21
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10051: [BEAM-7961] Add
tests for all runner native transforms for XLang
URL: https://github.com/apache/beam/pull/10051#discussion_r368175531
##
File path:
.test-infra/jenkins/job_PostCommit_CrossLanguageValidatesRunner_Flink.groovy
##
@@ -34,10 +34,19 @@
PostcommitJobBuilder.postCommitJob('beam_PostCommit_XVR_Flink',
// Gradle goals for this job.
steps {
+shell('echo *** RUN CROSS-LANGUAGE FLINK USING PYTHON 2.7 ***')
Review comment:
Followed the same convention in
https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Spark.groovy#L36
Should we also need to modify those?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373939)
Time Spent: 15h 50m (was: 15h 40m)
> Add tests for all runner native transforms and some widely used composite
> transforms to cross-language validates runner test suite
> --
>
> Key: BEAM-7961
> URL: https://issues.apache.org/jira/browse/BEAM-7961
> Project: Beam
> Issue Type: Improvement
> Components: testing
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 15h 50m
> Remaining Estimate: 0h
>
> Add tests for all runner native transforms and some widely used composite
> transforms to cross-language validates runner test suite
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7861) Make it easy to change between multi-process and multi-thread mode for Python Direct runners
[ https://issues.apache.org/jira/browse/BEAM-7861?focusedWorklogId=373938=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373938 ] ASF GitHub Bot logged work on BEAM-7861: Author: ASF GitHub Bot Created on: 17/Jan/20 23:20 Start Date: 17/Jan/20 23:20 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #10616: [BEAM-7861] update documentation about --direct_running_mode option with direct runner. URL: https://github.com/apache/beam/pull/10616#issuecomment-575831989 cc: @soyrice This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373938) Time Spent: 3.5h (was: 3h 20m) > Make it easy to change between multi-process and multi-thread mode for Python > Direct runners > > > Key: BEAM-7861 > URL: https://issues.apache.org/jira/browse/BEAM-7861 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.19.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > BEAM-3645 makes it possible to run a map task parallel. > However, users need to change runner when switch between multithreading and > multiprocessing mode. > We want to add a flag (ex: --use-multiprocess) to make the switch easy > without changing the runner each time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5504) PubsubAvroTable
[ https://issues.apache.org/jira/browse/BEAM-5504?focusedWorklogId=373936=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373936 ] ASF GitHub Bot logged work on BEAM-5504: Author: ASF GitHub Bot Created on: 17/Jan/20 23:16 Start Date: 17/Jan/20 23:16 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #10487: [BEAM-5504] Introduce PubsubAvroTable URL: https://github.com/apache/beam/pull/10487#issuecomment-575831150 Thanks @milantracy! I will take a look! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373936) Time Spent: 2h (was: 1h 50m) > PubsubAvroTable > --- > > Key: BEAM-5504 > URL: https://issues.apache.org/jira/browse/BEAM-5504 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Rui Wang >Assignee: Jing Chen >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-6936) Add a Jenkins job running Java examples on Java 11 Dataflow
[
https://issues.apache.org/jira/browse/BEAM-6936?focusedWorklogId=373935=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373935
]
ASF GitHub Bot logged work on BEAM-6936:
Author: ASF GitHub Bot
Created on: 17/Jan/20 23:16
Start Date: 17/Jan/20 23:16
Worklog Time Spent: 10m
Work Description: boyuanzz commented on pull request #8183: [BEAM-6936]
Added Jenkins jobs running Java examples on Dataflow with Java 11
URL: https://github.com/apache/beam/pull/8183#discussion_r368174524
##
File path: runners/google-cloud-dataflow-java/examples/build.gradle
##
@@ -42,57 +42,59 @@ def dockerImageName =
project(':beam-runners-google-cloud-dataflow-java').ext.do
// If -PuseExecutableStage is set, the use_executable_stage_bundle_execution
wil be enabled.
def fnapiExperiments = project.hasProperty('useExecutableStage') ?
'beam_fn_api,use_executable_stage_bundle_execution' : "beam_fn_api"
+def commonConfig = { dataflowWorkerJar, workerHarnessContainerImage = '',
additionalOptions = [] ->
+ // return the preevaluated configuration closure
+ return {
+ testClassesDirs =
files(project(":beam-examples-java").sourceSets.test.output.classesDirs)
+ include "**/WordCountIT.class"
+ include "**/WindowedWordCountIT.class"
+ forkEvery 1
+ maxParallelForks 4
+
+ def preCommitBeamTestPipelineOptions = [
+ "--project=${gcpProject}",
+ "--tempRoot=${gcsTempRoot}",
+ "--runner=TestDataflowRunner",
+ "--dataflowWorkerJar=${dataflowWorkerJar}",
+ workerHarnessContainerImage.isEmpty()
?'':"--workerHarnessContainerImage=${workerHarnessContainerImage}"
Review comment:
When `dataflowWorkerJar` is provided, the `workerHarnessContainerImage`
should be set to empty explicitly, otherwise, the dataflow service will still
try to pull harness image rather than using custom-built worker jar,
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373935)
Time Spent: 7h 10m (was: 7h)
> Add a Jenkins job running Java examples on Java 11 Dataflow
> ---
>
> Key: BEAM-6936
> URL: https://issues.apache.org/jira/browse/BEAM-6936
> Project: Beam
> Issue Type: Sub-task
> Components: examples-java, testing
>Reporter: Michal Walenia
>Assignee: Michal Walenia
>Priority: Minor
> Fix For: 2.14.0
>
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9138) beam_Release_Gradle_Build failure in Go
[
https://issues.apache.org/jira/browse/BEAM-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018395#comment-17018395
]
Udi Meiri edited comment on BEAM-9138 at 1/17/20 11:15 PM:
---
Trying this temporary hack: commenting out Go projects in settings.gradle:
{code}
include ":sdks:go" // is a dependency of other packages
//include ":sdks:go:container"
//include ":sdks:go:examples" // goVet failed on this project
//include ":sdks:go:test"
{code}
was (Author: udim):
Trying this temporary hack: commenting out Go projects in settings.gradle:
{code}
include ":sdks:go" // is a dependency of other packages
//include ":sdks:go:container"
//include ":sdks:go:examples" // goVet failed on this project
//include ":sdks:go:test"
{codE}
> beam_Release_Gradle_Build failure in Go
> ---
>
> Key: BEAM-9138
> URL: https://issues.apache.org/jira/browse/BEAM-9138
> Project: Beam
> Issue Type: Bug
> Components: build-system, test-failures
>Reporter: Udi Meiri
>Priority: Major
>
> Phrase: "Run Release Gradle Build"
> Recent failures:
> https://builds.apache.org/job/beam_Release_Gradle_Build/24/
> https://builds.apache.org/job/beam_Release_Gradle_Build/25/
> Seems like all of these are Go-related, and it seems like the culprit might
> be the failure in :sdks:go:resolveBuildDependencies (but I have very little
> knowledge of these tasks).
> CC: [~markflyhigh] who wrote the job, [~lostluck] who knows Go SDK
> Please help in triaging and/or assigning to someone who can look at this.
> {code}
> 13:00:15 Execution failed for task ':beam-test-tools:goVet'.
> 13:00:15 > Build failed due to return code 1 of:
> 13:00:15Command:
> 13:00:15 /usr/bin/go vet github.com/apache/beam/.test-infra/tools
> {code}
> {code}
> 13:00:15 Execution failed for task ':beam-test-tools:goTest'.
> 13:00:15 > There are 1 failed tests. Please see
> /home/jenkins/jenkins-slave/workspace/beam_Release_Gradle_Build/src/.test-infra/tools/.gogradle/reports/test/index.html
> for more details.
> {code}
> {code}
> 13:00:15 Execution failed for task ':beam-test-tools:buildLinuxAmd64'.
> 13:00:15 > Build failed due to return code 1 of:
> 13:00:15Command:
> 13:00:15 /usr/bin/go build -o ./.gogradle/beam-test-tools-linux-amd64
> github.com/apache/beam/.test-infra/tools
> {code}
> {code}
> 13:00:15 Execution failed for task ':sdks:go:resolveBuildDependencies'.
> 13:00:15 > Exception in resolution, message is:
> 13:00:15Cannot resolve dependency:google.golang.org/api:
> commit='386d4e5f4f92f86e6aec85985761bba4b938a2d5',
> urls=[https://code.googlesource.com/google-api-go-client]
> 13:00:15Resolution stack is:
> 13:00:15+- github.com/apache/beam/sdks/go
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9138) beam_Release_Gradle_Build failure in Go
[
https://issues.apache.org/jira/browse/BEAM-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018395#comment-17018395
]
Udi Meiri commented on BEAM-9138:
-
Trying this temporary hack: commenting out Go projects in settings.gradle:
{code}
include ":sdks:go" // is a dependency of other packages
//include ":sdks:go:container"
//include ":sdks:go:examples" // goVet failed on this project
//include ":sdks:go:test"
{codE}
> beam_Release_Gradle_Build failure in Go
> ---
>
> Key: BEAM-9138
> URL: https://issues.apache.org/jira/browse/BEAM-9138
> Project: Beam
> Issue Type: Bug
> Components: build-system, test-failures
>Reporter: Udi Meiri
>Priority: Major
>
> Phrase: "Run Release Gradle Build"
> Recent failures:
> https://builds.apache.org/job/beam_Release_Gradle_Build/24/
> https://builds.apache.org/job/beam_Release_Gradle_Build/25/
> Seems like all of these are Go-related, and it seems like the culprit might
> be the failure in :sdks:go:resolveBuildDependencies (but I have very little
> knowledge of these tasks).
> CC: [~markflyhigh] who wrote the job, [~lostluck] who knows Go SDK
> Please help in triaging and/or assigning to someone who can look at this.
> {code}
> 13:00:15 Execution failed for task ':beam-test-tools:goVet'.
> 13:00:15 > Build failed due to return code 1 of:
> 13:00:15Command:
> 13:00:15 /usr/bin/go vet github.com/apache/beam/.test-infra/tools
> {code}
> {code}
> 13:00:15 Execution failed for task ':beam-test-tools:goTest'.
> 13:00:15 > There are 1 failed tests. Please see
> /home/jenkins/jenkins-slave/workspace/beam_Release_Gradle_Build/src/.test-infra/tools/.gogradle/reports/test/index.html
> for more details.
> {code}
> {code}
> 13:00:15 Execution failed for task ':beam-test-tools:buildLinuxAmd64'.
> 13:00:15 > Build failed due to return code 1 of:
> 13:00:15Command:
> 13:00:15 /usr/bin/go build -o ./.gogradle/beam-test-tools-linux-amd64
> github.com/apache/beam/.test-infra/tools
> {code}
> {code}
> 13:00:15 Execution failed for task ':sdks:go:resolveBuildDependencies'.
> 13:00:15 > Exception in resolution, message is:
> 13:00:15Cannot resolve dependency:google.golang.org/api:
> commit='386d4e5f4f92f86e6aec85985761bba4b938a2d5',
> urls=[https://code.googlesource.com/google-api-go-client]
> 13:00:15Resolution stack is:
> 13:00:15+- github.com/apache/beam/sdks/go
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9138) beam_Release_Gradle_Build failure in Go
[
https://issues.apache.org/jira/browse/BEAM-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018392#comment-17018392
]
Udi Meiri commented on BEAM-9138:
-
The goVet failure is actually this previous one:
https://issues.apache.org/jira/browse/BEAM-8992
> beam_Release_Gradle_Build failure in Go
> ---
>
> Key: BEAM-9138
> URL: https://issues.apache.org/jira/browse/BEAM-9138
> Project: Beam
> Issue Type: Bug
> Components: build-system, test-failures
>Reporter: Udi Meiri
>Priority: Major
>
> Phrase: "Run Release Gradle Build"
> Recent failures:
> https://builds.apache.org/job/beam_Release_Gradle_Build/24/
> https://builds.apache.org/job/beam_Release_Gradle_Build/25/
> Seems like all of these are Go-related, and it seems like the culprit might
> be the failure in :sdks:go:resolveBuildDependencies (but I have very little
> knowledge of these tasks).
> CC: [~markflyhigh] who wrote the job, [~lostluck] who knows Go SDK
> Please help in triaging and/or assigning to someone who can look at this.
> {code}
> 13:00:15 Execution failed for task ':beam-test-tools:goVet'.
> 13:00:15 > Build failed due to return code 1 of:
> 13:00:15Command:
> 13:00:15 /usr/bin/go vet github.com/apache/beam/.test-infra/tools
> {code}
> {code}
> 13:00:15 Execution failed for task ':beam-test-tools:goTest'.
> 13:00:15 > There are 1 failed tests. Please see
> /home/jenkins/jenkins-slave/workspace/beam_Release_Gradle_Build/src/.test-infra/tools/.gogradle/reports/test/index.html
> for more details.
> {code}
> {code}
> 13:00:15 Execution failed for task ':beam-test-tools:buildLinuxAmd64'.
> 13:00:15 > Build failed due to return code 1 of:
> 13:00:15Command:
> 13:00:15 /usr/bin/go build -o ./.gogradle/beam-test-tools-linux-amd64
> github.com/apache/beam/.test-infra/tools
> {code}
> {code}
> 13:00:15 Execution failed for task ':sdks:go:resolveBuildDependencies'.
> 13:00:15 > Exception in resolution, message is:
> 13:00:15Cannot resolve dependency:google.golang.org/api:
> commit='386d4e5f4f92f86e6aec85985761bba4b938a2d5',
> urls=[https://code.googlesource.com/google-api-go-client]
> 13:00:15Resolution stack is:
> 13:00:15+- github.com/apache/beam/sdks/go
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9150) beam_PostRelease_Python_Candidate fails
Udi Meiri created BEAM-9150:
---
Summary: beam_PostRelease_Python_Candidate fails
Key: BEAM-9150
URL: https://issues.apache.org/jira/browse/BEAM-9150
Project: Beam
Issue Type: Bug
Components: build-system
Reporter: Udi Meiri
Assignee: Yifan Zou
https://builds.apache.org/job/beam_PostRelease_Python_Candidate/171/timestamps/?time=HH:mm:ss=GMT-8=en_US
Some interesting lines:
{code}
17:48:36
release/src/main/python-release/run_release_candidate_python_quickstart.sh:
line 144: 20935 Killed python -m
apache_beam.examples.streaming_wordcount --input_topic
projects/$PROJECT_ID/topics/$PUBSUB_TOPIC1 --output_topic
projects/$PROJECT_ID/topics/$PUBSUB_TOPIC2 --streaming
{code}
{code}
18:05:25
18:05:25 Running HourlyTeamScore example with DirectRunner
18:05:25
18:05:25
/tmp/tmp.gnpMNIFhcL/temp_virtualenv_python2.7/local/lib/python2.7/site-packages/apache_beam/__init__.py:84:
UserWarning: You are using Apache Beam with Python 2. New releases of Apache
Beam will soon support Python 3 only.
18:05:25'You are using Apache Beam with Python 2. '
18:05:33 INFO:root:Missing pipeline option (runner). Executing pipeline using
the default runner: DirectRunner.
18:05:33 INFO:root:Setting socket default timeout to 60 seconds.
18:05:33 INFO:root:socket default timeout is 60.0 seconds.
18:05:33 INFO:oauth2client.transport:Attempting refresh to obtain initial
access_token
18:05:34
/tmp/tmp.gnpMNIFhcL/temp_virtualenv_python2.7/local/lib/python2.7/site-packages/apache_beam/io/gcp/bigquery.py:1220:
BeamDeprecationWarning: options is deprecated since First stable release.
References to .options will not be supported
18:05:34experiments = p.options.view_as(DebugOptions).experiments or []
18:05:37 INFO:root:
18:05:37 INFO:root:
18:05:37 INFO:root:
18:05:37 INFO:root:
18:05:37 INFO:root:
18:05:37 INFO:root:
18:05:37 INFO:root:
18:05:37 INFO:root:
18:05:37 INFO:root:
18:05:37 INFO:root:
18:05:37 INFO:root:
18:05:37 INFO:root:
18:05:37 INFO:root:Creating state cache with size 100
18:05:37 INFO:root:Created Worker handler
for environment urn: "beam:env:embedded_python:v1"
18:05:37
18:05:37 INFO:root:Running
((ref_AppliedPTransform_ReadInputText/Read/_SDFBoundedSourceWrapper/Impulse_5)+(ReadInputText/Read/_SDFBoundedSourceWrapper/ParDo(SDFBoundedSourceDoFn)/PairWithRestriction))+((ReadInputText/Read/_SDFBoundedSourceWrapper/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction)+(ref_PCollection_PCollection_1_split/Write))
18:05:38 INFO:root:Running
(ref_PCollection_PCollection_1_split/Read)+ReadInputText/Read/_SDFBoundedSourceWrapper/ParDo(SDFBoundedSourceDoFn)/Process)+((ref_AppliedPTransform_HourlyTeamScore/ParseGameEventFn_8)+((ref_AppliedPTransform_HourlyTeamScore/FilterStartTime_9)+((ref_AppliedPTransform_HourlyTeamScore/FilterEndTime_10)+((ref_AppliedPTransform_HourlyTeamScore/AddEventTimestamps_11)+((ref_AppliedPTransform_HourlyTeamScore/FixedWindowsTeam_12)+(ref_AppliedPTransform_HourlyTeamScore/ExtractAndSumScore/Map()_14)))+(HourlyTeamScore/ExtractAndSumScore/CombinePerKey(sum)/Precombine))+(HourlyTeamScore/ExtractAndSumScore/CombinePerKey(sum)/Group/Write))
18:05:39 INFO:root:Running
((HourlyTeamScore/ExtractAndSumScore/CombinePerKey(sum)/Group/Read)+(HourlyTeamScore/ExtractAndSumScore/CombinePerKey(sum)/Merge))+((HourlyTeamScore/ExtractAndSumScore/CombinePerKey(sum)/ExtractOutputs)+((ref_AppliedPTransform_TeamScoresDict_22)+((ref_AppliedPTransform_WriteTeamScoreSums/ConvertToRow_24)+((ref_AppliedPTransform_WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/AppendDestination_27)+((ref_AppliedPTransform_WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/AddInsertIds_28)+((ref_AppliedPTransform_WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/AddRandomKeys_30)+((ref_AppliedPTransform_WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/Map(reify_timestamps)_32)+(WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/GroupByKey/Write
18:05:39 INFO:root:Running
[jira] [Work logged] (BEAM-5086) Beam Dependency Update Request: org.apache.kudu
[ https://issues.apache.org/jira/browse/BEAM-5086?focusedWorklogId=373932=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373932 ] ASF GitHub Bot logged work on BEAM-5086: Author: ASF GitHub Bot Created on: 17/Jan/20 22:36 Start Date: 17/Jan/20 22:36 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10561: [BEAM-5086] Update kudu-client to version 1.11.1 URL: https://github.com/apache/beam/pull/10561#issuecomment-575821994 If the linkage issues are with the client, it would be good to explicitly resolve them since upgrades may cause issues even if no new errors are being reported since the new version may use different code paths. In the end its a judgement call always when errors are still being reported that can be done based upon the number of unit/integration/manual testing that has been performed vs how popular something is and how many people might be broken by the change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373932) Time Spent: 1h 20m (was: 1h 10m) > Beam Dependency Update Request: org.apache.kudu > --- > > Key: BEAM-5086 > URL: https://issues.apache.org/jira/browse/BEAM-5086 > Project: Beam > Issue Type: Bug > Components: dependencies, io-java-kudu >Reporter: Beam JIRA Bot >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > 2018-08-06 12:13:43.606551 > Please review and upgrade the org.apache.kudu to the latest version > None > > cc: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8684) Beam Dependency Update Request: com.google.apis:google-api-services-bigquery
[ https://issues.apache.org/jira/browse/BEAM-8684?focusedWorklogId=373928=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373928 ] ASF GitHub Bot logged work on BEAM-8684: Author: ASF GitHub Bot Created on: 17/Jan/20 22:34 Start Date: 17/Jan/20 22:34 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10631: [BEAM-8684] Google proto and google-services library upgrades URL: https://github.com/apache/beam/pull/10631#issuecomment-575821457 R: @lukecwik Ready for review. There's no difference in linkage checks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373928) Time Spent: 50m (was: 40m) > Beam Dependency Update Request: com.google.apis:google-api-services-bigquery > > > Key: BEAM-8684 > URL: https://issues.apache.org/jira/browse/BEAM-8684 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > - 2019-11-15 19:39:07.113511 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:04:31.460554 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:09:35.656962 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:08:45.501496 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:08:46.385699 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:04:15.902178 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:07:51.708019 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:07:45.384750 >
[jira] [Work logged] (BEAM-8684) Beam Dependency Update Request: com.google.apis:google-api-services-bigquery
[
https://issues.apache.org/jira/browse/BEAM-8684?focusedWorklogId=373926=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373926
]
ASF GitHub Bot logged work on BEAM-8684:
Author: ASF GitHub Bot
Created on: 17/Jan/20 22:33
Start Date: 17/Jan/20 22:33
Worklog Time Spent: 10m
Work Description: suztomo commented on pull request #10631: [BEAM-8684]
Google proto and google-services library upgrades
URL: https://github.com/apache/beam/pull/10631#discussion_r368163674
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -365,11 +365,10 @@ class BeamModulePlugin implements Plugin {
def cassandra_driver_version = "3.8.0"
def classgraph_version = "4.8.56"
def gax_version = "1.52.0"
-def generated_grpc_beta_version = "0.44.0"
def generated_grpc_ga_version = "1.83.0"
-def generated_grpc_dc_beta_version = "0.27.0-alpha"
+def generated_grpc_dc_beta_version = "0.29.0-alpha"
Review comment:
This is not the latest but 0.30.0-alpha requires code change. Stopping at
0.29.0-alpha for simplicity.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373926)
Time Spent: 40m (was: 0.5h)
> Beam Dependency Update Request: com.google.apis:google-api-services-bigquery
>
>
> Key: BEAM-8684
> URL: https://issues.apache.org/jira/browse/BEAM-8684
> Project: Beam
> Issue Type: Sub-task
> Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> - 2019-11-15 19:39:07.113511
> -
> Please consider upgrading the dependency
> com.google.apis:google-api-services-bigquery.
> The current version is v2-rev20181221-1.28.0. The latest version is
> v2-rev20190917-1.30.3
> cc: [~chamikara],
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-11-19 21:04:31.460554
> -
> Please consider upgrading the dependency
> com.google.apis:google-api-services-bigquery.
> The current version is v2-rev20181221-1.28.0. The latest version is
> v2-rev20190917-1.30.3
> cc: [~chamikara],
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-12-02 12:09:35.656962
> -
> Please consider upgrading the dependency
> com.google.apis:google-api-services-bigquery.
> The current version is v2-rev20181221-1.28.0. The latest version is
> v2-rev20190917-1.30.3
> cc: [~chamikara],
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-12-09 12:08:45.501496
> -
> Please consider upgrading the dependency
> com.google.apis:google-api-services-bigquery.
> The current version is v2-rev20181221-1.28.0. The latest version is
> v2-rev20190917-1.30.3
> cc: [~chamikara],
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-12-23 12:08:46.385699
> -
> Please consider upgrading the dependency
> com.google.apis:google-api-services-bigquery.
> The current version is v2-rev20181221-1.28.0. The latest version is
> v2-rev20190917-1.30.3
> cc: [~chamikara],
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-12-30 14:04:15.902178
> -
> Please consider upgrading the dependency
> com.google.apis:google-api-services-bigquery.
> The current version is v2-rev20181221-1.28.0. The latest version is
> v2-rev20190917-1.30.3
> cc: [~chamikara],
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
[jira] [Commented] (BEAM-9144) Beam's own Avro TimeConversion class in beam-sdk-java-core
[
https://issues.apache.org/jira/browse/BEAM-9144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018367#comment-17018367
]
Ismaël Mejía commented on BEAM-9144:
The fix was merged in master, so it would be great if you can test it with the
most recent SNAPSHOT version 2.20.0-SNAPSHOT [~atdixon] The fix should be
published tomorrow. I also cherry picked it hoping it gets into 2.19.0 since
2.18.0 vote is almost done at this point.
> Beam's own Avro TimeConversion class in beam-sdk-java-core
> ---
>
> Key: BEAM-9144
> URL: https://issues.apache.org/jira/browse/BEAM-9144
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Fix For: 2.19.0
>
> Attachments: avro-beam-dependency-graph.png
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> From Aaron's comment in
> https://issues.apache.org/jira/browse/BEAM-8388?focusedCommentId=17016476=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17016476
> .
> {quote}My org must use Avro 1.9.x (due to some Avro schema resolution issues
> resolved in 1.9.x) so downgrading Avro is not possible for us.
> Beam 2.16.0 is compatible with our usage of Avro 1.9.x – but upgrading to
> 2.17.0 we are broken as 2.17.0 links to Java classes in Avro 1.8.x that are
> not available in 1.9.x.
> {quote}
> The Java class is
> {{org.apache.avro.data.TimeConversions.TimestampConversion}} in Avro 1.8.
> It's renamed to {{org.apache.avro.data.JodaTimeConversions}} in Avro 1.9.
> h1. Beam Java SDK cannot upgrade Avro to 1.9
> Beam has Spark runners and Spark has not yet upgraded to Avro 1.9.
> Illustration of the dependency
> !avro-beam-dependency-graph.png|width=799,height=385!
> h1. Short-term Solution
> As illustrated above, as long as Beam Java SDK uses only the intersection of
> Avro classes, method, and fields between Avro 1.8 and 1.9, it will provide
> flexibility in runtime Avro versions (as it did until Beam 2.16).
> h2. Difference of the TimeConversion Classes
> Avro 1.9's TimestampConversion overrides {{getRecommendedSchema}} method.
> Details below:
> Avro 1.8's TimeConversions.TimestampConversion:
> {code:java}
> public static class TimestampConversion extends Conversion {
> @Override
> public Class getConvertedType() {
> return DateTime.class;
> }
> @Override
> public String getLogicalTypeName() {
> return "timestamp-millis";
> }
> @Override
> public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType
> type) {
> return new DateTime(millisFromEpoch, DateTimeZone.UTC);
> }
> @Override
> public Long toLong(DateTime timestamp, Schema schema, LogicalType type) {
> return timestamp.getMillis();
> }
> }
> {code}
> Avro 1.9's JodaTimeConversions.TimestampConversion:
> {code:java}
> public static class TimestampConversion extends Conversion {
> @Override
> public Class getConvertedType() {
> return DateTime.class;
> }
> @Override
> public String getLogicalTypeName() {
> return "timestamp-millis";
> }
> @Override
> public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType
> type) {
> return new DateTime(millisFromEpoch, DateTimeZone.UTC);
> }
> @Override
> public Long toLong(DateTime timestamp, Schema schema, LogicalType type) {
> return timestamp.getMillis();
> }
> @Override
> public Schema getRecommendedSchema() {
> return
> LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG));
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9144) Beam's own Avro TimeConversion class in beam-sdk-java-core
[
https://issues.apache.org/jira/browse/BEAM-9144?focusedWorklogId=373925=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373925
]
ASF GitHub Bot logged work on BEAM-9144:
Author: ASF GitHub Bot
Created on: 17/Jan/20 22:29
Start Date: 17/Jan/20 22:29
Worklog Time Spent: 10m
Work Description: iemejia commented on pull request #10632:
[release-2.19.0][BEAM-9144] Beam's own Avro TimeConversion class in
beam-sdk-java-core
URL: https://github.com/apache/beam/pull/10632
A cherry pick that fixes an issue reported by an incompatibility with Avro
versions reported by an user on dev@
R: @boyuanzz
CC: @suztomo
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373925)
Time Spent: 1.5h (was: 1h 20m)
> Beam's own Avro TimeConversion class in beam-sdk-java-core
> ---
>
> Key: BEAM-9144
> URL: https://issues.apache.org/jira/browse/BEAM-9144
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Fix For: 2.19.0
>
> Attachments: avro-beam-dependency-graph.png
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> From Aaron's comment in
> https://issues.apache.org/jira/browse/BEAM-8388?focusedCommentId=17016476=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17016476
> .
> {quote}My org must use Avro 1.9.x (due to some Avro schema resolution issues
> resolved in 1.9.x) so downgrading Avro is not possible for us.
> Beam 2.16.0 is compatible with our usage of Avro 1.9.x – but upgrading to
> 2.17.0 we are broken as 2.17.0 links to Java classes in Avro 1.8.x that are
> not available in 1.9.x.
> {quote}
> The Java class is
> {{org.apache.avro.data.TimeConversions.TimestampConversion}} in Avro 1.8.
> It's renamed to {{org.apache.avro.data.JodaTimeConversions}} in Avro 1.9.
> h1. Beam Java SDK cannot upgrade Avro to 1.9
> Beam has Spark runners and Spark has not yet upgraded to Avro 1.9.
> Illustration of the dependency
> !avro-beam-dependency-graph.png|width=799,height=385!
> h1. Short-term Solution
> As illustrated above, as long as Beam Java SDK uses only the intersection of
> Avro classes, method, and fields between Avro 1.8 and 1.9, it will provide
> flexibility in runtime Avro versions (as it did until Beam 2.16).
> h2. Difference of the TimeConversion Classes
> Avro 1.9's TimestampConversion overrides {{getRecommendedSchema}} method.
> Details below:
> Avro 1.8's TimeConversions.TimestampConversion:
> {code:java}
> public static class TimestampConversion extends Conversion {
> @Override
> public Class getConvertedType() {
> return DateTime.class;
> }
> @Override
> public String getLogicalTypeName() {
> return "timestamp-millis";
> }
> @Override
> public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType
> type) {
> return new DateTime(millisFromEpoch, DateTimeZone.UTC);
> }
> @Override
> public Long toLong(DateTime timestamp, Schema schema, LogicalType type) {
> return timestamp.getMillis();
> }
> }
> {code}
> Avro 1.9's JodaTimeConversions.TimestampConversion:
> {code:java}
> public static class TimestampConversion extends Conversion {
> @Override
> public Class getConvertedType() {
> return DateTime.class;
> }
> @Override
> public String getLogicalTypeName() {
> return "timestamp-millis";
> }
> @Override
> public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType
> type) {
> return new DateTime(millisFromEpoch, DateTimeZone.UTC);
> }
> @Override
> public Long toLong(DateTime timestamp, Schema schema, LogicalType type) {
> return timestamp.getMillis();
> }
> @Override
> public Schema getRecommendedSchema() {
> return
> LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG));
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9140) Update to ZetaSQL 2020.01.1
[
https://issues.apache.org/jira/browse/BEAM-9140?focusedWorklogId=373924=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373924
]
ASF GitHub Bot logged work on BEAM-9140:
Author: ASF GitHub Bot
Created on: 17/Jan/20 22:27
Start Date: 17/Jan/20 22:27
Worklog Time Spent: 10m
Work Description: apilloud commented on pull request #10620: [BEAM-9140]
Upgrade to ZetaSQL 2020.01.1
URL: https://github.com/apache/beam/pull/10620#discussion_r368162123
##
File path: build.gradle
##
@@ -33,6 +33,12 @@ plugins {
id "org.sonarqube" version "2.7"
}
+repositories {
+ maven {
+url "https://oss.sonatype.org/content/repositories/comgooglezetasql-1013;
Review comment:
Don't tell anyone, running with Apache Beam is the current release
verification step for ZetaSQL. Once this PR is passing, I'll publish the
ZetaSQL release, delete this line and send it out for review.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373924)
Time Spent: 1h 10m (was: 1h)
> Update to ZetaSQL 2020.01.1
> ---
>
> Key: BEAM-9140
> URL: https://issues.apache.org/jira/browse/BEAM-9140
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> I hear ZetaSQL 2020.01.1 will be coming out in the next few hours. We should
> upgrade.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8684) Beam Dependency Update Request: com.google.apis:google-api-services-bigquery
[ https://issues.apache.org/jira/browse/BEAM-8684?focusedWorklogId=373923=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373923 ] ASF GitHub Bot logged work on BEAM-8684: Author: ASF GitHub Bot Created on: 17/Jan/20 22:25 Start Date: 17/Jan/20 22:25 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10631: [BEAM-8684] [WIP] Google proto and google-services library upgrades URL: https://github.com/apache/beam/pull/10631#issuecomment-575819172 I just added them. Now running diffs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373923) Time Spent: 0.5h (was: 20m) > Beam Dependency Update Request: com.google.apis:google-api-services-bigquery > > > Key: BEAM-8684 > URL: https://issues.apache.org/jira/browse/BEAM-8684 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > - 2019-11-15 19:39:07.113511 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:04:31.460554 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:09:35.656962 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:08:45.501496 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:08:46.385699 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:04:15.902178 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:07:51.708019 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:07:45.384750 > - > Please
[jira] [Work logged] (BEAM-8684) Beam Dependency Update Request: com.google.apis:google-api-services-bigquery
[ https://issues.apache.org/jira/browse/BEAM-8684?focusedWorklogId=373922=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373922 ] ASF GitHub Bot logged work on BEAM-8684: Author: ASF GitHub Bot Created on: 17/Jan/20 22:24 Start Date: 17/Jan/20 22:24 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10631: [BEAM-8684] [WIP] Google proto and google-services library upgrades URL: https://github.com/apache/beam/pull/10631#issuecomment-575818958 What about the before linkage checker results? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373922) Time Spent: 20m (was: 10m) > Beam Dependency Update Request: com.google.apis:google-api-services-bigquery > > > Key: BEAM-8684 > URL: https://issues.apache.org/jira/browse/BEAM-8684 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > - 2019-11-15 19:39:07.113511 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:04:31.460554 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:09:35.656962 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:08:45.501496 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:08:46.385699 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:04:15.902178 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:07:51.708019 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-bigquery. > The current version is v2-rev20181221-1.28.0. The latest version is > v2-rev20190917-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:07:45.384750 > - >
[jira] [Work logged] (BEAM-9144) Beam's own Avro TimeConversion class in beam-sdk-java-core
[
https://issues.apache.org/jira/browse/BEAM-9144?focusedWorklogId=373920=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373920
]
ASF GitHub Bot logged work on BEAM-9144:
Author: ASF GitHub Bot
Created on: 17/Jan/20 22:23
Start Date: 17/Jan/20 22:23
Worklog Time Spent: 10m
Work Description: iemejia commented on pull request #10628: [BEAM-9144]
Beam's own Avro TimeConversion class in beam-sdk-java-core
URL: https://github.com/apache/beam/pull/10628
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373920)
Time Spent: 1h 20m (was: 1h 10m)
> Beam's own Avro TimeConversion class in beam-sdk-java-core
> ---
>
> Key: BEAM-9144
> URL: https://issues.apache.org/jira/browse/BEAM-9144
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Fix For: 2.19.0
>
> Attachments: avro-beam-dependency-graph.png
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> From Aaron's comment in
> https://issues.apache.org/jira/browse/BEAM-8388?focusedCommentId=17016476=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17016476
> .
> {quote}My org must use Avro 1.9.x (due to some Avro schema resolution issues
> resolved in 1.9.x) so downgrading Avro is not possible for us.
> Beam 2.16.0 is compatible with our usage of Avro 1.9.x – but upgrading to
> 2.17.0 we are broken as 2.17.0 links to Java classes in Avro 1.8.x that are
> not available in 1.9.x.
> {quote}
> The Java class is
> {{org.apache.avro.data.TimeConversions.TimestampConversion}} in Avro 1.8.
> It's renamed to {{org.apache.avro.data.JodaTimeConversions}} in Avro 1.9.
> h1. Beam Java SDK cannot upgrade Avro to 1.9
> Beam has Spark runners and Spark has not yet upgraded to Avro 1.9.
> Illustration of the dependency
> !avro-beam-dependency-graph.png|width=799,height=385!
> h1. Short-term Solution
> As illustrated above, as long as Beam Java SDK uses only the intersection of
> Avro classes, method, and fields between Avro 1.8 and 1.9, it will provide
> flexibility in runtime Avro versions (as it did until Beam 2.16).
> h2. Difference of the TimeConversion Classes
> Avro 1.9's TimestampConversion overrides {{getRecommendedSchema}} method.
> Details below:
> Avro 1.8's TimeConversions.TimestampConversion:
> {code:java}
> public static class TimestampConversion extends Conversion {
> @Override
> public Class getConvertedType() {
> return DateTime.class;
> }
> @Override
> public String getLogicalTypeName() {
> return "timestamp-millis";
> }
> @Override
> public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType
> type) {
> return new DateTime(millisFromEpoch, DateTimeZone.UTC);
> }
> @Override
> public Long toLong(DateTime timestamp, Schema schema, LogicalType type) {
> return timestamp.getMillis();
> }
> }
> {code}
> Avro 1.9's JodaTimeConversions.TimestampConversion:
> {code:java}
> public static class TimestampConversion extends Conversion {
> @Override
> public Class getConvertedType() {
> return DateTime.class;
> }
> @Override
> public String getLogicalTypeName() {
> return "timestamp-millis";
> }
> @Override
> public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType
> type) {
> return new DateTime(millisFromEpoch, DateTimeZone.UTC);
> }
> @Override
> public Long toLong(DateTime timestamp, Schema schema, LogicalType type) {
> return timestamp.getMillis();
> }
> @Override
> public Schema getRecommendedSchema() {
> return
> LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG));
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9140) Update to ZetaSQL 2020.01.1
[
https://issues.apache.org/jira/browse/BEAM-9140?focusedWorklogId=373918=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373918
]
ASF GitHub Bot logged work on BEAM-9140:
Author: ASF GitHub Bot
Created on: 17/Jan/20 22:21
Start Date: 17/Jan/20 22:21
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10620: [BEAM-9140]
Upgrade to ZetaSQL 2020.01.1
URL: https://github.com/apache/beam/pull/10620#discussion_r368160408
##
File path: build.gradle
##
@@ -33,6 +33,12 @@ plugins {
id "org.sonarqube" version "2.7"
}
+repositories {
+ maven {
+url "https://oss.sonatype.org/content/repositories/comgooglezetasql-1013;
Review comment:
This is an unreleased repository. How do our clients get this dependency in
a maven project without this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373918)
Time Spent: 1h (was: 50m)
> Update to ZetaSQL 2020.01.1
> ---
>
> Key: BEAM-9140
> URL: https://issues.apache.org/jira/browse/BEAM-9140
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> I hear ZetaSQL 2020.01.1 will be coming out in the next few hours. We should
> upgrade.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9137) PR10338 breaks beam_PostCommit_Py_ValCont
[
https://issues.apache.org/jira/browse/BEAM-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018354#comment-17018354
]
Boyuan Zhang commented on BEAM-9137:
Thanks for your confirmation and update! Turns out this test target only fails
on Py2 and even the last successful commit also failed on the same error:
Error processing instruction -77. Original traceback is Traceback (most recent
call last): File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 161, in _execute response = task() File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 217, in lambda: self.create_worker().do_instruction(request),
request) File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 346, in do_instruction request.instruction_id) File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 374, in process_bundle instruction_id,
request.process_bundle_descriptor_id) File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 282, in get self.data_channel_factory) File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 717, in __init__ self.ops =
self.create_execution_tree(self.process_bundle_descriptor) File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 766, in create_execution_tree descriptor.transforms,
key=topological_height, reverse=True)]) File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 680, in wrapper result = cache[args] = func(*args) File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 748, in get_operation in
descriptor.transforms[transform_id].outputs.items() File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 747, in for tag, pcoll_id File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 680, in wrapper result = cache[args] = func(*args) File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 748, in get_operation in
descriptor.transforms[transform_id].outputs.items() File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 747, in for tag, pcoll_id File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 680, in wrapper result = cache[args] = func(*args) File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 748, in get_operation in
descriptor.transforms[transform_id].outputs.items() File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 747, in for tag, pcoll_id File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 680, in wrapper result = cache[args] = func(*args) File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 751, in get_operation transform_id, transform_consumers) File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 1053, in create_operation return creator(self, transform_id,
transform_proto, payload, consumers) File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 1168, in create output_coder =
factory.get_only_input_coder(transform_proto) File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 1099, in get_only_input_coder return
only_element(list(self.get_input_coders(transform_proto).values())) File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 1094, in get_input_coders for tag, pcoll_id in
transform_proto.inputs.items() File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 1094, in for tag, pcoll_id in transform_proto.inputs.items()
File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 1069, in get_windowed_coder coder =
self.get_coder(self.descriptor.pcollections[pcoll_id].coder_id) File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 1065, in get_coder json.loads(coder_proto.spec.payload.decode('utf-8')))
File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/operation_specs.py",
line 331, in get_coder_from_spec get_coder_from_spec(c) for c in
coder_spec['component_encodings']] File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/operation_specs.py",
line 349, in get_coder_from_spec
get_coder_from_spec(coder_spec['component_encodings'][0])) File
[jira] [Work logged] (BEAM-9140) Update to ZetaSQL 2020.01.1
[ https://issues.apache.org/jira/browse/BEAM-9140?focusedWorklogId=373901=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373901 ] ASF GitHub Bot logged work on BEAM-9140: Author: ASF GitHub Bot Created on: 17/Jan/20 22:01 Start Date: 17/Jan/20 22:01 Worklog Time Spent: 10m Work Description: apilloud commented on issue #10620: [BEAM-9140] Upgrade to ZetaSQL 2020.01.1 URL: https://github.com/apache/beam/pull/10620#issuecomment-575812048 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373901) Time Spent: 50m (was: 40m) > Update to ZetaSQL 2020.01.1 > --- > > Key: BEAM-9140 > URL: https://issues.apache.org/jira/browse/BEAM-9140 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Andrew Pilloud >Assignee: Andrew Pilloud >Priority: Major > Fix For: 2.20.0 > > Time Spent: 50m > Remaining Estimate: 0h > > I hear ZetaSQL 2020.01.1 will be coming out in the next few hours. We should > upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8889) Make GcsUtil use GoogleCloudStorage
[ https://issues.apache.org/jira/browse/BEAM-8889?focusedWorklogId=373899=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373899 ] ASF GitHub Bot logged work on BEAM-8889: Author: ASF GitHub Bot Created on: 17/Jan/20 22:00 Start Date: 17/Jan/20 22:00 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #10617: [BEAM-8889] adding gRPC connectivity to Beam/GCS connector URL: https://github.com/apache/beam/pull/10617#issuecomment-575811548 R: @veblush This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373899) Remaining Estimate: 167h 40m (was: 167h 50m) Time Spent: 20m (was: 10m) > Make GcsUtil use GoogleCloudStorage > --- > > Key: BEAM-8889 > URL: https://issues.apache.org/jira/browse/BEAM-8889 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Affects Versions: 2.16.0 >Reporter: Esun Kim >Assignee: VASU NORI >Priority: Major > Labels: gcs > Original Estimate: 168h > Time Spent: 20m > Remaining Estimate: 167h 40m > > [GcsUtil|https://github.com/apache/beam/blob/master/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java] > is a primary class to access Google Cloud Storage on Apache Beam. Current > implementation directly creates GoogleCloudStorageReadChannel and > GoogleCloudStorageWriteChannel by itself to read and write GCS data rather > than using > [GoogleCloudStorage|https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorage.java] > which is an abstract class providing basic IO capability which eventually > creates channel objects. This request is about updating GcsUtil to use > GoogleCloudStorage to create read and write channel, which is expected > flexible because it can easily pick up the new change; e.g. new channel > implementation using new protocol without code change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8889) Make GcsUtil use GoogleCloudStorage
[ https://issues.apache.org/jira/browse/BEAM-8889?focusedWorklogId=373897=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373897 ] ASF GitHub Bot logged work on BEAM-8889: Author: ASF GitHub Bot Created on: 17/Jan/20 21:59 Start Date: 17/Jan/20 21:59 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #10617: [BEAM-8889] adding gRPC connectivity to Beam/GCS connector URL: https://github.com/apache/beam/pull/10617#issuecomment-575811548 R: @vebl...@google.com This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373897) Remaining Estimate: 167h 50m (was: 168h) Time Spent: 10m > Make GcsUtil use GoogleCloudStorage > --- > > Key: BEAM-8889 > URL: https://issues.apache.org/jira/browse/BEAM-8889 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Affects Versions: 2.16.0 >Reporter: Esun Kim >Assignee: VASU NORI >Priority: Major > Labels: gcs > Original Estimate: 168h > Time Spent: 10m > Remaining Estimate: 167h 50m > > [GcsUtil|https://github.com/apache/beam/blob/master/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java] > is a primary class to access Google Cloud Storage on Apache Beam. Current > implementation directly creates GoogleCloudStorageReadChannel and > GoogleCloudStorageWriteChannel by itself to read and write GCS data rather > than using > [GoogleCloudStorage|https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorage.java] > which is an abstract class providing basic IO capability which eventually > creates channel objects. This request is about updating GcsUtil to use > GoogleCloudStorage to create read and write channel, which is expected > flexible because it can easily pick up the new change; e.g. new channel > implementation using new protocol without code change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7861) Make it easy to change between multi-process and multi-thread mode for Python Direct runners
[ https://issues.apache.org/jira/browse/BEAM-7861?focusedWorklogId=373896=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373896 ] ASF GitHub Bot logged work on BEAM-7861: Author: ASF GitHub Bot Created on: 17/Jan/20 21:57 Start Date: 17/Jan/20 21:57 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #10616: [BEAM-7861] update documentation about --direct_running_mode option with direct runner. URL: https://github.com/apache/beam/pull/10616#issuecomment-575810811 Looking at http://apache-beam-website-pull-requests.storage.googleapis.com/10616/documentation/runners/direct/index.html, we will need to fix formatting. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373896) Time Spent: 3h 20m (was: 3h 10m) > Make it easy to change between multi-process and multi-thread mode for Python > Direct runners > > > Key: BEAM-7861 > URL: https://issues.apache.org/jira/browse/BEAM-7861 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.19.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > BEAM-3645 makes it possible to run a map task parallel. > However, users need to change runner when switch between multithreading and > multiprocessing mode. > We want to add a flag (ex: --use-multiprocess) to make the switch easy > without changing the runner each time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9149) Support ZetaSQL positional parameters
Kyle Weaver created BEAM-9149: - Summary: Support ZetaSQL positional parameters Key: BEAM-9149 URL: https://issues.apache.org/jira/browse/BEAM-9149 Project: Beam Issue Type: New Feature Components: dsl-sql-zetasql Reporter: Kyle Weaver Assignee: Kyle Weaver While they are not yet exposed to the end user, ZetaSQL query parameters are currently being passed internally. However, the existing code assumes that all parameters are named parameters, not positional parameters. To support positional parameters, we will need to make at least the following changes: 1) Set mode to PARAMETER_POSITIONAL and use addPositionalQueryParameter instead of addQueryParameter in SqlAnalyzer: https://github.com/apache/beam/blob/671b02ac5f1be87a591de8f5f456d0e5a199d771/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlAnalyzer.java#L119 2) Code currently assumes that resolved parameters are named. While even positional parameters must be named when they are used as inputs, after they are resolved their names are removed. Thus this check will deref a null pointer and must be fixed: https://github.com/apache/beam/blob/8915d6e95c405aeee0f29152545d3210e8e09f1f/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java#L1004 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9149) Support ZetaSQL positional parameters
[ https://issues.apache.org/jira/browse/BEAM-9149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Weaver updated BEAM-9149: -- Status: Open (was: Triage Needed) > Support ZetaSQL positional parameters > - > > Key: BEAM-9149 > URL: https://issues.apache.org/jira/browse/BEAM-9149 > Project: Beam > Issue Type: New Feature > Components: dsl-sql-zetasql >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > > While they are not yet exposed to the end user, ZetaSQL query parameters are > currently being passed internally. However, the existing code assumes that > all parameters are named parameters, not positional parameters. To support > positional parameters, we will need to make at least the following changes: > 1) Set mode to PARAMETER_POSITIONAL and use addPositionalQueryParameter > instead of addQueryParameter in SqlAnalyzer: > https://github.com/apache/beam/blob/671b02ac5f1be87a591de8f5f456d0e5a199d771/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlAnalyzer.java#L119 > 2) Code currently assumes that resolved parameters are named. While even > positional parameters must be named when they are used as inputs, after they > are resolved their names are removed. Thus this check will deref a null > pointer and must be fixed: > https://github.com/apache/beam/blob/8915d6e95c405aeee0f29152545d3210e8e09f1f/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java#L1004 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9144) Beam's own Avro TimeConversion class in beam-sdk-java-core
[
https://issues.apache.org/jira/browse/BEAM-9144?focusedWorklogId=373879=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373879
]
ASF GitHub Bot logged work on BEAM-9144:
Author: ASF GitHub Bot
Created on: 17/Jan/20 21:20
Start Date: 17/Jan/20 21:20
Worklog Time Spent: 10m
Work Description: iemejia commented on issue #10628: [BEAM-9144] Beam's
own Avro TimeConversion class in beam-sdk-java-core
URL: https://github.com/apache/beam/pull/10628#issuecomment-575799224
Run Java PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373879)
Time Spent: 1h 10m (was: 1h)
> Beam's own Avro TimeConversion class in beam-sdk-java-core
> ---
>
> Key: BEAM-9144
> URL: https://issues.apache.org/jira/browse/BEAM-9144
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Fix For: 2.19.0
>
> Attachments: avro-beam-dependency-graph.png
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> From Aaron's comment in
> https://issues.apache.org/jira/browse/BEAM-8388?focusedCommentId=17016476=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17016476
> .
> {quote}My org must use Avro 1.9.x (due to some Avro schema resolution issues
> resolved in 1.9.x) so downgrading Avro is not possible for us.
> Beam 2.16.0 is compatible with our usage of Avro 1.9.x – but upgrading to
> 2.17.0 we are broken as 2.17.0 links to Java classes in Avro 1.8.x that are
> not available in 1.9.x.
> {quote}
> The Java class is
> {{org.apache.avro.data.TimeConversions.TimestampConversion}} in Avro 1.8.
> It's renamed to {{org.apache.avro.data.JodaTimeConversions}} in Avro 1.9.
> h1. Beam Java SDK cannot upgrade Avro to 1.9
> Beam has Spark runners and Spark has not yet upgraded to Avro 1.9.
> Illustration of the dependency
> !avro-beam-dependency-graph.png|width=799,height=385!
> h1. Short-term Solution
> As illustrated above, as long as Beam Java SDK uses only the intersection of
> Avro classes, method, and fields between Avro 1.8 and 1.9, it will provide
> flexibility in runtime Avro versions (as it did until Beam 2.16).
> h2. Difference of the TimeConversion Classes
> Avro 1.9's TimestampConversion overrides {{getRecommendedSchema}} method.
> Details below:
> Avro 1.8's TimeConversions.TimestampConversion:
> {code:java}
> public static class TimestampConversion extends Conversion {
> @Override
> public Class getConvertedType() {
> return DateTime.class;
> }
> @Override
> public String getLogicalTypeName() {
> return "timestamp-millis";
> }
> @Override
> public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType
> type) {
> return new DateTime(millisFromEpoch, DateTimeZone.UTC);
> }
> @Override
> public Long toLong(DateTime timestamp, Schema schema, LogicalType type) {
> return timestamp.getMillis();
> }
> }
> {code}
> Avro 1.9's JodaTimeConversions.TimestampConversion:
> {code:java}
> public static class TimestampConversion extends Conversion {
> @Override
> public Class getConvertedType() {
> return DateTime.class;
> }
> @Override
> public String getLogicalTypeName() {
> return "timestamp-millis";
> }
> @Override
> public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType
> type) {
> return new DateTime(millisFromEpoch, DateTimeZone.UTC);
> }
> @Override
> public Long toLong(DateTime timestamp, Schema schema, LogicalType type) {
> return timestamp.getMillis();
> }
> @Override
> public Schema getRecommendedSchema() {
> return
> LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG));
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9063) Migrate docker images to apache namespace.
[
https://issues.apache.org/jira/browse/BEAM-9063?focusedWorklogId=373871=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373871
]
ASF GitHub Bot logged work on BEAM-9063:
Author: ASF GitHub Bot
Created on: 17/Jan/20 21:02
Start Date: 17/Jan/20 21:02
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #10612: [NOT READY TO
MERGE][BEAM-9063] migrate docker images to apache
URL: https://github.com/apache/beam/pull/10612#discussion_r368134576
##
File path: runners/spark/job-server/container/build.gradle
##
@@ -54,9 +54,10 @@ task copyDockerfileDependencies(type: Copy) {
}
docker {
- name containerImageName(name: 'spark-job-server',
+ name containerImageName(name: ${project.docker_image_default_repo_prefix} +
'job-server',
Review comment:
Looks like we lost `spark` here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373871)
Time Spent: 1h (was: 50m)
> Migrate docker images to apache namespace.
> --
>
> Key: BEAM-9063
> URL: https://issues.apache.org/jira/browse/BEAM-9063
> Project: Beam
> Issue Type: Task
> Components: beam-community
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> https://hub.docker.com/u/apache
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9063) Migrate docker images to apache namespace.
[ https://issues.apache.org/jira/browse/BEAM-9063?focusedWorklogId=373870=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373870 ] ASF GitHub Bot logged work on BEAM-9063: Author: ASF GitHub Bot Created on: 17/Jan/20 21:02 Start Date: 17/Jan/20 21:02 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #10612: [NOT READY TO MERGE][BEAM-9063] migrate docker images to apache URL: https://github.com/apache/beam/pull/10612#discussion_r368133471 ## File path: gradle.properties ## @@ -27,3 +27,6 @@ version=2.20.0-SNAPSHOT sdk_version=2.20.0.dev javaVersion=1.8 + +docker_image_default_repo_root=apache +docker_image_default_repo_prefix=beam- Review comment: Elsewhere, we use underscores in our docker image names (e.g. `java_sdk`). We should use an underscore here to keep it consistent. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373870) Time Spent: 50m (was: 40m) > Migrate docker images to apache namespace. > -- > > Key: BEAM-9063 > URL: https://issues.apache.org/jira/browse/BEAM-9063 > Project: Beam > Issue Type: Task > Components: beam-community >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: Not applicable > > Time Spent: 50m > Remaining Estimate: 0h > > https://hub.docker.com/u/apache -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7246) Create a Spanner IO for Python
[ https://issues.apache.org/jira/browse/BEAM-7246?focusedWorklogId=373868=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373868 ] ASF GitHub Bot logged work on BEAM-7246: Author: ASF GitHub Bot Created on: 17/Jan/20 20:55 Start Date: 17/Jan/20 20:55 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #9606: [BEAM-7246] Add Google Spanner IO Read on Python SDK URL: https://github.com/apache/beam/pull/9606 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373868) Time Spent: 12.5h (was: 12h 20m) > Create a Spanner IO for Python > -- > > Key: BEAM-7246 > URL: https://issues.apache.org/jira/browse/BEAM-7246 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Reporter: Reuven Lax >Assignee: Shehzaad Nakhoda >Priority: Major > Time Spent: 12.5h > Remaining Estimate: 0h > > Add I/O support for Google Cloud Spanner for the Python SDK (Batch Only). > Testing in this work item will be in the form of DirectRunner tests and > manual testing. > Integration and performance tests are a separate work item (not included > here). > See https://beam.apache.org/documentation/io/built-in/. The goal is to add > Google Clound Spanner to the Database column for the Python/Batch row. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7246) Create a Spanner IO for Python
[ https://issues.apache.org/jira/browse/BEAM-7246?focusedWorklogId=373869=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373869 ] ASF GitHub Bot logged work on BEAM-7246: Author: ASF GitHub Bot Created on: 17/Jan/20 20:55 Start Date: 17/Jan/20 20:55 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #9606: [BEAM-7246] Add Google Spanner IO Read on Python SDK URL: https://github.com/apache/beam/pull/9606#issuecomment-575791689 Thank you. Let's get integration tests in so that we can move this out of experimental :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373869) Time Spent: 12h 40m (was: 12.5h) > Create a Spanner IO for Python > -- > > Key: BEAM-7246 > URL: https://issues.apache.org/jira/browse/BEAM-7246 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Reporter: Reuven Lax >Assignee: Shehzaad Nakhoda >Priority: Major > Time Spent: 12h 40m > Remaining Estimate: 0h > > Add I/O support for Google Cloud Spanner for the Python SDK (Batch Only). > Testing in this work item will be in the form of DirectRunner tests and > manual testing. > Integration and performance tests are a separate work item (not included > here). > See https://beam.apache.org/documentation/io/built-in/. The goal is to add > Google Clound Spanner to the Database column for the Python/Batch row. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8889) Make GcsUtil use GoogleCloudStorage
[ https://issues.apache.org/jira/browse/BEAM-8889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018305#comment-17018305 ] VASU NORI commented on BEAM-8889: - PR in progress [https://github.com/apache/beam/pull/10617] > Make GcsUtil use GoogleCloudStorage > --- > > Key: BEAM-8889 > URL: https://issues.apache.org/jira/browse/BEAM-8889 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Affects Versions: 2.16.0 >Reporter: Esun Kim >Assignee: VASU NORI >Priority: Major > Labels: gcs > Original Estimate: 168h > Remaining Estimate: 168h > > [GcsUtil|https://github.com/apache/beam/blob/master/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java] > is a primary class to access Google Cloud Storage on Apache Beam. Current > implementation directly creates GoogleCloudStorageReadChannel and > GoogleCloudStorageWriteChannel by itself to read and write GCS data rather > than using > [GoogleCloudStorage|https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorage.java] > which is an abstract class providing basic IO capability which eventually > creates channel objects. This request is about updating GcsUtil to use > GoogleCloudStorage to create read and write channel, which is expected > flexible because it can easily pick up the new change; e.g. new channel > implementation using new protocol without code change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7246) Create a Spanner IO for Python
[ https://issues.apache.org/jira/browse/BEAM-7246?focusedWorklogId=373866=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373866 ] ASF GitHub Bot logged work on BEAM-7246: Author: ASF GitHub Bot Created on: 17/Jan/20 20:36 Start Date: 17/Jan/20 20:36 Worklog Time Spent: 10m Work Description: shehzaadn-vd commented on issue #9606: [BEAM-7246] Add Google Spanner IO Read on Python SDK URL: https://github.com/apache/beam/pull/9606#issuecomment-575785491 Thanks @chamikaramj for your support! @aaltay looks like the tests are passing. Would you be able to merge this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373866) Time Spent: 12h 20m (was: 12h 10m) > Create a Spanner IO for Python > -- > > Key: BEAM-7246 > URL: https://issues.apache.org/jira/browse/BEAM-7246 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Reporter: Reuven Lax >Assignee: Shehzaad Nakhoda >Priority: Major > Time Spent: 12h 20m > Remaining Estimate: 0h > > Add I/O support for Google Cloud Spanner for the Python SDK (Batch Only). > Testing in this work item will be in the form of DirectRunner tests and > manual testing. > Integration and performance tests are a separate work item (not included > here). > See https://beam.apache.org/documentation/io/built-in/. The goal is to add > Google Clound Spanner to the Database column for the Python/Batch row. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8684) Beam Dependency Update Request: com.google.apis:google-api-services-bigquery
[ https://issues.apache.org/jira/browse/BEAM-8684?focusedWorklogId=373852=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373852 ] ASF GitHub Bot logged work on BEAM-8684: Author: ASF GitHub Bot Created on: 17/Jan/20 20:22 Start Date: 17/Jan/20 20:22 Worklog Time Spent: 10m Work Description: suztomo commented on pull request #10631: [BEAM-8684] Google proto and google-services library upgrades URL: https://github.com/apache/beam/pull/10631 https://issues.apache.org/jira/browse/BEAM-8684 Google proto and google-services library upgrades. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build
[jira] [Work logged] (BEAM-9140) Update to ZetaSQL 2020.01.1
[ https://issues.apache.org/jira/browse/BEAM-9140?focusedWorklogId=373851=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373851 ] ASF GitHub Bot logged work on BEAM-9140: Author: ASF GitHub Bot Created on: 17/Jan/20 20:22 Start Date: 17/Jan/20 20:22 Worklog Time Spent: 10m Work Description: apilloud commented on issue #10620: [BEAM-9140] Upgrade to ZetaSQL 2020.01.1 URL: https://github.com/apache/beam/pull/10620#issuecomment-575780988 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373851) Time Spent: 40m (was: 0.5h) > Update to ZetaSQL 2020.01.1 > --- > > Key: BEAM-9140 > URL: https://issues.apache.org/jira/browse/BEAM-9140 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Andrew Pilloud >Assignee: Andrew Pilloud >Priority: Major > Fix For: 2.20.0 > > Time Spent: 40m > Remaining Estimate: 0h > > I hear ZetaSQL 2020.01.1 will be coming out in the next few hours. We should > upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8626) Implement status api handler in python sdk harness
[
https://issues.apache.org/jira/browse/BEAM-8626?focusedWorklogId=373835=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373835
]
ASF GitHub Bot logged work on BEAM-8626:
Author: ASF GitHub Bot
Created on: 17/Jan/20 19:50
Start Date: 17/Jan/20 19:50
Worklog Time Spent: 10m
Work Description: y1chi commented on pull request #10598: [BEAM-8626]
Implement status fn api handler in python sdk
URL: https://github.com/apache/beam/pull/10598#discussion_r368108678
##
File path: sdks/python/apache_beam/runners/worker/worker_status.py
##
@@ -0,0 +1,139 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Worker status api handler for reporting SDK harness debug info."""
+
+from __future__ import absolute_import
+from __future__ import division
+
+import queue
+import sys
+import threading
+import traceback
+from collections import defaultdict
+
+import grpc
+
+from apache_beam.portability.api import beam_fn_api_pb2
+from apache_beam.portability.api import beam_fn_api_pb2_grpc
+from apache_beam.runners.worker.channel_factory import GRPCChannelFactory
+from apache_beam.runners.worker.worker_id_interceptor import
WorkerIdInterceptor
+
+
+def thread_dump():
+ # deduplicate threads with same stack trace
+ stack_traces = defaultdict(list)
+ frames = sys._current_frames() # pylint: disable=protected-access
+
+ for t in threading.enumerate():
+stack_trace = ''.join(traceback.format_stack(frames[t.ident]))
+thread_ident_name = (t.ident, t.name)
+stack_traces[stack_trace].append(thread_ident_name)
+
+ all_traces = ['=' * 10 + 'THREAD DUMP' + '=' * 10]
+ for stack, identity in stack_traces.items():
+ident, name = identity[0]
+trace = '--- Thread #%s name: %s %s---\n' % (
+ident, name, 'and other %d threads' %
+(len(identity) - 1) if len(identity) > 1 else '')
+if len(identity) > 1:
+ trace += 'threads: %s\n' % identity
+trace += stack
+all_traces.append(trace)
+ all_traces.append('=' * 30)
+ return '\n'.join(x.encode('utf-8') for x in all_traces)
+
+
+def active_processing_bundles_state(bundle_process_cache):
+ active_bundles = ['=' * 10 + 'ACTIVE PROCESSING BUNDLES' + '=' * 10]
+ if not bundle_process_cache.active_bundle_processors:
+active_bundles.append("No active processing bundles.")
+ else:
+cache = []
+for instruction in list(
+bundle_process_cache.active_bundle_processors.keys()):
+ processor = bundle_process_cache.lookup(instruction)
+ if processor:
+info = processor.state_sampler.get_info()
+cache.append((instruction,
+ processor.process_bundle_descriptor.id,
+ info.tracked_thread, info.time_since_transition))
+# reverse sort active bundle by time since last transition, keep top 10.
+cache.sort(key=lambda x: x[-1], reverse=True)
+for s in cache[:10]:
+ state = '--- instruction %s ---\n' % s[0]
+ state += 'ProcessBundleDescriptorId: %s\n' % s[1]
+ state += "tracked thread: %s\n" % s[2]
+ state += "time since transition: %.2f seconds\n" % (s[3] / 1e9)
+ active_bundles.append(state)
+
+ active_bundles.append('=' * 30)
+ return '\n'.join(x.encode('utf-8') for x in active_bundles)
+
+
+DONE = object()
+
+
+class FnApiWorkerStatusHandler(object):
+ def __init__(self, status_address, bundle_process_cache=None):
+self._alive = True
+self._bundle_process_cache = bundle_process_cache
+ch = GRPCChannelFactory.insecure_channel(status_address)
+grpc.channel_ready_future(ch).result(timeout=60)
+self._status_channel = grpc.intercept_channel(ch, WorkerIdInterceptor())
+self._status_stub = beam_fn_api_pb2_grpc.BeamFnWorkerStatusStub(
+self._status_channel)
+self._responses = queue.Queue()
+self._server = threading.Thread(target=lambda: self._serve(),
+name='fn_api_status_handler')
+self._server.daemon = True
+self._server.start()
+
+ def _get_responses(self):
+while True:
+ response = self._responses.get()
+ if response is DONE:
+self._alive =
[jira] [Work logged] (BEAM-8626) Implement status api handler in python sdk harness
[
https://issues.apache.org/jira/browse/BEAM-8626?focusedWorklogId=373834=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373834
]
ASF GitHub Bot logged work on BEAM-8626:
Author: ASF GitHub Bot
Created on: 17/Jan/20 19:50
Start Date: 17/Jan/20 19:50
Worklog Time Spent: 10m
Work Description: y1chi commented on pull request #10598: [BEAM-8626]
Implement status fn api handler in python sdk
URL: https://github.com/apache/beam/pull/10598#discussion_r368108652
##
File path: sdks/python/apache_beam/runners/worker/worker_status.py
##
@@ -0,0 +1,139 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Worker status api handler for reporting SDK harness debug info."""
+
+from __future__ import absolute_import
+from __future__ import division
+
+import queue
+import sys
+import threading
+import traceback
+from collections import defaultdict
+
+import grpc
+
+from apache_beam.portability.api import beam_fn_api_pb2
+from apache_beam.portability.api import beam_fn_api_pb2_grpc
+from apache_beam.runners.worker.channel_factory import GRPCChannelFactory
+from apache_beam.runners.worker.worker_id_interceptor import
WorkerIdInterceptor
+
+
+def thread_dump():
+ # deduplicate threads with same stack trace
+ stack_traces = defaultdict(list)
+ frames = sys._current_frames() # pylint: disable=protected-access
+
+ for t in threading.enumerate():
+stack_trace = ''.join(traceback.format_stack(frames[t.ident]))
+thread_ident_name = (t.ident, t.name)
+stack_traces[stack_trace].append(thread_ident_name)
+
+ all_traces = ['=' * 10 + 'THREAD DUMP' + '=' * 10]
+ for stack, identity in stack_traces.items():
+ident, name = identity[0]
+trace = '--- Thread #%s name: %s %s---\n' % (
+ident, name, 'and other %d threads' %
+(len(identity) - 1) if len(identity) > 1 else '')
+if len(identity) > 1:
+ trace += 'threads: %s\n' % identity
+trace += stack
+all_traces.append(trace)
+ all_traces.append('=' * 30)
+ return '\n'.join(x.encode('utf-8') for x in all_traces)
+
+
+def active_processing_bundles_state(bundle_process_cache):
+ active_bundles = ['=' * 10 + 'ACTIVE PROCESSING BUNDLES' + '=' * 10]
+ if not bundle_process_cache.active_bundle_processors:
+active_bundles.append("No active processing bundles.")
+ else:
+cache = []
+for instruction in list(
+bundle_process_cache.active_bundle_processors.keys()):
+ processor = bundle_process_cache.lookup(instruction)
+ if processor:
+info = processor.state_sampler.get_info()
+cache.append((instruction,
+ processor.process_bundle_descriptor.id,
+ info.tracked_thread, info.time_since_transition))
+# reverse sort active bundle by time since last transition, keep top 10.
+cache.sort(key=lambda x: x[-1], reverse=True)
+for s in cache[:10]:
+ state = '--- instruction %s ---\n' % s[0]
+ state += 'ProcessBundleDescriptorId: %s\n' % s[1]
+ state += "tracked thread: %s\n" % s[2]
+ state += "time since transition: %.2f seconds\n" % (s[3] / 1e9)
+ active_bundles.append(state)
+
+ active_bundles.append('=' * 30)
+ return '\n'.join(x.encode('utf-8') for x in active_bundles)
+
+
+DONE = object()
+
+
+class FnApiWorkerStatusHandler(object):
+ def __init__(self, status_address, bundle_process_cache=None):
+self._alive = True
+self._bundle_process_cache = bundle_process_cache
+ch = GRPCChannelFactory.insecure_channel(status_address)
+grpc.channel_ready_future(ch).result(timeout=60)
+self._status_channel = grpc.intercept_channel(ch, WorkerIdInterceptor())
+self._status_stub = beam_fn_api_pb2_grpc.BeamFnWorkerStatusStub(
+self._status_channel)
+self._responses = queue.Queue()
+self._server = threading.Thread(target=lambda: self._serve(),
+name='fn_api_status_handler')
+self._server.daemon = True
+self._server.start()
+
+ def _get_responses(self):
+while True:
+ response = self._responses.get()
+ if response is DONE:
+self._alive =
[jira] [Work logged] (BEAM-8626) Implement status api handler in python sdk harness
[
https://issues.apache.org/jira/browse/BEAM-8626?focusedWorklogId=373828=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373828
]
ASF GitHub Bot logged work on BEAM-8626:
Author: ASF GitHub Bot
Created on: 17/Jan/20 19:45
Start Date: 17/Jan/20 19:45
Worklog Time Spent: 10m
Work Description: y1chi commented on pull request #10598: [BEAM-8626]
Implement status fn api handler in python sdk
URL: https://github.com/apache/beam/pull/10598#discussion_r368106706
##
File path: sdks/python/apache_beam/runners/worker/worker_status.py
##
@@ -0,0 +1,139 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Worker status api handler for reporting SDK harness debug info."""
+
+from __future__ import absolute_import
+from __future__ import division
+
+import queue
+import sys
+import threading
+import traceback
+from collections import defaultdict
+
+import grpc
+
+from apache_beam.portability.api import beam_fn_api_pb2
+from apache_beam.portability.api import beam_fn_api_pb2_grpc
+from apache_beam.runners.worker.channel_factory import GRPCChannelFactory
+from apache_beam.runners.worker.worker_id_interceptor import
WorkerIdInterceptor
+
+
+def thread_dump():
+ # deduplicate threads with same stack trace
+ stack_traces = defaultdict(list)
+ frames = sys._current_frames() # pylint: disable=protected-access
+
+ for t in threading.enumerate():
+stack_trace = ''.join(traceback.format_stack(frames[t.ident]))
+thread_ident_name = (t.ident, t.name)
+stack_traces[stack_trace].append(thread_ident_name)
+
+ all_traces = ['=' * 10 + 'THREAD DUMP' + '=' * 10]
+ for stack, identity in stack_traces.items():
+ident, name = identity[0]
+trace = '--- Thread #%s name: %s %s---\n' % (
+ident, name, 'and other %d threads' %
+(len(identity) - 1) if len(identity) > 1 else '')
+if len(identity) > 1:
+ trace += 'threads: %s\n' % identity
+trace += stack
+all_traces.append(trace)
+ all_traces.append('=' * 30)
+ return '\n'.join(x.encode('utf-8') for x in all_traces)
+
+
+def active_processing_bundles_state(bundle_process_cache):
+ active_bundles = ['=' * 10 + 'ACTIVE PROCESSING BUNDLES' + '=' * 10]
+ if not bundle_process_cache.active_bundle_processors:
+active_bundles.append("No active processing bundles.")
+ else:
+cache = []
+for instruction in list(
+bundle_process_cache.active_bundle_processors.keys()):
+ processor = bundle_process_cache.lookup(instruction)
+ if processor:
+info = processor.state_sampler.get_info()
+cache.append((instruction,
+ processor.process_bundle_descriptor.id,
+ info.tracked_thread, info.time_since_transition))
+# reverse sort active bundle by time since last transition, keep top 10.
+cache.sort(key=lambda x: x[-1], reverse=True)
+for s in cache[:10]:
+ state = '--- instruction %s ---\n' % s[0]
+ state += 'ProcessBundleDescriptorId: %s\n' % s[1]
+ state += "tracked thread: %s\n" % s[2]
+ state += "time since transition: %.2f seconds\n" % (s[3] / 1e9)
+ active_bundles.append(state)
+
+ active_bundles.append('=' * 30)
+ return '\n'.join(x.encode('utf-8') for x in active_bundles)
+
+
+DONE = object()
+
+
+class FnApiWorkerStatusHandler(object):
+ def __init__(self, status_address, bundle_process_cache=None):
+self._alive = True
+self._bundle_process_cache = bundle_process_cache
+ch = GRPCChannelFactory.insecure_channel(status_address)
+grpc.channel_ready_future(ch).result(timeout=60)
+self._status_channel = grpc.intercept_channel(ch, WorkerIdInterceptor())
+self._status_stub = beam_fn_api_pb2_grpc.BeamFnWorkerStatusStub(
+self._status_channel)
+self._responses = queue.Queue()
+self._server = threading.Thread(target=lambda: self._serve(),
+name='fn_api_status_handler')
+self._server.daemon = True
+self._server.start()
+
+ def _get_responses(self):
+while True:
+ response = self._responses.get()
+ if response is DONE:
+self._alive =
[jira] [Work logged] (BEAM-8626) Implement status api handler in python sdk harness
[ https://issues.apache.org/jira/browse/BEAM-8626?focusedWorklogId=373825=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373825 ] ASF GitHub Bot logged work on BEAM-8626: Author: ASF GitHub Bot Created on: 17/Jan/20 19:43 Start Date: 17/Jan/20 19:43 Worklog Time Spent: 10m Work Description: y1chi commented on pull request #10598: [BEAM-8626] Implement status fn api handler in python sdk URL: https://github.com/apache/beam/pull/10598#discussion_r368105639 ## File path: sdks/python/apache_beam/runners/worker/sdk_worker.py ## @@ -110,6 +112,15 @@ def __init__(self, data_channel_factory=self._data_channel_factory, fns=self._fns) +if status_address: Review comment: I need to do the actual initialization inside sdk_worker since I want to pass the active bundle cache in sdk worker in order to report the dangling operation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373825) Time Spent: 1h 20m (was: 1h 10m) > Implement status api handler in python sdk harness > -- > > Key: BEAM-8626 > URL: https://issues.apache.org/jira/browse/BEAM-8626 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8626) Implement status api handler in python sdk harness
[ https://issues.apache.org/jira/browse/BEAM-8626?focusedWorklogId=373823=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373823 ] ASF GitHub Bot logged work on BEAM-8626: Author: ASF GitHub Bot Created on: 17/Jan/20 19:41 Start Date: 17/Jan/20 19:41 Worklog Time Spent: 10m Work Description: y1chi commented on pull request #10598: [BEAM-8626] Implement status fn api handler in python sdk URL: https://github.com/apache/beam/pull/10598#discussion_r368104854 ## File path: sdks/python/apache_beam/runners/worker/sdk_worker.py ## @@ -73,6 +74,7 @@ class SdkHarness(object): def __init__(self, control_address, # type: str + status_address=None, # type: Optional[str, unicode] Review comment: done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373823) Time Spent: 1h 10m (was: 1h) > Implement status api handler in python sdk harness > -- > > Key: BEAM-8626 > URL: https://issues.apache.org/jira/browse/BEAM-8626 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7246) Create a Spanner IO for Python
[ https://issues.apache.org/jira/browse/BEAM-7246?focusedWorklogId=373821=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373821 ] ASF GitHub Bot logged work on BEAM-7246: Author: ASF GitHub Bot Created on: 17/Jan/20 19:38 Start Date: 17/Jan/20 19:38 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #9606: [BEAM-7246] Add Google Spanner IO Read on Python SDK URL: https://github.com/apache/beam/pull/9606#issuecomment-575766434 LGTM. Thanks. We can get this in when tests pass. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373821) Time Spent: 12h 10m (was: 12h) > Create a Spanner IO for Python > -- > > Key: BEAM-7246 > URL: https://issues.apache.org/jira/browse/BEAM-7246 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Reporter: Reuven Lax >Assignee: Shehzaad Nakhoda >Priority: Major > Time Spent: 12h 10m > Remaining Estimate: 0h > > Add I/O support for Google Cloud Spanner for the Python SDK (Batch Only). > Testing in this work item will be in the form of DirectRunner tests and > manual testing. > Integration and performance tests are a separate work item (not included > here). > See https://beam.apache.org/documentation/io/built-in/. The goal is to add > Google Clound Spanner to the Database column for the Python/Batch row. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8626) Implement status api handler in python sdk harness
[ https://issues.apache.org/jira/browse/BEAM-8626?focusedWorklogId=373820=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373820 ] ASF GitHub Bot logged work on BEAM-8626: Author: ASF GitHub Bot Created on: 17/Jan/20 19:36 Start Date: 17/Jan/20 19:36 Worklog Time Spent: 10m Work Description: y1chi commented on pull request #10598: [BEAM-8626] Implement status fn api handler in python sdk URL: https://github.com/apache/beam/pull/10598#discussion_r368102764 ## File path: sdks/python/apache_beam/runners/worker/worker_status.py ## @@ -0,0 +1,139 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +"""Worker status api handler for reporting SDK harness debug info.""" + +from __future__ import absolute_import +from __future__ import division + +import queue +import sys +import threading +import traceback +from collections import defaultdict + +import grpc + +from apache_beam.portability.api import beam_fn_api_pb2 +from apache_beam.portability.api import beam_fn_api_pb2_grpc +from apache_beam.runners.worker.channel_factory import GRPCChannelFactory +from apache_beam.runners.worker.worker_id_interceptor import WorkerIdInterceptor + + +def thread_dump(): Review comment: I made few changes to the thread dump format. I'll reuse the function, I think eventually we probably won't need the status http server. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373820) Time Spent: 1h (was: 50m) > Implement status api handler in python sdk harness > -- > > Key: BEAM-8626 > URL: https://issues.apache.org/jira/browse/BEAM-8626 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9148) test flakiness: BigQueryQueryToTableIT.test_big_query_standard_sql
[
https://issues.apache.org/jira/browse/BEAM-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri reassigned BEAM-9148:
---
Assignee: Boyuan Zhang
> test flakiness: BigQueryQueryToTableIT.test_big_query_standard_sql
> --
>
> Key: BEAM-9148
> URL: https://issues.apache.org/jira/browse/BEAM-9148
> Project: Beam
> Issue Type: Bug
> Components: io-py-gcp, sdk-py-core, test-failures
>Reporter: Udi Meiri
>Assignee: Boyuan Zhang
>Priority: Major
>
> There might be other flaky test cases from the same class, but I'm focusing
> on test_big_query_standard_sql here.
> {code}
> 19:39:12
> ==
> 19:39:12 FAIL: test_big_query_standard_sql
> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
> 19:39:12
> --
> 19:39:12 Traceback (most recent call last):
> 19:39:12File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py",
> line 172, in test_big_query_standard_sql
> 19:39:12 big_query_query_to_table_pipeline.run_bq_pipeline(options)
> 19:39:12File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_pipeline.py",
> line 84, in run_bq_pipeline
> 19:39:12 result = p.run()
> 19:39:12File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/testing/test_pipeline.py",
> line 112, in run
> 19:39:12 else test_runner_api))
> 19:39:12File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
> line 461, in run
> 19:39:12 self._options).run(False)
> 19:39:12File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
> line 474, in run
> 19:39:12 return self.runner.run_pipeline(self, self._options)
> 19:39:12File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/runners/direct/test_direct_runner.py",
> line 53, in run_pipeline
> 19:39:12 hc_assert_that(self.result, pickler.loads(on_success_matcher))
> 19:39:12 AssertionError:
> 19:39:12 Expected: (Test pipeline expected terminated in state: DONE and
> Expected checksum is 158a8ea1c254fcf40d4ed3e7c0242c3ea0a29e72)
> 19:39:12 but: Expected checksum is
> 158a8ea1c254fcf40d4ed3e7c0242c3ea0a29e72 Actual checksum is
> da39a3ee5e6b4b0d3255bfef95601890afd80709
> 19:39:12
> 19:39:12 >> begin captured logging <<
>
> 19:39:12 root: DEBUG: Unhandled type_constraint: Union[]
> 19:39:12 root: DEBUG: Unhandled type_constraint: Union[]
> 19:39:12 apache_beam.runners.direct.direct_runner: INFO: Running pipeline
> with DirectRunner.
> 19:39:12 apache_beam.io.gcp.bigquery_tools: DEBUG: Query SELECT * FROM
> (SELECT "apple" as fruit) UNION ALL (SELECT "orange" as fruit) does not
> reference any tables.
> 19:39:12 apache_beam.io.gcp.bigquery_tools: WARNING: Dataset
> apache-beam-testing:temp_dataset_90f5797bdb5f4137af750399f91a8e66 does not
> exist so we will create it as temporary with location=None
> 19:39:12 apache_beam.io.gcp.bigquery: DEBUG: Creating or getting table
> 19:39:12 datasetId: 'python_query_to_table_15792323245106'
> 19:39:12 projectId: 'apache-beam-testing'
> 19:39:12 tableId: 'output_table'> with schema {'fields': [{'name': 'fruit',
> 'type': 'STRING', 'mode': 'NULLABLE'}]}.
> 19:39:12 apache_beam.io.gcp.bigquery_tools: DEBUG: Created the table with id
> output_table
> 19:39:12 apache_beam.io.gcp.bigquery_tools: INFO: Created table
> apache-beam-testing.python_query_to_table_15792323245106.output_table with
> schema 19:39:12 fields: [ 19:39:12 fields: []
> 19:39:12 mode: 'NULLABLE'
> 19:39:12 name: 'fruit'
> 19:39:12 type: 'STRING'>]>. Result: 19:39:12 creationTime: 1579232328576
> 19:39:12 etag: 'WYysl6UIvc8IWMmTiiKhbg=='
> 19:39:12 id:
> 'apache-beam-testing:python_query_to_table_15792323245106.output_table'
> 19:39:12 kind: 'bigquery#table'
> 19:39:12 lastModifiedTime: 1579232328629
> 19:39:12 location: 'US'
> 19:39:12 numBytes: 0
> 19:39:12 numLongTermBytes: 0
> 19:39:12 numRows: 0
> 19:39:12 schema: 19:39:12 fields: [ 19:39:12 fields: []
> 19:39:12 mode: 'NULLABLE'
> 19:39:12 name: 'fruit'
> 19:39:12 type: 'STRING'>]>
> 19:39:12 selfLink:
> 'https://www.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/python_query_to_table_15792323245106/tables/output_table'
> 19:39:12 tableReference: 19:39:12 datasetId:
[jira] [Work logged] (BEAM-7246) Create a Spanner IO for Python
[ https://issues.apache.org/jira/browse/BEAM-7246?focusedWorklogId=373814=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373814 ] ASF GitHub Bot logged work on BEAM-7246: Author: ASF GitHub Bot Created on: 17/Jan/20 19:29 Start Date: 17/Jan/20 19:29 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #9606: [BEAM-7246] Add Google Spanner IO Read on Python SDK URL: https://github.com/apache/beam/pull/9606#issuecomment-575763166 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373814) Time Spent: 11h 50m (was: 11h 40m) > Create a Spanner IO for Python > -- > > Key: BEAM-7246 > URL: https://issues.apache.org/jira/browse/BEAM-7246 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Reporter: Reuven Lax >Assignee: Shehzaad Nakhoda >Priority: Major > Time Spent: 11h 50m > Remaining Estimate: 0h > > Add I/O support for Google Cloud Spanner for the Python SDK (Batch Only). > Testing in this work item will be in the form of DirectRunner tests and > manual testing. > Integration and performance tests are a separate work item (not included > here). > See https://beam.apache.org/documentation/io/built-in/. The goal is to add > Google Clound Spanner to the Database column for the Python/Batch row. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7246) Create a Spanner IO for Python
[ https://issues.apache.org/jira/browse/BEAM-7246?focusedWorklogId=373815=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373815 ] ASF GitHub Bot logged work on BEAM-7246: Author: ASF GitHub Bot Created on: 17/Jan/20 19:29 Start Date: 17/Jan/20 19:29 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #9606: [BEAM-7246] Add Google Spanner IO Read on Python SDK URL: https://github.com/apache/beam/pull/9606#issuecomment-575763362 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373815) Time Spent: 12h (was: 11h 50m) > Create a Spanner IO for Python > -- > > Key: BEAM-7246 > URL: https://issues.apache.org/jira/browse/BEAM-7246 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Reporter: Reuven Lax >Assignee: Shehzaad Nakhoda >Priority: Major > Time Spent: 12h > Remaining Estimate: 0h > > Add I/O support for Google Cloud Spanner for the Python SDK (Batch Only). > Testing in this work item will be in the form of DirectRunner tests and > manual testing. > Integration and performance tests are a separate work item (not included > here). > See https://beam.apache.org/documentation/io/built-in/. The goal is to add > Google Clound Spanner to the Database column for the Python/Batch row. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9122) Add uses_keyed_state step property to python dataflow runner
[ https://issues.apache.org/jira/browse/BEAM-9122?focusedWorklogId=373808=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373808 ] ASF GitHub Bot logged work on BEAM-9122: Author: ASF GitHub Bot Created on: 17/Jan/20 19:24 Start Date: 17/Jan/20 19:24 Worklog Time Spent: 10m Work Description: y1chi commented on pull request #10596: [BEAM-9122] Add uses_keyed_state step property in python dataflow run… URL: https://github.com/apache/beam/pull/10596#discussion_r368097986 ## File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py ## @@ -950,6 +950,10 @@ def run_ParDo(self, transform_node, options): step.add_property(PropertyNames.RESTRICTION_ENCODING, self._get_cloud_encoding(restriction_coder)) +from apache_beam.runners.common import DoFnSignature +if DoFnSignature(transform.dofn).is_stateful_dofn(): Review comment: I see, done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373808) Time Spent: 1.5h (was: 1h 20m) > Add uses_keyed_state step property to python dataflow runner > > > Key: BEAM-9122 > URL: https://issues.apache.org/jira/browse/BEAM-9122 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > Add additional step property to dataflow job property when a DoFn is stateful > in python sdk. So that the backend runner can recognize stateful steps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7861) Make it easy to change between multi-process and multi-thread mode for Python Direct runners
[ https://issues.apache.org/jira/browse/BEAM-7861?focusedWorklogId=373798=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373798 ] ASF GitHub Bot logged work on BEAM-7861: Author: ASF GitHub Bot Created on: 17/Jan/20 19:13 Start Date: 17/Jan/20 19:13 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #10616: [BEAM-7861] update documentation URL: https://github.com/apache/beam/pull/10616#issuecomment-575757817 Run Website_Stage_GCS PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373798) Time Spent: 3h 10m (was: 3h) > Make it easy to change between multi-process and multi-thread mode for Python > Direct runners > > > Key: BEAM-7861 > URL: https://issues.apache.org/jira/browse/BEAM-7861 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.19.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > > BEAM-3645 makes it possible to run a map task parallel. > However, users need to change runner when switch between multithreading and > multiprocessing mode. > We want to add a flag (ex: --use-multiprocess) to make the switch easy > without changing the runner each time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (BEAM-8685) Beam Dependency Update Request: com.google.auth:google-auth-library-oauth2-http
[ https://issues.apache.org/jira/browse/BEAM-8685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomo Suzuki closed BEAM-8685. - Fix Version/s: 2.20.0 Resolution: Fixed Now it's 0.19 in master > Beam Dependency Update Request: > com.google.auth:google-auth-library-oauth2-http > --- > > Key: BEAM-8685 > URL: https://issues.apache.org/jira/browse/BEAM-8685 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Assignee: Tomo Suzuki >Priority: Major > Fix For: 2.20.0 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > - 2019-11-15 19:39:27.324449 > - > Please consider upgrading the dependency > com.google.auth:google-auth-library-oauth2-http. > The current version is 0.12.0. The latest version is 0.18.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:05:03.844285 > - > Please consider upgrading the dependency > com.google.auth:google-auth-library-oauth2-http. > The current version is 0.12.0. The latest version is 0.18.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:10:30.864371 > - > Please consider upgrading the dependency > com.google.auth:google-auth-library-oauth2-http. > The current version is 0.12.0. The latest version is 0.18.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:09:38.646889 > - > Please consider upgrading the dependency > com.google.auth:google-auth-library-oauth2-http. > The current version is 0.12.0. The latest version is 0.18.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:09:39.967215 > - > Please consider upgrading the dependency > com.google.auth:google-auth-library-oauth2-http. > The current version is 0.12.0. The latest version is 0.19.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:05:10.534268 > - > Please consider upgrading the dependency > com.google.auth:google-auth-library-oauth2-http. > The current version is 0.12.0. The latest version is 0.19.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:08:45.451960 > - > Please consider upgrading the dependency > com.google.auth:google-auth-library-oauth2-http. > The current version is 0.12.0. The latest version is 0.19.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7861) Make it easy to change between multi-process and multi-thread mode for Python Direct runners
[ https://issues.apache.org/jira/browse/BEAM-7861?focusedWorklogId=373796=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373796 ] ASF GitHub Bot logged work on BEAM-7861: Author: ASF GitHub Bot Created on: 17/Jan/20 19:10 Start Date: 17/Jan/20 19:10 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #10616: [BEAM-7861] update documentation URL: https://github.com/apache/beam/pull/10616#issuecomment-575756723 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373796) Time Spent: 3h (was: 2h 50m) > Make it easy to change between multi-process and multi-thread mode for Python > Direct runners > > > Key: BEAM-7861 > URL: https://issues.apache.org/jira/browse/BEAM-7861 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.19.0 > > Time Spent: 3h > Remaining Estimate: 0h > > BEAM-3645 makes it possible to run a map task parallel. > However, users need to change runner when switch between multithreading and > multiprocessing mode. > We want to add a flag (ex: --use-multiprocess) to make the switch easy > without changing the runner each time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (BEAM-9130) sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with ImportError: No module named google.protobuf.message
[
https://issues.apache.org/jira/browse/BEAM-9130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri closed BEAM-9130.
---
Fix Version/s: Not applicable
Resolution: Fixed
> sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with
> ImportError: No module named google.protobuf.message
> ---
>
> Key: BEAM-9130
> URL: https://issues.apache.org/jira/browse/BEAM-9130
> Project: Beam
> Issue Type: Improvement
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Udi Meiri
>Priority: Major
> Labels: currently-failing
> Fix For: Not applicable
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> From logs:
> {noformat}
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 267, in _bootstrap
> 16:33:50 [0m[91mself.run()
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 114, in run
> 16:33:50 self._target(*self._args, **self._kwargs)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 357, in
> _install_grpcio_tools_and_generate_proto_files
> 16:33:50 generate_proto_files(force=force)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 324, in
> generate_proto_files
> 16:33:50 generate_urn_files(log, out_dir)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 65, in
> generate_urn_files
> 16:33:50 import google.protobuf.message as message
> 16:33:50 [0m[91mImportError: No module named google.protobuf.message
> 16:33:50 [0m[91mTraceback (most recent call last):
> 16:33:50File "setup.py", line 305, in
> 16:33:50 'mypy': generate_protos_first(mypy),
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/__init__.py", line 145, in
> setup
> 16:33:50 [0m[91mreturn distutils.core.setup(**attrs)
> 16:33:50File "/usr/local/lib/python2.7/distutils/core.py", line 151, in
> setup
> 16:33:50 [0m[91mdist.run_commands()
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 953, in
> run_commands
> 16:33:50 [0m[91mself.run_command(cmd)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 cmd_obj.run()
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/command/sdist.py", line
> 44, in run
> 16:33:50 [0m[91mself.run_command('egg_info')
> 16:33:50File "/usr/local/lib/python2.7/distutils/cmd.py", line 326, in
> run_command
> 16:33:50 [0m[91mself.distribution.run_command(command)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 [0m[91mcmd_obj.run()
> 16:33:50File "setup.py", line 229, in run
> 16:33:50 [0m[91mgen_protos.generate_proto_files(log=log)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 291, in
> generate_proto_files
> 16:33:50 raise ValueError("Proto generation failed (see log for
> details).")
> 16:33:50 [0m[91mValueError: [0m[91mProto generation failed (see log for
> details
> {noformat}
> {noformat}
> import google.protobuf.message as message
> ImportError: No module named google.protobuf.message
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-9130) sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with ImportError: No module named google.protobuf.message
[
https://issues.apache.org/jira/browse/BEAM-9130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri reassigned BEAM-9130:
---
Assignee: Udi Meiri
> sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with
> ImportError: No module named google.protobuf.message
> ---
>
> Key: BEAM-9130
> URL: https://issues.apache.org/jira/browse/BEAM-9130
> Project: Beam
> Issue Type: Improvement
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Udi Meiri
>Priority: Major
> Labels: currently-failing
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> From logs:
> {noformat}
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 267, in _bootstrap
> 16:33:50 [0m[91mself.run()
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 114, in run
> 16:33:50 self._target(*self._args, **self._kwargs)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 357, in
> _install_grpcio_tools_and_generate_proto_files
> 16:33:50 generate_proto_files(force=force)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 324, in
> generate_proto_files
> 16:33:50 generate_urn_files(log, out_dir)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 65, in
> generate_urn_files
> 16:33:50 import google.protobuf.message as message
> 16:33:50 [0m[91mImportError: No module named google.protobuf.message
> 16:33:50 [0m[91mTraceback (most recent call last):
> 16:33:50File "setup.py", line 305, in
> 16:33:50 'mypy': generate_protos_first(mypy),
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/__init__.py", line 145, in
> setup
> 16:33:50 [0m[91mreturn distutils.core.setup(**attrs)
> 16:33:50File "/usr/local/lib/python2.7/distutils/core.py", line 151, in
> setup
> 16:33:50 [0m[91mdist.run_commands()
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 953, in
> run_commands
> 16:33:50 [0m[91mself.run_command(cmd)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 cmd_obj.run()
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/command/sdist.py", line
> 44, in run
> 16:33:50 [0m[91mself.run_command('egg_info')
> 16:33:50File "/usr/local/lib/python2.7/distutils/cmd.py", line 326, in
> run_command
> 16:33:50 [0m[91mself.distribution.run_command(command)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 [0m[91mcmd_obj.run()
> 16:33:50File "setup.py", line 229, in run
> 16:33:50 [0m[91mgen_protos.generate_proto_files(log=log)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 291, in
> generate_proto_files
> 16:33:50 raise ValueError("Proto generation failed (see log for
> details).")
> 16:33:50 [0m[91mValueError: [0m[91mProto generation failed (see log for
> details
> {noformat}
> {noformat}
> import google.protobuf.message as message
> ImportError: No module named google.protobuf.message
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9130) sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with ImportError: No module named google.protobuf.message
[
https://issues.apache.org/jira/browse/BEAM-9130?focusedWorklogId=373791=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373791
]
ASF GitHub Bot logged work on BEAM-9130:
Author: ASF GitHub Bot
Created on: 17/Jan/20 18:58
Start Date: 17/Jan/20 18:58
Worklog Time Spent: 10m
Work Description: udim commented on pull request #10629: [BEAM-9130]
Migrate HDFS IT to use tox env.
URL: https://github.com/apache/beam/pull/10629
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373791)
Time Spent: 1.5h (was: 1h 20m)
> sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with
> ImportError: No module named google.protobuf.message
> ---
>
> Key: BEAM-9130
> URL: https://issues.apache.org/jira/browse/BEAM-9130
> Project: Beam
> Issue Type: Improvement
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Priority: Major
> Labels: currently-failing
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> From logs:
> {noformat}
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 267, in _bootstrap
> 16:33:50 [0m[91mself.run()
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 114, in run
> 16:33:50 self._target(*self._args, **self._kwargs)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 357, in
> _install_grpcio_tools_and_generate_proto_files
> 16:33:50 generate_proto_files(force=force)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 324, in
> generate_proto_files
> 16:33:50 generate_urn_files(log, out_dir)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 65, in
> generate_urn_files
> 16:33:50 import google.protobuf.message as message
> 16:33:50 [0m[91mImportError: No module named google.protobuf.message
> 16:33:50 [0m[91mTraceback (most recent call last):
> 16:33:50File "setup.py", line 305, in
> 16:33:50 'mypy': generate_protos_first(mypy),
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/__init__.py", line 145, in
> setup
> 16:33:50 [0m[91mreturn distutils.core.setup(**attrs)
> 16:33:50File "/usr/local/lib/python2.7/distutils/core.py", line 151, in
> setup
> 16:33:50 [0m[91mdist.run_commands()
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 953, in
> run_commands
> 16:33:50 [0m[91mself.run_command(cmd)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 cmd_obj.run()
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/command/sdist.py", line
> 44, in run
> 16:33:50 [0m[91mself.run_command('egg_info')
> 16:33:50File "/usr/local/lib/python2.7/distutils/cmd.py", line 326, in
> run_command
> 16:33:50 [0m[91mself.distribution.run_command(command)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 [0m[91mcmd_obj.run()
> 16:33:50File "setup.py", line 229, in run
> 16:33:50 [0m[91mgen_protos.generate_proto_files(log=log)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 291, in
> generate_proto_files
> 16:33:50 raise ValueError("Proto generation failed (see log for
> details).")
> 16:33:50 [0m[91mValueError: [0m[91mProto generation failed (see log for
> details
> {noformat}
> {noformat}
> import google.protobuf.message as message
> ImportError: No module named google.protobuf.message
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9144) Beam's own Avro TimeConversion class in beam-sdk-java-core
[
https://issues.apache.org/jira/browse/BEAM-9144?focusedWorklogId=373789=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373789
]
ASF GitHub Bot logged work on BEAM-9144:
Author: ASF GitHub Bot
Created on: 17/Jan/20 18:57
Start Date: 17/Jan/20 18:57
Worklog Time Spent: 10m
Work Description: suztomo commented on issue #10628: [BEAM-9144] Beam's
own Avro TimeConversion class in beam-sdk-java-core
URL: https://github.com/apache/beam/pull/10628#issuecomment-575752067
Java PreCommit failed:
```
12:51:00 Execution failed for task ':sdks:java:io:rabbitmq:test'.
12:51:00 > Process 'Gradle Test Executor 101' finished with non-zero exit
value 143
```
https://builds.apache.org/job/beam_PreCommit_Java_Commit/9677/console
I believe it's transient (It worked in my local)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373789)
Time Spent: 1h (was: 50m)
> Beam's own Avro TimeConversion class in beam-sdk-java-core
> ---
>
> Key: BEAM-9144
> URL: https://issues.apache.org/jira/browse/BEAM-9144
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Fix For: 2.19.0
>
> Attachments: avro-beam-dependency-graph.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> From Aaron's comment in
> https://issues.apache.org/jira/browse/BEAM-8388?focusedCommentId=17016476=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17016476
> .
> {quote}My org must use Avro 1.9.x (due to some Avro schema resolution issues
> resolved in 1.9.x) so downgrading Avro is not possible for us.
> Beam 2.16.0 is compatible with our usage of Avro 1.9.x – but upgrading to
> 2.17.0 we are broken as 2.17.0 links to Java classes in Avro 1.8.x that are
> not available in 1.9.x.
> {quote}
> The Java class is
> {{org.apache.avro.data.TimeConversions.TimestampConversion}} in Avro 1.8.
> It's renamed to {{org.apache.avro.data.JodaTimeConversions}} in Avro 1.9.
> h1. Beam Java SDK cannot upgrade Avro to 1.9
> Beam has Spark runners and Spark has not yet upgraded to Avro 1.9.
> Illustration of the dependency
> !avro-beam-dependency-graph.png|width=799,height=385!
> h1. Short-term Solution
> As illustrated above, as long as Beam Java SDK uses only the intersection of
> Avro classes, method, and fields between Avro 1.8 and 1.9, it will provide
> flexibility in runtime Avro versions (as it did until Beam 2.16).
> h2. Difference of the TimeConversion Classes
> Avro 1.9's TimestampConversion overrides {{getRecommendedSchema}} method.
> Details below:
> Avro 1.8's TimeConversions.TimestampConversion:
> {code:java}
> public static class TimestampConversion extends Conversion {
> @Override
> public Class getConvertedType() {
> return DateTime.class;
> }
> @Override
> public String getLogicalTypeName() {
> return "timestamp-millis";
> }
> @Override
> public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType
> type) {
> return new DateTime(millisFromEpoch, DateTimeZone.UTC);
> }
> @Override
> public Long toLong(DateTime timestamp, Schema schema, LogicalType type) {
> return timestamp.getMillis();
> }
> }
> {code}
> Avro 1.9's JodaTimeConversions.TimestampConversion:
> {code:java}
> public static class TimestampConversion extends Conversion {
> @Override
> public Class getConvertedType() {
> return DateTime.class;
> }
> @Override
> public String getLogicalTypeName() {
> return "timestamp-millis";
> }
> @Override
> public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType
> type) {
> return new DateTime(millisFromEpoch, DateTimeZone.UTC);
> }
> @Override
> public Long toLong(DateTime timestamp, Schema schema, LogicalType type) {
> return timestamp.getMillis();
> }
> @Override
> public Schema getRecommendedSchema() {
> return
> LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG));
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9130) sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with ImportError: No module named google.protobuf.message
[
https://issues.apache.org/jira/browse/BEAM-9130?focusedWorklogId=373790=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373790
]
ASF GitHub Bot logged work on BEAM-9130:
Author: ASF GitHub Bot
Created on: 17/Jan/20 18:57
Start Date: 17/Jan/20 18:57
Worklog Time Spent: 10m
Work Description: udim commented on issue #10629: [BEAM-9130] Migrate
HDFS IT to use tox env.
URL: https://github.com/apache/beam/pull/10629#issuecomment-575752267
Opened https://issues.apache.org/jira/browse/BEAM-9148 for flaky postcommit
test (test_big_query_standard_sql).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373790)
Time Spent: 1h 20m (was: 1h 10m)
> sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with
> ImportError: No module named google.protobuf.message
> ---
>
> Key: BEAM-9130
> URL: https://issues.apache.org/jira/browse/BEAM-9130
> Project: Beam
> Issue Type: Improvement
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Priority: Major
> Labels: currently-failing
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> From logs:
> {noformat}
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 267, in _bootstrap
> 16:33:50 [0m[91mself.run()
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 114, in run
> 16:33:50 self._target(*self._args, **self._kwargs)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 357, in
> _install_grpcio_tools_and_generate_proto_files
> 16:33:50 generate_proto_files(force=force)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 324, in
> generate_proto_files
> 16:33:50 generate_urn_files(log, out_dir)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 65, in
> generate_urn_files
> 16:33:50 import google.protobuf.message as message
> 16:33:50 [0m[91mImportError: No module named google.protobuf.message
> 16:33:50 [0m[91mTraceback (most recent call last):
> 16:33:50File "setup.py", line 305, in
> 16:33:50 'mypy': generate_protos_first(mypy),
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/__init__.py", line 145, in
> setup
> 16:33:50 [0m[91mreturn distutils.core.setup(**attrs)
> 16:33:50File "/usr/local/lib/python2.7/distutils/core.py", line 151, in
> setup
> 16:33:50 [0m[91mdist.run_commands()
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 953, in
> run_commands
> 16:33:50 [0m[91mself.run_command(cmd)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 cmd_obj.run()
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/command/sdist.py", line
> 44, in run
> 16:33:50 [0m[91mself.run_command('egg_info')
> 16:33:50File "/usr/local/lib/python2.7/distutils/cmd.py", line 326, in
> run_command
> 16:33:50 [0m[91mself.distribution.run_command(command)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 [0m[91mcmd_obj.run()
> 16:33:50File "setup.py", line 229, in run
> 16:33:50 [0m[91mgen_protos.generate_proto_files(log=log)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 291, in
> generate_proto_files
> 16:33:50 raise ValueError("Proto generation failed (see log for
> details).")
> 16:33:50 [0m[91mValueError: [0m[91mProto generation failed (see log for
> details
> {noformat}
> {noformat}
> import google.protobuf.message as message
> ImportError: No module named google.protobuf.message
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9063) Migrate docker images to apache namespace.
[ https://issues.apache.org/jira/browse/BEAM-9063?focusedWorklogId=373787=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373787 ] ASF GitHub Bot logged work on BEAM-9063: Author: ASF GitHub Bot Created on: 17/Jan/20 18:56 Start Date: 17/Jan/20 18:56 Worklog Time Spent: 10m Work Description: ibzib commented on issue #10612: [NOT READY TO MERGE][BEAM-9063] migrate docker images to apache URL: https://github.com/apache/beam/pull/10612#issuecomment-575751659 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373787) Time Spent: 40m (was: 0.5h) > Migrate docker images to apache namespace. > -- > > Key: BEAM-9063 > URL: https://issues.apache.org/jira/browse/BEAM-9063 > Project: Beam > Issue Type: Task > Components: beam-community >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: Not applicable > > Time Spent: 40m > Remaining Estimate: 0h > > https://hub.docker.com/u/apache -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9148) test flakiness: BigQueryQueryToTableIT.test_big_query_standard_sql
[
https://issues.apache.org/jira/browse/BEAM-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri updated BEAM-9148:
Description:
There might be other flaky test cases from the same class, but I'm focusing on
test_big_query_standard_sql here.
{code}
19:39:12 ==
19:39:12 FAIL: test_big_query_standard_sql
(apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
19:39:12 --
19:39:12 Traceback (most recent call last):
19:39:12File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py",
line 172, in test_big_query_standard_sql
19:39:12 big_query_query_to_table_pipeline.run_bq_pipeline(options)
19:39:12File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_pipeline.py",
line 84, in run_bq_pipeline
19:39:12 result = p.run()
19:39:12File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/testing/test_pipeline.py",
line 112, in run
19:39:12 else test_runner_api))
19:39:12File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
line 461, in run
19:39:12 self._options).run(False)
19:39:12File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
line 474, in run
19:39:12 return self.runner.run_pipeline(self, self._options)
19:39:12File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/runners/direct/test_direct_runner.py",
line 53, in run_pipeline
19:39:12 hc_assert_that(self.result, pickler.loads(on_success_matcher))
19:39:12 AssertionError:
19:39:12 Expected: (Test pipeline expected terminated in state: DONE and
Expected checksum is 158a8ea1c254fcf40d4ed3e7c0242c3ea0a29e72)
19:39:12 but: Expected checksum is
158a8ea1c254fcf40d4ed3e7c0242c3ea0a29e72 Actual checksum is
da39a3ee5e6b4b0d3255bfef95601890afd80709
19:39:12
19:39:12 >> begin captured logging <<
19:39:12 root: DEBUG: Unhandled type_constraint: Union[]
19:39:12 root: DEBUG: Unhandled type_constraint: Union[]
19:39:12 apache_beam.runners.direct.direct_runner: INFO: Running pipeline with
DirectRunner.
19:39:12 apache_beam.io.gcp.bigquery_tools: DEBUG: Query SELECT * FROM (SELECT
"apple" as fruit) UNION ALL (SELECT "orange" as fruit) does not reference any
tables.
19:39:12 apache_beam.io.gcp.bigquery_tools: WARNING: Dataset
apache-beam-testing:temp_dataset_90f5797bdb5f4137af750399f91a8e66 does not
exist so we will create it as temporary with location=None
19:39:12 apache_beam.io.gcp.bigquery: DEBUG: Creating or getting table
with schema {'fields': [{'name': 'fruit',
'type': 'STRING', 'mode': 'NULLABLE'}]}.
19:39:12 apache_beam.io.gcp.bigquery_tools: DEBUG: Created the table with id
output_table
19:39:12 apache_beam.io.gcp.bigquery_tools: INFO: Created table
apache-beam-testing.python_query_to_table_15792323245106.output_table with
schema ]>. Result: ]>
19:39:12 selfLink:
'https://www.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/python_query_to_table_15792323245106/tables/output_table'
19:39:12 tableReference:
19:39:12 type: 'TABLE'>.
19:39:12 apache_beam.io.gcp.bigquery: DEBUG: Attempting to flush to all
destinations. Total buffered: 2
19:39:12 apache_beam.io.gcp.bigquery: DEBUG: Flushing data to
apache-beam-testing:python_query_to_table_15792323245106.output_table. Total 2
rows.
19:39:12 apache_beam.io.gcp.tests.bigquery_matcher: INFO: Attempting to
perform query SELECT fruit from
`python_query_to_table_15792323245106.output_table`; to BQ
19:39:12 google.auth.transport._http_client: DEBUG: Making request: GET
http://169.254.169.254
19:39:12 google.auth.transport._http_client: DEBUG: Making request: GET
http://metadata.google.internal/computeMetadata/v1/project/project-id
19:39:12 urllib3.util.retry: DEBUG: Converted retries value: 3 ->
Retry(total=3, connect=None, read=None, redirect=None, status=None)
19:39:12 google.auth.transport.requests: DEBUG: Making request: GET
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
19:39:12 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1):
metadata.google.internal:80
19:39:12 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80
"GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true
HTTP/1.1" 200 144
19:39:12 google.auth.transport.requests: DEBUG: Making request: GET
