[jira] [Work logged] (BEAM-9008) Add readAll() method to CassandraIO
[
https://issues.apache.org/jira/browse/BEAM-9008?focusedWorklogId=415904=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415904
]
ASF GitHub Bot logged work on BEAM-9008:
Author: ASF GitHub Bot
Created on: 04/Apr/20 02:27
Start Date: 04/Apr/20 02:27
Worklog Time Spent: 10m
Work Description: vmarquez commented on issue #10546: [BEAM-9008] Add
CassandraIO readAll method
URL: https://github.com/apache/beam/pull/10546#issuecomment-608958438
@iemejia I've fixed the merge conflicts now with a rebase and I'm ready for
further review. Thanks in advance!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 415904)
Time Spent: 5h 50m (was: 5h 40m)
> Add readAll() method to CassandraIO
> ---
>
> Key: BEAM-9008
> URL: https://issues.apache.org/jira/browse/BEAM-9008
> Project: Beam
> Issue Type: New Feature
> Components: io-java-cassandra
>Affects Versions: 2.16.0
>Reporter: vincent marquez
>Assignee: vincent marquez
>Priority: Minor
> Fix For: 2.21.0
>
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> When querying a large cassandra database, it's often *much* more useful to
> programatically generate the queries needed to to be run rather than reading
> all partitions and attempting some filtering.
> As an example:
> {code:java}
> public class Event {
>@PartitionKey(0) public UUID accountId;
>@PartitionKey(1)public String yearMonthDay;
>@ClusteringKey public UUID eventId;
>//other data...
> }{code}
> If there is ten years worth of data, you may want to only query one year's
> worth. Here each token range would represent one 'token' but all events for
> the day.
> {code:java}
> Set accounts = getRelevantAccounts();
> Set dateRange = generateDateRange("2018-01-01", "2019-01-01");
> PCollection tokens = generateTokens(accounts, dateRange);
> {code}
>
> I propose an additional _readAll()_ PTransform that can take a PCollection
> of token ranges and can return a PCollection of what the query would
> return.
> *Question: How much code should be in common between both methods?*
> Currently the read connector already groups all partitions into a List of
> Token Ranges, so it would be simple to refactor the current read() based
> method to a 'ParDo' based one and have them both share the same function.
> Reasons against sharing code between read and readAll
> * Not having the read based method return a BoundedSource connector would
> mean losing the ability to know the size of the data returned
> * Currently the CassandraReader executes all the grouped TokenRange queries
> *asynchronously* which is (maybe?) fine when all that's happening is
> splitting up all the partition ranges but terrible for executing potentially
> millions of queries.
> Reasons _for_ sharing code would be simplified code base and that both of
> the above issues would most likely have a negligable performance impact.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9008) Add readAll() method to CassandraIO
[
https://issues.apache.org/jira/browse/BEAM-9008?focusedWorklogId=415903=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415903
]
ASF GitHub Bot logged work on BEAM-9008:
Author: ASF GitHub Bot
Created on: 04/Apr/20 02:26
Start Date: 04/Apr/20 02:26
Worklog Time Spent: 10m
Work Description: vmarquez commented on issue #10546: [BEAM-9008] Add
CassandraIO readAll method
URL: https://github.com/apache/beam/pull/10546#issuecomment-607043571
Hi @iemejia I've been busy with work and life but was finally able to this
just about finished up! To get around the connection issue (not caching the
connection was causing the Cassandra tests to run for ~ten minutes!), by
passing in a ReadAll to the ReadFn, we're able to initiate a connection in
the setup method, but we can still 'dynamically' use the passed in Read to
generate specific queries or query ranges.
As for a more advanced connection pooling, I'd prefer to get this merged in
and then perhaps work on an additional PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 415903)
Time Spent: 5h 40m (was: 5.5h)
> Add readAll() method to CassandraIO
> ---
>
> Key: BEAM-9008
> URL: https://issues.apache.org/jira/browse/BEAM-9008
> Project: Beam
> Issue Type: New Feature
> Components: io-java-cassandra
>Affects Versions: 2.16.0
>Reporter: vincent marquez
>Assignee: vincent marquez
>Priority: Minor
> Fix For: 2.21.0
>
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> When querying a large cassandra database, it's often *much* more useful to
> programatically generate the queries needed to to be run rather than reading
> all partitions and attempting some filtering.
> As an example:
> {code:java}
> public class Event {
>@PartitionKey(0) public UUID accountId;
>@PartitionKey(1)public String yearMonthDay;
>@ClusteringKey public UUID eventId;
>//other data...
> }{code}
> If there is ten years worth of data, you may want to only query one year's
> worth. Here each token range would represent one 'token' but all events for
> the day.
> {code:java}
> Set accounts = getRelevantAccounts();
> Set dateRange = generateDateRange("2018-01-01", "2019-01-01");
> PCollection tokens = generateTokens(accounts, dateRange);
> {code}
>
> I propose an additional _readAll()_ PTransform that can take a PCollection
> of token ranges and can return a PCollection of what the query would
> return.
> *Question: How much code should be in common between both methods?*
> Currently the read connector already groups all partitions into a List of
> Token Ranges, so it would be simple to refactor the current read() based
> method to a 'ParDo' based one and have them both share the same function.
> Reasons against sharing code between read and readAll
> * Not having the read based method return a BoundedSource connector would
> mean losing the ability to know the size of the data returned
> * Currently the CassandraReader executes all the grouped TokenRange queries
> *asynchronously* which is (maybe?) fine when all that's happening is
> splitting up all the partition ranges but terrible for executing potentially
> millions of queries.
> Reasons _for_ sharing code would be simplified code base and that both of
> the above issues would most likely have a negligable performance impact.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-6217) Typos on javadoc
[
https://issues.apache.org/jira/browse/BEAM-6217?focusedWorklogId=415900=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415900
]
ASF GitHub Bot logged work on BEAM-6217:
Author: ASF GitHub Bot
Created on: 04/Apr/20 02:10
Start Date: 04/Apr/20 02:10
Worklog Time Spent: 10m
Work Description: pabloem commented on pull request #11310: [BEAM-6217]
Fixing typos on javadoc
URL: https://github.com/apache/beam/pull/11310
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 415900)
Time Spent: 1h 10m (was: 1h)
> Typos on javadoc
>
>
> Key: BEAM-6217
> URL: https://issues.apache.org/jira/browse/BEAM-6217
> Project: Beam
> Issue Type: Bug
> Components: website
>Reporter: Rose Nguyen
>Priority: Minor
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> On the javadocs [https://beam.apache.org/releases/javadoc/2.1.0/]
> *View*
> Change tranform->transform in "When a
> [{{ParDo}}|https://beam.apache.org/releases/javadoc/2.1.0/org/apache/beam/sdk/transforms/ParDo.html]
> tranform is processing a main input"
> *AfterWatermark*
> Change therafter->thereafter in "window and then immediately therafter when
> any late data arrives"
> Change Additionaly->Additionally in "Additionaly firings before or after the
> watermark "
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-6217) Typos on javadoc
[
https://issues.apache.org/jira/browse/BEAM-6217?focusedWorklogId=415899=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415899
]
ASF GitHub Bot logged work on BEAM-6217:
Author: ASF GitHub Bot
Created on: 04/Apr/20 02:09
Start Date: 04/Apr/20 02:09
Worklog Time Spent: 10m
Work Description: pabloem commented on issue #11310: [BEAM-6217] Fixing
typos on javadoc
URL: https://github.com/apache/beam/pull/11310#issuecomment-608955985
Thanks for your contribution!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 415899)
Time Spent: 1h (was: 50m)
> Typos on javadoc
>
>
> Key: BEAM-6217
> URL: https://issues.apache.org/jira/browse/BEAM-6217
> Project: Beam
> Issue Type: Bug
> Components: website
>Reporter: Rose Nguyen
>Priority: Minor
> Time Spent: 1h
> Remaining Estimate: 0h
>
> On the javadocs [https://beam.apache.org/releases/javadoc/2.1.0/]
> *View*
> Change tranform->transform in "When a
> [{{ParDo}}|https://beam.apache.org/releases/javadoc/2.1.0/org/apache/beam/sdk/transforms/ParDo.html]
> tranform is processing a main input"
> *AfterWatermark*
> Change therafter->thereafter in "window and then immediately therafter when
> any late data arrives"
> Change Additionaly->Additionally in "Additionaly firings before or after the
> watermark "
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?focusedWorklogId=415894=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415894 ] ASF GitHub Bot logged work on BEAM-8019: Author: ASF GitHub Bot Created on: 04/Apr/20 02:06 Start Date: 04/Apr/20 02:06 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11185: [BEAM-8019] Updates Python SDK to handle remote SDK coders and preserve tags added by remote SDKs and propagate restriction coders. URL: https://github.com/apache/beam/pull/11185#discussion_r403390955 ## File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py ## @@ -269,11 +277,9 @@ def visit_transform(self, transform_node): pcoll.element_type, transform_node.full_label) key_type, value_type = pcoll.element_type.tuple_types if transform_node.outputs: -from apache_beam.runners.portability.fn_api_runner.translations \ - import only_element key = ( -None if None in transform_node.outputs.keys() else only_element( -transform_node.outputs.keys())) +None if None in transform_node.outputs.keys() else +DataflowRunner._only_element(transform_node.outputs.keys())) Review comment: Done. I assume this is because there can be only one output here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415894) Time Spent: 16h (was: 15h 50m) > Support cross-language transforms for DataflowRunner > > > Key: BEAM-8019 > URL: https://issues.apache.org/jira/browse/BEAM-8019 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: Major > Time Spent: 16h > Remaining Estimate: 0h > > This is to capture the Beam changes needed for this task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8019) Support cross-language transforms for DataflowRunner
[
https://issues.apache.org/jira/browse/BEAM-8019?focusedWorklogId=415896=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415896
]
ASF GitHub Bot logged work on BEAM-8019:
Author: ASF GitHub Bot
Created on: 04/Apr/20 02:06
Start Date: 04/Apr/20 02:06
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #11185:
[BEAM-8019] Updates Python SDK to handle remote SDK coders and preserve tags
added by remote SDKs and propagate restriction coders.
URL: https://github.com/apache/beam/pull/11185#discussion_r403320577
##
File path: sdks/python/apache_beam/pipeline.py
##
@@ -1065,10 +1070,9 @@ def named_inputs(self):
input in enumerate(self.inputs)
if isinstance(input, pvalue.PCollection)
}
-side_inputs = {
-'side%s' % ix: si.pvalue
-for ix, si in enumerate(self.side_inputs)
-}
+side_inputs = {(SIDE_INPUT_PREFIX + '%s') % ix: si.pvalue
+ for ix,
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 415896)
Time Spent: 16h 10m (was: 16h)
> Support cross-language transforms for DataflowRunner
>
>
> Key: BEAM-8019
> URL: https://issues.apache.org/jira/browse/BEAM-8019
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py-core
>Reporter: Chamikara Madhusanka Jayalath
>Assignee: Chamikara Madhusanka Jayalath
>Priority: Major
> Time Spent: 16h 10m
> Remaining Estimate: 0h
>
> This is to capture the Beam changes needed for this task.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?focusedWorklogId=415895=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415895 ] ASF GitHub Bot logged work on BEAM-8019: Author: ASF GitHub Bot Created on: 04/Apr/20 02:06 Start Date: 04/Apr/20 02:06 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11185: [BEAM-8019] Updates Python SDK to handle remote SDK coders and preserve tags added by remote SDKs and propagate restriction coders. URL: https://github.com/apache/beam/pull/11185#discussion_r403391405 ## File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py ## @@ -597,14 +604,22 @@ def _get_side_input_encoding(self, input_encoding): }, } - def _get_encoded_output_coder(self, transform_node, window_value=True): + def _get_encoded_output_coder( + self, transform_node, window_value=True, output_tag=None): """Returns the cloud encoding of the coder for the output of a transform.""" -from apache_beam.runners.portability.fn_api_runner.translations import \ - only_element -if len(transform_node.outputs) == 1: - output_tag = only_element(transform_node.outputs.keys()) +external_transform = isinstance( Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415895) Time Spent: 16h 10m (was: 16h) > Support cross-language transforms for DataflowRunner > > > Key: BEAM-8019 > URL: https://issues.apache.org/jira/browse/BEAM-8019 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: Major > Time Spent: 16h 10m > Remaining Estimate: 0h > > This is to capture the Beam changes needed for this task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8019) Support cross-language transforms for DataflowRunner
[
https://issues.apache.org/jira/browse/BEAM-8019?focusedWorklogId=415892=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415892
]
ASF GitHub Bot logged work on BEAM-8019:
Author: ASF GitHub Bot
Created on: 04/Apr/20 02:06
Start Date: 04/Apr/20 02:06
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #11185:
[BEAM-8019] Updates Python SDK to handle remote SDK coders and preserve tags
added by remote SDKs and propagate restriction coders.
URL: https://github.com/apache/beam/pull/11185#discussion_r403351916
##
File path: sdks/python/apache_beam/pipeline.py
##
@@ -1133,29 +1141,67 @@ def from_runner_api(proto, # type:
beam_runner_api_pb2.PTransform
context # type: PipelineContext
):
# type: (...) -> AppliedPTransform
-def is_side_input(tag):
- # type: (str) -> bool
- # As per named_inputs() above.
- return tag.startswith('side')
+
+if common_urns.primitives.PAR_DO.urn == proto.spec.urn:
+ # Preserving side input tags.
+ from apache_beam.portability.api import beam_runner_api_pb2
+ payload = (
+ proto_utils.parse_Bytes(
+ proto.spec.payload, beam_runner_api_pb2.ParDoPayload))
+ side_input_tags = list(payload.side_inputs.keys())
+else:
+ side_input_tags = []
main_inputs = [
context.pcollections.get_by_id(id) for tag,
-id in proto.inputs.items() if not is_side_input(tag)
+id in proto.inputs.items() if tag not in side_input_tags
]
-# Ordering is important here.
-indexed_side_inputs = [
-(get_sideinput_index(tag), context.pcollections.get_by_id(id)) for tag,
-id in proto.inputs.items() if is_side_input(tag)
-]
-side_inputs = [si for _, si in sorted(indexed_side_inputs)]
+def is_python_side_input(tag):
+ # type: (str) -> bool
+ # As per named_inputs() above.
+ return re.match(SIDE_INPUT_REGEX, tag)
+
+uses_python_sideinput_tags = (
+is_python_side_input(side_input_tags[0]) if side_input_tags else False)
+
+if uses_python_sideinput_tags:
+ # Ordering is important here.
+ # TODO(BEAM-9635): use key, value pairs instead of depending on tags with
+ # index as a suffix.
+ indexed_side_inputs = [
+ (get_sideinput_index(tag), context.pcollections.get_by_id(id))
+ for tag,
+ id in proto.inputs.items() if tag in side_input_tags
+ ]
+ side_inputs = [si for _, si in sorted(indexed_side_inputs)]
+else:
+ side_inputs = [
+ context.pcollections.get_by_id(id) for tag,
+ id in proto.inputs.items() if tag in side_input_tags
+ ]
+
transform = ptransform.PTransform.from_runner_api(proto, context)
+if isinstance(transform, RunnerAPIPTransformHolder):
+ # For external transforms that are ParDos, we have to set side-inputs
+ # manually and preserve input tags.
+ transform.side_inputs = [pvalue.AsMultiMap(pc) for pc in side_inputs]
+ input_tags_to_preserve = {
+ context.pcollections.get_by_id(id): tag
+ for tag,
+ id in proto.inputs.items()
+ }
+else:
Review comment:
That's done in respective from_runner_api methods. For example,
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/core.py#L1361
But RunnerAPIPTransformHolder is a holder type that is constructed directly
so setting it here seemed like the best option (we may be able to move this to
constructor but that would require passing in additional parameters to get
access to inputs etc. since using Python only
pvalue.SideInputData.from_runner_api is not an option).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 415892)
Time Spent: 15h 50m (was: 15h 40m)
> Support cross-language transforms for DataflowRunner
>
>
> Key: BEAM-8019
> URL: https://issues.apache.org/jira/browse/BEAM-8019
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py-core
>Reporter: Chamikara Madhusanka Jayalath
>Assignee: Chamikara Madhusanka Jayalath
>Priority: Major
> Time Spent: 15h 50m
> Remaining Estimate: 0h
>
> This is to capture the Beam changes needed for this task.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?focusedWorklogId=415891=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415891 ] ASF GitHub Bot logged work on BEAM-8019: Author: ASF GitHub Bot Created on: 04/Apr/20 02:06 Start Date: 04/Apr/20 02:06 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11185: [BEAM-8019] Updates Python SDK to handle remote SDK coders and preserve tags added by remote SDKs and propagate restriction coders. URL: https://github.com/apache/beam/pull/11185#discussion_r403319053 ## File path: sdks/python/apache_beam/coders/coders.py ## @@ -1383,22 +1385,50 @@ def from_runner_api_parameter(payload, components, context): write_state_threshold=int(payload)) -class RunnerAPICoderHolder(Coder): +class CoderElementType(typehints.TypeConstraint): + """An element type that just holds a coder proto.""" + def __init__(self, coder_proto, context): +self.coder_proto = coder_proto Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415891) Time Spent: 15h 40m (was: 15.5h) > Support cross-language transforms for DataflowRunner > > > Key: BEAM-8019 > URL: https://issues.apache.org/jira/browse/BEAM-8019 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: Major > Time Spent: 15h 40m > Remaining Estimate: 0h > > This is to capture the Beam changes needed for this task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8019) Support cross-language transforms for DataflowRunner
[
https://issues.apache.org/jira/browse/BEAM-8019?focusedWorklogId=415897=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415897
]
ASF GitHub Bot logged work on BEAM-8019:
Author: ASF GitHub Bot
Created on: 04/Apr/20 02:06
Start Date: 04/Apr/20 02:06
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #11185:
[BEAM-8019] Updates Python SDK to handle remote SDK coders and preserve tags
added by remote SDKs and propagate restriction coders.
URL: https://github.com/apache/beam/pull/11185#discussion_r403393322
##
File path: sdks/python/apache_beam/typehints/typehints.py
##
@@ -1180,6 +1180,28 @@ def get_yielded_type(type_hint):
raise ValueError('%s is not iterable' % type_hint)
+def _coerce_to_kv_type_from_external_type(element_type_holder):
Review comment:
We don't. Removed :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 415897)
Time Spent: 16h 20m (was: 16h 10m)
> Support cross-language transforms for DataflowRunner
>
>
> Key: BEAM-8019
> URL: https://issues.apache.org/jira/browse/BEAM-8019
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py-core
>Reporter: Chamikara Madhusanka Jayalath
>Assignee: Chamikara Madhusanka Jayalath
>Priority: Major
> Time Spent: 16h 20m
> Remaining Estimate: 0h
>
> This is to capture the Beam changes needed for this task.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?focusedWorklogId=415893=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415893 ] ASF GitHub Bot logged work on BEAM-8019: Author: ASF GitHub Bot Created on: 04/Apr/20 02:06 Start Date: 04/Apr/20 02:06 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11185: [BEAM-8019] Updates Python SDK to handle remote SDK coders and preserve tags added by remote SDKs and propagate restriction coders. URL: https://github.com/apache/beam/pull/11185#discussion_r403323329 ## File path: sdks/python/apache_beam/pipeline.py ## @@ -1107,6 +,10 @@ def transform_to_runner_api(transform, # type: Optional[ptransform.PTransform] (transform_urn in Pipeline.sdk_transforms_with_environment())): environment_id = context.default_environment_id() +def _maybe_preserve_tag(new_tag, pc, input_tags_to_preserve): Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415893) Time Spent: 16h (was: 15h 50m) > Support cross-language transforms for DataflowRunner > > > Key: BEAM-8019 > URL: https://issues.apache.org/jira/browse/BEAM-8019 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: Major > Time Spent: 16h > Remaining Estimate: 0h > > This is to capture the Beam changes needed for this task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?focusedWorklogId=415890=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415890 ] ASF GitHub Bot logged work on BEAM-8019: Author: ASF GitHub Bot Created on: 04/Apr/20 02:06 Start Date: 04/Apr/20 02:06 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11185: [BEAM-8019] Updates Python SDK to handle remote SDK coders and preserve tags added by remote SDKs and propagate restriction coders. URL: https://github.com/apache/beam/pull/11185#discussion_r403251962 ## File path: sdks/python/apache_beam/coders/coders.py ## @@ -370,7 +370,8 @@ def from_runner_api(cls, coder_proto, context): except Exception: if context.allow_proto_holders: # ignore this typing scenario for now, since it can't be easily tracked -return RunnerAPICoderHolder(coder_proto) # type: ignore +return ExternalCoder( Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415890) Time Spent: 15.5h (was: 15h 20m) > Support cross-language transforms for DataflowRunner > > > Key: BEAM-8019 > URL: https://issues.apache.org/jira/browse/BEAM-8019 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: Major > Time Spent: 15.5h > Remaining Estimate: 0h > > This is to capture the Beam changes needed for this task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8019) Support cross-language transforms for DataflowRunner
[
https://issues.apache.org/jira/browse/BEAM-8019?focusedWorklogId=415898=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415898
]
ASF GitHub Bot logged work on BEAM-8019:
Author: ASF GitHub Bot
Created on: 04/Apr/20 02:06
Start Date: 04/Apr/20 02:06
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #11185:
[BEAM-8019] Updates Python SDK to handle remote SDK coders and preserve tags
added by remote SDKs and propagate restriction coders.
URL: https://github.com/apache/beam/pull/11185#discussion_r403354509
##
File path: sdks/python/apache_beam/pipeline.py
##
@@ -1128,29 +1136,67 @@ def from_runner_api(proto, # type:
beam_runner_api_pb2.PTransform
context # type: PipelineContext
):
# type: (...) -> AppliedPTransform
-def is_side_input(tag):
+def is_python_side_input(tag):
# type: (str) -> bool
# As per named_inputs() above.
- return tag.startswith('side')
+ return re.match(SIDE_INPUT_REGEX, tag)
+
+side_input_tags = []
+if common_urns.primitives.PAR_DO.urn == proto.spec.urn:
+ # Preserving side input tags.
+ from apache_beam.utils import proto_utils
+ from apache_beam.portability.api import beam_runner_api_pb2
+ payload = (
+ proto_utils.parse_Bytes(
+ proto.spec.payload, beam_runner_api_pb2.ParDoPayload))
+ for tag, si in payload.side_inputs.items():
+side_input_tags.append(tag)
main_inputs = [
context.pcollections.get_by_id(id) for tag,
-id in proto.inputs.items() if not is_side_input(tag)
+id in proto.inputs.items() if tag not in side_input_tags
]
-# Ordering is important here.
-indexed_side_inputs = [
-(get_sideinput_index(tag), context.pcollections.get_by_id(id)) for tag,
-id in proto.inputs.items() if is_side_input(tag)
-]
+# Using a list here so that we can pass this into a function
+# TODO: use nonlocal after fully migrated to Python3.
+next_index = [0]
+
+def _get_sideinput_index(tag, next_index):
+ if is_python_side_input(tag):
+return get_sideinput_index(tag)
+ else:
+index = next_index[0]
+next_index[0] = next_index[0] + 1
+return index
+
+# Ordering is important here for Python sideinputs.
+indexed_side_inputs = [(
+_get_sideinput_index(tag, next_index),
+context.pcollections.get_by_id(id)) for tag,
+ id in proto.inputs.items() if tag in
side_input_tags]
side_inputs = [si for _, si in sorted(indexed_side_inputs)]
+
+input_tags_to_preserve = {}
+
transform = ptransform.PTransform.from_runner_api(proto, context)
+if isinstance(transform, RunnerAPIPTransformHolder):
+ # For external transforms that are ParDos, we have to set side-inputs
+ # manually and preserve input tags.
+ transform.side_inputs = [pvalue.AsMultiMap(pc) for pc in side_inputs]
Review comment:
Unfortunately 'pvalue.SideInputData.from_runner_api' is Python only and
fails for external side inputs:
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/pvalue.py#L463
So enumerated the two supported types here and added a TODO.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 415898)
Time Spent: 16.5h (was: 16h 20m)
> Support cross-language transforms for DataflowRunner
>
>
> Key: BEAM-8019
> URL: https://issues.apache.org/jira/browse/BEAM-8019
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py-core
>Reporter: Chamikara Madhusanka Jayalath
>Assignee: Chamikara Madhusanka Jayalath
>Priority: Major
> Time Spent: 16.5h
> Remaining Estimate: 0h
>
> This is to capture the Beam changes needed for this task.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9578) Enumerating artifacts is too expensive in Java
[
https://issues.apache.org/jira/browse/BEAM-9578?focusedWorklogId=415882=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415882
]
ASF GitHub Bot logged work on BEAM-9578:
Author: ASF GitHub Bot
Created on: 04/Apr/20 01:44
Start Date: 04/Apr/20 01:44
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #11205: [BEAM-9578]
Enumerating artifacts is too expensive in Java
URL: https://github.com/apache/beam/pull/11205#discussion_r403407082
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1206,6 +1210,15 @@ message MavenPayload {
string repository_url = 2;
}
+message DeferredArtifactPayload {
+ // A unique string identifier assigned by the creator of this payload. The
creator may use this key to confirm
+ // whether they can parse the data.
+ string key = 1;
Review comment:
This is going to have to get revamped for XLang and since it isn't being
exported outside of the SDK for portable runners we can easily change it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 415882)
Time Spent: 6h 20m (was: 6h 10m)
> Enumerating artifacts is too expensive in Java
> --
>
> Key: BEAM-9578
> URL: https://issues.apache.org/jira/browse/BEAM-9578
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Heejong Lee
>Priority: Critical
> Labels: portability
> Fix For: 2.21.0
>
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> There are a lot of places (e.g. *ParDoTranslation#getParDoPayload*) which
> effectively call *Environments#createOrGetDefaultEnvironment* which causes
> [artifacts to be
> computed|https://github.com/apache/beam/blob/fc6cef9972780ca6b7525d4aadd65a8344221f1b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java#L114].
> This leads to zipping directories for non-jar dependencies.
> Similar problems may exist for Python/Go.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9578) Enumerating artifacts is too expensive in Java
[ https://issues.apache.org/jira/browse/BEAM-9578?focusedWorklogId=415880=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415880 ] ASF GitHub Bot logged work on BEAM-9578: Author: ASF GitHub Bot Created on: 04/Apr/20 01:43 Start Date: 04/Apr/20 01:43 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11205: [BEAM-9578] Enumerating artifacts is too expensive in Java URL: https://github.com/apache/beam/pull/11205#issuecomment-608952454 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415880) Time Spent: 6h 10m (was: 6h) > Enumerating artifacts is too expensive in Java > -- > > Key: BEAM-9578 > URL: https://issues.apache.org/jira/browse/BEAM-9578 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Luke Cwik >Assignee: Heejong Lee >Priority: Critical > Labels: portability > Fix For: 2.21.0 > > Time Spent: 6h 10m > Remaining Estimate: 0h > > There are a lot of places (e.g. *ParDoTranslation#getParDoPayload*) which > effectively call *Environments#createOrGetDefaultEnvironment* which causes > [artifacts to be > computed|https://github.com/apache/beam/blob/fc6cef9972780ca6b7525d4aadd65a8344221f1b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java#L114]. > This leads to zipping directories for non-jar dependencies. > Similar problems may exist for Python/Go. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9578) Enumerating artifacts is too expensive in Java
[
https://issues.apache.org/jira/browse/BEAM-9578?focusedWorklogId=415873=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415873
]
ASF GitHub Bot logged work on BEAM-9578:
Author: ASF GitHub Bot
Created on: 04/Apr/20 01:31
Start Date: 04/Apr/20 01:31
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #11205: [BEAM-9578]
Enumerating artifacts is too expensive in Java
URL: https://github.com/apache/beam/pull/11205#discussion_r403403766
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1206,6 +1210,15 @@ message MavenPayload {
string repository_url = 2;
}
+message DeferredArtifactPayload {
+ // A unique string identifier assigned by the creator of this payload. The
creator may use this key to confirm
+ // whether they can parse the data.
+ string key = 1;
Review comment:
Should this be uid? Any collisions here could be bad...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 415873)
Time Spent: 6h (was: 5h 50m)
> Enumerating artifacts is too expensive in Java
> --
>
> Key: BEAM-9578
> URL: https://issues.apache.org/jira/browse/BEAM-9578
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Heejong Lee
>Priority: Critical
> Labels: portability
> Fix For: 2.21.0
>
> Time Spent: 6h
> Remaining Estimate: 0h
>
> There are a lot of places (e.g. *ParDoTranslation#getParDoPayload*) which
> effectively call *Environments#createOrGetDefaultEnvironment* which causes
> [artifacts to be
> computed|https://github.com/apache/beam/blob/fc6cef9972780ca6b7525d4aadd65a8344221f1b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java#L114].
> This leads to zipping directories for non-jar dependencies.
> Similar problems may exist for Python/Go.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9578) Enumerating artifacts is too expensive in Java
[
https://issues.apache.org/jira/browse/BEAM-9578?focusedWorklogId=415874=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415874
]
ASF GitHub Bot logged work on BEAM-9578:
Author: ASF GitHub Bot
Created on: 04/Apr/20 01:31
Start Date: 04/Apr/20 01:31
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #11205: [BEAM-9578]
Enumerating artifacts is too expensive in Java
URL: https://github.com/apache/beam/pull/11205#discussion_r403405278
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
##
@@ -214,24 +220,90 @@ public static Environment createProcessEnvironment(
pathsToStage.addAll(stagingFiles);
}
-ImmutableList.Builder filesToStage =
ImmutableList.builder();
+ImmutableList.Builder> lazyArtifactsBuilder =
+ImmutableList.builder();
for (String path : pathsToStage) {
Review comment:
Don't we want this for loop to be lazy?
Rather than introducing intermediate streams of Suppliers, I think we could
just rename the existing `getArtifacts()` something like
`getNonDeferredArtifacts()` and then call it during resolution.
```
if (key.equals(deferredArtifactPayload.getKey())) {
return getNonDeferredArtifacts(options);
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 415874)
Time Spent: 6h (was: 5h 50m)
> Enumerating artifacts is too expensive in Java
> --
>
> Key: BEAM-9578
> URL: https://issues.apache.org/jira/browse/BEAM-9578
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Heejong Lee
>Priority: Critical
> Labels: portability
> Fix For: 2.21.0
>
> Time Spent: 6h
> Remaining Estimate: 0h
>
> There are a lot of places (e.g. *ParDoTranslation#getParDoPayload*) which
> effectively call *Environments#createOrGetDefaultEnvironment* which causes
> [artifacts to be
> computed|https://github.com/apache/beam/blob/fc6cef9972780ca6b7525d4aadd65a8344221f1b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java#L114].
> This leads to zipping directories for non-jar dependencies.
> Similar problems may exist for Python/Go.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9578) Enumerating artifacts is too expensive in Java
[
https://issues.apache.org/jira/browse/BEAM-9578?focusedWorklogId=415875=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415875
]
ASF GitHub Bot logged work on BEAM-9578:
Author: ASF GitHub Bot
Created on: 04/Apr/20 01:31
Start Date: 04/Apr/20 01:31
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #11205: [BEAM-9578]
Enumerating artifacts is too expensive in Java
URL: https://github.com/apache/beam/pull/11205#discussion_r403403332
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/DefaultArtifactResolver.java
##
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction;
+
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+
+/**
+ * A default artifact resolver. This resolver applies {@link ResolutionFn} in
the reversed order
+ * they registered i.e. the function registered later overrides the earlier
one if they resolve the
+ * same artifact.
+ */
+public class DefaultArtifactResolver implements ArtifactResolver {
+ public static final ArtifactResolver INSTANCE = new
DefaultArtifactResolver();
+
+ private List fns =
+ Lists.newArrayList(
+ (info) -> {
+if
(BeamUrns.getUrn(RunnerApi.StandardArtifacts.Types.FILE).equals(info.getTypeUrn()))
{
+ return ImmutableList.of(info);
+} else {
+ return ImmutableList.of();
Review comment:
Is the empty list special? In particular sometimes a deferred artifact may
resolve to nothing, which is different than not being able to be resolved... I
think we still need optional or null or an exception to denote unresolveable by
this resolver.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 415875)
Time Spent: 6h (was: 5h 50m)
> Enumerating artifacts is too expensive in Java
> --
>
> Key: BEAM-9578
> URL: https://issues.apache.org/jira/browse/BEAM-9578
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Heejong Lee
>Priority: Critical
> Labels: portability
> Fix For: 2.21.0
>
> Time Spent: 6h
> Remaining Estimate: 0h
>
> There are a lot of places (e.g. *ParDoTranslation#getParDoPayload*) which
> effectively call *Environments#createOrGetDefaultEnvironment* which causes
> [artifacts to be
> computed|https://github.com/apache/beam/blob/fc6cef9972780ca6b7525d4aadd65a8344221f1b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java#L114].
> This leads to zipping directories for non-jar dependencies.
> Similar problems may exist for Python/Go.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-6217) Typos on javadoc
[
https://issues.apache.org/jira/browse/BEAM-6217?focusedWorklogId=415867=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415867
]
ASF GitHub Bot logged work on BEAM-6217:
Author: ASF GitHub Bot
Created on: 04/Apr/20 01:14
Start Date: 04/Apr/20 01:14
Worklog Time Spent: 10m
Work Description: pabloem commented on issue #11310: [BEAM-6217] Fixing
typos on javadoc
URL: https://github.com/apache/beam/pull/11310#issuecomment-608948612
Run Java PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 415867)
Time Spent: 50m (was: 40m)
> Typos on javadoc
>
>
> Key: BEAM-6217
> URL: https://issues.apache.org/jira/browse/BEAM-6217
> Project: Beam
> Issue Type: Bug
> Components: website
>Reporter: Rose Nguyen
>Priority: Minor
> Time Spent: 50m
> Remaining Estimate: 0h
>
> On the javadocs [https://beam.apache.org/releases/javadoc/2.1.0/]
> *View*
> Change tranform->transform in "When a
> [{{ParDo}}|https://beam.apache.org/releases/javadoc/2.1.0/org/apache/beam/sdk/transforms/ParDo.html]
> tranform is processing a main input"
> *AfterWatermark*
> Change therafter->thereafter in "window and then immediately therafter when
> any late data arrives"
> Change Additionaly->Additionally in "Additionaly firings before or after the
> watermark "
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9578) Enumerating artifacts is too expensive in Java
[
https://issues.apache.org/jira/browse/BEAM-9578?focusedWorklogId=415866=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415866
]
ASF GitHub Bot logged work on BEAM-9578:
Author: ASF GitHub Bot
Created on: 04/Apr/20 01:12
Start Date: 04/Apr/20 01:12
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #11205: [BEAM-9578]
Enumerating artifacts is too expensive in Java
URL: https://github.com/apache/beam/pull/11205#discussion_r403402959
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1206,6 +1210,11 @@ message MavenPayload {
string repository_url = 2;
}
+message DeferredArtifactPayload {
+ // A id for deferred artifacts.
+ string id = 1;
Review comment:
As discussed (but putting here for the record) having a proxy artifact type
could solve this issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 415866)
Time Spent: 5h 50m (was: 5h 40m)
> Enumerating artifacts is too expensive in Java
> --
>
> Key: BEAM-9578
> URL: https://issues.apache.org/jira/browse/BEAM-9578
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Heejong Lee
>Priority: Critical
> Labels: portability
> Fix For: 2.21.0
>
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> There are a lot of places (e.g. *ParDoTranslation#getParDoPayload*) which
> effectively call *Environments#createOrGetDefaultEnvironment* which causes
> [artifacts to be
> computed|https://github.com/apache/beam/blob/fc6cef9972780ca6b7525d4aadd65a8344221f1b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java#L114].
> This leads to zipping directories for non-jar dependencies.
> Similar problems may exist for Python/Go.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9519) BeamZetaSqlCalcRel$CalcFn.processElement field is not nullable in Array field $arrayx
[
https://issues.apache.org/jira/browse/BEAM-9519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9519:
-
Description:
One failure in shard 12, five failures in shard 16, one failure in shard 19,
fourteen failures on shard 50
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: FieldType{typeName=STRING, nullable=false,
logicalType=null, collectionElementType=null, mapKeyType=null,
mapValueType=null, rowSchema=null, metadata={}} is not nullable in Array field
$arraytz
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: FieldType{typeName=STRING,
nullable=false, logicalType=null, collectionElementType=null, mapKeyType=null,
mapValueType=null, rowSchema=null, metadata={}} is not nullable in Array field
$arraytz
at
org.apache.beam.sdk.values.SchemaVerification.verifyArray(SchemaVerification.java:101)
at
org.apache.beam.sdk.values.SchemaVerification.verifyFieldValue(SchemaVerification.java:65)
at
org.apache.beam.sdk.values.SchemaVerification.verifyRowValues(SchemaVerification.java:57)
at org.apache.beam.sdk.values.Row$Builder.build(Row.java:673)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:195)
{code}
{code}
Apr 01, 2020 6:02:53 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select timestamp("2011-02-03 10:20:30+05", tz)
from (select tz from unnest(["+02", NULL, "America/Los_Angeles"]) tz);
Apr 01, 2020 6:05:07 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT [1, 4, 9, NULL, 10] non_empty_array
Apr 01, 2020 6:05:08 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT [CAST(5 AS INT64), NULL] int64_val
Apr 01, 2020 6:05:09 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT [true, NULL, false, true] bool_val
Apr 01, 2020 6:05:09 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT [CAST("8" AS STRING), NULL,
CAST("googlesql" AS STRING)] string_val
Apr 01, 2020 6:05:10 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT [NULL, CAST(b"9" AS BYTES)] bytes_val
Apr 01, 2020 5:58:24 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT x FROM UNNEST([3, NULL]) x;
Apr 01, 2020 6:01:36 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO:
[jira] [Updated] (BEAM-9518) BeamZetaSqlCalcRel$CalcFn.processElement Unsupported Beam fieldType: LOGICAL_TYPE
[
https://issues.apache.org/jira/browse/BEAM-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9518:
-
Description:
eight failures in shard 6, eight failures in shard 7, eight failures in shard
8, ten failures in shard 9, five failures in shard 10, four failures in shard
11, six failures in shard 12, two failures in shard 16, six failures in shard
31, three failures in shard 42, two failures in shard 43, eight failures in
shard 44, two failures in shard 50
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSqlUtils.zetaSqlValueToJavaObject(ZetaSqlUtils.java:221)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:193)
{code}
{code}
Apr 01, 2020 6:00:21 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:03:59 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT DISTINCT val.DATE
from (select DATE"2014-01-01" DATE union all
select cast(NULL as date) union all
select DATE"2014-01-02" union all
select DATE"2014-01-01" union all
select cast(NULL as date)) val
Apr 01, 2020 6:10:09 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT COALESCE(NULL, DATE '2014-12-18')
Apr 01, 2020 6:03:26 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(null AS TIME),
CAST('12:34:56.123456' AS TIME),
CAST(CAST(null AS TIME) AS STRING),
CAST(CAST('12:34:56.123456' AS TIME) AS STRING),
CAST(TIME '12:34:56.123456' AS STRING),
CAST(CAST(TIME '12:34:56.123456' AS STRING) AS TIME)
Apr 01, 2020 5:59:59 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT SAFE_CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:07:07 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT
CAST(CAST(TIME "00:00:00" as STRING) as TIME),
CAST(CAST(TIME "23:59:59.99" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.123456" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.12345" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.12" as STRING) as TIME),
CAST(CAST(TIME "01:02:3" as STRING) as TIME),
CAST(CAST(TIME
[jira] [Updated] (BEAM-9517) BeamZetaSqlCalcRel$CalcFn.processElement Field is not nullable
[
https://issues.apache.org/jira/browse/BEAM-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9517:
-
Description:
two failures in shard 4, one failure in shard 12, nine failures in shard 15,
six failures in shard 17, one failure in shard 40, one failure in shard 41, one
failure in shard 42, one failure in shard 43, one failure in shard 44, one
failure in shard 45, one failure in shard 46, six failures in shard 50
{code:java}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.sdk.values.SchemaVerification.verifyRowValues(SchemaVerification.java:53)
at org.apache.beam.sdk.values.Row$Builder.build(Row.java:673)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:195)
{code}
{code}
Apr 01, 2020 5:56:51 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
Apr 01, 2020 5:56:52 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
Apr 01, 2020 6:02:50 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select string(NULL),
string(NULL, 'Americas/Los_Angeles'),
string(timestamp '2010-02-03 04:05:06', NULL),
string(NULL, NULL)
Apr 01, 2020 6:01:33 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_add(@p0, INTERVAL @p1 HOUR) AS
ColA
Apr 01, 2020 6:02:00 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_sub(@p0, INTERVAL @p1 HOUR) AS
ColA
Apr 01, 2020 6:02:44 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select tz, parse_timestamp('%y', '0', tz) from
Timezones
Apr 01, 2020 6:02:45 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select tz, format_timestamp('%y', timestamp
'2000-01-01 00:00:00', tz) from Timezones
Apr 01, 2020 6:02:45 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select tz, string(timestamp '2014-02-28
10:20:30', tz) from Timezones
Apr 01, 2020 6:04:04 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement:
[jira] [Updated] (BEAM-9517) BeamZetaSqlCalcRel$CalcFn.processElement Field is not nullable
[
https://issues.apache.org/jira/browse/BEAM-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9517:
-
Description:
two failures in shard 4, one failure in shard 12, nine failures in shard 15,
six failures in shard 17, one failure in shard 40, one failure in shard 41, one
failure in shard 42, one failure in shard 43, one failure in shard 44
{code:java}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.sdk.values.SchemaVerification.verifyRowValues(SchemaVerification.java:53)
at org.apache.beam.sdk.values.Row$Builder.build(Row.java:673)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:195)
{code}
{code}
Apr 01, 2020 5:56:51 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
Apr 01, 2020 5:56:52 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
Apr 01, 2020 6:02:50 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select string(NULL),
string(NULL, 'Americas/Los_Angeles'),
string(timestamp '2010-02-03 04:05:06', NULL),
string(NULL, NULL)
Apr 01, 2020 6:01:33 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_add(@p0, INTERVAL @p1 HOUR) AS
ColA
Apr 01, 2020 6:02:00 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_sub(@p0, INTERVAL @p1 HOUR) AS
ColA
Apr 01, 2020 6:02:44 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select tz, parse_timestamp('%y', '0', tz) from
Timezones
Apr 01, 2020 6:02:45 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select tz, format_timestamp('%y', timestamp
'2000-01-01 00:00:00', tz) from Timezones
Apr 01, 2020 6:02:45 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select tz, string(timestamp '2014-02-28
10:20:30', tz) from Timezones
Apr 01, 2020 6:04:04 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_seconds(@p0) AS ColA
Apr 01, 2020 6:04:05 PM
[jira] [Updated] (BEAM-9518) BeamZetaSqlCalcRel$CalcFn.processElement Unsupported Beam fieldType: LOGICAL_TYPE
[
https://issues.apache.org/jira/browse/BEAM-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9518:
-
Description:
eight failures in shard 6, eight failures in shard 7, eight failures in shard
8, ten failures in shard 9, five failures in shard 10, four failures in shard
11, six failures in shard 12, two failures in shard 16, six failures in shard
31, three failures in shard 42, two failures in shard 43, eight failures in
shard 44
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSqlUtils.zetaSqlValueToJavaObject(ZetaSqlUtils.java:221)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:193)
{code}
{code}
Apr 01, 2020 6:00:21 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:03:59 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT DISTINCT val.DATE
from (select DATE"2014-01-01" DATE union all
select cast(NULL as date) union all
select DATE"2014-01-02" union all
select DATE"2014-01-01" union all
select cast(NULL as date)) val
Apr 01, 2020 6:10:09 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT COALESCE(NULL, DATE '2014-12-18')
Apr 01, 2020 6:03:26 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(null AS TIME),
CAST('12:34:56.123456' AS TIME),
CAST(CAST(null AS TIME) AS STRING),
CAST(CAST('12:34:56.123456' AS TIME) AS STRING),
CAST(TIME '12:34:56.123456' AS STRING),
CAST(CAST(TIME '12:34:56.123456' AS STRING) AS TIME)
Apr 01, 2020 5:59:59 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT SAFE_CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:07:07 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT
CAST(CAST(TIME "00:00:00" as STRING) as TIME),
CAST(CAST(TIME "23:59:59.99" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.123456" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.12345" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.12" as STRING) as TIME),
CAST(CAST(TIME "01:02:3" as STRING) as TIME),
CAST(CAST(TIME "01:2:03" as STRING) as TIME),
[jira] [Updated] (BEAM-9518) BeamZetaSqlCalcRel$CalcFn.processElement Unsupported Beam fieldType: LOGICAL_TYPE
[
https://issues.apache.org/jira/browse/BEAM-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9518:
-
Description:
eight failures in shard 6, eight failures in shard 7, eight failures in shard
8, ten failures in shard 9, five failures in shard 10, four failures in shard
11, six failures in shard 12, two failures in shard 16, six failures in shard
31, three failures in shard 42, two failures in shard 43
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSqlUtils.zetaSqlValueToJavaObject(ZetaSqlUtils.java:221)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:193)
{code}
{code}
Apr 01, 2020 6:00:21 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:03:59 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT DISTINCT val.DATE
from (select DATE"2014-01-01" DATE union all
select cast(NULL as date) union all
select DATE"2014-01-02" union all
select DATE"2014-01-01" union all
select cast(NULL as date)) val
Apr 01, 2020 6:10:09 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT COALESCE(NULL, DATE '2014-12-18')
Apr 01, 2020 6:03:26 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(null AS TIME),
CAST('12:34:56.123456' AS TIME),
CAST(CAST(null AS TIME) AS STRING),
CAST(CAST('12:34:56.123456' AS TIME) AS STRING),
CAST(TIME '12:34:56.123456' AS STRING),
CAST(CAST(TIME '12:34:56.123456' AS STRING) AS TIME)
Apr 01, 2020 5:59:59 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT SAFE_CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:07:07 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT
CAST(CAST(TIME "00:00:00" as STRING) as TIME),
CAST(CAST(TIME "23:59:59.99" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.123456" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.12345" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.12" as STRING) as TIME),
CAST(CAST(TIME "01:02:3" as STRING) as TIME),
CAST(CAST(TIME "01:2:03" as STRING) as TIME),
CAST(CAST(TIME "1:02:03" as
[jira] [Updated] (BEAM-9517) BeamZetaSqlCalcRel$CalcFn.processElement Field is not nullable
[
https://issues.apache.org/jira/browse/BEAM-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9517:
-
Description:
two failures in shard 4, one failure in shard 12, nine failures in shard 15,
six failures in shard 17, one failure in shard 40, one failure in shard 41, one
failure in shard 42, one failure in shard 43
{code:java}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.sdk.values.SchemaVerification.verifyRowValues(SchemaVerification.java:53)
at org.apache.beam.sdk.values.Row$Builder.build(Row.java:673)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:195)
{code}
{code}
Apr 01, 2020 5:56:51 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
Apr 01, 2020 5:56:52 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
Apr 01, 2020 6:02:50 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select string(NULL),
string(NULL, 'Americas/Los_Angeles'),
string(timestamp '2010-02-03 04:05:06', NULL),
string(NULL, NULL)
Apr 01, 2020 6:01:33 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_add(@p0, INTERVAL @p1 HOUR) AS
ColA
Apr 01, 2020 6:02:00 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_sub(@p0, INTERVAL @p1 HOUR) AS
ColA
Apr 01, 2020 6:02:44 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select tz, parse_timestamp('%y', '0', tz) from
Timezones
Apr 01, 2020 6:02:45 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select tz, format_timestamp('%y', timestamp
'2000-01-01 00:00:00', tz) from Timezones
Apr 01, 2020 6:02:45 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select tz, string(timestamp '2014-02-28
10:20:30', tz) from Timezones
Apr 01, 2020 6:04:04 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_seconds(@p0) AS ColA
Apr 01, 2020 6:04:05 PM
[jira] [Updated] (BEAM-9517) BeamZetaSqlCalcRel$CalcFn.processElement Field is not nullable
[
https://issues.apache.org/jira/browse/BEAM-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9517:
-
Description:
two failures in shard 4, one failure in shard 12, nine failures in shard 15,
six failures in shard 17, one failure in shard 40, one failure in shard 41, one
failure in shard 42
{code:java}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.sdk.values.SchemaVerification.verifyRowValues(SchemaVerification.java:53)
at org.apache.beam.sdk.values.Row$Builder.build(Row.java:673)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:195)
{code}
{code}
Apr 01, 2020 5:56:51 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
Apr 01, 2020 5:56:52 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
Apr 01, 2020 6:02:50 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select string(NULL),
string(NULL, 'Americas/Los_Angeles'),
string(timestamp '2010-02-03 04:05:06', NULL),
string(NULL, NULL)
Apr 01, 2020 6:01:33 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_add(@p0, INTERVAL @p1 HOUR) AS
ColA
Apr 01, 2020 6:02:00 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_sub(@p0, INTERVAL @p1 HOUR) AS
ColA
Apr 01, 2020 6:02:44 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select tz, parse_timestamp('%y', '0', tz) from
Timezones
Apr 01, 2020 6:02:45 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select tz, format_timestamp('%y', timestamp
'2000-01-01 00:00:00', tz) from Timezones
Apr 01, 2020 6:02:45 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select tz, string(timestamp '2014-02-28
10:20:30', tz) from Timezones
Apr 01, 2020 6:04:04 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_seconds(@p0) AS ColA
Apr 01, 2020 6:04:05 PM
[jira] [Updated] (BEAM-9518) BeamZetaSqlCalcRel$CalcFn.processElement Unsupported Beam fieldType: LOGICAL_TYPE
[
https://issues.apache.org/jira/browse/BEAM-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9518:
-
Description:
eight failures in shard 6, eight failures in shard 7, eight failures in shard
8, ten failures in shard 9, five failures in shard 10, four failures in shard
11, six failures in shard 12, two failures in shard 16, six failures in shard
31, three failures in shard 42
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSqlUtils.zetaSqlValueToJavaObject(ZetaSqlUtils.java:221)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:193)
{code}
{code}
Apr 01, 2020 6:00:21 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:03:59 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT DISTINCT val.DATE
from (select DATE"2014-01-01" DATE union all
select cast(NULL as date) union all
select DATE"2014-01-02" union all
select DATE"2014-01-01" union all
select cast(NULL as date)) val
Apr 01, 2020 6:10:09 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT COALESCE(NULL, DATE '2014-12-18')
Apr 01, 2020 6:03:26 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(null AS TIME),
CAST('12:34:56.123456' AS TIME),
CAST(CAST(null AS TIME) AS STRING),
CAST(CAST('12:34:56.123456' AS TIME) AS STRING),
CAST(TIME '12:34:56.123456' AS STRING),
CAST(CAST(TIME '12:34:56.123456' AS STRING) AS TIME)
Apr 01, 2020 5:59:59 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT SAFE_CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:07:07 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT
CAST(CAST(TIME "00:00:00" as STRING) as TIME),
CAST(CAST(TIME "23:59:59.99" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.123456" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.12345" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.12" as STRING) as TIME),
CAST(CAST(TIME "01:02:3" as STRING) as TIME),
CAST(CAST(TIME "01:2:03" as STRING) as TIME),
CAST(CAST(TIME "1:02:03" as STRING) as TIME),
[jira] [Updated] (BEAM-9518) BeamZetaSqlCalcRel$CalcFn.processElement Unsupported Beam fieldType: LOGICAL_TYPE
[
https://issues.apache.org/jira/browse/BEAM-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9518:
-
Description:
eight failures in shard 6, eight failures in shard 7, eight failures in shard
8, ten failures in shard 9, five failures in shard 10, four failures in shard
11, six failures in shard 12, two failures in shard 16, six failures in shard 31
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSqlUtils.zetaSqlValueToJavaObject(ZetaSqlUtils.java:221)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:193)
{code}
{code}
Apr 01, 2020 6:00:21 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:03:59 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT DISTINCT val.DATE
from (select DATE"2014-01-01" DATE union all
select cast(NULL as date) union all
select DATE"2014-01-02" union all
select DATE"2014-01-01" union all
select cast(NULL as date)) val
Apr 01, 2020 6:10:09 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT COALESCE(NULL, DATE '2014-12-18')
Apr 01, 2020 6:03:26 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(null AS TIME),
CAST('12:34:56.123456' AS TIME),
CAST(CAST(null AS TIME) AS STRING),
CAST(CAST('12:34:56.123456' AS TIME) AS STRING),
CAST(TIME '12:34:56.123456' AS STRING),
CAST(CAST(TIME '12:34:56.123456' AS STRING) AS TIME)
Apr 01, 2020 5:59:59 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT SAFE_CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:07:07 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT
CAST(CAST(TIME "00:00:00" as STRING) as TIME),
CAST(CAST(TIME "23:59:59.99" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.123456" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.12345" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.12" as STRING) as TIME),
CAST(CAST(TIME "01:02:3" as STRING) as TIME),
CAST(CAST(TIME "01:2:03" as STRING) as TIME),
CAST(CAST(TIME "1:02:03" as STRING) as TIME),
#Leap second cases
[jira] [Updated] (BEAM-9517) BeamZetaSqlCalcRel$CalcFn.processElement Field is not nullable
[
https://issues.apache.org/jira/browse/BEAM-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9517:
-
Description:
two failures in shard 4, one failure in shard 12, nine failures in shard 15,
six failures in shard 17, one failure in shard 40, one failure in shard 41
{code:java}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.sdk.values.SchemaVerification.verifyRowValues(SchemaVerification.java:53)
at org.apache.beam.sdk.values.Row$Builder.build(Row.java:673)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:195)
{code}
{code}
Apr 01, 2020 5:56:51 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
Apr 01, 2020 5:56:52 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
Apr 01, 2020 6:02:50 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select string(NULL),
string(NULL, 'Americas/Los_Angeles'),
string(timestamp '2010-02-03 04:05:06', NULL),
string(NULL, NULL)
Apr 01, 2020 6:01:33 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_add(@p0, INTERVAL @p1 HOUR) AS
ColA
Apr 01, 2020 6:02:00 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_sub(@p0, INTERVAL @p1 HOUR) AS
ColA
Apr 01, 2020 6:02:44 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select tz, parse_timestamp('%y', '0', tz) from
Timezones
Apr 01, 2020 6:02:45 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select tz, format_timestamp('%y', timestamp
'2000-01-01 00:00:00', tz) from Timezones
Apr 01, 2020 6:02:45 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select tz, string(timestamp '2014-02-28
10:20:30', tz) from Timezones
Apr 01, 2020 6:04:04 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_seconds(@p0) AS ColA
Apr 01, 2020 6:04:05 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO:
[jira] [Updated] (BEAM-9517) BeamZetaSqlCalcRel$CalcFn.processElement Field is not nullable
[
https://issues.apache.org/jira/browse/BEAM-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9517:
-
Description:
two failures in shard 4, one failure in shard 12, nine failures in shard 15,
six failures in shard 17, one failure in shard 40
{code:java}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.sdk.values.SchemaVerification.verifyRowValues(SchemaVerification.java:53)
at org.apache.beam.sdk.values.Row$Builder.build(Row.java:673)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:195)
{code}
{code}
Apr 01, 2020 5:56:51 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
Apr 01, 2020 5:56:52 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
Apr 01, 2020 6:02:50 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select string(NULL),
string(NULL, 'Americas/Los_Angeles'),
string(timestamp '2010-02-03 04:05:06', NULL),
string(NULL, NULL)
Apr 01, 2020 6:01:33 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_add(@p0, INTERVAL @p1 HOUR) AS
ColA
Apr 01, 2020 6:02:00 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_sub(@p0, INTERVAL @p1 HOUR) AS
ColA
Apr 01, 2020 6:02:44 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select tz, parse_timestamp('%y', '0', tz) from
Timezones
Apr 01, 2020 6:02:45 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select tz, format_timestamp('%y', timestamp
'2000-01-01 00:00:00', tz) from Timezones
Apr 01, 2020 6:02:45 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select tz, string(timestamp '2014-02-28
10:20:30', tz) from Timezones
Apr 01, 2020 6:04:04 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_seconds(@p0) AS ColA
Apr 01, 2020 6:04:05 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql
[jira] [Updated] (BEAM-9693) BeamZetaSqlCalcRel getBoolValue IllegalStateException
[
https://issues.apache.org/jira/browse/BEAM-9693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9693:
-
Description:
three failures in shard 3, two failures in shard 31
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalStateException
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:491)
at com.google.zetasql.Value.getBoolValue(Value.java:225)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:184)
{code}
{code}
Apr 01, 2020 6:01:11 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT SUM(double_val) FROM TableLarge WHERE
double_val > 0
Apr 01, 2020 6:01:26 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT MIN(double_val), MAX(double_val) FROM
TableLarge WHERE double_val > 0
Apr 01, 2020 6:01:28 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT MIN(double_val), MAX(double_val) FROM
TableLarge WHERE double_val <= 0
Apr 01, 2020 6:05:55 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select sum(distinct_4) from
TableDistinctsWithNulls
group by distinct_2
having sum(distinct_4) > 14 or null
INFO: Processing Sql statement: select sum(distinct_4) foo1, distinct_2 foo2
from TableDistinctsWithNulls
group by distinct_2
having foo1 > 14 + foo2
{code}
was:
three failures in shard 3
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalStateException
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
[jira] [Updated] (BEAM-9520) Forbidden IOException when reading from InputStream
[
https://issues.apache.org/jira/browse/BEAM-9520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9520:
-
Description:
Six failures in shard 6, twelve failures in shard 22
{code:java}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: Forbidden IOException when reading from
InputStream
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Forbidden IOException when
reading from InputStream
at
org.apache.beam.sdk.util.CoderUtils.decodeFromSafeStream(CoderUtils.java:118)
at
org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:98)
at
org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:92)
at org.apache.beam.sdk.util.CoderUtils.clone(CoderUtils.java:141)
at
org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.(MutationDetectors.java:115)
at
org.apache.beam.sdk.util.MutationDetectors.forValueWithCoder(MutationDetectors.java:46)
at
org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.add(ImmutabilityCheckingBundleFactory.java:112)
at
org.apache.beam.runners.direct.ParDoEvaluator$BundleOutputManager.output(ParDoEvaluator.java:299)
at
org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:259)
at
org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.access$800(SimpleDoFnRunner.java:79)
at
org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:405)
at
org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:393)
at
org.apache.beam.sdk.transforms.join.CoGroupByKey$ConstructUnionTableFn.processElement(CoGroupByKey.java:175)
{code}
variant 1:
{code}
Caused by: java.io.EOFException: reached end of stream after reading 0 bytes; 1
bytes expected
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams.readFully(ByteStreams.java:780)
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams.readFully(ByteStreams.java:762)
at
org.apache.beam.sdk.coders.ByteArrayCoder.decode(ByteArrayCoder.java:108)
at org.apache.beam.sdk.coders.BitSetCoder.decode(BitSetCoder.java:57)
at org.apache.beam.sdk.coders.BitSetCoder.decode(BitSetCoder.java:52)
at
org.apache.beam.sdk.coders.RowCoderGenerator$DecodeInstruction.decodeDelegate(RowCoderGenerator.java:339)
at org.apache.beam.sdk.coders.Coder$ByteBuddy$Mp4cMJnv.decode(Unknown
Source)
at org.apache.beam.sdk.coders.Coder$ByteBuddy$Mp4cMJnv.decode(Unknown
Source)
at
[jira] [Created] (BEAM-9698) BeamUncollectRel UncollectDoFn NullPointerException
Andrew Pilloud created BEAM-9698:
Summary: BeamUncollectRel UncollectDoFn NullPointerException
Key: BEAM-9698
URL: https://issues.apache.org/jira/browse/BEAM-9698
Project: Beam
Issue Type: Bug
Components: dsl-sql-zetasql
Reporter: Andrew Pilloud
two failures in shard 19
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.NullPointerException
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamUncollectRel$UncollectDoFn.process(BeamUncollectRel.java:103)
{code}
{code}
Apr 01, 2020 5:58:27 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT e FROM UNNEST(CAST(NULL AS
ARRAY)) e
Apr 01, 2020 5:58:27 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT e FROM UNNEST(CAST(NULL AS
ARRAY>)) e
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9519) BeamZetaSqlCalcRel$CalcFn.processElement field is not nullable in Array field $arrayx
[
https://issues.apache.org/jira/browse/BEAM-9519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9519:
-
Description:
One failure in shard 12, five failures in shard 16, one failure in shard 19
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: FieldType{typeName=STRING, nullable=false,
logicalType=null, collectionElementType=null, mapKeyType=null,
mapValueType=null, rowSchema=null, metadata={}} is not nullable in Array field
$arraytz
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: FieldType{typeName=STRING,
nullable=false, logicalType=null, collectionElementType=null, mapKeyType=null,
mapValueType=null, rowSchema=null, metadata={}} is not nullable in Array field
$arraytz
at
org.apache.beam.sdk.values.SchemaVerification.verifyArray(SchemaVerification.java:101)
at
org.apache.beam.sdk.values.SchemaVerification.verifyFieldValue(SchemaVerification.java:65)
at
org.apache.beam.sdk.values.SchemaVerification.verifyRowValues(SchemaVerification.java:57)
at org.apache.beam.sdk.values.Row$Builder.build(Row.java:673)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:195)
{code}
{code}
Apr 01, 2020 6:02:53 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select timestamp("2011-02-03 10:20:30+05", tz)
from (select tz from unnest(["+02", NULL, "America/Los_Angeles"]) tz);
Apr 01, 2020 6:05:07 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT [1, 4, 9, NULL, 10] non_empty_array
Apr 01, 2020 6:05:08 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT [CAST(5 AS INT64), NULL] int64_val
Apr 01, 2020 6:05:09 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT [true, NULL, false, true] bool_val
Apr 01, 2020 6:05:09 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT [CAST("8" AS STRING), NULL,
CAST("googlesql" AS STRING)] string_val
Apr 01, 2020 6:05:10 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT [NULL, CAST(b"9" AS BYTES)] bytes_val
Apr 01, 2020 5:58:24 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT x FROM UNNEST([3, NULL]) x;
{code}
was:
One failure in shard 12, five failures in shard 16
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
[jira] [Created] (BEAM-9697) zetaSqlTimestampValueToJodaInstant IllegalStateException
Andrew Pilloud created BEAM-9697:
Summary: zetaSqlTimestampValueToJodaInstant IllegalStateException
Key: BEAM-9697
URL: https://issues.apache.org/jira/browse/BEAM-9697
Project: Beam
Issue Type: Bug
Components: dsl-sql-zetasql
Reporter: Andrew Pilloud
one failure in shard 18
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalStateException
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:491)
at com.google.zetasql.Value.getTimestampUnixMicros(Value.java:325)
at
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSqlUtils.zetaSqlTimestampValueToJodaInstant(ZetaSqlUtils.java:226)
at
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSqlUtils.zetaSqlValueToJavaObject(ZetaSqlUtils.java:210)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:193)
{code}
{code}
Apr 01, 2020 5:58:04 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT date(@p0, @p1, @p2) AS ColA
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9696) UnionCoder IndexOutOfBoundsException
Andrew Pilloud created BEAM-9696:
Summary: UnionCoder IndexOutOfBoundsException
Key: BEAM-9696
URL: https://issues.apache.org/jira/browse/BEAM-9696
Project: Beam
Issue Type: Bug
Components: dsl-sql-zetasql
Reporter: Andrew Pilloud
one failure in shard 17
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
at java.util.ArrayList.rangeCheck(ArrayList.java:658)
at java.util.ArrayList.get(ArrayList.java:434)
at
org.apache.beam.sdk.transforms.join.UnionCoder.decode(UnionCoder.java:83)
at
org.apache.beam.sdk.transforms.join.UnionCoder.decode(UnionCoder.java:32)
at org.apache.beam.sdk.coders.KvCoder.decode(KvCoder.java:82)
at org.apache.beam.sdk.coders.KvCoder.decode(KvCoder.java:36)
at
org.apache.beam.sdk.util.CoderUtils.decodeFromSafeStream(CoderUtils.java:115)
at
org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:98)
at
org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:92)
at org.apache.beam.sdk.util.CoderUtils.clone(CoderUtils.java:141)
at
org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.(MutationDetectors.java:115)
at
org.apache.beam.sdk.util.MutationDetectors.forValueWithCoder(MutationDetectors.java:46)
at
org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.add(ImmutabilityCheckingBundleFactory.java:112)
at
org.apache.beam.runners.direct.ParDoEvaluator$BundleOutputManager.output(ParDoEvaluator.java:299)
at
org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:259)
at
org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.access$800(SimpleDoFnRunner.java:79)
at
org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:405)
at
org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:393)
at
org.apache.beam.sdk.transforms.join.CoGroupByKey$ConstructUnionTableFn.processElement(CoGroupByKey.java:175)
{code}
{code}
Apr 01, 2020 6:05:14 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT 1, 0 UNION DISTINCT
SELECT 1, NULL UNION DISTINCT
SELECT 1, NULL UNION DISTINCT
SELECT 2, NULL UNION DISTINCT
SELECT NULL, 0 UNION DISTINCT
SELECT NULL, 0 UNION DISTINCT
SELECT NULL, 1 UNION DISTINCT
SELECT NULL, NULL UNION DISTINCT
SELECT NULL, NULL
{code}
--
This
[jira] [Updated] (BEAM-9517) BeamZetaSqlCalcRel$CalcFn.processElement Field is not nullable
[
https://issues.apache.org/jira/browse/BEAM-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9517:
-
Description:
two failures in shard 4, one failure in shard 12, nine failures in shard 15,
six failures in shard 17
{code:java}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.sdk.values.SchemaVerification.verifyRowValues(SchemaVerification.java:53)
at org.apache.beam.sdk.values.Row$Builder.build(Row.java:673)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:195)
{code}
{code}
Apr 01, 2020 5:56:51 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
Apr 01, 2020 5:56:52 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
Apr 01, 2020 6:02:50 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select string(NULL),
string(NULL, 'Americas/Los_Angeles'),
string(timestamp '2010-02-03 04:05:06', NULL),
string(NULL, NULL)
Apr 01, 2020 6:01:33 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_add(@p0, INTERVAL @p1 HOUR) AS
ColA
Apr 01, 2020 6:02:00 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_sub(@p0, INTERVAL @p1 HOUR) AS
ColA
Apr 01, 2020 6:02:44 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select tz, parse_timestamp('%y', '0', tz) from
Timezones
Apr 01, 2020 6:02:45 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select tz, format_timestamp('%y', timestamp
'2000-01-01 00:00:00', tz) from Timezones
Apr 01, 2020 6:02:45 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select tz, string(timestamp '2014-02-28
10:20:30', tz) from Timezones
Apr 01, 2020 6:04:04 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_seconds(@p0) AS ColA
Apr 01, 2020 6:04:05 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT
[jira] [Updated] (BEAM-9519) BeamZetaSqlCalcRel$CalcFn.processElement field is not nullable in Array field $arrayx
[
https://issues.apache.org/jira/browse/BEAM-9519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9519:
-
Description:
One failure in shard 12, five failures in shard 16
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: FieldType{typeName=STRING, nullable=false,
logicalType=null, collectionElementType=null, mapKeyType=null,
mapValueType=null, rowSchema=null, metadata={}} is not nullable in Array field
$arraytz
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: FieldType{typeName=STRING,
nullable=false, logicalType=null, collectionElementType=null, mapKeyType=null,
mapValueType=null, rowSchema=null, metadata={}} is not nullable in Array field
$arraytz
at
org.apache.beam.sdk.values.SchemaVerification.verifyArray(SchemaVerification.java:101)
at
org.apache.beam.sdk.values.SchemaVerification.verifyFieldValue(SchemaVerification.java:65)
at
org.apache.beam.sdk.values.SchemaVerification.verifyRowValues(SchemaVerification.java:57)
at org.apache.beam.sdk.values.Row$Builder.build(Row.java:673)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:195)
{code}
{code}
Apr 01, 2020 6:02:53 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select timestamp("2011-02-03 10:20:30+05", tz)
from (select tz from unnest(["+02", NULL, "America/Los_Angeles"]) tz);
Apr 01, 2020 6:05:07 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT [1, 4, 9, NULL, 10] non_empty_array
Apr 01, 2020 6:05:08 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT [CAST(5 AS INT64), NULL] int64_val
Apr 01, 2020 6:05:09 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT [true, NULL, false, true] bool_val
Apr 01, 2020 6:05:09 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT [CAST("8" AS STRING), NULL,
CAST("googlesql" AS STRING)] string_val
Apr 01, 2020 6:05:10 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT [NULL, CAST(b"9" AS BYTES)] bytes_val
{code}
was:
One failure in shard 12, two failures in shard 16
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: FieldType{typeName=STRING, nullable=false,
logicalType=null, collectionElementType=null, mapKeyType=null,
mapValueType=null, rowSchema=null, metadata={}} is not nullable in Array field
[jira] [Work logged] (BEAM-9578) Enumerating artifacts is too expensive in Java
[
https://issues.apache.org/jira/browse/BEAM-9578?focusedWorklogId=415853=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415853
]
ASF GitHub Bot logged work on BEAM-9578:
Author: ASF GitHub Bot
Created on: 04/Apr/20 00:03
Start Date: 04/Apr/20 00:03
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #11205: [BEAM-9578]
Enumerating artifacts is too expensive in Java
URL: https://github.com/apache/beam/pull/11205#discussion_r403390422
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/DefaultArtifactResolver.java
##
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction;
+
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+
+public class DefaultArtifactResolver implements ArtifactResolver {
+ public static ArtifactResolver INSTANCE = new DefaultArtifactResolver();
+
+ private ResolutionFn resolver =
+ (info) -> {
+if
(BeamUrns.getUrn(RunnerApi.StandardArtifacts.Types.FILE).equals(info.getTypeUrn()))
{
+ return Optional.of(info);
+} else {
+ return Optional.empty();
+}
+ };
+
+ @Override
+ public void register(ResolutionFn fn) {
+resolver =
Review comment:
done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 415853)
Time Spent: 5h 40m (was: 5.5h)
> Enumerating artifacts is too expensive in Java
> --
>
> Key: BEAM-9578
> URL: https://issues.apache.org/jira/browse/BEAM-9578
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Heejong Lee
>Priority: Critical
> Labels: portability
> Fix For: 2.21.0
>
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> There are a lot of places (e.g. *ParDoTranslation#getParDoPayload*) which
> effectively call *Environments#createOrGetDefaultEnvironment* which causes
> [artifacts to be
> computed|https://github.com/apache/beam/blob/fc6cef9972780ca6b7525d4aadd65a8344221f1b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java#L114].
> This leads to zipping directories for non-jar dependencies.
> Similar problems may exist for Python/Go.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9519) BeamZetaSqlCalcRel$CalcFn.processElement field is not nullable in Array field $arrayx
[
https://issues.apache.org/jira/browse/BEAM-9519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9519:
-
Description:
One failure in shard 12, two failures in shard 16
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: FieldType{typeName=STRING, nullable=false,
logicalType=null, collectionElementType=null, mapKeyType=null,
mapValueType=null, rowSchema=null, metadata={}} is not nullable in Array field
$arraytz
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: FieldType{typeName=STRING,
nullable=false, logicalType=null, collectionElementType=null, mapKeyType=null,
mapValueType=null, rowSchema=null, metadata={}} is not nullable in Array field
$arraytz
at
org.apache.beam.sdk.values.SchemaVerification.verifyArray(SchemaVerification.java:101)
at
org.apache.beam.sdk.values.SchemaVerification.verifyFieldValue(SchemaVerification.java:65)
at
org.apache.beam.sdk.values.SchemaVerification.verifyRowValues(SchemaVerification.java:57)
at org.apache.beam.sdk.values.Row$Builder.build(Row.java:673)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:195)
{code}
{code}
Apr 01, 2020 6:02:53 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select timestamp("2011-02-03 10:20:30+05", tz)
from (select tz from unnest(["+02", NULL, "America/Los_Angeles"]) tz);
Apr 01, 2020 6:05:07 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT [1, 4, 9, NULL, 10] non_empty_array
Apr 01, 2020 6:05:08 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT [CAST(5 AS INT64), NULL] int64_val
{code}
was:
One failure in shard 12
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: FieldType{typeName=STRING, nullable=false,
logicalType=null, collectionElementType=null, mapKeyType=null,
mapValueType=null, rowSchema=null, metadata={}} is not nullable in Array field
$arraytz
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
[jira] [Updated] (BEAM-9518) BeamZetaSqlCalcRel$CalcFn.processElement Unsupported Beam fieldType: LOGICAL_TYPE
[
https://issues.apache.org/jira/browse/BEAM-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9518:
-
Description:
eight failures in shard 6, eight failures in shard 7, eight failures in shard
8, ten failures in shard 9, five failures in shard 10, four failures in shard
11, six failures in shard 12, two failures in shard 16
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSqlUtils.zetaSqlValueToJavaObject(ZetaSqlUtils.java:221)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:193)
{code}
{code}
Apr 01, 2020 6:00:21 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:03:59 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT DISTINCT val.DATE
from (select DATE"2014-01-01" DATE union all
select cast(NULL as date) union all
select DATE"2014-01-02" union all
select DATE"2014-01-01" union all
select cast(NULL as date)) val
Apr 01, 2020 6:10:09 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT COALESCE(NULL, DATE '2014-12-18')
Apr 01, 2020 6:03:26 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(null AS TIME),
CAST('12:34:56.123456' AS TIME),
CAST(CAST(null AS TIME) AS STRING),
CAST(CAST('12:34:56.123456' AS TIME) AS STRING),
CAST(TIME '12:34:56.123456' AS STRING),
CAST(CAST(TIME '12:34:56.123456' AS STRING) AS TIME)
Apr 01, 2020 5:59:59 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT SAFE_CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:07:07 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT
CAST(CAST(TIME "00:00:00" as STRING) as TIME),
CAST(CAST(TIME "23:59:59.99" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.123456" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.12345" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.12" as STRING) as TIME),
CAST(CAST(TIME "01:02:3" as STRING) as TIME),
CAST(CAST(TIME "01:2:03" as STRING) as TIME),
CAST(CAST(TIME "1:02:03" as STRING) as TIME),
#Leap second cases
CAST(CAST(TIME "23:59:60" as
[jira] [Work logged] (BEAM-9578) Enumerating artifacts is too expensive in Java
[ https://issues.apache.org/jira/browse/BEAM-9578?focusedWorklogId=415852=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415852 ] ASF GitHub Bot logged work on BEAM-9578: Author: ASF GitHub Bot Created on: 03/Apr/20 23:59 Start Date: 03/Apr/20 23:59 Worklog Time Spent: 10m Work Description: ihji commented on issue #11205: [BEAM-9578] Enumerating artifacts is too expensive in Java URL: https://github.com/apache/beam/pull/11205#issuecomment-608857763 @lukecwik Ready to merge. Please trigger the test and take a final look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415852) Time Spent: 5.5h (was: 5h 20m) > Enumerating artifacts is too expensive in Java > -- > > Key: BEAM-9578 > URL: https://issues.apache.org/jira/browse/BEAM-9578 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Luke Cwik >Assignee: Heejong Lee >Priority: Critical > Labels: portability > Fix For: 2.21.0 > > Time Spent: 5.5h > Remaining Estimate: 0h > > There are a lot of places (e.g. *ParDoTranslation#getParDoPayload*) which > effectively call *Environments#createOrGetDefaultEnvironment* which causes > [artifacts to be > computed|https://github.com/apache/beam/blob/fc6cef9972780ca6b7525d4aadd65a8344221f1b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java#L114]. > This leads to zipping directories for non-jar dependencies. > Similar problems may exist for Python/Go. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (BEAM-9660) StreamingDataflowWorker has confusing exception on commits over 2GB
[ https://issues.apache.org/jira/browse/BEAM-9660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Whittle closed BEAM-9660. - Fix Version/s: 2.21.0 Resolution: Fixed > StreamingDataflowWorker has confusing exception on commits over 2GB > --- > > Key: BEAM-9660 > URL: https://issues.apache.org/jira/browse/BEAM-9660 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Affects Versions: 2.18.0, 2.19.0 >Reporter: Sam Whittle >Assignee: Sam Whittle >Priority: Minor > Fix For: 2.21.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Commits over 2GB have a negative serialized commit size. > When not using streaming engine the max commit limit is 2GB. > https://github.com/apache/beam/blob/v2.19.0/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java#L450 > There appears to be a logging regression introduced by > https://github.com/apache/beam/pull/10013 > With the new code, if the serialization overflows the estimated bytes is set > to Integer.MAX which equals the commit limit for appliance. > Then the comparison here: > https://github.com/apache/beam/blob/v2.19.0/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java#L1371 > which uses > does not trigger and the large commit is just passed on to the > commit queue, triggering the exception seen in #3 [2] when the weigher uses > the negative serialized size for the semaphore acquire call. > So previously where we would have thrown a KeyCommitTooLargeException we are > throwing the IllegalArgumentException. > From that exception description: > https://github.com/apache/beam/blob/v2.19.0/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java#L236 > ". This may be caused by grouping a very " > + "large amount of data in a single window without using > Combine," > + " or by producing a large amount of data from a single input > element." > The overflow could be remembered explicitly instead of just comparing with > max. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9578) Enumerating artifacts is too expensive in Java
[ https://issues.apache.org/jira/browse/BEAM-9578?focusedWorklogId=415824=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415824 ] ASF GitHub Bot logged work on BEAM-9578: Author: ASF GitHub Bot Created on: 03/Apr/20 23:22 Start Date: 03/Apr/20 23:22 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11205: [BEAM-9578] Enumerating artifacts is too expensive in Java URL: https://github.com/apache/beam/pull/11205#issuecomment-608803659 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415824) Time Spent: 5h 20m (was: 5h 10m) > Enumerating artifacts is too expensive in Java > -- > > Key: BEAM-9578 > URL: https://issues.apache.org/jira/browse/BEAM-9578 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Luke Cwik >Assignee: Heejong Lee >Priority: Critical > Labels: portability > Fix For: 2.21.0 > > Time Spent: 5h 20m > Remaining Estimate: 0h > > There are a lot of places (e.g. *ParDoTranslation#getParDoPayload*) which > effectively call *Environments#createOrGetDefaultEnvironment* which causes > [artifacts to be > computed|https://github.com/apache/beam/blob/fc6cef9972780ca6b7525d4aadd65a8344221f1b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java#L114]. > This leads to zipping directories for non-jar dependencies. > Similar problems may exist for Python/Go. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9412) Fix linkage errors in vendored calcite
[
https://issues.apache.org/jira/browse/BEAM-9412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074965#comment-17074965
]
Rui Wang commented on BEAM-9412:
[~lcwik] how to check linkage error for a specific module. E.g.
vendor/calcite-1_20_0
> Fix linkage errors in vendored calcite
> --
>
> Key: BEAM-9412
> URL: https://issues.apache.org/jira/browse/BEAM-9412
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Luke Cwik
>Assignee: Kai Jiang
>Priority: Minor
>
> As of [https://github.com/apache/beam/pull/10559], the linkage errors are:
> {code:java}
> Class org.slf4j.LoggerFactory is not found;
> referenced by 29 class files
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.pretty.SqlPrettyWriter
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.AbstractMaterializedViewRule
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Benchmark
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RexImplicationChecker
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.VisitorDataContext
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlDialectFactoryImpl
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlDialect
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.validate.SqlValidatorException
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.runtime.ResultSetEnumerable
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.trace.CalciteTrace
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.runtime.CalciteException
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.remote.AvaticaHttpClientFactoryImpl
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.remote.RemoteProtobufService
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.remote.KerberosConnection
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.remote.AvaticaCommonsHttpClientImpl
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.remote.ClientKeytabJaasConf
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.remote.AvaticaCommonsHttpClientSpnegoImpl
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.remote.Driver
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.remote.ProtobufTranslationImpl
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.com.jayway.jsonpath.internal.filter.FilterCompiler
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.com.jayway.jsonpath.internal.filter.RelationalExpressionNode
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.com.jayway.jsonpath.internal.filter.ValueNode
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.com.jayway.jsonpath.internal.JsonContext
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.com.jayway.jsonpath.internal.path.ArrayPathToken
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.com.jayway.jsonpath.internal.path.CompiledPath
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.com.jayway.jsonpath.internal.path.PredicateContextImpl
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.com.jayway.jsonpath.spi.json.JsonOrgJsonProvider
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.com.jayway.jsonpath.spi.mapper.GsonMappingProvider
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
>
[jira] [Updated] (BEAM-9517) BeamZetaSqlCalcRel$CalcFn.processElement Field is not nullable
[
https://issues.apache.org/jira/browse/BEAM-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9517:
-
Description:
two failures in shard 4, one failure in shard 12, nine failures in shard 15
{code:java}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.sdk.values.SchemaVerification.verifyRowValues(SchemaVerification.java:53)
at org.apache.beam.sdk.values.Row$Builder.build(Row.java:673)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:195)
{code}
{code}
Apr 01, 2020 5:56:51 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
Apr 01, 2020 5:56:52 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
Apr 01, 2020 6:02:50 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select string(NULL),
string(NULL, 'Americas/Los_Angeles'),
string(timestamp '2010-02-03 04:05:06', NULL),
string(NULL, NULL)
Apr 01, 2020 6:01:33 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_add(@p0, INTERVAL @p1 HOUR) AS
ColA
Apr 01, 2020 6:02:00 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_sub(@p0, INTERVAL @p1 HOUR) AS
ColA
Apr 01, 2020 6:02:44 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select tz, parse_timestamp('%y', '0', tz) from
Timezones
Apr 01, 2020 6:02:45 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select tz, format_timestamp('%y', timestamp
'2000-01-01 00:00:00', tz) from Timezones
Apr 01, 2020 6:02:45 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select tz, string(timestamp '2014-02-28
10:20:30', tz) from Timezones
{code}
was:
two failures in shard 4, one failure in shard 12, nine failures in shard 15
{code:java}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
[jira] [Work logged] (BEAM-9136) Add LICENSES and NOTICES to docker images
[ https://issues.apache.org/jira/browse/BEAM-9136?focusedWorklogId=415817=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415817 ] ASF GitHub Bot logged work on BEAM-9136: Author: ASF GitHub Bot Created on: 03/Apr/20 23:10 Start Date: 03/Apr/20 23:10 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #11307: [BEAM-9136] (test remove behavior) remove nose URL: https://github.com/apache/beam/pull/11307#issuecomment-608785737 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415817) Time Spent: 18.5h (was: 18h 20m) > Add LICENSES and NOTICES to docker images > - > > Key: BEAM-9136 > URL: https://issues.apache.org/jira/browse/BEAM-9136 > Project: Beam > Issue Type: Task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.21.0 > > Time Spent: 18.5h > Remaining Estimate: 0h > > Scan dependencies and add licenses and notices of the dependencies to SDK > docker images. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9517) BeamZetaSqlCalcRel$CalcFn.processElement Field is not nullable
[
https://issues.apache.org/jira/browse/BEAM-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9517:
-
Description:
two failures in shard 4, one failure in shard 12, nine failures in shard 15
{code:java}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.sdk.values.SchemaVerification.verifyRowValues(SchemaVerification.java:53)
at org.apache.beam.sdk.values.Row$Builder.build(Row.java:673)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:195)
{code}
{code}
Apr 01, 2020 5:56:51 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
Apr 01, 2020 5:56:52 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
Apr 01, 2020 6:02:50 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select string(NULL),
string(NULL, 'Americas/Los_Angeles'),
string(timestamp '2010-02-03 04:05:06', NULL),
string(NULL, NULL)
{code}
was:
two failures in shard 4, one failure in shard 12
{code:java}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
[jira] [Work logged] (BEAM-9685) Don't release Go SDK container until Go is officially supported.
[ https://issues.apache.org/jira/browse/BEAM-9685?focusedWorklogId=415815=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415815 ] ASF GitHub Bot logged work on BEAM-9685: Author: ASF GitHub Bot Created on: 03/Apr/20 23:09 Start Date: 03/Apr/20 23:09 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #11308: [BEAM-9685] remove Go SDK container from release process from 2.22.0 URL: https://github.com/apache/beam/pull/11308#issuecomment-608784317 Run Go Postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415815) Time Spent: 1h 10m (was: 1h) > Don't release Go SDK container until Go is officially supported. > > > Key: BEAM-9685 > URL: https://issues.apache.org/jira/browse/BEAM-9685 > Project: Beam > Issue Type: Task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.21.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > 1. Remove Go SDK container from release process. > 2. Update document about it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9685) Don't release Go SDK container until Go is officially supported.
[ https://issues.apache.org/jira/browse/BEAM-9685?focusedWorklogId=415816=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415816 ] ASF GitHub Bot logged work on BEAM-9685: Author: ASF GitHub Bot Created on: 03/Apr/20 23:09 Start Date: 03/Apr/20 23:09 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #11308: [BEAM-9685] remove Go SDK container from release process from 2.22.0 URL: https://github.com/apache/beam/pull/11308#issuecomment-608784594 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415816) Time Spent: 1h 20m (was: 1h 10m) > Don't release Go SDK container until Go is officially supported. > > > Key: BEAM-9685 > URL: https://issues.apache.org/jira/browse/BEAM-9685 > Project: Beam > Issue Type: Task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.21.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > 1. Remove Go SDK container from release process. > 2. Update document about it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9519) BeamZetaSqlCalcRel$CalcFn.processElement field is not nullable in Array field $arrayx
[
https://issues.apache.org/jira/browse/BEAM-9519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9519:
-
Description:
One failure in shard 12
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: FieldType{typeName=STRING, nullable=false,
logicalType=null, collectionElementType=null, mapKeyType=null,
mapValueType=null, rowSchema=null, metadata={}} is not nullable in Array field
$arraytz
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: FieldType{typeName=STRING,
nullable=false, logicalType=null, collectionElementType=null, mapKeyType=null,
mapValueType=null, rowSchema=null, metadata={}} is not nullable in Array field
$arraytz
at
org.apache.beam.sdk.values.SchemaVerification.verifyArray(SchemaVerification.java:101)
at
org.apache.beam.sdk.values.SchemaVerification.verifyFieldValue(SchemaVerification.java:65)
at
org.apache.beam.sdk.values.SchemaVerification.verifyRowValues(SchemaVerification.java:57)
at org.apache.beam.sdk.values.Row$Builder.build(Row.java:673)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:195)
{code}
{code}
Apr 01, 2020 6:02:53 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select timestamp("2011-02-03 10:20:30+05", tz)
from (select tz from unnest(["+02", NULL, "America/Los_Angeles"]) tz);
{code}
was:
One failure in shard 12
{code}
> BeamZetaSqlCalcRel$CalcFn.processElement field is not nullable in Array field
> $arrayx
> -
>
> Key: BEAM-9519
> URL: https://issues.apache.org/jira/browse/BEAM-9519
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Priority: Minor
> Labels: zetasql-compliance
>
> One failure in shard 12
> {code}
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.lang.IllegalArgumentException: FieldType{typeName=STRING,
> nullable=false, logicalType=null, collectionElementType=null,
> mapKeyType=null, mapValueType=null, rowSchema=null, metadata={}} is not
> nullable in Array field $arraytz
> at
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
> at
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
> at
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
> at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
> at
[jira] [Work logged] (BEAM-9691) Ensure Dataflow BQ Native sink are not used on FnApi
[ https://issues.apache.org/jira/browse/BEAM-9691?focusedWorklogId=415811=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415811 ] ASF GitHub Bot logged work on BEAM-9691: Author: ASF GitHub Bot Created on: 03/Apr/20 23:07 Start Date: 03/Apr/20 23:07 Worklog Time Spent: 10m Work Description: pabloem commented on issue #11309: [BEAM-9691] Ensuring BQ Native Source is avoided on FnApi pipelines URL: https://github.com/apache/beam/pull/11309#issuecomment-608782560 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415811) Time Spent: 1h (was: 50m) > Ensure Dataflow BQ Native sink are not used on FnApi > > > Key: BEAM-9691 > URL: https://issues.apache.org/jira/browse/BEAM-9691 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9519) BeamZetaSqlCalcRel$CalcFn.processElement field is not nullable in Array field $arrayx
[
https://issues.apache.org/jira/browse/BEAM-9519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9519:
-
Description:
One failure in shard 12
{code}
was:
{code:java}
Mar 16, 2020 1:27:47 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT x FROM UNNEST([3, NULL]) x;
Mar 16, 2020 1:27:48 PM com.google.zetasql.io.grpc.internal.SerializingExecutor
run
SEVERE: Exception while executing runnable
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@39086547
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: FieldType{typeName=INT64, nullable=false,
logicalType=null, collectionElementType=null, mapKeyType=null,
mapValueType=null, rowSchema=null, metadata={}} is not nullable in Array field
$arrayx
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:244)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: FieldType{typeName=INT64,
nullable=false, logicalType=null, collectionElementType=null, mapKeyType=null,
mapValueType=null, rowSchema=null, metadata={}} is not nullable in Array field
$arrayx
at org.apache.beam.sdk.values.Row$Builder.verifyArray(Row.java:682)
at org.apache.beam.sdk.values.Row$Builder.verify(Row.java:647)
at org.apache.beam.sdk.values.Row$Builder.verify(Row.java:639)
at org.apache.beam.sdk.values.Row$Builder.build(Row.java:844)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:195)
{code}
> BeamZetaSqlCalcRel$CalcFn.processElement field is not nullable in Array field
> $arrayx
> -
>
> Key: BEAM-9519
> URL: https://issues.apache.org/jira/browse/BEAM-9519
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Priority: Minor
> Labels: zetasql-compliance
>
> One failure in shard 12
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9517) BeamZetaSqlCalcRel$CalcFn.processElement Field is not nullable
[
https://issues.apache.org/jira/browse/BEAM-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9517:
-
Description:
two failures in shard 4, one failure in shard 12
{code:java}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.sdk.values.SchemaVerification.verifyRowValues(SchemaVerification.java:53)
at org.apache.beam.sdk.values.Row$Builder.build(Row.java:673)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:195)
{code}
{code}
Apr 01, 2020 5:56:51 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
Apr 01, 2020 5:56:52 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
Apr 01, 2020 6:02:50 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: select string(NULL),
string(NULL, 'Americas/Los_Angeles'),
string(timestamp '2010-02-03 04:05:06', NULL),
string(NULL, NULL)
{code}
was:
two failures in shard 4
{code:java}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
[jira] [Updated] (BEAM-9517) BeamZetaSqlCalcRel$CalcFn.processElement Field is not nullable
[
https://issues.apache.org/jira/browse/BEAM-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9517:
-
Description:
two failures in shard 4
{code:java}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.sdk.values.SchemaVerification.verifyRowValues(SchemaVerification.java:53)
at org.apache.beam.sdk.values.Row$Builder.build(Row.java:673)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:195)
{code}
{code}
Apr 01, 2020 5:56:51 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
Apr 01, 2020 5:56:52 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
{code}
was:
two failures in shard 4
{code:java}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
[jira] [Updated] (BEAM-9517) BeamZetaSqlCalcRel$CalcFn.processElement Field is not nullable
[
https://issues.apache.org/jira/browse/BEAM-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9517:
-
Description:
two failures in shard 4
{code:java}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.sdk.values.SchemaVerification.verifyRowValues(SchemaVerification.java:53)
at org.apache.beam.sdk.values.Row$Builder.build(Row.java:673)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:195)
{code}
{code}
Apr 01, 2020 5:56:51 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
Apr 01, 2020 5:56:52 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
{code}
was:
two failures in shard 4
{code:java}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
[jira] [Work logged] (BEAM-6217) Typos on javadoc
[
https://issues.apache.org/jira/browse/BEAM-6217?focusedWorklogId=415809=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415809
]
ASF GitHub Bot logged work on BEAM-6217:
Author: ASF GitHub Bot
Created on: 03/Apr/20 23:02
Start Date: 03/Apr/20 23:02
Worklog Time Spent: 10m
Work Description: pabloem commented on issue #11310: [BEAM-6217] Fixing
typos on javadoc
URL: https://github.com/apache/beam/pull/11310#issuecomment-608774034
Run Portable_Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 415809)
Time Spent: 40m (was: 0.5h)
> Typos on javadoc
>
>
> Key: BEAM-6217
> URL: https://issues.apache.org/jira/browse/BEAM-6217
> Project: Beam
> Issue Type: Bug
> Components: website
>Reporter: Rose Nguyen
>Priority: Minor
> Time Spent: 40m
> Remaining Estimate: 0h
>
> On the javadocs [https://beam.apache.org/releases/javadoc/2.1.0/]
> *View*
> Change tranform->transform in "When a
> [{{ParDo}}|https://beam.apache.org/releases/javadoc/2.1.0/org/apache/beam/sdk/transforms/ParDo.html]
> tranform is processing a main input"
> *AfterWatermark*
> Change therafter->thereafter in "window and then immediately therafter when
> any late data arrives"
> Change Additionaly->Additionally in "Additionaly firings before or after the
> watermark "
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9518) BeamZetaSqlCalcRel$CalcFn.processElement Unsupported Beam fieldType: LOGICAL_TYPE
[
https://issues.apache.org/jira/browse/BEAM-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9518:
-
Description:
eight failures in shard 6, eight failures in shard 7, eight failures in shard
8, ten failures in shard 9, five failures in shard 10, four failures in shard
11, six failures in shard 12
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSqlUtils.zetaSqlValueToJavaObject(ZetaSqlUtils.java:221)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:193)
{code}
{code}
Apr 01, 2020 6:00:21 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:03:59 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT DISTINCT val.DATE
from (select DATE"2014-01-01" DATE union all
select cast(NULL as date) union all
select DATE"2014-01-02" union all
select DATE"2014-01-01" union all
select cast(NULL as date)) val
Apr 01, 2020 6:10:09 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT COALESCE(NULL, DATE '2014-12-18')
Apr 01, 2020 6:03:26 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(null AS TIME),
CAST('12:34:56.123456' AS TIME),
CAST(CAST(null AS TIME) AS STRING),
CAST(CAST('12:34:56.123456' AS TIME) AS STRING),
CAST(TIME '12:34:56.123456' AS STRING),
CAST(CAST(TIME '12:34:56.123456' AS STRING) AS TIME)
Apr 01, 2020 5:59:59 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT SAFE_CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:07:07 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT
CAST(CAST(TIME "00:00:00" as STRING) as TIME),
CAST(CAST(TIME "23:59:59.99" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.123456" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.12345" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.12" as STRING) as TIME),
CAST(CAST(TIME "01:02:3" as STRING) as TIME),
CAST(CAST(TIME "01:2:03" as STRING) as TIME),
CAST(CAST(TIME "1:02:03" as STRING) as TIME),
#Leap second cases
CAST(CAST(TIME "23:59:60" as STRING) as TIME),
[jira] [Work logged] (BEAM-3097) Allow BigQuerySource to take a ValueProvider as a table input.
[ https://issues.apache.org/jira/browse/BEAM-3097?focusedWorklogId=415808=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415808 ] ASF GitHub Bot logged work on BEAM-3097: Author: ASF GitHub Bot Created on: 03/Apr/20 22:58 Start Date: 03/Apr/20 22:58 Worklog Time Spent: 10m Work Description: pabloem commented on issue #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table URL: https://github.com/apache/beam/pull/11244#issuecomment-608768810 Run Python 3.7 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415808) Time Spent: 2h 50m (was: 2h 40m) > Allow BigQuerySource to take a ValueProvider as a table input. > -- > > Key: BEAM-3097 > URL: https://issues.apache.org/jira/browse/BEAM-3097 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Ed Mothershaw >Priority: Minor > Original Estimate: 2h > Time Spent: 2h 50m > Remaining Estimate: 0h > > In file sdks/python/apache_beam/io/gcp/bigquery.py, class BigQuery, line 389. > When a ValueProvider is input as table the script will fail. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9518) BeamZetaSqlCalcRel$CalcFn.processElement Unsupported Beam fieldType: LOGICAL_TYPE
[
https://issues.apache.org/jira/browse/BEAM-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9518:
-
Description:
eight failures in shard 6, eight failures in shard 7, eight failures in shard
8, ten failures in shard 9, five failures in shard 10
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSqlUtils.zetaSqlValueToJavaObject(ZetaSqlUtils.java:221)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:193)
{code}
{code}
Apr 01, 2020 6:00:21 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:03:59 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT DISTINCT val.DATE
from (select DATE"2014-01-01" DATE union all
select cast(NULL as date) union all
select DATE"2014-01-02" union all
select DATE"2014-01-01" union all
select cast(NULL as date)) val
Apr 01, 2020 6:10:09 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT COALESCE(NULL, DATE '2014-12-18')
Apr 01, 2020 6:03:26 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(null AS TIME),
CAST('12:34:56.123456' AS TIME),
CAST(CAST(null AS TIME) AS STRING),
CAST(CAST('12:34:56.123456' AS TIME) AS STRING),
CAST(TIME '12:34:56.123456' AS STRING),
CAST(CAST(TIME '12:34:56.123456' AS STRING) AS TIME)
Apr 01, 2020 5:59:59 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT SAFE_CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:07:07 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT
CAST(CAST(TIME "00:00:00" as STRING) as TIME),
CAST(CAST(TIME "23:59:59.99" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.123456" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.12345" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.12" as STRING) as TIME),
CAST(CAST(TIME "01:02:3" as STRING) as TIME),
CAST(CAST(TIME "01:2:03" as STRING) as TIME),
CAST(CAST(TIME "1:02:03" as STRING) as TIME),
#Leap second cases
CAST(CAST(TIME "23:59:60" as STRING) as TIME),
CAST(CAST(TIME "12:59:60" as STRING) as TIME),
[jira] [Updated] (BEAM-9518) BeamZetaSqlCalcRel$CalcFn.processElement Unsupported Beam fieldType: LOGICAL_TYPE
[
https://issues.apache.org/jira/browse/BEAM-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9518:
-
Description:
eight failures in shard 6, eight failures in shard 7, eight failures in shard
8, ten failures in shard 9
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSqlUtils.zetaSqlValueToJavaObject(ZetaSqlUtils.java:221)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:193)
{code}
{code}
Apr 01, 2020 6:00:21 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:03:59 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT DISTINCT val.DATE
from (select DATE"2014-01-01" DATE union all
select cast(NULL as date) union all
select DATE"2014-01-02" union all
select DATE"2014-01-01" union all
select cast(NULL as date)) val
Apr 01, 2020 6:10:09 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT COALESCE(NULL, DATE '2014-12-18')
Apr 01, 2020 6:03:26 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(null AS TIME),
CAST('12:34:56.123456' AS TIME),
CAST(CAST(null AS TIME) AS STRING),
CAST(CAST('12:34:56.123456' AS TIME) AS STRING),
CAST(TIME '12:34:56.123456' AS STRING),
CAST(CAST(TIME '12:34:56.123456' AS STRING) AS TIME)
Apr 01, 2020 5:59:59 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT SAFE_CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:07:07 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT
CAST(CAST(TIME "00:00:00" as STRING) as TIME),
CAST(CAST(TIME "23:59:59.99" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.123456" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.12345" as STRING) as TIME),
CAST(CAST(TIME "01:02:03.12" as STRING) as TIME),
CAST(CAST(TIME "01:02:3" as STRING) as TIME),
CAST(CAST(TIME "01:2:03" as STRING) as TIME),
CAST(CAST(TIME "1:02:03" as STRING) as TIME),
#Leap second cases
CAST(CAST(TIME "23:59:60" as STRING) as TIME),
CAST(CAST(TIME "12:59:60" as STRING) as TIME),
CAST(CAST(TIME "12:59:60.123456" as
[jira] [Work logged] (BEAM-9577) Update artifact staging and retrieval protocols to be dependency aware.
[ https://issues.apache.org/jira/browse/BEAM-9577?focusedWorklogId=415807=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415807 ] ASF GitHub Bot logged work on BEAM-9577: Author: ASF GitHub Bot Created on: 03/Apr/20 22:55 Start Date: 03/Apr/20 22:55 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11305: [BEAM-9577] Update container boot code to stage from dependencies, if present. URL: https://github.com/apache/beam/pull/11305 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415807) Time Spent: 12h 20m (was: 12h 10m) > Update artifact staging and retrieval protocols to be dependency aware. > --- > > Key: BEAM-9577 > URL: https://issues.apache.org/jira/browse/BEAM-9577 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 12h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9518) BeamZetaSqlCalcRel$CalcFn.processElement Unsupported Beam fieldType: LOGICAL_TYPE
[
https://issues.apache.org/jira/browse/BEAM-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9518:
-
Description:
eight failures in shard 6, eight failures in shard 7
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSqlUtils.zetaSqlValueToJavaObject(ZetaSqlUtils.java:221)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:193)
{code}
{code}
Apr 01, 2020 6:00:21 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:03:59 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT DISTINCT val.DATE
from (select DATE"2014-01-01" DATE union all
select cast(NULL as date) union all
select DATE"2014-01-02" union all
select DATE"2014-01-01" union all
select cast(NULL as date)) val
Apr 01, 2020 6:10:09 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT COALESCE(NULL, DATE '2014-12-18')
{code}
was:
eight failures in shard 6
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
[jira] [Updated] (BEAM-9518) BeamZetaSqlCalcRel$CalcFn.processElement Unsupported Beam fieldType: LOGICAL_TYPE
[
https://issues.apache.org/jira/browse/BEAM-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9518:
-
Description:
eight failures in shard 6, eight failures in shard 7, eight failures in shard 8
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSqlUtils.zetaSqlValueToJavaObject(ZetaSqlUtils.java:221)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:193)
{code}
{code}
Apr 01, 2020 6:00:21 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:03:59 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT DISTINCT val.DATE
from (select DATE"2014-01-01" DATE union all
select cast(NULL as date) union all
select DATE"2014-01-02" union all
select DATE"2014-01-01" union all
select cast(NULL as date)) val
Apr 01, 2020 6:10:09 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT COALESCE(NULL, DATE '2014-12-18')
Apr 01, 2020 6:03:26 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(null AS TIME),
CAST('12:34:56.123456' AS TIME),
CAST(CAST(null AS TIME) AS STRING),
CAST(CAST('12:34:56.123456' AS TIME) AS STRING),
CAST(TIME '12:34:56.123456' AS STRING),
CAST(CAST(TIME '12:34:56.123456' AS STRING) AS TIME)
{code}
was:
eight failures in shard 6, eight failures in shard 7
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
[jira] [Work logged] (BEAM-9577) Update artifact staging and retrieval protocols to be dependency aware.
[ https://issues.apache.org/jira/browse/BEAM-9577?focusedWorklogId=415806=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415806 ] ASF GitHub Bot logged work on BEAM-9577: Author: ASF GitHub Bot Created on: 03/Apr/20 22:54 Start Date: 03/Apr/20 22:54 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11305: [BEAM-9577] Update container boot code to stage from dependencies, if present. URL: https://github.com/apache/beam/pull/11305#issuecomment-608763196 Yeah, I'm tracking that bug. This doesn't make things worse. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415806) Time Spent: 12h 10m (was: 12h) > Update artifact staging and retrieval protocols to be dependency aware. > --- > > Key: BEAM-9577 > URL: https://issues.apache.org/jira/browse/BEAM-9577 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 12h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-6217) Typos on javadoc
[
https://issues.apache.org/jira/browse/BEAM-6217?focusedWorklogId=415797=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415797
]
ASF GitHub Bot logged work on BEAM-6217:
Author: ASF GitHub Bot
Created on: 03/Apr/20 22:50
Start Date: 03/Apr/20 22:50
Worklog Time Spent: 10m
Work Description: pabloem commented on issue #11310: [BEAM-6217] Fixing
typos on javadoc
URL: https://github.com/apache/beam/pull/11310#issuecomment-608757279
retest this please
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 415797)
Time Spent: 0.5h (was: 20m)
> Typos on javadoc
>
>
> Key: BEAM-6217
> URL: https://issues.apache.org/jira/browse/BEAM-6217
> Project: Beam
> Issue Type: Bug
> Components: website
>Reporter: Rose Nguyen
>Priority: Minor
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> On the javadocs [https://beam.apache.org/releases/javadoc/2.1.0/]
> *View*
> Change tranform->transform in "When a
> [{{ParDo}}|https://beam.apache.org/releases/javadoc/2.1.0/org/apache/beam/sdk/transforms/ParDo.html]
> tranform is processing a main input"
> *AfterWatermark*
> Change therafter->thereafter in "window and then immediately therafter when
> any late data arrives"
> Change Additionaly->Additionally in "Additionaly firings before or after the
> watermark "
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9520) Forbidden IOException when reading from InputStream
[
https://issues.apache.org/jira/browse/BEAM-9520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9520:
-
Description:
Six failures in shard 6
{code:java}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: Forbidden IOException when reading from
InputStream
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Forbidden IOException when
reading from InputStream
at
org.apache.beam.sdk.util.CoderUtils.decodeFromSafeStream(CoderUtils.java:118)
at
org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:98)
at
org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:92)
at org.apache.beam.sdk.util.CoderUtils.clone(CoderUtils.java:141)
at
org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.(MutationDetectors.java:115)
at
org.apache.beam.sdk.util.MutationDetectors.forValueWithCoder(MutationDetectors.java:46)
at
org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.add(ImmutabilityCheckingBundleFactory.java:112)
at
org.apache.beam.runners.direct.ParDoEvaluator$BundleOutputManager.output(ParDoEvaluator.java:299)
at
org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:259)
at
org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.access$800(SimpleDoFnRunner.java:79)
at
org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:405)
at
org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:393)
at
org.apache.beam.sdk.transforms.join.CoGroupByKey$ConstructUnionTableFn.processElement(CoGroupByKey.java:175)
{code}
variant 1:
{code}
Caused by: java.io.EOFException: reached end of stream after reading 0 bytes; 1
bytes expected
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams.readFully(ByteStreams.java:780)
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams.readFully(ByteStreams.java:762)
at
org.apache.beam.sdk.coders.ByteArrayCoder.decode(ByteArrayCoder.java:108)
at org.apache.beam.sdk.coders.BitSetCoder.decode(BitSetCoder.java:57)
at org.apache.beam.sdk.coders.BitSetCoder.decode(BitSetCoder.java:52)
at
org.apache.beam.sdk.coders.RowCoderGenerator$DecodeInstruction.decodeDelegate(RowCoderGenerator.java:339)
at org.apache.beam.sdk.coders.Coder$ByteBuddy$Mp4cMJnv.decode(Unknown
Source)
at org.apache.beam.sdk.coders.Coder$ByteBuddy$Mp4cMJnv.decode(Unknown
Source)
at
[jira] [Updated] (BEAM-9518) BeamZetaSqlCalcRel$CalcFn.processElement Unsupported Beam fieldType: LOGICAL_TYPE
[
https://issues.apache.org/jira/browse/BEAM-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9518:
-
Description:
eight failures in shard 6
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSqlUtils.zetaSqlValueToJavaObject(ZetaSqlUtils.java:221)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:193)
{code}
{code}
Apr 01, 2020 6:00:21 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:00:25 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:00:26 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:00:26 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:00:27 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:00:27 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:00:28 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:03:59 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT DISTINCT val.DATE
from (select DATE"2014-01-01" DATE union all
select cast(NULL as date) union all
select DATE"2014-01-02" union all
select DATE"2014-01-01" union all
select cast(NULL as date)) val
{code}
was:
eight failures in shard 6
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at
[jira] [Updated] (BEAM-9518) BeamZetaSqlCalcRel$CalcFn.processElement Unsupported Beam fieldType: LOGICAL_TYPE
[
https://issues.apache.org/jira/browse/BEAM-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9518:
-
Description:
eight failures in shard 6
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSqlUtils.zetaSqlValueToJavaObject(ZetaSqlUtils.java:221)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:193)
{code}
{code}
Apr 01, 2020 6:00:21 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:00:25 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:00:26 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:00:26 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(@p0 AS DATE) AS ColA
Apr 01, 2020 6:00:27 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT CAST(@p0 AS DATE) AS ColA
{code}
was:
Mar 16, 2020 1:23:15 PM com.google.zetasql.io.grpc.internal.SerializingExecutor
run
SEVERE: Exception while executing runnable
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@61d0fcb4
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.UnsupportedOperationException: Unsupported Beam fieldType:
LOGICAL_TYPE
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
[jira] [Closed] (BEAM-9695) beam_PostCommit_Go failing
[
https://issues.apache.org/jira/browse/BEAM-9695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Bradshaw closed BEAM-9695.
-
Fix Version/s: Not applicable
Resolution: Duplicate
> beam_PostCommit_Go failing
> --
>
> Key: BEAM-9695
> URL: https://issues.apache.org/jira/browse/BEAM-9695
> Project: Beam
> Issue Type: Bug
> Components: sdk-go, test-failures
>Reporter: Udi Meiri
>Assignee: Robert Bradshaw
>Priority: Major
> Fix For: Not applicable
>
>
> [~lostluck]
> I'm seeing this in stackdriver logs:
> {code}
> rpc error: code = Unimplemented desc = unknown service
> org.apache.beam.model.job_management.v1.LegacyArtifactRetrievalService
> {code}
> First failure:
> https://builds.apache.org/job/beam_PostCommit_Go/6624/
> which lists commits from https://issues.apache.org/jira/browse/BEAM-9577
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9517) BeamZetaSqlCalcRel$CalcFn.processElement Field is not nullable
[
https://issues.apache.org/jira/browse/BEAM-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9517:
-
Description:
two failures in shard 4
{code:java}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.sdk.values.SchemaVerification.verifyRowValues(SchemaVerification.java:53)
at org.apache.beam.sdk.values.Row$Builder.build(Row.java:673)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:195)
{code}
Apr 01, 2020 5:56:51 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
Apr 01, 2020 5:56:52 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
{code}
{code}
was:
{code:java}
Mar 16, 2020 1:24:42 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND) AS
ColA
Mar 16, 2020 1:24:43 PM com.google.zetasql.io.grpc.internal.SerializingExecutor
run
SEVERE: Exception while executing runnable
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@20ca06
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: Field ColA is not nullable
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:244)
[jira] [Updated] (BEAM-9695) beam_PostCommit_Go failing
[
https://issues.apache.org/jira/browse/BEAM-9695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri updated BEAM-9695:
Description:
[~lostluck]
I'm seeing this in stackdriver logs:
{code}
rpc error: code = Unimplemented desc = unknown service
org.apache.beam.model.job_management.v1.LegacyArtifactRetrievalService
{code}
First failure:
https://builds.apache.org/job/beam_PostCommit_Go/6624/
which lists commits from https://issues.apache.org/jira/browse/BEAM-9577
was:
[~lostluck]
I'm seeing this in the stackdriver logs:
{code}
rpc error: code = Unimplemented desc = unknown service
org.apache.beam.model.job_management.v1.LegacyArtifactRetrievalService
{code}
First failure:
https://builds.apache.org/job/beam_PostCommit_Go/6624/
which lists commits from https://issues.apache.org/jira/browse/BEAM-9577
> beam_PostCommit_Go failing
> --
>
> Key: BEAM-9695
> URL: https://issues.apache.org/jira/browse/BEAM-9695
> Project: Beam
> Issue Type: Bug
> Components: sdk-go, test-failures
>Reporter: Udi Meiri
>Assignee: Robert Bradshaw
>Priority: Major
>
> [~lostluck]
> I'm seeing this in stackdriver logs:
> {code}
> rpc error: code = Unimplemented desc = unknown service
> org.apache.beam.model.job_management.v1.LegacyArtifactRetrievalService
> {code}
> First failure:
> https://builds.apache.org/job/beam_PostCommit_Go/6624/
> which lists commits from https://issues.apache.org/jira/browse/BEAM-9577
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9695) beam_PostCommit_Go failing
Udi Meiri created BEAM-9695:
---
Summary: beam_PostCommit_Go failing
Key: BEAM-9695
URL: https://issues.apache.org/jira/browse/BEAM-9695
Project: Beam
Issue Type: Bug
Components: sdk-go, test-failures
Reporter: Udi Meiri
Assignee: Robert Bradshaw
[~lostluck]
I'm seeing this in the stackdriver logs:
{code}
rpc error: code = Unimplemented desc = unknown service
org.apache.beam.model.job_management.v1.LegacyArtifactRetrievalService
{code}
First failure:
https://builds.apache.org/job/beam_PostCommit_Go/6624/
which lists commits from https://issues.apache.org/jira/browse/BEAM-9577
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9577) Update artifact staging and retrieval protocols to be dependency aware.
[ https://issues.apache.org/jira/browse/BEAM-9577?focusedWorklogId=415789=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415789 ] ASF GitHub Bot logged work on BEAM-9577: Author: ASF GitHub Bot Created on: 03/Apr/20 22:31 Start Date: 03/Apr/20 22:31 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11312: [BEAM-9577] Plumb resources through Python job service and runner. URL: https://github.com/apache/beam/pull/11312 Also updates artifact staging service to handle dependencies from multiple environments. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-9577) Update artifact staging and retrieval protocols to be dependency aware.
[ https://issues.apache.org/jira/browse/BEAM-9577?focusedWorklogId=415788=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415788 ] ASF GitHub Bot logged work on BEAM-9577: Author: ASF GitHub Bot Created on: 03/Apr/20 22:27 Start Date: 03/Apr/20 22:27 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11311: [BEAM-9577] Attempt to stage resources via new API in portable runner. URL: https://github.com/apache/beam/pull/11311 R: @ihji This should be a fairly straightforward change for you. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-6217) Typos on javadoc
[
https://issues.apache.org/jira/browse/BEAM-6217?focusedWorklogId=415787=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415787
]
ASF GitHub Bot logged work on BEAM-6217:
Author: ASF GitHub Bot
Created on: 03/Apr/20 22:25
Start Date: 03/Apr/20 22:25
Worklog Time Spent: 10m
Work Description: AldairCoronel commented on issue #11310: [BEAM-6217]
Fixing typos on javadoc
URL: https://github.com/apache/beam/pull/11310#issuecomment-608727463
R: @pabloem
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 415787)
Time Spent: 20m (was: 10m)
> Typos on javadoc
>
>
> Key: BEAM-6217
> URL: https://issues.apache.org/jira/browse/BEAM-6217
> Project: Beam
> Issue Type: Bug
> Components: website
>Reporter: Rose Nguyen
>Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> On the javadocs [https://beam.apache.org/releases/javadoc/2.1.0/]
> *View*
> Change tranform->transform in "When a
> [{{ParDo}}|https://beam.apache.org/releases/javadoc/2.1.0/org/apache/beam/sdk/transforms/ParDo.html]
> tranform is processing a main input"
> *AfterWatermark*
> Change therafter->thereafter in "window and then immediately therafter when
> any late data arrives"
> Change Additionaly->Additionally in "Additionaly firings before or after the
> watermark "
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-6217) Typos on javadoc
[ https://issues.apache.org/jira/browse/BEAM-6217?focusedWorklogId=415786=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415786 ] ASF GitHub Bot logged work on BEAM-6217: Author: ASF GitHub Bot Created on: 03/Apr/20 22:24 Start Date: 03/Apr/20 22:24 Worklog Time Spent: 10m Work Description: AldairCoronel commented on pull request #11310: [BEAM-6217] Fixing typos on javadoc URL: https://github.com/apache/beam/pull/11310 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
[jira] [Commented] (BEAM-9690) Go build failing: undefined: primitives.Reshuffle(KV)
[ https://issues.apache.org/jira/browse/BEAM-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074936#comment-17074936 ] Robert Burke commented on BEAM-9690: I've been unable to replicate this issue locally, and the post commits are differently broken at present due to artifact issues, though when they were first committed, they did correctly run in post commit. > Go build failing: undefined: primitives.Reshuffle(KV) > - > > Key: BEAM-9690 > URL: https://issues.apache.org/jira/browse/BEAM-9690 > Project: Beam > Issue Type: Bug > Components: sdk-go >Reporter: Kyle Weaver >Assignee: Robert Burke >Priority: Major > > Go SDK build is failing on head (1d3e3ef9ffb4aaa913dc223d92626ca9f0f43207). I > tried ./gradlew sdks:go:clean but it didn't seem to make a difference. > Logs: > ./gradlew :sdks:go:container:docker > Resolving dependencies... > # github.com/apache/beam/sdks/go/test/integration > .gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/integration/driver.go:67:27: > undefined: primitives.Reshuffle > .gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/integration/driver.go:68:29: > undefined: primitives.ReshuffleKV > > Task :sdks:go:buildDarwinAmd64 FAILED > FAILURE: Build failed with an exception. > * What went wrong: > Execution failed for task ':sdks:go:buildDarwinAmd64'. > > Build failed due to return code 2 of: > Command: >/Users/kcweaver/.gradle/go/binary/1.12/go/bin/go build -o > ./build/bin/integration github.com/apache/beam/sdks/go/test/integration > Env: >GOEXE= > > GOPATH=/Users/kcweaver/go/src/github.com/apache/beam/sdks/go/.gogradle/project_gopath >GOROOT=/Users/kcweaver/.gradle/go/binary/1.12/go >GOOS=darwin >GOARCH=amd64 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9322) Python SDK ignores manually set PCollection tags
[ https://issues.apache.org/jira/browse/BEAM-9322?focusedWorklogId=415779=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415779 ] ASF GitHub Bot logged work on BEAM-9322: Author: ASF GitHub Bot Created on: 03/Apr/20 22:15 Start Date: 03/Apr/20 22:15 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11283: [BEAM-9322] [BEAM-1833] Better naming for composite transform output tags. URL: https://github.com/apache/beam/pull/11283#issuecomment-608712134 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415779) Time Spent: 3.5h (was: 3h 20m) > Python SDK ignores manually set PCollection tags > > > Key: BEAM-9322 > URL: https://issues.apache.org/jira/browse/BEAM-9322 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Critical > Fix For: 2.21.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > The Python SDK currently ignores any tags set on PCollections manually when > applying PTransforms when adding the PCollection to the PTransform > [outputs|[https://github.com/apache/beam/blob/688a4ea53f315ec2aa2d37602fd78496fca8bb4f/sdks/python/apache_beam/pipeline.py#L595]]. > In the > [add_output|[https://github.com/apache/beam/blob/688a4ea53f315ec2aa2d37602fd78496fca8bb4f/sdks/python/apache_beam/pipeline.py#L872]] > method, the tag is set to None for all PValues, meaning the output tags are > set to an enumeration index over the PCollection outputs. The tags are not > propagated to correctly which can be a problem on relying on the output > PCollection tags to match the user set values. > The fix is to correct BEAM-1833, and always pass in the tags. However, that > doesn't fix the problem for nested PCollections. If you have a dict of lists > of PCollections, what should their tags be correctly set to? In order to fix > this, first propagate the correct tag then talk with the community about the > best auto-generated tags. > Some users may rely on the old implementation, so a flag will be created: > "force_generated_pcollection_output_ids" and be default set to False. If > True, this will go to the old implementation and generate tags for > PCollections. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9322) Python SDK ignores manually set PCollection tags
[ https://issues.apache.org/jira/browse/BEAM-9322?focusedWorklogId=415780=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415780 ] ASF GitHub Bot logged work on BEAM-9322: Author: ASF GitHub Bot Created on: 03/Apr/20 22:15 Start Date: 03/Apr/20 22:15 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11283: [BEAM-9322] [BEAM-1833] Better naming for composite transform output tags. URL: https://github.com/apache/beam/pull/11283#issuecomment-608712255 @rohdesamuel is this good to go? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415780) Time Spent: 3h 40m (was: 3.5h) > Python SDK ignores manually set PCollection tags > > > Key: BEAM-9322 > URL: https://issues.apache.org/jira/browse/BEAM-9322 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Critical > Fix For: 2.21.0 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > The Python SDK currently ignores any tags set on PCollections manually when > applying PTransforms when adding the PCollection to the PTransform > [outputs|[https://github.com/apache/beam/blob/688a4ea53f315ec2aa2d37602fd78496fca8bb4f/sdks/python/apache_beam/pipeline.py#L595]]. > In the > [add_output|[https://github.com/apache/beam/blob/688a4ea53f315ec2aa2d37602fd78496fca8bb4f/sdks/python/apache_beam/pipeline.py#L872]] > method, the tag is set to None for all PValues, meaning the output tags are > set to an enumeration index over the PCollection outputs. The tags are not > propagated to correctly which can be a problem on relying on the output > PCollection tags to match the user set values. > The fix is to correct BEAM-1833, and always pass in the tags. However, that > doesn't fix the problem for nested PCollections. If you have a dict of lists > of PCollections, what should their tags be correctly set to? In order to fix > this, first propagate the correct tag then talk with the community about the > best auto-generated tags. > Some users may rely on the old implementation, so a flag will be created: > "force_generated_pcollection_output_ids" and be default set to False. If > True, this will go to the old implementation and generate tags for > PCollections. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9693) BeamZetaSqlCalcRel getBoolValue IllegalStateException
[
https://issues.apache.org/jira/browse/BEAM-9693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9693:
-
Description:
two failures in shard 3
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalStateException
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:491)
at com.google.zetasql.Value.getBoolValue(Value.java:225)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:184)
{code}
{code}
Apr 01, 2020 6:01:11 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT SUM(double_val) FROM TableLarge WHERE
double_val > 0
Apr 01, 2020 6:01:26 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT MIN(double_val), MAX(double_val) FROM
TableLarge WHERE double_val > 0
{code}
was:
one failure in shard 3
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalStateException
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
[jira] [Updated] (BEAM-9693) BeamZetaSqlCalcRel getBoolValue IllegalStateException
[
https://issues.apache.org/jira/browse/BEAM-9693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9693:
-
Description:
three failures in shard 3
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalStateException
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:491)
at com.google.zetasql.Value.getBoolValue(Value.java:225)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:184)
{code}
{code}
Apr 01, 2020 6:01:11 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT SUM(double_val) FROM TableLarge WHERE
double_val > 0
Apr 01, 2020 6:01:26 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT MIN(double_val), MAX(double_val) FROM
TableLarge WHERE double_val > 0
Apr 01, 2020 6:01:28 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT MIN(double_val), MAX(double_val) FROM
TableLarge WHERE double_val <= 0
{code}
was:
two failures in shard 3
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalStateException
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
[jira] [Created] (BEAM-9694) UNION ALL NULL StringUtf8Coder EOFException
Andrew Pilloud created BEAM-9694:
Summary: UNION ALL NULL StringUtf8Coder EOFException
Key: BEAM-9694
URL: https://issues.apache.org/jira/browse/BEAM-9694
Project: Beam
Issue Type: Bug
Components: dsl-sql-zetasql
Reporter: Andrew Pilloud
one failure in shard 3
{code:java}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.RuntimeException: org.apache.beam.sdk.coders.CoderException:
java.io.EOFException
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException:
org.apache.beam.sdk.coders.CoderException: java.io.EOFException
at
org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.add(ImmutabilityCheckingBundleFactory.java:114)
at
org.apache.beam.runners.direct.ParDoEvaluator$BundleOutputManager.output(ParDoEvaluator.java:299)
at
org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:259)
at
org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.access$800(SimpleDoFnRunner.java:79)
at
org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:405)
at
org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:393)
at
org.apache.beam.sdk.transforms.join.CoGroupByKey$ConstructUnionTableFn.processElement(CoGroupByKey.java:175)
Caused by: org.apache.beam.sdk.coders.CoderException: java.io.EOFException
at
org.apache.beam.sdk.coders.StringUtf8Coder.decode(StringUtf8Coder.java:104)
at
org.apache.beam.sdk.coders.StringUtf8Coder.decode(StringUtf8Coder.java:90)
at
org.apache.beam.sdk.coders.StringUtf8Coder.decode(StringUtf8Coder.java:37)
at
org.apache.beam.sdk.coders.IterableLikeCoder.decode(IterableLikeCoder.java:124)
at
org.apache.beam.sdk.coders.IterableLikeCoder.decode(IterableLikeCoder.java:60)
at
org.apache.beam.sdk.coders.RowCoderGenerator$DecodeInstruction.decodeDelegate(RowCoderGenerator.java:348)
at org.apache.beam.sdk.coders.Coder$ByteBuddy$a8Oia16G.decode(Unknown
Source)
at org.apache.beam.sdk.coders.Coder$ByteBuddy$a8Oia16G.decode(Unknown
Source)
at org.apache.beam.sdk.schemas.SchemaCoder.decode(SchemaCoder.java:120)
at org.apache.beam.sdk.coders.KvCoder.decode(KvCoder.java:81)
at org.apache.beam.sdk.coders.KvCoder.decode(KvCoder.java:36)
at
org.apache.beam.sdk.util.CoderUtils.decodeFromSafeStream(CoderUtils.java:115)
at
org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:98)
at
org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:92)
at
[jira] [Commented] (BEAM-9372) Drop support for Python 3.5 versions lower than 3.5.3
[ https://issues.apache.org/jira/browse/BEAM-9372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074930#comment-17074930 ] Valentyn Tymofieiev commented on BEAM-9372: --- Actually, Ubuntu 16.04 EOL is in 2024 [1], I was using a wrong link. End of support is April 2021. [1] https://wiki.ubuntu.com/Releases > Drop support for Python 3.5 versions lower than 3.5.3 > - > > Key: BEAM-9372 > URL: https://issues.apache.org/jira/browse/BEAM-9372 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: Major > > Supporting Python 3.5.1, 3.5.2 requires nontrivial amount of work in type > inference machinery, but may provide little benefit. Most recently, this > surfaced in [1]. > To require Python 3.5.3 and higher we need to migrate Jenkins infrastructure > to Python 3.5.3 (see [2]), and update python_requires section in setup.py > once we are ready to drop support. > > https://github.com/apache/beam/pull/10821 > https://issues.apache.org/jira/browse/BEAM-8152 > cc: [~udim] [~chadrik] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9693) BeamZetaSqlCalcRel getBoolValue IllegalStateException
[
https://issues.apache.org/jira/browse/BEAM-9693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9693:
-
Status: Open (was: Triage Needed)
> BeamZetaSqlCalcRel getBoolValue IllegalStateException
> -
>
> Key: BEAM-9693
> URL: https://issues.apache.org/jira/browse/BEAM-9693
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Priority: Minor
> Labels: zetasql-compliance
>
> one failure in shard 3
> {code}
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.lang.IllegalStateException
> at
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
> at
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
> at
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
> at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:491)
> at com.google.zetasql.Value.getBoolValue(Value.java:225)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:184)
> {code}
> {code}
> Apr 01, 2020 6:01:11 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: SELECT SUM(double_val) FROM TableLarge WHERE
> double_val > 0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9687) Names of temporary files created by interactive runner include characters invalid on some platforms.
[
https://issues.apache.org/jira/browse/BEAM-9687?focusedWorklogId=415774=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415774
]
ASF GitHub Bot logged work on BEAM-9687:
Author: ASF GitHub Bot
Created on: 03/Apr/20 22:05
Start Date: 03/Apr/20 22:05
Worklog Time Spent: 10m
Work Description: aaltay commented on pull request #11306: [BEAM-9687]
Change delimeter to a dash as the pipe is a reserved symbol in Windows in file
names
URL: https://github.com/apache/beam/pull/11306
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 415774)
Time Spent: 40m (was: 0.5h)
> Names of temporary files created by interactive runner include characters
> invalid on some platforms.
>
>
> Key: BEAM-9687
> URL: https://issues.apache.org/jira/browse/BEAM-9687
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Sam Rohde
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Temporary files created by interactive runner in streaming scenarios include
> pipe '|' characters, which are not allowed in filenames on Windows platform.
> This causes test failures on a Windows platform:
>
> python setup.py nosetests --tests
> apache_beam/runners/interactive/pipeline_instrument_test.py:PipelineInstrumentTest.test_instrument_example_unbounded_pipeline_to_multiple_read_cache
> ==
> ERROR: Tests that the instrumenter works for multiple unbounded sources.
> --
> Traceback (most recent call last):
> File
> "C:\projects\apache_beam\runners\interactive\pipeline_instrument_test.py",
> line 698, in test_instrument_example_unbounded_pipeline_to_multiple_read_cache
> self._mock_write_cache([b''], cache_key)
> File
> "C:\projects\apache_beam\runners\interactive\pipeline_instrument_test.py",
> line 227, in _mock_write_cache
> ie.current_env().cache_manager().write(values, *labels)
> File
> "C:\projects\apache_beam\runners\interactive\caching\streaming_cache.py",
> line 323, in write
> with open(filepath, 'ab') as f:
> IOError: [Errno 22] invalid mode ('ab') or filename:
> 'c:\\users\\deft-t~1\\appdata\\local\\temp\\2\\interactive-temp-xwg5qi\\full\\pcoll_1|149781752|149781920|1
> 49231600'
>
> [1]
> https://github.com/apache/beam/blob/e6b37c44d542969b6104fc97ee6f25b6f7d2ddba/sdks/python/apache_beam/runners/interactive/caching/streaming_cache.py#L323
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9693) BeamZetaSqlCalcRel getBoolValue IllegalStateException
Andrew Pilloud created BEAM-9693:
Summary: BeamZetaSqlCalcRel getBoolValue IllegalStateException
Key: BEAM-9693
URL: https://issues.apache.org/jira/browse/BEAM-9693
Project: Beam
Issue Type: Bug
Components: dsl-sql-zetasql
Reporter: Andrew Pilloud
one failure in shard 3
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalStateException
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:329)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:491)
at com.google.zetasql.Value.getBoolValue(Value.java:225)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:184)
{code}
{code}
Apr 01, 2020 6:01:11 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT SUM(double_val) FROM TableLarge WHERE
double_val > 0
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-9523) GROUP BY DOUBLE fails at runtime instead of planning time
[
https://issues.apache.org/jira/browse/BEAM-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud resolved BEAM-9523.
--
Fix Version/s: Not applicable
Resolution: Duplicate
> GROUP BY DOUBLE fails at runtime instead of planning time
> -
>
> Key: BEAM-9523
> URL: https://issues.apache.org/jira/browse/BEAM-9523
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Priority: Minor
> Labels: zetasql-compliance
> Fix For: Not applicable
>
>
> This should fail earlier.
> {code:java}
> Mar 16, 2020 1:27:32 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: SELECT COUNT(a) FROM (
> SELECT a FROM (SELECT 1.2 a UNION ALL SELECT 2.3 UNION ALL SELECT 3.4) LIMIT
> 1)
> Mar 16, 2020 1:27:33 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@7946abe5
> java.lang.IllegalStateException: the keyCoder of a GroupByKey must be
> deterministic
> at org.apache.beam.sdk.transforms.GroupByKey.expand(GroupByKey.java:234)
> at org.apache.beam.sdk.transforms.GroupByKey.expand(GroupByKey.java:110)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:542)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:493)
> at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:368)
> at
> org.apache.beam.sdk.transforms.join.CoGroupByKey.expand(CoGroupByKey.java:118)
> at
> org.apache.beam.sdk.transforms.join.CoGroupByKey.expand(CoGroupByKey.java:71)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:542)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:476)
> at
> org.apache.beam.sdk.transforms.join.KeyedPCollectionTuple.apply(KeyedPCollectionTuple.java:108)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSetOperatorRelBase.expand(BeamSetOperatorRelBase.java:96)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSetOperatorRelBase.expand(BeamSetOperatorRelBase.java:41)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:542)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:493)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.toPCollection(BeamSqlRelUtils.java:69)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.lambda$buildPCollectionList$0(BeamSqlRelUtils.java:50)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Iterator.forEachRemaining(Iterator.java:116)
> at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.buildPCollectionList(BeamSqlRelUtils.java:51)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.toPCollection(BeamSqlRelUtils.java:67)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.lambda$buildPCollectionList$0(BeamSqlRelUtils.java:50)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Iterator.forEachRemaining(Iterator.java:116)
> at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.buildPCollectionList(BeamSqlRelUtils.java:51)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.toPCollection(BeamSqlRelUtils.java:67)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.lambda$buildPCollectionList$0(BeamSqlRelUtils.java:50)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
[jira] [Work logged] (BEAM-9577) Update artifact staging and retrieval protocols to be dependency aware.
[ https://issues.apache.org/jira/browse/BEAM-9577?focusedWorklogId=415770=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415770 ] ASF GitHub Bot logged work on BEAM-9577: Author: ASF GitHub Bot Created on: 03/Apr/20 21:57 Start Date: 03/Apr/20 21:57 Worklog Time Spent: 10m Work Description: lostluck commented on issue #11305: [BEAM-9577] Update container boot code to stage from dependencies, if present. URL: https://github.com/apache/beam/pull/11305#issuecomment-608705326 That appears to have been the case since https://github.com/apache/beam/pull/11305 Edit: Whoopes, wrong queue. That's this PR. It's been broken since this PR PostCommit run https://builds.apache.org/job/beam_PostCommit_Go/6624/ But then it was complaining about the legacy service missing. This at least moves things forward a bit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415770) Time Spent: 11h 40m (was: 11.5h) > Update artifact staging and retrieval protocols to be dependency aware. > --- > > Key: BEAM-9577 > URL: https://issues.apache.org/jira/browse/BEAM-9577 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 11h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9577) Update artifact staging and retrieval protocols to be dependency aware.
[ https://issues.apache.org/jira/browse/BEAM-9577?focusedWorklogId=415769=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415769 ] ASF GitHub Bot logged work on BEAM-9577: Author: ASF GitHub Bot Created on: 03/Apr/20 21:55 Start Date: 03/Apr/20 21:55 Worklog Time Spent: 10m Work Description: lostluck commented on issue #11305: [BEAM-9577] Update container boot code to stage from dependencies, if present. URL: https://github.com/apache/beam/pull/11305#issuecomment-608705326 That appears to have been the case since https://github.com/apache/beam/pull/11305 Edit: Whoopes, wrong queue. That's this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415769) Time Spent: 11.5h (was: 11h 20m) > Update artifact staging and retrieval protocols to be dependency aware. > --- > > Key: BEAM-9577 > URL: https://issues.apache.org/jira/browse/BEAM-9577 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 11.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9577) Update artifact staging and retrieval protocols to be dependency aware.
[ https://issues.apache.org/jira/browse/BEAM-9577?focusedWorklogId=415768=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415768 ] ASF GitHub Bot logged work on BEAM-9577: Author: ASF GitHub Bot Created on: 03/Apr/20 21:54 Start Date: 03/Apr/20 21:54 Worklog Time Spent: 10m Work Description: lostluck commented on issue #11305: [BEAM-9577] Update container boot code to stage from dependencies, if present. URL: https://github.com/apache/beam/pull/11305#issuecomment-608705326 That appears to have been the case since https://github.com/apache/beam/pull/11305 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415768) Time Spent: 11h 20m (was: 11h 10m) > Update artifact staging and retrieval protocols to be dependency aware. > --- > > Key: BEAM-9577 > URL: https://issues.apache.org/jira/browse/BEAM-9577 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 11h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9577) Update artifact staging and retrieval protocols to be dependency aware.
[ https://issues.apache.org/jira/browse/BEAM-9577?focusedWorklogId=415766=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415766 ] ASF GitHub Bot logged work on BEAM-9577: Author: ASF GitHub Bot Created on: 03/Apr/20 21:53 Start Date: 03/Apr/20 21:53 Worklog Time Spent: 10m Work Description: lostluck commented on issue #11305: [BEAM-9577] Update container boot code to stage from dependencies, if present. URL: https://github.com/apache/beam/pull/11305#issuecomment-608704872 In the post commit, it doesn't look the dataflow jobs still don't like the current situation. They claim "no artifacts staged". This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415766) Time Spent: 11h 10m (was: 11h) > Update artifact staging and retrieval protocols to be dependency aware. > --- > > Key: BEAM-9577 > URL: https://issues.apache.org/jira/browse/BEAM-9577 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 11h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3097) Allow BigQuerySource to take a ValueProvider as a table input.
[ https://issues.apache.org/jira/browse/BEAM-3097?focusedWorklogId=415758=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415758 ] ASF GitHub Bot logged work on BEAM-3097: Author: ASF GitHub Bot Created on: 03/Apr/20 21:43 Start Date: 03/Apr/20 21:43 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table URL: https://github.com/apache/beam/pull/11244#discussion_r403350713 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -657,6 +664,10 @@ def estimate_size(self): kms_key=self.kms_key) size = int(job.statistics.totalBytesProcessed) return size +else: + # Size estimation is best effort. We return 0 as we have no + # access to the query that we're running. + return 0 Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415758) Time Spent: 2h 40m (was: 2.5h) > Allow BigQuerySource to take a ValueProvider as a table input. > -- > > Key: BEAM-3097 > URL: https://issues.apache.org/jira/browse/BEAM-3097 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Ed Mothershaw >Priority: Minor > Original Estimate: 2h > Time Spent: 2h 40m > Remaining Estimate: 0h > > In file sdks/python/apache_beam/io/gcp/bigquery.py, class BigQuery, line 389. > When a ValueProvider is input as table the script will fail. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3097) Allow BigQuerySource to take a ValueProvider as a table input.
[ https://issues.apache.org/jira/browse/BEAM-3097?focusedWorklogId=415756=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415756 ] ASF GitHub Bot logged work on BEAM-3097: Author: ASF GitHub Bot Created on: 03/Apr/20 21:42 Start Date: 03/Apr/20 21:42 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table URL: https://github.com/apache/beam/pull/11244#discussion_r403350549 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -641,12 +641,19 @@ def __init__( def estimate_size(self): bq = bigquery_tools.BigQueryWrapper() if self.table_reference is not None: + table_ref = self.table_reference + if (isinstance(self.table_reference, vp.ValueProvider) and + self.table_reference.is_accessible()): +table_ref = bigquery_tools.parse_table_reference( +self.table_reference.get(), self.dataset, self.project) + elif isinstance(self.table_reference, vp.ValueProvider): +# Size estimation is best effort. We return 0 as we have no +# access to the table that we're querying. +return 0 table = bq.get_table( - self.table_reference.projectId, - self.table_reference.datasetId, - self.table_reference.tableId) + table_ref.projectId, table_ref.datasetId, table_ref.tableId) return int(table.numBytes) -else: +elif self.query is not None and self.query.is_accessible(): job = bq._start_query_job( Review comment: if you look at line 663, it is a dry run : ) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415756) Time Spent: 2h 20m (was: 2h 10m) > Allow BigQuerySource to take a ValueProvider as a table input. > -- > > Key: BEAM-3097 > URL: https://issues.apache.org/jira/browse/BEAM-3097 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Ed Mothershaw >Priority: Minor > Original Estimate: 2h > Time Spent: 2h 20m > Remaining Estimate: 0h > > In file sdks/python/apache_beam/io/gcp/bigquery.py, class BigQuery, line 389. > When a ValueProvider is input as table the script will fail. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3097) Allow BigQuerySource to take a ValueProvider as a table input.
[ https://issues.apache.org/jira/browse/BEAM-3097?focusedWorklogId=415757=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415757 ] ASF GitHub Bot logged work on BEAM-3097: Author: ASF GitHub Bot Created on: 03/Apr/20 21:42 Start Date: 03/Apr/20 21:42 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table URL: https://github.com/apache/beam/pull/11244#discussion_r403350683 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -641,12 +641,19 @@ def __init__( def estimate_size(self): bq = bigquery_tools.BigQueryWrapper() if self.table_reference is not None: + table_ref = self.table_reference + if (isinstance(self.table_reference, vp.ValueProvider) and + self.table_reference.is_accessible()): +table_ref = bigquery_tools.parse_table_reference( +self.table_reference.get(), self.dataset, self.project) + elif isinstance(self.table_reference, vp.ValueProvider): +# Size estimation is best effort. We return 0 as we have no +# access to the table that we're querying. +return 0 Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415757) Time Spent: 2.5h (was: 2h 20m) > Allow BigQuerySource to take a ValueProvider as a table input. > -- > > Key: BEAM-3097 > URL: https://issues.apache.org/jira/browse/BEAM-3097 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Ed Mothershaw >Priority: Minor > Original Estimate: 2h > Time Spent: 2.5h > Remaining Estimate: 0h > > In file sdks/python/apache_beam/io/gcp/bigquery.py, class BigQuery, line 389. > When a ValueProvider is input as table the script will fail. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3097) Allow BigQuerySource to take a ValueProvider as a table input.
[ https://issues.apache.org/jira/browse/BEAM-3097?focusedWorklogId=415755=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415755 ] ASF GitHub Bot logged work on BEAM-3097: Author: ASF GitHub Bot Created on: 03/Apr/20 21:42 Start Date: 03/Apr/20 21:42 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table URL: https://github.com/apache/beam/pull/11244#discussion_r403350428 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -641,12 +641,19 @@ def __init__( def estimate_size(self): bq = bigquery_tools.BigQueryWrapper() if self.table_reference is not None: + table_ref = self.table_reference + if (isinstance(self.table_reference, vp.ValueProvider) and + self.table_reference.is_accessible()): +table_ref = bigquery_tools.parse_table_reference( +self.table_reference.get(), self.dataset, self.project) + elif isinstance(self.table_reference, vp.ValueProvider): +# Size estimation is best effort. We return 0 as we have no +# access to the table that we're querying. +return 0 table = bq.get_table( - self.table_reference.projectId, - self.table_reference.datasetId, - self.table_reference.tableId) + table_ref.projectId, table_ref.datasetId, table_ref.tableId) return int(table.numBytes) -else: +elif self.query is not None and self.query.is_accessible(): Review comment: It is supported now. In this case, the `self.query` parameter is wrapped by a ValueProvider already. This does not work for `self.table` because it may be a non-string. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415755) Time Spent: 2h 10m (was: 2h) > Allow BigQuerySource to take a ValueProvider as a table input. > -- > > Key: BEAM-3097 > URL: https://issues.apache.org/jira/browse/BEAM-3097 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Ed Mothershaw >Priority: Minor > Original Estimate: 2h > Time Spent: 2h 10m > Remaining Estimate: 0h > > In file sdks/python/apache_beam/io/gcp/bigquery.py, class BigQuery, line 389. > When a ValueProvider is input as table the script will fail. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9577) Update artifact staging and retrieval protocols to be dependency aware.
[ https://issues.apache.org/jira/browse/BEAM-9577?focusedWorklogId=415744=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415744 ] ASF GitHub Bot logged work on BEAM-9577: Author: ASF GitHub Bot Created on: 03/Apr/20 21:25 Start Date: 03/Apr/20 21:25 Worklog Time Spent: 10m Work Description: lostluck commented on issue #11305: [BEAM-9577] Update container boot code to stage from dependencies, if present. URL: https://github.com/apache/beam/pull/11305#issuecomment-608678863 Run Go PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415744) Time Spent: 11h (was: 10h 50m) > Update artifact staging and retrieval protocols to be dependency aware. > --- > > Key: BEAM-9577 > URL: https://issues.apache.org/jira/browse/BEAM-9577 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 11h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=415743=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415743 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 03/Apr/20 21:23 Start Date: 03/Apr/20 21:23 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11235: [BEAM-9618] Pull bundle descriptors. URL: https://github.com/apache/beam/pull/11235#issuecomment-608674803 @lukecwik Was there anything more you wanted me to do on this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415743) Time Spent: 1h 40m (was: 1.5h) > Allow SDKs to pull process bundle descriptors. > -- > > Key: BEAM-9618 > URL: https://issues.apache.org/jira/browse/BEAM-9618 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
