[jira] [Created] (BEAM-8441) Side-Input in Python3 fails to pickle class
Jannik Franz created BEAM-8441:
--
Summary: Side-Input in Python3 fails to pickle class
Key: BEAM-8441
URL: https://issues.apache.org/jira/browse/BEAM-8441
Project: Beam
Issue Type: Bug
Components: sdk-py-core
Reporter: Jannik Franz
When running Apache Beam with Python3 on Google Cloud Dataflow Sideinputs don't
work.
When testing it in the local/direct runner there seems to be no issue.
{code:java}
class FlattenCustomActions(beam.PTransform):
""" Transforms Facebook Day ActionsOnly retains actions with
custom_conversions
Flattens the actions
Adds custom conversions names using a side input
"""
def __init__(self, conversions):
super(FlattenCustomActions, self).__init__()
self.conversions = conversionsdef expand(self, input_or_inputs):
return (
input_or_inputs
| "FlattenActions" >> beam.ParDo(flatten_filter_actions)
| "AddConversionName" >> beam.Map(add_conversion_name,
self.conversions)
)
# ...
# in run():
pipeline_options = PipelineOptions(pipeline_args)
pipeline_options.view_as(SetupOptions).save_main_session = True
p = beam.Pipeline(options=pipeline_options)
conversions_output = (
p
| "ReadConversions" >> ReadFromText(known_args.input_conversions,
coder=JsonCoder())
| TransformConversionMetadata()
)(
conversions_output
| "WriteConversions"
>> WriteCoerced(
known_args.output_conversions,
known_args.output_type,
schema_path=BIGQUERY_SCHEMA_CONVERSIONS_PATH,
)
)(
p
| ReadFacebookJson(known_args.input, retain_root_fields=True)
| FlattenCustomActions(beam.pvalue.AsList(conversions_output))
| "WriteActions"
>> WriteCoerced(
known_args.output, known_args.output_type,
schema_path=BIGQUERY_SCHEMA_ACTIONS_PATH
)
){code}
I receive the following Traceback in Dataflow:
{code:java}
Traceback (most recent call last): File
"/usr/local/lib/python3.6/site-packages/dataflow_worker/batchworker.py", line
773, in run self._load_main_session(self.local_staging_directory) File
"/usr/local/lib/python3.6/site-packages/dataflow_worker/batchworker.py", line
489, in _load_main_session pickler.load_session(session_file) File
"/usr/local/lib/python3.6/site-packages/apache_beam/internal/pickler.py", line
287, in load_session return dill.load_session(file_path) File
"/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 410, in
load_session module = unpickler.load() File
"/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 474, in find_class
return StockUnpickler.find_class(self, module, name) AttributeError: Can't get
attribute 'FlattenCustomActions' on
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5690) Issue with GroupByKey in BeamSql using SparkRunner
[
https://issues.apache.org/jira/browse/BEAM-5690?focusedWorklogId=331255&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331255
]
ASF GitHub Bot logged work on BEAM-5690:
Author: ASF GitHub Bot
Created on: 21/Oct/19 08:04
Start Date: 21/Oct/19 08:04
Worklog Time Spent: 10m
Work Description: echauchot commented on issue #9567: [BEAM-5690] Fix
Zero value issue with GroupByKey/CountByKey in SparkRunner
URL: https://github.com/apache/beam/pull/9567#issuecomment-544397900
Run Spark ValidatesRunner
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331255)
Time Spent: 3h 50m (was: 3h 40m)
> Issue with GroupByKey in BeamSql using SparkRunner
> --
>
> Key: BEAM-5690
> URL: https://issues.apache.org/jira/browse/BEAM-5690
> Project: Beam
> Issue Type: Task
> Components: runner-spark
>Reporter: Kenneth Knowles
>Priority: Major
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> Reported on user@
> {quote}We are trying to setup a pipeline with using BeamSql and the trigger
> used is default (AfterWatermark crosses the window).
> Below is the pipeline:
>
>KafkaSource (KafkaIO)
>---> Windowing (FixedWindow 1min)
>---> BeamSql
>---> KafkaSink (KafkaIO)
>
> We are using Spark Runner for this.
> The BeamSql query is:
> {code}select Col3, count(*) as count_col1 from PCOLLECTION GROUP BY Col3{code}
> We are grouping by Col3 which is a string. It can hold values string[0-9].
>
> The records are getting emitted out at 1 min to kafka sink, but the output
> record in kafka is not as expected.
> Below is the output observed: (WST and WET are indicators for window start
> time and window end time)
> {code}
> {"count_col1":1,"Col3":"string5","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":3,"Col3":"string7","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":2,"Col3":"string8","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":1,"Col3":"string2","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":1,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 0}
> {code}
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5690) Issue with GroupByKey in BeamSql using SparkRunner
[
https://issues.apache.org/jira/browse/BEAM-5690?focusedWorklogId=331253&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331253
]
ASF GitHub Bot logged work on BEAM-5690:
Author: ASF GitHub Bot
Created on: 21/Oct/19 08:03
Start Date: 21/Oct/19 08:03
Worklog Time Spent: 10m
Work Description: echauchot commented on issue #9567: [BEAM-5690] Fix
Zero value issue with GroupByKey/CountByKey in SparkRunner
URL: https://github.com/apache/beam/pull/9567#issuecomment-544397376
Run Spark ValidatesRunner
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331253)
Time Spent: 3.5h (was: 3h 20m)
> Issue with GroupByKey in BeamSql using SparkRunner
> --
>
> Key: BEAM-5690
> URL: https://issues.apache.org/jira/browse/BEAM-5690
> Project: Beam
> Issue Type: Task
> Components: runner-spark
>Reporter: Kenneth Knowles
>Priority: Major
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Reported on user@
> {quote}We are trying to setup a pipeline with using BeamSql and the trigger
> used is default (AfterWatermark crosses the window).
> Below is the pipeline:
>
>KafkaSource (KafkaIO)
>---> Windowing (FixedWindow 1min)
>---> BeamSql
>---> KafkaSink (KafkaIO)
>
> We are using Spark Runner for this.
> The BeamSql query is:
> {code}select Col3, count(*) as count_col1 from PCOLLECTION GROUP BY Col3{code}
> We are grouping by Col3 which is a string. It can hold values string[0-9].
>
> The records are getting emitted out at 1 min to kafka sink, but the output
> record in kafka is not as expected.
> Below is the output observed: (WST and WET are indicators for window start
> time and window end time)
> {code}
> {"count_col1":1,"Col3":"string5","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":3,"Col3":"string7","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":2,"Col3":"string8","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":1,"Col3":"string2","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":1,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 0}
> {code}
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5690) Issue with GroupByKey in BeamSql using SparkRunner
[
https://issues.apache.org/jira/browse/BEAM-5690?focusedWorklogId=331251&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331251
]
ASF GitHub Bot logged work on BEAM-5690:
Author: ASF GitHub Bot
Created on: 21/Oct/19 08:02
Start Date: 21/Oct/19 08:02
Worklog Time Spent: 10m
Work Description: echauchot commented on pull request #9567: [BEAM-5690]
Fix Zero value issue with GroupByKey/CountByKey in SparkRunner
URL: https://github.com/apache/beam/pull/9567#discussion_r336881029
##
File path:
runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/SparkGroupAlsoByWindowViaWindowSet.java
##
@@ -338,6 +338,16 @@ public void outputWindowedValue(
outputHolder.getWindowedValues();
if (!outputs.isEmpty() || !stateInternals.getState().isEmpty()) {
+
+Collection expiredInternalTimers =
+LateDataUtils.getExpiredTimers(
+timerInternals.getTimers(),
+windowingStrategy,
+timerInternals.currentInputWatermarkTime());
+
+// Remove the expired timer from the timerInternals structure
+expiredInternalTimers.forEach(timerInternals::deleteTimer);
Review comment:
@bmv126 I would prefer that you add a method to LateDataUtils that drops
expired timers rather than a method that just lists them.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331251)
Time Spent: 3h 10m (was: 3h)
> Issue with GroupByKey in BeamSql using SparkRunner
> --
>
> Key: BEAM-5690
> URL: https://issues.apache.org/jira/browse/BEAM-5690
> Project: Beam
> Issue Type: Task
> Components: runner-spark
>Reporter: Kenneth Knowles
>Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Reported on user@
> {quote}We are trying to setup a pipeline with using BeamSql and the trigger
> used is default (AfterWatermark crosses the window).
> Below is the pipeline:
>
>KafkaSource (KafkaIO)
>---> Windowing (FixedWindow 1min)
>---> BeamSql
>---> KafkaSink (KafkaIO)
>
> We are using Spark Runner for this.
> The BeamSql query is:
> {code}select Col3, count(*) as count_col1 from PCOLLECTION GROUP BY Col3{code}
> We are grouping by Col3 which is a string. It can hold values string[0-9].
>
> The records are getting emitted out at 1 min to kafka sink, but the output
> record in kafka is not as expected.
> Below is the output observed: (WST and WET are indicators for window start
> time and window end time)
> {code}
> {"count_col1":1,"Col3":"string5","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":3,"Col3":"string7","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":2,"Col3":"string8","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":1,"Col3":"string2","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":1,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 0}
> {code}
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5690) Issue with GroupByKey in BeamSql using SparkRunner
[
https://issues.apache.org/jira/browse/BEAM-5690?focusedWorklogId=331256&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331256
]
ASF GitHub Bot logged work on BEAM-5690:
Author: ASF GitHub Bot
Created on: 21/Oct/19 08:04
Start Date: 21/Oct/19 08:04
Worklog Time Spent: 10m
Work Description: echauchot commented on issue #9567: [BEAM-5690] Fix
Zero value issue with GroupByKey/CountByKey in SparkRunner
URL: https://github.com/apache/beam/pull/9567#issuecomment-544398078
Run Spark ValidatesRunner
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331256)
Time Spent: 4h (was: 3h 50m)
> Issue with GroupByKey in BeamSql using SparkRunner
> --
>
> Key: BEAM-5690
> URL: https://issues.apache.org/jira/browse/BEAM-5690
> Project: Beam
> Issue Type: Task
> Components: runner-spark
>Reporter: Kenneth Knowles
>Priority: Major
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Reported on user@
> {quote}We are trying to setup a pipeline with using BeamSql and the trigger
> used is default (AfterWatermark crosses the window).
> Below is the pipeline:
>
>KafkaSource (KafkaIO)
>---> Windowing (FixedWindow 1min)
>---> BeamSql
>---> KafkaSink (KafkaIO)
>
> We are using Spark Runner for this.
> The BeamSql query is:
> {code}select Col3, count(*) as count_col1 from PCOLLECTION GROUP BY Col3{code}
> We are grouping by Col3 which is a string. It can hold values string[0-9].
>
> The records are getting emitted out at 1 min to kafka sink, but the output
> record in kafka is not as expected.
> Below is the output observed: (WST and WET are indicators for window start
> time and window end time)
> {code}
> {"count_col1":1,"Col3":"string5","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":3,"Col3":"string7","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":2,"Col3":"string8","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":1,"Col3":"string2","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":1,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 0}
> {code}
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5690) Issue with GroupByKey in BeamSql using SparkRunner
[
https://issues.apache.org/jira/browse/BEAM-5690?focusedWorklogId=331252&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331252
]
ASF GitHub Bot logged work on BEAM-5690:
Author: ASF GitHub Bot
Created on: 21/Oct/19 08:02
Start Date: 21/Oct/19 08:02
Worklog Time Spent: 10m
Work Description: echauchot commented on issue #9567: [BEAM-5690] Fix
Zero value issue with GroupByKey/CountByKey in SparkRunner
URL: https://github.com/apache/beam/pull/9567#issuecomment-544397376
Run Spark ValidatesRunner
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331252)
Time Spent: 3h 20m (was: 3h 10m)
> Issue with GroupByKey in BeamSql using SparkRunner
> --
>
> Key: BEAM-5690
> URL: https://issues.apache.org/jira/browse/BEAM-5690
> Project: Beam
> Issue Type: Task
> Components: runner-spark
>Reporter: Kenneth Knowles
>Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Reported on user@
> {quote}We are trying to setup a pipeline with using BeamSql and the trigger
> used is default (AfterWatermark crosses the window).
> Below is the pipeline:
>
>KafkaSource (KafkaIO)
>---> Windowing (FixedWindow 1min)
>---> BeamSql
>---> KafkaSink (KafkaIO)
>
> We are using Spark Runner for this.
> The BeamSql query is:
> {code}select Col3, count(*) as count_col1 from PCOLLECTION GROUP BY Col3{code}
> We are grouping by Col3 which is a string. It can hold values string[0-9].
>
> The records are getting emitted out at 1 min to kafka sink, but the output
> record in kafka is not as expected.
> Below is the output observed: (WST and WET are indicators for window start
> time and window end time)
> {code}
> {"count_col1":1,"Col3":"string5","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":3,"Col3":"string7","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":2,"Col3":"string8","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":1,"Col3":"string2","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":1,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 0}
> {code}
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5690) Issue with GroupByKey in BeamSql using SparkRunner
[
https://issues.apache.org/jira/browse/BEAM-5690?focusedWorklogId=331254&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331254
]
ASF GitHub Bot logged work on BEAM-5690:
Author: ASF GitHub Bot
Created on: 21/Oct/19 08:03
Start Date: 21/Oct/19 08:03
Worklog Time Spent: 10m
Work Description: echauchot commented on issue #9567: [BEAM-5690] Fix
Zero value issue with GroupByKey/CountByKey in SparkRunner
URL: https://github.com/apache/beam/pull/9567#issuecomment-544397900
Run Spark ValidatesRunner
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331254)
Time Spent: 3h 40m (was: 3.5h)
> Issue with GroupByKey in BeamSql using SparkRunner
> --
>
> Key: BEAM-5690
> URL: https://issues.apache.org/jira/browse/BEAM-5690
> Project: Beam
> Issue Type: Task
> Components: runner-spark
>Reporter: Kenneth Knowles
>Priority: Major
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Reported on user@
> {quote}We are trying to setup a pipeline with using BeamSql and the trigger
> used is default (AfterWatermark crosses the window).
> Below is the pipeline:
>
>KafkaSource (KafkaIO)
>---> Windowing (FixedWindow 1min)
>---> BeamSql
>---> KafkaSink (KafkaIO)
>
> We are using Spark Runner for this.
> The BeamSql query is:
> {code}select Col3, count(*) as count_col1 from PCOLLECTION GROUP BY Col3{code}
> We are grouping by Col3 which is a string. It can hold values string[0-9].
>
> The records are getting emitted out at 1 min to kafka sink, but the output
> record in kafka is not as expected.
> Below is the output observed: (WST and WET are indicators for window start
> time and window end time)
> {code}
> {"count_col1":1,"Col3":"string5","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":3,"Col3":"string7","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":2,"Col3":"string8","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":1,"Col3":"string2","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":1,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 0}
> {code}
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5690) Issue with GroupByKey in BeamSql using SparkRunner
[
https://issues.apache.org/jira/browse/BEAM-5690?focusedWorklogId=331261&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331261
]
ASF GitHub Bot logged work on BEAM-5690:
Author: ASF GitHub Bot
Created on: 21/Oct/19 08:16
Start Date: 21/Oct/19 08:16
Worklog Time Spent: 10m
Work Description: echauchot commented on issue #9567: [BEAM-5690] Fix
Zero value issue with GroupByKey/CountByKey in SparkRunner
URL: https://github.com/apache/beam/pull/9567#issuecomment-544402252
Trigerring of the validates runner tests from the PR seems to be broken. I'm
running them from my laptop on your PR branch and they pass.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331261)
Time Spent: 4h 10m (was: 4h)
> Issue with GroupByKey in BeamSql using SparkRunner
> --
>
> Key: BEAM-5690
> URL: https://issues.apache.org/jira/browse/BEAM-5690
> Project: Beam
> Issue Type: Task
> Components: runner-spark
>Reporter: Kenneth Knowles
>Priority: Major
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Reported on user@
> {quote}We are trying to setup a pipeline with using BeamSql and the trigger
> used is default (AfterWatermark crosses the window).
> Below is the pipeline:
>
>KafkaSource (KafkaIO)
>---> Windowing (FixedWindow 1min)
>---> BeamSql
>---> KafkaSink (KafkaIO)
>
> We are using Spark Runner for this.
> The BeamSql query is:
> {code}select Col3, count(*) as count_col1 from PCOLLECTION GROUP BY Col3{code}
> We are grouping by Col3 which is a string. It can hold values string[0-9].
>
> The records are getting emitted out at 1 min to kafka sink, but the output
> record in kafka is not as expected.
> Below is the output observed: (WST and WET are indicators for window start
> time and window end time)
> {code}
> {"count_col1":1,"Col3":"string5","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":3,"Col3":"string7","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":2,"Col3":"string8","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":1,"Col3":"string2","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":1,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 0}
> {code}
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5690) Issue with GroupByKey in BeamSql using SparkRunner
[
https://issues.apache.org/jira/browse/BEAM-5690?focusedWorklogId=331262&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331262
]
ASF GitHub Bot logged work on BEAM-5690:
Author: ASF GitHub Bot
Created on: 21/Oct/19 08:16
Start Date: 21/Oct/19 08:16
Worklog Time Spent: 10m
Work Description: echauchot commented on issue #9567: [BEAM-5690] Fix
Zero value issue with GroupByKey/CountByKey in SparkRunner
URL: https://github.com/apache/beam/pull/9567#issuecomment-544402252
Triggering of the validates runner tests from the PR seems to be broken. I'm
running them from my laptop on your PR branch and they pass.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331262)
Time Spent: 4h 20m (was: 4h 10m)
> Issue with GroupByKey in BeamSql using SparkRunner
> --
>
> Key: BEAM-5690
> URL: https://issues.apache.org/jira/browse/BEAM-5690
> Project: Beam
> Issue Type: Task
> Components: runner-spark
>Reporter: Kenneth Knowles
>Priority: Major
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> Reported on user@
> {quote}We are trying to setup a pipeline with using BeamSql and the trigger
> used is default (AfterWatermark crosses the window).
> Below is the pipeline:
>
>KafkaSource (KafkaIO)
>---> Windowing (FixedWindow 1min)
>---> BeamSql
>---> KafkaSink (KafkaIO)
>
> We are using Spark Runner for this.
> The BeamSql query is:
> {code}select Col3, count(*) as count_col1 from PCOLLECTION GROUP BY Col3{code}
> We are grouping by Col3 which is a string. It can hold values string[0-9].
>
> The records are getting emitted out at 1 min to kafka sink, but the output
> record in kafka is not as expected.
> Below is the output observed: (WST and WET are indicators for window start
> time and window end time)
> {code}
> {"count_col1":1,"Col3":"string5","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":3,"Col3":"string7","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":2,"Col3":"string8","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":1,"Col3":"string2","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":1,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 0}
> {code}
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8439) Flink fails ParDoLifecycleTests
[ https://issues.apache.org/jira/browse/BEAM-8439?focusedWorklogId=331273&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331273 ] ASF GitHub Bot logged work on BEAM-8439: Author: ASF GitHub Bot Created on: 21/Oct/19 08:29 Start Date: 21/Oct/19 08:29 Worklog Time Spent: 10m Work Description: je-ik commented on issue #9841: [BEAM-8439] enable empty bundles in ParDoLifecycleTest URL: https://github.com/apache/beam/pull/9841#issuecomment-544407395 Run Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331273) Time Spent: 20m (was: 10m) > Flink fails ParDoLifecycleTests > --- > > Key: BEAM-8439 > URL: https://issues.apache.org/jira/browse/BEAM-8439 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > ParDoLifecycleTest.testTeardownCalledAfterExceptionInFinishBundleStateful > fails consistently > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8439) Flink fails ParDoLifecycleTests
[ https://issues.apache.org/jira/browse/BEAM-8439?focusedWorklogId=331272&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331272 ] ASF GitHub Bot logged work on BEAM-8439: Author: ASF GitHub Bot Created on: 21/Oct/19 08:29 Start Date: 21/Oct/19 08:29 Worklog Time Spent: 10m Work Description: je-ik commented on pull request #9841: [BEAM-8439] enable empty bundles in ParDoLifecycleTest URL: https://github.com/apache/beam/pull/9841 Flink produces empty bundles (on some occasions), so it should be valid to see empty bundles in ParDoLifecycleTest. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[ Flink fails ParDoLifecycleTests
[ https://issues.apache.org/jira/browse/BEAM-8439?focusedWorklogId=331279&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331279 ] ASF GitHub Bot logged work on BEAM-8439: Author: ASF GitHub Bot Created on: 21/Oct/19 08:32 Start Date: 21/Oct/19 08:32 Worklog Time Spent: 10m Work Description: je-ik commented on issue #9841: [BEAM-8439] enable empty bundles in ParDoLifecycleTest URL: https://github.com/apache/beam/pull/9841#issuecomment-544408680 Run Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331279) Time Spent: 50m (was: 40m) > Flink fails ParDoLifecycleTests > --- > > Key: BEAM-8439 > URL: https://issues.apache.org/jira/browse/BEAM-8439 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > ParDoLifecycleTest.testTeardownCalledAfterExceptionInFinishBundleStateful > fails consistently > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8439) Flink fails ParDoLifecycleTests
[ https://issues.apache.org/jira/browse/BEAM-8439?focusedWorklogId=331275&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331275 ] ASF GitHub Bot logged work on BEAM-8439: Author: ASF GitHub Bot Created on: 21/Oct/19 08:32 Start Date: 21/Oct/19 08:32 Worklog Time Spent: 10m Work Description: je-ik commented on issue #9841: [BEAM-8439] enable empty bundles in ParDoLifecycleTest URL: https://github.com/apache/beam/pull/9841#issuecomment-544408386 R: @mxm This could be the proper fix. But please validate my logic: - because `checkInvokeStartBundle` is called in `DoFnOperator.processWatermark1`, it is possible that `startBundle` gets invoked even though there are no actual elements flowing into the operator (e.g. higher parallelism than the actual number of elements). Therefore, the sequence of calls of the lifecycle methods is `startBundle` -> `finishBundle` with no intermediate `processElement`. It should be therefore captured in the test directly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331275) Time Spent: 0.5h (was: 20m) > Flink fails ParDoLifecycleTests > --- > > Key: BEAM-8439 > URL: https://issues.apache.org/jira/browse/BEAM-8439 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > ParDoLifecycleTest.testTeardownCalledAfterExceptionInFinishBundleStateful > fails consistently > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8439) Flink fails ParDoLifecycleTests
[ https://issues.apache.org/jira/browse/BEAM-8439?focusedWorklogId=331278&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331278 ] ASF GitHub Bot logged work on BEAM-8439: Author: ASF GitHub Bot Created on: 21/Oct/19 08:32 Start Date: 21/Oct/19 08:32 Worklog Time Spent: 10m Work Description: je-ik commented on issue #9841: [BEAM-8439] enable empty bundles in ParDoLifecycleTest URL: https://github.com/apache/beam/pull/9841#issuecomment-544408612 Run Spark ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331278) Time Spent: 40m (was: 0.5h) > Flink fails ParDoLifecycleTests > --- > > Key: BEAM-8439 > URL: https://issues.apache.org/jira/browse/BEAM-8439 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > ParDoLifecycleTest.testTeardownCalledAfterExceptionInFinishBundleStateful > fails consistently > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8440) Temporarily disable ParDoLifecycleTest for flink
[ https://issues.apache.org/jira/browse/BEAM-8440?focusedWorklogId=331281&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331281 ] ASF GitHub Bot logged work on BEAM-8440: Author: ASF GitHub Bot Created on: 21/Oct/19 08:33 Start Date: 21/Oct/19 08:33 Worklog Time Spent: 10m Work Description: je-ik commented on pull request #9839: [BEAM-8440] temporarily disable ParDoLifecycleTest for flink runner URL: https://github.com/apache/beam/pull/9839 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331281) Time Spent: 1h 10m (was: 1h) > Temporarily disable ParDoLifecycleTest for flink > > > Key: BEAM-8440 > URL: https://issues.apache.org/jira/browse/BEAM-8440 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > Disable ParDoLifecycleTest validates runner category, until BEAM-8439 is > resolved. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8440) Temporarily disable ParDoLifecycleTest for flink
[ https://issues.apache.org/jira/browse/BEAM-8440?focusedWorklogId=331280&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331280 ] ASF GitHub Bot logged work on BEAM-8440: Author: ASF GitHub Bot Created on: 21/Oct/19 08:33 Start Date: 21/Oct/19 08:33 Worklog Time Spent: 10m Work Description: je-ik commented on issue #9839: [BEAM-8440] temporarily disable ParDoLifecycleTest for flink runner URL: https://github.com/apache/beam/pull/9839#issuecomment-544409137 Closing this, because it is probably not necessary to disable the test. We will fix that directly in [BEAM-8439]. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331280) Time Spent: 1h (was: 50m) > Temporarily disable ParDoLifecycleTest for flink > > > Key: BEAM-8440 > URL: https://issues.apache.org/jira/browse/BEAM-8440 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Disable ParDoLifecycleTest validates runner category, until BEAM-8439 is > resolved. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (BEAM-8440) Temporarily disable ParDoLifecycleTest for flink
[ https://issues.apache.org/jira/browse/BEAM-8440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Lukavský closed BEAM-8440. -- > Temporarily disable ParDoLifecycleTest for flink > > > Key: BEAM-8440 > URL: https://issues.apache.org/jira/browse/BEAM-8440 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Fix For: Not applicable > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Disable ParDoLifecycleTest validates runner category, until BEAM-8439 is > resolved. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8440) Temporarily disable ParDoLifecycleTest for flink
[ https://issues.apache.org/jira/browse/BEAM-8440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Lukavský resolved BEAM-8440. Fix Version/s: Not applicable Resolution: Won't Fix Closing, will fix directly by BEAM-8439. > Temporarily disable ParDoLifecycleTest for flink > > > Key: BEAM-8440 > URL: https://issues.apache.org/jira/browse/BEAM-8440 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Fix For: Not applicable > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Disable ParDoLifecycleTest validates runner category, until BEAM-8439 is > resolved. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8442) Unfiy bundle register in Python SDK harness
sunjincheng created BEAM-8442: - Summary: Unfiy bundle register in Python SDK harness Key: BEAM-8442 URL: https://issues.apache.org/jira/browse/BEAM-8442 Project: Beam Issue Type: Improvement Components: sdk-py-harness Reporter: sunjincheng Assignee: sunjincheng Fix For: 2.17.0 There are two methods for bundle register in Python SDK harness: `SdkHarness._request_register` and `SdkWorker.register.` It should be unfied. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8442) Unfiy bundle register in Python SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8442?focusedWorklogId=331324&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331324 ] ASF GitHub Bot logged work on BEAM-8442: Author: ASF GitHub Bot Created on: 21/Oct/19 10:02 Start Date: 21/Oct/19 10:02 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on pull request #9842: [BEAM-8442] Remove duplicate code for bundle register in Python SDK harness URL: https://github.com/apache/beam/pull/9842 There are two methods for bundle register in Python SDK harness: `SdkHarness._request_register` and `SdkWorker.register.` It should be unfied. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [ SdkWorker should be stopped in SdkHarness
[ https://issues.apache.org/jira/browse/BEAM-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955935#comment-16955935 ] sunjincheng commented on BEAM-8325: --- There are two approaches to solve this issue. We will describe them [here|https://docs.google.com/document/d/1sCgy9VQPf9zVXKRquK8P6N4x7aB62GEO8ozkujRSHZg/edit?usp=sharing]. > SdkWorker should be stopped in SdkHarness > - > > Key: BEAM-8325 > URL: https://issues.apache.org/jira/browse/BEAM-8325 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.17.0 > > > I found that in SdkHarness do not stop the SdkWorker when finish. We should > add the logic for sopt the SdkWorker in SdkHarness. See `sdk_worker.py`. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8410) JdbcIO should support setConnectionInitSqls in its DataSource
[ https://issues.apache.org/jira/browse/BEAM-8410?focusedWorklogId=331326&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331326 ] ASF GitHub Bot logged work on BEAM-8410: Author: ASF GitHub Bot Created on: 21/Oct/19 10:07 Start Date: 21/Oct/19 10:07 Worklog Time Spent: 10m Work Description: aromanenko-dev commented on issue #9808: [BEAM-8410] JdbcIO should support setConnectionInitSqls in its DataSource URL: https://github.com/apache/beam/pull/9808#issuecomment-55751 LGTM, thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331326) Time Spent: 1.5h (was: 1h 20m) > JdbcIO should support setConnectionInitSqls in its DataSource > - > > Key: BEAM-8410 > URL: https://issues.apache.org/jira/browse/BEAM-8410 > Project: Beam > Issue Type: Improvement > Components: io-java-jdbc >Reporter: Cam Mach >Assignee: Cam Mach >Priority: Minor > Time Spent: 1.5h > Remaining Estimate: 0h > > This property, connectionInitSqls, is very handy for anyone who use MySql and > Mariadb, to set any init sql statements to be executed at connection time. > Note: but it's not applicable across databases -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8410) JdbcIO should support setConnectionInitSqls in its DataSource
[ https://issues.apache.org/jira/browse/BEAM-8410?focusedWorklogId=331327&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331327 ] ASF GitHub Bot logged work on BEAM-8410: Author: ASF GitHub Bot Created on: 21/Oct/19 10:08 Start Date: 21/Oct/19 10:08 Worklog Time Spent: 10m Work Description: aromanenko-dev commented on pull request #9808: [BEAM-8410] JdbcIO should support setConnectionInitSqls in its DataSource URL: https://github.com/apache/beam/pull/9808 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331327) Time Spent: 1h 40m (was: 1.5h) > JdbcIO should support setConnectionInitSqls in its DataSource > - > > Key: BEAM-8410 > URL: https://issues.apache.org/jira/browse/BEAM-8410 > Project: Beam > Issue Type: Improvement > Components: io-java-jdbc >Reporter: Cam Mach >Assignee: Cam Mach >Priority: Minor > Time Spent: 1h 40m > Remaining Estimate: 0h > > This property, connectionInitSqls, is very handy for anyone who use MySql and > Mariadb, to set any init sql statements to be executed at connection time. > Note: but it's not applicable across databases -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-3788) Implement a Kafka IO for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955943#comment-16955943 ] Chethan UK commented on BEAM-3788: -- [~chamikara] Any Docs? Wanted to use Kafka in Dataflow pipelines... Thanks!. > Implement a Kafka IO for Python SDK > --- > > Key: BEAM-3788 > URL: https://issues.apache.org/jira/browse/BEAM-3788 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Priority: Major > > This will be implemented using the Splittable DoFn framework. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-3788) Implement a Kafka IO for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955943#comment-16955943 ] Chethan UK edited comment on BEAM-3788 at 10/21/19 10:34 AM: - What's the update on Kafka IO? [~chamikara] Any Docs? Wanted to use Kafka in Dataflow pipelines... Thanks!. was (Author: chethanuk): [~chamikara] Any Docs? Wanted to use Kafka in Dataflow pipelines... Thanks!. > Implement a Kafka IO for Python SDK > --- > > Key: BEAM-3788 > URL: https://issues.apache.org/jira/browse/BEAM-3788 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Priority: Major > > This will be implemented using the Splittable DoFn framework. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8439) Flink fails ParDoLifecycleTests
[
https://issues.apache.org/jira/browse/BEAM-8439?focusedWorklogId=331361&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331361
]
ASF GitHub Bot logged work on BEAM-8439:
Author: ASF GitHub Bot
Created on: 21/Oct/19 11:10
Start Date: 21/Oct/19 11:10
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #9841: [BEAM-8439] enable
empty bundles in ParDoLifecycleTest
URL: https://github.com/apache/beam/pull/9841#discussion_r336951787
##
File path:
sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoLifecycleTest.java
##
@@ -379,7 +379,7 @@ public void postBundle() throws Exception {
assertThat(
"processing bundle should have been called before finish bundle",
Review comment:
Might have to adjust the error message here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331361)
Time Spent: 1h (was: 50m)
> Flink fails ParDoLifecycleTests
> ---
>
> Key: BEAM-8439
> URL: https://issues.apache.org/jira/browse/BEAM-8439
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Jan Lukavský
>Assignee: Jan Lukavský
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> ParDoLifecycleTest.testTeardownCalledAfterExceptionInFinishBundleStateful
> fails consistently
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8439) Flink fails ParDoLifecycleTests
[
https://issues.apache.org/jira/browse/BEAM-8439?focusedWorklogId=331362&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331362
]
ASF GitHub Bot logged work on BEAM-8439:
Author: ASF GitHub Bot
Created on: 21/Oct/19 11:10
Start Date: 21/Oct/19 11:10
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #9841: [BEAM-8439] enable
empty bundles in ParDoLifecycleTest
URL: https://github.com/apache/beam/pull/9841#discussion_r336951898
##
File path:
sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoLifecycleTest.java
##
@@ -379,7 +379,7 @@ public void postBundle() throws Exception {
assertThat(
"processing bundle should have been called before finish bundle",
Review comment:
```suggestion
"processing bundle or start bundle should have been called before
finish bundle",
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331362)
Time Spent: 1h (was: 50m)
> Flink fails ParDoLifecycleTests
> ---
>
> Key: BEAM-8439
> URL: https://issues.apache.org/jira/browse/BEAM-8439
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Jan Lukavský
>Assignee: Jan Lukavský
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> ParDoLifecycleTest.testTeardownCalledAfterExceptionInFinishBundleStateful
> fails consistently
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-8274) Remove dependency on experiments=beam_fn_api flag from PortableRunner
[
https://issues.apache.org/jira/browse/BEAM-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maximilian Michels updated BEAM-8274:
-
Fix Version/s: (was: 2.17.0)
> Remove dependency on experiments=beam_fn_api flag from PortableRunner
> -
>
> Key: BEAM-8274
> URL: https://issues.apache.org/jira/browse/BEAM-8274
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Maximilian Michels
>Priority: Major
>
> The execution path of Python's {{PortableRunner}} depends on the
> {{experiments=beam_fn_api}} flag. This flag is set automatically but has been
> broken in the past, e.g. when the {{FlinkRunner}} was subclassed from the
> {{PortableRunner}} (BEAM-8037).
> We should get rid of the flag, as the only supported execution path for the
> PortableRunner is the BeamFnApi. Perhaps we can move the flag to the Dataflow
> runners which may still depend on it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8439) Flink fails ParDoLifecycleTests
[ https://issues.apache.org/jira/browse/BEAM-8439?focusedWorklogId=331365&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331365 ] ASF GitHub Bot logged work on BEAM-8439: Author: ASF GitHub Bot Created on: 21/Oct/19 11:37 Start Date: 21/Oct/19 11:37 Worklog Time Spent: 10m Work Description: je-ik commented on issue #9841: [BEAM-8439] enable empty bundles in ParDoLifecycleTest URL: https://github.com/apache/beam/pull/9841#issuecomment-544475075 @mxm I'm not sure if I follow - how would this change avoid creating empty bundles at the end of input, when watermark might move to the "infinity"? Couldn't this end up in the same situation? I'm not sure if a bundle creation time should be a concern here, because if an "empty bundle" is created, then it pretty much signals, that the pipeline is somewhat "empty" - e.g. it has too high parallelism, or no data is not flowing in. In either case, the performance of creating empty bundle should be negligible (because these will not be created too often). My biggest concern here would be, if modifying the test is actually aligned with Beam's model. I'll rise a question on mailing list and then we can decide what to do. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331365) Time Spent: 1h 10m (was: 1h) > Flink fails ParDoLifecycleTests > --- > > Key: BEAM-8439 > URL: https://issues.apache.org/jira/browse/BEAM-8439 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > ParDoLifecycleTest.testTeardownCalledAfterExceptionInFinishBundleStateful > fails consistently > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8372) Allow submission of Flink UberJar directly to flink cluster.
[ https://issues.apache.org/jira/browse/BEAM-8372?focusedWorklogId=331369&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331369 ] ASF GitHub Bot logged work on BEAM-8372: Author: ASF GitHub Bot Created on: 21/Oct/19 11:40 Start Date: 21/Oct/19 11:40 Worklog Time Spent: 10m Work Description: mxm commented on pull request #9803: [BEAM-8372] Follow-up to Flink UberJar submission. URL: https://github.com/apache/beam/pull/9803#discussion_r336969270 ## File path: sdks/python/apache_beam/runners/portability/flink_runner.py ## @@ -32,18 +32,18 @@ class FlinkRunner(portable_runner.PortableRunner): def default_job_server(self, options): -flink_master_url = options.view_as(FlinkRunnerOptions).flink_master_url -if flink_master_url == '[local]' or sys.version_info < (3, 6): +flink_master = options.view_as(FlinkRunnerOptions).flink_master Review comment: https://github.com/apache/beam/pull/9844 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331369) Time Spent: 5.5h (was: 5h 20m) > Allow submission of Flink UberJar directly to flink cluster. > > > Key: BEAM-8372 > URL: https://issues.apache.org/jira/browse/BEAM-8372 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 5.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8372) Allow submission of Flink UberJar directly to flink cluster.
[ https://issues.apache.org/jira/browse/BEAM-8372?focusedWorklogId=331368&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331368 ] ASF GitHub Bot logged work on BEAM-8372: Author: ASF GitHub Bot Created on: 21/Oct/19 11:40 Start Date: 21/Oct/19 11:40 Worklog Time Spent: 10m Work Description: mxm commented on pull request #9844: [BEAM-8372] Support both flink_master and flink_master_url parameter URL: https://github.com/apache/beam/pull/9844 The original parameter was named flink_master (defined in FlinkPipelineOptions) but since there have been inconsistencies. So it's best to support both now. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache
[jira] [Work logged] (BEAM-8372) Allow submission of Flink UberJar directly to flink cluster.
[
https://issues.apache.org/jira/browse/BEAM-8372?focusedWorklogId=331370&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331370
]
ASF GitHub Bot logged work on BEAM-8372:
Author: ASF GitHub Bot
Created on: 21/Oct/19 11:46
Start Date: 21/Oct/19 11:46
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #9844: [BEAM-8372]
Support both flink_master and flink_master_url parameter
URL: https://github.com/apache/beam/pull/9844#discussion_r336971171
##
File path: sdks/python/apache_beam/runners/portability/flink_runner.py
##
@@ -43,7 +43,8 @@ def default_job_server(self, options):
class FlinkRunnerOptions(pipeline_options.PipelineOptions):
@classmethod
def _add_argparse_args(cls, parser):
-parser.add_argument('--flink_master', default='[local]')
+parser.add_argument('--flink_master', '--flink_master_url',
+default='[local]')
Review comment:
The default for Java is `[auto]`. Should we change the Python SDK to match?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331370)
Time Spent: 5h 40m (was: 5.5h)
> Allow submission of Flink UberJar directly to flink cluster.
>
>
> Key: BEAM-8372
> URL: https://issues.apache.org/jira/browse/BEAM-8372
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8409) docker-credential-gcloud not installed or not available in PATH
[
https://issues.apache.org/jira/browse/BEAM-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955999#comment-16955999
]
Kamil Wasilewski commented on BEAM-8409:
[~yifanzou] Is there anything I can help you with?
> docker-credential-gcloud not installed or not available in PATH
> ---
>
> Key: BEAM-8409
> URL: https://issues.apache.org/jira/browse/BEAM-8409
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Kamil Wasilewski
>Assignee: Yifan Zou
>Priority: Major
> Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
> *
> [beam_PreCommit_CommunityMetrics_Commit|https://builds.apache.org/view/A-D/view/Beam/view/All/job/beam_PreCommit_CommunityMetrics_Commit/1355/]
> *
> [beam_PostCommit_Python2_PR|https://builds.apache.org/job/beam_PostCommit_Python2_PR]
> Initial investigation:
> Jenkins job fails when executing docker-compose script.
> It seems the only Jenkins worker affected is *apache-beam-jenkins-15.*
>
> Relevant logs:
> 1)
>
> {code:java}
> 11:56:24 Execution failed for task ':beam-test-infra-metrics:composeUp'.
> 11:56:24 > Exit-code 255 when calling docker-compose, stdout: postgresql uses
> an image, skipping
> 11:56:24 prometheus uses an image, skipping
> 11:56:24 pushgateway uses an image, skipping
> 11:56:24 alertmanager uses an image, skipping
> 11:56:24 Building grafana
> 11:56:24 [17038] Failed to execute script docker-compose
> 11:56:24 Traceback (most recent call last):
> 11:56:24 File "bin/docker-compose", line 6, in
> 11:56:24 File "compose/cli/main.py", line 71, in main
> 11:56:24 File "compose/cli/main.py", line 127, in perform_command
> 11:56:24 File "compose/cli/main.py", line 287, in build
> 11:56:24 File "compose/project.py", line 386, in build
> 11:56:24 File "compose/project.py", line 368, in build_service
> 11:56:24 File "compose/service.py", line 1084, in build
> 11:56:24 File "site-packages/docker/api/build.py", line 260, in build
> 11:56:24 File "site-packages/docker/api/build.py", line 307, in
> _set_auth_headers
> 11:56:24 File "site-packages/docker/auth.py", line 310, in
> get_all_credentials
> 11:56:24 File "site-packages/docker/auth.py", line 262, in
> _resolve_authconfig_credstore
> 11:56:24 File "site-packages/docker/auth.py", line 287, in
> _get_store_instance
> 11:56:24 File "site-packages/dockerpycreds/store.py", line 25, in __init__
> 11:56:24 dockerpycreds.errors.InitializationError: docker-credential-gcloud
> not installed or not available in PATH
> {code}
> 2)
> {code:java}
> 16:26:08 [9316] Failed to execute script docker-compose
> 16:26:08 Traceback (most recent call last):
> 16:26:08 File "bin/docker-compose", line 6, in
> 16:26:08 File "compose/cli/main.py", line 71, in main
> 16:26:08 File "compose/cli/main.py", line 127, in perform_command
> 16:26:08 File "compose/cli/main.py", line 287, in build
> 16:26:08 File "compose/project.py", line 386, in build
> 16:26:08 File "compose/project.py", line 368, in build_service
> 16:26:08 File "compose/service.py", line 1084, in build
> 16:26:08 File "site-packages/docker/api/build.py", line 260, in build
> 16:26:08 File "site-packages/docker/api/build.py", line 307, in
> _set_auth_headers
> 16:26:08 File "site-packages/docker/auth.py", line 310, in
> get_all_credentials
> 16:26:08 File "site-packages/docker/auth.py", line 262, in
> _resolve_authconfig_credstore
> 16:26:08 File "site-packages/docker/auth.py", line 287, in
> _get_store_instance
> 16:26:08 File "site-packages/dockerpycreds/store.py", line 25, in __init__
> 16:26:08 dockerpycreds.errors.InitializationError: docker-credential-gcloud
> not installed or not available in PATH
> {code}
> **
>
>
> _After you've filled out the above details, pl__ease [assign the issue to an
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
> Assignee should [treat test failures as
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
> helping to fix the issue or find a more appropriate owner. See [Apache Beam
> Post-Commit
> Policies|https://beam.apache.org/contribute/postcommits-policies]._
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-7792) Beam Dependency Update Request: com.github.spotbugs:spotbugs
[ https://issues.apache.org/jira/browse/BEAM-7792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-7792: Description: - 2019-07-22 12:15:41.216793 - Please consider upgrading the dependency com.github.spotbugs:spotbugs. The current version is 3.1.10. The latest version is 4.0.0-beta3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-07-29 12:06:22.055418 - Please consider upgrading the dependency com.github.spotbugs:spotbugs. The current version is 3.1.10. The latest version is 4.0.0-beta3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-05 12:03:56.940731 - Please consider upgrading the dependency com.github.spotbugs:spotbugs. The current version is 3.1.10. The latest version is 4.0.0-beta3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-12 12:05:04.611973 - Please consider upgrading the dependency com.github.spotbugs:spotbugs. The current version is 3.1.10. The latest version is 4.0.0-beta3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-19 12:03:13.035336 - Please consider upgrading the dependency com.github.spotbugs:spotbugs. The current version is 3.1.12. The latest version is 4.0.0-beta3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-26 12:02:56.981076 - Please consider upgrading the dependency com.github.spotbugs:spotbugs. The current version is 3.1.12. The latest version is 4.0.0-beta3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-02 12:03:52.780365 - Please consider upgrading the dependency com.github.spotbugs:spotbugs. The current version is 3.1.12. The latest version is 4.0.0-beta3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-09 12:02:54.192199 - Please consider upgrading the dependency com.github.spotbugs:spotbugs. The current version is 3.1.12. The latest version is 4.0.0-beta3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-16 12:02:39.812475 - Please consider upgrading the dependency com.github.spotbugs:spotbugs. The current version is 3.1.12. The latest version is 4.0.0-beta3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-23 12:03:12.396932 - Please consider upgrading the dependency com.github.spotbugs:spotbugs. The current version is 3.1.12. The latest version is 4.0.0-beta4 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-30 12:04:13.122038 - Please consider upgrading the dependency com.github.spotbugs:spotbugs. The current version is 3.1.12. The latest version is 4.0.0-beta4 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-10-07 12:03:31.042633 - Please consider upgrading the dependency com.github.spotbugs:spotbugs. The current version is 3.1.12. The latest version is 4.0.0-beta4 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]
[jira] [Updated] (BEAM-6089) Beam Dependency Update Request: oauth2client
[ https://issues.apache.org/jira/browse/BEAM-6089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-6089: Description: - 2018-11-19 12:11:53.801885 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-26 12:10:31.359164 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-03 12:11:18.194090 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-10 12:13:40.021791 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-17 12:12:09.88 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-31 15:20:14.935936 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-01-07 12:23:14.664558 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-01-14 12:12:14.081917 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-01-21 12:18:38.928775 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-01-28 12:10:22.371989 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-02-04 12:11:00.277439 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-02-11 12:11:25.422782 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-02-18 12:21:52.210498 - Please consider upgrading the dependency oauth2client. The current version is 3.0.0. The latest version is 4.1.3
[jira] [Created] (BEAM-8443) Beam Dependency Update Request: google-cloud-datastore
Beam JIRA Bot created BEAM-8443: --- Summary: Beam Dependency Update Request: google-cloud-datastore Key: BEAM-8443 URL: https://issues.apache.org/jira/browse/BEAM-8443 Project: Beam Issue Type: Bug Components: dependencies Reporter: Beam JIRA Bot - 2019-10-21 12:02:33.133724 - Please consider upgrading the dependency google-cloud-datastore. The current version is 1.7.4. The latest version is 1.10.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7369) Beam Dependency Update Request: mock
[ https://issues.apache.org/jira/browse/BEAM-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-7369: Description: - 2019-05-20 16:37:42.331317 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-06-17 12:32:01.319061 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-06-24 12:02:34.500201 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-07-01 12:03:46.078139 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-07-08 12:02:33.351570 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-07-15 12:02:47.637435 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-07-22 12:03:10.835914 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-07-29 12:04:50.359555 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-05 12:02:38.989136 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-12 12:03:38.282286 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-19 12:02:30.405376 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-26 12:02:24.272597 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-02 12:03:07.834005 - Please consider upgrading the dependency mock. The current version is 2.0.0. The latest version is 3.0.5 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/
[jira] [Updated] (BEAM-6645) Beam Dependency Update Request: com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin
[ https://issues.apache.org/jira/browse/BEAM-6645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-6645: Description: - 2019-02-11 12:12:17.710870 - Please consider upgrading the dependency com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin. The current version is 0.17.0. The latest version is 0.20.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-02-18 12:22:38.110512 - Please consider upgrading the dependency com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin. The current version is 0.17.0. The latest version is 0.20.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-02-25 12:12:13.220144 - Please consider upgrading the dependency com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin. The current version is 0.17.0. The latest version is 0.20.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-03-04 12:13:34.975335 - Please consider upgrading the dependency com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin. The current version is 0.17.0. The latest version is 0.21.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-03-11 12:13:20.550078 - Please consider upgrading the dependency com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin. The current version is 0.17.0. The latest version is 0.21.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-03-25 04:18:13.860760 - Please consider upgrading the dependency com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin. The current version is 0.17.0. The latest version is 0.21.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-04-01 12:09:02.686241 - Please consider upgrading the dependency com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin. The current version is 0.17.0. The latest version is 0.21.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-04-08 12:11:15.303458 - Please consider upgrading the dependency com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin. The current version is 0.17.0. The latest version is 0.21.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-04-15 12:30:47.279374 - Please consider upgrading the dependency com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin. The current version is 0.17.0. The latest version is 0.21.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-04-22 12:08:36.647085 - Please consider upgrading the dependency com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin. The current version is 0.17.0. The latest version is 0.21.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-05-20 16:38:23.480410 - Please consider upgrading the dependency com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin. The current version is 0.17.0. The latest version is 0.21.0 cc: Please refer to [Beam Dependency Guide |ht
[jira] [Updated] (BEAM-7370) Beam Dependency Update Request: Sphinx
[ https://issues.apache.org/jira/browse/BEAM-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-7370: Description: - 2019-05-20 16:38:07.937770 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-06-17 12:32:27.855338 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.1.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-06-24 12:02:59.052884 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.1.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-07-01 12:04:13.113613 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.1.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-07-08 12:03:15.091005 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.1.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-07-15 12:03:09.406918 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.1.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-07-22 12:03:31.157859 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.1.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-07-29 12:05:13.023604 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.1.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-05 12:03:03.242767 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.1.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-12 12:04:01.647619 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.1.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-19 12:02:52.342008 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.2.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-26 12:02:44.918642 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.2.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-02 12:03:28.319578 - Please consider upgrading the dependency Sphinx. The current version is 1.8.5. The latest version is 2.2.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.or
[jira] [Updated] (BEAM-5753) Beam Dependency Update Request: org.eclipse.jetty:jetty-servlet
[ https://issues.apache.org/jira/browse/BEAM-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-5753: Description: - 2018-10-15 12:13:19.256622 - Please consider upgrading the dependency org.eclipse.jetty:jetty-servlet. The current version is 9.2.10.v20150310. The latest version is 9.4.12.RC2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-10-22 12:13:32.240729 - Please consider upgrading the dependency org.eclipse.jetty:jetty-servlet. The current version is 9.2.10.v20150310. The latest version is 9.4.12.RC2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-10-29 12:17:15.388249 - Please consider upgrading the dependency org.eclipse.jetty:jetty-servlet. The current version is 9.2.10.v20150310. The latest version is 9.4.12.RC2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-05 12:15:02.810121 - Please consider upgrading the dependency org.eclipse.jetty:jetty-servlet. The current version is 9.2.10.v20150310. The latest version is 9.4.12.RC2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-12 12:15:02.741564 - Please consider upgrading the dependency org.eclipse.jetty:jetty-servlet. The current version is 9.2.10.v20150310. The latest version is 9.4.12.RC2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-19 12:15:44.462009 - Please consider upgrading the dependency org.eclipse.jetty:jetty-servlet. The current version is 9.2.10.v20150310. The latest version is 9.4.14.v20181114 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-26 12:14:43.766710 - Please consider upgrading the dependency org.eclipse.jetty:jetty-servlet. The current version is 9.2.10.v20150310. The latest version is 9.4.14.v20181114 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-03 12:15:09.151091 - Please consider upgrading the dependency org.eclipse.jetty:jetty-servlet. The current version is 9.2.10.v20150310. The latest version is 9.4.14.v20181114 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-10 12:17:38.451204 - Please consider upgrading the dependency org.eclipse.jetty:jetty-servlet. The current version is 9.2.10.v20150310. The latest version is 9.4.14.v20181114 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-17 12:17:59.925429 - Please consider upgrading the dependency org.eclipse.jetty:jetty-servlet. The current version is 9.2.10.v20150310. The latest version is 9.4.14.v20181114 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-31 15:24:18.611955 - Please consider upgrading the dependency org.eclipse.jetty:jetty-servlet. The current version is 9.2.10.v20150310. The latest version is 9.4.14.v20181114 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-01-07 12:27:30.751791 - Please consider upgrading the dependency org.eclipse.jetty:je
[jira] [Updated] (BEAM-6951) Beam Dependency Update Request: com.github.spotbugs:spotbugs-annotations
[ https://issues.apache.org/jira/browse/BEAM-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-6951: Description: - 2019-04-01 12:15:05.460427 - Please consider upgrading the dependency com.github.spotbugs:spotbugs-annotations. The current version is 3.1.11. The latest version is 4.0.0-beta1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-04-08 12:15:37.305259 - Please consider upgrading the dependency com.github.spotbugs:spotbugs-annotations. The current version is 3.1.11. The latest version is 4.0.0-beta1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-04-15 12:35:52.817108 - Please consider upgrading the dependency com.github.spotbugs:spotbugs-annotations. The current version is 3.1.11. The latest version is 4.0.0-beta1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-04-22 12:13:25.261372 - Please consider upgrading the dependency com.github.spotbugs:spotbugs-annotations. The current version is 3.1.11. The latest version is 4.0.0-beta1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-05-20 16:39:18.034675 - Please consider upgrading the dependency com.github.spotbugs:spotbugs-annotations. The current version is 3.1.11. The latest version is 4.0.0-beta1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-05-20 16:54:09.180503 - Please consider upgrading the dependency com.github.spotbugs:spotbugs-annotations. The current version is 3.1.11. The latest version is 4.0.0-beta1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-05-20 17:37:40.326607 - Please consider upgrading the dependency com.github.spotbugs:spotbugs-annotations. The current version is 3.1.11. The latest version is 4.0.0-beta1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-05-27 12:06:48.236836 - Please consider upgrading the dependency com.github.spotbugs:spotbugs-annotations. The current version is 3.1.11. The latest version is 4.0.0-beta2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-06-03 12:04:13.166255 - Please consider upgrading the dependency com.github.spotbugs:spotbugs-annotations. The current version is 3.1.11. The latest version is 4.0.0-beta2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-06-10 12:04:08.241217 - Please consider upgrading the dependency com.github.spotbugs:spotbugs-annotations. The current version is 3.1.11. The latest version is 4.0.0-beta2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-06-17 12:33:32.545061 - Please consider upgrading the dependency com.github.spotbugs:spotbugs-annotations. The current version is 3.1.11. The latest version is 4.0.0-beta2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-06-24 12:04:02.498856 - Please consider upgrading the dependency com.github.spotbugs:spotbugs-annotations. The current ver
[jira] [Updated] (BEAM-5750) Beam Dependency Update Request: javax.servlet:javax.servlet-api
[ https://issues.apache.org/jira/browse/BEAM-5750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-5750: Description: - 2018-10-15 12:13:14.253682 - Please consider upgrading the dependency javax.servlet:javax.servlet-api. The current version is 3.1.0. The latest version is 4.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-10-22 12:13:20.677351 - Please consider upgrading the dependency javax.servlet:javax.servlet-api. The current version is 3.1.0. The latest version is 4.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-10-29 12:17:03.954722 - Please consider upgrading the dependency javax.servlet:javax.servlet-api. The current version is 3.1.0. The latest version is 4.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-05 12:14:50.209888 - Please consider upgrading the dependency javax.servlet:javax.servlet-api. The current version is 3.1.0. The latest version is 4.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-12 12:14:50.401239 - Please consider upgrading the dependency javax.servlet:javax.servlet-api. The current version is 3.1.0. The latest version is 4.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-19 12:15:31.652290 - Please consider upgrading the dependency javax.servlet:javax.servlet-api. The current version is 3.1.0. The latest version is 4.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-26 12:14:29.396363 - Please consider upgrading the dependency javax.servlet:javax.servlet-api. The current version is 3.1.0. The latest version is 4.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-03 12:14:52.265627 - Please consider upgrading the dependency javax.servlet:javax.servlet-api. The current version is 3.1.0. The latest version is 4.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-10 12:17:21.331309 - Please consider upgrading the dependency javax.servlet:javax.servlet-api. The current version is 3.1.0. The latest version is 4.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-17 12:17:42.365110 - Please consider upgrading the dependency javax.servlet:javax.servlet-api. The current version is 3.1.0. The latest version is 4.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-31 15:24:01.296226 - Please consider upgrading the dependency javax.servlet:javax.servlet-api. The current version is 3.1.0. The latest version is 4.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-01-07 12:27:13.634593 - Please consider upgrading the dependency javax.servlet:javax.servlet-api. The current version is 3.1.0. The latest version is 4.0.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify
[jira] [Updated] (BEAM-5752) Beam Dependency Update Request: org.eclipse.jetty:jetty-server
[ https://issues.apache.org/jira/browse/BEAM-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-5752: Description: - 2018-10-15 12:13:17.575546 - Please consider upgrading the dependency org.eclipse.jetty:jetty-server. The current version is 9.2.10.v20150310. The latest version is 9.4.12.RC2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-10-22 12:13:26.908573 - Please consider upgrading the dependency org.eclipse.jetty:jetty-server. The current version is 9.2.10.v20150310. The latest version is 9.4.12.RC2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-10-29 12:17:10.104039 - Please consider upgrading the dependency org.eclipse.jetty:jetty-server. The current version is 9.2.10.v20150310. The latest version is 9.4.12.RC2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-05 12:14:56.891103 - Please consider upgrading the dependency org.eclipse.jetty:jetty-server. The current version is 9.2.10.v20150310. The latest version is 9.4.12.RC2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-12 12:14:57.086704 - Please consider upgrading the dependency org.eclipse.jetty:jetty-server. The current version is 9.2.10.v20150310. The latest version is 9.4.12.RC2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-19 12:15:38.454968 - Please consider upgrading the dependency org.eclipse.jetty:jetty-server. The current version is 9.2.10.v20150310. The latest version is 9.4.14.v20181114 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-26 12:14:38.329092 - Please consider upgrading the dependency org.eclipse.jetty:jetty-server. The current version is 9.2.10.v20150310. The latest version is 9.4.14.v20181114 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-03 12:15:03.778648 - Please consider upgrading the dependency org.eclipse.jetty:jetty-server. The current version is 9.2.10.v20150310. The latest version is 9.4.14.v20181114 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-10 12:17:33.026807 - Please consider upgrading the dependency org.eclipse.jetty:jetty-server. The current version is 9.2.10.v20150310. The latest version is 9.4.14.v20181114 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-17 12:17:54.249911 - Please consider upgrading the dependency org.eclipse.jetty:jetty-server. The current version is 9.2.10.v20150310. The latest version is 9.4.14.v20181114 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-31 15:24:13.071360 - Please consider upgrading the dependency org.eclipse.jetty:jetty-server. The current version is 9.2.10.v20150310. The latest version is 9.4.14.v20181114 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-01-07 12:27:25.376755 - Please consider upgrading the dependency org.eclipse.jetty:jetty-server.
[jira] [Updated] (BEAM-5748) Beam Dependency Update Request: org.conscrypt:conscrypt-openjdk
[ https://issues.apache.org/jira/browse/BEAM-5748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-5748: Description: - 2018-10-15 12:11:29.496210 - Please consider upgrading the dependency org.conscrypt:conscrypt-openjdk. The current version is 1.1.3. The latest version is 1.4.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-10-22 12:11:12.117107 - Please consider upgrading the dependency org.conscrypt:conscrypt-openjdk. The current version is 1.1.3. The latest version is 1.4.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-10-29 12:13:29.155766 - Please consider upgrading the dependency org.conscrypt:conscrypt-openjdk. The current version is 1.1.3. The latest version is 1.4.0 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-05 12:12:12.225263 - Please consider upgrading the dependency org.conscrypt:conscrypt-openjdk. The current version is 1.1.3. The latest version is 1.4.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-12 12:12:03.483941 - Please consider upgrading the dependency org.conscrypt:conscrypt-openjdk. The current version is 1.1.3. The latest version is 1.4.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-19 12:12:49.618981 - Please consider upgrading the dependency org.conscrypt:conscrypt-openjdk. The current version is 1.1.3. The latest version is 1.4.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-11-26 12:11:54.540783 - Please consider upgrading the dependency org.conscrypt:conscrypt-openjdk. The current version is 1.1.3. The latest version is 1.4.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-03 12:12:15.486522 - Please consider upgrading the dependency org.conscrypt:conscrypt-openjdk. The current version is 1.1.3. The latest version is 1.4.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-10 12:14:35.731081 - Please consider upgrading the dependency org.conscrypt:conscrypt-openjdk. The current version is 1.1.3. The latest version is 1.4.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-17 12:15:07.608265 - Please consider upgrading the dependency org.conscrypt:conscrypt-openjdk. The current version is 1.1.3. The latest version is 1.4.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2018-12-31 15:21:22.979404 - Please consider upgrading the dependency org.conscrypt:conscrypt-openjdk. The current version is 1.1.3. The latest version is 1.4.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-01-07 12:24:30.102231 - Please consider upgrading the dependency org.conscrypt:conscrypt-openjdk. The current version is 1.1.3. The latest version is 1.4.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify
[jira] [Updated] (BEAM-8002) Beam Dependency Update Request: Gradle:
[ https://issues.apache.org/jira/browse/BEAM-8002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-8002: Description: - 2019-08-19 12:04:19.192688 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1. The latest version is 5.6 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-26 12:04:17.802062 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1. The latest version is 5.6 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-02 12:04:53.057461 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1. The latest version is 5.6.1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-09 12:03:53.298241 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1. The latest version is 5.6.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-16 12:03:47.037326 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1. The latest version is 5.6.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-23 12:04:12.360299 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1. The latest version is 5.6.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-30 12:05:19.680040 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1. The latest version is 5.6.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-10-07 12:04:48.065365 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1. The latest version is 5.6.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-10-14 12:04:37.533903 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1. The latest version is 5.6.2 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-10-21 12:05:15.993810 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1 -> 5.6.3. The latest version is 6.0-rc-1 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. was: - 2019-08-19 12:04:19.192688 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1. The latest version is 5.6 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-08-26 12:04:17.802062 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1. The latest version is 5.6 cc: Please refer to [Beam Dependency Guide |https://beam.apache.org/contribute/dependencies/]for more information. Do Not Modify The Description Above. - 2019-09-02 12:04:53.057461 - Please consider upgrading the dependency Gradle:. The current version is 5.2.1. The latest version is 5.6.1 cc: Please refer to [Beam Dependency Guid
[jira] [Work logged] (BEAM-8430) sdk_worker_parallelism default is inconsistent between Py and Java SDKs
[ https://issues.apache.org/jira/browse/BEAM-8430?focusedWorklogId=331379&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331379 ] ASF GitHub Bot logged work on BEAM-8430: Author: ASF GitHub Bot Created on: 21/Oct/19 12:09 Start Date: 21/Oct/19 12:09 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9829: [BEAM-8430] Change py default sdk_worker_parallelism to 1 URL: https://github.com/apache/beam/pull/9829#issuecomment-544485050 @tweise I added a comment. I agree that seems like a lot of extra work to manually maintain consistency between SDKs. It seems like the current options setup was designed to prioritize flexibility over structure. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331379) Time Spent: 40m (was: 0.5h) > sdk_worker_parallelism default is inconsistent between Py and Java SDKs > --- > > Key: BEAM-8430 > URL: https://issues.apache.org/jira/browse/BEAM-8430 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core, sdk-py-core >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Default is currently 1 in Java, 0 in Python. > https://github.com/apache/beam/blob/7b67a926b8939ede8f2e33c85579b540d18afccf/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L73 > https://github.com/apache/beam/blob/7b67a926b8939ede8f2e33c85579b540d18afccf/sdks/python/apache_beam/options/pipeline_options.py#L848 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-6763) Publish Portability artifacts
[ https://issues.apache.org/jira/browse/BEAM-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956024#comment-16956024 ] Kyle Weaver commented on BEAM-6763: --- Is this complete? > Publish Portability artifacts > - > > Key: BEAM-6763 > URL: https://issues.apache.org/jira/browse/BEAM-6763 > Project: Beam > Issue Type: Task > Components: runner-core, runner-flink >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > > Publish the following artifacts to make it easy for the users to use portable > runner. > # Flink Job Server binary > # Python SdkHarness image > # Java SdkHarness image -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8207) KafkaIOITs generate different hashes each run, sometimes dropping records
[ https://issues.apache.org/jira/browse/BEAM-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukasz Gajowy updated BEAM-8207: Status: Open (was: Triage Needed) > KafkaIOITs generate different hashes each run, sometimes dropping records > - > > Key: BEAM-8207 > URL: https://issues.apache.org/jira/browse/BEAM-8207 > Project: Beam > Issue Type: Bug > Components: io-java-kafka, testing >Reporter: Michal Walenia >Priority: Major > > While working to adapt Java's KafkaIOIT to work with a large dataset > generated by a SyntheticSource I encountered a problem. I want to push 100M > records through a Kafka topic, verify data correctness and at the same time > check the performance of KafkaIO.Write and KafkaIO.Read. > > To perform the tests I'm using a Kafka cluster on Kubernetes from the Beam > repo > ([here|https://github.com/apache/beam/tree/master/.test-infra/kubernetes/kafka-cluster]). > > The expected result would be that first the records are generated in a > deterministic way (using hashes of list positions as Random seeds), next they > are written to Kafka - this concludes the write pipeline. > As for reading and correctness checking - first, the data is read from the > topic and after being decoded into String representations, a hashcode of the > whole PCollection is calculated (For details, check KafkaIOIT.java). > > During the testing I ran into several problems: > 1. When all the records are read from the Kafka topic, the hash is different > each time. > 2. Sometimes not all the records are read and the Dataflow task waits for the > input indefinitely, occasionally throwing exceptions. > > I believe there are two possible causes of this behavior: > > either there is something wrong with the Kafka cluster configuration > or KafkaIO behaves erratically on high data volumes, duplicating and/or > dropping records. > Second option seems troubling and I would be grateful for help with the first. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8385) Add an option to run KafkaIOIT with 10GB dataset
[ https://issues.apache.org/jira/browse/BEAM-8385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michal Walenia resolved BEAM-8385. -- Fix Version/s: 2.17.0 Resolution: Fixed > Add an option to run KafkaIOIT with 10GB dataset > > > Key: BEAM-8385 > URL: https://issues.apache.org/jira/browse/BEAM-8385 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Michal Walenia >Assignee: Michal Walenia >Priority: Minor > Fix For: 2.17.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8372) Allow submission of Flink UberJar directly to flink cluster.
[
https://issues.apache.org/jira/browse/BEAM-8372?focusedWorklogId=331414&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331414
]
ASF GitHub Bot logged work on BEAM-8372:
Author: ASF GitHub Bot
Created on: 21/Oct/19 13:30
Start Date: 21/Oct/19 13:30
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #9844: [BEAM-8372]
Support both flink_master and flink_master_url parameter
URL: https://github.com/apache/beam/pull/9844#discussion_r337017786
##
File path: sdks/python/apache_beam/runners/portability/flink_runner.py
##
@@ -43,7 +43,8 @@ def default_job_server(self, options):
class FlinkRunnerOptions(pipeline_options.PipelineOptions):
@classmethod
def _add_argparse_args(cls, parser):
-parser.add_argument('--flink_master', default='[local]')
+parser.add_argument('--flink_master', '--flink_master_url',
+default='[local]')
Review comment:
Good point! We should.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331414)
Time Spent: 5h 50m (was: 5h 40m)
> Allow submission of Flink UberJar directly to flink cluster.
>
>
> Key: BEAM-8372
> URL: https://issues.apache.org/jira/browse/BEAM-8372
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8430) sdk_worker_parallelism default is inconsistent between Py and Java SDKs
[ https://issues.apache.org/jira/browse/BEAM-8430?focusedWorklogId=331417&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331417 ] ASF GitHub Bot logged work on BEAM-8430: Author: ASF GitHub Bot Created on: 21/Oct/19 13:51 Start Date: 21/Oct/19 13:51 Worklog Time Spent: 10m Work Description: mxm commented on issue #9829: [BEAM-8430] Change py default sdk_worker_parallelism to 1 URL: https://github.com/apache/beam/pull/9829#issuecomment-544524559 Since we already have the option retrieval, we could leverage it to display the supported options from the Java SDK. However, it would be even better to have a language-agnostic option format which could be used by all SDKs to read from a single point of definition. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331417) Time Spent: 50m (was: 40m) > sdk_worker_parallelism default is inconsistent between Py and Java SDKs > --- > > Key: BEAM-8430 > URL: https://issues.apache.org/jira/browse/BEAM-8430 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core, sdk-py-core >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > Default is currently 1 in Java, 0 in Python. > https://github.com/apache/beam/blob/7b67a926b8939ede8f2e33c85579b540d18afccf/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L73 > https://github.com/apache/beam/blob/7b67a926b8939ede8f2e33c85579b540d18afccf/sdks/python/apache_beam/options/pipeline_options.py#L848 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (BEAM-8430) sdk_worker_parallelism default is inconsistent between Py and Java SDKs
[ https://issues.apache.org/jira/browse/BEAM-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maximilian Michels closed BEAM-8430. Fix Version/s: 2.17.0 Resolution: Fixed > sdk_worker_parallelism default is inconsistent between Py and Java SDKs > --- > > Key: BEAM-8430 > URL: https://issues.apache.org/jira/browse/BEAM-8430 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core, sdk-py-core >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Fix For: 2.17.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Default is currently 1 in Java, 0 in Python. > https://github.com/apache/beam/blob/7b67a926b8939ede8f2e33c85579b540d18afccf/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L73 > https://github.com/apache/beam/blob/7b67a926b8939ede8f2e33c85579b540d18afccf/sdks/python/apache_beam/options/pipeline_options.py#L848 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8430) sdk_worker_parallelism default is inconsistent between Py and Java SDKs
[ https://issues.apache.org/jira/browse/BEAM-8430?focusedWorklogId=331418&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331418 ] ASF GitHub Bot logged work on BEAM-8430: Author: ASF GitHub Bot Created on: 21/Oct/19 13:54 Start Date: 21/Oct/19 13:54 Worklog Time Spent: 10m Work Description: mxm commented on pull request #9829: [BEAM-8430] Change py default sdk_worker_parallelism to 1 URL: https://github.com/apache/beam/pull/9829 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331418) Time Spent: 1h (was: 50m) > sdk_worker_parallelism default is inconsistent between Py and Java SDKs > --- > > Key: BEAM-8430 > URL: https://issues.apache.org/jira/browse/BEAM-8430 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core, sdk-py-core >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Fix For: 2.17.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Default is currently 1 in Java, 0 in Python. > https://github.com/apache/beam/blob/7b67a926b8939ede8f2e33c85579b540d18afccf/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L73 > https://github.com/apache/beam/blob/7b67a926b8939ede8f2e33c85579b540d18afccf/sdks/python/apache_beam/options/pipeline_options.py#L848 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-4775) JobService should support returning metrics
[ https://issues.apache.org/jira/browse/BEAM-4775?focusedWorklogId=331419&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331419 ] ASF GitHub Bot logged work on BEAM-4775: Author: ASF GitHub Bot Created on: 21/Oct/19 13:56 Start Date: 21/Oct/19 13:56 Worklog Time Spent: 10m Work Description: kamilwu commented on issue #9843: [BEAM-4775] Converting MonitoringInfos to MetricResults in PortableRunner URL: https://github.com/apache/beam/pull/9843#issuecomment-544526591 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331419) Time Spent: 49h 50m (was: 49h 40m) > JobService should support returning metrics > --- > > Key: BEAM-4775 > URL: https://issues.apache.org/jira/browse/BEAM-4775 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Eugene Kirpichov >Assignee: Kamil Wasilewski >Priority: Major > Time Spent: 49h 50m > Remaining Estimate: 0h > > Design doc: [https://s.apache.org/get-metrics-api]. > Further discussion is ongoing on [this > doc|https://docs.google.com/document/d/1m83TsFvJbOlcLfXVXprQm1B7vUakhbLZMzuRrOHWnTg/edit?ts=5c826bb4#heading=h.faqan9rjc6dm]. > We want to report job metrics back to the portability harness from the runner > harness, for displaying to users. > h1. Relevant PRs in flight: > h2. Ready for Review: > * [#8022|https://github.com/apache/beam/pull/8022]: correct the Job RPC > protos from [#8018|https://github.com/apache/beam/pull/8018]. > h2. Iterating / Discussing: > * [#7971|https://github.com/apache/beam/pull/7971]: Flink portable metrics: > get ptransform from MonitoringInfo, not stage name > ** this is a simpler, Flink-specific PR that is basically duplicated inside > each of the following two, so may be worth trying to merge in first > * #[7915|https://github.com/apache/beam/pull/7915]: use MonitoringInfo data > model in Java SDK metrics > * [#7868|https://github.com/apache/beam/pull/7868]: MonitoringInfo URN tweaks > h2. Merged > * [#8018|https://github.com/apache/beam/pull/8018]: add job metrics RPC > protos > * [#7867|https://github.com/apache/beam/pull/7867]: key MetricResult by a > MetricKey > * [#7938|https://github.com/apache/beam/pull/7938]: move MonitoringInfo > protos to model/pipeline module > * [#7883|https://github.com/apache/beam/pull/7883]: Add > MetricQueryResults.allMetrics() helper > * [#7866|https://github.com/apache/beam/pull/7866]: move function helpers > from fn-harness to sdks/java/core > * [#7890|https://github.com/apache/beam/pull/7890]: consolidate MetricResult > implementations > h2. Closed > * [#7934|https://github.com/apache/beam/pull/7934]: job metrics RPC + SDK > support > * [#7876|https://github.com/apache/beam/pull/7876]: Clean up metric protos; > support integer distributions, gauges -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8439) Flink fails ParDoLifecycleTests
[ https://issues.apache.org/jira/browse/BEAM-8439?focusedWorklogId=331437&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331437 ] ASF GitHub Bot logged work on BEAM-8439: Author: ASF GitHub Bot Created on: 21/Oct/19 14:54 Start Date: 21/Oct/19 14:54 Worklog Time Spent: 10m Work Description: je-ik commented on issue #9841: [BEAM-8439] enable empty bundles in ParDoLifecycleTest URL: https://github.com/apache/beam/pull/9841#issuecomment-544552587 Run Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331437) Time Spent: 1.5h (was: 1h 20m) > Flink fails ParDoLifecycleTests > --- > > Key: BEAM-8439 > URL: https://issues.apache.org/jira/browse/BEAM-8439 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > ParDoLifecycleTest.testTeardownCalledAfterExceptionInFinishBundleStateful > fails consistently > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8439) Flink fails ParDoLifecycleTests
[ https://issues.apache.org/jira/browse/BEAM-8439?focusedWorklogId=331438&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331438 ] ASF GitHub Bot logged work on BEAM-8439: Author: ASF GitHub Bot Created on: 21/Oct/19 14:54 Start Date: 21/Oct/19 14:54 Worklog Time Spent: 10m Work Description: je-ik commented on issue #9841: [BEAM-8439] enable empty bundles in ParDoLifecycleTest URL: https://github.com/apache/beam/pull/9841#issuecomment-544552639 Run Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331438) Time Spent: 1h 40m (was: 1.5h) > Flink fails ParDoLifecycleTests > --- > > Key: BEAM-8439 > URL: https://issues.apache.org/jira/browse/BEAM-8439 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > ParDoLifecycleTest.testTeardownCalledAfterExceptionInFinishBundleStateful > fails consistently > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8439) Flink fails ParDoLifecycleTests
[ https://issues.apache.org/jira/browse/BEAM-8439?focusedWorklogId=331439&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331439 ] ASF GitHub Bot logged work on BEAM-8439: Author: ASF GitHub Bot Created on: 21/Oct/19 14:54 Start Date: 21/Oct/19 14:54 Worklog Time Spent: 10m Work Description: je-ik commented on issue #9841: [BEAM-8439] enable empty bundles in ParDoLifecycleTest URL: https://github.com/apache/beam/pull/9841#issuecomment-544552731 Run Spark ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331439) Time Spent: 1h 50m (was: 1h 40m) > Flink fails ParDoLifecycleTests > --- > > Key: BEAM-8439 > URL: https://issues.apache.org/jira/browse/BEAM-8439 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > ParDoLifecycleTest.testTeardownCalledAfterExceptionInFinishBundleStateful > fails consistently > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8439) Flink fails ParDoLifecycleTests
[ https://issues.apache.org/jira/browse/BEAM-8439?focusedWorklogId=331436&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331436 ] ASF GitHub Bot logged work on BEAM-8439: Author: ASF GitHub Bot Created on: 21/Oct/19 14:54 Start Date: 21/Oct/19 14:54 Worklog Time Spent: 10m Work Description: je-ik commented on issue #9841: [BEAM-8439] enable empty bundles in ParDoLifecycleTest URL: https://github.com/apache/beam/pull/9841#issuecomment-544552503 Run Direct ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331436) Time Spent: 1h 20m (was: 1h 10m) > Flink fails ParDoLifecycleTests > --- > > Key: BEAM-8439 > URL: https://issues.apache.org/jira/browse/BEAM-8439 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > ParDoLifecycleTest.testTeardownCalledAfterExceptionInFinishBundleStateful > fails consistently > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5690) Issue with GroupByKey in BeamSql using SparkRunner
[
https://issues.apache.org/jira/browse/BEAM-5690?focusedWorklogId=331443&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331443
]
ASF GitHub Bot logged work on BEAM-5690:
Author: ASF GitHub Bot
Created on: 21/Oct/19 15:09
Start Date: 21/Oct/19 15:09
Worklog Time Spent: 10m
Work Description: echauchot commented on pull request #9567: [BEAM-5690]
Fix Zero value issue with GroupByKey/CountByKey in SparkRunner
URL: https://github.com/apache/beam/pull/9567
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331443)
Time Spent: 4.5h (was: 4h 20m)
> Issue with GroupByKey in BeamSql using SparkRunner
> --
>
> Key: BEAM-5690
> URL: https://issues.apache.org/jira/browse/BEAM-5690
> Project: Beam
> Issue Type: Task
> Components: runner-spark
>Reporter: Kenneth Knowles
>Priority: Major
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> Reported on user@
> {quote}We are trying to setup a pipeline with using BeamSql and the trigger
> used is default (AfterWatermark crosses the window).
> Below is the pipeline:
>
>KafkaSource (KafkaIO)
>---> Windowing (FixedWindow 1min)
>---> BeamSql
>---> KafkaSink (KafkaIO)
>
> We are using Spark Runner for this.
> The BeamSql query is:
> {code}select Col3, count(*) as count_col1 from PCOLLECTION GROUP BY Col3{code}
> We are grouping by Col3 which is a string. It can hold values string[0-9].
>
> The records are getting emitted out at 1 min to kafka sink, but the output
> record in kafka is not as expected.
> Below is the output observed: (WST and WET are indicators for window start
> time and window end time)
> {code}
> {"count_col1":1,"Col3":"string5","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":3,"Col3":"string7","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":2,"Col3":"string8","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":1,"Col3":"string2","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":1,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09 09-55-00
> +","WET":"2018-10-09 09-56-00 0}
> {code}
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-4775) JobService should support returning metrics
[ https://issues.apache.org/jira/browse/BEAM-4775?focusedWorklogId=331444&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331444 ] ASF GitHub Bot logged work on BEAM-4775: Author: ASF GitHub Bot Created on: 21/Oct/19 15:14 Start Date: 21/Oct/19 15:14 Worklog Time Spent: 10m Work Description: kamilwu commented on issue #9843: [BEAM-4775] Converting MonitoringInfos to MetricResults in PortableRunner URL: https://github.com/apache/beam/pull/9843#issuecomment-544562055 R: @robertwb @ajamato Could you take a look? cc: @lgajowy This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331444) Time Spent: 50h (was: 49h 50m) > JobService should support returning metrics > --- > > Key: BEAM-4775 > URL: https://issues.apache.org/jira/browse/BEAM-4775 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Eugene Kirpichov >Assignee: Kamil Wasilewski >Priority: Major > Time Spent: 50h > Remaining Estimate: 0h > > Design doc: [https://s.apache.org/get-metrics-api]. > Further discussion is ongoing on [this > doc|https://docs.google.com/document/d/1m83TsFvJbOlcLfXVXprQm1B7vUakhbLZMzuRrOHWnTg/edit?ts=5c826bb4#heading=h.faqan9rjc6dm]. > We want to report job metrics back to the portability harness from the runner > harness, for displaying to users. > h1. Relevant PRs in flight: > h2. Ready for Review: > * [#8022|https://github.com/apache/beam/pull/8022]: correct the Job RPC > protos from [#8018|https://github.com/apache/beam/pull/8018]. > h2. Iterating / Discussing: > * [#7971|https://github.com/apache/beam/pull/7971]: Flink portable metrics: > get ptransform from MonitoringInfo, not stage name > ** this is a simpler, Flink-specific PR that is basically duplicated inside > each of the following two, so may be worth trying to merge in first > * #[7915|https://github.com/apache/beam/pull/7915]: use MonitoringInfo data > model in Java SDK metrics > * [#7868|https://github.com/apache/beam/pull/7868]: MonitoringInfo URN tweaks > h2. Merged > * [#8018|https://github.com/apache/beam/pull/8018]: add job metrics RPC > protos > * [#7867|https://github.com/apache/beam/pull/7867]: key MetricResult by a > MetricKey > * [#7938|https://github.com/apache/beam/pull/7938]: move MonitoringInfo > protos to model/pipeline module > * [#7883|https://github.com/apache/beam/pull/7883]: Add > MetricQueryResults.allMetrics() helper > * [#7866|https://github.com/apache/beam/pull/7866]: move function helpers > from fn-harness to sdks/java/core > * [#7890|https://github.com/apache/beam/pull/7890]: consolidate MetricResult > implementations > h2. Closed > * [#7934|https://github.com/apache/beam/pull/7934]: job metrics RPC + SDK > support > * [#7876|https://github.com/apache/beam/pull/7876]: Clean up metric protos; > support integer distributions, gauges -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8416) ZipFileArtifactServiceTest.test_concurrent_requests flaky
[ https://issues.apache.org/jira/browse/BEAM-8416?focusedWorklogId=331452&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331452 ] ASF GitHub Bot logged work on BEAM-8416: Author: ASF GitHub Bot Created on: 21/Oct/19 15:33 Start Date: 21/Oct/19 15:33 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #9845: [BEAM-8416] Hold the lock for the duration, not just the creation, of the iterator. URL: https://github.com/apache/beam/pull/9845 I was able to finally reproduce this on another machine. This fixes it. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[ Default to LOOPBACK mode for local flink (spark, ...) runner.
[ https://issues.apache.org/jira/browse/BEAM-8396?focusedWorklogId=331453&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331453 ] ASF GitHub Bot logged work on BEAM-8396: Author: ASF GitHub Bot Created on: 21/Oct/19 15:36 Start Date: 21/Oct/19 15:36 Worklog Time Spent: 10m Work Description: robertwb commented on issue #9833: [BEAM-8396] Default to LOOPBACK mode for local flink runner. URL: https://github.com/apache/beam/pull/9833#issuecomment-544572256 The unrelated failing tests is fixed by https://github.com/apache/beam/pull/9845 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331453) Time Spent: 0.5h (was: 20m) > Default to LOOPBACK mode for local flink (spark, ...) runner. > - > > Key: BEAM-8396 > URL: https://issues.apache.org/jira/browse/BEAM-8396 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > As well as being lower overhead, this will avoid surprises about workers > operating within the docker filesystem, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8396) Default to LOOPBACK mode for local flink (spark, ...) runner.
[ https://issues.apache.org/jira/browse/BEAM-8396?focusedWorklogId=331454&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331454 ] ASF GitHub Bot logged work on BEAM-8396: Author: ASF GitHub Bot Created on: 21/Oct/19 15:36 Start Date: 21/Oct/19 15:36 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #9833: [BEAM-8396] Default to LOOPBACK mode for local flink runner. URL: https://github.com/apache/beam/pull/9833 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331454) Time Spent: 40m (was: 0.5h) > Default to LOOPBACK mode for local flink (spark, ...) runner. > - > > Key: BEAM-8396 > URL: https://issues.apache.org/jira/browse/BEAM-8396 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > As well as being lower overhead, this will avoid surprises about workers > operating within the docker filesystem, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8439) Flink fails ParDoLifecycleTests
[ https://issues.apache.org/jira/browse/BEAM-8439?focusedWorklogId=331455&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331455 ] ASF GitHub Bot logged work on BEAM-8439: Author: ASF GitHub Bot Created on: 21/Oct/19 15:37 Start Date: 21/Oct/19 15:37 Worklog Time Spent: 10m Work Description: je-ik commented on issue #9841: [BEAM-8439] avoid creation of empty bundles URL: https://github.com/apache/beam/pull/9841#issuecomment-544572572 Run Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331455) Time Spent: 2h (was: 1h 50m) > Flink fails ParDoLifecycleTests > --- > > Key: BEAM-8439 > URL: https://issues.apache.org/jira/browse/BEAM-8439 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > ParDoLifecycleTest.testTeardownCalledAfterExceptionInFinishBundleStateful > fails consistently > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8439) Flink fails ParDoLifecycleTests
[ https://issues.apache.org/jira/browse/BEAM-8439?focusedWorklogId=331459&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331459 ] ASF GitHub Bot logged work on BEAM-8439: Author: ASF GitHub Bot Created on: 21/Oct/19 15:45 Start Date: 21/Oct/19 15:45 Worklog Time Spent: 10m Work Description: mxm commented on issue #9841: [BEAM-8439] avoid creation of empty bundles URL: https://github.com/apache/beam/pull/9841#issuecomment-544576185 Run Java Flink PortableValidatesRunner Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331459) Time Spent: 2h 10m (was: 2h) > Flink fails ParDoLifecycleTests > --- > > Key: BEAM-8439 > URL: https://issues.apache.org/jira/browse/BEAM-8439 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > ParDoLifecycleTest.testTeardownCalledAfterExceptionInFinishBundleStateful > fails consistently > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8444) Incompatibility between FnApiRunnerSplitTestWithMultiWorkers.test_checkpoint_sdf and beam_fn_api
Robert Bradshaw created BEAM-8444:
-
Summary: Incompatibility between
FnApiRunnerSplitTestWithMultiWorkers.test_checkpoint_sdf and beam_fn_api
Key: BEAM-8444
URL: https://issues.apache.org/jira/browse/BEAM-8444
Project: Beam
Issue Type: Bug
Components: sdk-py-harness
Reporter: Robert Bradshaw
{code}
test_checkpoint_sdf
(apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerSplitTestWithMultiWorkers)
... Traceback (most recent call last):
File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
line 170, in _execute
response = task()
File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
line 203, in
self._execute(lambda: worker.do_instruction(work), work)
File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
line 360, in do_instruction
request.instruction_id)
File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
line 386, in process_bundle
bundle_processor.process_bundle(instruction_id))
File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/bundle_processor.py",
line 663, in process_bundle
data.transform_id].process_encoded(data.data)
File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/bundle_processor.py",
line 143, in process_encoded
self.output(decoded_value)
File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/operations.py",
line 256, in output
cython.cast(Receiver, self.receivers[output_index]).receive(windowed_value)
File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/operations.py",
line 143, in receive
self.consumer.process(windowed_value)
File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/operations.py",
line 594, in process
delayed_application = self.dofn_receiver.receive(o)
File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/common.py",
line 776, in receive
self.process(windowed_value)
File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/common.py",
line 782, in process
self._reraise_augmented(exn)
File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/common.py",
line 780, in process
return self.do_fn_invoker.invoke_process(windowed_value)
File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/common.py",
line 588, in invoke_process
windowed_value, additional_args, additional_kwargs, output_processor)
File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/common.py",
line 660, in _invoke_process_per_window
windowed_value, self.process_method(*args_for_process))
File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/common.py",
line 919, in process_outputs
self.main_receivers.receive(windowed_value)
File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/operations.py",
line 143, in receive
self.consumer.process(windowed_value)
File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/operations.py",
line 594, in process
delayed_application = self.dofn_receiver.receive(o)
File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/common.py",
line 776, in re
[jira] [Work logged] (BEAM-8382) Add polling interval to KinesisIO.Read
[
https://issues.apache.org/jira/browse/BEAM-8382?focusedWorklogId=331463&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331463
]
ASF GitHub Bot logged work on BEAM-8382:
Author: ASF GitHub Bot
Created on: 21/Oct/19 15:58
Start Date: 21/Oct/19 15:58
Worklog Time Spent: 10m
Work Description: aromanenko-dev commented on issue #9765: [BEAM-8382]
Add polling interval to KinesisIO.Read
URL: https://github.com/apache/beam/pull/9765#issuecomment-544582003
@jfarr Sounds great for me! Thank you for your contribution on this!
Actually, I think it's very interesting and important topic for KinesisIO and
we have a field for improvement there.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331463)
Time Spent: 2h 40m (was: 2.5h)
> Add polling interval to KinesisIO.Read
> --
>
> Key: BEAM-8382
> URL: https://issues.apache.org/jira/browse/BEAM-8382
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kinesis
>Affects Versions: 2.13.0, 2.14.0, 2.15.0
>Reporter: Jonothan Farr
>Assignee: Jonothan Farr
>Priority: Major
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> With the current implementation we are observing Kinesis throttling due to
> ReadProvisionedThroughputExceeded on the order of hundreds of times per
> second, regardless of the actual Kinesis throughput. This is because the
> ShardReadersPool readLoop() method is polling getRecords() as fast as
> possible.
> From the KDS documentation:
> {quote}Each shard can support up to five read transactions per second.
> {quote}
> and
> {quote}For best results, sleep for at least 1 second (1,000 milliseconds)
> between calls to getRecords to avoid exceeding the limit on getRecords
> frequency.
> {quote}
> [https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html]
> [https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-sdk.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8418) Fix handling of Impulse transform in Dataflow runner.
[
https://issues.apache.org/jira/browse/BEAM-8418?focusedWorklogId=331464&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331464
]
ASF GitHub Bot logged work on BEAM-8418:
Author: ASF GitHub Bot
Created on: 21/Oct/19 16:07
Start Date: 21/Oct/19 16:07
Worklog Time Spent: 10m
Work Description: tvalentyn commented on issue #9822: [BEAM-8418] Use
base64 string for representing impulse payload in DF runner legacy codepath.
URL: https://github.com/apache/beam/pull/9822#issuecomment-544586597
R: @robertwb
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331464)
Time Spent: 2.5h (was: 2h 20m)
> Fix handling of Impulse transform in Dataflow runner.
> --
>
> Key: BEAM-8418
> URL: https://issues.apache.org/jira/browse/BEAM-8418
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Following pipeline fails on Dataflow runner unless we use beam_fn_api
> experiment.
> {noformat}
> class NoOpDoFn(beam.DoFn):
> def process(self, element):
> return element
> p = beam.Pipeline(options=pipeline_options)
> _ = p | beam.Impulse() | beam.ParDo(NoOpDoFn())
> result = p.run()
> {noformat}
> The reason is that we encode Impluse payload using url-escaping in [1], while
> Dataflow runner expects base64 encoding in non-fnapi mode. In FnApi mode, DF
> runner expects URL escaping.
> We should fix or reconcile the encoding in non-FnAPI path, and add a
> ValidatesRunner test that catches this error.
> [1]
> https://github.com/apache/beam/blob/12d07745835e1b9c1e824b83beeeadf63ab4b234/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L633
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8382) Add polling interval to KinesisIO.Read
[
https://issues.apache.org/jira/browse/BEAM-8382?focusedWorklogId=331468&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331468
]
ASF GitHub Bot logged work on BEAM-8382:
Author: ASF GitHub Bot
Created on: 21/Oct/19 16:16
Start Date: 21/Oct/19 16:16
Worklog Time Spent: 10m
Work Description: aromanenko-dev commented on issue #9765: [BEAM-8382]
Add polling interval to KinesisIO.Read
URL: https://github.com/apache/beam/pull/9765#issuecomment-544590712
@cmachgodaddy Hmm, what do you mean by "split"? One split is one shard, and
as you properly said before, only one thread from thread pool will consume
messages from this shard. Did I miss something?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331468)
Time Spent: 2h 50m (was: 2h 40m)
> Add polling interval to KinesisIO.Read
> --
>
> Key: BEAM-8382
> URL: https://issues.apache.org/jira/browse/BEAM-8382
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kinesis
>Affects Versions: 2.13.0, 2.14.0, 2.15.0
>Reporter: Jonothan Farr
>Assignee: Jonothan Farr
>Priority: Major
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> With the current implementation we are observing Kinesis throttling due to
> ReadProvisionedThroughputExceeded on the order of hundreds of times per
> second, regardless of the actual Kinesis throughput. This is because the
> ShardReadersPool readLoop() method is polling getRecords() as fast as
> possible.
> From the KDS documentation:
> {quote}Each shard can support up to five read transactions per second.
> {quote}
> and
> {quote}For best results, sleep for at least 1 second (1,000 milliseconds)
> between calls to getRecords to avoid exceeding the limit on getRecords
> frequency.
> {quote}
> [https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html]
> [https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-sdk.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7738) Support PubSubIO to be configured externally for use with other SDKs
[ https://issues.apache.org/jira/browse/BEAM-7738?focusedWorklogId=331472&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331472 ] ASF GitHub Bot logged work on BEAM-7738: Author: ASF GitHub Bot Created on: 21/Oct/19 16:20 Start Date: 21/Oct/19 16:20 Worklog Time Spent: 10m Work Description: chadrik commented on issue #9268: [BEAM-7738] Add external transform support to PubsubIO URL: https://github.com/apache/beam/pull/9268#issuecomment-544592670 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331472) Time Spent: 6h 10m (was: 6h) > Support PubSubIO to be configured externally for use with other SDKs > > > Key: BEAM-7738 > URL: https://issues.apache.org/jira/browse/BEAM-7738 > Project: Beam > Issue Type: New Feature > Components: io-java-gcp, runner-flink, sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Labels: portability > Time Spent: 6h 10m > Remaining Estimate: 0h > > Now that KafkaIO is supported via the external transform API (BEAM-7029) we > should add support for PubSub. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8382) Add polling interval to KinesisIO.Read
[
https://issues.apache.org/jira/browse/BEAM-8382?focusedWorklogId=331482&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331482
]
ASF GitHub Bot logged work on BEAM-8382:
Author: ASF GitHub Bot
Created on: 21/Oct/19 16:36
Start Date: 21/Oct/19 16:36
Worklog Time Spent: 10m
Work Description: cmachgodaddy commented on issue #9765: [BEAM-8382] Add
polling interval to KinesisIO.Read
URL: https://github.com/apache/beam/pull/9765#issuecomment-544599853
@aromanenko-dev , a bit technical, in KinesisSource, we have spit method,
this method generate a number of sources or readers (or KinesisReader) ? This
split method is now based on either the desiredNumSplits (which is configured
by user as a parallelism parameter) or partitions (number of shards) to
generate num of readers. However, in each reader (KinesisReader), we again
generate another number of threads and each read one shard. So this looks to me
a parallelism inside a parallelism (or nested parallelism). Here is in the
KinesisReader, which call ShareReadersPool::start(), and which uses
ExecutorService to execute 'readLoop' in parallel (in diff thread)
https://github.com/apache/beam/blob/master/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardReadersPool.java#L124.
Let me know if I am missing something?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331482)
Time Spent: 3h (was: 2h 50m)
> Add polling interval to KinesisIO.Read
> --
>
> Key: BEAM-8382
> URL: https://issues.apache.org/jira/browse/BEAM-8382
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kinesis
>Affects Versions: 2.13.0, 2.14.0, 2.15.0
>Reporter: Jonothan Farr
>Assignee: Jonothan Farr
>Priority: Major
> Time Spent: 3h
> Remaining Estimate: 0h
>
> With the current implementation we are observing Kinesis throttling due to
> ReadProvisionedThroughputExceeded on the order of hundreds of times per
> second, regardless of the actual Kinesis throughput. This is because the
> ShardReadersPool readLoop() method is polling getRecords() as fast as
> possible.
> From the KDS documentation:
> {quote}Each shard can support up to five read transactions per second.
> {quote}
> and
> {quote}For best results, sleep for at least 1 second (1,000 milliseconds)
> between calls to getRecords to avoid exceeding the limit on getRecords
> frequency.
> {quote}
> [https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html]
> [https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-sdk.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-3788) Implement a Kafka IO for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956261#comment-16956261 ] Chamikara Madhusanka Jayalath commented on BEAM-3788: - I believe unbounded SDF support for Python SDK is few months away. We are also working on adding cross-language transforms support to Dataflow but I don't have an ETA yet. As I mentioned in a previous comment either of these will pave the way for a KafkaIO in Python SDK on Dataflow. KafkaIO is already available as a cross-language transform for Flink: [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/external/kafka.py] > Implement a Kafka IO for Python SDK > --- > > Key: BEAM-3788 > URL: https://issues.apache.org/jira/browse/BEAM-3788 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Priority: Major > > This will be implemented using the Splittable DoFn framework. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8397) DataflowRunnerTest.test_remote_runner_display_data fails due to infinite recursion during pickling.
[
https://issues.apache.org/jira/browse/BEAM-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956268#comment-16956268
]
Valentyn Tymofieiev commented on BEAM-8397:
---
The test starts failing after [1] commit in dill.
We also established in [2] that this test also fails when running via [3], and
also starts failing if IPython>=6.0 is installed. Looking further to understand
the rootcause of the error.
[1]
https://github.com/uqfoundation/dill/commit/1cc66b404b539df76f8332440547c567a09b8b28
[2]
https://lists.apache.org/thread.html/fd77ce67ad14681a82b256312126ad115da5bcc67191dfbad2fae91a@%3Cdev.beam.apache.org%3E
[3] python ./setup.py nosetests --tests
'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data'
> DataflowRunnerTest.test_remote_runner_display_data fails due to infinite
> recursion during pickling.
> ---
>
> Key: BEAM-8397
> URL: https://issues.apache.org/jira/browse/BEAM-8397
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> `python ./setup.py test -s
> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
> passes.
> `tox -e py37-gcp` passes if Beam depends on dill==0.3.0, but fails if Beam
> depends on dill==0.3.1.1.`python ./setup.py nosetests --tests
> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data`
> fails currently if run on master.
> The failure indicates infinite recursion during pickling:
> {noformat}
> test_remote_runner_display_data
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) ...
> Fatal Python error: Cannot recover from stack overflow.
> Current thread 0x7f9d700ed740 (most recent call first):
> File "/usr/lib/python3.7/pickle.py", line 479 in get
> File "/usr/lib/python3.7/pickle.py", line 497 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 910 in save_module_dict
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 198 in new_save_module_dict
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 114 in wrapper
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 771 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1137 in save_cell
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 771 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 910 in save_module_dict
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 198 in new_save_module_dict
> ...
> {noformat}
> cc: [~lazylynx]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8439) Flink fails ParDoLifecycleTests
[ https://issues.apache.org/jira/browse/BEAM-8439?focusedWorklogId=331511&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331511 ] ASF GitHub Bot logged work on BEAM-8439: Author: ASF GitHub Bot Created on: 21/Oct/19 16:58 Start Date: 21/Oct/19 16:58 Worklog Time Spent: 10m Work Description: mxm commented on issue #9841: [BEAM-8439] avoid creation of empty bundles URL: https://github.com/apache/beam/pull/9841#issuecomment-544608902 Run Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331511) Time Spent: 2h 20m (was: 2h 10m) > Flink fails ParDoLifecycleTests > --- > > Key: BEAM-8439 > URL: https://issues.apache.org/jira/browse/BEAM-8439 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > ParDoLifecycleTest.testTeardownCalledAfterExceptionInFinishBundleStateful > fails consistently > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8439) Flink fails ParDoLifecycleTests
[ https://issues.apache.org/jira/browse/BEAM-8439?focusedWorklogId=331512&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331512 ] ASF GitHub Bot logged work on BEAM-8439: Author: ASF GitHub Bot Created on: 21/Oct/19 16:58 Start Date: 21/Oct/19 16:58 Worklog Time Spent: 10m Work Description: mxm commented on issue #9841: [BEAM-8439] avoid creation of empty bundles URL: https://github.com/apache/beam/pull/9841#issuecomment-544608937 Run Java Flink PortableValidatesRunner Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331512) Time Spent: 2.5h (was: 2h 20m) > Flink fails ParDoLifecycleTests > --- > > Key: BEAM-8439 > URL: https://issues.apache.org/jira/browse/BEAM-8439 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > > ParDoLifecycleTest.testTeardownCalledAfterExceptionInFinishBundleStateful > fails consistently > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8397) DataflowRunnerTest.test_remote_runner_display_data fails due to infinite recursion during pickling.
[
https://issues.apache.org/jira/browse/BEAM-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956291#comment-16956291
]
Valentyn Tymofieiev commented on BEAM-8397:
---
Similarly, via git bisect we can find that commit [1] in IPython causes this
error when running tox -e py37-gcp.
[1]
https://github.com/ipython/ipython/commit/3ff1be2ea8ef180a6f17a6a03a3f8452303b9abe
> DataflowRunnerTest.test_remote_runner_display_data fails due to infinite
> recursion during pickling.
> ---
>
> Key: BEAM-8397
> URL: https://issues.apache.org/jira/browse/BEAM-8397
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> `python ./setup.py test -s
> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
> passes.
> `tox -e py37-gcp` passes if Beam depends on dill==0.3.0, but fails if Beam
> depends on dill==0.3.1.1.`python ./setup.py nosetests --tests
> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data`
> fails currently if run on master.
> The failure indicates infinite recursion during pickling:
> {noformat}
> test_remote_runner_display_data
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) ...
> Fatal Python error: Cannot recover from stack overflow.
> Current thread 0x7f9d700ed740 (most recent call first):
> File "/usr/lib/python3.7/pickle.py", line 479 in get
> File "/usr/lib/python3.7/pickle.py", line 497 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 910 in save_module_dict
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 198 in new_save_module_dict
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 114 in wrapper
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 771 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1137 in save_cell
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 771 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 910 in save_module_dict
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 198 in new_save_module_dict
> ...
> {noformat}
> cc: [~lazylynx]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8382) Add polling interval to KinesisIO.Read
[
https://issues.apache.org/jira/browse/BEAM-8382?focusedWorklogId=331521&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331521
]
ASF GitHub Bot logged work on BEAM-8382:
Author: ASF GitHub Bot
Created on: 21/Oct/19 17:17
Start Date: 21/Oct/19 17:17
Worklog Time Spent: 10m
Work Description: aromanenko-dev commented on issue #9765: [BEAM-8382]
Add polling interval to KinesisIO.Read
URL: https://github.com/apache/beam/pull/9765#issuecomment-544616617
@cmachgodaddy My understanding is the following. `KinesisSource.split()`
method is called on driver, before running a pipeline. So, it will create a
list of `KinesisSource`s for every split/partition (based on
`initialCheckpointGenerator`) that will be distributed on workers and run
there. Then every `KinesisSource` will create `KinesisReader` with own
`CheckpointGenerator` (on worker) and instantiate `ShardReadersPool` which will
create thread for every shard iterator (since every reader can read from more
than one shard). So, I don't see nested parallelism here. Though, it would be
great to test it in case if we have a bug there since this logic is quite
complicated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331521)
Time Spent: 3h 10m (was: 3h)
> Add polling interval to KinesisIO.Read
> --
>
> Key: BEAM-8382
> URL: https://issues.apache.org/jira/browse/BEAM-8382
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kinesis
>Affects Versions: 2.13.0, 2.14.0, 2.15.0
>Reporter: Jonothan Farr
>Assignee: Jonothan Farr
>Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> With the current implementation we are observing Kinesis throttling due to
> ReadProvisionedThroughputExceeded on the order of hundreds of times per
> second, regardless of the actual Kinesis throughput. This is because the
> ShardReadersPool readLoop() method is polling getRecords() as fast as
> possible.
> From the KDS documentation:
> {quote}Each shard can support up to five read transactions per second.
> {quote}
> and
> {quote}For best results, sleep for at least 1 second (1,000 milliseconds)
> between calls to getRecords to avoid exceeding the limit on getRecords
> frequency.
> {quote}
> [https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html]
> [https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-sdk.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7389) Colab examples for element-wise transforms (Python)
[ https://issues.apache.org/jira/browse/BEAM-7389?focusedWorklogId=331525&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331525 ] ASF GitHub Bot logged work on BEAM-7389: Author: ASF GitHub Bot Created on: 21/Oct/19 17:27 Start Date: 21/Oct/19 17:27 Worklog Time Spent: 10m Work Description: davidcavazos commented on issue #9790: [BEAM-7389] Show code snippet outputs as stdout URL: https://github.com/apache/beam/pull/9790#issuecomment-544620767 @aaltay This is ready for review This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331525) Time Spent: 69h 40m (was: 69.5h) > Colab examples for element-wise transforms (Python) > --- > > Key: BEAM-7389 > URL: https://issues.apache.org/jira/browse/BEAM-7389 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Rose Nguyen >Assignee: David Cavazos >Priority: Minor > Time Spent: 69h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7926) Visualize PCollection with Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-7926?focusedWorklogId=331529&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331529 ] ASF GitHub Bot logged work on BEAM-7926: Author: ASF GitHub Bot Created on: 21/Oct/19 17:32 Start Date: 21/Oct/19 17:32 Worklog Time Spent: 10m Work Description: KevinGG commented on issue #9741: [BEAM-7926] Visualize PCollection URL: https://github.com/apache/beam/pull/9741#issuecomment-544622565 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331529) Time Spent: 6h 50m (was: 6h 40m) > Visualize PCollection with Interactive Beam > --- > > Key: BEAM-7926 > URL: https://issues.apache.org/jira/browse/BEAM-7926 > Project: Beam > Issue Type: New Feature > Components: runner-py-interactive >Reporter: Ning Kang >Assignee: Ning Kang >Priority: Major > Time Spent: 6h 50m > Remaining Estimate: 0h > > Support auto plotting / charting of materialized data of a given PCollection > with Interactive Beam. > Say an Interactive Beam pipeline defined as > p = create_pipeline() > pcoll = p | 'Transform' >> transform() > The use can call a single function and get auto-magical charting of the data > as materialized pcoll. > e.g., visualize(pcoll) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8341) basic bundling support for samza portable runner
[
https://issues.apache.org/jira/browse/BEAM-8341?focusedWorklogId=331534&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331534
]
ASF GitHub Bot logged work on BEAM-8341:
Author: ASF GitHub Bot
Created on: 21/Oct/19 17:42
Start Date: 21/Oct/19 17:42
Worklog Time Spent: 10m
Work Description: lhaiesp commented on pull request #9777: [BEAM-8341]:
basic bundling support for portable runner
URL: https://github.com/apache/beam/pull/9777#discussion_r337150103
##
File path:
runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/DoFnOp.java
##
@@ -237,9 +292,39 @@ private String getTimerStateId(DoFnSignature signature) {
return builder.toString();
}
+ private void attemptStartBundle() {
+if (isBundleStarted.compareAndSet(false, true)) {
+ currentBundleElementCount.set(0L);
+ bundleStartTime.set(System.currentTimeMillis());
+ pushbackFnRunner.startBundle();
Review comment:
There are two models. 1) the current model is to regularly check in the
background to see whether max bundling time is hit; this is only roughly
accurate but more efficient 2) schedule a timer for EVERY bundle start. This is
more accurate but we may end up scheduling too many timer when the QPS is too
high. Let's discuss this
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331534)
Time Spent: 1h (was: 50m)
> basic bundling support for samza portable runner
>
>
> Key: BEAM-8341
> URL: https://issues.apache.org/jira/browse/BEAM-8341
> Project: Beam
> Issue Type: Task
> Components: runner-samza
>Reporter: Hai Lu
>Assignee: Hai Lu
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> bundling support for samza portable runner
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8341) basic bundling support for samza portable runner
[
https://issues.apache.org/jira/browse/BEAM-8341?focusedWorklogId=331535&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331535
]
ASF GitHub Bot logged work on BEAM-8341:
Author: ASF GitHub Bot
Created on: 21/Oct/19 17:45
Start Date: 21/Oct/19 17:45
Worklog Time Spent: 10m
Work Description: lhaiesp commented on pull request #9777: [BEAM-8341]:
basic bundling support for portable runner
URL: https://github.com/apache/beam/pull/9777#discussion_r337151398
##
File path:
runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/DoFnOp.java
##
@@ -237,9 +292,39 @@ private String getTimerStateId(DoFnSignature signature) {
return builder.toString();
}
+ private void attemptStartBundle() {
+if (isBundleStarted.compareAndSet(false, true)) {
Review comment:
Make sense. Will do
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331535)
Time Spent: 1h 10m (was: 1h)
> basic bundling support for samza portable runner
>
>
> Key: BEAM-8341
> URL: https://issues.apache.org/jira/browse/BEAM-8341
> Project: Beam
> Issue Type: Task
> Components: runner-samza
>Reporter: Hai Lu
>Assignee: Hai Lu
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> bundling support for samza portable runner
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5967) ProtoCoder doesn't support DynamicMessage
[
https://issues.apache.org/jira/browse/BEAM-5967?focusedWorklogId=331541&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331541
]
ASF GitHub Bot logged work on BEAM-5967:
Author: ASF GitHub Bot
Created on: 21/Oct/19 17:56
Start Date: 21/Oct/19 17:56
Worklog Time Spent: 10m
Work Description: TheNeuralBit commented on issue #8496: [BEAM-5967] Add
handling of DynamicMessage in ProtoCoder
URL: https://github.com/apache/beam/pull/8496#issuecomment-544632827
Can this be merged before the 2.17 cut (scheduled for Wednesday, October 23)?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331541)
Time Spent: 6h 10m (was: 6h)
> ProtoCoder doesn't support DynamicMessage
> -
>
> Key: BEAM-5967
> URL: https://issues.apache.org/jira/browse/BEAM-5967
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Affects Versions: 2.8.0
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Major
> Fix For: 2.17.0
>
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> The ProtoCoder does make some assumptions about static messages being
> available. The DynamicMessage doesn't have some of them, mainly because the
> proto schema is defined at runtime and not at compile time.
> Does it make sense to make a special coder for DynamicMessage or build it
> into the normal ProtoCoder.
> Here is an example of the assumtion being made in the current Codec:
> {code:java}
> try {
> @SuppressWarnings("unchecked")
> T protoMessageInstance = (T)
> protoMessageClass.getMethod("getDefaultInstance").invoke(null);
> @SuppressWarnings("unchecked")
> Parser tParser = (Parser) protoMessageInstance.getParserForType();
> memoizedParser = tParser;
> } catch (IllegalAccessException | InvocationTargetException |
> NoSuchMethodException e) {
> throw new IllegalArgumentException(e);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7926) Visualize PCollection with Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-7926?focusedWorklogId=331543&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331543 ] ASF GitHub Bot logged work on BEAM-7926: Author: ASF GitHub Bot Created on: 21/Oct/19 18:01 Start Date: 21/Oct/19 18:01 Worklog Time Spent: 10m Work Description: rohdesamuel commented on issue #9741: [BEAM-7926] Visualize PCollection URL: https://github.com/apache/beam/pull/9741#issuecomment-544634753 lgtm This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331543) Time Spent: 7h (was: 6h 50m) > Visualize PCollection with Interactive Beam > --- > > Key: BEAM-7926 > URL: https://issues.apache.org/jira/browse/BEAM-7926 > Project: Beam > Issue Type: New Feature > Components: runner-py-interactive >Reporter: Ning Kang >Assignee: Ning Kang >Priority: Major > Time Spent: 7h > Remaining Estimate: 0h > > Support auto plotting / charting of materialized data of a given PCollection > with Interactive Beam. > Say an Interactive Beam pipeline defined as > p = create_pipeline() > pcoll = p | 'Transform' >> transform() > The use can call a single function and get auto-magical charting of the data > as materialized pcoll. > e.g., visualize(pcoll) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8439) Flink fails ParDoLifecycleTests
[ https://issues.apache.org/jira/browse/BEAM-8439?focusedWorklogId=331547&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331547 ] ASF GitHub Bot logged work on BEAM-8439: Author: ASF GitHub Bot Created on: 21/Oct/19 18:03 Start Date: 21/Oct/19 18:03 Worklog Time Spent: 10m Work Description: je-ik commented on pull request #9841: [BEAM-8439] avoid creation of empty bundles URL: https://github.com/apache/beam/pull/9841 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331547) Time Spent: 2h 50m (was: 2h 40m) > Flink fails ParDoLifecycleTests > --- > > Key: BEAM-8439 > URL: https://issues.apache.org/jira/browse/BEAM-8439 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > ParDoLifecycleTest.testTeardownCalledAfterExceptionInFinishBundleStateful > fails consistently > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8439) Flink fails ParDoLifecycleTests
[ https://issues.apache.org/jira/browse/BEAM-8439?focusedWorklogId=331546&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331546 ] ASF GitHub Bot logged work on BEAM-8439: Author: ASF GitHub Bot Created on: 21/Oct/19 18:03 Start Date: 21/Oct/19 18:03 Worklog Time Spent: 10m Work Description: je-ik commented on issue #9841: [BEAM-8439] avoid creation of empty bundles URL: https://github.com/apache/beam/pull/9841#issuecomment-544635386 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331546) Time Spent: 2h 40m (was: 2.5h) > Flink fails ParDoLifecycleTests > --- > > Key: BEAM-8439 > URL: https://issues.apache.org/jira/browse/BEAM-8439 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > ParDoLifecycleTest.testTeardownCalledAfterExceptionInFinishBundleStateful > fails consistently > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8439) Flink fails ParDoLifecycleTests
[ https://issues.apache.org/jira/browse/BEAM-8439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Lukavský resolved BEAM-8439. Fix Version/s: 2.17.0 Resolution: Fixed > Flink fails ParDoLifecycleTests > --- > > Key: BEAM-8439 > URL: https://issues.apache.org/jira/browse/BEAM-8439 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Fix For: 2.17.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > ParDoLifecycleTest.testTeardownCalledAfterExceptionInFinishBundleStateful > fails consistently > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (BEAM-8439) Flink fails ParDoLifecycleTests
[ https://issues.apache.org/jira/browse/BEAM-8439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Lukavský closed BEAM-8439. -- > Flink fails ParDoLifecycleTests > --- > > Key: BEAM-8439 > URL: https://issues.apache.org/jira/browse/BEAM-8439 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Fix For: 2.17.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > ParDoLifecycleTest.testTeardownCalledAfterExceptionInFinishBundleStateful > fails consistently > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7520) DirectRunner timers are not strictly time ordered
[ https://issues.apache.org/jira/browse/BEAM-7520?focusedWorklogId=331549&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331549 ] ASF GitHub Bot logged work on BEAM-7520: Author: ASF GitHub Bot Created on: 21/Oct/19 18:06 Start Date: 21/Oct/19 18:06 Worklog Time Spent: 10m Work Description: je-ik commented on issue #9190: [BEAM-7520] Fix timer firing order in DirectRunner URL: https://github.com/apache/beam/pull/9190#issuecomment-544636783 Run Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331549) Time Spent: 16h (was: 15h 50m) > DirectRunner timers are not strictly time ordered > - > > Key: BEAM-7520 > URL: https://issues.apache.org/jira/browse/BEAM-7520 > Project: Beam > Issue Type: Bug > Components: runner-direct >Affects Versions: 2.13.0 >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Time Spent: 16h > Remaining Estimate: 0h > > Let's suppose we have the following situation: > - statful ParDo with two timers - timerA and timerB > - timerA is set for window.maxTimestamp() + 1 > - timerB is set anywhere between timerB.timestamp > - input watermark moves to BoundedWindow.TIMESTAMP_MAX_VALUE > Then the order of timers is as follows (correct): > - timerB > - timerA > But, if timerB sets another timer (say for timerB.timestamp + 1), then the > order of timers will be: > - timerB (timerB.timestamp) > - timerA (BoundedWindow.TIMESTAMP_MAX_VALUE) > - timerB (timerB.timestamp + 1) > Which is not ordered by timestamp. The reason for this is that when the input > watermark update is evaluated, the WatermarkManager,extractFiredTimers() will > produce both timerA and timerB. That would be correct, but when timerB sets > another timer, that breaks this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8433) DataCatalogBigQueryIT runs for both Calcite and ZetaSQL dialects
[ https://issues.apache.org/jira/browse/BEAM-8433?focusedWorklogId=331550&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331550 ] ASF GitHub Bot logged work on BEAM-8433: Author: ASF GitHub Bot Created on: 21/Oct/19 18:07 Start Date: 21/Oct/19 18:07 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #9838: [BEAM-8433] Minor refactors to Data Catalog BigQuery integration test config & test class URL: https://github.com/apache/beam/pull/9838#issuecomment-544637123 Since these are opportunistic improvements while developing #9835, which is really just after-the-fact improvements to #9831 (speeding code review of the latter), I've chosen BEAM-8433 for all of these PRs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331550) Time Spent: 1.5h (was: 1h 20m) > DataCatalogBigQueryIT runs for both Calcite and ZetaSQL dialects > > > Key: BEAM-8433 > URL: https://issues.apache.org/jira/browse/BEAM-8433 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Fix For: Not applicable > > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8433) DataCatalogBigQueryIT runs for both Calcite and ZetaSQL dialects
[ https://issues.apache.org/jira/browse/BEAM-8433?focusedWorklogId=331551&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331551 ] ASF GitHub Bot logged work on BEAM-8433: Author: ASF GitHub Bot Created on: 21/Oct/19 18:08 Start Date: 21/Oct/19 18:08 Worklog Time Spent: 10m Work Description: kennknowles commented on pull request #9838: [BEAM-8433] Minor refactors to Data Catalog BigQuery integration test config & test class URL: https://github.com/apache/beam/pull/9838 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331551) Time Spent: 1h 40m (was: 1.5h) > DataCatalogBigQueryIT runs for both Calcite and ZetaSQL dialects > > > Key: BEAM-8433 > URL: https://issues.apache.org/jira/browse/BEAM-8433 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Fix For: Not applicable > > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7520) DirectRunner timers are not strictly time ordered
[ https://issues.apache.org/jira/browse/BEAM-7520?focusedWorklogId=331556&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331556 ] ASF GitHub Bot logged work on BEAM-7520: Author: ASF GitHub Bot Created on: 21/Oct/19 18:09 Start Date: 21/Oct/19 18:09 Worklog Time Spent: 10m Work Description: je-ik commented on issue #9190: [BEAM-7520] Fix timer firing order in DirectRunner URL: https://github.com/apache/beam/pull/9190#issuecomment-544637739 Run Samza ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331556) Time Spent: 16h 10m (was: 16h) > DirectRunner timers are not strictly time ordered > - > > Key: BEAM-7520 > URL: https://issues.apache.org/jira/browse/BEAM-7520 > Project: Beam > Issue Type: Bug > Components: runner-direct >Affects Versions: 2.13.0 >Reporter: Jan Lukavský >Assignee: Jan Lukavský >Priority: Major > Time Spent: 16h 10m > Remaining Estimate: 0h > > Let's suppose we have the following situation: > - statful ParDo with two timers - timerA and timerB > - timerA is set for window.maxTimestamp() + 1 > - timerB is set anywhere between timerB.timestamp > - input watermark moves to BoundedWindow.TIMESTAMP_MAX_VALUE > Then the order of timers is as follows (correct): > - timerB > - timerA > But, if timerB sets another timer (say for timerB.timestamp + 1), then the > order of timers will be: > - timerB (timerB.timestamp) > - timerA (BoundedWindow.TIMESTAMP_MAX_VALUE) > - timerB (timerB.timestamp + 1) > Which is not ordered by timestamp. The reason for this is that when the input > watermark update is evaluated, the WatermarkManager,extractFiredTimers() will > produce both timerA and timerB. That would be correct, but when timerB sets > another timer, that breaks this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8445) ZetaSQL translator returns null from unsupported methods
Kenneth Knowles created BEAM-8445: - Summary: ZetaSQL translator returns null from unsupported methods Key: BEAM-8445 URL: https://issues.apache.org/jira/browse/BEAM-8445 Project: Beam Issue Type: Bug Components: dsl-sql-zetasql Reporter: Kenneth Knowles Assignee: Kenneth Knowles This is an error-prone style that was missed because our null analysis is disabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8433) DataCatalogBigQueryIT runs for both Calcite and ZetaSQL dialects
[
https://issues.apache.org/jira/browse/BEAM-8433?focusedWorklogId=331559&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331559
]
ASF GitHub Bot logged work on BEAM-8433:
Author: ASF GitHub Bot
Created on: 21/Oct/19 18:10
Start Date: 21/Oct/19 18:10
Worklog Time Spent: 10m
Work Description: kennknowles commented on pull request #9838:
[BEAM-8433] Minor refactors to Data Catalog BigQuery integration test config &
test class
URL: https://github.com/apache/beam/pull/9838#discussion_r337163352
##
File path: sdks/java/extensions/sql/datacatalog/build.gradle
##
@@ -37,6 +36,8 @@ dependencies {
// Dependencies for the example
provided project(":sdks:java:io:google-cloud-platform")
provided library.java.slf4j_api
+
+ testCompile project(":sdks:java:extensions:sql:zetasql")
Review comment:
Ah, you are right. An artifact of an intermediate state of the PR in which I
had an enum that returned an actual class.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331559)
Time Spent: 1h 50m (was: 1h 40m)
> DataCatalogBigQueryIT runs for both Calcite and ZetaSQL dialects
>
>
> Key: BEAM-8433
> URL: https://issues.apache.org/jira/browse/BEAM-8433
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8433) DataCatalogBigQueryIT runs for both Calcite and ZetaSQL dialects
[
https://issues.apache.org/jira/browse/BEAM-8433?focusedWorklogId=331561&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331561
]
ASF GitHub Bot logged work on BEAM-8433:
Author: ASF GitHub Bot
Created on: 21/Oct/19 18:12
Start Date: 21/Oct/19 18:12
Worklog Time Spent: 10m
Work Description: kennknowles commented on pull request #9838:
[BEAM-8433] Minor refactors to Data Catalog BigQuery integration test config &
test class
URL: https://github.com/apache/beam/pull/9838#discussion_r337163927
##
File path: sdks/java/extensions/sql/datacatalog/build.gradle
##
@@ -37,6 +36,8 @@ dependencies {
// Dependencies for the example
provided project(":sdks:java:io:google-cloud-platform")
provided library.java.slf4j_api
+
+ testCompile project(":sdks:java:extensions:sql:zetasql")
Review comment:
Err, not intermediate state of this PR, but the fact that it was part of
#9835 where I use actual classes to eliminate the possibility of a typo.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331561)
Time Spent: 2h (was: 1h 50m)
> DataCatalogBigQueryIT runs for both Calcite and ZetaSQL dialects
>
>
> Key: BEAM-8433
> URL: https://issues.apache.org/jira/browse/BEAM-8433
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8382) Add polling interval to KinesisIO.Read
[
https://issues.apache.org/jira/browse/BEAM-8382?focusedWorklogId=331563&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331563
]
ASF GitHub Bot logged work on BEAM-8382:
Author: ASF GitHub Bot
Created on: 21/Oct/19 18:13
Start Date: 21/Oct/19 18:13
Worklog Time Spent: 10m
Work Description: cmachgodaddy commented on issue #9765: [BEAM-8382] Add
polling interval to KinesisIO.Read
URL: https://github.com/apache/beam/pull/9765#issuecomment-544639586
@alexvanboxel , I think we all understand it right, from what you described
above we just say it in different ways ;-) . In simple words, each split, run
on each worker, will create a number of threads base off num of shards. That's
means if we have 10 splits, we will have 10 guys/reader read one shard (of
course, it will read all shards)? Hope you agree with me this point? Now, what
Amazon recommend is we should have one client read one shard. Here is what they
say:
> Typically, when you use the KCL, you should ensure that the number of
instances does not exceed the number of shards (except for failure standby
purposes). Each shard is processed by exactly one KCL worker and has exactly
one corresponding record processor, so you never need multiple instances to
process one shard. However, one worker can process any number of shards, so
it's fine if the number of shards exceeds the number of instances.
And, here is why they do that,
https://docs.aws.amazon.com/streams/latest/dev/kinesis-low-latency.html
But, don't argue that, here they use KCL, so pls consider one KCL is one
KinesisClient that our split use to connect to Kinesis and read a shard. And
don't misunderstand the `one worker can read a number of shard`. Here in their
context, or the example in their doc, their worker is an application that run
on EC2 instance, and when that scale this application, it will loadbalance the
KCL, e.g. two EC2 will have two KCL, each read two shards (assume a stream has
4 shards).
The point is we don't want to have number readers greater number of shards
in one application (or one pipeline in our context)? Imagine, if we have 10
pipelines deployed in our runners, and which has parallelism of 10, then we
will have 10 x 10 readers reading one shard ? And this 10 x 10 readers are not
loadbalanced since we are not using KCL? (even with KCL, this number still
exceed num of shard?). I am sure we will get "throughput exception" in the log
:-)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331563)
Time Spent: 3h 20m (was: 3h 10m)
> Add polling interval to KinesisIO.Read
> --
>
> Key: BEAM-8382
> URL: https://issues.apache.org/jira/browse/BEAM-8382
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kinesis
>Affects Versions: 2.13.0, 2.14.0, 2.15.0
>Reporter: Jonothan Farr
>Assignee: Jonothan Farr
>Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> With the current implementation we are observing Kinesis throttling due to
> ReadProvisionedThroughputExceeded on the order of hundreds of times per
> second, regardless of the actual Kinesis throughput. This is because the
> ShardReadersPool readLoop() method is polling getRecords() as fast as
> possible.
> From the KDS documentation:
> {quote}Each shard can support up to five read transactions per second.
> {quote}
> and
> {quote}For best results, sleep for at least 1 second (1,000 milliseconds)
> between calls to getRecords to avoid exceeding the limit on getRecords
> frequency.
> {quote}
> [https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html]
> [https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-sdk.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8445) ZetaSQL translator returns null from unsupported methods
[
https://issues.apache.org/jira/browse/BEAM-8445?focusedWorklogId=331562&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331562
]
ASF GitHub Bot logged work on BEAM-8445:
Author: ASF GitHub Bot
Created on: 21/Oct/19 18:13
Start Date: 21/Oct/19 18:13
Worklog Time Spent: 10m
Work Description: kennknowles commented on pull request #9837:
[BEAM-8445] Fix some not allowed nulls in ZetaSQL translator
URL: https://github.com/apache/beam/pull/9837#discussion_r337164659
##
File path:
sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
##
@@ -98,27 +98,42 @@ public ZetaSQLPlannerImpl(FrameworkConfig config) {
@Override
public SqlNode parse(String s) throws SqlParseException {
-return null;
+throw new UnsupportedOperationException(
+String.format(
+"%s.parse(String) is not implemented and should need be called",
Review comment:
Ah "should _not_ need to be called". But yes, it can be simplified. My
intent was to signal to the client that this is a class with a bunch of extra
methods most of which are not needed for the functionality it delivers.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331562)
Remaining Estimate: 0h
Time Spent: 10m
> ZetaSQL translator returns null from unsupported methods
>
>
> Key: BEAM-8445
> URL: https://issues.apache.org/jira/browse/BEAM-8445
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This is an error-prone style that was missed because our null analysis is
> disabled.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8382) Add polling interval to KinesisIO.Read
[
https://issues.apache.org/jira/browse/BEAM-8382?focusedWorklogId=331564&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331564
]
ASF GitHub Bot logged work on BEAM-8382:
Author: ASF GitHub Bot
Created on: 21/Oct/19 18:14
Start Date: 21/Oct/19 18:14
Worklog Time Spent: 10m
Work Description: cmachgodaddy commented on issue #9765: [BEAM-8382] Add
polling interval to KinesisIO.Read
URL: https://github.com/apache/beam/pull/9765#issuecomment-544639586
@alexvanboxel , I think we all understand it right, from what you described
above we just say it in different ways ;-) . In simple words, each split, run
on each worker, will create a number of threads base off num of shards. That's
means if we have 10 splits, we will have 10 guys/reader read one shard (of
course, it will read all shards)? Hope you agree with me this point? Now, what
Amazon recommend is we should have one client read one shard. Here is what they
say:
> Typically, when you use the KCL, you should ensure that the number of
instances does not exceed the number of shards (except for failure standby
purposes). Each shard is processed by exactly one KCL worker and has exactly
one corresponding record processor, so you never need multiple instances to
process one shard. However, one worker can process any number of shards, so
it's fine if the number of shards exceeds the number of instances.
And, here is why they do that,
https://docs.aws.amazon.com/streams/latest/dev/kinesis-low-latency.html
But, don't argue that here they use KCL, so pls consider one KCL is one
KinesisClient that our split use to connect to Kinesis and read a shard. And
don't misunderstand the `one worker can read a number of shard`. Here in their
context, or the example in their doc, their worker is an application that run
on EC2 instance, and when that scale this application, it will loadbalance the
KCL, e.g. two EC2 will have two KCL, each read two shards (assume a stream has
4 shards).
The point is we don't want to have number readers greater number of shards
in one application (or one pipeline in our context)? Imagine, if we have 10
pipelines deployed in our runners, and which has parallelism of 10, then we
will have 10 x 10 readers reading one shard ? And this 10 x 10 readers are not
loadbalanced since we are not using KCL? (even with KCL, this number still
exceed num of shard?). I am sure we will get "throughput exception" in the log
:-)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331564)
Time Spent: 3.5h (was: 3h 20m)
> Add polling interval to KinesisIO.Read
> --
>
> Key: BEAM-8382
> URL: https://issues.apache.org/jira/browse/BEAM-8382
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kinesis
>Affects Versions: 2.13.0, 2.14.0, 2.15.0
>Reporter: Jonothan Farr
>Assignee: Jonothan Farr
>Priority: Major
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> With the current implementation we are observing Kinesis throttling due to
> ReadProvisionedThroughputExceeded on the order of hundreds of times per
> second, regardless of the actual Kinesis throughput. This is because the
> ShardReadersPool readLoop() method is polling getRecords() as fast as
> possible.
> From the KDS documentation:
> {quote}Each shard can support up to five read transactions per second.
> {quote}
> and
> {quote}For best results, sleep for at least 1 second (1,000 milliseconds)
> between calls to getRecords to avoid exceeding the limit on getRecords
> frequency.
> {quote}
> [https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html]
> [https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-sdk.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8382) Add polling interval to KinesisIO.Read
[
https://issues.apache.org/jira/browse/BEAM-8382?focusedWorklogId=331566&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331566
]
ASF GitHub Bot logged work on BEAM-8382:
Author: ASF GitHub Bot
Created on: 21/Oct/19 18:16
Start Date: 21/Oct/19 18:16
Worklog Time Spent: 10m
Work Description: cmachgodaddy commented on issue #9765: [BEAM-8382] Add
polling interval to KinesisIO.Read
URL: https://github.com/apache/beam/pull/9765#issuecomment-544639586
@alexvanboxel , I think we all understand it right, from what you described
above we just say it in different ways ;-) . In simple words, each split, run
on each worker, will create a number of threads base off num of shards. That's
means if we have 10 splits, we will have 10 guys/reader read one shard (of
course, it will read all shards)? Hope you agree with me this point? Now, what
Amazon recommend is we should have one client read one shard. Here is what they
say:
> Typically, when you use the KCL, you should ensure that the number of
instances does not exceed the number of shards (except for failure standby
purposes). Each shard is processed by exactly one KCL worker and has exactly
one corresponding record processor, so you never need multiple instances to
process one shard. However, one worker can process any number of shards, so
it's fine if the number of shards exceeds the number of instances.
And, here is why they do that,
https://docs.aws.amazon.com/streams/latest/dev/kinesis-low-latency.html
But, don't argue that here they use KCL, so pls consider one KCL is one
KinesisClient that our split use to connect to Kinesis and read a shard. And
don't misunderstand the `one worker can read a number of shard`. Here in their
context, or the example in their doc, their worker is an application that run
on EC2 instance, and when that scale this application, it will loadbalance the
KCL, e.g. two EC2 will have two KCL, each read two shards (assume a stream has
4 shards).
The point is we don't want to have number readers greater number of shards
in one application (or one pipeline in our context)? Imagine, if we have 10
pipelines deployed in our runners, and which has parallelism of 10, then we
will have 10 x 10 readers reading one shard ? And this 10 x 10 readers are not
loadbalanced since we are not using KCL? (even with KCL, this number still
exceed num of shard?). I am sure we will get "throughput exception" in the log
:-)
That's why in my subscribing POC (for the enhanced-fan-out), I design it in
a way that each split read only one shard (1:1).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331566)
Time Spent: 3h 40m (was: 3.5h)
> Add polling interval to KinesisIO.Read
> --
>
> Key: BEAM-8382
> URL: https://issues.apache.org/jira/browse/BEAM-8382
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kinesis
>Affects Versions: 2.13.0, 2.14.0, 2.15.0
>Reporter: Jonothan Farr
>Assignee: Jonothan Farr
>Priority: Major
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> With the current implementation we are observing Kinesis throttling due to
> ReadProvisionedThroughputExceeded on the order of hundreds of times per
> second, regardless of the actual Kinesis throughput. This is because the
> ShardReadersPool readLoop() method is polling getRecords() as fast as
> possible.
> From the KDS documentation:
> {quote}Each shard can support up to five read transactions per second.
> {quote}
> and
> {quote}For best results, sleep for at least 1 second (1,000 milliseconds)
> between calls to getRecords to avoid exceeding the limit on getRecords
> frequency.
> {quote}
> [https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html]
> [https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-sdk.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5967) ProtoCoder doesn't support DynamicMessage
[
https://issues.apache.org/jira/browse/BEAM-5967?focusedWorklogId=331567&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331567
]
ASF GitHub Bot logged work on BEAM-5967:
Author: ASF GitHub Bot
Created on: 21/Oct/19 18:17
Start Date: 21/Oct/19 18:17
Worklog Time Spent: 10m
Work Description: pabloem commented on issue #8496: [BEAM-5967] Add
handling of DynamicMessage in ProtoCoder
URL: https://github.com/apache/beam/pull/8496#issuecomment-544641391
> Can this be merged before the 2.17 cut (scheduled for Wednesday, October
23)?
@reuvenlax
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331567)
Time Spent: 6h 20m (was: 6h 10m)
> ProtoCoder doesn't support DynamicMessage
> -
>
> Key: BEAM-5967
> URL: https://issues.apache.org/jira/browse/BEAM-5967
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Affects Versions: 2.8.0
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Major
> Fix For: 2.17.0
>
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> The ProtoCoder does make some assumptions about static messages being
> available. The DynamicMessage doesn't have some of them, mainly because the
> proto schema is defined at runtime and not at compile time.
> Does it make sense to make a special coder for DynamicMessage or build it
> into the normal ProtoCoder.
> Here is an example of the assumtion being made in the current Codec:
> {code:java}
> try {
> @SuppressWarnings("unchecked")
> T protoMessageInstance = (T)
> protoMessageClass.getMethod("getDefaultInstance").invoke(null);
> @SuppressWarnings("unchecked")
> Parser tParser = (Parser) protoMessageInstance.getParserForType();
> memoizedParser = tParser;
> } catch (IllegalAccessException | InvocationTargetException |
> NoSuchMethodException e) {
> throw new IllegalArgumentException(e);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8382) Add polling interval to KinesisIO.Read
[
https://issues.apache.org/jira/browse/BEAM-8382?focusedWorklogId=331568&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331568
]
ASF GitHub Bot logged work on BEAM-8382:
Author: ASF GitHub Bot
Created on: 21/Oct/19 18:18
Start Date: 21/Oct/19 18:18
Worklog Time Spent: 10m
Work Description: alexvanboxel commented on issue #9765: [BEAM-8382] Add
polling interval to KinesisIO.Read
URL: https://github.com/apache/beam/pull/9765#issuecomment-544641514
> @alexvanboxel , I think we all understand it right, from what you
described above we just say it in different ways ;-) . In simple words, each
split, run on each worker, will create a number of threads
I think you're talking about another Alex, probably @aromanenko-dev :-)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331568)
Time Spent: 3h 50m (was: 3h 40m)
> Add polling interval to KinesisIO.Read
> --
>
> Key: BEAM-8382
> URL: https://issues.apache.org/jira/browse/BEAM-8382
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kinesis
>Affects Versions: 2.13.0, 2.14.0, 2.15.0
>Reporter: Jonothan Farr
>Assignee: Jonothan Farr
>Priority: Major
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> With the current implementation we are observing Kinesis throttling due to
> ReadProvisionedThroughputExceeded on the order of hundreds of times per
> second, regardless of the actual Kinesis throughput. This is because the
> ShardReadersPool readLoop() method is polling getRecords() as fast as
> possible.
> From the KDS documentation:
> {quote}Each shard can support up to five read transactions per second.
> {quote}
> and
> {quote}For best results, sleep for at least 1 second (1,000 milliseconds)
> between calls to getRecords to avoid exceeding the limit on getRecords
> frequency.
> {quote}
> [https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html]
> [https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-sdk.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
