[
https://issues.apache.org/jira/browse/BEAM-5758?focusedWorklogId=211802&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-211802
]
ASF GitHub Bot logged work on BEAM-5758:
----------------------------------------
Author: ASF GitHub Bot
Created on: 12/Mar/19 15:54
Start Date: 12/Mar/19 15:54
Worklog Time Spent: 10m
Work Description: lgajowy commented on pull request #8027: [BEAM-5758]
add fanout and reiteration to LoadTests
URL: https://github.com/apache/beam/pull/8027#discussion_r264747088
##########
File path: sdks/python/apache_beam/testing/load_tests/group_by_key_test.py
##########
@@ -166,18 +188,31 @@ def setUp(self):
raise ValueError('One or more of parameters for collecting metrics '
'are empty.')
+ class UngroupAndReiterate(beam.DoFn):
+ def process(self, element, *args, **kwargs):
+ key, value = element
+ for i in range(self.iterations):
+ for v in value:
+ if i == self.iterations - 1:
+ return (key, v)
+
def testGroupByKey(self):
- # pylint: disable=expression-not-assigned
- (self.pipeline
- | beam.io.Read(synthetic_pipeline.SyntheticSource(
- self.parseTestPipelineOptions()))
- | 'Measure time: Start' >> beam.ParDo(
- MeasureTime(self.metrics_namespace))
- | 'GroupByKey' >> beam.GroupByKey()
- | 'Ungroup' >> beam.FlatMap(
- lambda elm: [(elm[0], v) for v in elm[1]])
- | 'Measure time: End' >> beam.ParDo(MeasureTime(self.metrics_namespace))
- )
+ input = (self.pipeline
+ | beam.io.Read(synthetic_pipeline.SyntheticSource(
+ self.parseTestPipelineOptions()))
+ | 'Measure time: Start' >> beam.ParDo(
+ MeasureTime(self.metrics_namespace))
+ )
+
+ for branch in range(self.fanout):
+ # pylint: disable=expression-not-assigned
+ (input
+ | 'GroupByKey %i' % branch >> beam.GroupByKey()
+ | 'Ungroup %i' % branch >> beam.FlatMap(
Review comment:
It should use `UngroupAndReiterate` here, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 211802)
Time Spent: 2h 10m (was: 2h)
> Load tests for SyntheticSources in Python
> -----------------------------------------
>
> Key: BEAM-5758
> URL: https://issues.apache.org/jira/browse/BEAM-5758
> Project: Beam
> Issue Type: Test
> Components: testing
> Reporter: Kasia Kucharczyk
> Assignee: Kasia Kucharczyk
> Priority: Major
> Labels: triaged
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> For purpose of load testing the SyntheticSources there should be created
> tests with metrics which will be sent to BigQuery.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
