[
https://issues.apache.org/jira/browse/BEAM-6883?focusedWorklogId=284092&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-284092
]
ASF GitHub Bot logged work on BEAM-6883:
----------------------------------------
Author: ASF GitHub Bot
Created on: 29/Jul/19 08:15
Start Date: 29/Jul/19 08:15
Worklog Time Spent: 10m
Work Description: RyanSkraba commented on pull request #9178: [BEAM-6883]
Improve long-running spark streaming test.
URL: https://github.com/apache/beam/pull/9178#discussion_r308103884
##########
File path:
runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/StreamingSourceMetricsTest.java
##########
@@ -47,11 +52,18 @@
@Test
@Category(StreamingTest.class)
public void testUnboundedSourceMetrics() {
- final long numElements = 1000;
+ final long minElements = 1000;
- pipeline.apply(
- // Use maxReadTime to force unbounded mode.
-
GenerateSequence.from(0).to(numElements).withMaxReadTime(Duration.standardDays(1)));
+ // Use a GenerateSequence for the UnboundedSequence, but push the
watermark to infinity at
+ // minElements to let the test pipeline cleanly shut it down. Shutdown
will occur shortly
+ // afterwards, but at least minElements will be reported in the metrics.
+ PCollection<Long> pc =
+ pipeline.apply(
+ GenerateSequence.from(1)
+ .withRate(minElements / 10, Duration.millis(500L))
+ .withTimestampFn(
+ t -> t < minElements ? Instant.now() :
BoundedWindow.TIMESTAMP_MAX_VALUE));
+ assertThat(pc.isBounded(), is(PCollection.IsBounded.UNBOUNDED));
Review comment:
Note: this assert would have failed with the old GenerateSequence
configuration.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 284092)
Time Spent: 20m (was: 10m)
> StreamingSourceMetricsTest takes too long to finish
> ---------------------------------------------------
>
> Key: BEAM-6883
> URL: https://issues.apache.org/jira/browse/BEAM-6883
> Project: Beam
> Issue Type: Test
> Components: runner-spark
> Affects Versions: 2.11.0
> Reporter: Ismaël Mejía
> Assignee: Ryan Skraba
> Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> This test is part of Spark's ValidatesRunner suite and it takes more than 10
> minutes to end.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
