[ 
https://issues.apache.org/jira/browse/BEAM-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16480754#comment-16480754
 ] 

Dariusz Aniszewski commented on BEAM-4041:
------------------------------------------

Since those issues are still appearing, and [this PR to 
PerfKit|https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1641] 
was merged, I submitted [PR 5425|https://github.com/apache/beam/pull/5425] to 
increase retry timeout from 3 to 6 minutes - please check it and merge it if it 
makes sense. If increasing timeout is pointless please discard the PR.

> Performance tests fail due to kubernetes load balancer problems
> ---------------------------------------------------------------
>
>                 Key: BEAM-4041
>                 URL: https://issues.apache.org/jira/browse/BEAM-4041
>             Project: Beam
>          Issue Type: Bug
>          Components: testing
>            Reporter: Łukasz Gajowy
>            Assignee: Jason Kuster
>            Priority: Major
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Recently, as we added more IOITs to be run on jenkins using kubernetes, some 
> of them started to fail randomly, because they couldn't retrieve LoadBalancer 
> address. Normally obtaining the address took about one minute. Perfkit waits 
> for the address (actively checking for it) for 3 minutes. This should be 
> enough for getting the address, yet it recently started to exceed the 3 
> minutes limit. I also noticed that this error didn't happen when there were 
> fewer tests.
> Example logs:
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_Compressed_TextIOIT_HDFS/31/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to