[ https://issues.apache.org/jira/browse/BEAM-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16480754#comment-16480754 ]
Dariusz Aniszewski commented on BEAM-4041: ------------------------------------------ Since those issues are still appearing, and [this PR to PerfKit|https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1641] was merged, I submitted [PR 5425|https://github.com/apache/beam/pull/5425] to increase retry timeout from 3 to 6 minutes - please check it and merge it if it makes sense. If increasing timeout is pointless please discard the PR. > Performance tests fail due to kubernetes load balancer problems > --------------------------------------------------------------- > > Key: BEAM-4041 > URL: https://issues.apache.org/jira/browse/BEAM-4041 > Project: Beam > Issue Type: Bug > Components: testing > Reporter: Łukasz Gajowy > Assignee: Jason Kuster > Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Recently, as we added more IOITs to be run on jenkins using kubernetes, some > of them started to fail randomly, because they couldn't retrieve LoadBalancer > address. Normally obtaining the address took about one minute. Perfkit waits > for the address (actively checking for it) for 3 minutes. This should be > enough for getting the address, yet it recently started to exceed the 3 > minutes limit. I also noticed that this error didn't happen when there were > fewer tests. > Example logs: > https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_Compressed_TextIOIT_HDFS/31/console -- This message was sent by Atlassian JIRA (v7.6.3#76005)