[
https://issues.apache.org/jira/browse/CASSANDRA-21805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132375#comment-18132375
]
Dmitry Konstantinov commented on CASSANDRA-21805:
-------------------------------------------------
Observation: even with the retry and use download/dlcdn.apache.org we still
may fail for 6.0.0-alpha1. Because 6.0.0-alpha1 is old and only in archive +
downloading from archive can be just very slow (but without timeouts), so the
overall test timeout is happening.
So, we need to switch to 6.0.0-alpha2 (CASSANDRA-21803) and maybe even bake it
into docker image as we do for usual releases.
{code}
55 splits downloaded 6.0-alpha1, every one from http://archive.apache.org. 52
finished and 3 timed out.
┌───────────────────────┬──────────┬───────────┬───────────────────────┐
│ │ KiB/s │ Mbit/s │ time for 75.5 MiB │
├───────────────────────┼──────────┼───────────┼───────────────────────┤
│ slowest that finished │ 97 │ 0.8 │ 797s │
├───────────────────────┼──────────┼───────────┼───────────────────────┤
│ p10 │ 135 │ 1.1 │ │
├───────────────────────┼──────────┼───────────┼───────────────────────┤
│ median │ 237 │ 1.9 │ 327s │
├───────────────────────┼──────────┼───────────┼───────────────────────┤
│ p90 │ 369 │ 3.0 │ │
├───────────────────────┼──────────┼───────────┼───────────────────────┤
│ fastest │ 456 │ 3.7 │ 170s │
├───────────────────────┼──────────┼───────────┼───────────────────────┤
│ the 3 timeouts │ below 86 │ below 0.7 │ still running at 899s │
└───────────────────────┴──────────┴───────────┴───────────────────────┘
{code}
> CCM: download releases from dlcdn.apache.org before archive.apache.org, and
> time out stalled download
> -----------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21805
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21805
> Project: Apache Cassandra
> Issue Type: Improvement
> Components: Tool/CCM
> Reporter: Dmitry Konstantinov
> Assignee: Dmitry Konstantinov
> Priority: Normal
> Labels: green_ci
> Fix For: NA
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> ccm downloads release tarballs only from
> [http://archive.apache.org/dist/cassandra] and calls urlopen() without a
> timeout. archive.apache.org is a single throttled host. When many CI splits
> download from it at once, downloads slow down, and a stalled connection
> blocks until pytest kills the test.
> In Cassandra-trunk #2641 (
> [https://ci-cassandra.apache.org/job/Cassandra-trunk/2641]), every upgrade
> split downloaded 6.0-alpha1 from archive.apache.org. The median speed per
> split was 188 KB/s, 14 splits got stuck for more than 900s, and 20 upgrade
> tests from current_6_0_x failed with "Timeout >900.0s", most of them while
> still in repository.__download().
> Proposed change:
> * Try [https://dlcdn.apache.org|https://dlcdn.apache.org/], then
> [https://downloads.apache.org|https://downloads.apache.org/], then
> archive.apache.org. dlcdn and downloads only keep current releases; a 404
> moves on to the next mirror immediately. [repositories] cassandra in
> ~/.ccm/config still overrides the list.
> * Use a 45s socket timeout, so a stalled connection fails instead of hanging.
> * Retry each URL up to 3 times on timeouts, connection errors, truncated
> responses and HTTP 5xx/408/429. Do not retry other 4xx responses, certificate
> errors or [file://|file:///] URLs.
> To be used by dtests, the cassandra-test tag has to be moved after the merge.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
