Charles Allen created MESOS-7621: ------------------------------------ Summary: Fetcher does not handle content length in redirects Key: MESOS-7621 URL: https://issues.apache.org/jira/browse/MESOS-7621 Project: Mesos Issue Type: Bug Components: fetcher Affects Versions: 1.2.0 Reporter: Charles Allen
{code} $ curl -L -v -O -s http://HOSTNAME_REDACTED/PATH_REDACTED.tar.gz * Trying 172.17.4.10... * Connected to HOSTNAME_REDACTED (172.17.4.10) port 80 (#0) > GET /PATH_REDACTED.tar.gz HTTP/1.1 > Host: HOSTNAME_REDACTED > User-Agent: curl/7.43.0 > Accept: */* > < HTTP/1.1 302 FOUND < Server: nginx/1.4.6 (Ubuntu) < Date: Mon, 05 Jun 2017 17:58:04 GMT < Content-Type: text/html; charset=utf-8 < Content-Length: 1947 < Connection: keep-alive < Location: https://BUCKET_REDACTED.s3.amazonaws.com:443/PATH_REDACTED?Signature=REDACTED%3D&Expires=1496689084&AWSAccessKeyId=KEY_REDACTED&x-amz-security-token=TOKEN_REDACTED%3D < * Ignoring the response-body { [309 bytes data] * Connection #0 to host HOSTNAME_REDACTED left intact * Issue another request to this URL: 'https://BUCKET_REDACTED.s3.amazonaws.com:443/PATH_REDACTED.tar.gz?Signature=SIGNATURE_REDACTED%3D&Expires=1496689084&AWSAccessKeyId=KEY_REDACTED&x-amz-security-token=TOKEN_REDACTED%3D' * Trying 54.231.40.75... * Connected to BUCKET_REDACTED.s3.amazonaws.com (54.231.40.75) port 443 (#1) * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: *.s3.amazonaws.com * Server certificate: DigiCert Baltimore CA-2 G2 * Server certificate: Baltimore CyberTrust Root > GET > /PATH_REDACTED.tar.gz?Signature=REDACTED&Expires=1496689084&AWSAccessKeyId=KEY_REDACTED&x-amz-security-token=TOKEN_REDACTED%3D > HTTP/1.1 > Host: BUCKET_REDACTED.s3.amazonaws.com > User-Agent: curl/7.43.0 > Accept: */* > < HTTP/1.1 200 OK < x-amz-id-2: ID_REDACTED= < x-amz-request-id: REQUEST_ID_REDACTED < Date: Mon, 05 Jun 2017 17:58:07 GMT < Last-Modified: Thu, 01 Jun 2017 03:04:49 GMT < ETag: "ETAG_REDACTED" < Accept-Ranges: bytes < Content-Type: application/x-tar < Content-Length: 208245664 < Server: AmazonS3 < { [16360 bytes data] {code} We have a micro-service which signs temporary urls for services which can't speak natively with S3. The above is an example download using {{curl}}. But when using the mesos fetcher the agent logs contain the following information: {code} fetcher.cpp:479] Reverting to fetching directly into the sandbox for 'http://HOST_REDACTED/PATH_REDACTED.tar.gz', due to failure to fetch through the cache, with error: Could not determine size of cache file for 'USER_REDACTED@http://HOST_REDACTED/PATH_REDACTED.tar.gz' with error: No URL content-length available {code} Any idea why this error would occur? -- This message was sent by Atlassian JIRA (v6.3.15#6346)