shane knapp created SPARK-3745: ---------------------------------- Summary: curl on maven search repo apache rat url returns search status, not jar file Key: SPARK-3745 URL: https://issues.apache.org/jira/browse/SPARK-3745 Project: Spark Issue Type: Bug Components: Build Environment: centos 6.5 Reporter: shane knapp
in spark/dev/check-license, there are four attempts to download the apache rat jar from maven: URL1="http://search.maven.org/remotecontent?filepath=org/apache/rat/apache-rat/${RAT_VERSION}/apache-rat-${RAT_VERSION}.jar" URL2="http://repo1.maven.org/maven2/org/apache/rat/apache-rat/${RAT_VERSION}/apache-rat-${RAT_VERSION}.jar" *snip* if hash curl 2>/dev/null; then (curl --progress-bar ${URL1} > "$JAR_DL" || curl --progress-bar ${URL2} > "$JAR_DL") && mv "$JAR_DL" "$JAR" elif hash wget 2>/dev/null; then (wget --progress=bar ${URL1} -O "$JAR_DL" || wget --progress=bar ${URL2} -O "$JAR_DL") && mv "$JAR_DL" "$JAR" the first attempt is on the search repo via curl, which returns a "YEP! WE FOUND IT!" html blob: [root@test01 sknapp]# curl --progress-bar http://search.maven.org/remotecontent?filepath=org/apache/rat/apache-rat/0.10/apache-rat-0.10.jar > wtf.part ######################################################################## 100.0% [root@test01 sknapp]# cat wtf.part <html> <head><title>302 Found</title></head> <body bgcolor="white"> <center><h1>302 Found</h1></center> <hr><center>nginx/0.8.55</center> </body> </html> this is failing to DL for EVERY time the test is run. i've run curl on the 2nd url, which points at the repo itself and it successfully downloads. wget does the correct thing for both URLs. there is also no error checking on the downloaded file, short of file existence. potential fixes, in no particular order: 1) run unzip -tq ${$JAR}, check for 0 exist status to ensure it's a compressed archive 2) run wget before curl 3) only run curl on the 2nd URL (pointing directly to the repo) -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org