2. *Apache mirrors are inconvenient to use.* When you download
something from an Apache mirror, you get a link like this one
<https://www.apache.org/dyn/closer.lua/spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz>.
Instead of automatically redirecting you to your download, though,
you need to process the results you get back
<https://github.com/nchammas/flintrock/blob/67bf84a1b7cfa1c276cf57ecd8a0b27613ad2698/flintrock/scripts/download-hadoop.py#L21-L42>
to find your download target. And you need to handle the high
download failure rate, since sometimes the mirror you get doesn’t
have the file it claims to have.
It's not a full answer, just a note:
You can also use mirrors.cgi instead of parsing the json from closer.lua:
https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz
(Unfortunatelly it doesn't check the availibility of the file. If it's
moved to the archive you will be redirected to a 404)
Marton
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
