I think the download could use the Apache mirror, yeah. I don't know if there's a reason that it must though. What's good enough for releases is good enough for this purpose. People might not like the big download in the tests if it really came up as an issue we could find ways to cache it better locally. I brought it up more as a question than a problem to solve.
On Thu, Sep 14, 2017 at 5:02 PM Mark Hamstra <m...@clearstorydata.com> wrote: > The problem is that it's not really an "official" download link, but > rather just a supplemental convenience. While that may be ok when > distributing artifacts, it's more of a problem when actually building and > testing artifacts. In the latter case, the download should really only be > from an Apache mirror. > > On Thu, Sep 14, 2017 at 1:20 AM, Wenchen Fan <cloud0...@gmail.com> wrote: > >> That test case is trying to test the backward compatibility of >> `HiveExternalCatalog`. It downloads official Spark releases and creates >> tables with them, and then read these tables via the current Spark. >> >> About the download link, I just picked it from the Spark website, and >> this link is the default one when you choose "direct download". Do we have >> a better choice? >> >> On Thu, Sep 14, 2017 at 3:05 AM, Shivaram Venkataraman < >> shiva...@eecs.berkeley.edu> wrote: >> >>> Mark, I agree with your point on the risks of using Cloudfront while >>> building Spark. I was only trying to provide background on when we >>> started using Cloudfront. >>> >>> Personally, I don't have enough about context about the test case in >>> question (e.g. Why are we downloading Spark in a test case ?). >>> >>> Thanks >>> Shivaram >>> >>> On Wed, Sep 13, 2017 at 11:50 AM, Mark Hamstra <m...@clearstorydata.com> >>> wrote: >>> > Yeah, but that discussion and use case is a bit different -- providing >>> a >>> > different route to download the final released and approved artifacts >>> that >>> > were built using only acceptable artifacts and sources vs. building and >>> > checking prior to release using something that is not from an Apache >>> mirror. >>> > This new use case puts us in the position of approving spark artifacts >>> that >>> > weren't built entirely from canonical resources located in presumably >>> secure >>> > and monitored repositories. Incorporating something that is not >>> completely >>> > trusted or approved into the process of building something that we are >>> then >>> > going to approve as trusted is different from the prior use of >>> cloudfront. >>> > >>> > On Wed, Sep 13, 2017 at 10:26 AM, Shivaram Venkataraman >>> > <shiva...@eecs.berkeley.edu> wrote: >>> >> >>> >> The bucket comes from Cloudfront, a CDN thats part of AWS. There was a >>> >> bunch of discussion about this back in 2013 >>> >> >>> >> >>> https://lists.apache.org/thread.html/9a72ff7ce913dd85a6b112b1b2de536dcda74b28b050f70646aba0ac@1380147885@%3Cdev.spark.apache.org%3E >>> >> >>> >> Shivaram >>> >> >>> >> On Wed, Sep 13, 2017 at 9:30 AM, Sean Owen <so...@cloudera.com> >>> wrote: >>> >> > Not a big deal, but Mark noticed that this test now downloads Spark >>> >> > artifacts from the same 'direct download' link available on the >>> >> > downloads >>> >> > page: >>> >> > >>> >> > >>> >> > >>> https://github.com/apache/spark/blob/master/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala#L53 >>> >> > >>> >> > >>> https://d3kbcqa49mib13.cloudfront.net/spark-$version-bin-hadoop2.7.tgz >>> >> > >>> >> > I don't know of any particular problem with this, which is a >>> parallel >>> >> > download option in addition to the Apache mirrors. It's also the >>> >> > default. >>> >> > >>> >> > Does anyone know what this bucket is and if there's a strong reason >>> we >>> >> > can't >>> >> > just use mirrors? >>> >> >>> >> --------------------------------------------------------------------- >>> >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>> >> >>> > >>> >>> --------------------------------------------------------------------- >>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>> >>> >> >
