[
https://issues.apache.org/jira/browse/SPARK-7481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14534174#comment-14534174
]
Steve Loughran commented on SPARK-7481:
---------------------------------------
This doesn't contain any endorsement of the use of s3a in Hadoop 2.6; see
HADOOP-11571
I'm not planning to add any tests for this, but its something to consider for
regression testing all the object stores —the tests just need to:
* be skipped if there's no credentials
* make a best effort to stop anyone accidentally checking in their credentials
* work on deskop/jenkins rather than just on cloud.
* not run up massive bills
* not take forever
AWS publishes some free-to-read datasets, such as [this
one|http://datasets.elasticmapreduce.s3.amazonaws.com/] which won't need
credentials, work remote and don't ring up bills for the read part of the
process, but would take a long time to complete on a single executor.
> Add Hadoop 2.6+ profile to pull in object store FS accessors
> ------------------------------------------------------------
>
> Key: SPARK-7481
> URL: https://issues.apache.org/jira/browse/SPARK-7481
> Project: Spark
> Issue Type: Improvement
> Components: Build
> Affects Versions: 1.3.1
> Reporter: Steve Loughran
>
> To keep the s3n classpath right, to add s3a, swift & azure, the dependencies
> of spark in a 2.6+ profile need to add the relevant object store packages
> (hadoop-aws, hadoop-openstack, hadoop-azure)
> this adds more stuff to the client bundle, but will mean a single spark
> package can talk to all of the stores.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
