[
https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16767077#comment-16767077
]
Steve Loughran commented on SPARK-23534:
----------------------------------------
bq. am curious to know if hadoop3 offers much performance benefit
if you are using S3 as the destination of work you get an output committer
which is O(files), not O(data), and can cope with an inconsistent store
(HADOOP-13786). Not sure of what else you can point to and say "tangible
speedup", though can point to stuff and and say 'tangible functionality
improvement"
with Hadoop 3.2 spark can generate delegation tokens for an S3 filesystem
during spark-submit (HADOOP-14556), and include them in the Yarn app launch.
This lets you deploy a cluster in EC2 with the VMs deployed in an IAM role with
lower privileges than you: a generated session login and your encryption
secrets will come with the job. This is very slick. And if you ask for role
delegation tokens then the generated token is limited to the specific s3 bucket
and DDB table you are working with. Video of distcp in action:
https://www.youtube.com/watch?v=rpyLkDEzIxI
Also ships with the abfs:// connector to Azure Datalake Gen 2 storage;
Microsoft's latest iteration of Azure storage.
> Spark run on Hadoop 3.0.0
> -------------------------
>
> Key: SPARK-23534
> URL: https://issues.apache.org/jira/browse/SPARK-23534
> Project: Spark
> Issue Type: Improvement
> Components: Build
> Affects Versions: 2.3.0
> Reporter: Saisai Shao
> Priority: Major
>
> Major Hadoop vendors already/will step in Hadoop 3.0. So we should also make
> sure Spark can run with Hadoop 3.0. This Jira tracks the work to make Spark
> run on Hadoop 3.0.
> The work includes:
> # Add a Hadoop 3.0.0 new profile to make Spark build-able with Hadoop 3.0.
> # Test to see if there's dependency issues with Hadoop 3.0.
> # Investigating the feasibility to use shaded client jars (HADOOP-11804).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
