[
https://issues.apache.org/jira/browse/SPARK-21514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16865228#comment-16865228
]
Noritaka Sekiyama commented on SPARK-21514:
-------------------------------------------
To move data from S3 (s3a) to HDFS, there is a problem.
Current implementation of Hive 1.2 does not support data movement across
different file systems (Hive 2.0 supports it).
If we try to implement this without Hive version upgrade, it means we need to
backport some implementation from Hive 2.0. When I tried, the patch included so
much diff.
It would be better to upgrade Hive version at first, then I can submit the
patch without backporting.
> Hive has updated with new support for S3 and InsertIntoHiveTable.scala should
> update also
> -----------------------------------------------------------------------------------------
>
> Key: SPARK-21514
> URL: https://issues.apache.org/jira/browse/SPARK-21514
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Javier Ros
> Priority: Major
>
> Hive has updated adding new parameters to optimize the usage of S3, now you
> can avoid the usage of S3 as the stagingdir using the parameters
> hive.blobstore.supported.schemes & hive.blobstore.optimizations.enabled.
> The InsertIntoHiveTable.scala file should be updated with the same
> improvement to match the behavior of Hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
