[
https://issues.apache.org/jira/browse/HIVE-16686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-16686:
------------------------------------
Release Note:
This introduces parsing of additional parameters that are not directly used by
hive, but are passed on to distcp when hive invokes it. We now introduce the
ability to use the hive command to do "set" commands to pass along cli
arguments to distcp.
Any parameter set as "set distcp.options.blah=''" will result in an extra
"-blah" argument going into distcp, as well as any parameter set as "set
distcp.options.foo='bar'" will result in an extra "-foo bar" argument going to
distcp.
Currently, we always pass along "-update" and "-skipcrccheck" to distcp - that
is retained as defaults if no distcp.options.* params are found. If they are
found, then these options are not added by default, letting the user instead
provide an excplicit list.
Note that all of these properties affect how distcp runs when it is launched by
hive, but are not directly hive settings. Instead, hive will allow setting them
through the use of the "set" command.
was:
This introduces parsing of additional parameters that are not directly used by
hive, but are passed on to distcp when hive invokes it. We now introduce the
ability to use the hive command to do "set" commands to pass along cli
arguments to distcp.
Any parameter set as "set distcp.options.blah=''" will result in an extra
"-blah" argument going into distcp, as well as any parameter set as "set
distcp.options.foo='bar'" will result in an extra "-foo bar" argument going to
distcp.
Currently, we always pass along "-update" and "-skipcrccheck" to distcp - that
is retained as defaults if no distcp.options.* params are found. If they are
found, then these options are not added by default, letting the user instead
provide an excplicit list.
In addition, one new special option parameter, "distcp.option.privilegedUser"
is being added as a special option that is not passed along to distCp. Instead,
this option is used to make sure that hive will run distcp inside a
impersonation context as that specified user, if this parameter is specified,
and the user being impersonated is different from the current user. This,
however, will require that the user have impersonation proxy
privileges(something that a HS2 instance typically will have, but not a regular
end-user).
Note that all of these properties affect how distcp runs when it is launched by
hive, but are not directly hive settings. Instead, hive will allow setting them
through the use of the "set" command.
> repli invocations of distcp needs additional handling
> -----------------------------------------------------
>
> Key: HIVE-16686
> URL: https://issues.apache.org/jira/browse/HIVE-16686
> Project: Hive
> Issue Type: Sub-task
> Components: repl
> Reporter: Sushanth Sowmyan
> Assignee: Sushanth Sowmyan
> Labels: TODOC3.0
> Attachments: HIVE-16686.1.patch, HIVE-16686.2.patch
>
>
> When REPL LOAD invokes distcp, there needs to be a way for the user invoking
> REPL LOAD to pass on arguments to distcp. In addition, there is sometimes a
> need for distcp to be invoked from within an impersonated context, such as
> running as user "hdfs", asking distcp to preserve ownerships of individual
> files.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
