Re: Best way to pass configuration properties to MRv2 jobs
How true!! ;-)
Thanks,
Henning
On 18.04.2016 19:53, Dima Spivak wrote:
Probably better off asking on the Hadoop user mailing list (
u...@hadoop.apache.org) than the HBase one… :)
-Dima
On Mon, Apr 18, 2016 at 2:57 AM, Henning Blohm
wrote:
Hi,
in our Hadoop 2.6.0 cluster, we need to pass some properties to all Hadoop
processes so they can be referenced using ${...} syntax in configuration
files. This works reasonably well using HADOOP_NAMENODE_OPTS and the like.
For Map/Reduce jobs however, we need to speficy not only
mapred.child.java.opts
to pass system properties, in addition we need to set
yarn.app.mapreduce.am.command-opts
for anything that is referenced in Hadoop configuration files.
In the end however almost all the properties passed are available as
environment variables as well.
Hence my question:
* Is it possible to use reference environment variables in configuration
files directly?
* Does anybody know of a simpler way to make sure some system properties
are _always_ set for all Yarn processes?
Thanks,
Henning
Re: Best way to pass configuration properties to MRv2 jobs
Probably better off asking on the Hadoop user mailing list (
u...@hadoop.apache.org) than the HBase one… :)
-Dima
On Mon, Apr 18, 2016 at 2:57 AM, Henning Blohm
wrote:
> Hi,
>
> in our Hadoop 2.6.0 cluster, we need to pass some properties to all Hadoop
> processes so they can be referenced using ${...} syntax in configuration
> files. This works reasonably well using HADOOP_NAMENODE_OPTS and the like.
>
> For Map/Reduce jobs however, we need to speficy not only
>
> mapred.child.java.opts
>
> to pass system properties, in addition we need to set
>
> yarn.app.mapreduce.am.command-opts
>
> for anything that is referenced in Hadoop configuration files.
>
> In the end however almost all the properties passed are available as
> environment variables as well.
>
> Hence my question:
>
> * Is it possible to use reference environment variables in configuration
> files directly?
> * Does anybody know of a simpler way to make sure some system properties
> are _always_ set for all Yarn processes?
>
> Thanks,
> Henning
>
Best way to pass configuration properties to MRv2 jobs
Hi,
in our Hadoop 2.6.0 cluster, we need to pass some properties to all
Hadoop processes so they can be referenced using ${...} syntax in
configuration files. This works reasonably well using
HADOOP_NAMENODE_OPTS and the like.
For Map/Reduce jobs however, we need to speficy not only
mapred.child.java.opts
to pass system properties, in addition we need to set
yarn.app.mapreduce.am.command-opts
for anything that is referenced in Hadoop configuration files.
In the end however almost all the properties passed are available as
environment variables as well.
Hence my question:
* Is it possible to use reference environment variables in configuration
files directly?
* Does anybody know of a simpler way to make sure some system properties
are _always_ set for all Yarn processes?
Thanks,
Henning
