Re: Hadoop 2.8.0: Job console output suggesting non-existent rmserver 8088:proxy URI
On 9 September 2017 at 05:17, Ravi Prakash wrote: > I'm not sure my reply will be entirely helpful, but here goes. It sheds more light on things than I previously understood, Ravi, so cheers > The ResourceManager either proxies your request to the ApplicationMaster (if > the application is running), or (once the application is finished) serves it > itself if the job is in the "cache" (usually the last 1 applications) or > redirects to the MapReduce JHS if its a MapReduce job. That suggests that I don't have the ApplicationMaster(s) setup correctly (or possibly at all!) or that the caching setup is wrong because the ResourceManager and JobHistoryServer clearly both has the Job Info once the Jobs have ended, as I listed before > > http://rmserver.ecs.vuw.ac.nz:8088/cluster/app/application_1234567890123_4567/ > > Similarly, over on the Job History Server, we can get to a page, > related to the job > > http://jhserver.ecs.vuw.ac.nz:19888/jobhistory/job/job_1234567890123_4567/ however, trying to access those, though the "proxy channel" http://rmserver.ecs.vuw.ac.nz:8088/proxy/ URI doesn't take anyone anywhere. Thanks again for the insight: I think know where I need to look now, Kevin - To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org For additional commands, e-mail: user-h...@hadoop.apache.org
Hadoop 2.8.0: Job console output suggesting non-existent rmserver 8088:proxy URI
Hi again, my attempts to Kerberise our Hadoop instance seem to have things working OK, although one of the users has reported the following issue: The console output, from a running job, suggests following a link to the RM server's WebGUI, akin to http://rmserver.ecs.vuw.ac.nz:8088/proxy/application_1234567890123_4567/ but that URI doesn't appear to be being served by the RM server's WebGUI. However, starting from the RM server's WebGUI "About the Cluster" page's Applications view: http://rmserver.ecs.vuw.ac.nz:8088/cluster/apps we can get to the following page, related to the job http://rmserver.ecs.vuw.ac.nz:8088/cluster/app/application_1234567890123_4567/ Similarly, over on the Job History Server, we can get to a page, related to the job http://jhserver.ecs.vuw.ac.nz:19888/jobhistory/job/job_1234567890123_4567/ What, if anything, is likely to be missing from the configuration, that produces the "8088/proxy" path in the URIs that the console output presents ? Kevin --- Kevin M. Buckley eScience Consultant School of Engineering and Computer Science Victoria University of Wellington New Zealand - To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org For additional commands, e-mail: user-h...@hadoop.apache.org
Hadoop 2.8.0: Use of container-executor.cfg to restrict access to MapReduce jobs
Hi again early on in my attempts to Kerberise our Hadoop instance, I had seen an error message that suggested I needed to add a list of users who could run jobs into the last line of Hadoop's container-executor.cfg for which the default content is yarn.nodemanager.linux-container-executor.group=#configured value of yarn.nodemanager.linux-container-executor.group banned.users=#comma separated list of users who can not run applications min.user.id=1000#Prevent other super-users allowed.system.users=##comma separated list of system users who CAN run applications and after I had dropped the min.user.id to allow for the yarn user in our systems to run jobs AND added a list of users higher than that, those other users were able to run jobs. I now came to test out removing a user from the "allowed" list and I can't seem to prevent that user from running MapReduce jobs, no matter which of the various daemons I stop and start, including shutting down and restarting the whole thing. Should I be reading that allowed.system.users= list to be a list of UIDs from BELOW the min.user.id= list, rather than an actual "only allow users in the list" to run jobs list ? Clealry, one can't run jobs if one doesn't have access to directories to put data into, so that's a kind of "job control" ACL of itself but I was hoping that the underlying HDFS might contain a wider set of users than those allowed to run jobs at any given time, in which case, altering the ability via the container-executor.cfg list seemed a simple way to achieve that. Any clues/insight welcome, Kevin --- Kevin M. Buckley eScience Consultant School of Engineering and Computer Science Victoria University of Wellington New Zealand - To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org For additional commands, e-mail: user-h...@hadoop.apache.org
Re: Kerberised JobHistory Server not starting: User jhs trying to create the /mr-history/done directory
On 25 July 2017 at 03:21, Erik Krogen wrote: > Hey Kevin, > > Sorry, I missed your point about using auth_to_local. You're right that you > should be able to use that for what you're trying to achieve. I think it's > just that your rule is wrong; I believe it should be: > > RULE:[2:$1@$0](jhs/.*@ECS.VUW.AC.NZ)s/.*/mapred/ > > HTH If I ever get a chance to retrace my steps, Erik, then I'll give it a go. As things stand I went with the altered username in the keytab and things seem to be working, although I have one other issue that I'm about to start a new thread for. Cheers again, Kevin - To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org For additional commands, e-mail: user-h...@hadoop.apache.org
Re: Kerberised JobHistory Server not starting: User jhs trying to create the /mr-history/done directory
On 21 July 2017 at 13:25, Kevin Buckley
wrote:
> On 21 July 2017 at 04:04, Erik Krogen wrote:
>> Hi Kevin,
>>
>> Since you are using the "jhs" keytab with principal "jhs/_h...@realm.tld",
>> the JHS is authenticating itself as the jhs user (which is the actual
>> important part, rather than the user the process is running as). If you want
>> it to be the "mapred" user, you should change the keytab/principal you use
>> (mapred.jobhistory.{principal,keytab}).
>
> I'll certainly give that a go Erik, however, the way I read the
>
>>> The hadoop-2.8.0 docs SecureMode page also suggests that one would need to
>>> play around with the
>>>
>>> hadoop.security.auth_to_local
>
> bits suggested to me that if you set things up such that
>
> ===
> $ hadoop org.apache.hadoop.security.HadoopKerberosName
> jhs/co246a-9.ecs.vuw.ac...@ecs.vuw.ac.nz
> 17/07/20 17:42:50 INFO util.KerberosName: Non-simple name
> mapred/co246a-9.ecs.vuw.ac...@ecs.vuw.ac.nz after auth_to_local rule
> RULE:[2:$1/$2@$0](jhs/.*)s/jhs/mapred/
> Name: jhs/co246a-9.ecs.vuw.ac...@ecs.vuw.ac.nz to
> mapred/co246a-9.ecs.vuw.ac...@ecs.vuw.ac.nz
>
>
> (or even used a rule that just mapped the principal to a simple "mapred"
> because I tried that too !) told you it was remapping the user, then it would
> remap for all instances of the user, within the Hadoop instance..
>
> Let's see.
OK,
so it would appear that despite the Hadoop docs appearing to suggest that
you only need the three usernames, 'hdfs', 'yarn' and 'mapred'. if you do use
the principal from the docs, which has the jhs component, then even if you
do try to map users using 'hadoop.security.auth_to_local', your JobHistory
server will start up, inside Hadoop running as a 'jhs' user.
That would seem to be a bit of a trap for the unaware/unwary that the docs
could easily improve upon ?
Thanks again for the pointer to the correct interpreation of the docs, Erik,
Kevin
-
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org
Re: Kerberised JobHistory Server not starting: User jhs trying to create the /mr-history/done directory
On 21 July 2017 at 04:04, Erik Krogen wrote:
> Hi Kevin,
>
> Since you are using the "jhs" keytab with principal "jhs/_h...@realm.tld",
> the JHS is authenticating itself as the jhs user (which is the actual
> important part, rather than the user the process is running as). If you want
> it to be the "mapred" user, you should change the keytab/principal you use
> (mapred.jobhistory.{principal,keytab}).
I'll certainly give that a go Erik, however, the way I read the
>> The hadoop-2.8.0 docs SecureMode page also suggests that one would need to
>> play around with the
>>
>> hadoop.security.auth_to_local
bits suggested to me that if you set things up such that
===
$ hadoop org.apache.hadoop.security.HadoopKerberosName
jhs/co246a-9.ecs.vuw.ac...@ecs.vuw.ac.nz
17/07/20 17:42:50 INFO util.KerberosName: Non-simple name
mapred/co246a-9.ecs.vuw.ac...@ecs.vuw.ac.nz after auth_to_local rule
RULE:[2:$1/$2@$0](jhs/.*)s/jhs/mapred/
Name: jhs/co246a-9.ecs.vuw.ac...@ecs.vuw.ac.nz to
mapred/co246a-9.ecs.vuw.ac...@ecs.vuw.ac.nz
(or even used a rule that just mapped the principal to a simple "mapred"
because I tried that too !) told you it was remapping the user, then it would
remap for all instances of the user, within the Hadoop instance..
Let's see.
Cheers again for the feedback.
-
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org
Kerberised JobHistory Server not starting: User jhs trying to create the /mr-history/done directory
My Hadoop 2.8.0's /mr-history/done directory is owned by the mapred user, who is in the hadoop group, and the directory has the pemissions /mr-history":mapred:hadoop:drwxrwx--- If I run the Hadoop instance without any Kerberos config, and fire up the JobHistory server as the mapred user, everything works. If I flip over to a Kerberised environment, the NameNode and DataNodes, running as the 'hdfs' user, and the Resource and and Node Managers, running as the 'yarn' user, all start up OK and their respective web exposure can be used. When I try to start up the JobHistory server however /bin/su mapred -c '/local/Hadoop/hadoop-2.8.0/sbin/mr-jobhistory-daemon.sh --config /local/Hadoop/hadoop-2.8.0/etc/hadoop/ start historyserver I get a message in the logs telling me that, rather than the mapred user doing things, a user 'jhs' is trying to do stuff, vis 2017-07-20 18:15:09,667 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer: registered UNIX signal handlers for [TERM, HUP, INT] 2017-07-20 18:15:10,062 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user jhs/co246a-9.ecs.vuw.ac...@ecs.vuw.ac.nz using keytab file /local/Hadoop/krb/jhs.service.keytab 2017-07-20 18:15:10,107 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2017-07-20 18:15:10,142 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s). 2017-07-20 18:15:10,142 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobHistoryServer metrics system started 2017-07-20 18:15:10,145 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: JobHistory Init 2017-07-20 18:15:10,411 INFO org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://co246a-a.ecs.vuw.ac.nz:9000] 2017-07-20 18:15:10,518 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error creating done directory: [hdfs://co246a-a.ecs.vuw.ac.nz:9000/mr-history/done] org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error creating done directory: [hdfs://co246a-a.ecs.vuw.ac.nz:9000/mr-history/done] at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.tryCreatingHistoryDirs(HistoryFileManager.java:639) at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.createHistoryDirs(HistoryFileManager.java:585) at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.serviceInit(HistoryFileManager.java:550) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.mapreduce.v2.hs.JobHistory.serviceInit(JobHistory.java:95) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceInit(JobHistoryServer.java:151) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.launchJobHistoryServer(JobHistoryServer.java:231) at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.main(JobHistoryServer.java:241) Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=jhs, access=EXECUTE, inode="/mr-history":mapred:hadoop:drwxrwx--- But where has the jhs user come from ? Doesn't appear to be set anywhere in any of the config files. According to the hadoop-2.8.0 docs SecureMode page, https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-common/SecureMode.html = MapReduce JobHistory Server The MapReduce JobHistory Server keytab file, on that host, should look like the following: $ klist -e -k -t /etc/security/keytab/jhs.service.keytab Keytab name: FILE:/etc/security/keytab/jhs.service.keytab KVNO Timestamp Principal 4 07/18/11 21:08:09 jhs/full.qualified.domain.n...@realm.tld (AES-256 CTS mode with 96-bit SHA-1 HMAC) 4 07/18/11 21:08:09 jhs/full.qualified.domain.n...@realm.tld (AES-128 CTS mode with 96-bit SHA-1 HMAC) 4 07/18/11 21:08:09 jhs/full.qualified.domain.n...@realm.tld (ArcFour with HMAC/md5) 4 07/18/11 21:08:09 host/full.qualified.domain.n...@realm.tld (AES-256 CTS mode with 96-bit SHA-1 HMAC) 4 07/18/11 21:08:09 host/full.qualified.domain.n...@realm.tld (AES-128 CTS mode with 96-bit SHA-1 HMAC) 4 07/18/11 21:08:09 host/full.qualified.domain.n...@realm.tld (ArcFour with HMAC/md5) = and mine does. The hadoop-2.8.0 docs SecureMode page also suggests that one would need to play around with the hadoop.security.auth_to_local config value, but I haven't had to do that for the nn, dn, rm or nm keytabs. So is there something special about the jhs user ? Or perhap
v2.8.0: Setting PID dir EnvVars: libexec/thing-config.sh or etc/hadoop/thing-env.sh ?
I've noted in
https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-common/ClusterSetup.html
that it says
8<8<8<8<8<
See etc/hadoop/hadoop-env.sh for other examples.
Other useful configuration parameters that you can customize include:
HADOOP_PID_DIR - The directory where the daemons’ process id files
are stored.
HADOOP_LOG_DIR - The directory where the daemons’ log files are
stored. Log files are automatically created if
they don’t exist.
HADOOP_HEAPSIZE / YARN_HEAPSIZE - The maximum amount of heapsize
to use, in MB e.g. if the varibale is set to 1000
the heap will be set to 1000MB. This is used to
configure the heap size for the daemon. By
default, the value is 1000. If you want to
configure the values separately for each deamon
you can use.
In most cases, you should specify the HADOOP_PID_DIR and HADOOP_LOG_DIR
directories such that they can only be written to by the users that
are going to run the hadoop daemons. Otherwise there is the potential
for a symlink attack.
8<8<8<8<8<
and I have recently had a need to move my PID-files from their default,
/tmp, location, so as to avoid an over-aggressive /tmp cleanup.
So, if I am in
/path/to/hadoop-2.8.0/etc/hadoop
I can see
$ grep PID *
hadoop-env.cmd:set HADOOP_PID_DIR=%HADOOP_PID_DIR%
hadoop-env.cmd:set HADOOP_SECURE_DN_PID_DIR=%HADOOP_PID_DIR%
hadoop-env.sh:export HADOOP_PID_DIR=${HADOOP_PID_DIR}
hadoop-env.sh:export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR}
mapred-env.sh:#export HADOOP_MAPRED_PID_DIR= # The pid files are
stored. /tmp by default.
$
so clearly PID_DIR values could be overriden there, although I note that the
yarn-env.sh
file, unlike its "hadoop" and "mapred" bretheren, doesn't contain
any PID_DIR default lines, nor even commented suggestion lines for
YARN_PID_DIR
which is used in, for exmaple,
hadoop-2.8.0/sbin/yarn-daemon.sh
However, I've also noted that many of the Hadoop sbin scripts
will source a related file from
hadoop-2.8.0/libexec/
so for example, a desired value for the env var
YARN_PID_DIR
might thus be set in either (or both ?) of
hadoop-2.8.0/etc/hadoop/yarn-env.sh
hadoop-2.8.0/libexec/yarn-config.sh
and similarly,
HADOOP_PID_DIR
might be set in either (or both ?) of
hadoop-2.8.0/libexec/hadoop-config.sh
hadoop-2.8.0/etc/hadoop/hadoop-env.sh
So,
is there a "to be preferred" file choice, between those two,
within which to set certain classes of EnvVars ?
Any info/pointers welcome,
Kevin
---
Kevin M. Buckley
eScience Consultant
School of Engineering and Computer Science
Victoria University of Wellington
New Zealand
-
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org
