yarn REST api (controller for v1 not found)
Hi, Using HDP 2.0.6, Yarn 2.1 I am trying to access the REST api per the documentation here: http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/MapredAppMasterRest.html When I try to access this http://rm host:8088/ws/v1/cluster/app/application_1401899005478_2241 I get org.apache.hadoop.yarn.webapp.WebAppException: /v1/cluster/app/application_1401899005478_2241: controller for v1 not found However, I can access the page for that application through the web UI no problem, at http://rm host:8088/cluster/app/application_1401899005478_2241 Thanks in advance.
Re: yarn REST api (controller for v1 not found)
This is actually the link I was following: http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html On Thu, Jul 3, 2014 at 4:12 PM, Alex Nastetsky anastet...@spryinc.com wrote: Hi, Using HDP 2.0.6, Yarn 2.1 I am trying to access the REST api per the documentation here: http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/MapredAppMasterRest.html When I try to access this http://rm host:8088/ws/v1/cluster/app/application_1401899005478_2241 I get org.apache.hadoop.yarn.webapp.WebAppException: /v1/cluster/app/application_1401899005478_2241: controller for v1 not found However, I can access the page for that application through the web UI no problem, at http://rm host:8088/cluster/app/application_1401899005478_2241 Thanks in advance.
Re: Service Level Authorization
Juan, What kind of information are you looking for? The service level ACLs are for limiting which services can communicate under certain protocols, by username or user group. Perhaps you are looking for client level ACL, something like the MapReduce ACLs? https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html#Job+Authorization Alex. 2014-02-20 4:58 GMT-05:00 Juan Carlos jcfernan...@cediant.es: Where could I find some information about ACL? I only could find the available in http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/ServiceLevelAuth.html, which isn't so detailed. Regards Juan Carlos Fernández Rodríguez Consultor Tecnológico Telf: +34918105294 Móvil: +34639311788 CEDIANT Centro para el Desarrollo, Investigación y Aplicación de Nuevas Tecnologías HPC Business Solutions * AVISO LEGAL * Este mensaje es solamente para la persona a la que va dirigido. Puede contener información confidencial o legalmente protegida. No hay renuncia a la confidencialidad o privilegio por cualquier transmisión mala/errónea. Si usted ha recibido este mensaje por error,le rogamos que borre de su sistema inmediatamente el mensaje asi como todas sus copias, destruya todas las copias del mismo de su disco duro y notifique al remitente. No debe, directa o indirectamente, usar, revelar, distribuir, imprimir o copiar ninguna de las partes de este mensaje si no es usted el destinatario. Cualquier opinión expresada en este mensaje proviene del remitente, excepto cuando el mensaje establezca lo contrario y el remitente esté autorizado para establecer que dichas opiniones provienen de 'CEDIANT'. Nótese que el correo electrónico vía Internet no permite asegurar ni la confidencialidad de los mensajes que se transmiten ni la correcta recepción de los mismos. En el caso de que el destinatario de este mensaje no consintiera la utilización del correo electrónico vía Internet, rogamos lo ponga en nuestro conocimiento de manera inmediata. * DISCLAIMER * This message is intended exclusively for the named person. It may contain confidential, propietary or legally privileged information. No confidentiality or privilege is waived or lost by any mistransmission. If you receive this message in error, please immediately delete it and all copies of it from your system, destroy any hard copies of it an notify the sender. Your must not, directly or indirectly, use, disclose, distribute, print, or copy any part of this message if you are not the intended recipient. Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorised to state them to be the views of 'CEDIANT'. Please note that internet e-mail neither guarantees the confidentiality nor the proper receipt of the message sent. If the addressee of this message does not consent to the use of internet e-mail, please communicate it to us immediately.
Re: Service Level Authorization
If your test1 queue is under test queue, then you have to specify the path in the same way: yarn.scheduler.capacity.root.test.test1.acl_submit_applications (you are missing the test) Also, if your hadoop user is a member of user group hadoop, that is the default value of the mapreduce.cluster.administrators in mapred-site.xml. Users of that group can submit jobs to and administer all queues. On Thu, Feb 20, 2014 at 11:28 AM, Juan Carlos juc...@gmail.com wrote: Yes, that is what I'm looking for, but I couldn't find this information for hadoop 2.2.0. I saw mapreduce.cluster.acls.enabled it's now the parameter to use. But I don't know how to set my ACLs. I'm using capacity schedurler and I've created 3 new queues test (which is under root at the same level as default) and test1 and test2, which are under test. As I said, I enabled mapreduce.cluster.acls.enabled in mapred-site.xml and later added the parameter yarn.scheduler.capacity.root.test1.acl_submit_applications with value jcfernandez . If I submit a job to queue test1 with user hadoop, it allows it to run it. Which is my error? 2014-02-20 16:41 GMT+01:00 Alex Nastetsky anastet...@spryinc.com: Juan, What kind of information are you looking for? The service level ACLs are for limiting which services can communicate under certain protocols, by username or user group. Perhaps you are looking for client level ACL, something like the MapReduce ACLs? https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html#Job+Authorization Alex. 2014-02-20 4:58 GMT-05:00 Juan Carlos jcfernan...@cediant.es: Where could I find some information about ACL? I only could find the available in http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/ServiceLevelAuth.html, which isn't so detailed. Regards Juan Carlos Fernández Rodríguez Consultor Tecnológico Telf: +34918105294 Móvil: +34639311788 CEDIANT Centro para el Desarrollo, Investigación y Aplicación de Nuevas Tecnologías HPC Business Solutions * AVISO LEGAL * Este mensaje es solamente para la persona a la que va dirigido. Puede contener información confidencial o legalmente protegida. No hay renuncia a la confidencialidad o privilegio por cualquier transmisión mala/errónea. Si usted ha recibido este mensaje por error,le rogamos que borre de su sistema inmediatamente el mensaje asi como todas sus copias, destruya todas las copias del mismo de su disco duro y notifique al remitente. No debe, directa o indirectamente, usar, revelar, distribuir, imprimir o copiar ninguna de las partes de este mensaje si no es usted el destinatario. Cualquier opinión expresada en este mensaje proviene del remitente, excepto cuando el mensaje establezca lo contrario y el remitente esté autorizado para establecer que dichas opiniones provienen de 'CEDIANT'. Nótese que el correo electrónico vía Internet no permite asegurar ni la confidencialidad de los mensajes que se transmiten ni la correcta recepción de los mismos. En el caso de que el destinatario de este mensaje no consintiera la utilización del correo electrónico vía Internet, rogamos lo ponga en nuestro conocimiento de manera inmediata. * DISCLAIMER * This message is intended exclusively for the named person. It may contain confidential, propietary or legally privileged information. No confidentiality or privilege is waived or lost by any mistransmission. If you receive this message in error, please immediately delete it and all copies of it from your system, destroy any hard copies of it an notify the sender. Your must not, directly or indirectly, use, disclose, distribute, print, or copy any part of this message if you are not the intended recipient. Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorised to state them to be the views of 'CEDIANT'. Please note that internet e-mail neither guarantees the confidentiality nor the proper receipt of the message sent. If the addressee of this message does not consent to the use of internet e-mail, please communicate it to us immediately.
Capacity Scheduler capacity vs. maximum-capacity
Please help me understand how capacity and maximum-capacity on a queue work in the Capacity Scheduler. My understanding is that a queue is allocated capacity amount of resources, and if it needs more, it can stretch up to maximum-capacity resources. But if that's the case, why do we need capacity at all? It seems like maximum-capacity is the true limit and capacity is ignored? Thanks in advance, Alex.
Re: Capacity Scheduler capacity vs. maximum-capacity
Thanks, makes sense now. On Feb 19, 2014 9:35 PM, Jian He j...@hortonworks.com wrote: Yes, in the scenario you mentioned, the scheduler will take away the 10% from queue B and give it back to queue A On Wed, Feb 19, 2014 at 5:35 PM, Alex Nastetsky anastet...@spryinc.comwrote: Thanks Jian. In what sense is the capacity resources guaranteed? Let's say there are two queues at the root level, A and B. Both have capacity of 50% and max capacity of 70%. If queue A is only currently using 40% and queue B is using 50% and needs more, then it will be able to borrow 10% from queue A, so now queue A is using 40% and queue B is using 60%. But what happens if queue A now needs 50%? Will the scheduler take away the 10% from queue B and give it back to queue A even if queue B needs it? If not, it would seem that the scheduler is reneging on its guarantee. On Wed, Feb 19, 2014 at 6:01 PM, Jian He j...@hortonworks.com wrote: Hi Alex You can find good explanation from here: http://hortonworks.com/blog/understanding-apache-hadoops-capacity-scheduler/ Short term: Capacity is the soft limit that queue is guaranteed for such an amount of resource. For the purpose of necessary elasticity, queue can go beyond capacity limit but can not go beyond Max-Capacity limit which is the hard limit. Jian On Wed, Feb 19, 2014 at 11:08 AM, Alex Nastetsky anastet...@spryinc.com wrote: Please help me understand how capacity and maximum-capacity on a queue work in the Capacity Scheduler. My understanding is that a queue is allocated capacity amount of resources, and if it needs more, it can stretch up to maximum-capacity resources. But if that's the case, why do we need capacity at all? It seems like maximum-capacity is the true limit and capacity is ignored? Thanks in advance, Alex. CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
restrict job queues listing
Hi, I am trying to secure multiple MR job queues from being accessed from unauthorized users. I know there is acl_submit_applications and acl_administer_queue for a specific queue, but neither of those properties control who can view the list of jobs that have been or are executing in a specific queue. I would like to have a group of users only be able to view the list of jobs in some queues, but not others. Is there a way to do this? Even with mapreduce.cluster.acls.enabled set to true, which means that only the job owner and admins can view the details of a job, the listing of jobs itself has a lot of sensitive information. The only thing I can find is that I can filter the view by queue, which doesn't really help with security. Thanks in advance, Alex.
