service level authorization check the combination of host and user （hadoop-12628）
hi all, Service level authorization in hadoop2.2.x,hadoop2.5.x,hadoop2.6.x can only check the user from client. Service level authorization in hadoop2.7.x add the function of checking the host(ip) from client, but only can check host and user independently and cannot check the combination of host and user. I submitted a improvement issue and the patch which add the function of checking authorization of the combination of host and user for hadoop2.2.x, hadoop2.5.x, hadoop2.6.x : https://issues.apache.org/jira/browse/HADOOP-12628 . After put the patch,we can set the authorization of host-user pair in the hadoop-policy.xml.Take security.client.protocol.acl for example: If we only let the hadoop_user1 from 192.168.0.1(ip) has the authorization, we can set "hadoop_user1:192.168.0.1". So hadoop_user1 from other host but 192.168.0.1 doesn't have the authorization. If we add the authorization of hadoop_user2 from myhost.com.cn(hostname), we can set "hadoop_user2:myhost.com.cn"; if we authorize hadoop_user3 from any host,we just set "hadoop_user3" like before; if we want toauthorize any user from the host 192.168.10.10, we can set "*:192.168.10.10". example： security.client.protocol.acl hadoop_user1:192.168.0.1,hadoop_user2:myhost.com.cn,hadoop_user3,*:192.168.10.10 It is also applied to the blocked access control list after hadoop2.6.0： example： security.client.protocol.acl.blocked hadoop_user1:192.168.0.1,hadoop_user2:myhost.com.cn,hadoop_user3,*:192.168.10.10 The format of access control list is completely Compatible. The list of users and groups are both comma separated list of names. The two lists are separated by a space. Add a blank at the beginning of the line if only a list of groups is to be provided, equivalently a comma-separated list of users followed by a space or nothing implies only a set of given users.A special value of * implies that all users from any host are allowed to access the service. Example: user1,user2 group1,group2 （user1,user2,group1,group2 from any host have the authorization） user1:192.168.0.1,user2:myhost1.com.cn group1:192.168.0.2,group2:myhost2.com.cn （user1 from 192.168.0.1, user2 from myhost1.com.cn, group1 from 192.168.0.2,group2 from myhost2.com.cn have the authorization） *:192.168.0.1,*:myhost1.com.cn (any user from 192.168.0.1, any user from myhost1.com.cn have the authorization） * (any user from any host have the authorization) example1: security.client.protocol.acl * example2: security.client.protocol.acl user1,user2 group1,group2 example3: security.client.protocol.acl *:192.168.0.1,*:myhost1.com.cn example3: security.client.protocol.acl user1:192.168.0.1,user2:myhost1.com.cn group1:192.168.0.2,group2:myhost2.com.cn Thanks and happy weekend!
Re: Service Level Authorization
Thanks Alex, my path to the queue was a mistake when I was testing configurations and was unable to make work ACLs. My major problem was about mapreduce.cluster.administrators parameters. I didn't know anything about this parameter, I have been looking for it in http://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xmlbut it is missing there. Thanks for your help, it worked as soon as I set that property to my hadoop admin group. 2014-02-20 17:38 GMT+01:00 Alex Nastetsky anastet...@spryinc.com: If your test1 queue is under test queue, then you have to specify the path in the same way: yarn.scheduler.capacity.root.test.test1.acl_submit_applications (you are missing the test) Also, if your hadoop user is a member of user group hadoop, that is the default value of the mapreduce.cluster.administrators in mapred-site.xml. Users of that group can submit jobs to and administer all queues. On Thu, Feb 20, 2014 at 11:28 AM, Juan Carlos juc...@gmail.com wrote: Yes, that is what I'm looking for, but I couldn't find this information for hadoop 2.2.0. I saw mapreduce.cluster.acls.enabled it's now the parameter to use. But I don't know how to set my ACLs. I'm using capacity schedurler and I've created 3 new queues test (which is under root at the same level as default) and test1 and test2, which are under test. As I said, I enabled mapreduce.cluster.acls.enabled in mapred-site.xml and later added the parameter yarn.scheduler.capacity.root.test1.acl_submit_applications with value jcfernandez . If I submit a job to queue test1 with user hadoop, it allows it to run it. Which is my error? 2014-02-20 16:41 GMT+01:00 Alex Nastetsky anastet...@spryinc.com: Juan, What kind of information are you looking for? The service level ACLs are for limiting which services can communicate under certain protocols, by username or user group. Perhaps you are looking for client level ACL, something like the MapReduce ACLs? https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html#Job+Authorization Alex. 2014-02-20 4:58 GMT-05:00 Juan Carlos jcfernan...@cediant.es: Where could I find some information about ACL? I only could find the available in http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/ServiceLevelAuth.html, which isn't so detailed. Regards Juan Carlos Fernández Rodríguez Consultor Tecnológico Telf: +34918105294 Móvil: +34639311788 CEDIANT Centro para el Desarrollo, Investigación y Aplicación de Nuevas Tecnologías HPC Business Solutions * AVISO LEGAL * Este mensaje es solamente para la persona a la que va dirigido. Puede contener información confidencial o legalmente protegida. No hay renuncia a la confidencialidad o privilegio por cualquier transmisión mala/errónea. Si usted ha recibido este mensaje por error,le rogamos que borre de su sistema inmediatamente el mensaje asi como todas sus copias, destruya todas las copias del mismo de su disco duro y notifique al remitente. No debe, directa o indirectamente, usar, revelar, distribuir, imprimir o copiar ninguna de las partes de este mensaje si no es usted el destinatario. Cualquier opinión expresada en este mensaje proviene del remitente, excepto cuando el mensaje establezca lo contrario y el remitente esté autorizado para establecer que dichas opiniones provienen de 'CEDIANT'. Nótese que el correo electrónico vía Internet no permite asegurar ni la confidencialidad de los mensajes que se transmiten ni la correcta recepción de los mismos. En el caso de que el destinatario de este mensaje no consintiera la utilización del correo electrónico vía Internet, rogamos lo ponga en nuestro conocimiento de manera inmediata. * DISCLAIMER * This message is intended exclusively for the named person. It may contain confidential, propietary or legally privileged information. No confidentiality or privilege is waived or lost by any mistransmission. If you receive this message in error, please immediately delete it and all copies of it from your system, destroy any hard copies of it an notify the sender. Your must not, directly or indirectly, use, disclose, distribute, print, or copy any part of this message if you are not the intended recipient. Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorised to state them to be the views of 'CEDIANT'. Please note that internet e-mail neither guarantees the confidentiality nor the proper receipt of the message sent. If the addressee of this message does not consent to the use of internet e-mail, please communicate it to us immediately.
Service Level Authorization
Where could I find some information about ACL? I only could find the available in http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/ServiceLevelAuth.html, which isn't so detailed. Regards Juan Carlos Fernández Rodríguez Consultor Tecnológico Telf: +34918105294 Móvil: +34639311788 CEDIANT Centro para el Desarrollo, Investigación y Aplicación de Nuevas Tecnologías HPC Business Solutions * AVISO LEGAL * Este mensaje es solamente para la persona a la que va dirigido. Puede contener información confidencial o legalmente protegida. No hay renuncia a la confidencialidad o privilegio por cualquier transmisión mala/errónea. Si usted ha recibido este mensaje por error,le rogamos que borre de su sistema inmediatamente el mensaje asi como todas sus copias, destruya todas las copias del mismo de su disco duro y notifique al remitente. No debe, directa o indirectamente, usar, revelar, distribuir, imprimir o copiar ninguna de las partes de este mensaje si no es usted el destinatario. Cualquier opinión expresada en este mensaje proviene del remitente, excepto cuando el mensaje establezca lo contrario y el remitente esté autorizado para establecer que dichas opiniones provienen de 'CEDIANT'. Nótese que el correo electrónico vía Internet no permite asegurar ni la confidencialidad de los mensajes que se transmiten ni la correcta recepción de los mismos. En el caso de que el destinatario de este mensaje no consintiera la utilización del correo electrónico vía Internet, rogamos lo ponga en nuestro conocimiento de manera inmediata. * DISCLAIMER * This message is intended exclusively for the named person. It may contain confidential, propietary or legally privileged information. No confidentiality or privilege is waived or lost by any mistransmission. If you receive this message in error, please immediately delete it and all copies of it from your system, destroy any hard copies of it an notify the sender. Your must not, directly or indirectly, use, disclose, distribute, print, or copy any part of this message if you are not the intended recipient. Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorised to state them to be the views of 'CEDIANT'. Please note that internet e-mail neither guarantees the confidentiality nor the proper receipt of the message sent. If the addressee of this message does not consent to the use of internet e-mail, please communicate it to us immediately.
Re: Service Level Authorization
Juan, What kind of information are you looking for? The service level ACLs are for limiting which services can communicate under certain protocols, by username or user group. Perhaps you are looking for client level ACL, something like the MapReduce ACLs? https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html#Job+Authorization Alex. 2014-02-20 4:58 GMT-05:00 Juan Carlos jcfernan...@cediant.es: Where could I find some information about ACL? I only could find the available in http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/ServiceLevelAuth.html, which isn't so detailed. Regards Juan Carlos Fernández Rodríguez Consultor Tecnológico Telf: +34918105294 Móvil: +34639311788 CEDIANT Centro para el Desarrollo, Investigación y Aplicación de Nuevas Tecnologías HPC Business Solutions * AVISO LEGAL * Este mensaje es solamente para la persona a la que va dirigido. Puede contener información confidencial o legalmente protegida. No hay renuncia a la confidencialidad o privilegio por cualquier transmisión mala/errónea. Si usted ha recibido este mensaje por error,le rogamos que borre de su sistema inmediatamente el mensaje asi como todas sus copias, destruya todas las copias del mismo de su disco duro y notifique al remitente. No debe, directa o indirectamente, usar, revelar, distribuir, imprimir o copiar ninguna de las partes de este mensaje si no es usted el destinatario. Cualquier opinión expresada en este mensaje proviene del remitente, excepto cuando el mensaje establezca lo contrario y el remitente esté autorizado para establecer que dichas opiniones provienen de 'CEDIANT'. Nótese que el correo electrónico vía Internet no permite asegurar ni la confidencialidad de los mensajes que se transmiten ni la correcta recepción de los mismos. En el caso de que el destinatario de este mensaje no consintiera la utilización del correo electrónico vía Internet, rogamos lo ponga en nuestro conocimiento de manera inmediata. * DISCLAIMER * This message is intended exclusively for the named person. It may contain confidential, propietary or legally privileged information. No confidentiality or privilege is waived or lost by any mistransmission. If you receive this message in error, please immediately delete it and all copies of it from your system, destroy any hard copies of it an notify the sender. Your must not, directly or indirectly, use, disclose, distribute, print, or copy any part of this message if you are not the intended recipient. Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorised to state them to be the views of 'CEDIANT'. Please note that internet e-mail neither guarantees the confidentiality nor the proper receipt of the message sent. If the addressee of this message does not consent to the use of internet e-mail, please communicate it to us immediately.
Re: Service Level Authorization
Yes, that is what I'm looking for, but I couldn't find this information for hadoop 2.2.0. I saw mapreduce.cluster.acls.enabled it's now the parameter to use. But I don't know how to set my ACLs. I'm using capacity schedurler and I've created 3 new queues test (which is under root at the same level as default) and test1 and test2, which are under test. As I said, I enabled mapreduce.cluster.acls.enabled in mapred-site.xml and later added the parameter yarn.scheduler.capacity.root.test1.acl_submit_applications with value jcfernandez . If I submit a job to queue test1 with user hadoop, it allows it to run it. Which is my error? 2014-02-20 16:41 GMT+01:00 Alex Nastetsky anastet...@spryinc.com: Juan, What kind of information are you looking for? The service level ACLs are for limiting which services can communicate under certain protocols, by username or user group. Perhaps you are looking for client level ACL, something like the MapReduce ACLs? https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html#Job+Authorization Alex. 2014-02-20 4:58 GMT-05:00 Juan Carlos jcfernan...@cediant.es: Where could I find some information about ACL? I only could find the available in http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/ServiceLevelAuth.html, which isn't so detailed. Regards Juan Carlos Fernández Rodríguez Consultor Tecnológico Telf: +34918105294 Móvil: +34639311788 CEDIANT Centro para el Desarrollo, Investigación y Aplicación de Nuevas Tecnologías HPC Business Solutions * AVISO LEGAL * Este mensaje es solamente para la persona a la que va dirigido. Puede contener información confidencial o legalmente protegida. No hay renuncia a la confidencialidad o privilegio por cualquier transmisión mala/errónea. Si usted ha recibido este mensaje por error,le rogamos que borre de su sistema inmediatamente el mensaje asi como todas sus copias, destruya todas las copias del mismo de su disco duro y notifique al remitente. No debe, directa o indirectamente, usar, revelar, distribuir, imprimir o copiar ninguna de las partes de este mensaje si no es usted el destinatario. Cualquier opinión expresada en este mensaje proviene del remitente, excepto cuando el mensaje establezca lo contrario y el remitente esté autorizado para establecer que dichas opiniones provienen de 'CEDIANT'. Nótese que el correo electrónico vía Internet no permite asegurar ni la confidencialidad de los mensajes que se transmiten ni la correcta recepción de los mismos. En el caso de que el destinatario de este mensaje no consintiera la utilización del correo electrónico vía Internet, rogamos lo ponga en nuestro conocimiento de manera inmediata. * DISCLAIMER * This message is intended exclusively for the named person. It may contain confidential, propietary or legally privileged information. No confidentiality or privilege is waived or lost by any mistransmission. If you receive this message in error, please immediately delete it and all copies of it from your system, destroy any hard copies of it an notify the sender. Your must not, directly or indirectly, use, disclose, distribute, print, or copy any part of this message if you are not the intended recipient. Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorised to state them to be the views of 'CEDIANT'. Please note that internet e-mail neither guarantees the confidentiality nor the proper receipt of the message sent. If the addressee of this message does not consent to the use of internet e-mail, please communicate it to us immediately.
Re: Service Level Authorization
If your test1 queue is under test queue, then you have to specify the path in the same way: yarn.scheduler.capacity.root.test.test1.acl_submit_applications (you are missing the test) Also, if your hadoop user is a member of user group hadoop, that is the default value of the mapreduce.cluster.administrators in mapred-site.xml. Users of that group can submit jobs to and administer all queues. On Thu, Feb 20, 2014 at 11:28 AM, Juan Carlos juc...@gmail.com wrote: Yes, that is what I'm looking for, but I couldn't find this information for hadoop 2.2.0. I saw mapreduce.cluster.acls.enabled it's now the parameter to use. But I don't know how to set my ACLs. I'm using capacity schedurler and I've created 3 new queues test (which is under root at the same level as default) and test1 and test2, which are under test. As I said, I enabled mapreduce.cluster.acls.enabled in mapred-site.xml and later added the parameter yarn.scheduler.capacity.root.test1.acl_submit_applications with value jcfernandez . If I submit a job to queue test1 with user hadoop, it allows it to run it. Which is my error? 2014-02-20 16:41 GMT+01:00 Alex Nastetsky anastet...@spryinc.com: Juan, What kind of information are you looking for? The service level ACLs are for limiting which services can communicate under certain protocols, by username or user group. Perhaps you are looking for client level ACL, something like the MapReduce ACLs? https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html#Job+Authorization Alex. 2014-02-20 4:58 GMT-05:00 Juan Carlos jcfernan...@cediant.es: Where could I find some information about ACL? I only could find the available in http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/ServiceLevelAuth.html, which isn't so detailed. Regards Juan Carlos Fernández Rodríguez Consultor Tecnológico Telf: +34918105294 Móvil: +34639311788 CEDIANT Centro para el Desarrollo, Investigación y Aplicación de Nuevas Tecnologías HPC Business Solutions * AVISO LEGAL * Este mensaje es solamente para la persona a la que va dirigido. Puede contener información confidencial o legalmente protegida. No hay renuncia a la confidencialidad o privilegio por cualquier transmisión mala/errónea. Si usted ha recibido este mensaje por error,le rogamos que borre de su sistema inmediatamente el mensaje asi como todas sus copias, destruya todas las copias del mismo de su disco duro y notifique al remitente. No debe, directa o indirectamente, usar, revelar, distribuir, imprimir o copiar ninguna de las partes de este mensaje si no es usted el destinatario. Cualquier opinión expresada en este mensaje proviene del remitente, excepto cuando el mensaje establezca lo contrario y el remitente esté autorizado para establecer que dichas opiniones provienen de 'CEDIANT'. Nótese que el correo electrónico vía Internet no permite asegurar ni la confidencialidad de los mensajes que se transmiten ni la correcta recepción de los mismos. En el caso de que el destinatario de este mensaje no consintiera la utilización del correo electrónico vía Internet, rogamos lo ponga en nuestro conocimiento de manera inmediata. * DISCLAIMER * This message is intended exclusively for the named person. It may contain confidential, propietary or legally privileged information. No confidentiality or privilege is waived or lost by any mistransmission. If you receive this message in error, please immediately delete it and all copies of it from your system, destroy any hard copies of it an notify the sender. Your must not, directly or indirectly, use, disclose, distribute, print, or copy any part of this message if you are not the intended recipient. Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorised to state them to be the views of 'CEDIANT'. Please note that internet e-mail neither guarantees the confidentiality nor the proper receipt of the message sent. If the addressee of this message does not consent to the use of internet e-mail, please communicate it to us immediately.
