Re: Troubles with HDFS policies
I just created my account on JIRA. My user ID is the same than in Confluence : bartimeux. Regards, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-06-09 18:43 GMT+02:00 Don Bosco Durai : > Loïc, thanks > > Can you also create a JIRA to track it? Selva can you help here to add > Loïc to the contributor list? > > Thanks > > Bosco > > > From: Loïc Chanel > Reply-To: "user@ranger.incubator.apache.org" < > user@ranger.incubator.apache.org> > Date: Tuesday, June 9, 2015 at 7:52 AM > To: "user@ranger.incubator.apache.org" > Subject: Re: Troubles with HDFS policies > > Hi, > > Actually, I still have to modify it, but I will complete it as I go > further in Hadoop secured ecosystem deployment. > > The principal thing I wanted to document was the way to use Apache Knox, > as I noticed some mistakes in the URLs for Knox usage described by the > documentations I found on the Web (like unnecessary "/api"). > But as I am working on the deployment of a fully secured multi-tenant > cluster providing services such as Spark, Hive and HBase, I will have to > provide some documentation describing how to deploy Apache Ranger to manage > security on these components. > > Therefore, that documentation should improve and complete what I started > to write on Confluence. > > Regards, > > > Loïc > > Loïc CHANEL > Engineering student at TELECOM Nancy > Trainee at Worldline - Villeurbanne > > 2015-06-04 19:00 GMT+02:00 Don Bosco Durai : > >> Hi >> >> I apologize, I missed this email somehow. >> >> Thanks for putting this document together. It is looking good. I think, >> this will be good starting point to build our user guide. >> >> I feel, we should list out the topics we want to document and share the >> effort. >> >> Thanks again >> >> Bosco >> >> From: Chanel Loïc >> Reply-To: "user@ranger.incubator.apache.org" < >> user@ranger.incubator.apache.org> >> Date: Tuesday, May 26, 2015 at 6:33 AM >> To: "user@ranger.incubator.apache.org" >> Subject: RE: Troubles with HDFS policies >> >> Hi Bosco, >> >> >> >> I wrote some paragraphs on the page >> https://cwiki.apache.org/confluence/display/RANGER/Ranger+User+Guide >> >> As I only worked on Ranger and HDFS for now, it is the first part I >> created, but I will document the other components in the upcoming weeks. >> >> Feel free to make any remarks, and to tell me if this suits you. >> >> >> >> In the meantime, I noticed some missing things and typo in Ranger >> Hortonworks documentation. Can I help improving it somehow ? >> >> >> >> Thanks, >> >> >> >> >> >> Loïc >> >> >> >> *De :* Don Bosco Durai [mailto:bdu...@hortonworks.com >> ] *De la part de* Don Bosco Durai >> *Envoyé :* lundi 4 mai 2015 19:05 >> *À :* user@ranger.incubator.apache.org >> *Objet :* Re: Troubles with HDFS policies >> >> >> >> I have given you the permission. Let’s co-ordinate on creating the user >> guide page. >> >> >> >> Thanks >> >> >> >> Bosco >> >> >> >> >> >> *From: *Chanel Loïc >> *Reply-To: *"user@ranger.incubator.apache.org" < >> user@ranger.incubator.apache.org> >> *Date: *Monday, May 4, 2015 at 1:23 AM >> *To: *"user@ranger.incubator.apache.org" < >> user@ranger.incubator.apache.org> >> *Subject: *RE: Troubles with HDFS policies >> >> >> >> Hi Bosco, >> >> >> >> I just created an account on Confluence, my user ID is bartimeux. >> >> Thanks, >> >> >> >> >> >> Loïc >> >> >> >> *De :* Don Bosco Durai [mailto:bdu...@hortonworks.com >> ] *De la part de* Don Bosco Durai >> *Envoyé :* vendredi 1 mai 2015 06:44 >> *À :* user@ranger.incubator.apache.org >> *Objet :* Re: Troubles with HDFS policies >> >> >> >> Hi Loïc >> >> >> >> Thanks for the feedback. >> >> >> >> I think, you are referring to the Hortonworks documentation. >> >> >> >> We have a place holder in Apache Ranger Wiki site for user guide. We can >> start working on it. If you can give your confluence id, we can give you >> edit permission. >> >> >> >> Thanks >> >> >> >> Bosco >> >> &g
Re: Troubles with HDFS policies
Loïc, thanks Can you also create a JIRA to track it? Selva can you help here to add Loïc to the contributor list? Thanks Bosco From: Loïc Chanel Reply-To: "user@ranger.incubator.apache.org" Date: Tuesday, June 9, 2015 at 7:52 AM To: "user@ranger.incubator.apache.org" Subject: Re: Troubles with HDFS policies > Hi, > > Actually, I still have to modify it, but I will complete it as I go further in > Hadoop secured ecosystem deployment. > > The principal thing I wanted to document was the way to use Apache Knox, as I > noticed some mistakes in the URLs for Knox usage described by the > documentations I found on the Web (like unnecessary "/api"). > But as I am working on the deployment of a fully secured multi-tenant cluster > providing services such as Spark, Hive and HBase, I will have to provide some > documentation describing how to deploy Apache Ranger to manage security on > these components. > > Therefore, that documentation should improve and complete what I started to > write on Confluence. > > Regards, > > > Loïc > > Loïc CHANEL > Engineering student at TELECOM Nancy > Trainee at Worldline - Villeurbanne > > 2015-06-04 19:00 GMT+02:00 Don Bosco Durai : >> Hi >> >> I apologize, I missed this email somehow. >> >> Thanks for putting this document together. It is looking good. I think, this >> will be good starting point to build our user guide. >> >> I feel, we should list out the topics we want to document and share the >> effort. >> >> Thanks again >> >> Bosco >> >> From: Chanel Loïc >> Reply-To: "user@ranger.incubator.apache.org" >> >> Date: Tuesday, May 26, 2015 at 6:33 AM >> To: "user@ranger.incubator.apache.org" >> Subject: RE: Troubles with HDFS policies >> >>> Hi Bosco, >>> >>> I wrote some paragraphs on the page >>> https://cwiki.apache.org/confluence/display/RANGER/Ranger+User+Guide >>> As I only worked on Ranger and HDFS for now, it is the first part I created, >>> but I will document the other components in the upcoming weeks. >>> Feel free to make any remarks, and to tell me if this suits you. >>> >>> In the meantime, I noticed some missing things and typo in Ranger >>> Hortonworks documentation. Can I help improving it somehow ? >>> >>> Thanks, >>> >>> >>> Loïc >>> >>> >>> De : Don Bosco Durai [mailto:bdu...@hortonworks.com] De la part de Don Bosco >>> Durai >>> Envoyé : lundi 4 mai 2015 19:05 >>> À : user@ranger.incubator.apache.org >>> Objet : Re: Troubles with HDFS policies >>> >>> >>> I have given you the permission. Let¹s co-ordinate on creating the user >>> guide page. >>> >>> >>> >>> Thanks >>> >>> >>> >>> Bosco >>> >>> >>> >>> >>> >>> From: Chanel Loïc >>> Reply-To: "user@ranger.incubator.apache.org" >>> >>> Date: Monday, May 4, 2015 at 1:23 AM >>> To: "user@ranger.incubator.apache.org" >>> Subject: RE: Troubles with HDFS policies >>> >>> >>>> >>>> Hi Bosco, >>>> >>>> I just created an account on Confluence, my user ID is bartimeux. >>>> Thanks, >>>> >>>> >>>> Loïc >>>> >>>> >>>> De : Don Bosco Durai [mailto:bdu...@hortonworks.com] De la part de Don >>>> Bosco Durai >>>> Envoyé : vendredi 1 mai 2015 06:44 >>>> À : user@ranger.incubator.apache.org >>>> Objet : Re: Troubles with HDFS policies >>>> >>>> >>>> Hi Loïc >>>> >>>> >>>> >>>> Thanks for the feedback. >>>> >>>> >>>> >>>> I think, you are referring to the Hortonworks documentation. >>>> >>>> >>>> >>>> We have a place holder in Apache Ranger Wiki site for user guide. We can >>>> start working on it. If you can give your confluence id, we can give you >>>> edit permission. >>>> >>>> >>>> >>>> Thanks >>>> >>>> >>>> >>>> Bosco >>>> >>>> >>>> >>>> From: Chanel Loïc >>>> Reply-To: "user@ranger.incubator.apac
Re: Troubles with HDFS policies
Hi, Actually, I still have to modify it, but I will complete it as I go further in Hadoop secured ecosystem deployment. The principal thing I wanted to document was the way to use Apache Knox, as I noticed some mistakes in the URLs for Knox usage described by the documentations I found on the Web (like unnecessary "/api"). But as I am working on the deployment of a fully secured multi-tenant cluster providing services such as Spark, Hive and HBase, I will have to provide some documentation describing how to deploy Apache Ranger to manage security on these components. Therefore, that documentation should improve and complete what I started to write on Confluence. Regards, Loïc Loïc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-06-04 19:00 GMT+02:00 Don Bosco Durai : > Hi > > I apologize, I missed this email somehow. > > Thanks for putting this document together. It is looking good. I think, > this will be good starting point to build our user guide. > > I feel, we should list out the topics we want to document and share the > effort. > > Thanks again > > Bosco > > From: Chanel Loïc > Reply-To: "user@ranger.incubator.apache.org" < > user@ranger.incubator.apache.org> > Date: Tuesday, May 26, 2015 at 6:33 AM > To: "user@ranger.incubator.apache.org" > Subject: RE: Troubles with HDFS policies > > Hi Bosco, > > > > I wrote some paragraphs on the page > https://cwiki.apache.org/confluence/display/RANGER/Ranger+User+Guide > > As I only worked on Ranger and HDFS for now, it is the first part I > created, but I will document the other components in the upcoming weeks. > > Feel free to make any remarks, and to tell me if this suits you. > > > > In the meantime, I noticed some missing things and typo in Ranger > Hortonworks documentation. Can I help improving it somehow ? > > > > Thanks, > > > > > > Loïc > > > > *De :* Don Bosco Durai [mailto:bdu...@hortonworks.com > ] *De la part de* Don Bosco Durai > *Envoyé :* lundi 4 mai 2015 19:05 > *À :* user@ranger.incubator.apache.org > *Objet :* Re: Troubles with HDFS policies > > > > I have given you the permission. Let’s co-ordinate on creating the user > guide page. > > > > Thanks > > > > Bosco > > > > > > *From: *Chanel Loïc > *Reply-To: *"user@ranger.incubator.apache.org" < > user@ranger.incubator.apache.org> > *Date: *Monday, May 4, 2015 at 1:23 AM > *To: *"user@ranger.incubator.apache.org" > > *Subject: *RE: Troubles with HDFS policies > > > > Hi Bosco, > > > > I just created an account on Confluence, my user ID is bartimeux. > > Thanks, > > > > > > Loïc > > > > *De :* Don Bosco Durai [mailto:bdu...@hortonworks.com > ] *De la part de* Don Bosco Durai > *Envoyé :* vendredi 1 mai 2015 06:44 > *À :* user@ranger.incubator.apache.org > *Objet :* Re: Troubles with HDFS policies > > > > Hi Loïc > > > > Thanks for the feedback. > > > > I think, you are referring to the Hortonworks documentation. > > > > We have a place holder in Apache Ranger Wiki site for user guide. We can > start working on it. If you can give your confluence id, we can give you > edit permission. > > > > Thanks > > > > Bosco > > > > *From: *Chanel Loïc > *Reply-To: *"user@ranger.incubator.apache.org" < > user@ranger.incubator.apache.org> > *Date: *Thursday, April 30, 2015 at 1:32 AM > *To: *"user@ranger.incubator.apache.org" > > *Subject: *RE: Troubles with HDFS policies > > > > Hi, > > > > Indeed, the page 10 of the Ranger User Guide specifies : > > > > ”Through configuration, Apache Ranger enables both Ranger policies and > HDFS permissions to be checked for a user request. When the NameNode > receives a user request, the Ranger plugin checks for policies set through > the Ranger Policy Manager. If there are no policies, the Ranger plugin > checks for permissions set in HDFS. > > We recommend that permissions be created at the Ranger Policy Manager, and > to have restrictive permissions at the HDFS level.” > > > > So setting very restrictive permissions with HDFS allows to manage > entirely the cluster security with Ranger. > > Still, as I noticed some small mistakes, do you know how I can contribute > to the documentation improvement ? > > > > Thanks for your help, > > > > > > Loïc > > > > > > *De :* Don Bosco Durai [mailto:bdu...@hortonworks.com > ] *De la part de* Don Bosco Durai > *Envoyé :* mercredi 29 avril 2015 17
Re: Troubles with HDFS policies
Hi I apologize, I missed this email somehow. Thanks for putting this document together. It is looking good. I think, this will be good starting point to build our user guide. I feel, we should list out the topics we want to document and share the effort. Thanks again Bosco From: Chanel Loïc Reply-To: "user@ranger.incubator.apache.org" Date: Tuesday, May 26, 2015 at 6:33 AM To: "user@ranger.incubator.apache.org" Subject: RE: Troubles with HDFS policies > Hi Bosco, > > I wrote some paragraphs on the page > https://cwiki.apache.org/confluence/display/RANGER/Ranger+User+Guide > As I only worked on Ranger and HDFS for now, it is the first part I created, > but I will document the other components in the upcoming weeks. > Feel free to make any remarks, and to tell me if this suits you. > > In the meantime, I noticed some missing things and typo in Ranger Hortonworks > documentation. Can I help improving it somehow ? > > Thanks, > > > Loïc > > > De : Don Bosco Durai [mailto:bdu...@hortonworks.com] De la part de Don Bosco > Durai > Envoyé : lundi 4 mai 2015 19:05 > À : user@ranger.incubator.apache.org > Objet : Re: Troubles with HDFS policies > > > I have given you the permission. Let¹s co-ordinate on creating the user guide > page. > > > > Thanks > > > > Bosco > > > > > > From: Chanel Loïc > Reply-To: "user@ranger.incubator.apache.org" > > Date: Monday, May 4, 2015 at 1:23 AM > To: "user@ranger.incubator.apache.org" > Subject: RE: Troubles with HDFS policies > > >> >> Hi Bosco, >> >> I just created an account on Confluence, my user ID is bartimeux. >> Thanks, >> >> >> Loïc >> >> >> De : Don Bosco Durai [mailto:bdu...@hortonworks.com] De la part de Don Bosco >> Durai >> Envoyé : vendredi 1 mai 2015 06:44 >> À : user@ranger.incubator.apache.org >> Objet : Re: Troubles with HDFS policies >> >> >> Hi Loïc >> >> >> >> Thanks for the feedback. >> >> >> >> I think, you are referring to the Hortonworks documentation. >> >> >> >> We have a place holder in Apache Ranger Wiki site for user guide. We can >> start working on it. If you can give your confluence id, we can give you edit >> permission. >> >> >> >> Thanks >> >> >> >> Bosco >> >> >> >> From: Chanel Loïc >> Reply-To: "user@ranger.incubator.apache.org" >> >> Date: Thursday, April 30, 2015 at 1:32 AM >> To: "user@ranger.incubator.apache.org" >> Subject: RE: Troubles with HDFS policies >> >> >>> >>> Hi, >>> >>> Indeed, the page 10 of the Ranger User Guide specifies : >>> >>> ²Through configuration, Apache Ranger enables both Ranger policies and HDFS >>> permissions to be checked for a user request. When the NameNode receives a >>> user request, the Ranger plugin checks for policies set through the Ranger >>> Policy Manager. If there are no policies, the Ranger plugin checks for >>> permissions set in HDFS. >>> We recommend that permissions be created at the Ranger Policy Manager, and >>> to have restrictive permissions at the HDFS level.² >>> >>> So setting very restrictive permissions with HDFS allows to manage entirely >>> the cluster security with Ranger. >>> Still, as I noticed some small mistakes, do you know how I can contribute to >>> the documentation improvement ? >>> >>> Thanks for your help, >>> >>> >>> Loïc >>> >>> >>> >>> De : Don Bosco Durai [mailto:bdu...@hortonworks.com] De la part de Don Bosco >>> Durai >>> Envoyé : mercredi 29 avril 2015 17:45 >>> À : user@ranger.incubator.apache.org >>> Objet : Re: Troubles with HDFS policies >>> >>> >>> Check hdfs dfs -ls $folderName. In the case of HDFS, if Ranger doesn¹t find >>> any permission in it¹s policy database, then it falls back to HDFS >>> permission check. So make sure in the HDFS level, you have 700 or even 000 >>> for the given folder and manage all the permissions via Ranger. We recommend >>> pick all relevant folders (e.g Hive data warehouse folder) and do hdfs dfs >>> -chown -R hdfs:hdfs $folderName and hdfs dfs chmod 000 R $folderName. >>> >>> >>> >>> Please note, falling back to native permission is
RE: Troubles with HDFS policies
Hi Bosco, I wrote some paragraphs on the page https://cwiki.apache.org/confluence/display/RANGER/Ranger+User+Guide As I only worked on Ranger and HDFS for now, it is the first part I created, but I will document the other components in the upcoming weeks. Feel free to make any remarks, and to tell me if this suits you. In the meantime, I noticed some missing things and typo in Ranger Hortonworks documentation. Can I help improving it somehow ? Thanks, Loïc De : Don Bosco Durai [mailto:bdu...@hortonworks.com] De la part de Don Bosco Durai Envoyé : lundi 4 mai 2015 19:05 À : user@ranger.incubator.apache.org Objet : Re: Troubles with HDFS policies I have given you the permission. Let's co-ordinate on creating the user guide page. Thanks Bosco From: Chanel Loïc mailto:loic.cha...@worldline.com>> Reply-To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" mailto:user@ranger.incubator.apache.org>> Date: Monday, May 4, 2015 at 1:23 AM To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" mailto:user@ranger.incubator.apache.org>> Subject: RE: Troubles with HDFS policies Hi Bosco, I just created an account on Confluence, my user ID is bartimeux. Thanks, Loïc De : Don Bosco Durai [mailto:bdu...@hortonworks.com] De la part de Don Bosco Durai Envoyé : vendredi 1 mai 2015 06:44 À : user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org> Objet : Re: Troubles with HDFS policies Hi Loïc Thanks for the feedback. I think, you are referring to the Hortonworks documentation. We have a place holder in Apache Ranger Wiki site for user guide. We can start working on it. If you can give your confluence id, we can give you edit permission. Thanks Bosco From: Chanel Loïc mailto:loic.cha...@worldline.com>> Reply-To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" mailto:user@ranger.incubator.apache.org>> Date: Thursday, April 30, 2015 at 1:32 AM To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" mailto:user@ranger.incubator.apache.org>> Subject: RE: Troubles with HDFS policies Hi, Indeed, the page 10 of the Ranger User Guide specifies : "Through configuration, Apache Ranger enables both Ranger policies and HDFS permissions to be checked for a user request. When the NameNode receives a user request, the Ranger plugin checks for policies set through the Ranger Policy Manager. If there are no policies, the Ranger plugin checks for permissions set in HDFS. We recommend that permissions be created at the Ranger Policy Manager, and to have restrictive permissions at the HDFS level." So setting very restrictive permissions with HDFS allows to manage entirely the cluster security with Ranger. Still, as I noticed some small mistakes, do you know how I can contribute to the documentation improvement ? Thanks for your help, Loïc De : Don Bosco Durai [mailto:bdu...@hortonworks.com] De la part de Don Bosco Durai Envoyé : mercredi 29 avril 2015 17:45 À : user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org> Objet : Re: Troubles with HDFS policies Check hdfs dfs -ls $folderName. In the case of HDFS, if Ranger doesn't find any permission in it's policy database, then it falls back to HDFS permission check. So make sure in the HDFS level, you have 700 or even 000 for the given folder and manage all the permissions via Ranger. We recommend pick all relevant folders (e.g Hive data warehouse folder) and do hdfs dfs -chown -R hdfs:hdfs $folderName and hdfs dfs -chmod 000 -R $folderName. Please note, falling back to native permission is only available in HDFS. There is a switch to turn it off, but you have to be cautious when using it. Thanks Bosco From: Chanel Loïc mailto:loic.cha...@worldline.com>> Reply-To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" mailto:user@ranger.incubator.apache.org>> Date: Wednesday, April 29, 2015 at 5:24 AM To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" mailto:user@ranger.incubator.apache.org>> Subject: Troubles with HDFS policies Hi All, As I am trying to set a Hadoop secured cluster with Ranger, I encountered some troubles. The principal one consists in the fact that even if I have no rights to read, write or execute files in a directory, I still can execute a ls command (hdfs dfs -ls /testdir) showing me the files that I should not be able to read, or even see. I can even see the file contents by making a cat on these files (hdfs dfs -cat /testdir/testfile) that I should not be able to read, which is even more problematic to me. In parallel, I am not able to put any files in the directory (Permission denied for hdfs dfs -put myotherfile /testdir/myoth
Re: Troubles with HDFS policies
I have given you the permission. Let¹s co-ordinate on creating the user guide page. Thanks Bosco From: Chanel Loïc Reply-To: "user@ranger.incubator.apache.org" Date: Monday, May 4, 2015 at 1:23 AM To: "user@ranger.incubator.apache.org" Subject: RE: Troubles with HDFS policies > Hi Bosco, > > I just created an account on Confluence, my user ID is bartimeux. > Thanks, > > > Loïc > > > De : Don Bosco Durai [mailto:bdu...@hortonworks.com] De la part de Don Bosco > Durai > Envoyé : vendredi 1 mai 2015 06:44 > À : user@ranger.incubator.apache.org > Objet : Re: Troubles with HDFS policies > > > Hi Loïc > > > > Thanks for the feedback. > > > > I think, you are referring to the Hortonworks documentation. > > > > We have a place holder in Apache Ranger Wiki site for user guide. We can start > working on it. If you can give your confluence id, we can give you edit > permission. > > > > Thanks > > > > Bosco > > > > From: Chanel Loïc > Reply-To: "user@ranger.incubator.apache.org" > > Date: Thursday, April 30, 2015 at 1:32 AM > To: "user@ranger.incubator.apache.org" > Subject: RE: Troubles with HDFS policies > > >> >> Hi, >> >> Indeed, the page 10 of the Ranger User Guide specifies : >> >> ²Through configuration, Apache Ranger enables both Ranger policies and HDFS >> permissions to be checked for a user request. When the NameNode receives a >> user request, the Ranger plugin checks for policies set through the Ranger >> Policy Manager. If there are no policies, the Ranger plugin checks for >> permissions set in HDFS. >> We recommend that permissions be created at the Ranger Policy Manager, and to >> have restrictive permissions at the HDFS level.² >> >> So setting very restrictive permissions with HDFS allows to manage entirely >> the cluster security with Ranger. >> Still, as I noticed some small mistakes, do you know how I can contribute to >> the documentation improvement ? >> >> Thanks for your help, >> >> >> Loïc >> >> >> >> De : Don Bosco Durai [mailto:bdu...@hortonworks.com] De la part de Don Bosco >> Durai >> Envoyé : mercredi 29 avril 2015 17:45 >> À : user@ranger.incubator.apache.org >> Objet : Re: Troubles with HDFS policies >> >> >> Check hdfs dfs -ls $folderName. In the case of HDFS, if Ranger doesn¹t find >> any permission in it¹s policy database, then it falls back to HDFS permission >> check. So make sure in the HDFS level, you have 700 or even 000 for the given >> folder and manage all the permissions via Ranger. We recommend pick all >> relevant folders (e.g Hive data warehouse folder) and do hdfs dfs -chown -R >> hdfs:hdfs $folderName and hdfs dfs chmod 000 R $folderName. >> >> >> >> Please note, falling back to native permission is only available in HDFS. >> There is a switch to turn it off, but you have to be cautious when using it. >> >> >> >> Thanks >> >> >> >> Bosco >> >> >> >> >> >> From: Chanel Loïc >> Reply-To: "user@ranger.incubator.apache.org" >> >> Date: Wednesday, April 29, 2015 at 5:24 AM >> To: "user@ranger.incubator.apache.org" >> Subject: Troubles with HDFS policies >> >> >>> >>> Hi All, >>> >>> As I am trying to set a Hadoop secured cluster with Ranger, I encountered >>> some troubles. >>> The principal one consists in the fact that even if I have no rights to >>> read, write or execute files in a directory, I still can execute a ls >>> command (hdfs dfs ls /testdir) showing me the files that I should not be >>> able to read, or even see. I can even see the file contents by making a cat >>> on these files (hdfs dfs cat /testdir/testfile) that I should not be able >>> to read, which is even more problematic to me. >>> In parallel, I am not able to put any files in the directory (Permission >>> denied for hdfs dfs put myotherfile /testdir/myotherfile), which makes me >>> think the policies are correctly set. >>> >>> Does that sound quite normal to you ? Do you see a solution to make sure my >>> user toto cannot see what is in the repository of my user tata ? >>> Thanks for your help, >>> >>> >>> Loïc Chanel >>> >>> >>> >>> >>> Ce message et les pièces join
RE: Troubles with HDFS policies
Hi Bosco, I just created an account on Confluence, my user ID is bartimeux. Thanks, Loïc De : Don Bosco Durai [mailto:bdu...@hortonworks.com] De la part de Don Bosco Durai Envoyé : vendredi 1 mai 2015 06:44 À : user@ranger.incubator.apache.org Objet : Re: Troubles with HDFS policies Hi Loïc Thanks for the feedback. I think, you are referring to the Hortonworks documentation. We have a place holder in Apache Ranger Wiki site for user guide. We can start working on it. If you can give your confluence id, we can give you edit permission. Thanks Bosco From: Chanel Loïc mailto:loic.cha...@worldline.com>> Reply-To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" mailto:user@ranger.incubator.apache.org>> Date: Thursday, April 30, 2015 at 1:32 AM To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" mailto:user@ranger.incubator.apache.org>> Subject: RE: Troubles with HDFS policies Hi, Indeed, the page 10 of the Ranger User Guide specifies : "Through configuration, Apache Ranger enables both Ranger policies and HDFS permissions to be checked for a user request. When the NameNode receives a user request, the Ranger plugin checks for policies set through the Ranger Policy Manager. If there are no policies, the Ranger plugin checks for permissions set in HDFS. We recommend that permissions be created at the Ranger Policy Manager, and to have restrictive permissions at the HDFS level." So setting very restrictive permissions with HDFS allows to manage entirely the cluster security with Ranger. Still, as I noticed some small mistakes, do you know how I can contribute to the documentation improvement ? Thanks for your help, Loïc De : Don Bosco Durai [mailto:bdu...@hortonworks.com] De la part de Don Bosco Durai Envoyé : mercredi 29 avril 2015 17:45 À : user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org> Objet : Re: Troubles with HDFS policies Check hdfs dfs -ls $folderName. In the case of HDFS, if Ranger doesn't find any permission in it's policy database, then it falls back to HDFS permission check. So make sure in the HDFS level, you have 700 or even 000 for the given folder and manage all the permissions via Ranger. We recommend pick all relevant folders (e.g Hive data warehouse folder) and do hdfs dfs -chown -R hdfs:hdfs $folderName and hdfs dfs -chmod 000 -R $folderName. Please note, falling back to native permission is only available in HDFS. There is a switch to turn it off, but you have to be cautious when using it. Thanks Bosco From: Chanel Loïc mailto:loic.cha...@worldline.com>> Reply-To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" mailto:user@ranger.incubator.apache.org>> Date: Wednesday, April 29, 2015 at 5:24 AM To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" mailto:user@ranger.incubator.apache.org>> Subject: Troubles with HDFS policies Hi All, As I am trying to set a Hadoop secured cluster with Ranger, I encountered some troubles. The principal one consists in the fact that even if I have no rights to read, write or execute files in a directory, I still can execute a ls command (hdfs dfs -ls /testdir) showing me the files that I should not be able to read, or even see. I can even see the file contents by making a cat on these files (hdfs dfs -cat /testdir/testfile) that I should not be able to read, which is even more problematic to me. In parallel, I am not able to put any files in the directory (Permission denied for hdfs dfs -put myotherfile /testdir/myotherfile), which makes me think the policies are correctly set. Does that sound quite normal to you ? Do you see a solution to make sure my user toto cannot see what is in the repository of my user tata ? Thanks for your help, Loïc Chanel Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the
Re: Troubles with HDFS policies
Hi Loïc Thanks for the feedback. I think, you are referring to the Hortonworks documentation. We have a place holder in Apache Ranger Wiki site for user guide. We can start working on it. If you can give your confluence id, we can give you edit permission. Thanks Bosco From: Chanel Loïc Reply-To: "user@ranger.incubator.apache.org" Date: Thursday, April 30, 2015 at 1:32 AM To: "user@ranger.incubator.apache.org" Subject: RE: Troubles with HDFS policies > Hi, > > Indeed, the page 10 of the Ranger User Guide specifies : > > ²Through configuration, Apache Ranger enables both Ranger policies and HDFS > permissions to be checked for a user request. When the NameNode receives a > user request, the Ranger plugin checks for policies set through the Ranger > Policy Manager. If there are no policies, the Ranger plugin checks for > permissions set in HDFS. > We recommend that permissions be created at the Ranger Policy Manager, and to > have restrictive permissions at the HDFS level.² > > So setting very restrictive permissions with HDFS allows to manage entirely > the cluster security with Ranger. > Still, as I noticed some small mistakes, do you know how I can contribute to > the documentation improvement ? > > Thanks for your help, > > > Loïc > > > > De : Don Bosco Durai [mailto:bdu...@hortonworks.com] De la part de Don Bosco > Durai > Envoyé : mercredi 29 avril 2015 17:45 > À : user@ranger.incubator.apache.org > Objet : Re: Troubles with HDFS policies > > > Check hdfs dfs -ls $folderName. In the case of HDFS, if Ranger doesn¹t find > any permission in it¹s policy database, then it falls back to HDFS permission > check. So make sure in the HDFS level, you have 700 or even 000 for the given > folder and manage all the permissions via Ranger. We recommend pick all > relevant folders (e.g Hive data warehouse folder) and do hdfs dfs -chown -R > hdfs:hdfs $folderName and hdfs dfs chmod 000 R $folderName. > > > > Please note, falling back to native permission is only available in HDFS. > There is a switch to turn it off, but you have to be cautious when using it. > > > > Thanks > > > > Bosco > > > > > > From: Chanel Loïc > Reply-To: "user@ranger.incubator.apache.org" > > Date: Wednesday, April 29, 2015 at 5:24 AM > To: "user@ranger.incubator.apache.org" > Subject: Troubles with HDFS policies > > >> >> Hi All, >> >> As I am trying to set a Hadoop secured cluster with Ranger, I encountered >> some troubles. >> The principal one consists in the fact that even if I have no rights to read, >> write or execute files in a directory, I still can execute a ls command (hdfs >> dfs ls /testdir) showing me the files that I should not be able to read, or >> even see. I can even see the file contents by making a cat on these files >> (hdfs dfs cat /testdir/testfile) that I should not be able to read, which is >> even more problematic to me. >> In parallel, I am not able to put any files in the directory (Permission >> denied for hdfs dfs put myotherfile /testdir/myotherfile), which makes me >> think the policies are correctly set. >> >> Does that sound quite normal to you ? Do you see a solution to make sure my >> user toto cannot see what is in the repository of my user tata ? >> Thanks for your help, >> >> >> Loïc Chanel >> >> >> >> >> Ce message et les pièces jointes sont confidentiels et réservés à l'usage >> exclusif de ses destinataires. Il peut également être protégé par le secret >> professionnel. Si vous recevez ce message par erreur, merci d'en avertir >> immédiatement l'expéditeur et de le détruire. L'intégrité du message ne >> pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra >> être recherchée quant au contenu de ce message. Bien que les meilleurs >> efforts soient faits pour maintenir cette transmission exempte de tout virus, >> l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne >> saurait être recherchée pour tout dommage résultant d'un virus transmis. >> >> This e-mail and the documents attached are confidential and intended solely >> for the addressee; it may also be privileged. If you receive this e-mail in >> error, please notify the sender immediately and destroy it. As its integrity >> cannot be secured on the Internet, the Worldline liability cannot be >> triggered for the message content. Although the sender endeavours to maintain >> a computer virus-free network, the sender does not warra
RE: Troubles with HDFS policies
Hi, Indeed, the page 10 of the Ranger User Guide specifies : "Through configuration, Apache Ranger enables both Ranger policies and HDFS permissions to be checked for a user request. When the NameNode receives a user request, the Ranger plugin checks for policies set through the Ranger Policy Manager. If there are no policies, the Ranger plugin checks for permissions set in HDFS. We recommend that permissions be created at the Ranger Policy Manager, and to have restrictive permissions at the HDFS level." So setting very restrictive permissions with HDFS allows to manage entirely the cluster security with Ranger. Still, as I noticed some small mistakes, do you know how I can contribute to the documentation improvement ? Thanks for your help, Loïc De : Don Bosco Durai [mailto:bdu...@hortonworks.com] De la part de Don Bosco Durai Envoyé : mercredi 29 avril 2015 17:45 À : user@ranger.incubator.apache.org Objet : Re: Troubles with HDFS policies Check hdfs dfs -ls $folderName. In the case of HDFS, if Ranger doesn't find any permission in it's policy database, then it falls back to HDFS permission check. So make sure in the HDFS level, you have 700 or even 000 for the given folder and manage all the permissions via Ranger. We recommend pick all relevant folders (e.g Hive data warehouse folder) and do hdfs dfs -chown -R hdfs:hdfs $folderName and hdfs dfs -chmod 000 -R $folderName. Please note, falling back to native permission is only available in HDFS. There is a switch to turn it off, but you have to be cautious when using it. Thanks Bosco From: Chanel Loïc mailto:loic.cha...@worldline.com>> Reply-To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" mailto:user@ranger.incubator.apache.org>> Date: Wednesday, April 29, 2015 at 5:24 AM To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" mailto:user@ranger.incubator.apache.org>> Subject: Troubles with HDFS policies Hi All, As I am trying to set a Hadoop secured cluster with Ranger, I encountered some troubles. The principal one consists in the fact that even if I have no rights to read, write or execute files in a directory, I still can execute a ls command (hdfs dfs -ls /testdir) showing me the files that I should not be able to read, or even see. I can even see the file contents by making a cat on these files (hdfs dfs -cat /testdir/testfile) that I should not be able to read, which is even more problematic to me. In parallel, I am not able to put any files in the directory (Permission denied for hdfs dfs -put myotherfile /testdir/myotherfile), which makes me think the policies are correctly set. Does that sound quite normal to you ? Do you see a solution to make sure my user toto cannot see what is in the repository of my user tata ? Thanks for your help, Loïc Chanel Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis. This e-mail and the document
Re: Troubles with HDFS policies
Check hdfs dfs -ls $folderName. In the case of HDFS, if Ranger doesn¹t find any permission in it¹s policy database, then it falls back to HDFS permission check. So make sure in the HDFS level, you have 700 or even 000 for the given folder and manage all the permissions via Ranger. We recommend pick all relevant folders (e.g Hive data warehouse folder) and do hdfs dfs -chown -R hdfs:hdfs $folderName and hdfs dfs chmod 000 R $folderName. Please note, falling back to native permission is only available in HDFS. There is a switch to turn it off, but you have to be cautious when using it. Thanks Bosco From: Chanel Loïc Reply-To: "user@ranger.incubator.apache.org" Date: Wednesday, April 29, 2015 at 5:24 AM To: "user@ranger.incubator.apache.org" Subject: Troubles with HDFS policies > Hi All, > > As I am trying to set a Hadoop secured cluster with Ranger, I encountered some > troubles. > The principal one consists in the fact that even if I have no rights to read, > write or execute files in a directory, I still can execute a ls command (hdfs > dfs ls /testdir) showing me the files that I should not be able to read, or > even see. I can even see the file contents by making a cat on these files > (hdfs dfs cat /testdir/testfile) that I should not be able to read, which is > even more problematic to me. > In parallel, I am not able to put any files in the directory (Permission > denied for hdfs dfs put myotherfile /testdir/myotherfile), which makes me > think the policies are correctly set. > > Does that sound quite normal to you ? Do you see a solution to make sure my > user toto cannot see what is in the repository of my user tata ? > Thanks for your help, > > > Loïc Chanel > > > > Ce message et les pièces jointes sont confidentiels et réservés à l'usage > exclusif de ses destinataires. Il peut également être protégé par le secret > professionnel. Si vous recevez ce message par erreur, merci d'en avertir > immédiatement l'expéditeur et de le détruire. L'intégrité du message ne > pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra > être recherchée quant au contenu de ce message. Bien que les meilleurs efforts > soient faits pour maintenir cette transmission exempte de tout virus, > l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne > saurait être recherchée pour tout dommage résultant d'un virus transmis. > > This e-mail and the documents attached are confidential and intended solely > for the addressee; it may also be privileged. If you receive this e-mail in > error, please notify the sender immediately and destroy it. As its integrity > cannot be secured on the Internet, the Worldline liability cannot be triggered > for the message content. Although the sender endeavours to maintain a computer > virus-free network, the sender does not warrant that this transmission is > virus-free and will not be liable for any damages resulting from any virus > transmitted.
