Cool. Can we check RangerAdmin->Audit to see if there are any audit records?
Thanks Bosco From: Aneela Saleem <ane...@platalytics.com> Reply-To: "user@ranger.incubator.apache.org" <user@ranger.incubator.apache.org> Date: Tuesday, July 28, 2015 at 1:14 PM To: "user@ranger.incubator.apache.org" <user@ranger.incubator.apache.org> Subject: Re: Error Setting up Ranger-Admin > Yes the command is working > > On Wed, Jul 29, 2015 at 1:11 AM, Don Bosco Durai <bo...@apache.org> wrote: >> Did the original startup issue get resolved? >> >> Your errors seems to be coming from WebHDFS. Can we check whether “hdfs dfs >> -ls /“ works from command line? >> >> Thanks >> >> Bosco >> >> >> From: Aneela Saleem <ane...@platalytics.com> >> Reply-To: "user@ranger.incubator.apache.org" >> <user@ranger.incubator.apache.org> >> Date: Tuesday, July 28, 2015 at 1:05 PM >> >> To: "user@ranger.incubator.apache.org" <user@ranger.incubator.apache.org> >> Subject: Re: Error Setting up Ranger-Admin >> >>> That error has been resolved. >>> >>> Following is the latest error: >>> >>> class org.apache.hadoop.hdfs.web.resources.UserProvider >>> class org.apache.hadoop.hdfs.web.resources.ExceptionHandler >>> Jul 28, 2015 10:39:46 PM >>> com.sun.jersey.server.impl.application.WebApplicationImpl _initiate >>> INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 >>> AM' >>> Jul 28, 2015 10:39:51 PM com.sun.jersey.spi.inject.Errors >>> processErrorMessages >>> WARNING: The following warnings have been detected with resource and/or >>> provider classes: >>> WARNING: A sub-resource method, public javax.ws.rs.core.Response >>> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods. >>> getRoot(org.apache.hadoop.security.UserGroupInformation,org.apache.hadoop.hd >>> fs.web.resources.DelegationParam,org.apache.hadoop.hdfs.web.resources.UserPa >>> ram,org.apache.hadoop.hdfs.web.resources.DoAsParam,org.apache.hadoop.hdfs.we >>> b.resources.GetOpParam,org.apache.hadoop.hdfs.web.resources.OffsetParam,org. >>> apache.hadoop.hdfs.web.resources.LengthParam,org.apache.hadoop.hdfs.web.reso >>> urces.RenewerParam,org.apache.hadoop.hdfs.web.resources.BufferSizeParam,java >>> .util.List,org.apache.hadoop.hdfs.web.resources.XAttrEncodingParam,org.apach >>> e.hadoop.hdfs.web.resources.ExcludeDatanodesParam,org.apache.hadoop.hdfs.web >>> .resources.FsActionParam,org.apache.hadoop.hdfs.web.resources.TokenKindParam >>> ,org.apache.hadoop.hdfs.web.resources.TokenServiceParam) throws >>> java.io.IOException,java.lang.InterruptedException, with URI template, "/", >>> is treated as a resource method >>> >>> >>> >>> On Wed, Jul 29, 2015 at 12:50 AM, Don Bosco Durai <bo...@apache.org> wrote: >>>> I have seen this issue before in OpenStack environment where the DFS >>>> folders were configured to /tmp/… and they were wiped out after restart. >>>> >>>> If it is a test env, you can try formatting. First check dfs.name.dir >>>> property in HDFS. >>>> % $HADOOP_INSTALL/hadoop/bin/hadoop namenode -format >>>> >>>> >>>> http://wiki.apache.org/hadoop/GettingStartedWithHadoop >>>> Formatting the Namenode >>>> The first step to starting up your Hadoop installation is formatting the >>>> Hadoop filesystem, which is implemented on top of the local filesystems of >>>> your cluster. You need to do this the first time you set up a Hadoop >>>> installation. Do not format a running Hadoop filesystem, this will cause >>>> all your data to be erased. Before formatting, ensure that the dfs.name.dir >>>> directory exists. If you just used the default, then mkdir -p >>>> /tmp/hadoop-username/dfs/name will create the directory. To format the >>>> filesystem (which simply initializes the directory specified by the >>>> dfs.name.dir variable), run the command: >>>> % $HADOOP_INSTALL/hadoop/bin/hadoop namenode -format >>>> >>>> If asked to [re]format, you must reply Y (not just y) if you want to >>>> reformat, else Hadoop will abort the format. >>>> >>>> >>>> Thanks >>>> >>>> Bosco >>>> >>>> From: Aneela Saleem <ane...@platalytics.com> >>>> Reply-To: "user@ranger.incubator.apache.org" >>>> <user@ranger.incubator.apache.org> >>>> Date: Tuesday, July 28, 2015 at 12:40 PM >>>> >>>> To: "user@ranger.incubator.apache.org" <user@ranger.incubator.apache.org> >>>> Subject: Re: Error Setting up Ranger-Admin >>>> >>>>> NameNode error Logs: >>>>> >>>>> 2015-06-30 00:30:05,933 FATAL >>>>> org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode. >>>>> java.io.IOException: NameNode is not formatted. >>>>> >>>>> On Wed, Jul 29, 2015 at 12:22 AM, Don Bosco Durai <bo...@apache.org> >>>>> wrote: >>>>>> Anything in the namenode log file? >>>>>> >>>>>> Thanks >>>>>> >>>>>> Bosco >>>>>> >>>>>> From: Aneela Saleem <ane...@platalytics.com> >>>>>> Reply-To: "user@ranger.incubator.apache.org" >>>>>> <user@ranger.incubator.apache.org> >>>>>> Date: Tuesday, July 28, 2015 at 12:18 PM >>>>>> To: "user@ranger.incubator.apache.org" <user@ranger.incubator.apache.org> >>>>>> Subject: Re: Error Setting up Ranger-Admin >>>>>> >>>>>>> Hdfs Plugin request is not being shown in Audit > Plugins tab. I >>>>>>> followed all steps and verified with your steps. Restarted namenode but >>>>>>> still can't see any plugin request >>>>>>> >>>>>>> On Wed, Jul 29, 2015 at 12:03 AM, Aneela Saleem <ane...@platalytics.com> >>>>>>> wrote: >>>>>>> Thank you so much Bosco. I created JIRA for this. Can be found here >>>>>>> <https://issues.apache.org/jira/browse/RANGER-582> >>>>>>> >>>>>>> Let me try UserSync, then i will approach yyou in case of any problem. >>>>>>> >>>>>>> On Tue, Jul 28, 2015 at 11:28 PM, Don Bosco Durai >>>>>>> <bdu...@hortonworks.com> wrote: >>>>>>> Good. I already typed it, so let me paste it for the benefit of others. >>>>>>> We need to update the doc also. If you don’t mind, can you create a JIRA >>>>>>> to track this? >>>>>>> 1. Go to RangerAdmin UI -> AccessManager (first menu tab is already >>>>>>> opened by default) >>>>>>> 2. Click on “+” in the HDFS panel title >>>>>>> 3. Enter Service Name: e.g. “${cluster_name}_hdfs. Cluster name could be >>>>>>> anything. You can call the repo “test_hdfs” if you want to >>>>>>> 4. UserName: ranger (doesn’t matter whether it exists because you are >>>>>>> not using kerberos) >>>>>>> 5. Password: ranger (doesn’t matter what you give) >>>>>>> 6. Namenode URL: hdfs://<namenode_host>:8020 >>>>>>> 7. Authorization Enabled: No >>>>>>> 8. Authentication Type: Simple >>>>>>> 9. Rest you can leave it as default. >>>>>>> Save this screen. Then use the same name e.g “test_hdfs” in your >>>>>>> install.properties for enabling Ranger in HDFS. Then restart namenode. >>>>>>> >>>>>>> Is the plugin request showing up in the Audit->Plugin tab? >>>>>>> >>>>>>> If the plugin is showing up, then that means authorization and auditing >>>>>>> is already working. >>>>>>> >>>>>>> For configuring policies, we need to configure UserSync. The >>>>>>> install/configure process is same for all processes. You need to update >>>>>>> the install.properties with the appropriate values. >>>>>>> Set SYNC_SOURCE = ldap >>>>>>> And check the section "Table: LDAP/AD Properties with sample values” for >>>>>>> reference values. >>>>>>> >>>>>>> Please note that for using LDAP within Hadoop, you need to configure >>>>>>> Hadoop core-site.xml to use LDAP. You might be already knowing it. Just >>>>>>> want to make sure. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Bosco >>>>>>> >>>>>>> >>>>>>> From: Aneela Saleem <ane...@platalytics.com> >>>>>>> Reply-To: "user@ranger.incubator.apache.org" >>>>>>> <user@ranger.incubator.apache.org> >>>>>>> Date: Tuesday, July 28, 2015 at 11:13 AM >>>>>>> >>>>>>> To: "user@ranger.incubator.apache.org" >>>>>>> <user@ranger.incubator.apache.org> >>>>>>> Subject: Re: Error Setting up Ranger-Admin >>>>>>> >>>>>>> Luckily i have created the service myself. Thanks for your timely help. >>>>>>> >>>>>>> Can you please tell me now what should be the next step. I want to >>>>>>> explore all the functionality i.e., >>>>>>> >>>>>>> sync LDAP contacts >>>>>>> perform authorization etc >>>>>>> >>>>>>> On Tue, Jul 28, 2015 at 11:07 PM, Aneela Saleem <ane...@platalytics.com> >>>>>>> wrote: >>>>>>> There are no errors in NameNode logs. >>>>>>> >>>>>>> Can you please tell me how to create service/repo using Ranger Admin? >>>>>>> >>>>>>> On Tue, Jul 28, 2015 at 10:48 PM, Don Bosco Durai <bo...@apache.org> >>>>>>> wrote: >>>>>>> Yes, this is link I was about to redirect you to. >>>>>>> >>>>>>> Sorry, I might have assumed incorrectly on your initial question. >>>>>>> ./setup.sh is called only for installing RangerAdmin. Good it seems you >>>>>>> have made progress on installing RangerAdmin. >>>>>>> >>>>>>> I also assume, your ./enable-hdfs-plugin.sh went through fine. >>>>>>> >>>>>>> Looking into the Apache Ranger document, it seems it doesn’t explicitly >>>>>>> call out to create a repo/service for HDFS in RangerAdmin. In the >>>>>>> install.properties for HDFS, what value have you given for the property >>>>>>> “REPOSITORY_NAME”? You need to create a service/repo in RangerAdmin with >>>>>>> the same name. >>>>>>> >>>>>>> Also, after you restart HDFS namenode, can you check the namenode logs >>>>>>> for any error? >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Bosco >>>>>>> >>>>>>> >>>>>>> From: Aneela Saleem <ane...@platalytics.com> >>>>>>> Reply-To: "user@ranger.incubator.apache.org" >>>>>>> <user@ranger.incubator.apache.org> >>>>>>> Date: Tuesday, July 28, 2015 at 10:16 AM >>>>>>> >>>>>>> To: "user@ranger.incubator.apache.org" >>>>>>> <user@ranger.incubator.apache.org> >>>>>>> Subject: Re: Error Setting up Ranger-Admin >>>>>>> >>>>>>> And i'm using following link for installation: >>>>>>> >>>>>>> https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+0.5.0+I >>>>>>> nstallation >>>>>>> >>>>>>> I'm upto Enabling Ranger HDFS Plugins section (e) >>>>>>> * You can verify by logging into the Ranger Admin Web interface > Audit >>>>>>> > Agents. >>>>>>> * You can verify the plugin is communicating to Ranger admin in >>>>>>> Audit->plugins tab >>>>>>> >>>>>>> >>>>>>> where i don't see any audits ensuring hdfs plugin is communicating to >>>>>>> Ranger Admin. >>>>>>> >>>>>>> On Tue, Jul 28, 2015 at 10:11 PM, Aneela Saleem <ane...@platalytics.com> >>>>>>> wrote: >>>>>>> I'm planning to secure HDFS and i'm using Knox for authentication using >>>>>>> REST API. >>>>>>> >>>>>>> Yes i'm still able to access the RangerAdmin UI. I'm just curious how to >>>>>>> sync LDAP contacts in APache Ranger. >>>>>>> >>>>>>> Secondly if the document is related to 0.4 and i'm linked with 0.5, then >>>>>>> how should i proceed? Do you have any fruitful link, i should follow? >>>>>>> (being a novice) >>>>>>> >>>>>>> On Tue, Jul 28, 2015 at 10:04 PM, Don Bosco Durai <bo...@apache.org> >>>>>>> wrote: >>>>>>> Aneela >>>>>>> >>>>>>> The document you linked is for Apache Ranger 0.4. There subtle changes >>>>>>> in Apache Ranger 0.5. >>>>>>> >>>>>>> Just curious, how you were able to access RangerAdmin UI? Did the >>>>>>> installation continued after the install failure? >>>>>>> >>>>>>> Also, can you give additional information of your env? I can try >>>>>>> reproducing it. >>>>>>> >>>>>>> Also, which components are you planning secure? (HDFS, Hive, Hbase, >>>>>>> Solr, YARN, Storm, etc) >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Bosco >>>>>>> >>>>>>> >>>>>>> From: Aneela Saleem <ane...@platalytics.com> >>>>>>> Reply-To: "user@ranger.incubator.apache.org" >>>>>>> <user@ranger.incubator.apache.org> >>>>>>> Date: Tuesday, July 28, 2015 at 5:19 AM >>>>>>> To: "user@ranger.incubator.apache.org" >>>>>>> <user@ranger.incubator.apache.org> >>>>>>> Subject: Re: Error Setting up Ranger-Admin >>>>>>> >>>>>>> And secondly my Ranger UI does not Match the UI as shown in the >>>>>>> following link: >>>>>>> >>>>>>> http://pivotalhd.docs.pivotal.io/docs/ranger-user-guide.html#Item1.4.2 >>>>>>> >>>>>>> There is no Policy Manager Tab and no Manage Repository Thing. >>>>>>> >>>>>>> On Tue, Jul 28, 2015 at 5:17 PM, Aneela Saleem <ane...@platalytics.com> >>>>>>> wrote: >>>>>>> Hi Bosco, >>>>>>> >>>>>>> I did as you suggested but still getting same error. >>>>>>> >>>>>>> On Tue, Jul 28, 2015 at 4:49 AM, Don Bosco Durai <bo...@apache.org> >>>>>>> wrote: >>>>>>> The properties file looks good.. >>>>>>> >>>>>>> I am not sure whether it is because of python 2.7. Is it possible for >>>>>>> you to use python 2.6? >>>>>>> >>>>>>> Try changing the below to where 2.6 is installed. >>>>>>> >>>>>>> PYTHON_COMMAND_INVOKER=python >>>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Bosco >>>>>>> >>>>>>> >>>>>>> From: Aneela Saleem <ane...@platalytics.com> >>>>>>> Reply-To: "user@ranger.incubator.apache.org" >>>>>>> <user@ranger.incubator.apache.org> >>>>>>> Date: Monday, July 27, 2015 at 2:02 PM >>>>>>> To: "user@ranger.incubator.apache.org" >>>>>>> <user@ranger.incubator.apache.org> >>>>>>> Subject: Error Setting up Ranger-Admin >>>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I have followed this link >>>>>>> >>>>>>> https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+0.5.0+I >>>>>>> nstallation >>>>>>> >>>>>>> But when i run ./setupsh, i get the following error at the end: >>>>>>> >>>>>>> Traceback (most recent call last): >>>>>>> File "update_property.py", line 40, in <module> >>>>>>> >>>>>>> write_properties_to_xml(ranger_admin_site_xml_path,parameter_name,parame >>>>>>> ter_value) >>>>>>> File "update_property.py", line 21, in write_properties_to_xml >>>>>>> if(os.path.isfile(xml_path)): >>>>>>> File "/usr/lib/python2.7/genericpath.py", line 29, in isfile >>>>>>> st = os.stat(path) >>>>>>> TypeError: coercing to Unicode: need string or buffer, NoneType found >>>>>>> 2015-07-28 01:38:57,308 [E] Update property failed for: >>>>>>> >>>>>>> Attached is the install.proprties file. >>>>>>> >>>>>>> I'm new to Apache Knox, can anyone please guide me. I'm having troubles >>>>>>> in installing Apache Ranger. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>> >>> >
