Yes the command is working On Wed, Jul 29, 2015 at 1:11 AM, Don Bosco Durai <bo...@apache.org> wrote:
> Did the original startup issue get resolved? > > Your errors seems to be coming from WebHDFS. Can we check whether “hdfs > dfs -ls /“ works from command line? > > Thanks > > Bosco > > > From: Aneela Saleem <ane...@platalytics.com> > Reply-To: "user@ranger.incubator.apache.org" < > user@ranger.incubator.apache.org> > Date: Tuesday, July 28, 2015 at 1:05 PM > > To: "user@ranger.incubator.apache.org" <user@ranger.incubator.apache.org> > Subject: Re: Error Setting up Ranger-Admin > > That error has been resolved. > > Following is the latest error: > > class org.apache.hadoop.hdfs.web.resources.UserProvider > class org.apache.hadoop.hdfs.web.resources.ExceptionHandler > Jul 28, 2015 10:39:46 PM > com.sun.jersey.server.impl.application.WebApplicationImpl _initiate > INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 > AM' > Jul 28, 2015 10:39:51 PM com.sun.jersey.spi.inject.Errors > processErrorMessages > WARNING: The following warnings have been detected with resource and/or > provider classes: > WARNING: A sub-resource method, public javax.ws.rs.core.Response > org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.getRoot(org.apache.hadoop.security.UserGroupInformation,org.apache.hadoop.hdfs.web.resources.DelegationParam,org.apache.hadoop.hdfs.web.resources.UserParam,org.apache.hadoop.hdfs.web.resources.DoAsParam,org.apache.hadoop.hdfs.web.resources.GetOpParam,org.apache.hadoop.hdfs.web.resources.OffsetParam,org.apache.hadoop.hdfs.web.resources.LengthParam,org.apache.hadoop.hdfs.web.resources.RenewerParam,org.apache.hadoop.hdfs.web.resources.BufferSizeParam,java.util.List,org.apache.hadoop.hdfs.web.resources.XAttrEncodingParam,org.apache.hadoop.hdfs.web.resources.ExcludeDatanodesParam,org.apache.hadoop.hdfs.web.resources.FsActionParam,org.apache.hadoop.hdfs.web.resources.TokenKindParam,org.apache.hadoop.hdfs.web.resources.TokenServiceParam) > throws java.io.IOException,java.lang.InterruptedException, with URI > template, "/", is treated as a resource method > > > > On Wed, Jul 29, 2015 at 12:50 AM, Don Bosco Durai <bo...@apache.org> > wrote: > >> I have seen this issue before in OpenStack environment where the DFS >> folders were configured to /tmp/… and they were wiped out after restart. >> >> If it is a test env, you can try formatting. First check dfs.name.dir >> property in HDFS. >> >> % $HADOOP_INSTALL/hadoop/bin/hadoop namenode -format >> >> http://wiki.apache.org/hadoop/GettingStartedWithHadoop >> Formatting the Namenode >> >> The first step to starting up your Hadoop installation is formatting the >> Hadoop filesystem, which is implemented on top of the local filesystems of >> your cluster. You need to do this the first time you set up a Hadoop >> installation. *Do not* format a running Hadoop filesystem, this will >> cause all your data to be erased. Before formatting, ensure that the >> dfs.name.dir directory exists. If you just used the default, then >> mkdir -p /tmp/hadoop-username/dfs/name will create the directory. To >> format the filesystem (which simply initializes the directory specified by >> the dfs.name.dir variable), run the command: >> % $HADOOP_INSTALL/hadoop/bin/hadoop namenode -format >> >> If asked to [re]format, you must reply Y (not just y) if you want to >> reformat, else Hadoop will abort the format. >> >> >> Thanks >> >> Bosco >> >> From: Aneela Saleem <ane...@platalytics.com> >> Reply-To: "user@ranger.incubator.apache.org" < >> user@ranger.incubator.apache.org> >> Date: Tuesday, July 28, 2015 at 12:40 PM >> >> To: "user@ranger.incubator.apache.org" <user@ranger.incubator.apache.org> >> Subject: Re: Error Setting up Ranger-Admin >> >> NameNode error Logs: >> >> 2015-06-30 00:30:05,933 FATAL >> org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode. >> java.io.IOException: NameNode is not formatted. >> >> On Wed, Jul 29, 2015 at 12:22 AM, Don Bosco Durai <bo...@apache.org> >> wrote: >> >>> Anything in the namenode log file? >>> >>> Thanks >>> >>> Bosco >>> >>> From: Aneela Saleem <ane...@platalytics.com> >>> Reply-To: "user@ranger.incubator.apache.org" < >>> user@ranger.incubator.apache.org> >>> Date: Tuesday, July 28, 2015 at 12:18 PM >>> To: "user@ranger.incubator.apache.org" <user@ranger.incubator.apache.org >>> > >>> Subject: Re: Error Setting up Ranger-Admin >>> >>> Hdfs Plugin request is not being shown in Audit > Plugins tab. I >>> followed all steps and verified with your steps. Restarted namenode but >>> still can't see any plugin request >>> >>> On Wed, Jul 29, 2015 at 12:03 AM, Aneela Saleem <ane...@platalytics.com> >>> wrote: >>> >>>> Thank you so much Bosco. I created JIRA for this. Can be found here >>>> <https://issues.apache.org/jira/browse/RANGER-582> >>>> >>>> Let me try UserSync, then i will approach yyou in case of any problem. >>>> >>>> On Tue, Jul 28, 2015 at 11:28 PM, Don Bosco Durai < >>>> bdu...@hortonworks.com> wrote: >>>> >>>>> Good. I already typed it, so let me paste it for the benefit of >>>>> others. We need to update the doc also. If you don’t mind, can you create >>>>> a >>>>> JIRA to track this? >>>>> >>>>> 1. Go to RangerAdmin UI -> AccessManager (first menu tab is >>>>> already opened by default) >>>>> 2. Click on “+” in the HDFS panel title >>>>> 3. Enter Service Name: e.g. “${cluster_name}_hdfs. Cluster name >>>>> could be anything. You can call the repo “test_hdfs” if you want to >>>>> 4. UserName: ranger (doesn’t matter whether it exists because you >>>>> are not using kerberos) >>>>> 5. Password: ranger (doesn’t matter what you give) >>>>> 6. Namenode URL: hdfs://<namenode_host>:8020 >>>>> 7. Authorization Enabled: No >>>>> 8. Authentication Type: Simple >>>>> 9. Rest you can leave it as default. >>>>> >>>>> Save this screen. Then use the same name e.g “test_hdfs” in your >>>>> install.properties for enabling Ranger in HDFS. Then restart namenode. >>>>> >>>>> Is the plugin request showing up in the Audit->Plugin tab? >>>>> >>>>> If the plugin is showing up, then that means authorization and >>>>> auditing is already working. >>>>> >>>>> For configuring policies, we need to configure UserSync. The >>>>> install/configure process is same for all processes. You need to update >>>>> the >>>>> install.properties with the appropriate values. >>>>> Set SYNC_SOURCE = ldap >>>>> And check the section "Table: LDAP/AD Properties with sample values” >>>>> for reference values. >>>>> >>>>> Please note that for using LDAP within Hadoop, you need to configure >>>>> Hadoop core-site.xml to use LDAP. You might be already knowing it. Just >>>>> want to make sure. >>>>> >>>>> Thanks >>>>> >>>>> Bosco >>>>> >>>>> >>>>> From: Aneela Saleem <ane...@platalytics.com> >>>>> Reply-To: "user@ranger.incubator.apache.org" < >>>>> user@ranger.incubator.apache.org> >>>>> Date: Tuesday, July 28, 2015 at 11:13 AM >>>>> >>>>> To: "user@ranger.incubator.apache.org" < >>>>> user@ranger.incubator.apache.org> >>>>> Subject: Re: Error Setting up Ranger-Admin >>>>> >>>>> Luckily i have created the service myself. Thanks for your timely >>>>> help. >>>>> >>>>> Can you please tell me now what should be the next step. I want to >>>>> explore all the functionality i.e., >>>>> >>>>> sync LDAP contacts >>>>> perform authorization etc >>>>> >>>>> On Tue, Jul 28, 2015 at 11:07 PM, Aneela Saleem < >>>>> ane...@platalytics.com> wrote: >>>>> >>>>>> There are no errors in NameNode logs. >>>>>> >>>>>> Can you please tell me how to create service/repo using Ranger Admin? >>>>>> >>>>>> On Tue, Jul 28, 2015 at 10:48 PM, Don Bosco Durai <bo...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> Yes, this is link I was about to redirect you to. >>>>>>> >>>>>>> Sorry, I might have assumed incorrectly on your initial question. >>>>>>> ./setup.sh is called only for installing RangerAdmin. Good it seems you >>>>>>> have made progress on installing RangerAdmin. >>>>>>> >>>>>>> I also assume, your ./enable-hdfs-plugin.sh went through fine. >>>>>>> >>>>>>> Looking into the Apache Ranger document, it seems it doesn’t >>>>>>> explicitly call out to create a repo/service for HDFS in RangerAdmin. In >>>>>>> the install.properties for HDFS, what value have you given for the >>>>>>> property >>>>>>> “REPOSITORY_NAME”? You need to create a service/repo in RangerAdmin >>>>>>> with the same name. >>>>>>> >>>>>>> Also, after you restart HDFS namenode, can you check the namenode >>>>>>> logs for any error? >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Bosco >>>>>>> >>>>>>> >>>>>>> From: Aneela Saleem <ane...@platalytics.com> >>>>>>> Reply-To: "user@ranger.incubator.apache.org" < >>>>>>> user@ranger.incubator.apache.org> >>>>>>> Date: Tuesday, July 28, 2015 at 10:16 AM >>>>>>> >>>>>>> To: "user@ranger.incubator.apache.org" < >>>>>>> user@ranger.incubator.apache.org> >>>>>>> Subject: Re: Error Setting up Ranger-Admin >>>>>>> >>>>>>> And i'm using following link for installation: >>>>>>> >>>>>>> >>>>>>> https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+0.5.0+Installation >>>>>>> >>>>>>> I'm upto *Enabling Ranger HDFS Plugins *section (e) >>>>>>> >>>>>>> - You can verify by logging into the Ranger Admin Web interface >>>>>>> > Audit > Agents. >>>>>>> - >>>>>>> >>>>>>> You can verify the plugin is communicating to Ranger admin in >>>>>>> Audit->plugins tab >>>>>>> >>>>>>> >>>>>>> where i don't see any audits ensuring hdfs plugin is communicating >>>>>>> to Ranger Admin. >>>>>>> >>>>>>> On Tue, Jul 28, 2015 at 10:11 PM, Aneela Saleem < >>>>>>> ane...@platalytics.com> wrote: >>>>>>> >>>>>>>> I'm planning to secure HDFS and i'm using Knox for authentication >>>>>>>> using REST API. >>>>>>>> >>>>>>>> Yes i'm still able to access the RangerAdmin UI. I'm just curious >>>>>>>> how to sync LDAP contacts in APache Ranger. >>>>>>>> >>>>>>>> Secondly if the document is related to 0.4 and i'm linked with 0.5, >>>>>>>> then how should i proceed? Do you have any fruitful link, i should >>>>>>>> follow? >>>>>>>> (being a novice) >>>>>>>> >>>>>>>> On Tue, Jul 28, 2015 at 10:04 PM, Don Bosco Durai <bo...@apache.org >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Aneela >>>>>>>>> >>>>>>>>> The document you linked is for Apache Ranger 0.4. There subtle >>>>>>>>> changes in Apache Ranger 0.5. >>>>>>>>> >>>>>>>>> Just curious, how you were able to access RangerAdmin UI? Did the >>>>>>>>> installation continued after the install failure? >>>>>>>>> >>>>>>>>> Also, can you give additional information of your env? I can try >>>>>>>>> reproducing it. >>>>>>>>> >>>>>>>>> Also, which components are you planning secure? (HDFS, Hive, >>>>>>>>> Hbase, Solr, YARN, Storm, etc) >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> Bosco >>>>>>>>> >>>>>>>>> >>>>>>>>> From: Aneela Saleem <ane...@platalytics.com> >>>>>>>>> Reply-To: "user@ranger.incubator.apache.org" < >>>>>>>>> user@ranger.incubator.apache.org> >>>>>>>>> Date: Tuesday, July 28, 2015 at 5:19 AM >>>>>>>>> To: "user@ranger.incubator.apache.org" < >>>>>>>>> user@ranger.incubator.apache.org> >>>>>>>>> Subject: Re: Error Setting up Ranger-Admin >>>>>>>>> >>>>>>>>> And secondly my Ranger UI does not Match the UI as shown in the >>>>>>>>> following link: >>>>>>>>> >>>>>>>>> >>>>>>>>> http://pivotalhd.docs.pivotal.io/docs/ranger-user-guide.html#Item1.4.2 >>>>>>>>> >>>>>>>>> There is no Policy Manager Tab and no Manage Repository Thing. >>>>>>>>> >>>>>>>>> On Tue, Jul 28, 2015 at 5:17 PM, Aneela Saleem < >>>>>>>>> ane...@platalytics.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Bosco, >>>>>>>>>> >>>>>>>>>> I did as you suggested but still getting same error. >>>>>>>>>> >>>>>>>>>> On Tue, Jul 28, 2015 at 4:49 AM, Don Bosco Durai < >>>>>>>>>> bo...@apache.org> wrote: >>>>>>>>>> >>>>>>>>>>> The properties file looks good.. >>>>>>>>>>> >>>>>>>>>>> I am not sure whether it is because of python 2.7. Is it >>>>>>>>>>> possible for you to use python 2.6? >>>>>>>>>>> >>>>>>>>>>> Try changing the below to where 2.6 is installed. >>>>>>>>>>> >>>>>>>>>>> PYTHON_COMMAND_INVOKER=python >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> >>>>>>>>>>> Bosco >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> From: Aneela Saleem <ane...@platalytics.com> >>>>>>>>>>> Reply-To: "user@ranger.incubator.apache.org" < >>>>>>>>>>> user@ranger.incubator.apache.org> >>>>>>>>>>> Date: Monday, July 27, 2015 at 2:02 PM >>>>>>>>>>> To: "user@ranger.incubator.apache.org" < >>>>>>>>>>> user@ranger.incubator.apache.org> >>>>>>>>>>> Subject: Error Setting up Ranger-Admin >>>>>>>>>>> >>>>>>>>>>> Hi all, >>>>>>>>>>> >>>>>>>>>>> I have followed this link >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+0.5.0+Installation >>>>>>>>>>> >>>>>>>>>>> But when i run *./setupsh, *i get the following error at the >>>>>>>>>>> end: >>>>>>>>>>> >>>>>>>>>>> Traceback (most recent call last): >>>>>>>>>>> File "update_property.py", line 40, in <module> >>>>>>>>>>> >>>>>>>>>>> write_properties_to_xml(ranger_admin_site_xml_path,parameter_name,parameter_value) >>>>>>>>>>> File "update_property.py", line 21, in write_properties_to_xml >>>>>>>>>>> if(os.path.isfile(xml_path)): >>>>>>>>>>> File "/usr/lib/python2.7/genericpath.py", line 29, in isfile >>>>>>>>>>> st = os.stat(path) >>>>>>>>>>> TypeError: coercing to Unicode: need string or buffer, NoneType >>>>>>>>>>> found >>>>>>>>>>> 2015-07-28 01:38:57,308 [E] Update property failed for: >>>>>>>>>>> >>>>>>>>>>> Attached is the *install.proprties* file. >>>>>>>>>>> >>>>>>>>>>> I'm new to Apache Knox, can anyone please guide me. I'm having >>>>>>>>>>> troubles in installing Apache Ranger. >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
