We just reinstalled MapR client (4.0.2) with security patch on our NiFi servers as our IT enabled security. I will test NiFi connectivity to MapR tomorrow and update MapR-client.md with my findings. I need to explore how to save data in parquet format on MapR HDFS. Anybody has success using kites processor with MapR? Thanks -Sumo Sent from my iPad
> On Jun 14, 2016, at 3:55 PM, Andre <andre-li...@fucs.org> wrote: > > Ravi, > > "1. > https://github.com/xmlking/mapr-nifi-hadoop-libraries-bundle/blob/master/mapr-client.md > – has a step to configure fs.defaultFS, should I configure the cluster > resource manage here? Or should I get maprfs url from our IT who is > supporting the cluster?" > > Depends on your configuration, but generally the fs.defaultFS can be > configured just by using your own settings (and pointing the NiFi processor > to your settings). > > Ideally you should be connecting to a cluster defined under > /opt/mapr/conf/mapr-clusters.conf (assuming you haven't played with client > configs) > > Also have in mind that mapr-client.md file seems to be focused on insecure > MapR clusters. Are you using insecure (terrible choice), MapR Hybrid security > (not ideal IMHO) or Kerberos (recommended IMHO)? > > Cheers > >> On Wed, Jun 15, 2016 at 3:36 AM, Ravi Papisetti (rpapiset) >> <rpapi...@cisco.com> wrote: >> Hi, >> >> I have configure the way it is mentioned in below e-mail, still no luck:-(. >> >> I have two questions: >> 1. >> https://github.com/xmlking/mapr-nifi-hadoop-libraries-bundle/blob/master/mapr-client.md >> – has a step to configure fs.defaultFS, should I configure the cluster >> resource manage here? Or should I get maprfs url from our IT who is >> supporting the cluster? >> 2. Matt: What is MaprPR? Can you please elaborate >> >> Appreciate all your responses. >> >> Thanks, >> Ravi Papisetti >> Technical Leader >> Services Technology Incubation Center >> rpapi...@cisco.com >> Phone: +1 512 340 3377 >> >> <0B65E9AA-485A-49EE-9FB4-8600F4D55880[28].png> >> >> From: Matt Burgess <mattyb...@gmail.com> >> Reply-To: "users@nifi.apache.org" <users@nifi.apache.org> >> Date: Monday, June 13, 2016 at 8:26 PM >> To: "users@nifi.apache.org" <users@nifi.apache.org> >> Subject: Re: Writing files to MapR File system using putHDFS >> >> Sumo, >> >> I'll try the MapR PR with your additional settings below. If they work, >> they'll need to be added to the doc (or ideally, the profile if possible). >> That's what I suspected had been missing but haven't had a chance to try >> yet, will do that shortly :) >> >> Thanks, >> Matt >> >> On Jun 13, 2016, at 9:17 PM, Sumanth Chinthagunta <xmlk...@gmail.com> wrote: >> >>> >>> I had been using custom build nifi-hadoop-libraries-nar-0.6.1.nar that >>> worked with MapR 4.02 >>> make sure you add java.security.auth.login.config and follow mapR client >>> setup on the NiFi server >>> (https://github.com/xmlking/mapr-nifi-hadoop-libraries-bundle/blob/master/mapr-client.md) >>> >>> $NIFI_HOME/conf/bootstrap.conf >>> >>> java.arg.15=-Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf >>> >>> I just build Nar with MapR 2.7.0-mapr-1602 libs. I haven’t tested with MapR >>> 5.1 but you can try and let us know. >>> >>> Hadoop bundle for NiFi v0.6.1 and MapR 2.7.0-mapr-1602 >>> https://github.com/xmlking/mapr-nifi-hadoop-libraries-bundle/releases >>> >>> -Sumo >>> >>> >>>> On Jun 13, 2016, at 5:57 PM, Bryan Bende <bbe...@gmail.com> wrote: >>>> >>>> I'm not sure if this would make a difference, but typically the >>>> configuration resources would be the full paths to core-site.xml and >>>> hdfs-site.xml. Wondering if using those instead of yarn-site.xml changes >>>> anything. >>>> >>>>> On Monday, June 13, 2016, Ravi Papisetti (rpapiset) <rpapi...@cisco.com> >>>>> wrote: >>>>> Yes, Aldrin. Tried listHDFS, gets the similar error complaining directory >>>>> doesn't exist. >>>>> >>>>> NiFi – 0.6.1 >>>>> MapR – 5.1 >>>>> >>>>> NiFi is local standalone instance. Target cluster is enabled with token >>>>> based authentication. I am able to execute "hadoop fs –ls <path>" from >>>>> cli on the node with NiFi installed. >>>>> >>>>> Thanks, >>>>> Ravi Papisetti >>>>> Technical Leader >>>>> Services Technology Incubation Center >>>>> rpapi...@cisco.com >>>>> Phone: +1 512 340 3377 >>>>> >>>>> <0B65E9AA-485A-49EE-9FB4-8600F4D55880[22].png> >>>>> >>>>> From: Aldrin Piri <aldrinp...@gmail.com> >>>>> Reply-To: "users@nifi.apache.org" <users@nifi.apache.org> >>>>> Date: Monday, June 13, 2016 at 6:24 PM >>>>> To: "users@nifi.apache.org" <users@nifi.apache.org> >>>>> Subject: Re: Writing files to MapR File system using putHDFS >>>>> >>>>> Hi Ravi, >>>>> >>>>> Could you provide some additional details in terms of both your NiFi >>>>> environment and the MapR destination? >>>>> >>>>> Is your NiFi a single instance or clustered? In the case of the latter, >>>>> is security established for your ZooKeeper ensemble? >>>>> >>>>> Is your target cluster Kerberized? What version are you running? Have >>>>> you attempted to use the List/GetHDFS processors? Do they also have >>>>> errors in reading? >>>>> >>>>> Thanks! >>>>> --aldrin >>>>> >>>>>> On Mon, Jun 13, 2016 at 5:19 PM, Ravi Papisetti (rpapiset) >>>>>> <rpapi...@cisco.com> wrote: >>>>>> Thanks Conrad for your reply. >>>>>> >>>>>> Yes, I have configured putHDFS with "Remove Owner" and "Renive Group" >>>>>> with same values as on HDFS. Also, nifi service is started under the >>>>>> same user. >>>>>> >>>>>> >>>>>> >>>>>> Thanks, >>>>>> Ravi Papisetti >>>>>> Technical Leader >>>>>> Services Technology Incubation Center >>>>>> rpapi...@cisco.com >>>>>> Phone: +1 512 340 3377 >>>>>> >>>>>> <0B65E9AA-485A-49EE-9FB4-8600F4D55880[21].png> >>>>>> >>>>>> From: Conrad Crampton <conrad.cramp...@secdata.com> >>>>>> Reply-To: "users@nifi.apache.org" <users@nifi.apache.org> >>>>>> Date: Monday, June 13, 2016 at 4:01 PM >>>>>> To: "users@nifi.apache.org" <users@nifi.apache.org> >>>>>> Subject: Re: Writing files to MapR File system using putHDFS >>>>>> >>>>>> Hi, >>>>>> >>>>>> Sounds like a permissions problem. Have you set the Remote Owner and >>>>>> Remote Groups settings in the processor appropriate for HDFS permissions? >>>>>> >>>>>> Conrad >>>>>> >>>>>> >>>>>> >>>>>> From: "Ravi Papisetti (rpapiset)" <rpapi...@cisco.com> >>>>>> Reply-To: "users@nifi.apache.org" <users@nifi.apache.org> >>>>>> Date: Monday, 13 June 2016 at 21:25 >>>>>> To: "users@nifi.apache.org" <users@nifi.apache.org>, >>>>>> "d...@nifi.apache.org" <d...@nifi.apache.org> >>>>>> Subject: Writing files to MapR File system using putHDFS >>>>>> >>>>>> >>>>>> >>>>>> Hi, >>>>>> >>>>>> >>>>>> >>>>>> We just started exploring apache nifi for data onboarding into MapR >>>>>> distribution. Have configured putHDFS with yarn-site.xml from on local >>>>>> mapr client where cluster information is provided, configured the >>>>>> "Directory" with mapr fs directory to write the files, configured nifi >>>>>> to run as user has permission to write to mapr fs, inspie of that we are >>>>>> getting below error while writing the file into given file system path. >>>>>> I am doubting, nifi is not talking to the cluster or talking with wrong >>>>>> user, appreciate if you some can guide me to troubleshoot this issue or >>>>>> any solutions if we are doing something wrong: >>>>>> >>>>>> Nifi workflow is very simple: GetFile is configure to read from locla >>>>>> file system, connected to PutHDFS with yarn-site.xml and directory >>>>>> information configured. >>>>>> >>>>>> 2016-06-13 15:14:36,305 INFO [Timer-Driven Process Thread-2] >>>>>> o.apache.nifi.processors.hadoop.PutHDFS >>>>>> PutHDFS[id=07abcfaa-fa8d-496b-81f0-b1b770672719] Kerberos relogin >>>>>> successful or ticket still valid >>>>>> 2016-06-13 15:14:36,324 ERROR [Timer-Driven Process Thread-2] >>>>>> o.apache.nifi.processors.hadoop.PutHDFS >>>>>> PutHDFS[id=07abcfaa-fa8d-496b-81f0-b1b770672719] Failed to write to HDFS >>>>>> due to java.io.IOException: /app/DataAnalyticsFramework/catalog/nifi >>>>>> could not be created: java.io.IOException: >>>>>> /app/DataAnalyticsFramework/catalog/nifi could not be created >>>>>> 2016-06-13 15:14:36,330 ERROR [Timer-Driven Process Thread-2] >>>>>> o.apache.nifi.processors.hadoop.PutHDFS >>>>>> java.io.IOException: /app/DataAnalyticsFramework/catalog/nifi could not >>>>>> be created >>>>>> at >>>>>> org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:238) >>>>>> ~[nifi-hdfs-processors-0.6.1.jar:0.6.1] >>>>>> at >>>>>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) >>>>>> [nifi-api-0.6.1.jar:0.6.1] >>>>>> at >>>>>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1059) >>>>>> [nifi-framework-core-0.6.1.jar:0.6.1] >>>>>> at >>>>>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) >>>>>> [nifi-framework-core-0.6.1.jar:0.6.1] >>>>>> at >>>>>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) >>>>>> [nifi-framework-core-0.6.1.jar:0.6.1] >>>>>> at >>>>>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123) >>>>>> [nifi-framework-core-0.6.1.jar:0.6.1] >>>>>> at >>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>> [na:1.7.0_101] >>>>>> at >>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >>>>>> [na:1.7.0_101] >>>>>> at >>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >>>>>> [na:1.7.0_101] >>>>>> at >>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>>>>> [na:1.7.0_101] >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>> [na:1.7.0_101] >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>> [na:1.7.0_101] >>>>>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101] >>>>>> >>>>>> Appreciate any help. >>>>>> >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Ravi Papisetti >>>>>> >>>>>> Technical Leader >>>>>> >>>>>> Services Technology Incubation Center >>>>>> >>>>>> rpapi...@cisco.com >>>>>> >>>>>> Phone: +1 512 340 3377 >>>>>> >>>>>> >>>>>> >>>>>> <image001.png> >>>>>> >>>>>> >>>>>> >>>>>> ***This email originated outside SecureData*** >>>>>> >>>>>> Click here to report this email as spam. >>>>>> >>>>>> >>>>>> >>>>>> SecureData, combating cyber threats >>>>>> >>>>>> The information contained in this message or any of its attachments may >>>>>> be privileged and confidential and intended for the exclusive use of the >>>>>> intended recipient. If you are not the intended recipient any >>>>>> disclosure, reproduction, distribution or other dissemination or use of >>>>>> this communications is strictly prohibited. The views expressed in this >>>>>> email are those of the individual and not necessarily of SecureData >>>>>> Europe Ltd. Any prices quoted are only valid if followed up by a formal >>>>>> written quote. >>>>>> >>>>>> SecureData Europe Limited. Registered in England & Wales 04365896. >>>>>> Registered Address: SecureData House, Hermitage Court, Hermitage Lane, >>>>>> Maidstone, Kent, ME16 9NT >>>>>> >>>> >>>> >>>> -- >>>> Sent from Gmail Mobile >
