Hello Sudeep, Sorry, not following your emails, did you need more help importing the processor? Currently the way you would clear a DistributedMapCache is to just remove the DistributedMapCacheServer controller service and make a new one. Joe - - - - - - Joseph Percivalllinkedin.com/in/Percivalle: joeperciv...@yahoo.com
On Thursday, January 14, 2016 7:04 AM, sudeep mishra <sudeepshekh...@gmail.com> wrote: Thanks Joe. The GetDistributedMapCache seems to be working fine. Is there a way to clear DistributedMapCache on demand? Regards, Sudeep On Thu, Jan 14, 2016 at 12:42 PM, sudeep mishra <sudeepshekh...@gmail.com> wrote: Upon building the repository we get different .nar files which can be updated in the lib for my requirement. Thanks for your help. On Thu, Jan 14, 2016 at 9:27 AM, sudeep mishra <sudeepshekh...@gmail.com> wrote: Is it possible to build the code for only a particular processor? Just curious if we can build and deploy a particular processor in an existing NiFi environment. On Wed, Jan 13, 2016 at 9:33 PM, sudeep mishra <sudeepshekh...@gmail.com> wrote: Thanks Joe. I will try out the patch. On Wed, Jan 13, 2016 at 9:31 PM, Joe Percivall <joeperciv...@yahoo.com> wrote: You would need to clone the nifi source from github and then apply the patch using git. Here is how to clone a repo: https://help.github.com/articles/cloning-a-repository/ Along with the nifi repo itself: https://github.com/apache/nifi and how to apply a patch: http://makandracards.com/makandra/2521-git-how-to-create-and-apply-patches Let me know if you have any other questions, Joe - - - - - - Joseph Percivall linkedin.com/in/Percivall e: joeperciv...@yahoo.com On Wednesday, January 13, 2016 10:56 AM, sudeep mishra <sudeepshekh...@gmail.com> wrote: Thank you very much Joe. Can you please let me know how I can use the .patch file? I am using the NiFi via the binaries... Do I need to setup the source code and build the same along with the patch? Thanks & Regards, Sudeep On Wed, Jan 13, 2016 at 9:02 PM, Joe Percivall <joeperciv...@yahoo.com> wrote: Hello Sudeep, > >I put up a patch on the GetDistributedMapCache ticket[1]. Let me know what you >think. > >The PutDistributedMapCache processor and GetDistributedMapCache work with the >data as a byte[] so it should be format agnostic. That being said it will be >up to you to know what is in there in order to use it later. > >[1] https://issues.apache.org/jira/browse/NIFI-1382 > >Joe >- - - - - - >Joseph Percivall >linkedin.com/in/Percivall >e: joeperciv...@yahoo.com > > > > >On Tuesday, January 12, 2016 11:34 PM, sudeep mishra ><sudeepshekh...@gmail.com> wrote: > > > >Thanks Joe. > >I do not have specific configuration as of now as I am still exploring NiFi. >Though I think it would be helpful to let user store and retrieve the cache >values in different formats json, avro etc. > >Thanks & Regards, > >Sudeep > > > > > >On Tue, Jan 12, 2016 at 9:15 PM, Joe Percivall <joeperciv...@yahoo.com> wrote: > >Hello Sudeep, >> >> >>We are currently lacking a "GetDistributedMapCache" processor that >>corresponds to the "PutDistributedMapCache". I created a ticket[1] and will >>be working on it today. If you have any comments, configuration suggestions, >>etc. please let me know or comment on the ticket. >> >> >>[1] https://issues.apache.org/jira/browse/NIFI-1382 >> >>Joe >>- - - - - - >>Joseph Percivall >>linkedin.com/in/Percivall >>e: joeperciv...@yahoo.com >> >> >> >> >> >>On Tuesday, January 12, 2016 9:46 AM, sudeep mishra >><sudeepshekh...@gmail.com> wrote: >> >> >> >>Thanks Matt. >> >> >>In my data flow I am expected to perform certain validations on data. I am >>loading some SQLServer data into HDFSusing Sqoop (not part of NiFi flow). For >>each record in HDFS file I have to query another database and then save the >>validated record again in HDFS which will be processed bysome Spark jobs. >> >> >>Since I have to query for each record thus I was planning to cache the >>database records against which I have to validate the HDFS. Thus I was >>evaluating the DistributedCacheServer. But looks like its purpose is >>different. Alternatively can we integrate Redis or another distributed cache >>with NiFi as I do not see any processor for it. >> >> >>Appreciate your help. >> >> >>Thanks & Regards, >> >> >>Sudeep >> >> >> >> >>On Tue, Jan 12, 2016 at 6:59 PM, Matthew Clarke <matt.clarke....@gmail.com> >>wrote: >> >>Sudeep, >>> I was a little off on my second scenario. The detectduplicate >>>processor uses the distributedcache service all on its own.. Files that are >>>route through it are loaded into the cache if they do not already exist in >>>the cache. if they do already exist they are routed to duplicate. The >>>putDistributedCache processor was a community contribution to which there >>>are no processor that make use of the info that it caches. >>> >>> We should probably build a processor that would make use of the data >>>that can be loaded by the putDistributeCache processor. Is there a >>>particular use case you are trying to solve where this would be applicable? >>> >>> >>>Thanks, >>>Matt >>> >>> >>>On Tue, Jan 12, 2016 at 8:11 AM, Matthew Clarke <matt.clarke....@gmail.com> >>>wrote: >>> >>>Sudeep, >>>> The DistributedMapCache is typically used to prevent the consumption of >>>>duplicate data by some of the ingest type processors (GetHBASE, ListHDFS, >>>>and ListSFTP). NiFi uses the service to keep a listing of what has been >>>>consumed so the same files are not consumed multiple times. The Service can >>>>also be used to detect if duplicate data already exists within a NiFi >>>>Instance or cluster. This would be the scenario where some source is >>>>pushing data to your NiFi and perhaps they push the same data more than >>>>once. You want to catch these duplicates so you can perhaps kick them out >>>>of your flow. For this you would use the PutDistributedCache processor to >>>>cache all incoming data and then use the DetectDuplicate processor to find >>>>those duplicates. >>>> >>>> Was there a different use case you were looking to solve using the >>>>Distributed cache service? >>>> >>>> >>>>Thanks, >>>>Matt >>>> >>>> >>>>On Tue, Jan 12, 2016 at 4:36 AM, sudeep mishra <sudeepshekh...@gmail.com> >>>>wrote: >>>> >>>>Hi, >>>>> >>>>> >>>>>I can cache some data to be used in NiFi flow. I can see the processor >>>>>PutDistributedMapCache in the documentation which saves key-value pairs in >>>>>DistributedMapCache for NiFi but I do not see any processor to red this >>>>>data. How can I read data from DistributedMapCache in my data flow? >>>>> >>>>> >>>>> >>>>> >>>>>Thanks & Regards, >>>>> >>>>> >>>>>Sudeep Shekhar Mishra >>>>> >>>>> >>>> >>> >> >> >> >>-- >> >>Thanks & Regards, >> >> >>Sudeep Shekhar Mishra >> >> >>+91-9167519029 >>sudeepshekh...@gmail.com >> >> > > >-- > >Thanks & Regards, > >Sudeep Shekhar Mishra > >+91-9167519029 >sudeepshekh...@gmail.com > -- Thanks & Regards, Sudeep Shekhar Mishra +91-9167519029 sudeepshekh...@gmail.com -- Thanks & Regards, Sudeep Shekhar Mishra +91-9167519029sudeepshekh...@gmail.com -- Thanks & Regards, Sudeep Shekhar Mishra +91-9167519029sudeepshekh...@gmail.com -- Thanks & Regards, Sudeep Shekhar Mishra +91-9167519029sudeepshekh...@gmail.com -- Thanks & Regards, Sudeep Shekhar Mishra +91-9167519029sudeepshekh...@gmail.com