Re: Reg: NIFI-4787
Hi Joe, Please assign it to me. Regards, Sourav Gulati On Tue, 23 Jan 2018 at 9:58 PM, Sourav Gulati <soravgul...@gmail.com> wrote: > Hi Joe, > > I am still not able to assign it to myself. I can see two users with id: > soravgulati . Mine is one with Avatar. Please check > > Regards, > Sourav Gulati > > On Mon, Jan 22, 2018 at 11:13 PM, Joe Witt <joew...@apache.org> wrote: > >> Hello >> >> I've added you as a contributor so you should be good to go now. >> >> Thanks for contributing. >> >> Joe >> >> On Mon, Jan 22, 2018 at 10:00 AM, Sourav Gulati <soravgul...@gmail.com> >> wrote: >> > Hi Jewitt, >> > >> > I have started working on NIFI-4787. My username is : soravgulati . How >> can >> > I assign this issue to me? >> > >> > Regards, >> > Sourav Gulati >> > >
Re: Reg: NIFI-4787
Hi Joe, I am still not able to assign it to myself. I can see two users with id: soravgulati . Mine is one with Avatar. Please check Regards, Sourav Gulati On Mon, Jan 22, 2018 at 11:13 PM, Joe Witt <joew...@apache.org> wrote: > Hello > > I've added you as a contributor so you should be good to go now. > > Thanks for contributing. > > Joe > > On Mon, Jan 22, 2018 at 10:00 AM, Sourav Gulati <soravgul...@gmail.com> > wrote: > > Hi Jewitt, > > > > I have started working on NIFI-4787. My username is : soravgulati . How > can > > I assign this issue to me? > > > > Regards, > > Sourav Gulati >
RE: Exception while restarting the Nifi Cluster
Hi Nifi Team, We analyzed the issue and found that it is because of custom processor library missing in lib. Steps to reproduce: 1. Create nar of custom processor and add it to the lib. 2. Start nifi and create a flow using that custom processor. 3. Stop nifi and remove the nar of custom processor 4. Start nifi While starting it throws this exception. I think instead of Nifi instance not getting started and throwing error in such case. It should start and that particular flow remains disabled which is using this library. Regards, Sourav Gulati -Original Message- From: dale.chang13 [mailto:dale.chan...@outlook.com] Sent: Wednesday, May 11, 2016 5:24 PM To: dev@nifi.apache.org Subject: Re: Exception while restarting the Nifi Cluster Rahul Dahiya wrote > Hi Team, > > > I am getting below exception while trying to restart the NiFi nodes : > > > java.lang.Exception: Unable to load flow due to: java.io.IOException: > org.apache.nifi.cluster.ConnectionException: Failed to connect node to > cluster because local flow controller partially updated. > Administrator should disconnect node and review flow for corruption. > at > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:783) > ~[nifi-jetty-0.6.1.jar:0.6.1] > at org.apache.nifi.NiFi. > > (NiFi.java:137) [nifi-runtime-0.6.1.jar:0.6.1] > at org.apache.nifi.NiFi.main(NiFi.java:227) > [nifi-runtime-0.6.1.jar:0.6.1] > > > > Based on the following link : > > https://mail-archives.apache.org/mod_mbox/nifi-dev/201508.mbox/% > 3CBAY172-W19DF8C4EDE001FC6B8CA0ECE6B0@ > %3E > > > it seems that the issue could be because of trailing while space / > incorrect entries in the following properties files > > > nifi.sensitive.props.key > nifi.sensitive.props.algorithm > nifi.sensitive.props.provider > > > I've checked these properties files on all the nodes and they are > exactly the same on all nodes with no trailing white space. > > > Can someone please help on what could be the root cause of this > problem and how can it be resolved . > > > Also I don't want to clean the nifi working directories (I know it > will work fine on cleaning the directories). Thanks in advance for the help. > > > Regards, > > Rahul Could you give us the rest of the stack trace? It should contain more information that would help us further diagnose your problem -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Exception-while-restarting-the-Nifi-Cluster-tp10135p10137.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com. NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Reg: Get files from ftp
Mark, Does zookeeper process runs inside Nifi JVM? If yes, what is the default path of zookeeper data directory? Regards, Sourav Gulati -Original Message- From: Mark Payne [mailto:marka...@hotmail.com] Sent: Wednesday, May 11, 2016 6:37 PM To: dev@nifi.apache.org Subject: Re: Reg: Get files from ftp Sourav, If your NiFi instance is clustered, it will store the information in ZooKeeper. If not clustered, it will store the state in a local file. This is done because in a cluster, you typically want to run your List*** Processors on Primary Node only, and this allows another node to pick up where the previous one left off if the Primary Node changes. Of course, storing all of the files that have been listed can become very verbose so it stores only a small amount of data -- the timestamp of the latest file discovered and the timestamp of the latest file process/listed. It can then use this information to determine if files are new or modified without storing much info. Thanks -Mark > On May 11, 2016, at 12:39 AM, Sourav Gulati <sourav.gul...@impetus.co.in> > wrote: > > Thanks Matthew, > A quick question: Where does it store the state of files already listed? > > > Regards, > Sourav Gulati > > -Original Message- > From: Matthew Clarke [mailto:matt.clarke@gmail.com] > Sent: Wednesday, May 11, 2016 3:37 AM > To: dev@nifi.apache.org > Subject: Re: Reg: Get files from ftp > > The list type processors are designed to use NiFi state management to keep > from listing the same files twice. The fetch type processors with retrieve > files based on the FlowFiles it is fed. Typically those FlowFiles it works > from come from the corresponding list processor. > On May 10, 2016 8:56 AM, "Mark Payne" <marka...@hotmail.com> wrote: > >> Sourav, >> >> Sure. Within the nifi-standard-processors bundle are a few classes >> that would be important here. >> First is the AbstractListProcessor. You'll want to use this as your >> base class for ListFTP. Also, FetchFileTransfer will be the class >> that you'll extend for the FetchFTP processor. >> >> The ListSFTP and FetchSFTP are great examples to look at as examples. >> >> Additionally, the GetFTP and GetSFTP are good examples to look at as >> to how the FTP & SFTP implementations differ. They basically differ >> in the Property Descriptors provided and the FileTransfer object that >> is used. >> >> If you have any questions, please feel free to reach out to this >> mailing list. Very happy to help however we can! >> >> Thanks >> -Mark >> >> >>> On May 10, 2016, at 1:30 AM, Sourav Gulati >>> <sourav.gul...@impetus.co.in> >> wrote: >>> >>> Sure Mark. I am interested to work on it. Please provide some >>> pointers >> regarding that. >>> >>> Also, I will check if Sftp can be used. So ListSFTP / FetchSFTP >>> won't >> pick files more than once? >>> >>> Regards, >>> Sourav Gulati >>> >>> -Original Message- >>> From: Mark Payne [mailto:marka...@hotmail.com] >>> Sent: Monday, May 09, 2016 5:34 PM >>> To: dev@nifi.apache.org >>> Subject: Re: Reg: Get files from ftp >>> >>> Sourav, >>> >>> We have begun transitioning from many of the Get*** Processors to >> List*** and Fetch*** Processors. >>> There is a ListSFTP / FetchSFTP processor set but not currently a >> List/Fetch FTP. Is SFTP a possibility for you? Would you be >> interested in working on a List/Fetch FTP Processor set? >>> >>> Thanks >>> -Mark >>> >>>> On May 9, 2016, at 5:48 AM, Sourav Gulati >>>> <sourav.gul...@impetus.co.in> >> wrote: >>>> >>>> Hi Team, >>>> >>>> I need a suggestion. >>>> >>>> I want to get files from ftp server for which GetFtp processor is >> available. However, as I cannot delete files from source, I need to >> put a check that this processor does not pick a file more than once. >> What is the best way to do that? >>>> >>>> Regards, >>>> Sourav Gulati >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> NOTE: This message may contain information that is confidential, >> proprietary, privileged or otherwise protected by law. The message is >> intended solely for the named addressee. If received in error, please >> destroy and notify the sender.
RE: Reg: Get files from ftp
Thanks Matthew, A quick question: Where does it store the state of files already listed? Regards, Sourav Gulati -Original Message- From: Matthew Clarke [mailto:matt.clarke@gmail.com] Sent: Wednesday, May 11, 2016 3:37 AM To: dev@nifi.apache.org Subject: Re: Reg: Get files from ftp The list type processors are designed to use NiFi state management to keep from listing the same files twice. The fetch type processors with retrieve files based on the FlowFiles it is fed. Typically those FlowFiles it works from come from the corresponding list processor. On May 10, 2016 8:56 AM, "Mark Payne" <marka...@hotmail.com> wrote: > Sourav, > > Sure. Within the nifi-standard-processors bundle are a few classes > that would be important here. > First is the AbstractListProcessor. You'll want to use this as your > base class for ListFTP. Also, FetchFileTransfer will be the class that > you'll extend for the FetchFTP processor. > > The ListSFTP and FetchSFTP are great examples to look at as examples. > > Additionally, the GetFTP and GetSFTP are good examples to look at as > to how the FTP & SFTP implementations differ. They basically differ in > the Property Descriptors provided and the FileTransfer object that is > used. > > If you have any questions, please feel free to reach out to this > mailing list. Very happy to help however we can! > > Thanks > -Mark > > > > On May 10, 2016, at 1:30 AM, Sourav Gulati > > <sourav.gul...@impetus.co.in> > wrote: > > > > Sure Mark. I am interested to work on it. Please provide some > > pointers > regarding that. > > > > Also, I will check if Sftp can be used. So ListSFTP / FetchSFTP > > won't > pick files more than once? > > > > Regards, > > Sourav Gulati > > > > -Original Message- > > From: Mark Payne [mailto:marka...@hotmail.com] > > Sent: Monday, May 09, 2016 5:34 PM > > To: dev@nifi.apache.org > > Subject: Re: Reg: Get files from ftp > > > > Sourav, > > > > We have begun transitioning from many of the Get*** Processors to > List*** and Fetch*** Processors. > > There is a ListSFTP / FetchSFTP processor set but not currently a > List/Fetch FTP. Is SFTP a possibility for you? Would you be interested > in working on a List/Fetch FTP Processor set? > > > > Thanks > > -Mark > > > >> On May 9, 2016, at 5:48 AM, Sourav Gulati > >> <sourav.gul...@impetus.co.in> > wrote: > >> > >> Hi Team, > >> > >> I need a suggestion. > >> > >> I want to get files from ftp server for which GetFtp processor is > available. However, as I cannot delete files from source, I need to > put a check that this processor does not pick a file more than once. > What is the best way to do that? > >> > >> Regards, > >> Sourav Gulati > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> NOTE: This message may contain information that is confidential, > proprietary, privileged or otherwise protected by law. The message is > intended solely for the named addressee. If received in error, please > destroy and notify the sender. Any use of this email is prohibited > when received in error. Impetus does not represent, warrant and/or > guarantee, that the integrity of this communication has been > maintained nor that the communication is free of errors, virus, interception > or interference. > > > > > > > > > > > > > > > > > > > > NOTE: This message may contain information that is confidential, > proprietary, privileged or otherwise protected by law. The message is > intended solely for the named addressee. If received in error, please > destroy and notify the sender. Any use of this email is prohibited > when received in error. Impetus does not represent, warrant and/or > guarantee, that the integrity of this communication has been > maintained nor that the communication is free of errors, virus, interception > or interference. > > NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Reg: Nifi Clustering
Another question Mark, If I have provided URL of one of the node and that node goes down. Will it still work? Regards, Sourav Gulati -Original Message- From: Sourav Gulati [mailto:sourav.gul...@impetus.co.in] Sent: Tuesday, May 10, 2016 10:57 AM To: dev@nifi.apache.org Subject: RE: Reg: Nifi Clustering Thanks Mark for the information Regards, Sourav Gulati -Original Message- From: Mark Payne [mailto:marka...@hotmail.com] Sent: Monday, May 09, 2016 5:33 PM To: dev@nifi.apache.org Subject: Re: Reg: Nifi Clustering Sourav, Absolutely, Site-to-Site is a key enabler of NiFi and certainly will continue to be supported. Whereas right now you enter the URL of the NCM, you would simply enter the URL of any of the nodes in the cluster. Your instance would then reach out to that node and get a list of other nodes in the cluster. This will be similar to how the user will access the application, as well. In 1.0, since there will be no NCM to connect to, you will be able to connect to any node in the cluster, and it should be able to perform the necessary functions. Thanks -Mark > On May 9, 2016, at 5:15 AM, Sourav Gulati <sourav.gul...@impetus.co.in> wrote: > > Hello Team, > > We are doing some poc with Nifi . IF successful, we may be using it in > production some time in future. We are using site-to-site in our poc. As > Joe, in the upcoming 1.0 release work is being done to eliminate this notion > of master/slave altogether. > > Will site-to-site be still supported? Reason I am asking this is currently > while configuring site-to-site we need to provide URL of NCM in Remote > process Group. > > > Regards, > Sourav Gulati > > -Original Message- > From: Matthew Clarke [mailto:matt.clarke@gmail.com] > Sent: Tuesday, May 03, 2016 4:41 PM > To: dev@nifi.apache.org > Subject: RE: Reg: Nifi Clustering > > Sourav, > A single instance of NiFi cannot be both a NCM and a Node at the same time. > In order to have both a NCM and a Node on a single server, you will need to > install two copies of NiFi. One will be configured to be the NCM while the > other is configured to be the Node. As Joe pointed out, these two instance of > NiFi cannot share any listening ports (HTTP or S2S ports) in their configs. > > Matt > On May 3, 2016 2:03 AM, "Sourav Gulati" <sourav.gul...@impetus.co.in> wrote: > > Hi Joe, > > This is what I am getting while running both master and node > > IllegalStateException: Application may be configured as a cluster > manager or a node, but not both > > > > Regards, > Sourav Gulati > > -Original Message- > From: Joe Witt [mailto:joe.w...@gmail.com] > Sent: Monday, May 02, 2016 6:45 PM > To: dev@nifi.apache.org > Subject: Re: Reg: Nifi Clustering > > Sourav, > > The installation procedure does not differ. However, you do need to be > careful to ensure you're not trying to use the same ports for both the master > and the node. > > If you're having trouble go ahead and send some of the configuration details > over and someone can try to assist. > > Thanks > Joe > > On Mon, May 2, 2016 at 9:05 AM, Sourav Gulati > <sourav.gul...@impetus.co.in> > wrote: >> Hello Joe, >> >> Could you please provide steps of creating cluster on single node in > current version? >> >> Regards, >> Sourav Gulati >> >> -Original Message- >> From: Joe Witt [mailto:joe.w...@gmail.com] >> Sent: Monday, May 02, 2016 6:28 PM >> To: dev@nifi.apache.org >> Subject: Re: Reg: Nifi Clustering >> >> Sourav, >> >> Yes. And in the upcoming 1.0 release work is being done to eliminate > this notion of master/slave altogether and move to instead that at any time > any node can carry the responsibility of being a coordinator for the cluster. >> >> Thanks >> Joe >> >> On Mon, May 2, 2016 at 8:49 AM, Sourav Gulati >> <sourav.gul...@impetus.co.in> > wrote: >>> Hi Team, >>> >>> While creating Nifi Cluster, Can I run master and slave on same node? >>> >>> Regards, >>> Sourav Gulati >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> NOTE: This message may contain information that is confidential, > proprietary, privileged or otherwise protected by law. The message is > intended solely for the named addressee. If received in error, please destroy > and notify the sender. Any use of this email is prohibited when received in > error. Impetus does not represent, warrant and/or guarantee, that the > integrity of this comm
RE: Reg: Nifi Clustering
Thanks Mark for the information Regards, Sourav Gulati -Original Message- From: Mark Payne [mailto:marka...@hotmail.com] Sent: Monday, May 09, 2016 5:33 PM To: dev@nifi.apache.org Subject: Re: Reg: Nifi Clustering Sourav, Absolutely, Site-to-Site is a key enabler of NiFi and certainly will continue to be supported. Whereas right now you enter the URL of the NCM, you would simply enter the URL of any of the nodes in the cluster. Your instance would then reach out to that node and get a list of other nodes in the cluster. This will be similar to how the user will access the application, as well. In 1.0, since there will be no NCM to connect to, you will be able to connect to any node in the cluster, and it should be able to perform the necessary functions. Thanks -Mark > On May 9, 2016, at 5:15 AM, Sourav Gulati <sourav.gul...@impetus.co.in> wrote: > > Hello Team, > > We are doing some poc with Nifi . IF successful, we may be using it in > production some time in future. We are using site-to-site in our poc. As > Joe, in the upcoming 1.0 release work is being done to eliminate this notion > of master/slave altogether. > > Will site-to-site be still supported? Reason I am asking this is currently > while configuring site-to-site we need to provide URL of NCM in Remote > process Group. > > > Regards, > Sourav Gulati > > -Original Message- > From: Matthew Clarke [mailto:matt.clarke@gmail.com] > Sent: Tuesday, May 03, 2016 4:41 PM > To: dev@nifi.apache.org > Subject: RE: Reg: Nifi Clustering > > Sourav, > A single instance of NiFi cannot be both a NCM and a Node at the same time. > In order to have both a NCM and a Node on a single server, you will need to > install two copies of NiFi. One will be configured to be the NCM while the > other is configured to be the Node. As Joe pointed out, these two instance of > NiFi cannot share any listening ports (HTTP or S2S ports) in their configs. > > Matt > On May 3, 2016 2:03 AM, "Sourav Gulati" <sourav.gul...@impetus.co.in> wrote: > > Hi Joe, > > This is what I am getting while running both master and node > > IllegalStateException: Application may be configured as a cluster > manager or a node, but not both > > > > Regards, > Sourav Gulati > > -Original Message- > From: Joe Witt [mailto:joe.w...@gmail.com] > Sent: Monday, May 02, 2016 6:45 PM > To: dev@nifi.apache.org > Subject: Re: Reg: Nifi Clustering > > Sourav, > > The installation procedure does not differ. However, you do need to be > careful to ensure you're not trying to use the same ports for both the master > and the node. > > If you're having trouble go ahead and send some of the configuration details > over and someone can try to assist. > > Thanks > Joe > > On Mon, May 2, 2016 at 9:05 AM, Sourav Gulati > <sourav.gul...@impetus.co.in> > wrote: >> Hello Joe, >> >> Could you please provide steps of creating cluster on single node in > current version? >> >> Regards, >> Sourav Gulati >> >> -Original Message- >> From: Joe Witt [mailto:joe.w...@gmail.com] >> Sent: Monday, May 02, 2016 6:28 PM >> To: dev@nifi.apache.org >> Subject: Re: Reg: Nifi Clustering >> >> Sourav, >> >> Yes. And in the upcoming 1.0 release work is being done to eliminate > this notion of master/slave altogether and move to instead that at any time > any node can carry the responsibility of being a coordinator for the cluster. >> >> Thanks >> Joe >> >> On Mon, May 2, 2016 at 8:49 AM, Sourav Gulati >> <sourav.gul...@impetus.co.in> > wrote: >>> Hi Team, >>> >>> While creating Nifi Cluster, Can I run master and slave on same node? >>> >>> Regards, >>> Sourav Gulati >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> NOTE: This message may contain information that is confidential, > proprietary, privileged or otherwise protected by law. The message is > intended solely for the named addressee. If received in error, please destroy > and notify the sender. Any use of this email is prohibited when received in > error. Impetus does not represent, warrant and/or guarantee, that the > integrity of this communication has been maintained nor that the > communication is free of errors, virus, interception or interference. >> >> >> >> >> >> >> >> >> NOTE: This message may contain information that is confidential, > proprietary, privileg
Reg: Get files from ftp
Hi Team, I need a suggestion. I want to get files from ftp server for which GetFtp processor is available. However, as I cannot delete files from source, I need to put a check that this processor does not pick a file more than once. What is the best way to do that? Regards, Sourav Gulati NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: PutKafka-Message Delimiter
Hello Joe,Oleg, I was trying "\n" for new line .Please refer nifi_kafka_1.png for the same. Then I tried pressing ctrl+enter and click ok. It says "no value set". Please refer nifi_kafka_2.png for the same Regards, Sourav Gulati -Original Message- From: Joe Witt [mailto:joe.w...@gmail.com] Sent: Thursday, May 05, 2016 6:34 PM To: dev@nifi.apache.org Subject: Re: PutKafka-Message Delimiter Sourav, Could you please send a screenshot of your processor configuration? What we're looking to help you do is enter a new-line character to the delimiter property. You cannot just enter the literal string '\n' but you can hit the "enter" key. To do that you click on that property value like you're about to type characters so it brings up the text box. Then you press "CTRL+Enter" and then hit enter or click the ok button. You should then notice that it will no longer suggest 'no value set'. So we'll see that in the screen shot. You're on the right track we just need to get this last bit done. Thanks Joe On Thu, May 5, 2016 at 8:51 AM, Sourav Gulati <sourav.gul...@impetus.co.in> wrote: > Oleg, > > Yes it is not a required property. > > As per the documentation, If we do not provide any message delimiter to > PutKafka, it writes all the content of a flowfile as single message to > Kafka. My requirement is that I want to write all messages in the flow file > as separate messages on Kafka. > > Messages in the flowfile is delimited by "/n" . That is why I was providing > "/n" as message delimiter. However, it is not working. > > Regards, > Sourav Gulati > > -Original Message- > From: Oleg Zhurakousky [mailto:ozhurakou...@hortonworks.com] > Sent: Thursday, May 05, 2016 6:08 PM > To: dev@nifi.apache.org > Subject: Re: PutKafka-Message Delimiter > > I am not sure what you’re saying since it is not a required property and it > uses non empty validator so anything you enter there is a valid value. In > other words it can never produce 'no value set’ message. > I just tested it myself as well to be sure. > Could you provide more details as to what exactly you are doing? > > Cheers > Oleg >> On May 5, 2016, at 7:31 AM, Sourav Gulati <sourav.gul...@impetus.co.in> >> wrote: >> >> Could you please provide me that documentation content? >> >> Regards, >> Sourav Gulati >> >> -Original Message- >> From: Sourav Gulati >> Sent: Thursday, May 05, 2016 5:01 PM >> To: dev@nifi.apache.org >> Subject: RE: PutKafka-Message Delimiter >> >> If I press " CTRL+Enter" on the value table of "Message delimiter" key. It >> says "no value set". >> >> >> >> Regards, >> Sourav Gulati >> >> -Original Message- >> From: Oleg Zhurakousky [mailto:ozhurakou...@hortonworks.com] >> Sent: Thursday, May 05, 2016 4:54 PM >> To: dev@nifi.apache.org >> Subject: Re: PutKafka-Message Delimiter >> >> Yes, we've also updated documentation as well for the upcoming >> release >> >> Sent from my iPhone >> >>> On May 5, 2016, at 07:22, Sourav Gulati <sourav.gul...@impetus.co.in> wrote: >>> >>> Should I press CTRL+Enter on value tab of "Message delimiter" key? >>> >>> Regards, >>> Sourav Gulati >>> >>> -Original Message- >>> From: Oleg Zhurakousky [mailto:ozhurakou...@hortonworks.com] >>> Sent: Thursday, May 05, 2016 4:41 PM >>> To: dev@nifi.apache.org >>> Subject: Re: PutKafka-Message Delimiter >>> >>> Please use Ctrl+Enter. >>> The '\n' is interpreted literally as a string (not new line). >>> Let me know how it goes >>> Cheers >>> Oleg >>> >>>> On May 5, 2016, at 7:01 AM, Sourav Gulati <sourav.gul...@impetus.co.in> >>>> wrote: >>>> >>>> Hi All, >>>> >>>> I am tried setting message delimiter in put kafka to /n . But it is not >>>> working. Please guide. >>>> >>>> Regards, >>>> Sourav Gulati >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> NOTE: This message may contain information that is confidential, >>>> proprietary, privileged or otherwise protected by law. The message is >>>> intended solely for the named addressee. If received in error, please >>>> destroy and notify the sender. Any use of this
RE: PutKafka-Message Delimiter
Oleg, Yes it is not a required property. As per the documentation, If we do not provide any message delimiter to PutKafka, it writes all the content of a flowfile as single message to Kafka. My requirement is that I want to write all messages in the flow file as separate messages on Kafka. Messages in the flowfile is delimited by "/n" . That is why I was providing "/n" as message delimiter. However, it is not working. Regards, Sourav Gulati -Original Message- From: Oleg Zhurakousky [mailto:ozhurakou...@hortonworks.com] Sent: Thursday, May 05, 2016 6:08 PM To: dev@nifi.apache.org Subject: Re: PutKafka-Message Delimiter I am not sure what you’re saying since it is not a required property and it uses non empty validator so anything you enter there is a valid value. In other words it can never produce 'no value set’ message. I just tested it myself as well to be sure. Could you provide more details as to what exactly you are doing? Cheers Oleg > On May 5, 2016, at 7:31 AM, Sourav Gulati <sourav.gul...@impetus.co.in> wrote: > > Could you please provide me that documentation content? > > Regards, > Sourav Gulati > > -Original Message- > From: Sourav Gulati > Sent: Thursday, May 05, 2016 5:01 PM > To: dev@nifi.apache.org > Subject: RE: PutKafka-Message Delimiter > > If I press " CTRL+Enter" on the value table of "Message delimiter" key. It > says "no value set". > > > > Regards, > Sourav Gulati > > -Original Message- > From: Oleg Zhurakousky [mailto:ozhurakou...@hortonworks.com] > Sent: Thursday, May 05, 2016 4:54 PM > To: dev@nifi.apache.org > Subject: Re: PutKafka-Message Delimiter > > Yes, we've also updated documentation as well for the upcoming release > > Sent from my iPhone > >> On May 5, 2016, at 07:22, Sourav Gulati <sourav.gul...@impetus.co.in> wrote: >> >> Should I press CTRL+Enter on value tab of "Message delimiter" key? >> >> Regards, >> Sourav Gulati >> >> -Original Message- >> From: Oleg Zhurakousky [mailto:ozhurakou...@hortonworks.com] >> Sent: Thursday, May 05, 2016 4:41 PM >> To: dev@nifi.apache.org >> Subject: Re: PutKafka-Message Delimiter >> >> Please use Ctrl+Enter. >> The '\n' is interpreted literally as a string (not new line). >> Let me know how it goes >> Cheers >> Oleg >> >>> On May 5, 2016, at 7:01 AM, Sourav Gulati <sourav.gul...@impetus.co.in> >>> wrote: >>> >>> Hi All, >>> >>> I am tried setting message delimiter in put kafka to /n . But it is not >>> working. Please guide. >>> >>> Regards, >>> Sourav Gulati >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> NOTE: This message may contain information that is confidential, >>> proprietary, privileged or otherwise protected by law. The message is >>> intended solely for the named addressee. If received in error, please >>> destroy and notify the sender. Any use of this email is prohibited when >>> received in error. Impetus does not represent, warrant and/or guarantee, >>> that the integrity of this communication has been maintained nor that the >>> communication is free of errors, virus, interception or interference. >> >> >> >> >> >> >> >> >> >> NOTE: This message may contain information that is confidential, >> proprietary, privileged or otherwise protected by law. The message is >> intended solely for the named addressee. If received in error, please >> destroy and notify the sender. Any use of this email is prohibited when >> received in error. Impetus does not represent, warrant and/or guarantee, >> that the integrity of this communication has been maintained nor that the >> communication is free of errors, virus, interception or interference. > > > > > > > > > NOTE: This message may contain information that is confidential, proprietary, > privileged or otherwise protected by law. The message is intended solely for > the named addressee. If received in error, please destroy and notify the > sender. Any use of this email is prohibited when received in error. Impetus > does not represent, warrant and/or guarantee, that the integrity of this > communication has been maintained nor that the communication is free of > errors, virus, interception or interference. > NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: PutKafka-Message Delimiter
Could you please provide me that documentation content? Regards, Sourav Gulati -Original Message- From: Sourav Gulati Sent: Thursday, May 05, 2016 5:01 PM To: dev@nifi.apache.org Subject: RE: PutKafka-Message Delimiter If I press " CTRL+Enter" on the value table of "Message delimiter" key. It says "no value set". Regards, Sourav Gulati -Original Message- From: Oleg Zhurakousky [mailto:ozhurakou...@hortonworks.com] Sent: Thursday, May 05, 2016 4:54 PM To: dev@nifi.apache.org Subject: Re: PutKafka-Message Delimiter Yes, we've also updated documentation as well for the upcoming release Sent from my iPhone > On May 5, 2016, at 07:22, Sourav Gulati <sourav.gul...@impetus.co.in> wrote: > > Should I press CTRL+Enter on value tab of "Message delimiter" key? > > Regards, > Sourav Gulati > > -Original Message- > From: Oleg Zhurakousky [mailto:ozhurakou...@hortonworks.com] > Sent: Thursday, May 05, 2016 4:41 PM > To: dev@nifi.apache.org > Subject: Re: PutKafka-Message Delimiter > > Please use Ctrl+Enter. > The '\n' is interpreted literally as a string (not new line). > Let me know how it goes > Cheers > Oleg > >> On May 5, 2016, at 7:01 AM, Sourav Gulati <sourav.gul...@impetus.co.in> >> wrote: >> >> Hi All, >> >> I am tried setting message delimiter in put kafka to /n . But it is not >> working. Please guide. >> >> Regards, >> Sourav Gulati >> >> >> >> >> >> >> >> >> >> NOTE: This message may contain information that is confidential, >> proprietary, privileged or otherwise protected by law. The message is >> intended solely for the named addressee. If received in error, please >> destroy and notify the sender. Any use of this email is prohibited when >> received in error. Impetus does not represent, warrant and/or guarantee, >> that the integrity of this communication has been maintained nor that the >> communication is free of errors, virus, interception or interference. > > > > > > > > > > NOTE: This message may contain information that is confidential, proprietary, > privileged or otherwise protected by law. The message is intended solely for > the named addressee. If received in error, please destroy and notify the > sender. Any use of this email is prohibited when received in error. Impetus > does not represent, warrant and/or guarantee, that the integrity of this > communication has been maintained nor that the communication is free of > errors, virus, interception or interference. NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: PutKafka-Message Delimiter
Should I press CTRL+Enter on value tab of "Message delimiter" key? Regards, Sourav Gulati -Original Message- From: Oleg Zhurakousky [mailto:ozhurakou...@hortonworks.com] Sent: Thursday, May 05, 2016 4:41 PM To: dev@nifi.apache.org Subject: Re: PutKafka-Message Delimiter Please use Ctrl+Enter. The ‘\n’ is interpreted literally as a string (not new line). Let me know how it goes Cheers Oleg > On May 5, 2016, at 7:01 AM, Sourav Gulati <sourav.gul...@impetus.co.in> wrote: > > Hi All, > > I am tried setting message delimiter in put kafka to /n . But it is not > working. Please guide. > > Regards, > Sourav Gulati > > > > > > > > > > NOTE: This message may contain information that is confidential, proprietary, > privileged or otherwise protected by law. The message is intended solely for > the named addressee. If received in error, please destroy and notify the > sender. Any use of this email is prohibited when received in error. Impetus > does not represent, warrant and/or guarantee, that the integrity of this > communication has been maintained nor that the communication is free of > errors, virus, interception or interference. NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
PutKafka-Message Delimiter
Hi All, I am tried setting message delimiter in put kafka to /n . But it is not working. Please guide. Regards, Sourav Gulati NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Quick Question- Nifi Kafka
I upgraded kafka to 0.9.0 and now its working Regards, Sourav Gulati -Original Message- From: Oleg Zhurakousky [mailto:ozhurakou...@hortonworks.com] Sent: Tuesday, May 03, 2016 5:25 PM To: dev@nifi.apache.org Subject: Re: Quick Question- Nifi Kafka It will be vey hard to suggest anything without any details. Have you looked at Kafka logs? Oleg > On May 3, 2016, at 7:30 AM, Sourav Gulati <sourav.gul...@impetus.co.in> wrote: > > Hi, > > > I am using 0.8.1.1 , I am not seeing any exception in logs . however, still > it is not writing to kafka. Any suggestions? > > > Regards, > Sourav Gulati > > -Original Message- > From: Oleg Zhurakousky [mailto:ozhurakou...@hortonworks.com] > Sent: Tuesday, May 03, 2016 4:14 PM > To: dev@nifi.apache.org > Subject: Re: Quick Question- Nifi Kafka > > Hi > > It was tested with 0.8.2 and 0.9, but it does not use new consumer API. We > are working on a new version of Kafka support slated for 0.7 release > > Cheers > Oleg > > > >> On May 3, 2016, at 02:20, Sourav Gulati <sourav.gul...@impetus.co.in> wrote: >> >> Hi All, >> >> Nifi-0.6.1 is compatible with which kafka version? >> >> >> Regards, >> Sourav Gulati >> >> >> >> >> >> >> >> >> >> NOTE: This message may contain information that is confidential, >> proprietary, privileged or otherwise protected by law. The message is >> intended solely for the named addressee. If received in error, please >> destroy and notify the sender. Any use of this email is prohibited when >> received in error. Impetus does not represent, warrant and/or guarantee, >> that the integrity of this communication has been maintained nor that the >> communication is free of errors, virus, interception or interference. > > > > > > > > > NOTE: This message may contain information that is confidential, proprietary, > privileged or otherwise protected by law. The message is intended solely for > the named addressee. If received in error, please destroy and notify the > sender. Any use of this email is prohibited when received in error. Impetus > does not represent, warrant and/or guarantee, that the integrity of this > communication has been maintained nor that the communication is free of > errors, virus, interception or interference. > NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Quick Question- Nifi Kafka
Hi, I am using 0.8.1.1 , I am not seeing any exception in logs . however, still it is not writing to kafka. Any suggestions? Regards, Sourav Gulati -Original Message- From: Oleg Zhurakousky [mailto:ozhurakou...@hortonworks.com] Sent: Tuesday, May 03, 2016 4:14 PM To: dev@nifi.apache.org Subject: Re: Quick Question- Nifi Kafka Hi It was tested with 0.8.2 and 0.9, but it does not use new consumer API. We are working on a new version of Kafka support slated for 0.7 release Cheers Oleg > On May 3, 2016, at 02:20, Sourav Gulati <sourav.gul...@impetus.co.in> wrote: > > Hi All, > > Nifi-0.6.1 is compatible with which kafka version? > > > Regards, > Sourav Gulati > > > > > > > > > > NOTE: This message may contain information that is confidential, proprietary, > privileged or otherwise protected by law. The message is intended solely for > the named addressee. If received in error, please destroy and notify the > sender. Any use of this email is prohibited when received in error. Impetus > does not represent, warrant and/or guarantee, that the integrity of this > communication has been maintained nor that the communication is free of > errors, virus, interception or interference. NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
Quick Question- Nifi Kafka
Hi All, Nifi-0.6.1 is compatible with which kafka version? Regards, Sourav Gulati NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Reg: Nifi Clustering
Hi Joe, This is what I am getting while running both master and node IllegalStateException: Application may be configured as a cluster manager or a node, but not both Regards, Sourav Gulati -Original Message- From: Joe Witt [mailto:joe.w...@gmail.com] Sent: Monday, May 02, 2016 6:45 PM To: dev@nifi.apache.org Subject: Re: Reg: Nifi Clustering Sourav, The installation procedure does not differ. However, you do need to be careful to ensure you're not trying to use the same ports for both the master and the node. If you're having trouble go ahead and send some of the configuration details over and someone can try to assist. Thanks Joe On Mon, May 2, 2016 at 9:05 AM, Sourav Gulati <sourav.gul...@impetus.co.in> wrote: > Hello Joe, > > Could you please provide steps of creating cluster on single node in current > version? > > Regards, > Sourav Gulati > > -Original Message- > From: Joe Witt [mailto:joe.w...@gmail.com] > Sent: Monday, May 02, 2016 6:28 PM > To: dev@nifi.apache.org > Subject: Re: Reg: Nifi Clustering > > Sourav, > > Yes. And in the upcoming 1.0 release work is being done to eliminate this > notion of master/slave altogether and move to instead that at any time any > node can carry the responsibility of being a coordinator for the cluster. > > Thanks > Joe > > On Mon, May 2, 2016 at 8:49 AM, Sourav Gulati <sourav.gul...@impetus.co.in> > wrote: >> Hi Team, >> >> While creating Nifi Cluster, Can I run master and slave on same node? >> >> Regards, >> Sourav Gulati >> >> >> >> >> >> >> >> >> >> NOTE: This message may contain information that is confidential, >> proprietary, privileged or otherwise protected by law. The message is >> intended solely for the named addressee. If received in error, please >> destroy and notify the sender. Any use of this email is prohibited when >> received in error. Impetus does not represent, warrant and/or guarantee, >> that the integrity of this communication has been maintained nor that the >> communication is free of errors, virus, interception or interference. > > > > > > > > > NOTE: This message may contain information that is confidential, proprietary, > privileged or otherwise protected by law. The message is intended solely for > the named addressee. If received in error, please destroy and notify the > sender. Any use of this email is prohibited when received in error. Impetus > does not represent, warrant and/or guarantee, that the integrity of this > communication has been maintained nor that the communication is free of > errors, virus, interception or interference. NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Reg: Nifi Clustering
Hello Joe, Could you please provide steps of creating cluster on single node in current version? Regards, Sourav Gulati -Original Message- From: Joe Witt [mailto:joe.w...@gmail.com] Sent: Monday, May 02, 2016 6:28 PM To: dev@nifi.apache.org Subject: Re: Reg: Nifi Clustering Sourav, Yes. And in the upcoming 1.0 release work is being done to eliminate this notion of master/slave altogether and move to instead that at any time any node can carry the responsibility of being a coordinator for the cluster. Thanks Joe On Mon, May 2, 2016 at 8:49 AM, Sourav Gulati <sourav.gul...@impetus.co.in> wrote: > Hi Team, > > While creating Nifi Cluster, Can I run master and slave on same node? > > Regards, > Sourav Gulati > > > > > > > > > > NOTE: This message may contain information that is confidential, proprietary, > privileged or otherwise protected by law. The message is intended solely for > the named addressee. If received in error, please destroy and notify the > sender. Any use of this email is prohibited when received in error. Impetus > does not represent, warrant and/or guarantee, that the integrity of this > communication has been maintained nor that the communication is free of > errors, virus, interception or interference. NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Reg: Issues in Custom Processor
Hi Bryan,
Following is the code snapshot:
try (FileChannel fileChannel = FileChannel.open(file.toPath(),
StandardOpenOption.READ)) {
fileChannel.position(lastKnownLocation);
FlowFile flowFile = session.create();
flowFile = writeToFlowFile(session, flowFile, fileChannel,
positionHolder);
tranferFlowFile(session, flowFile);
} catch (IOException e) {
logger.error("Error while creating/closing file reader", e);
}
session.commit();
context.yield();
It is working even if I set yield to 1 sec(default setting) . However, If I
don't run yield function , it gives the exception.
Regards,
Sourav Gulati
-Original Message-
From: Bryan Bende [mailto:bbe...@gmail.com]
Sent: Wednesday, April 27, 2016 7:58 PM
To: dev@nifi.apache.org
Subject: Re: Reg: Issues in Custom Processor
Hi Sourav,
Are you able to share any of the code from your onTrigger method so we can see
how you are opening/closing the file reader?
-Bryan
On Wed, Apr 27, 2016 at 7:43 AM, Sourav Gulati <sourav.gul...@impetus.co.in>
wrote:
> I have introduced yield of 5 secs and after that it started working as
> expected.
>
> Regards,
> Sourav Gulati
>
> -Original Message-
> From: Sourav Gulati [mailto:sourav.gul...@impetus.co.in]
> Sent: Wednesday, April 27, 2016 11:34 AM
> To: dev@nifi.apache.org
> Subject: RE: Reg: Issues in Custom Processor
>
> Joe,
>
> Now I have increased number of open files limit.
> ulimit -n
> 50000
>
> How still I am seeing same exception
>
> Regards,
> Sourav Gulati
>
> -Original Message-
> From: Sourav Gulati [mailto:sourav.gul...@impetus.co.in]
> Sent: Tuesday, April 26, 2016 11:48 PM
> To: dev@nifi.apache.org
> Subject: Re: Reg: Issues in Custom Processor
>
> Thanks Joe. I will try and update you
>
> Regards,
> Sourav Gulati
> Big Data
>
>
>
> From: Joe Witt <joe.w...@gmail.com>
> Sent: Tuesday, April 26, 2016 11:26 PM
> To: dev@nifi.apache.org
> Subject: Re: Reg: Issues in Custom Processor
>
> Sourav,
>
> On twitter you asked "why is it opening so many file handles"
>
> Your processor is not. It may well be just fine. However, at startup
> nifi loads many files into the classloader and it is at the point now
> where it references more than 1024 objects (files, sockets, etc..) so
> it is important to set a higher default.
>
> I encourage you to go ahead and follow the best practices mentioned in
> the documentation and I suspect you'll find your custom processor is
> doing what you expected.
>
> Thanks
> Joe
>
> On Tue, Apr 26, 2016 at 1:47 PM, Joe Witt <joe.w...@gmail.com> wrote:
> > Sourav,
> >
> > Please make sure you've followed these recommendations
> > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.htm
> > l#
> > configuration-best-practices
> >
> > Specifically make sure you update the max open file handles.
> >
> > Thanks
> > Joe
> >
> > On Tue, Apr 26, 2016 at 9:34 AM, Sourav Gulati
> > <sourav.gul...@impetus.co.in> wrote:
> >> Hi Team,
> >>
> >> I am using "@TriggerSerially" annotation while creating processor
> >> and
> inside "On Trigger" method I am creating a file reader and after
> doing the processing I am closing the reader connection as well. As
> per my assumption, if I am using @TriggerSerially, then "On trigger"
> method runs in serial fashion.
> >>
> >> However, while executing processor I am getting "too many open files"
> error . I check with lsof command as well, it has opened lot of reader
> connections to file.
> >>
> >> What should I change ? Need urgent help
> >>
> >>
> >>
> >> Regards,
> >> Sourav Gulati
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited
> when received in error. Impetus does not represent, warrant and/or
> guarantee, that the integrity of this communication has been
> maintained nor that the communication is free of errors, virus, interception
> or interference.
>
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely
Re: Reg: Issues in Custom Processor
Thanks Joe. I will try and update you Regards, Sourav Gulati Big Data From: Joe Witt <joe.w...@gmail.com> Sent: Tuesday, April 26, 2016 11:26 PM To: dev@nifi.apache.org Subject: Re: Reg: Issues in Custom Processor Sourav, On twitter you asked "why is it opening so many file handles" Your processor is not. It may well be just fine. However, at startup nifi loads many files into the classloader and it is at the point now where it references more than 1024 objects (files, sockets, etc..) so it is important to set a higher default. I encourage you to go ahead and follow the best practices mentioned in the documentation and I suspect you'll find your custom processor is doing what you expected. Thanks Joe On Tue, Apr 26, 2016 at 1:47 PM, Joe Witt <joe.w...@gmail.com> wrote: > Sourav, > > Please make sure you've followed these recommendations > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#configuration-best-practices > > Specifically make sure you update the max open file handles. > > Thanks > Joe > > On Tue, Apr 26, 2016 at 9:34 AM, Sourav Gulati > <sourav.gul...@impetus.co.in> wrote: >> Hi Team, >> >> I am using "@TriggerSerially" annotation while creating processor and >> inside "On Trigger" method I am creating a file reader and after doing the >> processing I am closing the reader connection as well. As per my assumption, >> if I am using @TriggerSerially, then "On trigger" method runs in serial >> fashion. >> >> However, while executing processor I am getting "too many open files" error >> . I check with lsof command as well, it has opened lot of reader connections >> to file. >> >> What should I change ? Need urgent help >> >> >> >> Regards, >> Sourav Gulati >> >> >> >> >> >> >> >> >> >> NOTE: This message may contain information that is confidential, >> proprietary, privileged or otherwise protected by law. The message is >> intended solely for the named addressee. If received in error, please >> destroy and notify the sender. Any use of this email is prohibited when >> received in error. Impetus does not represent, warrant and/or guarantee, >> that the integrity of this communication has been maintained nor that the >> communication is free of errors, virus, interception or interference. NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
Reg: Issues in Custom Processor
Hi Team, I am using "@TriggerSerially" annotation while creating processor and inside "On Trigger" method I am creating a file reader and after doing the processing I am closing the reader connection as well. As per my assumption, if I am using @TriggerSerially, then "On trigger" method runs in serial fashion. However, while executing processor I am getting "too many open files" error . I check with lsof command as well, it has opened lot of reader connections to file. What should I change ? Need urgent help Regards, Sourav Gulati NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
Reg: Issue with Custom tail processor
HI Team, I was trying to write a custom processor. When I execute it, it given following message in log: 2016-04-25 05:39:58,705 DEBUG [Timer-Driven Process Thread-4] org.apache.nifi.engine.FlowEngine A flow controller execution task 'java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@3655664d' has been cancelled. There is no error as such in logs Also, I can "1" written on top rigt corner of the processor. [cid:image001.png@01D19EE3.A50C8140] I am not able to see any output. Could you please provide some suggestions as I am stuck. Regards, Sourav Gulati NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Reg: Tailing a remote file
Simon, I see an issue with tail processor . I have to give the exact name of the file to tail. I can give something like /home/user/abc*.log Is there anyway I can do this? Regards, Sourav Gulati -Original Message- From: Simon Ball [mailto:sb...@hortonworks.com] Sent: Thursday, April 21, 2016 3:40 PM To: dev@nifi.apache.org Subject: Re: Reg: Tailing a remote file If you're using a rolling file name pattern you will want to tail the current log file, and essentially ignore the rolled history, unless you need to represses or miss data. TailFile is a local processor, so you would run a nifi agent on the box that has the files on, use tail file -> remote process group -> input port on core nifi instance -> wherever your data is going next. The local/remote setting comes up when clustering nifi and using something like an NFS mount to get to the files being tailed, which is a different way of doing this. The remote will lead to state about where you've gotten to being stored differently to support the cluster continuing to tail from alternative nifi cluster nodes. Simon > On 21 Apr 2016, at 11:38, Sourav Gulati <sourav.gul...@impetus.co.in> wrote: > > Hi Simon/Team. > > Please provide some good example of tail processor. > > I need some example for " Rolling Filename Pattern". > > Also, as per my knowledge TailFile processor is used for tailing local file > then what does location attribute signifies, if I set it to remote. > > Regards, > Sourav Gulati > > -Original Message- > From: Simon Ball [mailto:sb...@hortonworks.com] > Sent: Wednesday, April 20, 2016 2:27 PM > To: dev@nifi.apache.org > Subject: Re: Reg: Tailing a remote file > > Site-to-site is in pretty much every version of nifi released under apache. > >> On 20 Apr 2016, at 10:48, Sourav Gulati <sourav.gul...@impetus.co.in> wrote: >> >> Thanks Simon. Which version of Nifi support site to site? >> >> Regards, >> Sourav Gulati >> >> -Original Message- >> From: Simon Ball [mailto:sb...@hortonworks.com] >> Sent: Wednesday, April 20, 2016 1:28 PM >> To: dev@nifi.apache.org >> Subject: Re: Reg: Tailing a remote file >> >> The best way to do this would be to run a nifi agent on the remote system, >> using the TailFile processor, and then use site-to-site to get that into >> your core nifi. >> >> Simon >> >> >>> On 20 Apr 2016, at 09:49, Sourav Gulati <sourav.gul...@impetus.co.in> wrote: >>> >>> Hi Team, >>> >>> We want to process logs from a file which is getting updated continuously >>> on remote system. >>> >>> What is the best way to do that in NiFi? >>> >>> Regards, >>> Sourav Gulati >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> NOTE: This message may contain information that is confidential, >>> proprietary, privileged or otherwise protected by law. The message is >>> intended solely for the named addressee. If received in error, please >>> destroy and notify the sender. Any use of this email is prohibited when >>> received in error. Impetus does not represent, warrant and/or guarantee, >>> that the integrity of this communication has been maintained nor that the >>> communication is free of errors, virus, interception or interference. >> >> >> >> >> >> >> >> >> NOTE: This message may contain information that is confidential, >> proprietary, privileged or otherwise protected by law. The message is >> intended solely for the named addressee. If received in error, please >> destroy and notify the sender. Any use of this email is prohibited when >> received in error. Impetus does not represent, warrant and/or guarantee, >> that the integrity of this communication has been maintained nor that the >> communication is free of errors, virus, interception or interference. > > > > > > > > > > NOTE: This message may contain information that is confidential, proprietary, > privileged or otherwise protected by law. The message is intended solely for > the named addressee. If received in error, please destroy and notify the > sender. Any use of this email is prohibited when received in error. Impetus > does not represent, warrant and/or guarantee, that the integrity of this > communication has been maintained nor that the communication is free of > errors, virus, interception or interference. > NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
Reg: Tailing a remote file
Hi Team, We want to process logs from a file which is getting updated continuously on remote system. What is the best way to do that in NiFi? Regards, Sourav Gulati NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
Re: Reg: starting and Stopping processor
Matt,
I tried it:
curl -i -X PUT -H 'Content-Type: application/json' -d
'{"revision":{"clientId":"561b4443-8be5-43ee-b1f7-b07740e87a0f"},"processor":{"id":"fad172d5-5d29-4a8b-923f-e888af039c52","state":"STOPPED"}}'
http://localhost:8080/nifi-api/controller/process-groups/42cb10e8-1856-4155-9aa2-bef736bf330e/processors/fad172d5-5d29-4a8b-923f-e888af039c52
This is the response i am getting:
HTTP/1.1 409 Conflict
Date: Sat, 12 Mar 2016 18:22:34 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Server: Jetty(9.2.11.v20150529)
This NiFi instance has been updated by 'anonymous'. Please refresh to
synchronize the view
But it did not stop the processor.
Please provide some suggesstions
Regards,
Sourav Gulati
Big Data
From: Matt Gilman <matt.c.gil...@gmail.com>
Sent: Friday, March 11, 2016 8:00 PM
To: dev@nifi.apache.org
Subject: Re: Reg: starting and Stopping processor
Sourav,
The command would like the following with substituting the relevant values.
curl -i -X PUT -H 'Content-Type: application/json' -d
'{"revision":{"version":,"clientId":""},"processor":{"id":"d6d28ec3-75c8-4194-b171-79c3e813f7e3","state":"STOPPED"}}'
http://localhost:8080/nifi-api/controller/process-groups/92e70e95-31c4-4eb3-beab-e6fd411378bd/processors/d6d28ec3-75c8-4194-b171-79c3e813f7e3
Also, check out this blog post that describes how NiFi uses optimistic
locking to know when another user has modified the data flow [1]. The
bottom contains a great sequence diagram describing the necessary values.
Matt
[1]
https://community.hortonworks.com/content/kbentry/3160/update-nifi-flow-on-the-fly-via-api.html
On Thu, Mar 10, 2016 at 11:57 PM, Sourav Gulati <sourav.gul...@impetus.co.in
> wrote:
> Hi Team,
>
> Is there a way to start and stop processor using some rest api or command
> line ? If yes, please provide the steps.
>
> Regards,
> Sourav Gulati
>
>
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
NOTE: This message may contain information that is confidential, proprietary,
privileged or otherwise protected by law. The message is intended solely for
the named addressee. If received in error, please destroy and notify the
sender. Any use of this email is prohibited when received in error. Impetus
does not represent, warrant and/or guarantee, that the integrity of this
communication has been maintained nor that the communication is free of errors,
virus, interception or interference.
Reg: starting and Stopping processor
Hi Team, Is there a way to start and stop processor using some rest api or command line ? If yes, please provide the steps. Regards, Sourav Gulati NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
