Sv: Re: ParseCEF processor fails when Msg field exceed 1023 characters

Hi, I will test your idea to use RouteOnContent with a length check before the ParseCEF processor. Since all message needs to be parsed, even if the msg field exceed 1023 character, I will probably have to save the content of the msg field temporarily, truncate the msg field in the message, pars

Sv: Re: ParseCEF processor fails when Msg field exceed 1023 characters

Hi, I`m quite sure, I have verified it in a small test Flow. A propert to set the do_validate flag to false would be helpful in this case. Regards, Lj På fre, jan. 2019 klockan 0:05, Otto Fowler skrev: #yiv4068205678 body{font-family:Helvetica, Arial;font-size:13px;}Can I ask how you are

Re: ParseCEF processor fails when Msg field exceed 1023 characters

Hi Felix, As you noticed, the CEF standard indeed states that 1023 characters is the limit and I such when I wrote the underlying parser (ParCEFone) I enforced a length check and failed the parsing in case of exceeding that value. Could I just truncate? Possibly... But what to do when the vendor

Re: ParseCEF processor fails when Msg field exceed 1023 characters

Otto, The parser should fail in that case (and many others). https://github.com/fluenda/ParCEFone/blob/master/src/main/java/com/fluenda/parcefone/event/CefRev23.java#L327 Cheers On Fri, Jan 4, 2019 at 10:05 AM Otto Fowler wrote: > Can I ask how you are sure it is the message size that is caus

A question about [MergeContent] processor

Hi all, I have a job consist of following steps: first consuming data from kafka, and then packing data every 5 minutes into one file, finally put the packed file into hdfs. I use the [MergeContent] processor to accomplish the “packing” step. The properties of MergeContent I configured is list belo

Re: A question about [MergeContent] processor

Hi Jianan As you have “Minimum Number of Entries: 1” it is normal that you can see merges with only one flowfile. In my opinion the “Minimum Number of Entries” is stronger than the “Max Bin Age” (first is written bold and second not). Additionally it is called “Max Bin Age” and not “Bin Age”. S

Re: A question about [MergeContent] processor

Hi Josef, Thanks for reply. In my opinion the “Minimum Number of Entries” is should not and can not stronger than the “Max Bin Age”. Suppose I have only ONE flowfile from datasource put into MergeContent processor, and I set "Minimum Number of Entries" = 2, then this ONE flowfile will never coming

Re: A question about [MergeContent] processor

Hi Jianan I just say that as soon as “Minimum Number of Entries” is reached the flow can be flushed out, and further if the minimum number isn’t reached I would expect that the “Max Bin Age” takes place. Have you tried that? Cheers Josef From: Jianan Zhang Reply-To: "users@nifi.apache.org" D

Re: ParseCEF processor fails when Msg field exceed 1023 characters

Yeah, I went through the code after my mail last night. Would having validation optional not be a valid setting? How often would the output feeding this flow be invalid for some other reason? On January 4, 2019 at 04:33:54, Andre (andre-li...@fucs.org) wrote: Otto, The parser should fail in

Re: NiFi (De-)"Compress Content" Processor causes to fill up content_repo insanly fast by corrupt GZIP files

Josef, Thanks for the info! There are a few things to consider here. Firstly, you said that you are using NiFi 1.8.0. Are you using the new Load Balancing capability? I.e., do you have any Connections configured to balance load across your cluster? And if so, are you load-balancing any 0-byte fi

Re: A question about [MergeContent] processor

The inner workings of MergeContent is certainly a FAQ. This message [1] to the users list from a long time ago may help. I think it's still accurate. [1] - https://lists.apache.org/thread.html/5ab5d9d0bcd0eef8ace391d00f5f5678427bee4b2fbf1e48d78ea8c8@1445464430@%3Cusers.nifi.apache.org%3E Regard

Re: NiFi (De-)"Compress Content" Processor causes to fill up content_repo insanly fast by corrupt GZIP files

Mark, Yes we are using Load Balancing capability and we do that after the ListSFTP processor, so yes we loadbalance 0-Byte files. Seems that we probably facing your Bug here. Thanks a lot for explaining in detail what happens regarding the flowfile/content repo in NiFi. Additionally we have s

Re: NiFi (De-)"Compress Content" Processor causes to fill up content_repo insanly fast by corrupt GZIP files

Josef, OK, thanks for confirming. My suspicion is that the Load-Balancing bug is what is biting you, and that when you tried to replicate with the CompressContent in a simple case, you may have just been experiencing the "cleanup lag" related to the way that the repositories interact with one a

Re: ParseCEF processor fails when Msg field exceed 1023 characters

Otto, I considered that but never implemented it in NiFi as I had concerns people would have the incorrect assumption the parser can help on those cases. >From the top of my head, I have seen type mismatch, length overflows, malformed CEF headers (the pipe delimited section of the CEF message). I

Re: Wait/Notify inconsistent behavior

Please avoid sending more copies of the question. Hopefully someone familiar with the processors in question will be available in time. Thanks On Fri, Jan 4, 2019 at 9:14 PM Luis Carmona wrote: > Hi everyone, > > Im having a strange behavior with Wait / Notify mechanism. Attached is > the im

Re: Wait/Notify inconsistent behavior

I'm sorry, got messages from nifi-users saying "UNCHECKED", and reading about understood the message did not get trough. Thanks for letting me know. LC De: "Joe Witt" Para: "users" Enviados: Viernes, 4 de Enero 2019 23:23:02 Asunto: Re: Wait/Notify inconsistent behavior Please avoi

Re: Wait/Notify inconsistent behavior

thanks for letting us know. the lists can be a bit awkward from a user experience pov. no worries On Fri, Jan 4, 2019, 9:26 PM Luis Carmona I'm sorry, > > got messages from nifi-users saying "UNCHECKED", and reading about > understood the message did not get trough. > > Thanks for letting me kn