Both hdfs and Elasticsearch batch sizes. There is no error in the logs. It mpacts topology error rate and cause almost 90% error rate on indexing tuples.
On 6 Dec. 2017 00:20, "Otto Fowler" <ottobackwa...@gmail.com> wrote: Where are you seeing the errors? Screenshot? On December 5, 2017 at 08:03:46, Otto Fowler (ottobackwa...@gmail.com) wrote: Which of the indexing options are you changing the batch size for? HDFS? Elasticsearch? Both? Can you give an example? On December 5, 2017 at 02:09:29, Ali Nazemian (alinazem...@gmail.com) wrote: No specific error in the logs. I haven't enabled debug/trace, though. On Tue, Dec 5, 2017 at 11:54 AM, Otto Fowler <ottobackwa...@gmail.com> wrote: > My first thought is what are the errors when you get a high error rate? > > > On December 4, 2017 at 19:34:29, Ali Nazemian (alinazem...@gmail.com) > wrote: > > Any thoughts? > > On Sun, Dec 3, 2017 at 11:27 PM, Ali Nazemian <alinazem...@gmail.com> > wrote: > > > Hi, > > > > We have noticed recently that no matter what batch size we use for Metron > > indexing feeds, as long as we start using different batch size for > > different Metron feeds, indexing topology throughput will start dropping > > due to the high error rate! So I was wondering whether based on the > current > > indexing topology design, we have to choose the same batch size for all > the > > feeds or not. Otherwise, throughout will be dropped. I assume since it is > > acceptable to use different batch sizes for different feeds, it is not > > expected by design. > > > > Moreover, I have noticed in practice that even if we change the batch > > size, it will not affect the messages that are already in enrichments or > > indexing topics, and it will only affect the new messages that are coming > > to the parser. Therefore, we need to let all the messages pass the > indexing > > topology so that we can change the batch size! > > > > It would be great if we can have more details regarding the design of > this > > section so we can understand our observations are based on the design or > > some kind of bug. > > > > Regards, > > Ali > > > > > > -- > A.Nazemian > > -- A.Nazemian
