Hey Laurens, My kafka processor says "Failed while waiting for acks from Kafka"
On 15 January 2018 at 21:00, Laurens Vets <laur...@daemon.be> wrote: > Hi Gaurav, > > If you click on the red squares in the upper right corners of your > processors, what error messages do you see? > > On 2018-01-14 19:29, Gaurav Bapat wrote: > > Hey Jon, > > I have Storm UI and the logs are coming from firewalls, servers, etc from > other machines(HP ArcSight Logger). > > I have attached the NiFi screenshots, my logs are coming but there is some > error with Kafka and I am having issues with configuring Kafka broker > > > > On 12 January 2018 at 18:14, zeo...@gmail.com <zeo...@gmail.com> wrote: > >> In Ambari under storm you can find the UI under quick links at the top. >> That said, the issue seems to be upstream of Metron, in NiFi. That is >> something I can't help with as much, but if you can share the listensyslog >> processor config that would be a start. Also, share the config of the >> thing that is sending syslog as well (are these local syslog, is that >> machine aggregating syslog from other machines, etc.). Thanks, >> >> Jon >> >> On Fri, Jan 12, 2018, 01:00 Gaurav Bapat <gauravb3...@gmail.com> wrote: >> >>> I have created a Kafka topic "cef" but my Listen Syslogs is not getting >>> logs in the processor. >>> >>> Also I checked using tcpdump -i and it is getting logs in my machine but >>> ListenSyslogs is not getting the logs >>> >>> On 12 January 2018 at 11:13, Gaurav Bapat <gauravb3...@gmail.com> wrote: >>> >>>> [root@metron incubator-metron]# ./metron-deployment/scripts/pl >>>> atform-info.sh >>>> Metron 0.4.3 >>>> -- >>>> * master >>>> -- >>>> commit c559ed7e1838ec71344eae3d9e37771db2641635 >>>> Author: cstella <ceste...@gmail.com> >>>> Date: Tue Jan 9 15:28:47 2018 -0500 >>>> >>>> METRON-1379: Add an OBJECT_GET stellar function closes >>>> apache/incubator-metron#880 >>>> -- >>>> metron-deployment/vagrant/full-dev-platform/Vagrantfile | 2 +- >>>> 1 file changed, 1 insertion(+), 1 deletion(-) >>>> -- >>>> ansible 2.0.0.2 >>>> config file = >>>> configured module search path = Default w/o overrides >>>> -- >>>> Vagrant 1.9.6 >>>> -- >>>> Python 2.7.5 >>>> -- >>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; >>>> 2015-11-10T22:11:47+05:30) >>>> Maven home: /opt/maven/current >>>> Java version: 1.8.0_151, vendor: Oracle Corporation >>>> Java home: /opt/jdk1.8.0_151/jre >>>> Default locale: en_US, platform encoding: UTF-8 >>>> OS name: "linux", version: "3.10.0-693.11.6.el7.x86_64", arch: "amd64", >>>> family: "unix" >>>> -- >>>> Docker version 1.12.6, build ec8512b/1.12.6 >>>> -- >>>> node >>>> v8.9.3 >>>> -- >>>> npm >>>> 5.5.1 >>>> -- >>>> g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16) >>>> Copyright (C) 2015 Free Software Foundation, Inc. >>>> This is free software; see the source for copying conditions. There is >>>> NO >>>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR >>>> PURPOSE. >>>> >>>> -- >>>> Compiler is C++11 compliant >>>> -- >>>> Linux metron.com 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4 01:06:37 >>>> UTC 2018 x86_64 x86_64 x86_64 GNU/Linux >>>> -- >>>> Total System Memory = 15773.3 MB >>>> Processor Model: Intel(R) Core(TM) i5-3450 CPU @ 3.10GHz >>>> Processor Speed: 3320.875 MHz >>>> Processor Speed: 3307.191 MHz >>>> Processor Speed: 3376.699 MHz >>>> Processor Speed: 3338.917 MHz >>>> Total Physical Processors: 4 >>>> Total cores: 16 >>>> Disk information: >>>> /dev/mapper/centos-root 200G 22G 179G 11% / >>>> /dev/sda1 2.0G 224M 1.8G 11% /boot >>>> /dev/sda2 1022M 12K 1022M 1% /boot/efi >>>> /dev/mapper/centos-home 247G 10G 237G 5% /home >>>> This CPU appears to support virtualization >>>> >>>> On 12 January 2018 at 09:25, Gaurav Bapat <gauravb3...@gmail.com> >>>> wrote: >>>> >>>>> Hey Jon, >>>>> >>>>> Appreciate your timely reply. >>>>> >>>>> I gone through your answer but still I can't figure out how do I do >>>>> parsing/indexing in Storm UI as I cant find any option for the same. >>>>> >>>>> Is there any other UI to do parsing/indexing? >>>>> >>>>> >>>>> >>>>> On 11 January 2018 at 21:22, zeo...@gmail.com <zeo...@gmail.com> >>>>> wrote: >>>>> >>>>>> So, you created a new cef topic, and set up the appropriate parser >>>>>> config for it (if not, this >>>>>> <https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source> >>>>>> may be helpful)? If so: >>>>>> >>>>>> Here are some basic troubleshooting steps: >>>>>> 1. Validate that the logs are getting onto the kafka topic that you >>>>>> are sending to. If they aren't there, the problem is upstream from >>>>>> Metron. >>>>>> 2. If they are getting onto the kafka topic they are being directly >>>>>> sent to, check the indexing kafka topic for an enriched version of those >>>>>> same logs. >>>>>> 3. Do a binary search of the various components involved with ingest. >>>>>> a. If the logs are *not* on the indexing kafka topic, check the >>>>>> enrichments topic for those logs. >>>>>> b. If the logs are *not* on the enrichments topic, check the >>>>>> parser storm topology. >>>>>> c. If the logs are on the enrichments topic, but *not* indexing, >>>>>> check the enrichments storm topology. >>>>>> d. If the logs are on the indexing but *not* Kibana, check the >>>>>> indexing storm topic. >>>>>> e. If the logs are in on the indexing topic and indexing storm >>>>>> topic is in good shape, check elasticsearch directly. >>>>>> 4. You should have identified where the issue is at this point. >>>>>> Report back here with what you observed, any relevant error messages, >>>>>> etc. >>>>>> >>>>>> Side note: We should document a decision tree for troubleshooting >>>>>> data ingest. It is fairly straightforward and makes me wonder if we >>>>>> already have this somewhere and I'm not aware of it? It would also be a >>>>>> good place to put pointers to some common errors. >>>>>> >>>>>> Jon >>>>>> >>>>>> On Thu, Jan 11, 2018 at 1:44 AM Gaurav Bapat <gauravb3...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hello everyone, I have deployed Metron on a single node machine and >>>>>>> I would like to know how do I get Syslogs from NiFi into Kibana >>>>>>> dashboard? >>>>>>> >>>>>>> I have created a Kafka topic by the name "cef" and I can see that >>>>>>> the topic exists in >>>>>>> Metron Configuration but I am unable to connect it with Kibana >>>>>>> >>>>>>> Need Help!! >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Jon >>>>>> >>>>> -- >> >> Jon >> > >
