Hey Laurens,

My kafka processor says "Failed while waiting for acks from Kafka"

On 15 January 2018 at 21:00, Laurens Vets <laur...@daemon.be> wrote:

> Hi Gaurav,
>
> If you click on the red squares in the upper right corners of your
> processors, what error messages do you see?
>
> On 2018-01-14 19:29, Gaurav Bapat wrote:
>
> Hey Jon,
>
> I have Storm UI and the logs are coming from firewalls, servers, etc from
> other machines(HP ArcSight Logger).
>
> I have attached the NiFi screenshots, my logs are coming but there is some
> error with Kafka and I am having issues with configuring Kafka broker
>
>
>
> On 12 January 2018 at 18:14, zeo...@gmail.com <zeo...@gmail.com> wrote:
>
>> In Ambari under storm you can find the UI under quick links at the top.
>> That said, the issue seems to be upstream of Metron, in NiFi.  That is
>> something I can't help with as much, but if you can share the listensyslog
>> processor config that would be a start.  Also, share the config of the
>> thing that is sending syslog as well (are these local syslog, is that
>> machine aggregating syslog from other machines, etc.).  Thanks,
>>
>> Jon
>>
>> On Fri, Jan 12, 2018, 01:00 Gaurav Bapat <gauravb3...@gmail.com> wrote:
>>
>>> I have created a Kafka topic "cef" but my Listen Syslogs is not getting
>>> logs in the processor.
>>>
>>> Also I checked using tcpdump -i and it is getting logs in my machine but
>>> ListenSyslogs is not getting the logs
>>>
>>> On 12 January 2018 at 11:13, Gaurav Bapat <gauravb3...@gmail.com> wrote:
>>>
>>>> [root@metron incubator-metron]# ./metron-deployment/scripts/pl
>>>> atform-info.sh
>>>> Metron 0.4.3
>>>> --
>>>> * master
>>>> --
>>>> commit c559ed7e1838ec71344eae3d9e37771db2641635
>>>> Author: cstella <ceste...@gmail.com>
>>>> Date:   Tue Jan 9 15:28:47 2018 -0500
>>>>
>>>>     METRON-1379: Add an OBJECT_GET stellar function closes
>>>> apache/incubator-metron#880
>>>> --
>>>>  metron-deployment/vagrant/full-dev-platform/Vagrantfile | 2 +-
>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>> --
>>>> ansible 2.0.0.2
>>>>   config file =
>>>>   configured module search path = Default w/o overrides
>>>> --
>>>> Vagrant 1.9.6
>>>> --
>>>> Python 2.7.5
>>>> --
>>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
>>>> 2015-11-10T22:11:47+05:30)
>>>> Maven home: /opt/maven/current
>>>> Java version: 1.8.0_151, vendor: Oracle Corporation
>>>> Java home: /opt/jdk1.8.0_151/jre
>>>> Default locale: en_US, platform encoding: UTF-8
>>>> OS name: "linux", version: "3.10.0-693.11.6.el7.x86_64", arch: "amd64",
>>>> family: "unix"
>>>> --
>>>> Docker version 1.12.6, build ec8512b/1.12.6
>>>> --
>>>> node
>>>> v8.9.3
>>>> --
>>>> npm
>>>> 5.5.1
>>>> --
>>>> g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16)
>>>> Copyright (C) 2015 Free Software Foundation, Inc.
>>>> This is free software; see the source for copying conditions.  There is
>>>> NO
>>>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
>>>> PURPOSE.
>>>>
>>>> --
>>>> Compiler is C++11 compliant
>>>> --
>>>> Linux metron.com 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4 01:06:37
>>>> UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
>>>> --
>>>> Total System Memory = 15773.3 MB
>>>> Processor Model: Intel(R) Core(TM) i5-3450 CPU @ 3.10GHz
>>>> Processor Speed: 3320.875 MHz
>>>> Processor Speed: 3307.191 MHz
>>>> Processor Speed: 3376.699 MHz
>>>> Processor Speed: 3338.917 MHz
>>>> Total Physical Processors: 4
>>>> Total cores: 16
>>>> Disk information:
>>>> /dev/mapper/centos-root  200G   22G  179G  11% /
>>>> /dev/sda1                2.0G  224M  1.8G  11% /boot
>>>> /dev/sda2               1022M   12K 1022M   1% /boot/efi
>>>> /dev/mapper/centos-home  247G   10G  237G   5% /home
>>>> This CPU appears to support virtualization
>>>>
>>>> On 12 January 2018 at 09:25, Gaurav Bapat <gauravb3...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hey Jon,
>>>>>
>>>>> Appreciate your timely reply.
>>>>>
>>>>> I gone through your answer but still I can't figure out how do I do
>>>>> parsing/indexing in Storm UI as I cant find any option for the same.
>>>>>
>>>>> Is there any other UI to do parsing/indexing?
>>>>>
>>>>>
>>>>>
>>>>> On 11 January 2018 at 21:22, zeo...@gmail.com <zeo...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> So, you created a new cef topic, and set up the appropriate parser
>>>>>> config for it (if not, this
>>>>>> <https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source>
>>>>>> may be helpful)?  If so:
>>>>>>
>>>>>> Here are some basic troubleshooting steps:
>>>>>> 1.  Validate that the logs are getting onto the kafka topic that you
>>>>>> are sending to.  If they aren't there, the problem is upstream from 
>>>>>> Metron.
>>>>>> 2.  If they are getting onto the kafka topic they are being directly
>>>>>> sent to, check the indexing kafka topic for an enriched version of those
>>>>>> same logs.
>>>>>> 3.  Do a binary search of the various components involved with ingest.
>>>>>>     a. If the logs are *not* on the indexing kafka topic, check the
>>>>>> enrichments topic for those logs.
>>>>>>     b. If the logs are *not* on the enrichments topic, check the
>>>>>> parser storm topology.
>>>>>>     c. If the logs are on the enrichments topic, but *not* indexing,
>>>>>> check the enrichments storm topology.
>>>>>>     d. If the logs are on the indexing but *not* Kibana, check the
>>>>>> indexing storm topic.
>>>>>>     e. If the logs are in on the indexing topic and indexing storm
>>>>>> topic is in good shape, check elasticsearch directly.
>>>>>> 4.  You should have identified where the issue is at this point.
>>>>>> Report back here with what you observed, any relevant error messages, 
>>>>>> etc.
>>>>>>
>>>>>> Side note:  We should document a decision tree for troubleshooting
>>>>>> data ingest.  It is fairly straightforward and makes me wonder if we
>>>>>> already have this somewhere and I'm not aware of it?  It would also be a
>>>>>> good place to put pointers to some common errors.
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>> On Thu, Jan 11, 2018 at 1:44 AM Gaurav Bapat <gauravb3...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello everyone, I have deployed Metron on a single node machine and
>>>>>>> I would like to know how do I get Syslogs from NiFi into Kibana 
>>>>>>> dashboard?
>>>>>>>
>>>>>>> I have created a Kafka topic by the name "cef" and I can see that
>>>>>>> the topic exists in
>>>>>>> Metron Configuration but I am unable to connect it with Kibana
>>>>>>>
>>>>>>> Need Help!!
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Jon
>>>>>>
>>>>> --
>>
>> Jon
>>
>
>

Reply via email to