Can you tell me is your KAFKA Topic getting data ? What are you machine specifications ?
On Mon, Jan 15, 2018 at 2:56 PM, Gaurav Bapat <gauravb3...@gmail.com> wrote: > Thanks Farrukh, > > I am not getting data in my kafka topic even after creating one, the issue > seems to be with broker config, how to configure Kafka and Zookeeper port? > > On 15 January 2018 at 13:23, Farrukh Naveed Anjum <anjum.farr...@gmail.com > > wrote: > >> Hi, >> >> I had similar issue it turned out to be the issue in STROM >> >> No worker is assigned to togolgoy all you need is to add additional port >> in >> >> Ambari -> Storm -> Configs -> supervisor.slot.ports by assigning an >> additional port to the list >> >> >> https://community.hortonworks.com/questions/32499/no-workers >> -in-storm-for-squid-topology.html >> >> >> I had similar issue and finally got it fixed >> >> On Mon, Jan 15, 2018 at 8:45 AM, Gaurav Bapat <gauravb3...@gmail.com> >> wrote: >> >>> Storm UI >>> >>> On 15 January 2018 at 08:59, Gaurav Bapat <gauravb3...@gmail.com> wrote: >>> >>>> Hey Jon, >>>> >>>> I have Storm UI and the logs are coming from firewalls, servers, etc >>>> from other machines(HP ArcSight Logger). >>>> >>>> I have attached the NiFi screenshots, my logs are coming but there is >>>> some error with Kafka and I am having issues with configuring Kafka broker >>>> >>>> >>>> >>>> On 12 January 2018 at 18:14, zeo...@gmail.com <zeo...@gmail.com> wrote: >>>> >>>>> In Ambari under storm you can find the UI under quick links at the >>>>> top. That said, the issue seems to be upstream of Metron, in NiFi. That >>>>> is something I can't help with as much, but if you can share the >>>>> listensyslog processor config that would be a start. Also, share the >>>>> config of the thing that is sending syslog as well (are these local >>>>> syslog, >>>>> is that machine aggregating syslog from other machines, etc.). Thanks, >>>>> >>>>> Jon >>>>> >>>>> On Fri, Jan 12, 2018, 01:00 Gaurav Bapat <gauravb3...@gmail.com> >>>>> wrote: >>>>> >>>>>> I have created a Kafka topic "cef" but my Listen Syslogs is not >>>>>> getting logs in the processor. >>>>>> >>>>>> Also I checked using tcpdump -i and it is getting logs in my machine >>>>>> but ListenSyslogs is not getting the logs >>>>>> >>>>>> On 12 January 2018 at 11:13, Gaurav Bapat <gauravb3...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> [root@metron incubator-metron]# ./metron-deployment/scripts/pl >>>>>>> atform-info.sh >>>>>>> Metron 0.4.3 >>>>>>> -- >>>>>>> * master >>>>>>> -- >>>>>>> commit c559ed7e1838ec71344eae3d9e37771db2641635 >>>>>>> Author: cstella <ceste...@gmail.com> >>>>>>> Date: Tue Jan 9 15:28:47 2018 -0500 >>>>>>> >>>>>>> METRON-1379: Add an OBJECT_GET stellar function closes >>>>>>> apache/incubator-metron#880 >>>>>>> -- >>>>>>> metron-deployment/vagrant/full-dev-platform/Vagrantfile | 2 +- >>>>>>> 1 file changed, 1 insertion(+), 1 deletion(-) >>>>>>> -- >>>>>>> ansible 2.0.0.2 >>>>>>> config file = >>>>>>> configured module search path = Default w/o overrides >>>>>>> -- >>>>>>> Vagrant 1.9.6 >>>>>>> -- >>>>>>> Python 2.7.5 >>>>>>> -- >>>>>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; >>>>>>> 2015-11-10T22:11:47+05:30) >>>>>>> Maven home: /opt/maven/current >>>>>>> Java version: 1.8.0_151, vendor: Oracle Corporation >>>>>>> Java home: /opt/jdk1.8.0_151/jre >>>>>>> Default locale: en_US, platform encoding: UTF-8 >>>>>>> OS name: "linux", version: "3.10.0-693.11.6.el7.x86_64", arch: >>>>>>> "amd64", family: "unix" >>>>>>> -- >>>>>>> Docker version 1.12.6, build ec8512b/1.12.6 >>>>>>> -- >>>>>>> node >>>>>>> v8.9.3 >>>>>>> -- >>>>>>> npm >>>>>>> 5.5.1 >>>>>>> -- >>>>>>> g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16) >>>>>>> Copyright (C) 2015 Free Software Foundation, Inc. >>>>>>> This is free software; see the source for copying conditions. There >>>>>>> is NO >>>>>>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR >>>>>>> PURPOSE. >>>>>>> >>>>>>> -- >>>>>>> Compiler is C++11 compliant >>>>>>> -- >>>>>>> Linux metron.com 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4 >>>>>>> 01:06:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux >>>>>>> -- >>>>>>> Total System Memory = 15773.3 MB >>>>>>> Processor Model: Intel(R) Core(TM) i5-3450 CPU @ 3.10GHz >>>>>>> Processor Speed: 3320.875 MHz >>>>>>> Processor Speed: 3307.191 MHz >>>>>>> Processor Speed: 3376.699 MHz >>>>>>> Processor Speed: 3338.917 MHz >>>>>>> Total Physical Processors: 4 >>>>>>> Total cores: 16 >>>>>>> Disk information: >>>>>>> /dev/mapper/centos-root 200G 22G 179G 11% / >>>>>>> /dev/sda1 2.0G 224M 1.8G 11% /boot >>>>>>> /dev/sda2 1022M 12K 1022M 1% /boot/efi >>>>>>> /dev/mapper/centos-home 247G 10G 237G 5% /home >>>>>>> This CPU appears to support virtualization >>>>>>> >>>>>>> On 12 January 2018 at 09:25, Gaurav Bapat <gauravb3...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hey Jon, >>>>>>>> >>>>>>>> Appreciate your timely reply. >>>>>>>> >>>>>>>> I gone through your answer but still I can't figure out how do I do >>>>>>>> parsing/indexing in Storm UI as I cant find any option for the same. >>>>>>>> >>>>>>>> Is there any other UI to do parsing/indexing? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 11 January 2018 at 21:22, zeo...@gmail.com <zeo...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> So, you created a new cef topic, and set up the appropriate parser >>>>>>>>> config for it (if not, this >>>>>>>>> <https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source> >>>>>>>>> may be helpful)? If so: >>>>>>>>> >>>>>>>>> Here are some basic troubleshooting steps: >>>>>>>>> 1. Validate that the logs are getting onto the kafka topic that >>>>>>>>> you are sending to. If they aren't there, the problem is upstream >>>>>>>>> from >>>>>>>>> Metron. >>>>>>>>> 2. If they are getting onto the kafka topic they are being >>>>>>>>> directly sent to, check the indexing kafka topic for an enriched >>>>>>>>> version of >>>>>>>>> those same logs. >>>>>>>>> 3. Do a binary search of the various components involved with >>>>>>>>> ingest. >>>>>>>>> a. If the logs are *not* on the indexing kafka topic, check >>>>>>>>> the enrichments topic for those logs. >>>>>>>>> b. If the logs are *not* on the enrichments topic, check the >>>>>>>>> parser storm topology. >>>>>>>>> c. If the logs are on the enrichments topic, but *not* >>>>>>>>> indexing, check the enrichments storm topology. >>>>>>>>> d. If the logs are on the indexing but *not* Kibana, check >>>>>>>>> the indexing storm topic. >>>>>>>>> e. If the logs are in on the indexing topic and indexing storm >>>>>>>>> topic is in good shape, check elasticsearch directly. >>>>>>>>> 4. You should have identified where the issue is at this point. >>>>>>>>> Report back here with what you observed, any relevant error messages, >>>>>>>>> etc. >>>>>>>>> >>>>>>>>> Side note: We should document a decision tree for troubleshooting >>>>>>>>> data ingest. It is fairly straightforward and makes me wonder if we >>>>>>>>> already have this somewhere and I'm not aware of it? It would also >>>>>>>>> be a >>>>>>>>> good place to put pointers to some common errors. >>>>>>>>> >>>>>>>>> Jon >>>>>>>>> >>>>>>>>> On Thu, Jan 11, 2018 at 1:44 AM Gaurav Bapat < >>>>>>>>> gauravb3...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hello everyone, I have deployed Metron on a single node machine >>>>>>>>>> and I would like to know how do I get Syslogs from NiFi into Kibana >>>>>>>>>> dashboard? >>>>>>>>>> >>>>>>>>>> I have created a Kafka topic by the name "cef" and I can see that >>>>>>>>>> the topic exists in >>>>>>>>>> Metron Configuration but I am unable to connect it with Kibana >>>>>>>>>> >>>>>>>>>> Need Help!! >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Jon >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> -- >>>>> >>>>> Jon >>>>> >>>> >>>> >>> >> >> >> -- >> With Regards >> Farrukh Naveed Anjum >> > > -- With Regards Farrukh Naveed Anjum
