Hi Farrukh, I cant find any folder by my topic
On 15 January 2018 at 16:33, Farrukh Naveed Anjum <anjum.farr...@gmail.com> wrote: > Can you check /kafaka-logs on your VM box (It should have a folder named > your topic). Can you check if it is there ? > > On Mon, Jan 15, 2018 at 3:49 PM, Gaurav Bapat <gauravb3...@gmail.com> > wrote: > >> I am not getting data into my Kafka topic >> >> I have used i5 4 Core Processor with 16 GB RAM and I have allocated 12 GB >> RAM to my vagrant VM. >> >> I dont understand how to configure Kafka broker because it is giving me >> failed while waiting for acks to Kafka >> >> >> >> On 15 January 2018 at 16:10, Farrukh Naveed Anjum < >> anjum.farr...@gmail.com> wrote: >> >>> Can you tell me is your KAFKA Topic getting data ? What are you machine >>> specifications ? >>> >>> >>> On Mon, Jan 15, 2018 at 2:56 PM, Gaurav Bapat <gauravb3...@gmail.com> >>> wrote: >>> >>>> Thanks Farrukh, >>>> >>>> I am not getting data in my kafka topic even after creating one, the >>>> issue seems to be with broker config, how to configure Kafka and Zookeeper >>>> port? >>>> >>>> On 15 January 2018 at 13:23, Farrukh Naveed Anjum < >>>> anjum.farr...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> I had similar issue it turned out to be the issue in STROM >>>>> >>>>> No worker is assigned to togolgoy all you need is to add additional >>>>> port in >>>>> >>>>> Ambari -> Storm -> Configs -> supervisor.slot.ports by assigning an >>>>> additional port to the list >>>>> >>>>> >>>>> https://community.hortonworks.com/questions/32499/no-workers >>>>> -in-storm-for-squid-topology.html >>>>> >>>>> >>>>> I had similar issue and finally got it fixed >>>>> >>>>> On Mon, Jan 15, 2018 at 8:45 AM, Gaurav Bapat <gauravb3...@gmail.com> >>>>> wrote: >>>>> >>>>>> Storm UI >>>>>> >>>>>> On 15 January 2018 at 08:59, Gaurav Bapat <gauravb3...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hey Jon, >>>>>>> >>>>>>> I have Storm UI and the logs are coming from firewalls, servers, etc >>>>>>> from other machines(HP ArcSight Logger). >>>>>>> >>>>>>> I have attached the NiFi screenshots, my logs are coming but there >>>>>>> is some error with Kafka and I am having issues with configuring Kafka >>>>>>> broker >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 12 January 2018 at 18:14, zeo...@gmail.com <zeo...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> In Ambari under storm you can find the UI under quick links at the >>>>>>>> top. That said, the issue seems to be upstream of Metron, in NiFi. >>>>>>>> That >>>>>>>> is something I can't help with as much, but if you can share the >>>>>>>> listensyslog processor config that would be a start. Also, share the >>>>>>>> config of the thing that is sending syslog as well (are these local >>>>>>>> syslog, >>>>>>>> is that machine aggregating syslog from other machines, etc.). Thanks, >>>>>>>> >>>>>>>> Jon >>>>>>>> >>>>>>>> On Fri, Jan 12, 2018, 01:00 Gaurav Bapat <gauravb3...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I have created a Kafka topic "cef" but my Listen Syslogs is not >>>>>>>>> getting logs in the processor. >>>>>>>>> >>>>>>>>> Also I checked using tcpdump -i and it is getting logs in my >>>>>>>>> machine but ListenSyslogs is not getting the logs >>>>>>>>> >>>>>>>>> On 12 January 2018 at 11:13, Gaurav Bapat <gauravb3...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> [root@metron incubator-metron]# ./metron-deployment/scripts/pl >>>>>>>>>> atform-info.sh >>>>>>>>>> Metron 0.4.3 >>>>>>>>>> -- >>>>>>>>>> * master >>>>>>>>>> -- >>>>>>>>>> commit c559ed7e1838ec71344eae3d9e37771db2641635 >>>>>>>>>> Author: cstella <ceste...@gmail.com> >>>>>>>>>> Date: Tue Jan 9 15:28:47 2018 -0500 >>>>>>>>>> >>>>>>>>>> METRON-1379: Add an OBJECT_GET stellar function closes >>>>>>>>>> apache/incubator-metron#880 >>>>>>>>>> -- >>>>>>>>>> metron-deployment/vagrant/full-dev-platform/Vagrantfile | 2 +- >>>>>>>>>> 1 file changed, 1 insertion(+), 1 deletion(-) >>>>>>>>>> -- >>>>>>>>>> ansible 2.0.0.2 >>>>>>>>>> config file = >>>>>>>>>> configured module search path = Default w/o overrides >>>>>>>>>> -- >>>>>>>>>> Vagrant 1.9.6 >>>>>>>>>> -- >>>>>>>>>> Python 2.7.5 >>>>>>>>>> -- >>>>>>>>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; >>>>>>>>>> 2015-11-10T22:11:47+05:30) >>>>>>>>>> Maven home: /opt/maven/current >>>>>>>>>> Java version: 1.8.0_151, vendor: Oracle Corporation >>>>>>>>>> Java home: /opt/jdk1.8.0_151/jre >>>>>>>>>> Default locale: en_US, platform encoding: UTF-8 >>>>>>>>>> OS name: "linux", version: "3.10.0-693.11.6.el7.x86_64", arch: >>>>>>>>>> "amd64", family: "unix" >>>>>>>>>> -- >>>>>>>>>> Docker version 1.12.6, build ec8512b/1.12.6 >>>>>>>>>> -- >>>>>>>>>> node >>>>>>>>>> v8.9.3 >>>>>>>>>> -- >>>>>>>>>> npm >>>>>>>>>> 5.5.1 >>>>>>>>>> -- >>>>>>>>>> g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16) >>>>>>>>>> Copyright (C) 2015 Free Software Foundation, Inc. >>>>>>>>>> This is free software; see the source for copying conditions. >>>>>>>>>> There is NO >>>>>>>>>> warranty; not even for MERCHANTABILITY or FITNESS FOR A >>>>>>>>>> PARTICULAR PURPOSE. >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Compiler is C++11 compliant >>>>>>>>>> -- >>>>>>>>>> Linux metron.com 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4 >>>>>>>>>> 01:06:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux >>>>>>>>>> -- >>>>>>>>>> Total System Memory = 15773.3 MB >>>>>>>>>> Processor Model: Intel(R) Core(TM) i5-3450 CPU @ 3.10GHz >>>>>>>>>> Processor Speed: 3320.875 MHz >>>>>>>>>> Processor Speed: 3307.191 MHz >>>>>>>>>> Processor Speed: 3376.699 MHz >>>>>>>>>> Processor Speed: 3338.917 MHz >>>>>>>>>> Total Physical Processors: 4 >>>>>>>>>> Total cores: 16 >>>>>>>>>> Disk information: >>>>>>>>>> /dev/mapper/centos-root 200G 22G 179G 11% / >>>>>>>>>> /dev/sda1 2.0G 224M 1.8G 11% /boot >>>>>>>>>> /dev/sda2 1022M 12K 1022M 1% /boot/efi >>>>>>>>>> /dev/mapper/centos-home 247G 10G 237G 5% /home >>>>>>>>>> This CPU appears to support virtualization >>>>>>>>>> >>>>>>>>>> On 12 January 2018 at 09:25, Gaurav Bapat <gauravb3...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hey Jon, >>>>>>>>>>> >>>>>>>>>>> Appreciate your timely reply. >>>>>>>>>>> >>>>>>>>>>> I gone through your answer but still I can't figure out how do I >>>>>>>>>>> do parsing/indexing in Storm UI as I cant find any option for the >>>>>>>>>>> same. >>>>>>>>>>> >>>>>>>>>>> Is there any other UI to do parsing/indexing? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 11 January 2018 at 21:22, zeo...@gmail.com <zeo...@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> So, you created a new cef topic, and set up the appropriate >>>>>>>>>>>> parser config for it (if not, this >>>>>>>>>>>> <https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source> >>>>>>>>>>>> may be helpful)? If so: >>>>>>>>>>>> >>>>>>>>>>>> Here are some basic troubleshooting steps: >>>>>>>>>>>> 1. Validate that the logs are getting onto the kafka topic >>>>>>>>>>>> that you are sending to. If they aren't there, the problem is >>>>>>>>>>>> upstream >>>>>>>>>>>> from Metron. >>>>>>>>>>>> 2. If they are getting onto the kafka topic they are being >>>>>>>>>>>> directly sent to, check the indexing kafka topic for an enriched >>>>>>>>>>>> version of >>>>>>>>>>>> those same logs. >>>>>>>>>>>> 3. Do a binary search of the various components involved with >>>>>>>>>>>> ingest. >>>>>>>>>>>> a. If the logs are *not* on the indexing kafka topic, >>>>>>>>>>>> check the enrichments topic for those logs. >>>>>>>>>>>> b. If the logs are *not* on the enrichments topic, check >>>>>>>>>>>> the parser storm topology. >>>>>>>>>>>> c. If the logs are on the enrichments topic, but *not* >>>>>>>>>>>> indexing, check the enrichments storm topology. >>>>>>>>>>>> d. If the logs are on the indexing but *not* Kibana, check >>>>>>>>>>>> the indexing storm topic. >>>>>>>>>>>> e. If the logs are in on the indexing topic and indexing >>>>>>>>>>>> storm topic is in good shape, check elasticsearch directly. >>>>>>>>>>>> 4. You should have identified where the issue is at this >>>>>>>>>>>> point. Report back here with what you observed, any relevant error >>>>>>>>>>>> messages, etc. >>>>>>>>>>>> >>>>>>>>>>>> Side note: We should document a decision tree for >>>>>>>>>>>> troubleshooting data ingest. It is fairly straightforward and >>>>>>>>>>>> makes me >>>>>>>>>>>> wonder if we already have this somewhere and I'm not aware of it? >>>>>>>>>>>> It would >>>>>>>>>>>> also be a good place to put pointers to some common errors. >>>>>>>>>>>> >>>>>>>>>>>> Jon >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Jan 11, 2018 at 1:44 AM Gaurav Bapat < >>>>>>>>>>>> gauravb3...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hello everyone, I have deployed Metron on a single node >>>>>>>>>>>>> machine and I would like to know how do I get Syslogs from NiFi >>>>>>>>>>>>> into Kibana >>>>>>>>>>>>> dashboard? >>>>>>>>>>>>> >>>>>>>>>>>>> I have created a Kafka topic by the name "cef" and I can see >>>>>>>>>>>>> that the topic exists in >>>>>>>>>>>>> Metron Configuration but I am unable to connect it with Kibana >>>>>>>>>>>>> >>>>>>>>>>>>> Need Help!! >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> Jon >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>> >>>>>>>> Jon >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> With Regards >>>>> Farrukh Naveed Anjum >>>>> >>>> >>>> >>> >>> >>> -- >>> With Regards >>> Farrukh Naveed Anjum >>> >> >> > > > -- > With Regards > Farrukh Naveed Anjum >
