It looks like your Nifi instance is running on your laptop/desktop (e.g. the VM host). My guess would be that name resolution or networking is not properly configured between the host and the guest preventing the data from getting from Nifi to Kafka. What's the contents of /etc/hosts on the VM host? Can you ping node1 from the VM host by name and by IP address?
-Kyle On Mon, Jan 15, 2018 at 6:55 AM, Gaurav Bapat <gauravb3...@gmail.com> wrote: > Failed while waiting for acks from Kafka is what I am getting in Kafka, am > I missing some configuration with Kafka? > > On 15 January 2018 at 16:50, Gaurav Bapat <gauravb3...@gmail.com> wrote: > >> Hi Farrukh, >> >> I cant find any folder by my topic >> >> On 15 January 2018 at 16:33, Farrukh Naveed Anjum < >> anjum.farr...@gmail.com> wrote: >> >>> Can you check /kafaka-logs on your VM box (It should have a folder named >>> your topic). Can you check if it is there ? >>> >>> On Mon, Jan 15, 2018 at 3:49 PM, Gaurav Bapat <gauravb3...@gmail.com> >>> wrote: >>> >>>> I am not getting data into my Kafka topic >>>> >>>> I have used i5 4 Core Processor with 16 GB RAM and I have allocated 12 >>>> GB RAM to my vagrant VM. >>>> >>>> I dont understand how to configure Kafka broker because it is giving me >>>> failed while waiting for acks to Kafka >>>> >>>> >>>> >>>> On 15 January 2018 at 16:10, Farrukh Naveed Anjum < >>>> anjum.farr...@gmail.com> wrote: >>>> >>>>> Can you tell me is your KAFKA Topic getting data ? What are you >>>>> machine specifications ? >>>>> >>>>> >>>>> On Mon, Jan 15, 2018 at 2:56 PM, Gaurav Bapat <gauravb3...@gmail.com> >>>>> wrote: >>>>> >>>>>> Thanks Farrukh, >>>>>> >>>>>> I am not getting data in my kafka topic even after creating one, the >>>>>> issue seems to be with broker config, how to configure Kafka and >>>>>> Zookeeper >>>>>> port? >>>>>> >>>>>> On 15 January 2018 at 13:23, Farrukh Naveed Anjum < >>>>>> anjum.farr...@gmail.com> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I had similar issue it turned out to be the issue in STROM >>>>>>> >>>>>>> No worker is assigned to togolgoy all you need is to add additional >>>>>>> port in >>>>>>> >>>>>>> Ambari -> Storm -> Configs -> supervisor.slot.ports by assigning an >>>>>>> additional port to the list >>>>>>> >>>>>>> >>>>>>> https://community.hortonworks.com/questions/32499/no-workers >>>>>>> -in-storm-for-squid-topology.html >>>>>>> >>>>>>> >>>>>>> I had similar issue and finally got it fixed >>>>>>> >>>>>>> On Mon, Jan 15, 2018 at 8:45 AM, Gaurav Bapat <gauravb3...@gmail.com >>>>>>> > wrote: >>>>>>> >>>>>>>> Storm UI >>>>>>>> >>>>>>>> On 15 January 2018 at 08:59, Gaurav Bapat <gauravb3...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hey Jon, >>>>>>>>> >>>>>>>>> I have Storm UI and the logs are coming from firewalls, servers, >>>>>>>>> etc from other machines(HP ArcSight Logger). >>>>>>>>> >>>>>>>>> I have attached the NiFi screenshots, my logs are coming but there >>>>>>>>> is some error with Kafka and I am having issues with configuring Kafka >>>>>>>>> broker >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 12 January 2018 at 18:14, zeo...@gmail.com <zeo...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> In Ambari under storm you can find the UI under quick links at >>>>>>>>>> the top. That said, the issue seems to be upstream of Metron, in >>>>>>>>>> NiFi. >>>>>>>>>> That is something I can't help with as much, but if you can share the >>>>>>>>>> listensyslog processor config that would be a start. Also, share the >>>>>>>>>> config of the thing that is sending syslog as well (are these local >>>>>>>>>> syslog, >>>>>>>>>> is that machine aggregating syslog from other machines, etc.). >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Jon >>>>>>>>>> >>>>>>>>>> On Fri, Jan 12, 2018, 01:00 Gaurav Bapat <gauravb3...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> I have created a Kafka topic "cef" but my Listen Syslogs is not >>>>>>>>>>> getting logs in the processor. >>>>>>>>>>> >>>>>>>>>>> Also I checked using tcpdump -i and it is getting logs in my >>>>>>>>>>> machine but ListenSyslogs is not getting the logs >>>>>>>>>>> >>>>>>>>>>> On 12 January 2018 at 11:13, Gaurav Bapat <gauravb3...@gmail.com >>>>>>>>>>> > wrote: >>>>>>>>>>> >>>>>>>>>>>> [root@metron incubator-metron]# ./metron-deployment/scripts/pl >>>>>>>>>>>> atform-info.sh >>>>>>>>>>>> Metron 0.4.3 >>>>>>>>>>>> -- >>>>>>>>>>>> * master >>>>>>>>>>>> -- >>>>>>>>>>>> commit c559ed7e1838ec71344eae3d9e37771db2641635 >>>>>>>>>>>> Author: cstella <ceste...@gmail.com> >>>>>>>>>>>> Date: Tue Jan 9 15:28:47 2018 -0500 >>>>>>>>>>>> >>>>>>>>>>>> METRON-1379: Add an OBJECT_GET stellar function closes >>>>>>>>>>>> apache/incubator-metron#880 >>>>>>>>>>>> -- >>>>>>>>>>>> metron-deployment/vagrant/full-dev-platform/Vagrantfile | 2 +- >>>>>>>>>>>> 1 file changed, 1 insertion(+), 1 deletion(-) >>>>>>>>>>>> -- >>>>>>>>>>>> ansible 2.0.0.2 >>>>>>>>>>>> config file = >>>>>>>>>>>> configured module search path = Default w/o overrides >>>>>>>>>>>> -- >>>>>>>>>>>> Vagrant 1.9.6 >>>>>>>>>>>> -- >>>>>>>>>>>> Python 2.7.5 >>>>>>>>>>>> -- >>>>>>>>>>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; >>>>>>>>>>>> 2015-11-10T22:11:47+05:30) >>>>>>>>>>>> Maven home: /opt/maven/current >>>>>>>>>>>> Java version: 1.8.0_151, vendor: Oracle Corporation >>>>>>>>>>>> Java home: /opt/jdk1.8.0_151/jre >>>>>>>>>>>> Default locale: en_US, platform encoding: UTF-8 >>>>>>>>>>>> OS name: "linux", version: "3.10.0-693.11.6.el7.x86_64", arch: >>>>>>>>>>>> "amd64", family: "unix" >>>>>>>>>>>> -- >>>>>>>>>>>> Docker version 1.12.6, build ec8512b/1.12.6 >>>>>>>>>>>> -- >>>>>>>>>>>> node >>>>>>>>>>>> v8.9.3 >>>>>>>>>>>> -- >>>>>>>>>>>> npm >>>>>>>>>>>> 5.5.1 >>>>>>>>>>>> -- >>>>>>>>>>>> g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16) >>>>>>>>>>>> Copyright (C) 2015 Free Software Foundation, Inc. >>>>>>>>>>>> This is free software; see the source for copying conditions. >>>>>>>>>>>> There is NO >>>>>>>>>>>> warranty; not even for MERCHANTABILITY or FITNESS FOR A >>>>>>>>>>>> PARTICULAR PURPOSE. >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Compiler is C++11 compliant >>>>>>>>>>>> -- >>>>>>>>>>>> Linux metron.com 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4 >>>>>>>>>>>> 01:06:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux >>>>>>>>>>>> -- >>>>>>>>>>>> Total System Memory = 15773.3 MB >>>>>>>>>>>> Processor Model: Intel(R) Core(TM) i5-3450 CPU @ 3.10GHz >>>>>>>>>>>> Processor Speed: 3320.875 MHz >>>>>>>>>>>> Processor Speed: 3307.191 MHz >>>>>>>>>>>> Processor Speed: 3376.699 MHz >>>>>>>>>>>> Processor Speed: 3338.917 MHz >>>>>>>>>>>> Total Physical Processors: 4 >>>>>>>>>>>> Total cores: 16 >>>>>>>>>>>> Disk information: >>>>>>>>>>>> /dev/mapper/centos-root 200G 22G 179G 11% / >>>>>>>>>>>> /dev/sda1 2.0G 224M 1.8G 11% /boot >>>>>>>>>>>> /dev/sda2 1022M 12K 1022M 1% /boot/efi >>>>>>>>>>>> /dev/mapper/centos-home 247G 10G 237G 5% /home >>>>>>>>>>>> This CPU appears to support virtualization >>>>>>>>>>>> >>>>>>>>>>>> On 12 January 2018 at 09:25, Gaurav Bapat < >>>>>>>>>>>> gauravb3...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hey Jon, >>>>>>>>>>>>> >>>>>>>>>>>>> Appreciate your timely reply. >>>>>>>>>>>>> >>>>>>>>>>>>> I gone through your answer but still I can't figure out how do >>>>>>>>>>>>> I do parsing/indexing in Storm UI as I cant find any option for >>>>>>>>>>>>> the same. >>>>>>>>>>>>> >>>>>>>>>>>>> Is there any other UI to do parsing/indexing? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 11 January 2018 at 21:22, zeo...@gmail.com < >>>>>>>>>>>>> zeo...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> So, you created a new cef topic, and set up the appropriate >>>>>>>>>>>>>> parser config for it (if not, this >>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source> >>>>>>>>>>>>>> may be helpful)? If so: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Here are some basic troubleshooting steps: >>>>>>>>>>>>>> 1. Validate that the logs are getting onto the kafka topic >>>>>>>>>>>>>> that you are sending to. If they aren't there, the problem is >>>>>>>>>>>>>> upstream >>>>>>>>>>>>>> from Metron. >>>>>>>>>>>>>> 2. If they are getting onto the kafka topic they are being >>>>>>>>>>>>>> directly sent to, check the indexing kafka topic for an enriched >>>>>>>>>>>>>> version of >>>>>>>>>>>>>> those same logs. >>>>>>>>>>>>>> 3. Do a binary search of the various components involved >>>>>>>>>>>>>> with ingest. >>>>>>>>>>>>>> a. If the logs are *not* on the indexing kafka topic, >>>>>>>>>>>>>> check the enrichments topic for those logs. >>>>>>>>>>>>>> b. If the logs are *not* on the enrichments topic, check >>>>>>>>>>>>>> the parser storm topology. >>>>>>>>>>>>>> c. If the logs are on the enrichments topic, but *not* >>>>>>>>>>>>>> indexing, check the enrichments storm topology. >>>>>>>>>>>>>> d. If the logs are on the indexing but *not* Kibana, >>>>>>>>>>>>>> check the indexing storm topic. >>>>>>>>>>>>>> e. If the logs are in on the indexing topic and indexing >>>>>>>>>>>>>> storm topic is in good shape, check elasticsearch directly. >>>>>>>>>>>>>> 4. You should have identified where the issue is at this >>>>>>>>>>>>>> point. Report back here with what you observed, any relevant >>>>>>>>>>>>>> error >>>>>>>>>>>>>> messages, etc. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Side note: We should document a decision tree for >>>>>>>>>>>>>> troubleshooting data ingest. It is fairly straightforward and >>>>>>>>>>>>>> makes me >>>>>>>>>>>>>> wonder if we already have this somewhere and I'm not aware of >>>>>>>>>>>>>> it? It would >>>>>>>>>>>>>> also be a good place to put pointers to some common errors. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Jon >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Jan 11, 2018 at 1:44 AM Gaurav Bapat < >>>>>>>>>>>>>> gauravb3...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hello everyone, I have deployed Metron on a single node >>>>>>>>>>>>>>> machine and I would like to know how do I get Syslogs from NiFi >>>>>>>>>>>>>>> into Kibana >>>>>>>>>>>>>>> dashboard? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I have created a Kafka topic by the name "cef" and I can see >>>>>>>>>>>>>>> that the topic exists in >>>>>>>>>>>>>>> Metron Configuration but I am unable to connect it with >>>>>>>>>>>>>>> Kibana >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Need Help!! >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>>> Jon >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Jon >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> With Regards >>>>>>> Farrukh Naveed Anjum >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> With Regards >>>>> Farrukh Naveed Anjum >>>>> >>>> >>>> >>> >>> >>> -- >>> With Regards >>> Farrukh Naveed Anjum >>> >> >> >
