Hi Kyle, I saw that I can ping from my OS to VM and from VM to OS. Looks like this is some Kafka or Zookeeper environment variables setup issue, do I need to configure that in vagrant ssh?
On 16 January 2018 at 08:59, Gaurav Bapat <gauravb3...@gmail.com> wrote: > Hey Kyle, > > I am running NiFi not on Ambari but on localhost:8089, I can ping from my > OS terminal to node1 but can't ping from node1 to my OS terminal, I have > attached few screenshots and the contents of /etc/hosts > > Thank You! > > On 15 January 2018 at 20:04, Kyle Richardson <kylerichards...@gmail.com> > wrote: > >> It looks like your Nifi instance is running on your laptop/desktop (e.g. >> the VM host). My guess would be that name resolution or networking is not >> properly configured between the host and the guest preventing the data from >> getting from Nifi to Kafka. What's the contents of /etc/hosts on the VM >> host? Can you ping node1 from the VM host by name and by IP address? >> >> -Kyle >> >> On Mon, Jan 15, 2018 at 6:55 AM, Gaurav Bapat <gauravb3...@gmail.com> >> wrote: >> >>> Failed while waiting for acks from Kafka is what I am getting in Kafka, >>> am I missing some configuration with Kafka? >>> >>> On 15 January 2018 at 16:50, Gaurav Bapat <gauravb3...@gmail.com> wrote: >>> >>>> Hi Farrukh, >>>> >>>> I cant find any folder by my topic >>>> >>>> On 15 January 2018 at 16:33, Farrukh Naveed Anjum < >>>> anjum.farr...@gmail.com> wrote: >>>> >>>>> Can you check /kafaka-logs on your VM box (It should have a folder >>>>> named your topic). Can you check if it is there ? >>>>> >>>>> On Mon, Jan 15, 2018 at 3:49 PM, Gaurav Bapat <gauravb3...@gmail.com> >>>>> wrote: >>>>> >>>>>> I am not getting data into my Kafka topic >>>>>> >>>>>> I have used i5 4 Core Processor with 16 GB RAM and I have allocated >>>>>> 12 GB RAM to my vagrant VM. >>>>>> >>>>>> I dont understand how to configure Kafka broker because it is giving >>>>>> me failed while waiting for acks to Kafka >>>>>> >>>>>> >>>>>> >>>>>> On 15 January 2018 at 16:10, Farrukh Naveed Anjum < >>>>>> anjum.farr...@gmail.com> wrote: >>>>>> >>>>>>> Can you tell me is your KAFKA Topic getting data ? What are you >>>>>>> machine specifications ? >>>>>>> >>>>>>> >>>>>>> On Mon, Jan 15, 2018 at 2:56 PM, Gaurav Bapat <gauravb3...@gmail.com >>>>>>> > wrote: >>>>>>> >>>>>>>> Thanks Farrukh, >>>>>>>> >>>>>>>> I am not getting data in my kafka topic even after creating one, >>>>>>>> the issue seems to be with broker config, how to configure Kafka and >>>>>>>> Zookeeper port? >>>>>>>> >>>>>>>> On 15 January 2018 at 13:23, Farrukh Naveed Anjum < >>>>>>>> anjum.farr...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I had similar issue it turned out to be the issue in STROM >>>>>>>>> >>>>>>>>> No worker is assigned to togolgoy all you need is to add >>>>>>>>> additional port in >>>>>>>>> >>>>>>>>> Ambari -> Storm -> Configs -> supervisor.slot.ports by assigning >>>>>>>>> an additional port to the list >>>>>>>>> >>>>>>>>> >>>>>>>>> https://community.hortonworks.com/questions/32499/no-workers >>>>>>>>> -in-storm-for-squid-topology.html >>>>>>>>> >>>>>>>>> >>>>>>>>> I had similar issue and finally got it fixed >>>>>>>>> >>>>>>>>> On Mon, Jan 15, 2018 at 8:45 AM, Gaurav Bapat < >>>>>>>>> gauravb3...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Storm UI >>>>>>>>>> >>>>>>>>>> On 15 January 2018 at 08:59, Gaurav Bapat <gauravb3...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hey Jon, >>>>>>>>>>> >>>>>>>>>>> I have Storm UI and the logs are coming from firewalls, servers, >>>>>>>>>>> etc from other machines(HP ArcSight Logger). >>>>>>>>>>> >>>>>>>>>>> I have attached the NiFi screenshots, my logs are coming but >>>>>>>>>>> there is some error with Kafka and I am having issues with >>>>>>>>>>> configuring >>>>>>>>>>> Kafka broker >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 12 January 2018 at 18:14, zeo...@gmail.com <zeo...@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> In Ambari under storm you can find the UI under quick links at >>>>>>>>>>>> the top. That said, the issue seems to be upstream of Metron, in >>>>>>>>>>>> NiFi. >>>>>>>>>>>> That is something I can't help with as much, but if you can share >>>>>>>>>>>> the >>>>>>>>>>>> listensyslog processor config that would be a start. Also, share >>>>>>>>>>>> the >>>>>>>>>>>> config of the thing that is sending syslog as well (are these >>>>>>>>>>>> local syslog, >>>>>>>>>>>> is that machine aggregating syslog from other machines, etc.). >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Jon >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Jan 12, 2018, 01:00 Gaurav Bapat <gauravb3...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I have created a Kafka topic "cef" but my Listen Syslogs is >>>>>>>>>>>>> not getting logs in the processor. >>>>>>>>>>>>> >>>>>>>>>>>>> Also I checked using tcpdump -i and it is getting logs in my >>>>>>>>>>>>> machine but ListenSyslogs is not getting the logs >>>>>>>>>>>>> >>>>>>>>>>>>> On 12 January 2018 at 11:13, Gaurav Bapat < >>>>>>>>>>>>> gauravb3...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> [root@metron incubator-metron]# >>>>>>>>>>>>>> ./metron-deployment/scripts/platform-info.sh >>>>>>>>>>>>>> Metron 0.4.3 >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> * master >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> commit c559ed7e1838ec71344eae3d9e37771db2641635 >>>>>>>>>>>>>> Author: cstella <ceste...@gmail.com> >>>>>>>>>>>>>> Date: Tue Jan 9 15:28:47 2018 -0500 >>>>>>>>>>>>>> >>>>>>>>>>>>>> METRON-1379: Add an OBJECT_GET stellar function closes >>>>>>>>>>>>>> apache/incubator-metron#880 >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> metron-deployment/vagrant/full-dev-platform/Vagrantfile | 2 >>>>>>>>>>>>>> +- >>>>>>>>>>>>>> 1 file changed, 1 insertion(+), 1 deletion(-) >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> ansible 2.0.0.2 >>>>>>>>>>>>>> config file = >>>>>>>>>>>>>> configured module search path = Default w/o overrides >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Vagrant 1.9.6 >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Python 2.7.5 >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; >>>>>>>>>>>>>> 2015-11-10T22:11:47+05:30) >>>>>>>>>>>>>> Maven home: /opt/maven/current >>>>>>>>>>>>>> Java version: 1.8.0_151, vendor: Oracle Corporation >>>>>>>>>>>>>> Java home: /opt/jdk1.8.0_151/jre >>>>>>>>>>>>>> Default locale: en_US, platform encoding: UTF-8 >>>>>>>>>>>>>> OS name: "linux", version: "3.10.0-693.11.6.el7.x86_64", >>>>>>>>>>>>>> arch: "amd64", family: "unix" >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Docker version 1.12.6, build ec8512b/1.12.6 >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> node >>>>>>>>>>>>>> v8.9.3 >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> npm >>>>>>>>>>>>>> 5.5.1 >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16) >>>>>>>>>>>>>> Copyright (C) 2015 Free Software Foundation, Inc. >>>>>>>>>>>>>> This is free software; see the source for copying >>>>>>>>>>>>>> conditions. There is NO >>>>>>>>>>>>>> warranty; not even for MERCHANTABILITY or FITNESS FOR A >>>>>>>>>>>>>> PARTICULAR PURPOSE. >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Compiler is C++11 compliant >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Linux metron.com 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4 >>>>>>>>>>>>>> 01:06:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Total System Memory = 15773.3 MB >>>>>>>>>>>>>> Processor Model: Intel(R) Core(TM) i5-3450 CPU @ 3.10GHz >>>>>>>>>>>>>> Processor Speed: 3320.875 MHz >>>>>>>>>>>>>> Processor Speed: 3307.191 MHz >>>>>>>>>>>>>> Processor Speed: 3376.699 MHz >>>>>>>>>>>>>> Processor Speed: 3338.917 MHz >>>>>>>>>>>>>> Total Physical Processors: 4 >>>>>>>>>>>>>> Total cores: 16 >>>>>>>>>>>>>> Disk information: >>>>>>>>>>>>>> /dev/mapper/centos-root 200G 22G 179G 11% / >>>>>>>>>>>>>> /dev/sda1 2.0G 224M 1.8G 11% /boot >>>>>>>>>>>>>> /dev/sda2 1022M 12K 1022M 1% /boot/efi >>>>>>>>>>>>>> /dev/mapper/centos-home 247G 10G 237G 5% /home >>>>>>>>>>>>>> This CPU appears to support virtualization >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 12 January 2018 at 09:25, Gaurav Bapat < >>>>>>>>>>>>>> gauravb3...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hey Jon, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Appreciate your timely reply. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I gone through your answer but still I can't figure out how >>>>>>>>>>>>>>> do I do parsing/indexing in Storm UI as I cant find any option >>>>>>>>>>>>>>> for the same. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Is there any other UI to do parsing/indexing? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 11 January 2018 at 21:22, zeo...@gmail.com < >>>>>>>>>>>>>>> zeo...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> So, you created a new cef topic, and set up the appropriate >>>>>>>>>>>>>>>> parser config for it (if not, this >>>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source> >>>>>>>>>>>>>>>> may be helpful)? If so: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Here are some basic troubleshooting steps: >>>>>>>>>>>>>>>> 1. Validate that the logs are getting onto the kafka topic >>>>>>>>>>>>>>>> that you are sending to. If they aren't there, the problem is >>>>>>>>>>>>>>>> upstream >>>>>>>>>>>>>>>> from Metron. >>>>>>>>>>>>>>>> 2. If they are getting onto the kafka topic they are being >>>>>>>>>>>>>>>> directly sent to, check the indexing kafka topic for an >>>>>>>>>>>>>>>> enriched version of >>>>>>>>>>>>>>>> those same logs. >>>>>>>>>>>>>>>> 3. Do a binary search of the various components involved >>>>>>>>>>>>>>>> with ingest. >>>>>>>>>>>>>>>> a. If the logs are *not* on the indexing kafka topic, >>>>>>>>>>>>>>>> check the enrichments topic for those logs. >>>>>>>>>>>>>>>> b. If the logs are *not* on the enrichments topic, >>>>>>>>>>>>>>>> check the parser storm topology. >>>>>>>>>>>>>>>> c. If the logs are on the enrichments topic, but *not* >>>>>>>>>>>>>>>> indexing, check the enrichments storm topology. >>>>>>>>>>>>>>>> d. If the logs are on the indexing but *not* Kibana, >>>>>>>>>>>>>>>> check the indexing storm topic. >>>>>>>>>>>>>>>> e. If the logs are in on the indexing topic and >>>>>>>>>>>>>>>> indexing storm topic is in good shape, check elasticsearch >>>>>>>>>>>>>>>> directly. >>>>>>>>>>>>>>>> 4. You should have identified where the issue is at this >>>>>>>>>>>>>>>> point. Report back here with what you observed, any relevant >>>>>>>>>>>>>>>> error >>>>>>>>>>>>>>>> messages, etc. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Side note: We should document a decision tree for >>>>>>>>>>>>>>>> troubleshooting data ingest. It is fairly straightforward and >>>>>>>>>>>>>>>> makes me >>>>>>>>>>>>>>>> wonder if we already have this somewhere and I'm not aware of >>>>>>>>>>>>>>>> it? It would >>>>>>>>>>>>>>>> also be a good place to put pointers to some common errors. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Jon >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Thu, Jan 11, 2018 at 1:44 AM Gaurav Bapat < >>>>>>>>>>>>>>>> gauravb3...@gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hello everyone, I have deployed Metron on a single node >>>>>>>>>>>>>>>>> machine and I would like to know how do I get Syslogs from >>>>>>>>>>>>>>>>> NiFi into Kibana >>>>>>>>>>>>>>>>> dashboard? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I have created a Kafka topic by the name "cef" and I can >>>>>>>>>>>>>>>>> see that the topic exists in >>>>>>>>>>>>>>>>> Metron Configuration but I am unable to connect it with >>>>>>>>>>>>>>>>> Kibana >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Need Help!! >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Jon >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> Jon >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> With Regards >>>>>>>>> Farrukh Naveed Anjum >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> With Regards >>>>>>> Farrukh Naveed Anjum >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> With Regards >>>>> Farrukh Naveed Anjum >>>>> >>>> >>>> >>> >> >
