But I cant find how to configure it On 16 January 2018 at 11:38, Farrukh Naveed Anjum <anjum.farr...@gmail.com> wrote:
> yes, do configure it as per metron reference usecase > > On Tue, Jan 16, 2018 at 8:35 AM, Gaurav Bapat <gauravb3...@gmail.com> > wrote: > >> Hi Kyle, >> >> I saw that I can ping from my OS to VM and from VM to OS. Looks like this >> is some Kafka or Zookeeper environment variables setup issue, do I need to >> configure that in vagrant ssh? >> >> On 16 January 2018 at 08:59, Gaurav Bapat <gauravb3...@gmail.com> wrote: >> >>> Hey Kyle, >>> >>> I am running NiFi not on Ambari but on localhost:8089, I can ping from >>> my OS terminal to node1 but can't ping from node1 to my OS terminal, I have >>> attached few screenshots and the contents of /etc/hosts >>> >>> Thank You! >>> >>> On 15 January 2018 at 20:04, Kyle Richardson <kylerichards...@gmail.com> >>> wrote: >>> >>>> It looks like your Nifi instance is running on your laptop/desktop >>>> (e.g. the VM host). My guess would be that name resolution or networking is >>>> not properly configured between the host and the guest preventing the data >>>> from getting from Nifi to Kafka. What's the contents of /etc/hosts on the >>>> VM host? Can you ping node1 from the VM host by name and by IP address? >>>> >>>> -Kyle >>>> >>>> On Mon, Jan 15, 2018 at 6:55 AM, Gaurav Bapat <gauravb3...@gmail.com> >>>> wrote: >>>> >>>>> Failed while waiting for acks from Kafka is what I am getting in >>>>> Kafka, am I missing some configuration with Kafka? >>>>> >>>>> On 15 January 2018 at 16:50, Gaurav Bapat <gauravb3...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Farrukh, >>>>>> >>>>>> I cant find any folder by my topic >>>>>> >>>>>> On 15 January 2018 at 16:33, Farrukh Naveed Anjum < >>>>>> anjum.farr...@gmail.com> wrote: >>>>>> >>>>>>> Can you check /kafaka-logs on your VM box (It should have a folder >>>>>>> named your topic). Can you check if it is there ? >>>>>>> >>>>>>> On Mon, Jan 15, 2018 at 3:49 PM, Gaurav Bapat <gauravb3...@gmail.com >>>>>>> > wrote: >>>>>>> >>>>>>>> I am not getting data into my Kafka topic >>>>>>>> >>>>>>>> I have used i5 4 Core Processor with 16 GB RAM and I have allocated >>>>>>>> 12 GB RAM to my vagrant VM. >>>>>>>> >>>>>>>> I dont understand how to configure Kafka broker because it is >>>>>>>> giving me failed while waiting for acks to Kafka >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 15 January 2018 at 16:10, Farrukh Naveed Anjum < >>>>>>>> anjum.farr...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Can you tell me is your KAFKA Topic getting data ? What are you >>>>>>>>> machine specifications ? >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Jan 15, 2018 at 2:56 PM, Gaurav Bapat < >>>>>>>>> gauravb3...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Thanks Farrukh, >>>>>>>>>> >>>>>>>>>> I am not getting data in my kafka topic even after creating one, >>>>>>>>>> the issue seems to be with broker config, how to configure Kafka and >>>>>>>>>> Zookeeper port? >>>>>>>>>> >>>>>>>>>> On 15 January 2018 at 13:23, Farrukh Naveed Anjum < >>>>>>>>>> anjum.farr...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I had similar issue it turned out to be the issue in STROM >>>>>>>>>>> >>>>>>>>>>> No worker is assigned to togolgoy all you need is to add >>>>>>>>>>> additional port in >>>>>>>>>>> >>>>>>>>>>> Ambari -> Storm -> Configs -> supervisor.slot.ports by >>>>>>>>>>> assigning an additional port to the list >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> https://community.hortonworks.com/questions/32499/no-workers >>>>>>>>>>> -in-storm-for-squid-topology.html >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I had similar issue and finally got it fixed >>>>>>>>>>> >>>>>>>>>>> On Mon, Jan 15, 2018 at 8:45 AM, Gaurav Bapat < >>>>>>>>>>> gauravb3...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Storm UI >>>>>>>>>>>> >>>>>>>>>>>> On 15 January 2018 at 08:59, Gaurav Bapat < >>>>>>>>>>>> gauravb3...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hey Jon, >>>>>>>>>>>>> >>>>>>>>>>>>> I have Storm UI and the logs are coming from firewalls, >>>>>>>>>>>>> servers, etc from other machines(HP ArcSight Logger). >>>>>>>>>>>>> >>>>>>>>>>>>> I have attached the NiFi screenshots, my logs are coming but >>>>>>>>>>>>> there is some error with Kafka and I am having issues with >>>>>>>>>>>>> configuring >>>>>>>>>>>>> Kafka broker >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 12 January 2018 at 18:14, zeo...@gmail.com < >>>>>>>>>>>>> zeo...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> In Ambari under storm you can find the UI under quick links >>>>>>>>>>>>>> at the top. That said, the issue seems to be upstream of >>>>>>>>>>>>>> Metron, in NiFi. >>>>>>>>>>>>>> That is something I can't help with as much, but if you can >>>>>>>>>>>>>> share the >>>>>>>>>>>>>> listensyslog processor config that would be a start. Also, >>>>>>>>>>>>>> share the >>>>>>>>>>>>>> config of the thing that is sending syslog as well (are these >>>>>>>>>>>>>> local syslog, >>>>>>>>>>>>>> is that machine aggregating syslog from other machines, etc.). >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Jon >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Jan 12, 2018, 01:00 Gaurav Bapat < >>>>>>>>>>>>>> gauravb3...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I have created a Kafka topic "cef" but my Listen Syslogs is >>>>>>>>>>>>>>> not getting logs in the processor. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Also I checked using tcpdump -i and it is getting logs in my >>>>>>>>>>>>>>> machine but ListenSyslogs is not getting the logs >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 12 January 2018 at 11:13, Gaurav Bapat < >>>>>>>>>>>>>>> gauravb3...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [root@metron incubator-metron]# >>>>>>>>>>>>>>>> ./metron-deployment/scripts/platform-info.sh >>>>>>>>>>>>>>>> Metron 0.4.3 >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> * master >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> commit c559ed7e1838ec71344eae3d9e37771db2641635 >>>>>>>>>>>>>>>> Author: cstella <ceste...@gmail.com> >>>>>>>>>>>>>>>> Date: Tue Jan 9 15:28:47 2018 -0500 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> METRON-1379: Add an OBJECT_GET stellar function closes >>>>>>>>>>>>>>>> apache/incubator-metron#880 >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> metron-deployment/vagrant/full-dev-platform/Vagrantfile | >>>>>>>>>>>>>>>> 2 +- >>>>>>>>>>>>>>>> 1 file changed, 1 insertion(+), 1 deletion(-) >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> ansible 2.0.0.2 >>>>>>>>>>>>>>>> config file = >>>>>>>>>>>>>>>> configured module search path = Default w/o overrides >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Vagrant 1.9.6 >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Python 2.7.5 >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; >>>>>>>>>>>>>>>> 2015-11-10T22:11:47+05:30) >>>>>>>>>>>>>>>> Maven home: /opt/maven/current >>>>>>>>>>>>>>>> Java version: 1.8.0_151, vendor: Oracle Corporation >>>>>>>>>>>>>>>> Java home: /opt/jdk1.8.0_151/jre >>>>>>>>>>>>>>>> Default locale: en_US, platform encoding: UTF-8 >>>>>>>>>>>>>>>> OS name: "linux", version: "3.10.0-693.11.6.el7.x86_64", >>>>>>>>>>>>>>>> arch: "amd64", family: "unix" >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Docker version 1.12.6, build ec8512b/1.12.6 >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> node >>>>>>>>>>>>>>>> v8.9.3 >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> npm >>>>>>>>>>>>>>>> 5.5.1 >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16) >>>>>>>>>>>>>>>> Copyright (C) 2015 Free Software Foundation, Inc. >>>>>>>>>>>>>>>> This is free software; see the source for copying >>>>>>>>>>>>>>>> conditions. There is NO >>>>>>>>>>>>>>>> warranty; not even for MERCHANTABILITY or FITNESS FOR A >>>>>>>>>>>>>>>> PARTICULAR PURPOSE. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Compiler is C++11 compliant >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Linux metron.com 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan >>>>>>>>>>>>>>>> 4 01:06:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Total System Memory = 15773.3 MB >>>>>>>>>>>>>>>> Processor Model: Intel(R) Core(TM) i5-3450 CPU @ 3.10GHz >>>>>>>>>>>>>>>> Processor Speed: 3320.875 MHz >>>>>>>>>>>>>>>> Processor Speed: 3307.191 MHz >>>>>>>>>>>>>>>> Processor Speed: 3376.699 MHz >>>>>>>>>>>>>>>> Processor Speed: 3338.917 MHz >>>>>>>>>>>>>>>> Total Physical Processors: 4 >>>>>>>>>>>>>>>> Total cores: 16 >>>>>>>>>>>>>>>> Disk information: >>>>>>>>>>>>>>>> /dev/mapper/centos-root 200G 22G 179G 11% / >>>>>>>>>>>>>>>> /dev/sda1 2.0G 224M 1.8G 11% /boot >>>>>>>>>>>>>>>> /dev/sda2 1022M 12K 1022M 1% /boot/efi >>>>>>>>>>>>>>>> /dev/mapper/centos-home 247G 10G 237G 5% /home >>>>>>>>>>>>>>>> This CPU appears to support virtualization >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 12 January 2018 at 09:25, Gaurav Bapat < >>>>>>>>>>>>>>>> gauravb3...@gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hey Jon, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Appreciate your timely reply. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I gone through your answer but still I can't figure out >>>>>>>>>>>>>>>>> how do I do parsing/indexing in Storm UI as I cant find any >>>>>>>>>>>>>>>>> option for the >>>>>>>>>>>>>>>>> same. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Is there any other UI to do parsing/indexing? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 11 January 2018 at 21:22, zeo...@gmail.com < >>>>>>>>>>>>>>>>> zeo...@gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> So, you created a new cef topic, and set up the >>>>>>>>>>>>>>>>>> appropriate parser config for it (if not, this >>>>>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source> >>>>>>>>>>>>>>>>>> may be helpful)? If so: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Here are some basic troubleshooting steps: >>>>>>>>>>>>>>>>>> 1. Validate that the logs are getting onto the >>>>>>>>>>>>>>>>>> kafka topic that you are sending to. If they aren't there, >>>>>>>>>>>>>>>>>> the problem is >>>>>>>>>>>>>>>>>> upstream from Metron. >>>>>>>>>>>>>>>>>> 2. If they are getting onto the kafka topic they are >>>>>>>>>>>>>>>>>> being directly sent to, check the indexing kafka topic for >>>>>>>>>>>>>>>>>> an enriched >>>>>>>>>>>>>>>>>> version of those same logs. >>>>>>>>>>>>>>>>>> 3. Do a binary search of the various components involved >>>>>>>>>>>>>>>>>> with ingest. >>>>>>>>>>>>>>>>>> a. If the logs are *not* on the indexing kafka >>>>>>>>>>>>>>>>>> topic, check the enrichments topic for those logs. >>>>>>>>>>>>>>>>>> b. If the logs are *not* on the enrichments topic, >>>>>>>>>>>>>>>>>> check the parser storm topology. >>>>>>>>>>>>>>>>>> c. If the logs are on the enrichments topic, but >>>>>>>>>>>>>>>>>> *not* indexing, check the enrichments storm topology. >>>>>>>>>>>>>>>>>> d. If the logs are on the indexing but *not* Kibana, >>>>>>>>>>>>>>>>>> check the indexing storm topic. >>>>>>>>>>>>>>>>>> e. If the logs are in on the indexing topic and >>>>>>>>>>>>>>>>>> indexing storm topic is in good shape, check >>>>>>>>>>>>>>>>>> elasticsearch directly. >>>>>>>>>>>>>>>>>> 4. You should have identified where the issue is at this >>>>>>>>>>>>>>>>>> point. Report back here with what you observed, any >>>>>>>>>>>>>>>>>> relevant error >>>>>>>>>>>>>>>>>> messages, etc. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Side note: We should document a decision tree for >>>>>>>>>>>>>>>>>> troubleshooting data ingest. It is fairly straightforward >>>>>>>>>>>>>>>>>> and makes me >>>>>>>>>>>>>>>>>> wonder if we already have this somewhere and I'm not aware >>>>>>>>>>>>>>>>>> of it? It would >>>>>>>>>>>>>>>>>> also be a good place to put pointers to some common errors. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Jon >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Thu, Jan 11, 2018 at 1:44 AM Gaurav Bapat < >>>>>>>>>>>>>>>>>> gauravb3...@gmail.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hello everyone, I have deployed Metron on a single node >>>>>>>>>>>>>>>>>>> machine and I would like to know how do I get Syslogs from >>>>>>>>>>>>>>>>>>> NiFi into Kibana >>>>>>>>>>>>>>>>>>> dashboard? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I have created a Kafka topic by the name "cef" and I can >>>>>>>>>>>>>>>>>>> see that the topic exists in >>>>>>>>>>>>>>>>>>> Metron Configuration but I am unable to connect it with >>>>>>>>>>>>>>>>>>> Kibana >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Need Help!! >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Jon >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>>> Jon >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> With Regards >>>>>>>>>>> Farrukh Naveed Anjum >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> With Regards >>>>>>>>> Farrukh Naveed Anjum >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> With Regards >>>>>>> Farrukh Naveed Anjum >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> > > > -- > With Regards > Farrukh Naveed Anjum >
