yes, do configure it as per metron reference usecase On Tue, Jan 16, 2018 at 8:35 AM, Gaurav Bapat <gauravb3...@gmail.com> wrote:
> Hi Kyle, > > I saw that I can ping from my OS to VM and from VM to OS. Looks like this > is some Kafka or Zookeeper environment variables setup issue, do I need to > configure that in vagrant ssh? > > On 16 January 2018 at 08:59, Gaurav Bapat <gauravb3...@gmail.com> wrote: > >> Hey Kyle, >> >> I am running NiFi not on Ambari but on localhost:8089, I can ping from my >> OS terminal to node1 but can't ping from node1 to my OS terminal, I have >> attached few screenshots and the contents of /etc/hosts >> >> Thank You! >> >> On 15 January 2018 at 20:04, Kyle Richardson <kylerichards...@gmail.com> >> wrote: >> >>> It looks like your Nifi instance is running on your laptop/desktop (e.g. >>> the VM host). My guess would be that name resolution or networking is not >>> properly configured between the host and the guest preventing the data from >>> getting from Nifi to Kafka. What's the contents of /etc/hosts on the VM >>> host? Can you ping node1 from the VM host by name and by IP address? >>> >>> -Kyle >>> >>> On Mon, Jan 15, 2018 at 6:55 AM, Gaurav Bapat <gauravb3...@gmail.com> >>> wrote: >>> >>>> Failed while waiting for acks from Kafka is what I am getting in Kafka, >>>> am I missing some configuration with Kafka? >>>> >>>> On 15 January 2018 at 16:50, Gaurav Bapat <gauravb3...@gmail.com> >>>> wrote: >>>> >>>>> Hi Farrukh, >>>>> >>>>> I cant find any folder by my topic >>>>> >>>>> On 15 January 2018 at 16:33, Farrukh Naveed Anjum < >>>>> anjum.farr...@gmail.com> wrote: >>>>> >>>>>> Can you check /kafaka-logs on your VM box (It should have a folder >>>>>> named your topic). Can you check if it is there ? >>>>>> >>>>>> On Mon, Jan 15, 2018 at 3:49 PM, Gaurav Bapat <gauravb3...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> I am not getting data into my Kafka topic >>>>>>> >>>>>>> I have used i5 4 Core Processor with 16 GB RAM and I have allocated >>>>>>> 12 GB RAM to my vagrant VM. >>>>>>> >>>>>>> I dont understand how to configure Kafka broker because it is giving >>>>>>> me failed while waiting for acks to Kafka >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 15 January 2018 at 16:10, Farrukh Naveed Anjum < >>>>>>> anjum.farr...@gmail.com> wrote: >>>>>>> >>>>>>>> Can you tell me is your KAFKA Topic getting data ? What are you >>>>>>>> machine specifications ? >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Jan 15, 2018 at 2:56 PM, Gaurav Bapat < >>>>>>>> gauravb3...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Thanks Farrukh, >>>>>>>>> >>>>>>>>> I am not getting data in my kafka topic even after creating one, >>>>>>>>> the issue seems to be with broker config, how to configure Kafka and >>>>>>>>> Zookeeper port? >>>>>>>>> >>>>>>>>> On 15 January 2018 at 13:23, Farrukh Naveed Anjum < >>>>>>>>> anjum.farr...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I had similar issue it turned out to be the issue in STROM >>>>>>>>>> >>>>>>>>>> No worker is assigned to togolgoy all you need is to add >>>>>>>>>> additional port in >>>>>>>>>> >>>>>>>>>> Ambari -> Storm -> Configs -> supervisor.slot.ports by assigning >>>>>>>>>> an additional port to the list >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> https://community.hortonworks.com/questions/32499/no-workers >>>>>>>>>> -in-storm-for-squid-topology.html >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I had similar issue and finally got it fixed >>>>>>>>>> >>>>>>>>>> On Mon, Jan 15, 2018 at 8:45 AM, Gaurav Bapat < >>>>>>>>>> gauravb3...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Storm UI >>>>>>>>>>> >>>>>>>>>>> On 15 January 2018 at 08:59, Gaurav Bapat <gauravb3...@gmail.com >>>>>>>>>>> > wrote: >>>>>>>>>>> >>>>>>>>>>>> Hey Jon, >>>>>>>>>>>> >>>>>>>>>>>> I have Storm UI and the logs are coming from firewalls, >>>>>>>>>>>> servers, etc from other machines(HP ArcSight Logger). >>>>>>>>>>>> >>>>>>>>>>>> I have attached the NiFi screenshots, my logs are coming but >>>>>>>>>>>> there is some error with Kafka and I am having issues with >>>>>>>>>>>> configuring >>>>>>>>>>>> Kafka broker >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 12 January 2018 at 18:14, zeo...@gmail.com <zeo...@gmail.com >>>>>>>>>>>> > wrote: >>>>>>>>>>>> >>>>>>>>>>>>> In Ambari under storm you can find the UI under quick links at >>>>>>>>>>>>> the top. That said, the issue seems to be upstream of Metron, in >>>>>>>>>>>>> NiFi. >>>>>>>>>>>>> That is something I can't help with as much, but if you can share >>>>>>>>>>>>> the >>>>>>>>>>>>> listensyslog processor config that would be a start. Also, share >>>>>>>>>>>>> the >>>>>>>>>>>>> config of the thing that is sending syslog as well (are these >>>>>>>>>>>>> local syslog, >>>>>>>>>>>>> is that machine aggregating syslog from other machines, etc.). >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> >>>>>>>>>>>>> Jon >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Jan 12, 2018, 01:00 Gaurav Bapat < >>>>>>>>>>>>> gauravb3...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I have created a Kafka topic "cef" but my Listen Syslogs is >>>>>>>>>>>>>> not getting logs in the processor. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Also I checked using tcpdump -i and it is getting logs in my >>>>>>>>>>>>>> machine but ListenSyslogs is not getting the logs >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 12 January 2018 at 11:13, Gaurav Bapat < >>>>>>>>>>>>>> gauravb3...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> [root@metron incubator-metron]# >>>>>>>>>>>>>>> ./metron-deployment/scripts/platform-info.sh >>>>>>>>>>>>>>> Metron 0.4.3 >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> * master >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> commit c559ed7e1838ec71344eae3d9e37771db2641635 >>>>>>>>>>>>>>> Author: cstella <ceste...@gmail.com> >>>>>>>>>>>>>>> Date: Tue Jan 9 15:28:47 2018 -0500 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> METRON-1379: Add an OBJECT_GET stellar function closes >>>>>>>>>>>>>>> apache/incubator-metron#880 >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> metron-deployment/vagrant/full-dev-platform/Vagrantfile | >>>>>>>>>>>>>>> 2 +- >>>>>>>>>>>>>>> 1 file changed, 1 insertion(+), 1 deletion(-) >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> ansible 2.0.0.2 >>>>>>>>>>>>>>> config file = >>>>>>>>>>>>>>> configured module search path = Default w/o overrides >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Vagrant 1.9.6 >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Python 2.7.5 >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; >>>>>>>>>>>>>>> 2015-11-10T22:11:47+05:30) >>>>>>>>>>>>>>> Maven home: /opt/maven/current >>>>>>>>>>>>>>> Java version: 1.8.0_151, vendor: Oracle Corporation >>>>>>>>>>>>>>> Java home: /opt/jdk1.8.0_151/jre >>>>>>>>>>>>>>> Default locale: en_US, platform encoding: UTF-8 >>>>>>>>>>>>>>> OS name: "linux", version: "3.10.0-693.11.6.el7.x86_64", >>>>>>>>>>>>>>> arch: "amd64", family: "unix" >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Docker version 1.12.6, build ec8512b/1.12.6 >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> node >>>>>>>>>>>>>>> v8.9.3 >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> npm >>>>>>>>>>>>>>> 5.5.1 >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16) >>>>>>>>>>>>>>> Copyright (C) 2015 Free Software Foundation, Inc. >>>>>>>>>>>>>>> This is free software; see the source for copying >>>>>>>>>>>>>>> conditions. There is NO >>>>>>>>>>>>>>> warranty; not even for MERCHANTABILITY or FITNESS FOR A >>>>>>>>>>>>>>> PARTICULAR PURPOSE. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Compiler is C++11 compliant >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Linux metron.com 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan >>>>>>>>>>>>>>> 4 01:06:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Total System Memory = 15773.3 MB >>>>>>>>>>>>>>> Processor Model: Intel(R) Core(TM) i5-3450 CPU @ 3.10GHz >>>>>>>>>>>>>>> Processor Speed: 3320.875 MHz >>>>>>>>>>>>>>> Processor Speed: 3307.191 MHz >>>>>>>>>>>>>>> Processor Speed: 3376.699 MHz >>>>>>>>>>>>>>> Processor Speed: 3338.917 MHz >>>>>>>>>>>>>>> Total Physical Processors: 4 >>>>>>>>>>>>>>> Total cores: 16 >>>>>>>>>>>>>>> Disk information: >>>>>>>>>>>>>>> /dev/mapper/centos-root 200G 22G 179G 11% / >>>>>>>>>>>>>>> /dev/sda1 2.0G 224M 1.8G 11% /boot >>>>>>>>>>>>>>> /dev/sda2 1022M 12K 1022M 1% /boot/efi >>>>>>>>>>>>>>> /dev/mapper/centos-home 247G 10G 237G 5% /home >>>>>>>>>>>>>>> This CPU appears to support virtualization >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 12 January 2018 at 09:25, Gaurav Bapat < >>>>>>>>>>>>>>> gauravb3...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hey Jon, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Appreciate your timely reply. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I gone through your answer but still I can't figure out how >>>>>>>>>>>>>>>> do I do parsing/indexing in Storm UI as I cant find any option >>>>>>>>>>>>>>>> for the same. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Is there any other UI to do parsing/indexing? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 11 January 2018 at 21:22, zeo...@gmail.com < >>>>>>>>>>>>>>>> zeo...@gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> So, you created a new cef topic, and set up the >>>>>>>>>>>>>>>>> appropriate parser config for it (if not, this >>>>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source> >>>>>>>>>>>>>>>>> may be helpful)? If so: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Here are some basic troubleshooting steps: >>>>>>>>>>>>>>>>> 1. Validate that the logs are getting onto the >>>>>>>>>>>>>>>>> kafka topic that you are sending to. If they aren't there, >>>>>>>>>>>>>>>>> the problem is >>>>>>>>>>>>>>>>> upstream from Metron. >>>>>>>>>>>>>>>>> 2. If they are getting onto the kafka topic they are >>>>>>>>>>>>>>>>> being directly sent to, check the indexing kafka topic for an >>>>>>>>>>>>>>>>> enriched >>>>>>>>>>>>>>>>> version of those same logs. >>>>>>>>>>>>>>>>> 3. Do a binary search of the various components involved >>>>>>>>>>>>>>>>> with ingest. >>>>>>>>>>>>>>>>> a. If the logs are *not* on the indexing kafka topic, >>>>>>>>>>>>>>>>> check the enrichments topic for those logs. >>>>>>>>>>>>>>>>> b. If the logs are *not* on the enrichments topic, >>>>>>>>>>>>>>>>> check the parser storm topology. >>>>>>>>>>>>>>>>> c. If the logs are on the enrichments topic, but *not* >>>>>>>>>>>>>>>>> indexing, check the enrichments storm topology. >>>>>>>>>>>>>>>>> d. If the logs are on the indexing but *not* Kibana, >>>>>>>>>>>>>>>>> check the indexing storm topic. >>>>>>>>>>>>>>>>> e. If the logs are in on the indexing topic and >>>>>>>>>>>>>>>>> indexing storm topic is in good shape, check elasticsearch >>>>>>>>>>>>>>>>> directly. >>>>>>>>>>>>>>>>> 4. You should have identified where the issue is at this >>>>>>>>>>>>>>>>> point. Report back here with what you observed, any relevant >>>>>>>>>>>>>>>>> error >>>>>>>>>>>>>>>>> messages, etc. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Side note: We should document a decision tree for >>>>>>>>>>>>>>>>> troubleshooting data ingest. It is fairly straightforward >>>>>>>>>>>>>>>>> and makes me >>>>>>>>>>>>>>>>> wonder if we already have this somewhere and I'm not aware of >>>>>>>>>>>>>>>>> it? It would >>>>>>>>>>>>>>>>> also be a good place to put pointers to some common errors. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Jon >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Thu, Jan 11, 2018 at 1:44 AM Gaurav Bapat < >>>>>>>>>>>>>>>>> gauravb3...@gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hello everyone, I have deployed Metron on a single node >>>>>>>>>>>>>>>>>> machine and I would like to know how do I get Syslogs from >>>>>>>>>>>>>>>>>> NiFi into Kibana >>>>>>>>>>>>>>>>>> dashboard? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I have created a Kafka topic by the name "cef" and I can >>>>>>>>>>>>>>>>>> see that the topic exists in >>>>>>>>>>>>>>>>>> Metron Configuration but I am unable to connect it with >>>>>>>>>>>>>>>>>> Kibana >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Need Help!! >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Jon >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>> >>>>>>>>>>>>> Jon >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> With Regards >>>>>>>>>> Farrukh Naveed Anjum >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> With Regards >>>>>>>> Farrukh Naveed Anjum >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> With Regards >>>>>> Farrukh Naveed Anjum >>>>>> >>>>> >>>>> >>>> >>> >> > -- With Regards Farrukh Naveed Anjum
