Re: Getting Syslogs to Metron

[Farrukh Naveed Anjum]

Hi Guys,

Its seems like we are able to make NIFI connection and data indeed is going
through KAFKA Topic yet using CEF Parser (SysLogs) we are unable to create
the elastic search index.




On Mon, Jan 22, 2018 at 12:32 PM, Farrukh Naveed Anjum <
anjum.farr...@gmail.com> wrote:

> Hi, Gaurav,
>
> Did you solved it ? I am also following same usecase for SysLog using UDP
> (Rsyslogs)
>
> It seems like data is coming to KAFKA Topic. As you can see its showing up.
>
> But Elasticsearch index is not created.
>
>
>
> On Tue, Jan 16, 2018 at 12:37 PM, Gaurav Bapat 
> wrote:
>
>> But I cant find how to configure it
>>
>> On 16 January 2018 at 11:38, Farrukh Naveed Anjum <
>> anjum.farr...@gmail.com> wrote:
>>
>>> yes, do configure it as per metron reference usecase
>>>
>>> On Tue, Jan 16, 2018 at 8:35 AM, Gaurav Bapat 
>>> wrote:
>>>
 Hi Kyle,

 I saw that I can ping from my OS to VM and from VM to OS. Looks like
 this is some Kafka or Zookeeper environment variables setup issue, do I
 need to configure that in vagrant ssh?

 On 16 January 2018 at 08:59, Gaurav Bapat 
 wrote:

> Hey Kyle,
>
> I am running NiFi not on Ambari but on localhost:8089, I can ping from
> my OS terminal to node1 but can't ping from node1 to my OS terminal, I 
> have
> attached few screenshots and the contents of /etc/hosts
>
> Thank You!
>
> On 15 January 2018 at 20:04, Kyle Richardson <
> kylerichards...@gmail.com> wrote:
>
>> It looks like your Nifi instance is running on your laptop/desktop
>> (e.g. the VM host). My guess would be that name resolution or networking 
>> is
>> not properly configured between the host and the guest preventing the 
>> data
>> from getting from Nifi to Kafka. What's the contents of /etc/hosts on the
>> VM host? Can you ping node1 from the VM host by name and by IP address?
>>
>> -Kyle
>>
>> On Mon, Jan 15, 2018 at 6:55 AM, Gaurav Bapat 
>> wrote:
>>
>>> Failed while waiting for acks from Kafka is what I am getting in
>>> Kafka, am I missing some configuration with Kafka?
>>>
>>> On 15 January 2018 at 16:50, Gaurav Bapat 
>>> wrote:
>>>
 Hi Farrukh,

 I cant find any folder by my topic

 On 15 January 2018 at 16:33, Farrukh Naveed Anjum <
 anjum.farr...@gmail.com> wrote:

> Can you check /kafaka-logs on your VM box (It should have a folder
> named your topic). Can you check if it is there ?
>
> On Mon, Jan 15, 2018 at 3:49 PM, Gaurav Bapat <
> gauravb3...@gmail.com> wrote:
>
>> I am not getting data into my Kafka topic
>>
>> I have used i5 4 Core Processor with 16 GB RAM and I have
>> allocated 12 GB RAM to my vagrant VM.
>>
>> I dont understand how to configure Kafka broker because it is
>> giving me failed while waiting for acks to Kafka
>>
>>
>>
>> On 15 January 2018 at 16:10, Farrukh Naveed Anjum <
>> anjum.farr...@gmail.com> wrote:
>>
>>> Can you tell me is your KAFKA Topic getting data ? What are you
>>> machine specifications ?
>>>
>>>
>>> On Mon, Jan 15, 2018 at 2:56 PM, Gaurav Bapat <
>>> gauravb3...@gmail.com> wrote:
>>>
 Thanks Farrukh,

 I am not getting data in my kafka topic even after creating
 one, the issue seems to be with broker config, how to configure 
 Kafka and
 Zookeeper port?

 On 15 January 2018 at 13:23, Farrukh Naveed Anjum <
 anjum.farr...@gmail.com> wrote:

> Hi,
>
> I had similar issue it turned out to be the issue in STROM
>
> No worker is assigned to togolgoy all you need is to add
> additional port in
>
>  Ambari -> Storm -> Configs -> supervisor.slot.ports by
> assigning an additional port to the list
>
>
> https://community.hortonworks.com/questions/32499/no-workers
> -in-storm-for-squid-topology.html
>
>
> I had similar issue and finally got it fixed
>
> On Mon, Jan 15, 2018 at 8:45 AM, Gaurav Bapat <
> gauravb3...@gmail.com> wrote:
>
>> Storm UI
>>
>> On 15 January 2018 at 08:59, Gaurav Bapat <
>> gauravb3...@gmail.com> wrote:
>>
>>> Hey Jon,
>>>
>>> I have Storm UI and the logs are coming from firewalls,
>>> servers, etc from other machines(HP ArcSight Logger).
>>>

Re: Getting Syslogs to Metron

[Farrukh Naveed Anjum]

Hi, Gaurav,

Did you solved it ? I am also following same usecase for SysLog using UDP
(Rsyslogs)

It seems like data is coming to KAFKA Topic. As you can see its showing up.

But Elasticsearch index is not created.



On Tue, Jan 16, 2018 at 12:37 PM, Gaurav Bapat 
wrote:

> But I cant find how to configure it
>
> On 16 January 2018 at 11:38, Farrukh Naveed Anjum  > wrote:
>
>> yes, do configure it as per metron reference usecase
>>
>> On Tue, Jan 16, 2018 at 8:35 AM, Gaurav Bapat 
>> wrote:
>>
>>> Hi Kyle,
>>>
>>> I saw that I can ping from my OS to VM and from VM to OS. Looks like
>>> this is some Kafka or Zookeeper environment variables setup issue, do I
>>> need to configure that in vagrant ssh?
>>>
>>> On 16 January 2018 at 08:59, Gaurav Bapat  wrote:
>>>
 Hey Kyle,

 I am running NiFi not on Ambari but on localhost:8089, I can ping from
 my OS terminal to node1 but can't ping from node1 to my OS terminal, I have
 attached few screenshots and the contents of /etc/hosts

 Thank You!

 On 15 January 2018 at 20:04, Kyle Richardson  wrote:

> It looks like your Nifi instance is running on your laptop/desktop
> (e.g. the VM host). My guess would be that name resolution or networking 
> is
> not properly configured between the host and the guest preventing the data
> from getting from Nifi to Kafka. What's the contents of /etc/hosts on the
> VM host? Can you ping node1 from the VM host by name and by IP address?
>
> -Kyle
>
> On Mon, Jan 15, 2018 at 6:55 AM, Gaurav Bapat 
> wrote:
>
>> Failed while waiting for acks from Kafka is what I am getting in
>> Kafka, am I missing some configuration with Kafka?
>>
>> On 15 January 2018 at 16:50, Gaurav Bapat 
>> wrote:
>>
>>> Hi Farrukh,
>>>
>>> I cant find any folder by my topic
>>>
>>> On 15 January 2018 at 16:33, Farrukh Naveed Anjum <
>>> anjum.farr...@gmail.com> wrote:
>>>
 Can you check /kafaka-logs on your VM box (It should have a folder
 named your topic). Can you check if it is there ?

 On Mon, Jan 15, 2018 at 3:49 PM, Gaurav Bapat <
 gauravb3...@gmail.com> wrote:

> I am not getting data into my Kafka topic
>
> I have used i5 4 Core Processor with 16 GB RAM and I have
> allocated 12 GB RAM to my vagrant VM.
>
> I dont understand how to configure Kafka broker because it is
> giving me failed while waiting for acks to Kafka
>
>
>
> On 15 January 2018 at 16:10, Farrukh Naveed Anjum <
> anjum.farr...@gmail.com> wrote:
>
>> Can you tell me is your KAFKA Topic getting data ? What are you
>> machine specifications ?
>>
>>
>> On Mon, Jan 15, 2018 at 2:56 PM, Gaurav Bapat <
>> gauravb3...@gmail.com> wrote:
>>
>>> Thanks Farrukh,
>>>
>>> I am not getting data in my kafka topic even after creating one,
>>> the issue seems to be with broker config, how to configure Kafka and
>>> Zookeeper port?
>>>
>>> On 15 January 2018 at 13:23, Farrukh Naveed Anjum <
>>> anjum.farr...@gmail.com> wrote:
>>>
 Hi,

 I had similar issue it turned out to be the issue in STROM

 No worker is assigned to togolgoy all you need is to add
 additional port in

  Ambari -> Storm -> Configs -> supervisor.slot.ports by
 assigning an additional port to the list


 https://community.hortonworks.com/questions/32499/no-workers
 -in-storm-for-squid-topology.html


 I had similar issue and finally got it fixed

 On Mon, Jan 15, 2018 at 8:45 AM, Gaurav Bapat <
 gauravb3...@gmail.com> wrote:

> Storm UI
>
> On 15 January 2018 at 08:59, Gaurav Bapat <
> gauravb3...@gmail.com> wrote:
>
>> Hey Jon,
>>
>> I have Storm UI and the logs are coming from firewalls,
>> servers, etc from other machines(HP ArcSight Logger).
>>
>> I have attached the NiFi screenshots, my logs are coming but
>> there is some error with Kafka and I am having issues with 
>> configuring
>> Kafka broker
>>
>>
>>
>> On 12 January 2018 at 18:14, zeo...@gmail.com <
>> zeo...@gmail.com> wrote:
>>
>>> In Ambari under storm you can find the UI under quick links