Getting Syslogs to Metron

[Gaurav Bapat]

Hello everyone, I have deployed Metron on a single node machine and I would
like to know how do I get Syslogs from NiFi into Kibana dashboard?

I have created a Kafka topic by the name "cef" and I can see that the topic
exists in
Metron Configuration but I am unable to connect it with Kibana

Need Help!!

Re: Intro & Question

[Matt Foley]

BTW, any community member can open a jira, but to assign it to yourself, as the 
instructions say, requires being added to the list of contributors.  Just 
forward this thread to dev@ with a request to Casey Stella, our PMC Chair, to 
be added as a contributor.

From: Matt Foley 
Reply-To: "user@metron.apache.org" 
Date: Wednesday, January 10, 2018 at 10:41 AM
To: "user@metron.apache.org" , Ahmed Shah 

Subject: Re: Intro & Question

Ahmed, please see

https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines

Feel free to ask the dev community questions.  Suggest this discussion (as 
regards the contribution) be moved to 
d...@metron.apache.org instead of user@.

Cheers,
--Matt

From: Otto Fowler 
Reply-To: "user@metron.apache.org" 
Date: Wednesday, January 10, 2018 at 7:28 AM
To: Ahmed Shah , "user@metron.apache.org" 

Subject: Re: Intro & Question

So, what would work would be:

1.  Create a jira like “Ability to deploy metron full dev to aws with vagrant”
With a description of the use case, and how the vagrant file will fill it.
2. create a pr, with the new file in metron-deployment/vagrant/aws
3. update the readme

I think



On January 10, 2018 at 10:16:03, Ahmed Shah 
(ahmeds...@cmail.carleton.ca) wrote:

Would be glad to.

Where in github should I put it?


-Ahmed
___
Ahmed Shah (PMP, M. Eng.)
Cybersecurity Analyst & Developer
GCR - Cybersecurity Operations Center
Carleton University - cugcr.com


From: Otto Fowler >
Sent: January 9, 2018 11:51 AM
To: Ahmed Shah; user@metron.apache.org
Subject: Re: Intro & Question

Any interest in submitting this?



On January 9, 2018 at 10:42:08, Ahmed Shah 
(ahmeds...@cmail.carleton.ca) wrote:

Hello Srikanth,



Our team adapted the Metron 0.4.1 Single Node VM install (Original Code Here: 
https://github.com/apache/metron/tree/master/metron-deployment/vagrant/full-dev-platform)
  to deploy a single node to AWS.

Our Vagrent file is here:

https://github.com/LTW-GCR-CSOC/csoc-installation-scripts/blob/master/amazon-deploy/Metron/Vagrantfile

You can define your AWS Elastic IP,  Subnet ID, VPC, and Security Group ID 
before running the file.



Hope it helps.



-Ahmed
___
Ahmed Shah (PMP, M. Eng.)
Cybersecurity Analyst & Developer
GCR - Cybersecurity Operations Center
Carleton University - cugcr.com


From: Srikanth Nagarajan 
>
Sent: January 9, 2018 2:39 AM
To: user@metron.apache.org
Subject: Intro & Question


Hi

My name is Srikanth and work for a Cyber Security firm.   We are building 
Metron to test in our lab environment using AWS.

1. Is there a single VM version for Cloud install available ?   If yes, please 
share procedure.

2. During the Amazon-Ec2 install for the multi node version provided in the 
metron git-hub docs

https://github.com/apache/metron/tree/master/metron-deployment/amazon-ec2  get 
an error

[WARNING]:  * Failed to parse 
/Users/sri/metron/metron-deployment/amazon-ec2/ec2.py with script plugin: 
Inventory script (/Users/sri/metron/metron-deployment/amazon-ec2/ec2.py) had an 
execution

error: ERROR: "Forbidden", while: getting ElastiCache clusters

Any assistance would be appreciated.

Thanks

Srikanth

__

Srikanth Nagarajan
Principal

Gandiva Networks Inc

732.690.1884 Mobile

s...@gandivanetworks.com

www.gandivanetworks.com

Please consider the environment before printing this. NOTICE: The information 
contained in this e-mail message is intended for addressee(s) only. If you have 
received this message in error please notify the sender.

Re: Intro & Question

[Matt Foley]

Ahmed, please see

https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines

Feel free to ask the dev community questions.  Suggest this discussion (as 
regards the contribution) be moved to 
d...@metron.apache.org instead of user@.

Cheers,
--Matt

From: Otto Fowler 
Reply-To: "user@metron.apache.org" 
Date: Wednesday, January 10, 2018 at 7:28 AM
To: Ahmed Shah , "user@metron.apache.org" 

Subject: Re: Intro & Question

So, what would work would be:

1.  Create a jira like “Ability to deploy metron full dev to aws with vagrant”
With a description of the use case, and how the vagrant file will fill it.
2. create a pr, with the new file in metron-deployment/vagrant/aws
3. update the readme

I think



On January 10, 2018 at 10:16:03, Ahmed Shah 
(ahmeds...@cmail.carleton.ca) wrote:

Would be glad to.

Where in github should I put it?


-Ahmed
___
Ahmed Shah (PMP, M. Eng.)
Cybersecurity Analyst & Developer
GCR - Cybersecurity Operations Center
Carleton University - cugcr.com


From: Otto Fowler >
Sent: January 9, 2018 11:51 AM
To: Ahmed Shah; user@metron.apache.org
Subject: Re: Intro & Question

Any interest in submitting this?



On January 9, 2018 at 10:42:08, Ahmed Shah 
(ahmeds...@cmail.carleton.ca) wrote:

Hello Srikanth,



Our team adapted the Metron 0.4.1 Single Node VM install (Original Code Here: 
https://github.com/apache/metron/tree/master/metron-deployment/vagrant/full-dev-platform)
  to deploy a single node to AWS.

Our Vagrent file is here:

https://github.com/LTW-GCR-CSOC/csoc-installation-scripts/blob/master/amazon-deploy/Metron/Vagrantfile

You can define your AWS Elastic IP,  Subnet ID, VPC, and Security Group ID 
before running the file.



Hope it helps.



-Ahmed
___
Ahmed Shah (PMP, M. Eng.)
Cybersecurity Analyst & Developer
GCR - Cybersecurity Operations Center
Carleton University - cugcr.com


From: Srikanth Nagarajan 
>
Sent: January 9, 2018 2:39 AM
To: user@metron.apache.org
Subject: Intro & Question


Hi

My name is Srikanth and work for a Cyber Security firm.   We are building 
Metron to test in our lab environment using AWS.

1. Is there a single VM version for Cloud install available ?   If yes, please 
share procedure.

2. During the Amazon-Ec2 install for the multi node version provided in the 
metron git-hub docs

https://github.com/apache/metron/tree/master/metron-deployment/amazon-ec2  get 
an error

[WARNING]:  * Failed to parse 
/Users/sri/metron/metron-deployment/amazon-ec2/ec2.py with script plugin: 
Inventory script (/Users/sri/metron/metron-deployment/amazon-ec2/ec2.py) had an 
execution

error: ERROR: "Forbidden", while: getting ElastiCache clusters

Any assistance would be appreciated.

Thanks

Srikanth

__

Srikanth Nagarajan
Principal

Gandiva Networks Inc

732.690.1884 Mobile

s...@gandivanetworks.com

www.gandivanetworks.com

Please consider the environment before printing this. NOTICE: The information 
contained in this e-mail message is intended for addressee(s) only. If you have 
received this message in error please notify the sender.

Re: Metron Rest Kerberos -- Kafka topic ACL

[Simon Elliston Ball]

The ansible roles and playbooks included with Metron install Ambari to handle 
the setup of the Metron and the Hadoop, Kafka etc. components, so yes. 

> On 10 Jan 2018, at 03:18, varsha mordi  wrote:
> 
> Can Ambari UI work with Ansible?
> 
> On Wed, Jan 10, 2018 at 3:46 PM, Mohan Venkateshaiah 
> > 
> wrote:
> Srikanth,
> 
>  
> 
> There is no way you can list all topics to particular user, there is PR for 
> adding REST endpoints to provide required ACL to topic.
> 
>  
> 
> Thanks
> 
> Mohan DV
> 
>  
> 
> From: prakash r >
> Reply-To: "user@metron.apache.org " 
> >
> Date: Wednesday, January 10, 2018 at 7:50 AM
> To: "user@metron.apache.org " 
> >
> Subject: Metron Rest Kerberos -- Kafka topic ACL
> 
>  
> 
> Hello,
> 
>  
> 
> We have kerberosed Hadoop Cluster.
> 
>  
> 
> Metron is trying to access all the Kafka topics (ir-respective of Kafka 
> topics which needed for Metron)
> 
>  
> 
> Since it does not have access to all topics, in UI Kafka related infos are 
> not displayed.
> 
>  
> 
> For Ex :
> 
>  
> 
> Kafka has some other topics like (checking123) Metron need authorization for 
> those topic as well.
> 
>  
> 
> 2018-01-10T11:17:39.576 DEBUG 
> [org.springframework.web.servlet.mvc.method.annotation.HttpEntityMethodProcessor]
>  - Written [{timestamp=Wed Jan 10 11:17:39 AEDT 2018, status=500, 
> error=Internal Server Error, 
> exception=org.apache.kafka.common.errors.TopicAuthorizationException, 
> message=Not authorized to access topics: [checking123], 
> path=/api/v1/kafka/topic/snort}] as "application/json" using 
> [org.springframework.http.converter.json.MappingJackson2HttpMessageConverter@ab327c]
> 
>  
> 
>  
> 
> Can Metron Rest restrict access only to those topics which is needed for the 
> same, thanks
> 
> 
> Regards,
> 
> Prakash R
> 
> 
> 
> 
> -- 
> Thanks & Regards,
> Varsha Mordi
> Prodevans Technologies LLP.
> M: +91 9637109734  | L: +91 80 64533365 | www.prodevans.com 
> 
> 

Re: Intro & Question

[Otto Fowler]

So, what would work would be:

1.  Create a jira like “Ability to deploy metron full dev to aws with
vagrant”
With a description of the use case, and how the vagrant file will fill it.
2. create a pr, with the new file in metron-deployment/vagrant/aws
3. update the readme

I think


On January 10, 2018 at 10:16:03, Ahmed Shah (ahmeds...@cmail.carleton.ca)
wrote:

Would be glad to.

Where in github should I put it?


-Ahmed
___
Ahmed Shah (PMP, M. Eng.)
Cybersecurity Analyst & Developer
GCR - Cybersecurity Operations Center
Carleton University - cugcr.com 


--
*From:* Otto Fowler 
*Sent:* January 9, 2018 11:51 AM
*To:* Ahmed Shah; user@metron.apache.org
*Subject:* Re: Intro & Question

Any interest in submitting this?


On January 9, 2018 at 10:42:08, Ahmed Shah (ahmeds...@cmail.carleton.ca)
wrote:

Hello Srikanth,


Our team adapted the Metron 0.4.1 Single Node VM install (Original Code
Here:
https://github.com/apache/metron/tree/master/metron-deployment/vagrant/full-dev-platform)
 to
deploy a single node to AWS.


Our Vagrent file is here:

https://github.com/LTW-GCR-CSOC/csoc-installation-scripts/blob/master/amazon-deploy/Metron/Vagrantfile

You can define your AWS Elastic IP,  Subnet ID, VPC, and Security Group ID
before running the file.


Hope it helps.


-Ahmed
___
Ahmed Shah (PMP, M. Eng.)
Cybersecurity Analyst & Developer
GCR - Cybersecurity Operations Center
Carleton University - cugcr.com 


--
*From:* Srikanth Nagarajan 
*Sent:* January 9, 2018 2:39 AM
*To:* user@metron.apache.org
*Subject:* Intro & Question


Hi

My name is Srikanth and work for a Cyber Security firm.   We are building
Metron to test in our lab environment using AWS.

1. Is there a single VM version for Cloud install available ?   If yes,
please share procedure.

2. During the Amazon-Ec2 install for the multi node version provided in the
metron git-hub docs

https://github.com/apache/metron/tree/master/metron-deployment/amazon-ec2
get an error

[WARNING]:  * Failed to parse
/Users/sri/metron/metron-deployment/amazon-ec2/ec2.py with script plugin:
Inventory script (/Users/sri/metron/metron-deployment/amazon-ec2/ec2.py)
had an execution

error: ERROR: "Forbidden", while: getting ElastiCache clusters

Any assistance would be appreciated.

Thanks

Srikanth

__

*Srikanth Nagarajan*
*Principal*

* Gandiva Networks Inc*

*732.690.1884* Mobile

s...@gandivanetworks.com

www.gandivanetworks.com

Please consider the environment before printing this. NOTICE: The
information contained in this e-mail message is intended for addressee(s)
only. If you have received this message in error please notify the sender.

Re: Intro & Question

[Ahmed Shah]

Would be glad to.

Where in github should I put it?


-Ahmed
___
Ahmed Shah (PMP, M. Eng.)
Cybersecurity Analyst & Developer
GCR - Cybersecurity Operations Center
Carleton University - cugcr.com



From: Otto Fowler 
Sent: January 9, 2018 11:51 AM
To: Ahmed Shah; user@metron.apache.org
Subject: Re: Intro & Question

Any interest in submitting this?



On January 9, 2018 at 10:42:08, Ahmed Shah 
(ahmeds...@cmail.carleton.ca) wrote:

Hello Srikanth,


Our team adapted the Metron 0.4.1 Single Node VM install (Original Code Here: 
https://github.com/apache/metron/tree/master/metron-deployment/vagrant/full-dev-platform)
  to deploy a single node to AWS.

Our Vagrent file is here:

https://github.com/LTW-GCR-CSOC/csoc-installation-scripts/blob/master/amazon-deploy/Metron/Vagrantfile


You can define your AWS Elastic IP,  Subnet ID, VPC, and Security Group ID 
before running the file.


Hope it helps.



-Ahmed
___
Ahmed Shah (PMP, M. Eng.)
Cybersecurity Analyst & Developer
GCR - Cybersecurity Operations Center
Carleton University - cugcr.com



From: Srikanth Nagarajan 
>
Sent: January 9, 2018 2:39 AM
To: user@metron.apache.org
Subject: Intro & Question


Hi

My name is Srikanth and work for a Cyber Security firm.   We are building 
Metron to test in our lab environment using AWS.

1. Is there a single VM version for Cloud install available ?   If yes, please 
share procedure.

2. During the Amazon-Ec2 install for the multi node version provided in the 
metron git-hub docs

https://github.com/apache/metron/tree/master/metron-deployment/amazon-ec2  get 
an error

[WARNING]:  * Failed to parse 
/Users/sri/metron/metron-deployment/amazon-ec2/ec2.py with script plugin: 
Inventory script (/Users/sri/metron/metron-deployment/amazon-ec2/ec2.py) had an 
execution

error: ERROR: "Forbidden", while: getting ElastiCache clusters

Any assistance would be appreciated.

Thanks

Srikanth

__

Srikanth Nagarajan
Principal

Gandiva Networks Inc

732.690.1884 Mobile

s...@gandivanetworks.com

www.gandivanetworks.com

Please consider the environment before printing this. NOTICE: The information 
contained in this e-mail message is intended for addressee(s) only. If you have 
received this message in error please notify the sender.

Re: ElasticSearch Indexing not working (Strom Error)

[Farrukh Naveed Anjum]

Please see the image...

java.lang.IllegalStateException: [Metron] Unable to update MaxMind database
at 
org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase.update(GeoLiteDatabase.java:150)
at org.apache.metron.enrichm

Its unable to do find MaxMind database, I installed it but still now
elasticsearch index.

Any Idea where can I debug why indcies are not being created. I can confirm
Elasticsearch is up and running.

On Wed, Jan 10, 2018 at 7:45 PM, Otto Fowler 
wrote:

> Please reply to the list
>
>
> On January 10, 2018 at 09:37:45, Farrukh Naveed Anjum (
> anjum.farr...@gmail.com) wrote:
>
> Please see the image...
>
> java.lang.IllegalStateException: [Metron] Unable to update MaxMind
> database at 
> org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase.update(GeoLiteDatabase.java:150)
> at org.apache.metron.enrichm
>
> Its unable to do find MaxMind database, I installed it but still now
> elasticsearch index.
>
>
>
> On Wed, Jan 10, 2018 at 7:04 PM, Otto Fowler 
> wrote:
>
>> Can we get the complete exception?  There may be a ‘caused by’ listing
>> that could help.
>>
>>
>>
>> On January 10, 2018 at 08:53:37, Farrukh Naveed Anjum (
>> anjum.farr...@gmail.com) wrote:
>>
>> Please some one respond
>>
>> On Mon, Jan 8, 2018 at 1:10 PM, Farrukh Naveed Anjum <
>> anjum.farr...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am unable to see any ElasticSearch Index in kibana or in elasticsearch
>>> plugin
>>>
>>> http://node1:9200/_plugin/head/
>>>
>>> After looking into strom, it seems like GeoLiteDatabase Exception in
>>> Strom bolts
>>>
>>> How can I fix it.
>>>
>>>
>>> java.lang.IllegalStateException: [Metron] Unable to update MaxMind
>>> database at 
>>> org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase.update(GeoLiteDatabase.java:150)
>>> at org.apache.metron.enrichm
>>>
>>> --
>>> With Regards
>>> Farrukh Naveed Anjum
>>>
>>
>>
>>
>> --
>> With Regards
>> Farrukh Naveed Anjum
>>
>>
>
>
> --
> With Regards
> Farrukh Naveed Anjum
>
>


-- 
With Regards
Farrukh Naveed Anjum

Re: ElasticSearch Indexing not working (Strom Error)

[Otto Fowler]

Can we get the complete exception?  There may be a ‘caused by’ listing that
could help.



On January 10, 2018 at 08:53:37, Farrukh Naveed Anjum (
anjum.farr...@gmail.com) wrote:

Please some one respond

On Mon, Jan 8, 2018 at 1:10 PM, Farrukh Naveed Anjum <
anjum.farr...@gmail.com> wrote:

> Hi,
>
> I am unable to see any ElasticSearch Index in kibana or in elasticsearch
> plugin
>
> http://node1:9200/_plugin/head/
>
> After looking into strom, it seems like GeoLiteDatabase Exception in
> Strom bolts
>
> How can I fix it.
>
>
> java.lang.IllegalStateException: [Metron] Unable to update MaxMind
> database at org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase.
> update(GeoLiteDatabase.java:150) at org.apache.metron.enrichm
>
> --
> With Regards
> Farrukh Naveed Anjum
>



--
With Regards
Farrukh Naveed Anjum

Re: ElasticSearch Indexing not working (Strom Error)

[Farrukh Naveed Anjum]

Please some one respond

On Mon, Jan 8, 2018 at 1:10 PM, Farrukh Naveed Anjum <
anjum.farr...@gmail.com> wrote:

> Hi,
>
> I am unable to see any ElasticSearch Index in kibana or in elasticsearch
> plugin
>
> http://node1:9200/_plugin/head/
>
> After looking into strom, it seems like GeoLiteDatabase Exception in
> Strom bolts
>
> How can I fix it.
>
>
> java.lang.IllegalStateException: [Metron] Unable to update MaxMind
> database at org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase.
> update(GeoLiteDatabase.java:150) at org.apache.metron.enrichm
>
> --
> With Regards
> Farrukh Naveed Anjum
>



-- 
With Regards
Farrukh Naveed Anjum

Re: Metron Rest Kerberos -- Kafka topic ACL

[varsha mordi]

Can Ambari UI work with Ansible?

On Wed, Jan 10, 2018 at 3:46 PM, Mohan Venkateshaiah <
mvenkatesha...@hortonworks.com> wrote:

> Srikanth,
>
>
>
> There is no way you can list all topics to particular user, there is PR
> for adding REST endpoints to provide required ACL to topic.
>
>
>
> Thanks
>
> Mohan DV
>
>
>
> *From: *prakash r 
> *Reply-To: *"user@metron.apache.org" 
> *Date: *Wednesday, January 10, 2018 at 7:50 AM
> *To: *"user@metron.apache.org" 
> *Subject: *Metron Rest Kerberos -- Kafka topic ACL
>
>
>
> Hello,
>
>
>
> We have kerberosed Hadoop Cluster.
>
>
>
> Metron is trying to access all the Kafka topics (ir-respective of Kafka
> topics which needed for Metron)
>
>
>
> Since it does not have access to all topics, in UI Kafka related infos are
> not displayed.
>
>
>
> For Ex :
>
>
>
> Kafka has some other topics like (checking123) Metron need authorization
> for those topic as well.
>
>
>
> 2018-01-10T11:17:39.576 DEBUG [org.springframework.web.
> servlet.mvc.method.annotation.HttpEntityMethodProcessor] - Written
> [{timestamp=Wed Jan 10 11:17:39 AEDT 2018, status=500, error=Internal
> Server Error, 
> exception=org.apache.kafka.common.errors.TopicAuthorizationException,
> message=Not authorized to access topics: [checking123],
> path=/api/v1/kafka/topic/snort}] as "application/json" using
> [org.springframework.http.converter.json.MappingJackson2HttpMessageConv
> erter@ab327c]
>
>
>
>
>
> Can Metron Rest restrict access only to those topics which is needed for
> the same, thanks
>
>
> Regards,
>
> Prakash R
>



-- 
Thanks & Regards,

Varsha Mordi

Prodevans Technologies LLP.

M: +91 9637109734  *| *L: +91 80 64533365 *|* www.prodevans.com

Re: Metron Rest Kerberos -- Kafka topic ACL

[Mohan Venkateshaiah]

Srikanth,

There is no way you can list all topics to particular user, there is PR for 
adding REST endpoints to provide required ACL to topic.

Thanks
Mohan DV

From: prakash r 
Reply-To: "user@metron.apache.org" 
Date: Wednesday, January 10, 2018 at 7:50 AM
To: "user@metron.apache.org" 
Subject: Metron Rest Kerberos -- Kafka topic ACL

Hello,

We have kerberosed Hadoop Cluster.

Metron is trying to access all the Kafka topics (ir-respective of Kafka topics 
which needed for Metron)

Since it does not have access to all topics, in UI Kafka related infos are not 
displayed.

For Ex :

Kafka has some other topics like (checking123) Metron need authorization for 
those topic as well.

2018-01-10T11:17:39.576 DEBUG 
[org.springframework.web.servlet.mvc.method.annotation.HttpEntityMethodProcessor]
 - Written [{timestamp=Wed Jan 10 11:17:39 AEDT 2018, status=500, 
error=Internal Server Error, 
exception=org.apache.kafka.common.errors.TopicAuthorizationException, 
message=Not authorized to access topics: [checking123], 
path=/api/v1/kafka/topic/snort}] as "application/json" using 
[org.springframework.http.converter.json.MappingJackson2HttpMessageConverter@ab327c]


Can Metron Rest restrict access only to those topics which is needed for the 
same, thanks

Regards,
Prakash R