Areas Stuck

[Gaurav Bapat]

These are the following ares where we are currently stuck:-

1.> Getting data from Logger to NiFi, using Kafka topic.
2.> Configuring Kafka & Zookeeper
3.> Indexing - spout & bolt
4.> Parsing syslog data
5.> Visualizing logs in Kibana
6.> Connecting Spark and Metron

Re: Deployment help needed.

[Gaurav Bapat]

1.>In Full development folder, do vagrant destroy -f
2.>In metron root directory do mvn clean compile -DskipTests
3.>start docker
4.>vagrant up

On 24 January 2018 at 08:13, Sujay Jaladi  wrote:

> Hello,
>
> Everytime I attempt to deploy apache metron on AWS, I get the following
> error and all the servers are up and running expect Metron or its
> components are not installed. Please help.
>
> fatal: [ec2-52-10-94-22.us-west-2.compute.amazonaws.com -> localhost]:
> FAILED! => {"changed": true, "cmd": "cd /Users/sujay/Downloads/apache-
> metron-0.4.2-rc2/metron-deployment/amazon-ec2/../playbooks/../.. && mvn
> clean package -DskipTests -T 2C -P HDP-2.5.0.0,mpack", "delta":
> "0:00:04.845260", "end": "2018-01-23 18:28:27.608265", "failed": true,
> "rc": 1, "start": "2018-01-23 18:28:22.763005", "stderr": "", "stdout":
> "[INFO] Scanning for projects...\n[INFO] --
> --\n[INFO] Reactor Build
> Order:\n[INFO] \n[INFO] Metron\n[INFO] metron-stellar\n[INFO]
> stellar-common\n[INFO] metron-analytics\n[INFO] metron-maas-common\n[INFO]
> metron-platform\n[INFO] metron-zookeeper\n[INFO]
> metron-test-utilities\n[INFO] metron-integration-test\n[INFO]
> metron-maas-service\n[INFO] metron-common\n[INFO] metron-statistics\n[INFO]
> metron-writer\n[INFO] metron-storm-kafka-override\n[INFO]
> metron-storm-kafka\n[INFO] metron-hbase\n[INFO]
> metron-profiler-common\n[INFO] metron-profiler-client\n[INFO]
> metron-profiler\n[INFO] metron-hbase-client\n[INFO]
> metron-enrichment\n[INFO] metron-indexing\n[INFO] metron-solr\n[INFO]
> metron-pcap\n[INFO] metron-parsers\n[INFO] metron-pcap-backend\n[INFO]
> metron-data-management\n[INFO] metron-api\n[INFO] metron-management\n[INFO]
> elasticsearch-shaded\n[INFO] metron-elasticsearch\n[INFO]
> metron-deployment\n[INFO] Metron Ambari Management Pack\n[INFO]
> metron-contrib\n[INFO] metron-docker\n[INFO] metron-interface\n[INFO]
> metron-config\n[INFO] metron-alerts\n[INFO] metron-rest-client\n[INFO]
> metron-rest\n[INFO] site-book\n[INFO] 3rd party Functions (just for
> tests)\n[INFO] \n[INFO] Using the MultiThreadedBuilder implementation with
> a thread count of 8\n[INFO]
>   \n[INFO] --
> --\n[INFO] Building Metron
> 0.4.2\n[INFO] 
> \n[INFO]
> \n[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ Metron
> ---\n[INFO] \n[INFO] --- maven-enforcer-plugin:1.4.1:enforce
> (enforce-versions) @ Metron ---\n[INFO] \n[INFO] ---
> jacoco-maven-plugin:0.7.9:prepare-agent (default) @ Metron ---\n[INFO]
> argLine set to -javaagent:/Users/sujay/.m2/repository/org/jacoco/org.
> jacoco.agent/0.7.9/org.jacoco.agent-0.7.9-runtime.jar=
> destfile=/Users/sujay/Downloads/apache-metron-0.4.2-rc2/target/jacoco.exec\n[INFO]
> \n[INFO] --- jacoco-maven-plugin:0.7.9:report (report) @ Metron
> ---\n[INFO] Skipping JaCoCo execution due to missing execution data
> file.\n[INFO]
> \n[INFO] --
> --\n[INFO] Building
> metron-stellar 0.4.2\n[INFO] --
> --\n[INFO]
>   \n[INFO]
>   \n[INFO]
> \n[INFO]
> Building metron-platform 0.4.2\n[INFO] --
> --\n[INFO]
> \n[INFO]
> Building metron-analytics 0.4.2\n[INFO] --
> --\n[INFO]
>   \n[INFO]
> \n[INFO]
> 
> \n[INFO]
> \n[INFO]
> Building metron-contrib 0.4.2\n[INFO] --
> --\n[INFO] Building
> metron-deployment 0.4.2\n[INFO] --
> --\n[INFO]
>   \n[INFO]
> \n[INFO]
> Building metron-interface 0.4.2\n[INFO] --
> --\n[INFO]
>   \n[INFO]
> \n[INFO]
> Building site-book 0.4.2\n[INFO] --
> --\n[INFO] \n[INFO] ---
> 

Re: Getting Syslogs to Metron

[Gaurav Bapat]

But I cant find how to configure it

On 16 January 2018 at 11:38, Farrukh Naveed Anjum <anjum.farr...@gmail.com>
wrote:

> yes, do configure it as per metron reference usecase
>
> On Tue, Jan 16, 2018 at 8:35 AM, Gaurav Bapat <gauravb3...@gmail.com>
> wrote:
>
>> Hi Kyle,
>>
>> I saw that I can ping from my OS to VM and from VM to OS. Looks like this
>> is some Kafka or Zookeeper environment variables setup issue, do I need to
>> configure that in vagrant ssh?
>>
>> On 16 January 2018 at 08:59, Gaurav Bapat <gauravb3...@gmail.com> wrote:
>>
>>> Hey Kyle,
>>>
>>> I am running NiFi not on Ambari but on localhost:8089, I can ping from
>>> my OS terminal to node1 but can't ping from node1 to my OS terminal, I have
>>> attached few screenshots and the contents of /etc/hosts
>>>
>>> Thank You!
>>>
>>> On 15 January 2018 at 20:04, Kyle Richardson <kylerichards...@gmail.com>
>>> wrote:
>>>
>>>> It looks like your Nifi instance is running on your laptop/desktop
>>>> (e.g. the VM host). My guess would be that name resolution or networking is
>>>> not properly configured between the host and the guest preventing the data
>>>> from getting from Nifi to Kafka. What's the contents of /etc/hosts on the
>>>> VM host? Can you ping node1 from the VM host by name and by IP address?
>>>>
>>>> -Kyle
>>>>
>>>> On Mon, Jan 15, 2018 at 6:55 AM, Gaurav Bapat <gauravb3...@gmail.com>
>>>> wrote:
>>>>
>>>>> Failed while waiting for acks from Kafka is what I am getting in
>>>>> Kafka, am I missing some configuration with Kafka?
>>>>>
>>>>> On 15 January 2018 at 16:50, Gaurav Bapat <gauravb3...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Farrukh,
>>>>>>
>>>>>> I cant find any folder by my topic
>>>>>>
>>>>>> On 15 January 2018 at 16:33, Farrukh Naveed Anjum <
>>>>>> anjum.farr...@gmail.com> wrote:
>>>>>>
>>>>>>> Can you check /kafaka-logs on your VM box (It should have a folder
>>>>>>> named your topic). Can you check if it is there ?
>>>>>>>
>>>>>>> On Mon, Jan 15, 2018 at 3:49 PM, Gaurav Bapat <gauravb3...@gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> I am not getting data into my Kafka topic
>>>>>>>>
>>>>>>>> I have used i5 4 Core Processor with 16 GB RAM and I have allocated
>>>>>>>> 12 GB RAM to my vagrant VM.
>>>>>>>>
>>>>>>>> I dont understand how to configure Kafka broker because it is
>>>>>>>> giving me failed while waiting for acks to Kafka
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 15 January 2018 at 16:10, Farrukh Naveed Anjum <
>>>>>>>> anjum.farr...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Can you tell me is your KAFKA Topic getting data ? What are you
>>>>>>>>> machine specifications ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jan 15, 2018 at 2:56 PM, Gaurav Bapat <
>>>>>>>>> gauravb3...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks Farrukh,
>>>>>>>>>>
>>>>>>>>>> I am not getting data in my kafka topic even after creating one,
>>>>>>>>>> the issue seems to be with broker config, how to configure Kafka and
>>>>>>>>>> Zookeeper port?
>>>>>>>>>>
>>>>>>>>>> On 15 January 2018 at 13:23, Farrukh Naveed Anjum <
>>>>>>>>>> anjum.farr...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I had similar issue it turned out to be the issue in STROM
>>>>>>>>>>>
>>>>>>>>>>> No worker is assigned to togolgoy all you need is to add
>>>>>>>>>>> additional port in
>>>>>>>

Re: Getting Syslogs to Metron

[Gaurav Bapat]

Hi Kyle,

I saw that I can ping from my OS to VM and from VM to OS. Looks like this
is some Kafka or Zookeeper environment variables setup issue, do I need to
configure that in vagrant ssh?

On 16 January 2018 at 08:59, Gaurav Bapat <gauravb3...@gmail.com> wrote:

> Hey Kyle,
>
> I am running NiFi not on Ambari but on localhost:8089, I can ping from my
> OS terminal to node1 but can't ping from node1 to my OS terminal, I have
> attached few screenshots and the contents of /etc/hosts
>
> Thank You!
>
> On 15 January 2018 at 20:04, Kyle Richardson <kylerichards...@gmail.com>
> wrote:
>
>> It looks like your Nifi instance is running on your laptop/desktop (e.g.
>> the VM host). My guess would be that name resolution or networking is not
>> properly configured between the host and the guest preventing the data from
>> getting from Nifi to Kafka. What's the contents of /etc/hosts on the VM
>> host? Can you ping node1 from the VM host by name and by IP address?
>>
>> -Kyle
>>
>> On Mon, Jan 15, 2018 at 6:55 AM, Gaurav Bapat <gauravb3...@gmail.com>
>> wrote:
>>
>>> Failed while waiting for acks from Kafka is what I am getting in Kafka,
>>> am I missing some configuration with Kafka?
>>>
>>> On 15 January 2018 at 16:50, Gaurav Bapat <gauravb3...@gmail.com> wrote:
>>>
>>>> Hi Farrukh,
>>>>
>>>> I cant find any folder by my topic
>>>>
>>>> On 15 January 2018 at 16:33, Farrukh Naveed Anjum <
>>>> anjum.farr...@gmail.com> wrote:
>>>>
>>>>> Can you check /kafaka-logs on your VM box (It should have a folder
>>>>> named your topic). Can you check if it is there ?
>>>>>
>>>>> On Mon, Jan 15, 2018 at 3:49 PM, Gaurav Bapat <gauravb3...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I am not getting data into my Kafka topic
>>>>>>
>>>>>> I have used i5 4 Core Processor with 16 GB RAM and I have allocated
>>>>>> 12 GB RAM to my vagrant VM.
>>>>>>
>>>>>> I dont understand how to configure Kafka broker because it is giving
>>>>>> me failed while waiting for acks to Kafka
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 15 January 2018 at 16:10, Farrukh Naveed Anjum <
>>>>>> anjum.farr...@gmail.com> wrote:
>>>>>>
>>>>>>> Can you tell me is your KAFKA Topic getting data ? What are you
>>>>>>> machine specifications ?
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jan 15, 2018 at 2:56 PM, Gaurav Bapat <gauravb3...@gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Thanks Farrukh,
>>>>>>>>
>>>>>>>> I am not getting data in my kafka topic even after creating one,
>>>>>>>> the issue seems to be with broker config, how to configure Kafka and
>>>>>>>> Zookeeper port?
>>>>>>>>
>>>>>>>> On 15 January 2018 at 13:23, Farrukh Naveed Anjum <
>>>>>>>> anjum.farr...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I had similar issue it turned out to be the issue in STROM
>>>>>>>>>
>>>>>>>>> No worker is assigned to togolgoy all you need is to add
>>>>>>>>> additional port in
>>>>>>>>>
>>>>>>>>>  Ambari -> Storm -> Configs -> supervisor.slot.ports by assigning
>>>>>>>>> an additional port to the list
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://community.hortonworks.com/questions/32499/no-workers
>>>>>>>>> -in-storm-for-squid-topology.html
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I had similar issue and finally got it fixed
>>>>>>>>>
>>>>>>>>> On Mon, Jan 15, 2018 at 8:45 AM, Gaurav Bapat <
>>>>>>>>> gauravb3...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Storm UI
>>>>>>>>>>
>>>>>>>>>> On 15 January 2018 at 08:59, Gau

Re: Getting Syslogs to Metron

[Gaurav Bapat]

Hey Laurens,

My kafka processor says "Failed while waiting for acks from Kafka"

On 15 January 2018 at 21:00, Laurens Vets <laur...@daemon.be> wrote:

> Hi Gaurav,
>
> If you click on the red squares in the upper right corners of your
> processors, what error messages do you see?
>
> On 2018-01-14 19:29, Gaurav Bapat wrote:
>
> Hey Jon,
>
> I have Storm UI and the logs are coming from firewalls, servers, etc from
> other machines(HP ArcSight Logger).
>
> I have attached the NiFi screenshots, my logs are coming but there is some
> error with Kafka and I am having issues with configuring Kafka broker
>
>
>
> On 12 January 2018 at 18:14, zeo...@gmail.com <zeo...@gmail.com> wrote:
>
>> In Ambari under storm you can find the UI under quick links at the top.
>> That said, the issue seems to be upstream of Metron, in NiFi.  That is
>> something I can't help with as much, but if you can share the listensyslog
>> processor config that would be a start.  Also, share the config of the
>> thing that is sending syslog as well (are these local syslog, is that
>> machine aggregating syslog from other machines, etc.).  Thanks,
>>
>> Jon
>>
>> On Fri, Jan 12, 2018, 01:00 Gaurav Bapat <gauravb3...@gmail.com> wrote:
>>
>>> I have created a Kafka topic "cef" but my Listen Syslogs is not getting
>>> logs in the processor.
>>>
>>> Also I checked using tcpdump -i and it is getting logs in my machine but
>>> ListenSyslogs is not getting the logs
>>>
>>> On 12 January 2018 at 11:13, Gaurav Bapat <gauravb3...@gmail.com> wrote:
>>>
>>>> [root@metron incubator-metron]# ./metron-deployment/scripts/pl
>>>> atform-info.sh
>>>> Metron 0.4.3
>>>> --
>>>> * master
>>>> --
>>>> commit c559ed7e1838ec71344eae3d9e37771db2641635
>>>> Author: cstella <ceste...@gmail.com>
>>>> Date:   Tue Jan 9 15:28:47 2018 -0500
>>>>
>>>> METRON-1379: Add an OBJECT_GET stellar function closes
>>>> apache/incubator-metron#880
>>>> --
>>>>  metron-deployment/vagrant/full-dev-platform/Vagrantfile | 2 +-
>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>> --
>>>> ansible 2.0.0.2
>>>>   config file =
>>>>   configured module search path = Default w/o overrides
>>>> --
>>>> Vagrant 1.9.6
>>>> --
>>>> Python 2.7.5
>>>> --
>>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
>>>> 2015-11-10T22:11:47+05:30)
>>>> Maven home: /opt/maven/current
>>>> Java version: 1.8.0_151, vendor: Oracle Corporation
>>>> Java home: /opt/jdk1.8.0_151/jre
>>>> Default locale: en_US, platform encoding: UTF-8
>>>> OS name: "linux", version: "3.10.0-693.11.6.el7.x86_64", arch: "amd64",
>>>> family: "unix"
>>>> --
>>>> Docker version 1.12.6, build ec8512b/1.12.6
>>>> --
>>>> node
>>>> v8.9.3
>>>> --
>>>> npm
>>>> 5.5.1
>>>> --
>>>> g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16)
>>>> Copyright (C) 2015 Free Software Foundation, Inc.
>>>> This is free software; see the source for copying conditions.  There is
>>>> NO
>>>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
>>>> PURPOSE.
>>>>
>>>> --
>>>> Compiler is C++11 compliant
>>>> --
>>>> Linux metron.com 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4 01:06:37
>>>> UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
>>>> --
>>>> Total System Memory = 15773.3 MB
>>>> Processor Model: Intel(R) Core(TM) i5-3450 CPU @ 3.10GHz
>>>> Processor Speed: 3320.875 MHz
>>>> Processor Speed: 3307.191 MHz
>>>> Processor Speed: 3376.699 MHz
>>>> Processor Speed: 3338.917 MHz
>>>> Total Physical Processors: 4
>>>> Total cores: 16
>>>> Disk information:
>>>> /dev/mapper/centos-root  200G   22G  179G  11% /
>>>> /dev/sda12.0G  224M  1.8G  11% /boot
>>>> /dev/sda2   1022M   12K 1022M   1% /boot/efi
>>>> /dev/mapper/centos-home  247G   10G  237G   5% /home
>>>> This CPU appears to support virtualization
>>>>
>>>> On 12 January 2018 at 09:25, Gaurav Bapat <gauravb3...@gmail.com>
&

Re: Getting Syslogs to Metron

[Gaurav Bapat]

Hi Farrukh,

I cant find any folder by my topic

On 15 January 2018 at 16:33, Farrukh Naveed Anjum <anjum.farr...@gmail.com>
wrote:

> Can you check /kafaka-logs on your VM box (It should have a folder named
> your topic). Can you check if it is there ?
>
> On Mon, Jan 15, 2018 at 3:49 PM, Gaurav Bapat <gauravb3...@gmail.com>
> wrote:
>
>> I am not getting data into my Kafka topic
>>
>> I have used i5 4 Core Processor with 16 GB RAM and I have allocated 12 GB
>> RAM to my vagrant VM.
>>
>> I dont understand how to configure Kafka broker because it is giving me
>> failed while waiting for acks to Kafka
>>
>>
>>
>> On 15 January 2018 at 16:10, Farrukh Naveed Anjum <
>> anjum.farr...@gmail.com> wrote:
>>
>>> Can you tell me is your KAFKA Topic getting data ? What are you machine
>>> specifications ?
>>>
>>>
>>> On Mon, Jan 15, 2018 at 2:56 PM, Gaurav Bapat <gauravb3...@gmail.com>
>>> wrote:
>>>
>>>> Thanks Farrukh,
>>>>
>>>> I am not getting data in my kafka topic even after creating one, the
>>>> issue seems to be with broker config, how to configure Kafka and Zookeeper
>>>> port?
>>>>
>>>> On 15 January 2018 at 13:23, Farrukh Naveed Anjum <
>>>> anjum.farr...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I had similar issue it turned out to be the issue in STROM
>>>>>
>>>>> No worker is assigned to togolgoy all you need is to add additional
>>>>> port in
>>>>>
>>>>>  Ambari -> Storm -> Configs -> supervisor.slot.ports by assigning an
>>>>> additional port to the list
>>>>>
>>>>>
>>>>> https://community.hortonworks.com/questions/32499/no-workers
>>>>> -in-storm-for-squid-topology.html
>>>>>
>>>>>
>>>>> I had similar issue and finally got it fixed
>>>>>
>>>>> On Mon, Jan 15, 2018 at 8:45 AM, Gaurav Bapat <gauravb3...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Storm UI
>>>>>>
>>>>>> On 15 January 2018 at 08:59, Gaurav Bapat <gauravb3...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hey Jon,
>>>>>>>
>>>>>>> I have Storm UI and the logs are coming from firewalls, servers, etc
>>>>>>> from other machines(HP ArcSight Logger).
>>>>>>>
>>>>>>> I have attached the NiFi screenshots, my logs are coming but there
>>>>>>> is some error with Kafka and I am having issues with configuring Kafka
>>>>>>> broker
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 12 January 2018 at 18:14, zeo...@gmail.com <zeo...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> In Ambari under storm you can find the UI under quick links at the
>>>>>>>> top.  That said, the issue seems to be upstream of Metron, in NiFi.  
>>>>>>>> That
>>>>>>>> is something I can't help with as much, but if you can share the
>>>>>>>> listensyslog processor config that would be a start.  Also, share the
>>>>>>>> config of the thing that is sending syslog as well (are these local 
>>>>>>>> syslog,
>>>>>>>> is that machine aggregating syslog from other machines, etc.).  Thanks,
>>>>>>>>
>>>>>>>> Jon
>>>>>>>>
>>>>>>>> On Fri, Jan 12, 2018, 01:00 Gaurav Bapat <gauravb3...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I have created a Kafka topic "cef" but my Listen Syslogs is not
>>>>>>>>> getting logs in the processor.
>>>>>>>>>
>>>>>>>>> Also I checked using tcpdump -i and it is getting logs in my
>>>>>>>>> machine but ListenSyslogs is not getting the logs
>>>>>>>>>
>>>>>>>>> On 12 January 2018 at 11:13, Gaurav Bapat <gauravb3...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> [root@m

Re: Getting Syslogs to Metron

[Gaurav Bapat]

Thanks Farrukh,

I am not getting data in my kafka topic even after creating one, the issue
seems to be with broker config, how to configure Kafka and Zookeeper port?

On 15 January 2018 at 13:23, Farrukh Naveed Anjum <anjum.farr...@gmail.com>
wrote:

> Hi,
>
> I had similar issue it turned out to be the issue in STROM
>
> No worker is assigned to togolgoy all you need is to add additional port in
>
>  Ambari -> Storm -> Configs -> supervisor.slot.ports by assigning an
> additional port to the list
>
>
> https://community.hortonworks.com/questions/32499/no-
> workers-in-storm-for-squid-topology.html
>
>
> I had similar issue and finally got it fixed
>
> On Mon, Jan 15, 2018 at 8:45 AM, Gaurav Bapat <gauravb3...@gmail.com>
> wrote:
>
>> Storm UI
>>
>> On 15 January 2018 at 08:59, Gaurav Bapat <gauravb3...@gmail.com> wrote:
>>
>>> Hey Jon,
>>>
>>> I have Storm UI and the logs are coming from firewalls, servers, etc
>>> from other machines(HP ArcSight Logger).
>>>
>>> I have attached the NiFi screenshots, my logs are coming but there is
>>> some error with Kafka and I am having issues with configuring Kafka broker
>>>
>>>
>>>
>>> On 12 January 2018 at 18:14, zeo...@gmail.com <zeo...@gmail.com> wrote:
>>>
>>>> In Ambari under storm you can find the UI under quick links at the
>>>> top.  That said, the issue seems to be upstream of Metron, in NiFi.  That
>>>> is something I can't help with as much, but if you can share the
>>>> listensyslog processor config that would be a start.  Also, share the
>>>> config of the thing that is sending syslog as well (are these local syslog,
>>>> is that machine aggregating syslog from other machines, etc.).  Thanks,
>>>>
>>>> Jon
>>>>
>>>> On Fri, Jan 12, 2018, 01:00 Gaurav Bapat <gauravb3...@gmail.com> wrote:
>>>>
>>>>> I have created a Kafka topic "cef" but my Listen Syslogs is not
>>>>> getting logs in the processor.
>>>>>
>>>>> Also I checked using tcpdump -i and it is getting logs in my machine
>>>>> but ListenSyslogs is not getting the logs
>>>>>
>>>>> On 12 January 2018 at 11:13, Gaurav Bapat <gauravb3...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> [root@metron incubator-metron]# ./metron-deployment/scripts/pl
>>>>>> atform-info.sh
>>>>>> Metron 0.4.3
>>>>>> --
>>>>>> * master
>>>>>> --
>>>>>> commit c559ed7e1838ec71344eae3d9e37771db2641635
>>>>>> Author: cstella <ceste...@gmail.com>
>>>>>> Date:   Tue Jan 9 15:28:47 2018 -0500
>>>>>>
>>>>>> METRON-1379: Add an OBJECT_GET stellar function closes
>>>>>> apache/incubator-metron#880
>>>>>> --
>>>>>>  metron-deployment/vagrant/full-dev-platform/Vagrantfile | 2 +-
>>>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>> --
>>>>>> ansible 2.0.0.2
>>>>>>   config file =
>>>>>>   configured module search path = Default w/o overrides
>>>>>> --
>>>>>> Vagrant 1.9.6
>>>>>> --
>>>>>> Python 2.7.5
>>>>>> --
>>>>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
>>>>>> 2015-11-10T22:11:47+05:30)
>>>>>> Maven home: /opt/maven/current
>>>>>> Java version: 1.8.0_151, vendor: Oracle Corporation
>>>>>> Java home: /opt/jdk1.8.0_151/jre
>>>>>> Default locale: en_US, platform encoding: UTF-8
>>>>>> OS name: "linux", version: "3.10.0-693.11.6.el7.x86_64", arch:
>>>>>> "amd64", family: "unix"
>>>>>> --
>>>>>> Docker version 1.12.6, build ec8512b/1.12.6
>>>>>> --
>>>>>> node
>>>>>> v8.9.3
>>>>>> --
>>>>>> npm
>>>>>> 5.5.1
>>>>>> --
>>>>>> g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16)
>>>>>> Copyright (C) 2015 Free Software Foundation, Inc.
>>>>>> This is free software; see the source for copying conditions.  There
>>>>>> is NO
>>>>>> warranty; not even for MERCHANTABILITY

Getting Syslogs to Metron

[Gaurav Bapat]

Hello everyone, I have deployed Metron on a single node machine and I would
like to know how do I get Syslogs from NiFi into Kibana dashboard?

I have created a Kafka topic by the name "cef" and I can see that the topic
exists in
Metron Configuration but I am unable to connect it with Kibana

Need Help!!

Re: Metron Installation on EC2

[Gaurav Bapat]

Hey James,

10 M4 Xlarges means that I will have 10 nodes or just one? Also how much
HDD capacity will I need? Atleast 1TB to start with?

On 9 January 2018 at 09:14, James Byrne <james.by...@intrepidtravel.com>
wrote:

> https://github.com/apache/metron/tree/master/metron-deployment/amazon-ec2
>
> 10 M4 Xlarges for ya. Or just run up a few instances and manually build it.
>
>
>
> *From:* Gaurav Bapat [mailto:gauravb3...@gmail.com]
> *Sent:* Tuesday, 9 January 2018 2:40 PM
> *To:* user@metron.apache.org
> *Cc:* ruchi.yag...@sequretek.com
> *Subject:* Metron Installation on EC2
>
>
>
> Hi,
>
> How do I deploy Metron on Amazon EC2? What kind of hardware will I need
> for this and also the pricing for the same?
>
> Previously I had deployed Metron on a single node machine with 16 GB RAM.
>
> Please Help!!
>
> Regards,
>
> Gaurav
>

Re: Metron Version

[Gaurav Bapat]

There are no errors in Storm, the topic is emitting just like Snort & Bro
but I still cant understand the problem

On Fri, Jan 5, 2018 at 19:54 zeo...@gmail.com <zeo...@gmail.com> wrote:

> Are you able to look through the storm UI and identify any errors?  Also,
> did you look at the Metron error dashboard?  Thanks,
>
> Jon
>
> On Thu, Jan 4, 2018, 22:47 Gaurav Bapat <gauravb3...@gmail.com> wrote:
>
>> Also when I enter indices in Kibana, it fails to search for my Kafka
>> topic and I dont know why the cef logs are not coming into Kibana
>>
>>
>>
>> On 5 January 2018 at 00:23, Simon Elliston Ball <
>> si...@simonellistonball.com> wrote:
>>
>>> Are the logs you’re sending with syslog in CEF format? You will note
>>> that the CEF sensor uses the CEF parser, which means unless your logs are
>>> in CEF format, they will fail to parse and be dropped into the error index
>>> (worth checking the error index in kibana via the Metron Error Dashboard.
>>> That will likely tell you why things aren’t parsing.
>>>
>>> The most likely scenario is that you are sending something non-CEF on
>>> the syslog feed, in which case you will need something like a Grok parser.
>>> I suggest reading through the Squid example in the documentation on how to
>>> do this.
>>>
>>> Simon
>>>
>>> > On 4 Jan 2018, at 18:49, Gaurav Bapat <gauravb3...@gmail.com> wrote:
>>> >
>>> > They are syslogs and my topic name is cef, I get one parsed logs out
>>> of 1000+ and I want to do analytics using Spark but I cant find a way out.
>>>
>>> --
>
> Jon
>

Re: [ANNOUNCE] Apache Metron release 0.4.2 and Apache Metron bro plugin for Kafka release 0.1

[Gaurav Bapat]

Hi,

I have deployed Metron with 16 GB RAM, but all components are losing
heartbeats and they dont go up

On 5 January 2018 at 02:41, Matt Foley  wrote:

> Metron Community:  Happy New Year.
>
> I’m happy to announce the release of Metron 0.4.2.  A great deal of work
> from across the community went into this, with over 100 enhancements,
> improvements, and bug fixes since 0.4.1.  Thanks to all contributors, and
> may all users enjoy the new features!
>
> This release also includes the first official release of the
> apache-metron-bro-plugin-kafka, version 0.1.
>
> Details:
> The official release source code tarballs may be obtained at any of the
> mirrors listed in
> http://www.apache.org/dyn/closer.cgi/metron/0.4.2/
>
> As usual, the secure signatures and confirming hashes may be obtained at
> https://dist.apache.org/repos/dist/release/metron/0.4.2/
>
> The release branches in github are
> https://github.com/apache/metron/tree/Metron_0.4.2 (tag
> apache-metron-0.4.2-release)
> https://github.com/apache/metron-bro-plugin-kafka/tree/0.1 (tag 0.1)
>
> The release doc book is at http://metron.apache.org/
> current-book/index.html
> The Apache Metron web site at http://metron.apache.org/ has been updated;
> please refresh your web browser cache if the new links do not immediately
> appear.
>
> Change lists and Release Notes may be obtained at the same locations as
> the tarballs.
> For your reading pleasure, the change list is appended to this message.
>
> Best regards,
> --Matt Foley
> release manager
>
> Metron CHANGES (in reverse chron order):
> METRON-1373 RAT failure for metron-interface/metron-alerts
> (mattf-horton) closes apache/metron#875
> METRON-1313 Update metron-deployment to use bro-pkg to install the
> kafka plugin (JonZeolla) closes apache/metron#847
> METRON-1346 Add new PMC members to web site (ottobackwards) closes
> apache/metron#860
> METRON-1336 Patching Can Result in Bad Configuration (nickwallen)
> closes apache/metron#851
> METRON-1335 Install metron-maas-service RPM as a part of the full-dev
> deployment (anandsubbu via ottobackwards) closes apache/metron#850
> METRON-1308 Fix Metron Documentation (JonZeolla) closes
> apache/metron#836
> METRON-1338 Rat Check Should Ignore Vagrant Retry Files (nickwallen)
> closes apache/metron#855
> METRON-1286 Add MIN & MAX Stellar functions (jasper-k via justinleet)
> closes apache/metron#823
> METRON-1334 Add C++11 Compliance Check to platform-info.sh
> (nickwallen) closes apache/metron#849
> METRON-1277 Add match statement to Stellar language closes
> apache/incubator-metron#814
> METRON-1239 Drop extra dev environments (nickwallen) closes
> apache/metron#852
> METRON-1328 Enhance platform-info.sh script to check if docker daemon
> is running  (anandsubbu via nickwallen) closes apache/metron#846
> METRON-1333 Ansible-Docker can no longer build metron (ottobackwards)
> closes apache/metron#848
> METRON-1252 Build UI for grouping alerts into meta-alerts (iraghumitra
> via nickwallen) closes apache/metron#803
> METRON-1316 Fastcapa Fails to Compile in Test Environment (nickwallen)
> closes apache/metron#841
> METRON-1088 Upgrade bro to 2.5.2 (JonZeolla) closes apache/metron#844
> METRON-1319 Column Metadata REST service should use default indices on
> empty input (merrimanr) closes apache/metron#843
> METRON-1321 Metaalert Threat Score Type Does Not Match Sensor Indices
> (nickwallen) closes apache/metron#845
> METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly Filters
> Some Records (nickwallen) closes apache/metron#832
> METRON-1294 IP addresses are not formatted correctly in facet and
> group results (merrimanr) closes apache/metron#827
> METRON-1291 Kafka produce REST endpoint does not work in a Kerberized
> cluster (merrimanr) closes apache/metron#826
> METRON-1290 Only first 10 alerts are update when a MetaAlert status is
> changed to inactive (justinleet) closes apache/metron#842
> METRON-1311 Service Check Should Check Elasticsearch Index Templates
> (nickwallen) closes apache/metron#839
> METRON-1289 Alert fields are lost when a MetaAlert is created
> (merrimanr) closes apache/metron#824
> METRON-1309 Change metron-deployment to pull the plugin from
> apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
> METRON-1310 Template Delete Action Deletes Search Indices (nickwallen)
> closes apache/metron#838
> METRON-1275 Fix Metron Documentation closes apache/incubator-metron#833
> METRON-1295 Unable to Configure Logging for REST API (nickwallen)
> closes apache/metron#828
> METRON-1307 Force install of java8 since java9 does not appear to work
> with the scripts (brianhurley via ottobackwards) closes apache/metron#835
> METRON-1296 Full Dev Fails to Deploy Index Templates (nickwallen via
> cestella) closes apache/incubator-metron#829
> METRON-1281 Remove hard-coded indices from 

Re: [ANNOUNCE] Apache Metron release 0.4.2 and Apache Metron bro plugin for Kafka release 0.1

[Gaurav Bapat]

Hi,

That's a great news!!

I am forwarding syslogs into Kafka through Nifi but I can't see the logs in
Kibana, I have added the CEF Parser with cef topic, is there any issue with
parsing and does the newer version directly parse and index telemetry in ES
and then to Kibana.

Request you to add Syslog Parser to make life easier.

Thanks,
Gaurav

On 5 January 2018 at 02:41, Matt Foley  wrote:

> Metron Community:  Happy New Year.
>
> I’m happy to announce the release of Metron 0.4.2.  A great deal of work
> from across the community went into this, with over 100 enhancements,
> improvements, and bug fixes since 0.4.1.  Thanks to all contributors, and
> may all users enjoy the new features!
>
> This release also includes the first official release of the
> apache-metron-bro-plugin-kafka, version 0.1.
>
> Details:
> The official release source code tarballs may be obtained at any of the
> mirrors listed in
> http://www.apache.org/dyn/closer.cgi/metron/0.4.2/
>
> As usual, the secure signatures and confirming hashes may be obtained at
> https://dist.apache.org/repos/dist/release/metron/0.4.2/
>
> The release branches in github are
> https://github.com/apache/metron/tree/Metron_0.4.2 (tag
> apache-metron-0.4.2-release)
> https://github.com/apache/metron-bro-plugin-kafka/tree/0.1 (tag 0.1)
>
> The release doc book is at http://metron.apache.org/
> current-book/index.html
> The Apache Metron web site at http://metron.apache.org/ has been updated;
> please refresh your web browser cache if the new links do not immediately
> appear.
>
> Change lists and Release Notes may be obtained at the same locations as
> the tarballs.
> For your reading pleasure, the change list is appended to this message.
>
> Best regards,
> --Matt Foley
> release manager
>
> Metron CHANGES (in reverse chron order):
> METRON-1373 RAT failure for metron-interface/metron-alerts
> (mattf-horton) closes apache/metron#875
> METRON-1313 Update metron-deployment to use bro-pkg to install the
> kafka plugin (JonZeolla) closes apache/metron#847
> METRON-1346 Add new PMC members to web site (ottobackwards) closes
> apache/metron#860
> METRON-1336 Patching Can Result in Bad Configuration (nickwallen)
> closes apache/metron#851
> METRON-1335 Install metron-maas-service RPM as a part of the full-dev
> deployment (anandsubbu via ottobackwards) closes apache/metron#850
> METRON-1308 Fix Metron Documentation (JonZeolla) closes
> apache/metron#836
> METRON-1338 Rat Check Should Ignore Vagrant Retry Files (nickwallen)
> closes apache/metron#855
> METRON-1286 Add MIN & MAX Stellar functions (jasper-k via justinleet)
> closes apache/metron#823
> METRON-1334 Add C++11 Compliance Check to platform-info.sh
> (nickwallen) closes apache/metron#849
> METRON-1277 Add match statement to Stellar language closes
> apache/incubator-metron#814
> METRON-1239 Drop extra dev environments (nickwallen) closes
> apache/metron#852
> METRON-1328 Enhance platform-info.sh script to check if docker daemon
> is running  (anandsubbu via nickwallen) closes apache/metron#846
> METRON-1333 Ansible-Docker can no longer build metron (ottobackwards)
> closes apache/metron#848
> METRON-1252 Build UI for grouping alerts into meta-alerts (iraghumitra
> via nickwallen) closes apache/metron#803
> METRON-1316 Fastcapa Fails to Compile in Test Environment (nickwallen)
> closes apache/metron#841
> METRON-1088 Upgrade bro to 2.5.2 (JonZeolla) closes apache/metron#844
> METRON-1319 Column Metadata REST service should use default indices on
> empty input (merrimanr) closes apache/metron#843
> METRON-1321 Metaalert Threat Score Type Does Not Match Sensor Indices
> (nickwallen) closes apache/metron#845
> METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly Filters
> Some Records (nickwallen) closes apache/metron#832
> METRON-1294 IP addresses are not formatted correctly in facet and
> group results (merrimanr) closes apache/metron#827
> METRON-1291 Kafka produce REST endpoint does not work in a Kerberized
> cluster (merrimanr) closes apache/metron#826
> METRON-1290 Only first 10 alerts are update when a MetaAlert status is
> changed to inactive (justinleet) closes apache/metron#842
> METRON-1311 Service Check Should Check Elasticsearch Index Templates
> (nickwallen) closes apache/metron#839
> METRON-1289 Alert fields are lost when a MetaAlert is created
> (merrimanr) closes apache/metron#824
> METRON-1309 Change metron-deployment to pull the plugin from
> apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
> METRON-1310 Template Delete Action Deletes Search Indices (nickwallen)
> closes apache/metron#838
> METRON-1275 Fix Metron Documentation closes apache/incubator-metron#833
> METRON-1295 Unable to Configure Logging for REST API (nickwallen)
> closes apache/metron#828
> METRON-1307 Force install of java8 since java9 does not appear to work
> with 

Re: Metron Version

[Gaurav Bapat]

They are syslogs and my topic name is cef, I get one parsed logs out of
1000+ and I want to do analytics using Spark but I cant find a way out.

Re: bro kafka plugin build error on --bro-init=$BRO_SRC option doesn't exist

[Gaurav Bapat]

Can I send syslogs to HDFS using NiFi without using Kafka Topic?

On 21 Dec 2017 5:16 p.m., "zeo...@gmail.com"  wrote:

> Where did you get the plugin from, and do you have $BRO_SRC set?  This
> plugin has recently moved, had a release, and became a package.  The
> documentation you point to is outdated at this point, and updated
> documentation is a part of a release that's currently being voted on.
>
> Please use bro-pkg to install this, or go directly to
> https://github.com/apache/metron-bro-plugin-kafka
>
> Bro dist is definitely in configure, https://github.com/apache/
> metron-bro-plugin-kafka/blob/master/configure#L86
>
> Jon
>
> On Thu, Dec 21, 2017, 00:16 pele_smk  wrote:
>
>> I'm following the instructions for building the bro Kafka plugin and I'm
>> getting an error with the --bro-dist option does not exist in ./configure
>>
>> The command:
>> ./configure --bro-dist=$BRO_SRC
>>
>> The instructions I'm following:
>> https://metron.apache.org/current-book/metron-sensors/
>> bro-plugin-kafka/index.html
>>
>> The --enable-sasl option exists in the configure and works fine
>>
>> ./configure --enable-sasl
>>
>>
>>
>> Am I missing something obvious?
>>
>> Thanks,
>> Daniel
>>
> --
>
> Jon
>

Re: Single Node Metron Installation

[Gaurav Bapat]

Can you give me the link??

On 18 Dec 2017 5:14 p.m., "zeo...@gmail.com" <zeo...@gmail.com> wrote:

> You can send them via email or if they're too long upload to a
> pastebin website and send a link.
>
> Jon
>
> On Mon, Dec 18, 2017 at 6:14 AM Gaurav Bapat <gauravb3...@gmail.com>
> wrote:
>
>> The metron component was up for 3 hours but later lost the heartbeats of
>> all component.
>>
>> From were do I send you the logs??
>>
>>
>>
>> On 18 December 2017 at 15:58, zeo...@gmail.com <zeo...@gmail.com> wrote:
>>
>>> Can you provide the logs of the initial failure?
>>>
>>> Jon
>>>
>>> On Sun, Dec 17, 2017, 23:55 Gaurav Bapat <gauravb3...@gmail.com> wrote:
>>>
>>>> [gaurav@metron full-dev-platform]$ cat Vagrantfile
>>>> #
>>>> #  Licensed to the Apache Software Foundation (ASF) under one or more
>>>> #  contributor license agreements.  See the NOTICE file distributed with
>>>> #  this work for additional information regarding copyright ownership.
>>>> #  The ASF licenses this file to You under the Apache License, Version
>>>> 2.0
>>>> #  (the "License"); you may not use this file except in compliance with
>>>> #  the License.  You may obtain a copy of the License at
>>>> #
>>>> #  http://www.apache.org/licenses/LICENSE-2.0
>>>> #
>>>> #  Unless required by applicable law or agreed to in writing, software
>>>> #  distributed under the License is distributed on an "AS IS" BASIS,
>>>> #  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>>> implied.
>>>> #  See the License for the specific language governing permissions and
>>>> #  limitations under the License.
>>>> #
>>>> require 'getoptlong'
>>>>
>>>> ansibleTags=''
>>>> ansibleSkipTags='sensors'
>>>>
>>>> begin
>>>>opts = GetoptLong.new(
>>>>  [ '--ansible-tags', GetoptLong::OPTIONAL_ARGUMENT ],
>>>>  [ '--ansible-skip-tags', GetoptLong::OPTIONAL_ARGUMENT ]
>>>>)
>>>>
>>>>opts.quiet = TRUE
>>>>
>>>>opts.each do |opt, arg|
>>>>  case opt
>>>>when '--ansible-tags'
>>>>  ansibleTags=arg
>>>>when '--ansible-skip-tags'
>>>>  ansibleSkipTags=arg
>>>>  end
>>>>end
>>>> rescue Exception => ignored
>>>> #Ignore to allow other opts to be passed to Vagrant
>>>> end
>>>>
>>>> puts " Running with ansible-tags: " + ansibleTags.split(",").to_s if
>>>> ansibleTags != ''
>>>> puts " Running with ansible-skip-tags: " +
>>>> ansibleSkipTags.split(",").to_s if ansibleSkipTags != ''
>>>>
>>>> hosts = [{
>>>> hostname: "node1",
>>>> ip: "192.168.66.121",
>>>> memory: "8192",
>>>> cpus: 4,
>>>> promisc: 2  # enables promisc on the 'Nth' network interface
>>>> }]
>>>>
>>>> Vagrant.configure(2) do |config|
>>>>
>>>>   # all hosts built on centos 6
>>>>   config.vm.box = "metron/centos_base"
>>>>   config.ssh.insert_key = true
>>>>
>>>>   # enable the hostmanager plugin
>>>>   config.hostmanager.enabled = true
>>>>   config.hostmanager.manage_host = true
>>>>
>>>>   # host definition
>>>>   hosts.each_with_index do |host, index|
>>>> config.vm.define host[:hostname] do |node|
>>>>
>>>>   # host settings
>>>>   node.vm.hostname = host[:hostname]
>>>>   node.vm.network "private_network", ip: host[:ip]
>>>>
>>>>   # vm settings
>>>>   node.vm.provider "virtualbox" do |vb|
>>>> vb.memory = host[:memory]
>>>> vb.cpus = host[:cpus]
>>>>
>>>> # enable promisc mode on the network interface
>>>> if host.has_key?(:promisc)
>>>>   vb.customize ["modifyvm", :id, "--nicpromisc#{host[:promisc]}",
>>>> "allow-all"]
>>>> end
>>>>   end
>>>> end
>>>>   end
>&g

Re: Single Node Metron Installation

[Gaurav Bapat]

[gaurav@metron full-dev-platform]$ cat Vagrantfile
#
#  Licensed to the Apache Software Foundation (ASF) under one or more
#  contributor license agreements.  See the NOTICE file distributed with
#  this work for additional information regarding copyright ownership.
#  The ASF licenses this file to You under the Apache License, Version 2.0
#  (the "License"); you may not use this file except in compliance with
#  the License.  You may obtain a copy of the License at
#
#  http://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
#  limitations under the License.
#
require 'getoptlong'

ansibleTags=''
ansibleSkipTags='sensors'

begin
   opts = GetoptLong.new(
 [ '--ansible-tags', GetoptLong::OPTIONAL_ARGUMENT ],
 [ '--ansible-skip-tags', GetoptLong::OPTIONAL_ARGUMENT ]
   )

   opts.quiet = TRUE

   opts.each do |opt, arg|
 case opt
   when '--ansible-tags'
 ansibleTags=arg
   when '--ansible-skip-tags'
 ansibleSkipTags=arg
 end
   end
rescue Exception => ignored
#Ignore to allow other opts to be passed to Vagrant
end

puts " Running with ansible-tags: " + ansibleTags.split(",").to_s if
ansibleTags != ''
puts " Running with ansible-skip-tags: " + ansibleSkipTags.split(",").to_s
if ansibleSkipTags != ''

hosts = [{
hostname: "node1",
ip: "192.168.66.121",
memory: "8192",
cpus: 4,
promisc: 2  # enables promisc on the 'Nth' network interface
}]

Vagrant.configure(2) do |config|

  # all hosts built on centos 6
  config.vm.box = "metron/centos_base"
  config.ssh.insert_key = true

  # enable the hostmanager plugin
  config.hostmanager.enabled = true
  config.hostmanager.manage_host = true

  # host definition
  hosts.each_with_index do |host, index|
config.vm.define host[:hostname] do |node|

  # host settings
  node.vm.hostname = host[:hostname]
  node.vm.network "private_network", ip: host[:ip]

  # vm settings
  node.vm.provider "virtualbox" do |vb|
vb.memory = host[:memory]
vb.cpus = host[:cpus]

# enable promisc mode on the network interface
if host.has_key?(:promisc)
  vb.customize ["modifyvm", :id, "--nicpromisc#{host[:promisc]}",
"allow-all"]
end
  end
end
  end

  # provisioning
  config.vm.provision :ansible do |ansible|
ansible.playbook = "../../playbooks/metron_full_install.yml"
ansible.sudo = true
ansible.tags = ansibleTags.split(",") if ansibleTags != ''
ansible.skip_tags = ansibleSkipTags.split(",") if ansibleSkipTags != ''
ansible.inventory_path = "../../inventory/full-dev-platform"
  end
end


On 18 December 2017 at 09:43, pele_smk <pele...@gmail.com> wrote:

> Hey Gaurav,
> I'm also in the process of building a single node VM. I'm running into a
> different set of errors. If you would share your Vagrantfile and any other
> changes to ansible or playbooks you made I'll replicate your setup on my
> end and see if I hit the TLS issue and troubleshoot it from there.
>
> Thanks,
> Daniel
>
> On Sun, Dec 17, 2017 at 7:02 PM, Gaurav Bapat <gauravb3...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> I have deployed Metron on single node on centOS 7.2 and used 16 GB RAM
>> with i5 processor but I am stuck with some network issues.
>>
>> I am getting connection refused error(Error No 111), also when I check
>> the logs it says TLS handshake failed.
>>
>> And when I restart all services, it gives me heartbeat lost error
>>
>> Please help, as I am not able to deploy Metron.
>>
>> Thanks,
>> Gaurav
>>
>
>

Re: AWS elastic IP configuration for metron?

[Gaurav Bapat]

I am facing issues in Metron Deployment, when I load maven it gives me
dependency errors, need help

On Fri, Dec 8, 2017 at 20:21 Ahmed Shah  wrote:

> Hello Pele_smk,
>
>
> Our team adapted the Metron Single Node VM install to deploy a single node
> to AWS.
>
> https://cwiki.apache.org/confluence/display/METRON/Dev+VM+Install
>
>
>
> Our Vagrent file is here:
>
>
> https://github.com/LTW-GCR-CSOC/csoc-installation-scripts/blob/master/amazon-deploy/Metron/Vagrantfile
>
> You can define your AWS Elastic IP,  Subnet ID, and Security Group ID.
>
>
> Hope it helps.
>
>
> -Ahmed
> ___
> Ahmed Shah (PMP, M. Eng.)
> Cybersecurity Analyst & Developer
> GCR - Cybersecurity Operations Center
> Carleton University - cugcr.com 
>
>
> --
> *From:* zeo...@gmail.com 
> *Sent:* December 8, 2017 9:06 AM
> *To:* user@metron.apache.org
> *Subject:* Re: AWS elastic IP configuration for metron?
>
> Are you using AWS for a testing, or a more static infrastructure?  If it's
> just to test I've found that just spinning up everything on one big box in
> AWS could help that problem a bit (if you would point to localhost or
> similar during setup).  Requires a bit of a hack - I have a very old
> version of what I did here
>  to spin it up
> on one big machine (mostly this
> ).
>
>
> Jon
>
> On Fri, Dec 8, 2017 at 12:50 AM pele_smk  wrote:
>
> When spinning up metron is it possible to define the IPs/hostnames to use?
> Otherwise, upon shutting down the cluster in AWS, hostnames change and
> updating ambari, etc... a mess seems to ensue. Any tips and tricks to
> manage turning up a cluster using EIP?
>
> --
>
> Jon
>

Re: machine learning libraries supported

[Gaurav Bapat]

Hi Moshe,

Even I want to know about ML libraries on Metron, I think Spark might help
but I dont know how will I setup Metron

Be in touch!!

Thank You,
Gaurav

On 6 December 2017 at 13:15, moshe jarusalem  wrote:

> Hi All,
> Would you please suggest some documentation about machine learning
> libraries can be used in metron architecture? and how ? any examples
> appretiated.
>
> regards,
>
>

Metron Setup

[Gaurav Bapat]

Hi Metron Team,

I want to install Metron on a single node VM but I am getting Metron
components heartbeat lost errors

Please help