PCAP on dashboard

2017-05-26 Thread tkg_cangkul 

hi,

i'm trying to using PCAP on metron. i'm using pycapa now and i've 
success to store it into kafka and hdfs.
So, what's the main function of PCAP on metron? can i show it to the 
dashboard? or it's just stored in hdfs only and read it by CLI.?



Need Your Advice,


Best Regards,

Tkg_Cangkul

Re: PCAP on dashboard

2017-06-07 Thread tkg_cangkul 

hi Nick,

thx for your reply.
Ok, so i can use Zeppelin for this pcap dashboard.
is there any way to connected the kibana to Zeppelin? i mean is there 
any way to select the pcap data to the kibana ui if i embed it in zeppelin?


By the way, i've tried to use pcap_query tool. but i've found some error 
message below :




any suggestion for that?

Best Regards,

On 29/05/17 20:49, Nick Allen wrote:
Right now it is stored in HDFS and then retrieved with the pcap_query 
tool. The pcap_query tool can also be embedded in a Zeppelin 
Notebook.  Of course with this data is in HDFS, you can integrate with 
it using your tool of choice.


If you have use cases in mind, please feel free to share.

On Fri, May 26, 2017 at 6:47 AM, tkg_cangkul <mailto:yuza.ras...@gmail.com>> wrote:


hi,

i'm trying to using PCAP on metron. i'm using pycapa now and i've
success to store it into kafka and hdfs.
So, what's the main function of PCAP on metron? can i show it to
the dashboard? or it's just stored in hdfs only and read it by CLI.?


    Need Your Advice,


Best Regards,

Tkg_Cangkul

Re: PCAP on dashboard

2017-06-07 Thread tkg_cangkul 

I'm using metron 0.3.0 and i run on my cluster machines.
this is the command that i run :

/usr/metron/0.3.0/bin/pcap_query.sh query -st "20160617" -df "MMdd" 
-bop "/apps/metron/out" --query "ip_src_addr == '10.1.80.220' and 
ip_src_port == '6667' and ip_dst_addr == '10.1.80.221' and ip_dst_port 
== '42885' and protocol == '6'"


i also have try to using pcap_query.sh with fixed option but i still got 
the message Could not initialize class java.net.NetworkInterface


Best Regards,

On 07/06/17 20:19, Nick Allen wrote:
You need to provide more information about your environment.  What 
version of Metron?  How are you running Metron (metal, VM, cloud)? 
What command did you run?


On Wed, Jun 7, 2017 at 3:27 AM, tkg_cangkul <mailto:yuza.ras...@gmail.com>> wrote:


hi Nick,

thx for your reply.
Ok, so i can use Zeppelin for this pcap dashboard.
is there any way to connected the kibana to Zeppelin? i mean is
there any way to select the pcap data to the kibana ui if i embed
it in zeppelin?

By the way, i've tried to use pcap_query tool. but i've found some
error message below :



any suggestion for that?

Best Regards,


On 29/05/17 20:49, Nick Allen wrote:

Right now it is stored in HDFS and then retrieved with the
pcap_query tool.  The pcap_query tool can also be embedded in a
Zeppelin Notebook.  Of course with this data is in HDFS, you can
integrate with it using your tool of choice.

If you have use cases in mind, please feel free to share.

On Fri, May 26, 2017 at 6:47 AM, tkg_cangkul
mailto:yuza.ras...@gmail.com>> wrote:

hi,

i'm trying to using PCAP on metron. i'm using pycapa now and
i've success to store it into kafka and hdfs.
So, what's the main function of PCAP on metron? can i show it
to the dashboard? or it's just stored in hdfs only and read
it by CLI.?


Need Your Advice,


Best Regards,

Tkg_Cangkul

Re: PCAP on dashboard

2017-06-07 Thread tkg_cangkul 

yes i've try to run other MR job there and it's succeed.
there is no problem when i run other MR job on my cluster.

On 08/06/17 02:13, Nick Allen wrote:
Are you able to run any MR jobs in your environment?  You could test 
that by using one of the Hadoop examples, if you don't normally run 
anything else.



$ find /usr/hdp/ -name "*hadoop*example*.jar"
/usr/hdp/2.5.3.0-37/hadoop-mapreduce/hadoop-mapreduce-examples.jar

/usr/hdp/2.5.3.0-37/hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3.2.5.3.0-37.jar

$ yarn jar
/usr/hdp/2.5.3.0-37/hadoop-mapreduce/hadoop-mapreduce-examples.jar
pi 16 1000
Number of Maps  = 16
Samples per Map = 1000
Wrote input for Map #0
Wrote input for Map #1
...
Job Finished in 34.842 seconds
Estimated value of Pi is 3.1425


On Wed, Jun 7, 2017 at 1:00 PM, tkg_cangkul <mailto:yuza.ras...@gmail.com>> wrote:


I'm using metron 0.3.0 and i run on my cluster machines.
this is the command that i run :

/usr/metron/0.3.0/bin/pcap_query.sh query -st "20160617" -df
"MMdd" -bop "/apps/metron/out" --query "ip_src_addr ==
'10.1.80.220' and ip_src_port == '6667' and ip_dst_addr ==
'10.1.80.221' and ip_dst_port == '42885' and protocol == '6'"

i also have try to using pcap_query.sh with fixed option but i
still got the message Could not initialize class
java.net.NetworkInterface

Best Regards,

On 07/06/17 20:19, Nick Allen wrote:
You need to provide more information about your environment. 
What version of Metron?  How are you running Metron (metal, VM,

cloud)?  What command did you run?

On Wed, Jun 7, 2017 at 3:27 AM, tkg_cangkul
mailto:yuza.ras...@gmail.com>> wrote:

hi Nick,

thx for your reply.
Ok, so i can use Zeppelin for this pcap dashboard.
is there any way to connected the kibana to Zeppelin? i mean
is there any way to select the pcap data to the kibana ui if
i embed it in zeppelin?

By the way, i've tried to use pcap_query tool. but i've found
some error message below :



any suggestion for that?

Best Regards,


On 29/05/17 20:49, Nick Allen wrote:

Right now it is stored in HDFS and then retrieved with the
pcap_query tool.  The pcap_query tool can also be embedded
in a Zeppelin Notebook.  Of course with this data is in
HDFS, you can integrate with it using your tool of choice.

If you have use cases in mind, please feel free to share.

On Fri, May 26, 2017 at 6:47 AM, tkg_cangkul
mailto:yuza.ras...@gmail.com>> wrote:

hi,

i'm trying to using PCAP on metron. i'm using pycapa now
and i've success to store it into kafka and hdfs.
So, what's the main function of PCAP on metron? can i
    show it to the dashboard? or it's just stored in hdfs
only and read it by CLI.?


Need Your Advice,


Best Regards,

Tkg_Cangkul

integrate Pcap with Zeppelin Notebook

2017-06-08 Thread tkg_cangkul 

hi,

is it possible to integrate pcap with Zeppelin Notebook on metron 0.3.0 ?
if possible, how can i integrate it?
i've read on the github repository and i found if zeppelin script is 
build on metron 0.4.0.


Thanks before for your help.


Best Regards,

tkg_cangkul

indexing topology stop emitted after few minutes

2017-09-11 Thread tkg_cangkul 

Hi,

i wanna asking a question about indexing topology on metron.
when indexing topology running, it always stopped emitted data after a 
few minutes or few hours.

and then sometimes it can be continue emitted data.
I've already check on enrichment and parser topology but all of them 
stills emitted data normally.


Any suggestion for this pls?
is there any configuration that i must set for indexing topology or my 
elasticsearch?

Pls help.


Best Regards,

Re: Installation Issues

2017-09-26 Thread tkg_cangkul 

Maybe you can check the ambari-agent status first from the terminal.
If the service stopped. just start it then you can check the ambari again.

On 27/09/17 13:16, Syed Hammad Tahir wrote:
This is what I see when I login into ambari. How do I check where 
cluster deployment failed?


Inline image 1

On Wed, Sep 27, 2017 at 10:54 AM, Aaron Harris 
mailto:aaron.s.har...@outlook.com>> wrote:


Syed,


Have you checked if Ambari is running on the node? And if it is
can you login and check what part the cluster deploy failed at.


Regards,

Aaron


From: Syed Hammad Tahir
Sent: Wednesday, 27 September, 06:28
Subject: Installation Issues
To: user@metron.apache.org 
Cc: Muhammad Umar Janjua


Ok, Re-did every thing again and got this error. This time on 12
GB RAM

Will try on 16GB ram next time but is it actually related to RAM?

Re: Installation Issues

2017-09-26 Thread tkg_cangkul 

Maybe you can check the ambari-agent service first from the terminal.
If it stopped, just start it manually and then you can check the ambari 
again.


On 27/09/17 13:16, Syed Hammad Tahir wrote:
This is what I see when I login into ambari. How do I check where 
cluster deployment failed?


Inline image 1

On Wed, Sep 27, 2017 at 10:54 AM, Aaron Harris 
mailto:aaron.s.har...@outlook.com>> wrote:


Syed,


Have you checked if Ambari is running on the node? And if it is
can you login and check what part the cluster deploy failed at.


Regards,

Aaron


From: Syed Hammad Tahir
Sent: Wednesday, 27 September, 06:28
Subject: Installation Issues
To: user@metron.apache.org 
Cc: Muhammad Umar Janjua


Ok, Re-did every thing again and got this error. This time on 12
GB RAM

Will try on 16GB ram next time but is it actually related to RAM?

Re: Installation Issues

2017-09-27 Thread tkg_cangkul 
what alert that you see on ambari? there are 24 alert on your screenshot 
below.


On 27/09/17 13:50, Syed Hammad Tahir wrote:

Ambari server and agent both are running

On Wed, Sep 27, 2017 at 11:49 AM, tkg_cangkul <mailto:yuza.ras...@gmail.com>> wrote:


Maybe you can check the ambari-agent service first from the terminal.
If it stopped, just start it manually and then you can check the
ambari again.

On 27/09/17 13:16, Syed Hammad Tahir wrote:

This is what I see when I login into ambari. How do I check where
cluster deployment failed?

Inline image 1

On Wed, Sep 27, 2017 at 10:54 AM, Aaron Harris
mailto:aaron.s.har...@outlook.com>>
wrote:

Syed,


Have you checked if Ambari is running on the node? And if it
is can you login and check what part the cluster deploy
failed at.


Regards,

Aaron


From: Syed Hammad Tahir
Sent: Wednesday, 27 September, 06:28
Subject: Installation Issues
To: user@metron.apache.org <mailto:user@metron.apache.org>
Cc: Muhammad Umar Janjua


Ok, Re-did every thing again and got this error. This time on
12 GB RAM

Will try on 16GB ram next time but is it actually related to RAM?

Re: Installation Issues

2017-09-27 Thread tkg_cangkul 
maybe you can capture one of the alert? or maybe you can check the logs 
on /var/logs/ambari-server/


On 27/09/17 15:06, Syed Hammad Tahir wrote:

yes, which one should I pursue in order to find the issue?

On Wed, Sep 27, 2017 at 12:50 PM, tkg_cangkul <mailto:yuza.ras...@gmail.com>> wrote:


what alert that you see on ambari? there are 24 alert on your
screenshot below.


On 27/09/17 13:50, Syed Hammad Tahir wrote:

Ambari server and agent both are running

On Wed, Sep 27, 2017 at 11:49 AM, tkg_cangkul
mailto:yuza.ras...@gmail.com>> wrote:

Maybe you can check the ambari-agent service first from the
terminal.
If it stopped, just start it manually and then you can check
the ambari again.

On 27/09/17 13:16, Syed Hammad Tahir wrote:

This is what I see when I login into ambari. How do I check
where cluster deployment failed?

Inline image 1

On Wed, Sep 27, 2017 at 10:54 AM, Aaron Harris
mailto:aaron.s.har...@outlook.com>> wrote:

Syed,


Have you checked if Ambari is running on the node? And
if it is can you login and check what part the cluster
deploy failed at.


Regards,

Aaron


From: Syed Hammad Tahir
Sent: Wednesday, 27 September, 06:28
Subject: Installation Issues
To: user@metron.apache.org <mailto:user@metron.apache.org>
Cc: Muhammad Umar Janjua


Ok, Re-did every thing again and got this error. This
time on 12 GB RAM

Will try on 16GB ram next time but is it actually
related to RAM?

Re: Installation Issues

2017-09-27 Thread tkg_cangkul 

what about the ambari-server logs? have you check it?

On 27/09/17 15:13, Syed Hammad Tahir wrote:

Here is the output of platform-info.sh


Metron Version 0.4.1

ansible 2.0.0.2
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
--
Vagrant 1.9.1
--
Python 2.7.12
--
Apache Maven 3.3.9
Maven home: /usr/share/maven
Java version: 1.8.0_144, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-oracle/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.10.0-35-generic", arch: "amd64", family: 
"unix"

--
Docker version 1.12.6, build 78d1802
--
node
./platform-info.sh: line 69: node: command not found
--
npm
./platform-info.sh: line 74: npm: command not found
--
Linux everyone 4.10.0-35-generic #39~16.04.1-Ubuntu SMP Wed Sep 13 
09:02:42 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

--
Total System Memory = 7946.98 MB
Processor Model: Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz
Processor Speed: 3158.087 MHz
Processor Speed: 3114.001 MHz
Processor Speed: 2981.933 MHz
Processor Speed: 2458.770 MHz
Total Physical Processors: 4
Total cores: 16
Disk information:
/dev/sda1   268G   21G  234G   9% /
This CPU appears to support virtualization


On Wed, Sep 27, 2017 at 1:06 PM, Syed Hammad Tahir 
mailto:mscs16...@itu.edu.pk>> wrote:


yes, which one should I pursue in order to find the issue?

    On Wed, Sep 27, 2017 at 12:50 PM, tkg_cangkul
mailto:yuza.ras...@gmail.com>> wrote:

what alert that you see on ambari? there are 24 alert on your
screenshot below.


On 27/09/17 13:50, Syed Hammad Tahir wrote:

Ambari server and agent both are running

    On Wed, Sep 27, 2017 at 11:49 AM, tkg_cangkul
mailto:yuza.ras...@gmail.com>> wrote:

Maybe you can check the ambari-agent service first from
the terminal.
If it stopped, just start it manually and then you can
check the ambari again.

On 27/09/17 13:16, Syed Hammad Tahir wrote:

This is what I see when I login into ambari. How do I
check where cluster deployment failed?

Inline image 1

On Wed, Sep 27, 2017 at 10:54 AM, Aaron Harris
mailto:aaron.s.har...@outlook.com>> wrote:

Syed,


Have you checked if Ambari is running on the node?
And if it is can you login and check what part the
cluster deploy failed at.


Regards,

Aaron


From: Syed Hammad Tahir
Sent: Wednesday, 27 September, 06:28
Subject: Installation Issues
To: user@metron.apache.org
<mailto:user@metron.apache.org>
Cc: Muhammad Umar Janjua


Ok, Re-did every thing again and got this error.
This time on 12 GB RAM

Will try on 16GB ram next time but is it actually
related to RAM?

SSL error when using vagrant on metron 0.4.0

2017-10-02 Thread tkg_cangkul 

HI, i'm trying to using vagrant deployment on metron 0.4.0.
But when i execute command *vagrant up *, there is an error message 
below when downloading the box :


/*An error occurred while downloading the remote file. The error*//*
*//*message, if any, is reproduced below. Please fix this error and try*//*
*//*again.*//*
*//*
*//*SSL read: error::lib(0):func(0):reason(0), errno 104*/

Any suggest for this pls?

Best Regards,

metron dashboard timeout when loads many data

2017-10-09 Thread tkg_cangkul 

Hi,

anyone have experienced with query heavy data on metron dashboard?
i have 30Gb data. But when i try to load it all with metron dashboard in 
kibana, i have an error msg like below:


*Request Timeout after 3ms

*and then after that i've got this error msg too :

*Caused by: EsRejectedExecutionException[rejected execution of 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler@eba98f2 
on EsThreadPoolExecutor[search, queue capacity = 100, 
org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@52fd7ae9[Running, 
pool size = 20, active threads = 20, queued tasks = 100, completed tasks 
= 192656]]]


*For your information,
my ES heap is 10Gb
with 3 master and 4 datanodes.

Pls advice,

Best Regards,

Re: metron dashboard timeout when loads many data

2017-10-10 Thread tkg_cangkul 

Hi James,

Thanks for your reply,
This is the output of the API's :





For GET /_cluster/allocation/ API, there is an error like below :



For your information, I'm using Elasticsearch 2.3.3

On 10/10/17 23:49, James Sirota wrote:
I suspect your Elasticsearch may be in a bad state. If you are using 
Chrome, can you download the sense plugin and then run the following 
commands:

GET /_cluster/health?pretty
GET _cat/pending_tasks?v
GET /_cat/nodes?v
GET /_cluster/allocation/
And paste the output here?
Thanks,
James


09.10.2017, 22:25, "tkg_cangkul" :

Hi,

anyone have experienced with query heavy data on metron dashboard?
i have 30Gb data. But when i try to load it all with metron dashboard 
in kibana, i have an error msg like below:


*Request Timeout after 3ms

*and then after that i've got this error msg too :

*Caused by: EsRejectedExecutionException[rejected execution of 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler@eba98f2 
on EsThreadPoolExecutor[search, queue capacity = 100, 
org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@52fd7ae9[Running, 
pool size = 20, active threads = 20, queued tasks = 100, completed 
tasks = 192656]]]


*For your information,
my ES heap is 10Gb
with 3 master and 4 datanodes.

Pls advice,

Best Regards,




---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org

Re: metron dashboard timeout when loads many data

2017-10-11 Thread tkg_cangkul 
First time i install metron on cluster, i just only set one master node. 
But there are many problems when recovery starts. after i restart the ES 
cluster, sometimes there are some nodes that disconnected randomly from 
cluster when recovery proccess. When i set 3 master nodes, the 
disconnected node problem before was solved.


On 12/10/17 08:13, James Sirota wrote:
Your problem is that all your master nodes are also your data nodes. 
For such a small cluster you have to make one master node and the rest 
of the nodes would be data nodes. Ideally you don't want to colocate 
master nodes and data nodes. If you fix that I think it should work



10.10.2017, 19:50, "tkg_cangkul" :

Hi James,

Thanks for your reply,
This is the output of the API's :





For GET /_cluster/allocation/ API, there is an error like below :



For your information, I'm using Elasticsearch 2.3.3

On 10/10/17 23:49, James Sirota wrote:
I suspect your Elasticsearch may be in a bad state. If you are using 
Chrome, can you download the sense plugin and then run the following 
commands:

GET /_cluster/health?pretty
GET _cat/pending_tasks?v
GET /_cat/nodes?v
GET /_cluster/allocation/
And paste the output here?
Thanks,
James


09.10.2017, 22:25, "tkg_cangkul"  
<mailto:yuza.ras...@gmail.com>:

Hi,

anyone have experienced with query heavy data on metron dashboard?
i have 30Gb data. But when i try to load it all with metron 
dashboard in kibana, i have an error msg like below:


*Request Timeout after 3ms

*and then after that i've got this error msg too :

*Caused by: EsRejectedExecutionException[rejected execution of 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler@eba98f2 
on EsThreadPoolExecutor[search, queue capacity = 100, 
org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@52fd7ae9[Running, 
pool size = 20, active threads = 20, queued tasks = 100, completed 
tasks = 192656]]]


*For your information,
my ES heap is 10Gb
with 3 master and 4 datanodes.

Pls advice,

Best Regards,




---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org






---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org

event correlation on metron

2017-10-16 Thread tkg_cangkul 

hi,

anyone could explain me about event correlation using apache metron?
does metron support event correlation?


Pls Advice

Re: event correlation on metron

2017-10-17 Thread tkg_cangkul 

for example,

i wanna try to correlate between logs.
how many times user A have login failed and how many times user A have 
login succeed. include detail IP, timestamp etc.

is this possible to do with metron?



On 17/10/17 02:56, James Sirota wrote:

What specifically are you looking to correlate?  Can you talk a little more 
about your use case?

16.10.2017, 02:23, "tkg_cangkul" :

hi,

anyone could explain me about event correlation using apache metron?
does metron support event correlation?

Pls Advice

---
Thank you,

James Sirota
PMC- Apache Metron
jsirota AT apache DOT org

Re: metron dashboard timeout when loads many data

2017-10-19 Thread tkg_cangkul 

Hi James,

I've try your suggestion about this but it doesn't works.
i've got msg master_not_discovered_exception oftenly.

On 17/10/17 02:48, James Sirota wrote:

that is correct. please let us know if that works for you


12.10.2017, 10:14, "Youzha" :
so i must restart the datanodes one by one. and set only one master 
node right?
Ok i got it. thank you so much James for your explanation. i will try 
it soon.


On Thu, 12 Oct 2017 at 21.55 James Sirota <mailto:jsir...@apache.org>> wrote:


You have to restart the ES cluster in a rolling fashion. Meaning
restart one data node, then the other, then the other, etc. If
you restart them all at once, this will happen


    11.10.2017, 19:37, "tkg_cangkul" mailto:yuza.ras...@gmail.com>>:

First time i install metron on cluster, i just only set one
master node. But there are many problems when recovery starts.
after i restart the ES cluster, sometimes there are some nodes
that disconnected randomly from cluster when recovery proccess.
When i set 3 master nodes, the disconnected node problem before
was solved.

On 12/10/17 08:13, James Sirota wrote:

Your problem is that all your master nodes are also your data
nodes. For such a small cluster you have to make one master
node and the rest of the nodes would be data nodes. Ideally you
don't want to colocate master nodes and data nodes. If you fix
    that I think it should work


10.10.2017, 19:50, "tkg_cangkul" 
<mailto:yuza.ras...@gmail.com>:

Hi James,

Thanks for your reply,
This is the output of the API's :





For GET /_cluster/allocation/ API, there is an error like below :



For your information, I'm using Elasticsearch 2.3.3

On 10/10/17 23:49, James Sirota wrote:

I suspect your Elasticsearch may be in a bad state. If you
are using Chrome, can you download the sense plugin and then
run the following commands:
GET /_cluster/health?pretty
GET _cat/pending_tasks?v
GET /_cat/nodes?v
GET /_cluster/allocation/
And paste the output here?
Thanks,
James


09.10.2017, 22:25, "tkg_cangkul" 
<mailto:yuza.ras...@gmail.com>:

Hi,

anyone have experienced with query heavy data on metron
dashboard?
i have 30Gb data. But when i try to load it all with metron
dashboard in kibana, i have an error msg like below:

*Request Timeout after 3ms

*and then after that i've got this error msg too :

*Caused by: EsRejectedExecutionException[rejected execution
of

org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler@eba98f2
on EsThreadPoolExecutor[search, queue capacity = 100,

org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@52fd7ae9[Running,
pool size = 20, active threads = 20, queued tasks = 100,
completed tasks = 192656]]]

*For your information,
my ES heap is 10Gb
with 3 master and 4 datanodes.

Pls advice,

Best Regards,




---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org






---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org






---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org




---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org

profiler logs

2017-10-20 Thread tkg_cangkul 

HI,

is there any logs for profiler feature on metron?
i'm looking for it because i have problem with profiler. there is no 
emitted data on storm topology.

Re: multiple pattern grok parser in 1 file

2017-10-22 Thread tkg_cangkul 

Hi Wasim,

thx for your reply.
So it means i should use logstash parser for metron?
Is there any documentation about use logstash parser for metron?
I didn't found any documentation about that on metron.
i just find logstash basic parser but there is no documentation about that.



On 23/10/17 10:33, Wasim Halani wrote:

Hi Youzha,

It should be possible to add multiple patterns in a single config 
file. For reference, you can check out the use of multiple patterns in 
a repo I maintain [1].

You would find the patterns in [2] useful for your use-case.

However, do note that there is a cost to every grok failure [3] - so 
you need to ensure that your most common event patterns are at the top 
of the list.


As a side-note, if you have any logstash parsers which are not 
available in the repo, please feel to submit a PR to [4]



[1] 
https://bitbucket.org/networkintelligence/logstash-configs/raw/aae3d61bb6c53beb0678536e2e9b33d7996e2960/cisco-asa.conf
[2] 
https://bitbucket.org/networkintelligence/logstash-configs/raw/aae3d61bb6c53beb0678536e2e9b33d7996e2960/linux-system.conf

[3] https://www.elastic.co/blog/do-you-grok-grok
[4] https://bitbucket.org/networkintelligence/logstash-configs/

Regards,
---
Wasim Halani
http://twitter.com/washalsec
http://securitythoughts.wordpress.com
--
To keep silent when you can say something wise and useful is as bad as 
keeping on propagating foolish and unwise thoughts. -- Imam Ali (p.b.u.h.)


On Mon, Oct 23, 2017 at 8:08 AM, Youzha > wrote:


Hi, is that possible to using multiple pattern grok parser ini 1
pattern file?
i’m trying to parsing authlog file in /var/log/secure into metron.
the problem is there are different structures of logs inside
/var/log/secure. any suggest for this pls?


Best Regards,

Re: multiple pattern grok parser in 1 file

2017-10-23 Thread tkg_cangkul 

FYI,

i've trying to using Grok parser metron with multiple pattern in single 
file but it doesn't work. this is my sample grok pattern on 
/apps/metron/patterns/authlog :


AUTHLOG %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
%{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} password for 
%{USERNAME:username} from %{IP:ip} %{GREEDYDATA}
AUTHLOG %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
%{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} closed for user 
%{USERNAME:username}


When the sensor started, the second grok pattern doesn't work. Only 
first pattern works.

There is an error message like this on storm logs:

Caused by: java.lang.RuntimeException: Grok statement produced a null message.



On 23/10/17 10:49, tkg_cangkul wrote:

Hi Wasim,

thx for your reply.
So it means i should use logstash parser for metron?
Is there any documentation about use logstash parser for metron?
I didn't found any documentation about that on metron.
i just find logstash basic parser but there is no documentation about 
that.




On 23/10/17 10:33, Wasim Halani wrote:

Hi Youzha,

It should be possible to add multiple patterns in a single config 
file. For reference, you can check out the use of multiple patterns 
in a repo I maintain [1].

You would find the patterns in [2] useful for your use-case.

However, do note that there is a cost to every grok failure [3] - so 
you need to ensure that your most common event patterns are at the 
top of the list.


As a side-note, if you have any logstash parsers which are not 
available in the repo, please feel to submit a PR to [4]



[1] 
https://bitbucket.org/networkintelligence/logstash-configs/raw/aae3d61bb6c53beb0678536e2e9b33d7996e2960/cisco-asa.conf
[2] 
https://bitbucket.org/networkintelligence/logstash-configs/raw/aae3d61bb6c53beb0678536e2e9b33d7996e2960/linux-system.conf

[3] https://www.elastic.co/blog/do-you-grok-grok
[4] https://bitbucket.org/networkintelligence/logstash-configs/

Regards,
---
Wasim Halani
http://twitter.com/washalsec
http://securitythoughts.wordpress.com
--
To keep silent when you can say something wise and useful is as bad 
as keeping on propagating foolish and unwise thoughts. -- Imam Ali 
(p.b.u.h.)


On Mon, Oct 23, 2017 at 8:08 AM, Youzha <mailto:yuza.ras...@gmail.com>> wrote:


Hi, is that possible to using multiple pattern grok parser ini 1
pattern file?
i’m trying to parsing authlog file in /var/log/secure into
metron. the problem is there are different structures of logs
inside /var/log/secure. any suggest for this pls?


Best Regards,

Re: multiple pattern grok parser in 1 file

2017-10-23 Thread tkg_cangkul 

Hi Simon,

I've tried your suggestion but i have an error msg like below :



On 23/10/17 16:22, Simon Elliston Ball wrote:

That is not valid grok. Pattern names should be unique in the grok.

What you probably mean is something like:

AUTHLOG1 %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
%{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} password for 
%{USERNAME:username} from %{IP:ip} %{GREEDYDATA}
AUTHLOG2 %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
%{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} closed for user 
%{USERNAME:username}
AUTHLOG (?:%{AUTHLOG1}|%{AUTHLOG2})

Simon


On 23 Oct 2017, at 08:53, tkg_cangkul  wrote:

FYI,

i've trying to using Grok parser metron with multiple pattern in single file 
but it doesn't work. this is my sample grok pattern on 
/apps/metron/patterns/authlog :

AUTHLOG %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
%{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} password for 
%{USERNAME:username} from %{IP:ip} %{GREEDYDATA}
AUTHLOG %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
%{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} closed for user 
%{USERNAME:username}

When the sensor started, the second grok pattern doesn't work. Only first 
pattern works.
There is an error message like this on storm logs:

Caused by: java.lang.RuntimeException: Grok statement produced a null message.


On 23/10/17 10:49, tkg_cangkul wrote:

Hi Wasim,

thx for your reply.
So it means i should use logstash parser for metron?
Is there any documentation about use logstash parser for metron?
I didn't found any documentation about that on metron.
i just find logstash basic parser but there is no documentation about that.



On 23/10/17 10:33, Wasim Halani wrote:

Hi Youzha,

It should be possible to add multiple patterns in a single config file. For 
reference, you can check out the use of multiple patterns in a repo I maintain 
[1].
You would find the patterns in [2] useful for your use-case.

However, do note that there is a cost to every grok failure [3] - so you need 
to ensure that your most common event patterns are at the top of the list.

As a side-note, if you have any logstash parsers which are not available in the 
repo, please feel to submit a PR to [4]


[1] 
https://bitbucket.org/networkintelligence/logstash-configs/raw/aae3d61bb6c53beb0678536e2e9b33d7996e2960/cisco-asa.conf
[2] 
https://bitbucket.org/networkintelligence/logstash-configs/raw/aae3d61bb6c53beb0678536e2e9b33d7996e2960/linux-system.conf
[3] https://www.elastic.co/blog/do-you-grok-grok
[4] https://bitbucket.org/networkintelligence/logstash-configs/

Regards,
---
Wasim Halani
http://twitter.com/washalsec
http://securitythoughts.wordpress.com
--
To keep silent when you can say something wise and useful is as bad as keeping 
on propagating foolish and unwise thoughts. -- Imam Ali (p.b.u.h.)

On Mon, Oct 23, 2017 at 8:08 AM, Youzha  wrote:
Hi, is that possible to using multiple pattern grok parser ini 1 pattern file?
i’m trying to parsing authlog file in /var/log/secure into metron. the problem 
is there are different structures of logs inside /var/log/secure. any suggest 
for this pls?


Best Regards,

ask about profiler rule

2017-10-23 Thread tkg_cangkul 

Hi,

anybody can explained to me this rule of profiler config please ?

   {
  "profile": "failed-logins",
  "foreach": "user.name ",
  "onlyif": "source.type == 'activedirectory' and event.type ==
   'failed_login'"
  "init": { "count": 0 },
  "update": { "count" : "count + 1" },
  "result": "count"
   }


what is "source.type == 'activedirectory' and event.type == 
'failed_login'" means?
does it means the profiler will read from ES index that have condition 
source.type == 'activedirectory' . if yes, so i must index to ES first 
where source type = activedirectory ?


I've just read on Nick article here :

https://www.slideshare.net/NickAllen4/apache-metron-profiler

In the other rules config there are "source.type == 'yaf'" , 
"source.type == 'bro'". What i know that "source.type == 'yaf'" & 
"source.type == 'bro'" have indexed by default on metron. how about 
activedirectory?



Best Regards,

Re: ask about profiler rule

2017-10-24 Thread tkg_cangkul 
Do you have any sample configuration or something like that to setup 
activedirectory sensor?

i've trying many ways but it stills not succeed.
that's because there are so many format log on there. i wanna get the 
login status (failed, success, logout, etc) with this profiler.

Is it possible to me include logstash into metron?

On 24/10/17 15:50, Mohan Venkateshaiah wrote:


Hi,

The Profiler will consume messages from the input kafka topic defined 
in the Profiler's configuration (see Configuring the Profiler 
<https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#configuring-the-profiler>). 
By default, this is the indexing topic.


Thanks

Mohan DV

*From: *Simon Elliston Ball 
*Reply-To: *"user@metron.apache.org" 
*Date: *Tuesday, October 24, 2017 at 2:02 PM
*To: *"user@metron.apache.org" 
*Subject: *Re: ask about profiler rule

The profiler reads direct from the ingest stream, so sees data before 
it gets to ES.


The onlyif config you are asking about is a filter condition, so only 
data which matches that expression will be considered by this 
particular profile.


The activedirectory example here assumes that you have a sensor setup 
from something like active directory, that has fields called user.name 
and event.type in. It will then count those failures per user.name.


Simon

On 24 Oct 2017, at 07:38, tkg_cangkul mailto:yuza.ras...@gmail.com>> wrote:

Hi,

anybody can explained to me this rule of profiler config please ?

{

"profile": "failed-logins",

"foreach": "user.name <http://user.name/>",

"onlyif": "source.type == 'activedirectory' and event.type ==
'failed_login'"

"init": { "count": 0 },

"update": { "count" : "count + 1" },

"result": "count"

}


what is "source.type == 'activedirectory' and event.type ==
'failed_login'" means?
does it means the profiler will read from ES index that have
condition source.type == 'activedirectory' . if yes, so i must
index to ES first where source type = activedirectory ?

I've just read on Nick article here :

https://www.slideshare.net/NickAllen4/apache-metron-profiler

In the other rules config there are  "source.type == 'yaf'" ,
"source.type == 'bro'". What i know that "source.type == 'yaf'" &
"source.type == 'bro'" have indexed by default on metron. how
about activedirectory?


Best Regards,

Re: ask about profiler rule

2017-10-24 Thread tkg_cangkul 

OK Nick,

I think i've succeed to do this.
i'm using logstash like what you suggest below and use JSONMap parser in 
Metron to parse it.


Thank You so much for your help.
Have a nice day :)

On 24/10/17 22:14, Nick Allen wrote:
> Do you have any sample configuration or something like that to setup 
activedirectory sensor?


I assuming you are not yet ingesting AD logs into Metron. There is not 
currently something out-of-the-box for AD logs, but it should not be 
too hard.  Feel free to contribute as many example AD logs as you can 
(after cleaning them of sensitive information) to either of these JIRAs.


https://issues.apache.org/jira/browse/METRON-1149
https://issues.apache.org/jira/browse/METRON-161


> i've trying many ways but it stills not succeed. that's because there 
are so many format log on there. i wanna get the login status (failed, 
success, logout, etc) with this profiler.


What have you tried?  I assume you are still talking about parsing the 
AD logs, which has nothing to do with the Profiler.


Just to level set, first step is to parse the AD logs and get them 
into Metron.  Then we can use that data in the Profiler.



> Is it possible to me include logstash into metron?

You can use Logstash to push data into Kafka.  Metron would then 
consume it from Kafka.





On Tue, Oct 24, 2017 at 4:59 AM, tkg_cangkul <mailto:yuza.ras...@gmail.com>> wrote:


Do you have any sample configuration or something like that to
setup activedirectory sensor?
i've trying many ways but it stills not succeed.
that's because there are so many format log on there. i wanna get
the login status (failed, success, logout, etc) with this profiler.
Is it possible to me include logstash into metron?


On 24/10/17 15:50, Mohan Venkateshaiah wrote:


Hi,

The Profiler will consume messages from the input kafka topic
defined in the Profiler's configuration (see Configuring the
Profiler

<https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#configuring-the-profiler>).
By default, this is the indexing topic.

Thanks

Mohan DV

*From: *Simon Elliston Ball 
<mailto:si...@simonellistonball.com>
*Reply-To: *"user@metron.apache.org"
<mailto:user@metron.apache.org> 
<mailto:user@metron.apache.org>
*Date: *Tuesday, October 24, 2017 at 2:02 PM
*To: *"user@metron.apache.org" <mailto:user@metron.apache.org>
 <mailto:user@metron.apache.org>
*Subject: *Re: ask about profiler rule

The profiler reads direct from the ingest stream, so sees data
before it gets to ES.

The onlyif config you are asking about is a filter condition, so
only data which matches that expression will be considered by
this particular profile.

The activedirectory example here assumes that you have a sensor
setup from something like active directory, that has fields
called user.name <http://user.name> and event.type in. It will
then count those failures per user.name <http://user.name>.

Simon

On 24 Oct 2017, at 07:38, tkg_cangkul mailto:yuza.ras...@gmail.com>> wrote:

Hi,

anybody can explained to me this rule of profiler config please ?

{

"profile": "failed-logins",

"foreach": "user.name <http://user.name/>",

"onlyif": "source.type == 'activedirectory' and
event.type == 'failed_login'"

"init": { "count": 0 },

"update": { "count" : "count + 1" },

"result": "count"

}


what is "source.type == 'activedirectory' and event.type ==
'failed_login'" means?
does it means the profiler will read from ES index that have
condition source.type == 'activedirectory' . if yes, so i
must index to ES first where source type = activedirectory ?

I've just read on Nick article here :

https://www.slideshare.net/NickAllen4/apache-metron-profiler
<https://www.slideshare.net/NickAllen4/apache-metron-profiler>

In the other rules config there are "source.type == 'yaf'" ,
"source.type == 'bro'". What i know that "source.type ==
'yaf'" & "source.type == 'bro'" have indexed by default on
metron. how about activedirectory?


Best Regards,

profiler on metron 0.3.x

2017-11-10 Thread tkg_cangkul 

Hi i'm trying to implement profiler on metron 0.3.x

but i've a problem when push config profiler.json into zookeeper
when i'm trying command :

/usr/metron/0.3.0/bin/zk_load_configs.sh -m DUMP -z node1:2181

there is no profiler config there.

Is metron 0.3.x support profiler function?

Pls advice.

[ask] upgrade metron

2019-01-01 Thread tkg_cangkul 

Hi all,

Does apache metron support upgrade version with patching file?
how if i want to upgrade the version without reinstall all components?

Pls advice.

Best Regards,
Tkg_cangkul

Re: [ask] upgrade metron

2019-01-01 Thread tkg_cangkul 

Is there a safe way to do an upgrade ?
do you have some advice to do this ?

Best Regards,
Tkg_cangkul

On 02/01/19 14:03, Pieter Baele wrote:

AFAIK currently not.
But with a bit planning (and testing), an upgrade is quite fast.

Sincerely
Pieter

On Wed, Jan 2, 2019 at 7:44 AM tkg_cangkul <mailto:yuza.ras...@gmail.com>> wrote:


Hi all,

Does apache metron support upgrade version with patching file?
how if i want to upgrade the version without reinstall all components?

Pls advice.

Best Regards,
Tkg_cangkul

what version metron on HCP 1.8.0

2019-01-21 Thread tkg_cangkul 

Hi,

I've downloaded hcp 1.8.0 mpack from this link :
https://docs.hortonworks.com/HDPDocuments/HCP1/HCP-1.8.0/release-notes/content/hcp_repositories.html

on hortonworks docs website, i've read if hcp 1.8.0 components is metron 
0.7.0.
but in mpack.json file of hcp 1.8.0, the metron version is 
service_version" : "0.6.0.1.8.0.0"


i've tried to install it on ambari and the stack version is metron 0.6.0

pls help.


Best Regards,

Tkg_Cangkul

use another geoIP db for enrichment

2019-04-01 Thread tkg_cangkul 

Hi,

Is there any ways to use another geoIP for metron.?
I wanna try to use another geoIP other than geolite.
if it's possible, pls give me some reference link to do this.


Best Regards,

Tkg_cangkul

Re: use another geoIP db for enrichment

2019-04-01 Thread tkg_cangkul 

Hi,

well actually i'm looking for free geoIP db.

Cheers,

Tkg_cangkul

On 01/04/19 22:51, Yerex, Tom wrote:

Good morning,

Does it have to be free or not?

Cheers,

Tom.
  


On 2019-04-01, 8:49 AM, "tkg_cangkul"  wrote:

 Hi,
 
 Is there any ways to use another geoIP for metron.?

 I wanna try to use another geoIP other than geolite.
 if it's possible, pls give me some reference link to do this.
 
 
 Best Regards,
     
 Tkg_cangkul

Re: use another geoIP db for enrichment

2019-04-01 Thread tkg_cangkul 

ah i see. so i just need to create the stellar function to do this.

Ok i'll try it.

Thanks a lot for your help nick

Best Regards,

Tkg_cangkul

On 01/04/19 23:08, Nick Allen wrote:
You would just have to create your own Stellar function that performs 
the geo-IP lookup using your alternative database.  The existing 
`GEO_GET` functionality is targeted specifically at the Maxmind database.


On Mon, Apr 1, 2019 at 11:49 AM tkg_cangkul <mailto:yuza.ras...@gmail.com>> wrote:


Hi,

Is there any ways to use another geoIP for metron.?
I wanna try to use another geoIP other than geolite.
if it's possible, pls give me some reference link to do this.


Best Regards,

Tkg_cangkul

Re: use another geoIP db for enrichment

2019-04-01 Thread tkg_cangkul 

Well Ok i'll try to create my own stellar function first .

Thanks a lot for your help Yerex :)

Best Regards,

Tkg_cangkul
On 01/04/19 23:13, Yerex, Tom wrote:

I don't know of any Geo IP that is free and provides better accuracy than geolite. There 
are some Geo IP sites that offer a certain number of requests for a period of time 
"free", which might provide better accuracy and cost less depending on your 
work load.

As an example: https://ipstack.com/product

You will need to write the functionality as Nick Allen mentioned in his 
response.

--Tom.

On 2019-04-01, 9:09 AM, "tkg_cangkul"  wrote:

 Hi,
 
 well actually i'm looking for free geoIP db.
 
 Cheers,
 
 Tkg_cangkul
 
 On 01/04/19 22:51, Yerex, Tom wrote:

 > Good morning,
 >
 > Does it have to be free or not?
 >
 > Cheers,
 >
 > Tom.
 >
 >
 > On 2019-04-01, 8:49 AM, "tkg_cangkul"  wrote:
 >
 >  Hi,
 >
 >  Is there any ways to use another geoIP for metron.?
 >  I wanna try to use another geoIP other than geolite.
 >  if it's possible, pls give me some reference link to do this.
 >
 >
 >  Best Regards,
 >
 >  Tkg_cangkul
 >

[ask] detect unsual login duration

2019-05-15 Thread tkg_cangkul 

Hi,

Does metron support to do detection an unusual login duration?

For example.
IP A login for 3 days without logout. then metron will give some alert 
to us.


If this possible, how to do that?
Pls help.


Best Regards,

Tkg_cangkul

Re: [ask] detect unsual login duration

2019-05-16 Thread tkg_cangkul 

Hi Simmon,

Thanks for your response,
Could you explain about *add some sort of state table and a triggered* ?
Is that the hbase table?
For your information, i don't want to wait for the logout from the user.

Best Regards,

Tkg_cangkul

On 16/05/19 15:59, Simon Elliston Ball wrote:
You could pull that out in a report in zeppelin easily enough, but to 
do it real-time we would need to add some sort of state table and a 
triggered check of that state, unless you say wanted to alert only on 
logout (I’m assuming you don’t want to wait for the logout, but alerts 
after some fixed duration or better some anomalous duration?)


Is that the sort of use case?

Simon

On Thu, 16 May 2019 at 03:59, tkg_cangkul <mailto:yuza.ras...@gmail.com>> wrote:


Hi,

Does metron support to do detection an unusual login duration?

For example.
IP A login for 3 days without logout. then metron will give some
alert
to us.

If this possible, how to do that?
Pls help.


Best Regards,

    Tkg_cangkul

--
--
simon elliston ball
@sireb

[ask] problem about hbase profiler

2019-07-01 Thread tkg_cangkul 
Hi, i've a problem about hbase profiler on metron. i've found if there 
are some inconsistentcy data that insert to  hbase. sometimes insert 
normally but sometimes doesn't inserted. if not inserted, there are an 
error msg on storm (rebalance... Max poll()).


I've set 2 worker for profiler. profiler.hbase.batch 10. 
profiler.hbase.flush.interval.seconds 30


any suggest about this? pls help

Re: [ask] problem about hbase profiler

2019-07-02 Thread tkg_cangkul 

Hi nick,

Please find attachment for my profiler.properties file

   *Start with 1 worker and increase the number of executors first*

What executor you mean here? in my profiler properties attached, i've 
set profiler.worker = 1 & profiler.executors = 15
I've try to increase the executor component from rebalance command like 
below :


*storm rebalance profiler -n 1 -e splitterBolt=3 -e hbaseBolt=3*

when i check it after rebalance process finish, splitterBolt and 
hbaseBolt executor doesn't change. it still have 1 executor.
for your information. the data that proccessed is about 1000 rows in 5 
seconds


any suggestion about this pls?

On 01/07/19 19:51, Nick Allen wrote:
I would assume that in those cases where you see the "rebalance... max 
poll()" message that topology is unable to keep up with the input 
throughput.  The messages are not ack'd quickly enough, they fail, and 
are retried.  I would not focus on HBase because that is not likely 
your bottleneck.


You need to tune the Profiler topology to keep up with your peak 
incoming throughput.  The same ideas for tuning any Storm topology 
apply here.  Start with 1 worker and increase the number of executors 
first.  You will also want to explore reducing your window lag and 
tuning other parameters before increasing the number of workers.  Try 
to max out the performance of a single worker before adding more workers.


You will probably first start to see performance issues on the 
Splitter bolt that has to consume every message and determine if that 
message is needed by any of the profilers.  Using the Storm UI watch 
the metrics generated for that bolt first.


If asking for more in-depth help, these are the types of questions 
that I would ask.


  * What is your peak input throughput to the Profiler?
  * What are your Profiler properties?
  * How many profiles do you have and what are they doing? Provide the
profile definitions.






On Mon, Jul 1, 2019 at 3:04 AM tkg_cangkul <mailto:yuza.ras...@gmail.com>> wrote:


Hi, i've a problem about hbase profiler on metron. i've found if
there
are some inconsistentcy data that insert to  hbase. sometimes insert
normally but sometimes doesn't inserted. if not inserted, there
are an
error msg on storm (rebalance... Max poll()).

I've set 2 worker for profiler. profiler.hbase.batch 10.
profiler.hbase.flush.interval.seconds 30

any suggest about this? pls help



#
#
#  Licensed to the Apache Software Foundation (ASF) under one
#  or more contributor license agreements.  See the NOTICE file
#  distributed with this work for additional information
#  regarding copyright ownership.  The ASF licenses this file
#  to you under the Apache License, Version 2.0 (the
#  "License"); you may not use this file except in compliance
#  with the License.  You may obtain a copy of the License at
#
#  http://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
#  limitations under the License.
#
#

# Storm #

topology.worker.childopts=
topology.auto-credentials=[]
profiler.workers=1
profiler.executors=15
topology.message.timeout.secs=1500
topology.max.spout.pending=1000
topology.fall.back.on.java.serialization=true
topology.testing.always.try.serialize=false
topology.kryo.register=[ org.apache.metron.profiler.ProfileMeasurement, \
org.apache.metron.profiler.ProfilePeriod, \
org.apache.metroncommon.configuration.profiler.ProfileResult, \
org.apache.metroncommon.configuration.profiler.ProfileResultExpressions, \
org.apache.metroncommon.configuration.profiler.ProfileTriageExpressions, \
org.apache.metroncommon.configuration.profiler.ProfilerConfig, \
org.apache.metroncommon.configuration.profiler.ProfileConfig, \
org.json.simple.JSONObject, \
org.json.simple.JSONArray, \
java.util.LinkedHashMap, \
org.apache.metron.statistics.OnlineStatisticsProvider ]

# Profiler #

profiler.input.topic=indexing
profiler.output.topic=enrichments
profiler.period.duration=10
profiler.period.duration.units=MINUTES
profiler.window.duration=30
profiler.window.duration.units=SECONDS
profiler.ttl=30
profiler.ttl.units=MINUTES
profiler.window.lag=1
profiler.window.lag.units=MINUTES
profiler.max.routes.per.bolt=10

# HBase #

profiler.hbase.salt.divisor=1000
profiler.hbase.table=profiler
profiler.hbase.column.family=P
profiler.hbase.batch=10
profiler.hbase.flush.interval.seconds=30

# Kafka #

kafka.zk=zookeeper3.metron.com:2181,zookeeper1.metron.com:2181,zookeeper2.metron.com:2181
kafka.broker=dn1.metron.com:6667,dn3.metron.com:6667,dn2.metron.com:6667
kafka.start=UNCOMMITTED_EARLIEST

Re: Invite for Merton slack channel

2019-07-08 Thread tkg_cangkul 

could you invite me too please?

On 08/07/19 23:05, zeo...@gmail.com wrote:

You got it.

- Jon Zeolla
zeo...@gmail.com


On Mon, Jul 8, 2019 at 10:15 AM David Auclair > wrote:


Could I also get an invite please?

Thanks in advance,

Dave

*From:*zeo...@gmail.com mailto:zeo...@gmail.com>>
*Sent:* July 8, 2019 9:30 AM
*To:* Srikanth Nagarajan mailto:s...@gandivanetworks.com>>
*Cc:* user@metron.apache.org 
*Subject:* Re: Invite for Merton slack channel

Done


- Jon Zeolla
zeo...@gmail.com 

On Mon, Jul 8, 2019 at 9:18 AM Srikanth Nagarajan
mailto:s...@gandivanetworks.com>> wrote:

Hi

I would appreciate an invite to the Metron slack channel .

Thank you

Srikanth

__
*Srikanth Nagarajan *
Principal
*Gandiva Networks Inc*
*732.690.1884 * Mobile
s...@gandivanetworks.com 
www.gandivanetworks.com

Re: tuning search query on alert UI

2019-09-04 Thread tkg_cangkul 

Hi James.

I'm using ES to index the data.

On 04/09/19 00:46, James Sirota wrote:
Are you using Solr or ES?  there is a different process based on the 
indexer used.



14.08.2019, 09:18, "Youzha" :

Hi,

is there any ways to optimize search query on Alert UI ?

i try to query all data on my alert UI but the proccess run too
slow. especially on my first execute search button. sometimes i
had “request time out” from the response.

pls advice,

Best Regards,

tkg_cangkul



---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org