Re: Metron HBase conditional enrichment

2017-05-25 Thread Nick Allen
> everywhere I can use Stellar DSL, all of the functions have been implemented and ready to use? Generally, yes, you are right. I vaguely remember a couple instances of functions that are useful in the REPL only, but I cannot remember what those are right now. Hopefully we have those doc'd appro

Re: Metron HBase conditional enrichment

2017-05-25 Thread Otto Fowler
I think most of those restricted functions are in the metron-managment section. On May 25, 2017 at 07:27:24, Nick Allen (n...@nickallen.org) wrote: > everywhere I can use Stellar DSL, all of the functions have been implemented and ready to use? Generally, yes, you are right. I vaguely remember

AWS deployment with 5 hosts.

2017-05-25 Thread Laurens Vets
Deploying the standard 10 instance setup works. However, for our current needs, 10 m4.xlarge instances seem overkill and we want to deploy Metron on only 5 hosts for now. I would think that editing metron/metron-deployment/amazon-ec2/playbook.yml would be enough. I changed the following:

Re: Metron HBase conditional enrichment

2017-05-25 Thread Matt Foley
Hi Ali, When writing Stellar statements, it is convenient to test them out in the REPL, which can be invoked via some variant of the commands in https://github.com/apache/metron/blob/master/metron-platform/metron-common/src/main/scripts/stellar , depending on the particular environment you’re wo

Re: AWS deployment with 5 hosts.

2017-05-25 Thread Nick Allen
What host groups did ec2-34-210-245-155.us-west-2.compute.amazonaws.com get assigned to? You can look at the labels that get attached to the host in the EC2 web interface. I have a feeling it has something to do with the Ambari blueprint that is used (see metron-deployment/roles/ambari_config/var

Re: Metron HBase conditional enrichment

2017-05-25 Thread Nick Allen
Thanks, Matt. That command-line doesn't work for me; even after updating version number. Not sure why. This is what I tend to run after a build and install of Metron. Maybe something is different in my environment. mvn clean install -DskipTests mvn exec:java -Dexec.mainClass="org.apache.metr

Re: AWS deployment with 5 hosts.

2017-05-25 Thread Laurens Vets
I'm not quite following. Unfortunately, I removed the previous ec2 instances and started again to test. Same error for host ec2-52-40-75-137.us-west-2.compute.amazonaws.com with the following configuration: - include: tasks/create-hosts.yml host_count=1 host_type=sensors,ambari_master,ec2,mon

Re: Metron HBase conditional enrichment

2017-05-25 Thread Nick Allen
Correct me if I am wrong, Matt, but I believe that changing the project that you pass to the -pl switch will allow you to see exactly what Stellar functions would be available in each topology. You just have to refer to whichever project drives the topology. This might help answer Ali's previous

Re: Metron HBase conditional enrichment

2017-05-25 Thread Matt Foley
Hmm. I just refreshed a current copy of master, ran “mvn clean install –DskipTests”, and found: java -cp metron-platform/metron-common/target/metron-common-0.4.0.jar org.apache.metron.common.stellar.shell.StellarShell fails with “ClassNotFoundException: org.slf4j.LoggerFactory”, b

Re: Metron HBase conditional enrichment

2017-05-25 Thread Matt Foley
Nick is correct that in any given environment, only Stellar functions defined in jars on the current classpath will be available. When running with the maven exec:java plugin, as below, this means only jars declared (or transitively required) as dependencies to the given project. In the install

Re: Metron HBase conditional enrichment

2017-05-25 Thread Nick Allen
Each topology has its own uber-jar that is built from all of it's dependencies. It's classpath is basically whatever is in the uber-jar. That's why running with -pl against the project from which the uber jar is built should identify the Stellar functions available to each topology. When running

Re: Metron HBase conditional enrichment

2017-05-25 Thread Matt Foley
That sheds light. Thanks, Nick. --Matt From: Nick Allen Reply-To: "user@metron.apache.org" Date: Thursday, May 25, 2017 at 3:06 PM To: "user@metron.apache.org" Subject: Re: Metron HBase conditional enrichment Each topology has its own uber-jar that is built from all of it's dependencies. It