AWS is a good option in that if you have the money for the resources to spare, it's a nice way to spin up a 10 node cluster and be off and running. Alternatively, we also have a development environment that all of the committers and contributors on the project tend to use. It spins up some basic sensors for bro, yaf, and snort and runs the telemetry data through Metron. The only downside is that due to the Hadoop components we depend on, it's a rather beefy setup for some personal computers. You need 8GB mem for the Vagrant VM to run. See here - https://github.com/apache/metron/tree/master/metron-deployment/development. Centos 6 is currently the most stable/seasoned. Once we finish our Hadoop upgrade this may be deprecated in favor of Centos 7, but this has not yet been decided.
On Thu, Sep 12, 2019 at 2:47 PM Eric Jacksch <e...@jacksch.com> wrote: > Hi Michael, > > Thank you very much for your reply. > > I'm working with the current release downloaded from > https://metron.apache.org/documentation/#releases 0.7.1 > > At this point, our objective is just to get Metron running in AWS so > we can try it out. If there is a faster/better/more reliable way to do > so, I'm open to suggestions :) > > Thanks, > Eric > > On Thu, 12 Sep 2019 at 16:39, Michael Miklavcic > <michael.miklav...@gmail.com> wrote: > > > > I think it's been quite some time since this has been tested (that I'm > aware of, anyhow). YMMV. I don't see any errors in the stacktrace you > copied - those are all warnings. Near the bottom, the Maven output > references "metron-profiler-spark", which is the part of the output we'd > want to see. Are you deploying a specific release of Metron or just the > latest master? > > > > On Thu, Sep 12, 2019 at 2:27 PM Eric Jacksch <e...@jacksch.com> wrote: > >> > >> Greetings, > >> > >> I've been trying to deploy in AWS to ec2 instances using the playbook. > >> > >> The VPC is created, instances spun up, etc, but I get hundreds (or > >> thousands) of lines of errors during the build. > >> > >> Can anyone help? > >> > >> I'm running the playbook on another EC2 instance, so it is certainly > >> possible that I am missing a dependency, etc... > >> > >> Thanks! > >> > >> > >> > >> TASK [metron-builder : Build Metron] > >> > ********************************************************************************************** > >> failed: [ec2-35-183-121-120.ca-central-1.compute.amazonaws.com -> > >> localhost] (item=mvn package -DskipTests -T 2C -P HDP-2.5.0.0,mpack) > >> => {"changed": true, "cmd": "mvn package -DskipTests -T 2C -P > >> HDP-2.5.0.0,mpack", "delta": "0:01:12.526650", "end": "2019-09-12 > >> 19:58:48.231138", "failed": true, "item": "mvn package -DskipTests -T > >> 2C -P HDP-2.5.0.0,mpack", "msg": "non-zero return code", "rc": 1, > >> "start": "2019-09-12 19:57:35.704488", "stderr": "warning: No > >> SupportedSourceVersion annotation found on > >> org.adrianwalker.multilinestring.MultilineProcessor, returning > >> RELEASE_6.\nwarning: Supported source version 'RELEASE_6' from > >> annotation processor > >> 'org.adrianwalker.multilinestring.MultilineProcessor' less than > >> -source '1.8'\n2 warnings\nwarning: No SupportedSourceVersion > >> annotation found on > >> org.adrianwalker.multilinestring.MultilineProcessor, returning > >> RELEASE_6.\nwarning: Supported source version 'RELEASE_6' from > >> annotation processor > >> 'org.adrianwalker.multilinestring.MultilineProcessor' less than > >> -source > '1.8'\n/home/ec2-user/metron/metron-platform/metron-test-utilities/src/main/java/org/apache/metron/test/bolt/BaseBoltTest.java:87: > >> warning: [unchecked] unchecked method invocation: method copyOf in > >> class ImmutableSet is applied to given types\n ImmutableSet keys = > >> ImmutableSet.copyOf(message.keySet());\n > >> ^\n required: Collection<? extends E>\n found: Set\n > >> where E is a type-variable:\n E extends Object declared in method > >> <E>copyOf(Collection<? extends > >> > E>)\n/home/ec2-user/metron/metron-platform/metron-test-utilities/src/main/java/org/apache/metron/test/bolt/BaseBoltTest.java:87: > >> warning: [unchecked] unchecked conversion\n ImmutableSet keys = > >> ImmutableSet.copyOf(message.keySet());\n > >> ^\n required: Collection<? extends E>\n > >> found: Set\n where E is a type-variable:\n E extends Object > >> declared in method <E>copyOf(Collection<? extends > >> > E>)\n/home/ec2-user/metron/metron-platform/metron-test-utilities/src/main/java/org/apache/metron/test/utils/KafkaLoader.java:67: > >> warning: [unchecked] unchecked call to send(ProducerRecord<K,V>) as a > >> member of the raw type KafkaProducer\n kafkaProducer.send(new > >> ProducerRecord<String, String>(topic, line));\n > >> ^\n where K,V are type-variables:\n K extends Object declared > >> in class KafkaProducer\n V extends Object declared in class > >> KafkaProducer\n5 > >> > warnings\n/home/ec2-user/metron/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RestFunctions.java:174: > >> warning: [unchecked] unchecked conversion\n functionRestConfig > >> = getArg(1, Map.class, args);\n ^\n > >> required: Map<String,Object>\n found: > >> > Map\n/home/ec2-user/metron/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RestFunctions.java:176: > >> warning: [unchecked] unchecked conversion\n queryParameters = > >> getArg(2, Map.class, args);\n ^\n > >> required: Map<String,Object>\n found: > >> > Map\n/home/ec2-user/metron/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RestFunctions.java:181: > >> warning: [unchecked] unchecked cast\n Map<String, Object> > >> globalRestConfig = (Map<String, Object>) > >> getGlobalConfig(context).get(STELLAR_REST_SETTINGS);\n > >> > >> ^\n required: Map<String,Object>\n found: > >> > Object\n/home/ec2-user/metron/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RestFunctions.java:182: > >> warning: [unchecked] unchecked cast\n Map<String, Object> > >> getRestConfig = (Map<String, Object>) > >> getGlobalConfig(context).get(STELLAR_REST_GET_SETTINGS);\n > >> > >> ^\n required: Map<String,Object>\n found: > >> > Object\n/home/ec2-user/metron/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RestFunctions.java:183: > >> warning: [unchecked] unchecked generic array creation for varargs > >> parameter of type Map<String,Object>[]\n RestConfig restConfig = > >> buildRestConfig(globalRestConfig, getRestConfig, > >> functionRestConfig);\n > >> > ^\n/home/ec2-user/metron/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RestFunctions.java:254: > >> warning: [unchecked] unchecked conversion\n functionRestConfig > >> = getArg(2, Map.class, args);\n ^\n > >> required: Map<String,Object>\n found: > >> > Map\n/home/ec2-user/metron/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RestFunctions.java:256: > >> warning: [unchecked] unchecked conversion\n queryParameters = > >> getArg(3, Map.class, args);\n ^\n > >> required: Map<String,Object>\n found: > >> > Map\n/home/ec2-user/metron/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RestFunctions.java:261: > >> warning: [unchecked] unchecked cast\n Map<String, Object> > >> globalRestConfig = (Map<String, Object>) > >> getGlobalConfig(context).get(STELLAR_REST_SETTINGS);\n > >> > >> ^\n required: Map<String,Object>\n found: > >> > Object\n/home/ec2-user/metron/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RestFunctions.java:262: > >> warning: [unchecked] unchecked cast\n Map<String, Object> > >> postRestConfig = (Map<String, Object>) > >> getGlobalConfig(context).get(STELLAR_REST_POST_SETTINGS);\n > >> > >> ^\n required: Map<String,Object>\n found: > >> > Object\n/home/ec2-user/metron/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RestFunctions.java:263: > >> warning: [unchecked] unchecked generic array creation for varargs > >> parameter of type Map<String,Object>[]\n RestConfig restConfig = > >> buildRestConfig(globalRestConfig, postRestConfig, > >> functionRestConfig);\n > >> > ^\n/home/ec2-user/metron/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RestFunctions.java:347: > >> warning: [unchecked] unchecked generic array creation for varargs > >> parameter of type Map<String,Object>[]\n RestConfig restConfig = > >> buildRestConfig(getGlobalConfig(context));\n > >> > >> > ^\n/home/ec2-user/metron/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/generated/StellarParser.java:3045: > >> warning: [MissingOverride] sempred overrides method in Recognizer; > >> expected @Override\n\tpublic boolean sempred(RuleContext _localctx, > >> int ruleIndex, int predIndex) {\n\t ^\n (see > >> http://errorprone.info/bugpattern/MissingOverride)\n Did you mean > >> '@Override public boolean sempred(RuleContext _localctx, int > >> ruleIndex, int predIndex) > >> > {'?\n/home/ec2-user/metron/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/SerDeUtils.java:136: > >> warning: [unchecked] newInstantiatorOf(Class) in > >> DefaultInstantiatorStrategy implements <T>newInstantiatorOf(Class<T>) > >> in InstantiatorStrategy\n public ObjectInstantiator > >> newInstantiatorOf(final Class type) {\n > >> ^\n return type requires unchecked conversion from ObjectInstantiator > >> to ObjectInstantiator<T>\n where T is a type-variable:\n T extends > >> Object declared in method <T>newInstantiatorOf(Class<T>)\n13 > >> warnings\nwarning: No SupportedSourceVersion annotation found on > >> org.adrianwalker.multilinestring.MultilineProcessor, returning > >> RELEASE_6.\nwarning: Supported source version 'RELEASE_6' from > >> annotation processor > >> 'org.adrianwalker.multilinestring.MultilineProcessor' less than > >> -source > '1.8'\n/home/ec2-user/metron/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/BasicStellarTest.java:654: > >> warning: [unchecked] unchecked method invocation: method run in class > >> StellarProcessorUtils is applied to given types\n > >> assertThat(run(query, vars), instanceOf(Map.class));\n > >> ^\n required: String,Map<String,Object>\n found: > >> > String,Map\n/home/ec2-user/metron/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/BasicStellarTest.java:654: > >> warning: [unchecked] unchecked conversion\n assertThat(run(query, > >> vars), instanceOf(Map.class));\n ^\n > >> required: Map<String,Object>\n found: > >> > Map\n/home/ec2-user/metron/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/BasicStellarTest.java:656: > >> warning: [unchecked] unchecked method invocation: method run in class > >> StellarProcessorUtils is applied to given types\n > >> assertThat(run(query, vars), equalTo(\"bar\"));\n ^\n > >> required: String,Map<String,Object>\n found: > >> > String,Map\n/home/ec2-user/metron/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/BasicStellarTest.java:656: > >> warning: [unchecked] unchecked conversion\n assertThat(run(query, > >> vars), equalTo(\"bar\"));\n ^\n required: > >> Map<String,Object>\n found: > >> > Map\n/home/ec2-user/metron/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/BasicStellarTest.java:662: > >> warning: [unchecked] unchecked call to put(K,V) as a member of the raw > >> type HashMap\n put(\"mymap\", null);\n ^\n where K,V are > >> type-variables:\n K extends Object declared in class HashMap\n V > >> extends Object declared in class > >> > HashMap\n/home/ec2-user/metron/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/fu > >> > >> <snip...many, many, many lines omitted> > >> > >> (Wall Clock)", "[INFO] Finished at: 2019-09-12T19:58:47Z", "[INFO] > >> Final Memory: 738M/2811M", "[INFO] > >> > ------------------------------------------------------------------------", > >> "[ERROR] Failed to execute goal > >> org.apache.maven.plugins:maven-compiler-plugin:3.5.1:compile > >> (default-compile) on project metron-profiler-spark: Compilation > >> failure -> [Help 1]", "[ERROR] ", "[ERROR] To see the full stack trace > >> of the errors, re-run Maven with the -e switch.", "[ERROR] Re-run > >> Maven using the -X switch to enable full debug logging.", "[ERROR] ", > >> "[ERROR] For more information about the errors and possible solutions, > >> please read the following articles:", "[ERROR] [Help 1] > >> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException";, > >> "[ERROR] ", "[ERROR] After correcting the problems, you can resume the > >> build with the command", "[ERROR] mvn <goals> -rf > >> :metron-profiler-spark", "[INFO] ", "[INFO] --- > >> maven-enforcer-plugin:1.4.1:enforce (enforce-versions) @ > >> metron-data-management ---"]} > >> > >> NO MORE HOSTS LEFT > >> > **************************************************************************************************************** > >> to retry, use: --limit > >> @/home/ec2-user/metron/metron-deployment/amazon-ec2/playbook.retry > >> > >> PLAY RECAP > ************************************************************************************************************************ > >> ec2-15-222-1-43.ca-central-1.compute.amazonaws.com : ok=14 changed=8 > >> unreachable=0 failed=0 > >> ec2-15-222-9-90.ca-central-1.compute.amazonaws.com : ok=14 changed=8 > >> unreachable=0 failed=0 > >> ec2-35-182-158-204.ca-central-1.compute.amazonaws.com : ok=14 > >> changed=8 unreachable=0 failed=0 > >> ec2-35-183-110-49.ca-central-1.compute.amazonaws.com : ok=14 > >> changed=8 unreachable=0 failed=0 > >> ec2-35-183-121-120.ca-central-1.compute.amazonaws.com : ok=15 > >> changed=9 unreachable=0 failed=1 > >> ec2-35-183-121-202.ca-central-1.compute.amazonaws.com : ok=14 > >> changed=8 unreachable=0 failed=0 > >> ec2-35-183-210-180.ca-central-1.compute.amazonaws.com : ok=14 > >> changed=8 unreachable=0 failed=0 > >> ec2-99-79-193-27.ca-central-1.compute.amazonaws.com : ok=14 > >> changed=8 unreachable=0 failed=0 > >> ec2-99-79-52-74.ca-central-1.compute.amazonaws.com : ok=14 changed=8 > >> unreachable=0 failed=0 > >> ec2-99-79-60-74.ca-central-1.compute.amazonaws.com : ok=14 changed=8 > >> unreachable=0 failed=0 > >> localhost : ok=20 changed=15 unreachable=0 > failed=0 > > > > -- > Eric Jacksch, CPP, CISM, CISSP > e...@jacksch.com > Twitter: @EricJacksch > https://SecurityShelf.com >
