[GitHub] incubator-metron issue #533: METRON-856: Ansible rpm build wipes out prior b...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/533
  
Does the current vagrant up count on this working the old way?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #530: METRON-777 Metron Extension System and Parser E...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/530
  
if you don't review the parsers, because the code didn't change, you still 
need to review or think about the concept that each parser, as an extension 
should have all of the things ( configuration etc ) that it needs within it's 
package.  So the parsers have their grok statements, their configuration ( 
index, enrichment , and parsers  and in the future ES + log rotate )


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #530: METRON-777 Metron Extension System and Parser E...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/530
  
So do you like want a listing of files?
All the areas of functionality that are not bundle or parser extensions 
would be the areas to look at.
Although I would say that the bundle-lib should still be reviewed in 
general.

Here is a high level by the file guide, but I may be missing something

// changes to OOM issues in travis container
.travis.yml
README.md

// packaging and deployment

metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml

metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml

metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py

metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py

metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py

metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_commands.py

metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/bundle.properties.j2
metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
metron-deployment/packaging/docker/rpm-docker/pom.xml

metron-extensions/README.md
metron-extensions/metron-parser-extensions/README.md

// asa integration test, but from bundle, not dep on asa

metron-extensions/metron-parser-extensions/metron-parser-bundle-tests/pom.xml

metron-extensions/metron-parser-extensions/metron-parser-bundle-tests/src/test/java/org/apache/metron/parsers/ASABundleHDFSIntegrationTest.java

metron-extensions/metron-parser-extensions/metron-parser-bundle-tests/src/test/resources/metron/extension_contrib_lib/metron-parser-test-bundle-1.0-SNAPSHOT.bundle

metron-extensions/metron-parser-extensions/metron-parser-bundle-tests/src/test/resources/metron/extension_lib/metron-parser-asa-bundle-0.3.1.bundle

metron-extensions/metron-parser-extensions/metron-parser-bundle-tests/src/test/resources/zookeeper/bundle.properties

metron-extensions/metron-parser-extensions/metron-parser-bundle-tests/src/test/resources/zookeeper/enrichments/test.json

metron-extensions/metron-parser-extensions/metron-parser-bundle-tests/src/test/resources/zookeeper/global.json

metron-extensions/metron-parser-extensions/metron-parser-bundle-tests/src/test/resources/zookeeper/indexing/test.json

// fixes for tests

metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronRestConstants.java

metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/GrokControllerIntegrationTest.java

metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/KafkaControllerIntegrationTest.java

metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/SensorParserConfigControllerIntegrationTest.java

metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/GrokServiceImplTest.java

metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/SensorParserConfigServiceImplTest.java

//new functions for bundle.properties and tests - dealing with paths etc

metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java

metron-platform/metron-common/src/test/java/org/apache/metron/common/cli/ConfigurationsUtilsTest.java

metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/SensorEnrichmentConfigTest.java

//support new directories

metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/components/ConfigUploadComponent.java

// load parsers from bundles

metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java

metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserLoader.java

metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyBuilder.java

// the system bundle properties to tests

metron-platform/metron-integration-test/src/main/config/zookeeper/bundle.properties

// test function fixes for new paths

metron-platform/metron-management/src/test/java/org/apache/metron/management/ConfigurationFunctionsTest.java

metron-platform/metron-management/src/test/java/org/apache/metron/management/ParserConfigFunctionsTest.java

// fix to work with new paths

metron-platform/metron-test-utilities/src/main/java/org/apache

[GitHub] incubator-metron issue #530: METRON-777 Metron Extension System and Parser E...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/530
  
RE: parser size
The metron-parser jar is still shaded. it is 97M JAR, 87M archive
The parsers, individually like the ASA are 44k JAR, 44k Bundle, 49K tar.gz 
each


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #530: METRON-777 Metron Extension System and Parser E...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/530
  
RE: Travis
I was getting these and going to vm was the only way I could get rid of 
them:


The system is out of resources.
Consult the following stack trace for details.
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3181)
at java.util.ArrayList.grow(ArrayList.java:261)
at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:235)
at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:227)
at java.util.ArrayList.add(ArrayList.java:458)
at 
com.sun.tools.javac.file.ZipFileIndex$ZipDirectory.readEntry(ZipFileIndex.java:674)
at 
com.sun.tools.javac.file.ZipFileIndex$ZipDirectory.buildIndex(ZipFileIndex.java:578)
at 
com.sun.tools.javac.file.ZipFileIndex$ZipDirectory.access$000(ZipFileIndex.java:485)
at 
com.sun.tools.javac.file.ZipFileIndex.checkIndex(ZipFileIndex.java:193)
at com.sun.tools.javac.file.ZipFileIndex.(ZipFileIndex.java:137)
at 
com.sun.tools.javac.file.ZipFileIndexCache.getZipFileIndex(ZipFileIndexCache.java:100)
at 
com.sun.tools.javac.file.JavacFileManager.openArchive(JavacFileManager.java:598)
at 
com.sun.tools.javac.file.JavacFileManager.openArchive(JavacFileManager.java:545)
at 
com.sun.tools.javac.file.JavacFileManager.listContainer(JavacFileManager.java:429)
at 
com.sun.tools.javac.file.JavacFileManager.list(JavacFileManager.java:676)
at 
com.sun.tools.javac.code.ClassFinder.scanUserPaths(ClassFinder.java:564)
at com.sun.tools.javac.code.ClassFinder.fillIn(ClassFinder.java:504)
at com.sun.tools.javac.code.ClassFinder.complete(ClassFinder.java:287)
at com.sun.tools.javac.code.ClassFinder.access$000(ClassFinder.java:72)
at com.sun.tools.javac.code.ClassFinder$1.complete(ClassFinder.java:159)
at com.sun.tools.javac.code.Symbol.complete(Symbol.java:579)
at com.sun.tools.javac.comp.Enter.visitTopLevel(Enter.java:299)
at 
com.sun.tools.javac.tree.JCTree$JCCompilationUnit.accept(JCTree.java:509)
at com.sun.tools.javac.comp.Enter.classEnter(Enter.java:255)
at com.sun.tools.javac.comp.Enter.classEnter(Enter.java:270)
at com.sun.tools.javac.comp.Enter.complete(Enter.java:483)
at com.sun.tools.javac.comp.Enter.main(Enter.java:467)
at 
com.sun.tools.javac.main.JavaCompiler.enterTrees(JavaCompiler.java:952)
at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:833)
at com.sun.tools.javac.main.Main.compile(Main.java:253)
at 
com.google.errorprone.BaseErrorProneCompiler.run(BaseErrorProneCompiler.java:214)
at 
com.google.errorprone.BaseErrorProneCompiler.run(BaseErrorProneCompiler.java:106)
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.5.1:compile (default-compile) 
on project metron-parser-lancope: Compilation failure -> [Help 1]
[ERROR] 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #530: METRON-777 Metron Extension System and P...

[ottobackwards]

GitHub user ottobackwards opened a pull request:

https://github.com/apache/incubator-metron/pull/530

METRON-777 Metron Extension System and Parser Extensions

## Contributor Comments
[Please place any comments here.  A description of the problem/enhancement, 
how to reproduce the issue, your testing methodology, etc.]

The pr. introduces an extension system for metron, along with refactoring 
the metron parsers on top of it.  This is the base work for METRON-258 - 
Sideloading parsers, which is to follow,  as it enables the creation and 
management of extensions outside of the metron codebase.  The work for enabling 
side loading is the ability to install and deploy 3rd party extensions/parsers.

There is a lot that can be done with this, but I could nibble at it 
forever, and I'd like to get feedback and improvements going.  There is still 
more documentation work that can be done for example.

The areas of change:

### Metron Maven Bundle Plugin
- adaptation of the nifi plugin
- more configurable wrt file extension/dependency and metadata naming
- new pre-build step on clean systems to install plugin

### bundle-lib
My goal here was not to make any radical changes
 - adaptation of nifi-nar-utils to be used outside of the nifi project
 - rudimentary extensibility to allow configuration and injection of 
service types and other things that were hard coded to nifi 
 - refactored from File based to VFS based
 - introduced class cluster for FileUtils to allow for specialized HDFS 
file functionality ( HDFS with VFS is read only )
 - rebranding to Bundle from Nar ( although the lib and the plugin allow 
that to be configured now )
 - added capability to the properties class to write to stream, adapted to 
uri from paths
 - added integration tests for hdfs

### Metron Maven Parser Extension Archetype
 - locally installable archetype for creating Metron Parsers
 -  dependencies instead of shading
 - builds Bundle
 - creates an assembly with bundle and configuration
 - configuration for parsers now includes all parser related configurations 
( except ES and Logrotate )
 - Includes sample data for testing, global config etc. ( such that you 
don't need metron code to build and test ).
 - Can be used with configuration only parsers, so that you can still unit 
and integration test them, common deployment
 - Creates documentation readme.

### Metron-Extensions / Metron-Parser-Extensions/**
- Module area for extensions, first extension type is parsers
- All parsers re-based on archetype generated projects
- All parser test data/ configuration located with parser ( see above )
- Each parser had a readme that should be filled out, but I didn't do that

### Metron-Parsers
 - Removed all parsers and their tests etc except CSV, JSON, GROK
 - Still shaded, still the storm loaded jar
 - extended or fix tests so that they work when derived outside of code tree
 - Parser bolt no longer takes MessageParser<> instance, loads it as from 
extension/bundle system

### Metron-configuration
 - changes to support new parser locations
 - added functionality to load and store bundle.properties to zk

### Metron Tests
 - Extended to work with relative path / formatted paths

### RPM-Build
 - Copy all the parser extensions
 - Include in the spec

/usr/metron/V/ now has a new directories for extensions:

/extension_etc/PARSER_NAME/  -> that parser's configuration
/extension_alt_etc/ -> location for 3rd Party extension configuration
/extension_lib -> location on disk for rpm to place bundles
/extension_alt_lib -> location on disk for staging 3rd party bundles

### METRON-SERVICE ambari
 - Load zk configurations for parsers from their location
 - filles out the properties template and deploys to hdfs
 - create HDFS directories
 - deploy/copy bundles to HDFS

### the metron workflow that this enables
We need a new parser:
*create with archetype under metron-extensions/metron-parser-extensions
*implement including tests and test data, all configurations
*add to the copy-resources of RPM-Docker pom
*add to the spec file
*add to the all_parsers variable in params
 - this will get it installed but not started, no ES no log rotate
* add to parsers variable in the env.xml to get it to start as well ( still 
no ES or Log rotate )
* other steps to get the ES template integrated with indexing scripts and 
log rotate with ansible

I have been working in Full Dev to get this going, and I believe it is 
working enough to get this started.
At the end of vagrant up with full dev, you should have data in kibana, as 
if nothing had changed ;)

There are issues however:
 

[GitHub] incubator-metron issue #528: METRON-838 Incorrect set of ts in FireEye parse...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/528
  
No just the appropriate ones you can see.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #528: METRON-838 Incorrect set of ts in FireEye parse...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/528
  
I created a gist of the template that you can copy and paste.
Please go through and fill out the appropriate things.

https://gist.github.com/ottobackwards/9b7ea0689a79b9510c52ad49045aea7e


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #528: METRON-838 Incorrect set of ts in FireEye parse...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/528
  
Hi bjigmp!
Thanks for the contribution.
We have a template for our PR's, that help us review and accept them.  This 
PR is missing that,
can you please add it in? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #516: METRON-830 Adding StringFunctions to Stellar - ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/516
  
+1, thanks for the contribution


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/516#discussion_r110629490
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java
 ---
@@ -343,4 +343,89 @@ public Object apply(List args) {
   return String.format(format, formatArgs);
 }
   }
+
+  @Stellar( name="CHOP"
--- End diff --

That is not to say the CHOP cannot be useful however


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/516#discussion_r110629309
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java
 ---
@@ -343,4 +343,89 @@ public Object apply(List args) {
   return String.format(format, formatArgs);
 }
   }
+
+  @Stellar( name="CHOP"
--- End diff --

If we are going to do URL/URI work, we should create functions off of the 
URI / URL classes and use their specific parsing for uri named component parts


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #517: METRON-831: Add lambda expressions and rudiment...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/517
  
+1 by review.  The 'assumes one argument' part, actually the 'assumes' part 
isn't my favorite thing, but we can look at that when we have interactive 
stellar writing in the ui ( and can validate before deploy ) better sorted out.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #489: METRON-623: Management UI

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/489
  
I agree +1



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #516: METRON-830 Adding StringFunctions to Stellar - ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/516
  
Thanks for the contribution!

Quick review, first thing please follow the style guide for naming classes. 
 You can see from the other stellar function classes.

It would be good to have some tests that aren't with string or null, to 
verify the behavior.

What happens if I pass in a number literal?




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/516#discussion_r110292921
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java
 ---
@@ -343,4 +343,89 @@ public Object apply(List args) {
   return String.format(format, formatArgs);
 }
   }
+
+  @Stellar( name="CHOP"
+  , description = "Remove the last character from a String"
+  , params = { "the String to chop last character from, may be 
null"}
+  , returns = "String without last character, null if null String 
input"
+  )
+  public static class chop extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+  if(strings.size() == 0) {
+throw new IllegalArgumentException("[CHOP] missing argument: 
string to be chopped");
+  }
+
+  String chop = StringUtils.chop((String) strings.get(0));
+  return chop;
+}
+  }
+
+  @Stellar( name = "PREPENDIFMISSING"
+  , description = "Prepends the prefix to the start of the string 
if the string does not already start with any of the prefixes"
+  , params = { "str - The string.", "prefix - The prefix to 
prepend to the start of the string", "prefixes - Additional prefixes that are 
valid" }
+  , returns = "A new String if prefix was prepended, the same 
string otherwise."
+  )
+  public static class prependifmissing extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
--- End diff --

Invalid Cast exception?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #508: METRON-820: StellarProcessor should have a stat...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/508
  
+1 by inspection, although I'm inclined to think of a reason not to have a 
static cache, but instead an instance based one, I can't think of a specific 
case.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #508: METRON-820: StellarProcessor should have a stat...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/508
  
1 question -> does this require any relation to the VFS cache?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #505: METRON-817: Customise output file path p...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/505#discussion_r109428569
  
--- Diff: 
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java
 ---
@@ -74,17 +91,43 @@ public BulkWriterResponse write(String sourceType
) throws Exception
   {
 BulkWriterResponse response = new BulkWriterResponse();
-SourceHandler handler = 
getSourceHandler(configurations.getIndex(sourceType));
+// Currently treating all the messages in a group for pass/failure.
 try {
-  handler.handle(messages);
-} catch(Exception e) {
+  // Messages can all result in different HDFS paths, because of 
Stellar Expressions, so we'll need to iterate through
+  for(JSONObject message : messages) {
+Map val = 
configurations.getSensorConfig(sourceType);
+String path = getHdfsPathExtension(
+sourceType,
+
(String)configurations.getSensorConfig(sourceType).getOrDefault(IndexingConfigurations.OUTPUT_PATH_FUNCTION_CONF,
 ""),
+message
+);
+SourceHandler handler = getSourceHandler(sourceType, path);
+handler.handle(message);
+  }
+} catch (Exception e) {
   response.addAllErrors(e, tuples);
 }
 
 response.addAllSuccesses(tuples);
 return response;
   }
 
+  public String getHdfsPathExtension(String sourceType, String 
stellarFunction, JSONObject message) {
+// If no function is provided, just use the sourceType directly
+if(stellarFunction == null || stellarFunction.trim().isEmpty()) {
+  return sourceType;
+}
+
+StellarCompiler.Expression expression = 
sourceTypeExpressionMap.computeIfAbsent(stellarFunction, s -> 
stellarProcessor.compile(stellarFunction));
+VariableResolver resolver = new MapVariableResolver(message);
--- End diff --

This makes me think of the UI case.  We configure the index configuration 
but have no way of validation before they save and deploy.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #505: METRON-817: Customise output file path p...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/505#discussion_r109428017
  
--- Diff: 
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java
 ---
@@ -74,17 +91,43 @@ public BulkWriterResponse write(String sourceType
) throws Exception
   {
 BulkWriterResponse response = new BulkWriterResponse();
-SourceHandler handler = 
getSourceHandler(configurations.getIndex(sourceType));
+// Currently treating all the messages in a group for pass/failure.
 try {
-  handler.handle(messages);
-} catch(Exception e) {
+  // Messages can all result in different HDFS paths, because of 
Stellar Expressions, so we'll need to iterate through
+  for(JSONObject message : messages) {
+Map val = 
configurations.getSensorConfig(sourceType);
+String path = getHdfsPathExtension(
+sourceType,
+
(String)configurations.getSensorConfig(sourceType).getOrDefault(IndexingConfigurations.OUTPUT_PATH_FUNCTION_CONF,
 ""),
+message
+);
+SourceHandler handler = getSourceHandler(sourceType, path);
+handler.handle(message);
+  }
+} catch (Exception e) {
   response.addAllErrors(e, tuples);
 }
 
 response.addAllSuccesses(tuples);
 return response;
   }
 
+  public String getHdfsPathExtension(String sourceType, String 
stellarFunction, JSONObject message) {
+// If no function is provided, just use the sourceType directly
+if(stellarFunction == null || stellarFunction.trim().isEmpty()) {
+  return sourceType;
+}
+
+StellarCompiler.Expression expression = 
sourceTypeExpressionMap.computeIfAbsent(stellarFunction, s -> 
stellarProcessor.compile(stellarFunction));
+VariableResolver resolver = new MapVariableResolver(message);
--- End diff --

We should be able to find out from the function metadata/annotation the 
return type, without doing all this work shouldn't we?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r109204340
  
--- Diff: 
metron-interface/metron-config/e2e/sensor-config-readonly/sensor-config-readonly.po.ts
 ---
@@ -0,0 +1,125 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import { browser, element, by, protractor } from 'protractor/globals';
+import {changeURL, waitForElementVisibility} from '../utils/e2e_util';
+
+export class SensorDetailsPage {
+
+private enableButton;
+private disableButton;
+private startButton;
+private stopButton;
+
+constructor() {
+this.enableButton = 
element(by.cssContainingText('metron-config-sensor-parser-readonly .btn', 
'ENABLE'));
+this.disableButton = 
element(by.cssContainingText('metron-config-sensor-parser-readonly .btn', 
'DISABLE'));
+this.startButton = 
element(by.cssContainingText('metron-config-sensor-parser-readonly .btn', 
'START'));
+this.stopButton = 
element(by.cssContainingText('metron-config-sensor-parser-readonly .btn', 
'STOP'));
+}
+
+clickToggleShowMoreLess(text: string, index: number) {
+return element.all(by.linkText(text)).get(index).click().then(() 
=> {
+browser.sleep(1000);
+return true;
+})
+}
+
+closePane(name: string) {
+return element(by.css('metron-config-sensor-parser-readonly 
.fa-times')).click().then(() => {
+return true;
+});
+}
+
+disableParser() {
+return waitForElementVisibility(this.disableButton).then(() => {
+return this.disableButton.click().then(() => {
+return waitForElementVisibility(this.enableButton).then(() 
=> {
+return true;
+})
+});
+});
+}
+
+enableParser() {
+return waitForElementVisibility(this.enableButton).then(() => {
+return this.enableButton.click().then(() => {
+return 
waitForElementVisibility(this.disableButton).then(() => {
+return true;
+})
+});
+});
+}
+
+startParser() {
+return waitForElementVisibility(this.startButton).then(() => {
+return this.startButton.click().then(() => {
+return waitForElementVisibility(this.stopButton).then(() 
=> {
+return true;
+})
+});
+});
+}
+
+stopParser() {
+return waitForElementVisibility(this.stopButton).then(() => {
+return this.stopButton.click().then(() => {
+return waitForElementVisibility(this.startButton).then(() 
=> {
+return true;
+})
+});
+});
+}
+
+getButtons() {
+return element.all(by.css('metron-config-sensor-parser-readonly 
button:not([hidden=""])')).getText();
+}
+
+getGrokStatement() {
--- End diff --

I think I should help with that



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #489: METRON-623: Management UI

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/489
  
@simonellistonball I totally agree.  I would rather have this in to work 
off of.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #489: METRON-623: Management UI

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/489
  
I agree with @cestella about making this grok centric.  

We should have a discussion around making the rest service work with 
plugins ( for example each plugin/extension/parser supplies a configuration 
service, etc.

When I land the side loading or parsers, I'll possibly have to look at this.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #501: METRON-812: Make the bro-kafka plugin work with...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/501
  
+1 by inspection


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #494: METRON-806 set posix

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/494
  
Thank *you*.  My maven foo is not that strong, it didn't occur to me, and 
this is like the third time fixing this!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #494: METRON-806 set posix

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/494
  
Ok - removed from child poms, put in parent


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #494: METRON-806 set posix

[ottobackwards]

GitHub user ottobackwards opened a pull request:

https://github.com/apache/incubator-metron/pull/494

METRON-806 set  posix for metron-rest

## Contributor Comments

Any new assemblies added to the project need to have the tarLongFileMode set

Tested by running on a mac attached to an active directory domain where 
build failed prior

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron (Incubating).  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x ] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```
[my travis 
build](https://travis-ci.org/ottobackwards/incubator-metron/builds/215546710)

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  bin/generate-md.sh
  mvn site:site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommened that [travis-ci](https://travis-ci.org) is set up for 
your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/incubator-metron METRON-806

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/494.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #494


commit 668a118e93e74d8f96805b346610ab96c27d8e67
Author: Otto Fowler 
Date:   2017-03-27T14:08:29Z

 set  posix for metron-rest




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/439#discussion_r105328056
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java
 ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+   public static class ExecuteScript implements StellarFunction {
+
+private ThreadedStreamHandler inStream;
+private ThreadedStreamHandler errStream;
+private boolean isOnTheList = false;
+
+@Stellar(name="EXEC_SCRIPT",
+description = "Executes an external shell function via 
stellar.",
+params = {
+"exec - the executing cmd (ie. bash, sh, python)",
+"name - name of the script, located in /scripts " +
+"Do NOT include any special chars 
except(_), Do include file extension"
+},
+returns = "the return value of the function"
+)
+
+   @Override
+public Object apply(List args, Context context) throws 
ParseException {
+String exec = "";
+String name = "";
+String path = "";
+
+// if args are provided, get args, only if in whitelist
+if (args.size() >= 1) {
+Object execObj = args.get(0);
+if (!(execObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) execObj).length() > 0) {
+exec = (String) execObj;
+}
+else {
+return null;
+}
+
+Object nameObj = args.get(1);
+if (!(nameObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) nameObj).length() > 0) {
+name = (String) nameObj;
+}
+else {
+return null;
+}
+
+if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+return null; //if not on whitelist
+}
+
+path = "/scripts" + name;
+try {
+File script = new File(path);
+if (!script.exists() || script.isDirectory()) {
+return null;
+}
+}
+catch (NullPointerException e)  {
+System.err.println("Error: " + e.toString());
+return null;
+}
+
+switch (exec) { //check if matches extensi

[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/439#discussion_r105327868
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java
 ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+   public static class ExecuteScript implements StellarFunction {
+
+private ThreadedStreamHandler inStream;
+private ThreadedStreamHandler errStream;
+private boolean isOnTheList = false;
+
+@Stellar(name="EXEC_SCRIPT",
+description = "Executes an external shell function via 
stellar.",
+params = {
+"exec - the executing cmd (ie. bash, sh, python)",
+"name - name of the script, located in /scripts " +
+"Do NOT include any special chars 
except(_), Do include file extension"
+},
+returns = "the return value of the function"
+)
+
+   @Override
+public Object apply(List args, Context context) throws 
ParseException {
+String exec = "";
+String name = "";
+String path = "";
+
--- End diff --

The ShellFunctions btw, are limited to the management package.
Is that were this needs to live?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/439#discussion_r105327240
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java
 ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+   public static class ExecuteScript implements StellarFunction {
+
+private ThreadedStreamHandler inStream;
+private ThreadedStreamHandler errStream;
+private boolean isOnTheList = false;
+
+@Stellar(name="EXEC_SCRIPT",
+description = "Executes an external shell function via 
stellar.",
+params = {
+"exec - the executing cmd (ie. bash, sh, python)",
+"name - name of the script, located in /scripts " +
+"Do NOT include any special chars 
except(_), Do include file extension"
+},
+returns = "the return value of the function"
+)
+
+   @Override
+public Object apply(List args, Context context) throws 
ParseException {
+String exec = "";
+String name = "";
+String path = "";
+
--- End diff --

Do we need to do:
Optional console =  context.getCapability(CONSOLE, true);

to make sure we have console for this?

see: ShellFunctions.java


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #439: METRON-571 add stellar external functions featu...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/439
  
This is missing documentation.  There are going to be requirements to the 
types of scripts that can be called, and they all need to be listed out.  A 
sample script or even a template should be provided.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/439#discussion_r105325446
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java
 ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+   public static class ExecuteScript implements StellarFunction {
+
+private ThreadedStreamHandler inStream;
+private ThreadedStreamHandler errStream;
+private boolean isOnTheList = false;
+
+@Stellar(name="EXEC_SCRIPT",
+description = "Executes an external shell function via 
stellar.",
+params = {
+"exec - the executing cmd (ie. bash, sh, python)",
+"name - name of the script, located in /scripts " +
+"Do NOT include any special chars 
except(_), Do include file extension"
+},
+returns = "the return value of the function"
+)
+
+   @Override
+public Object apply(List args, Context context) throws 
ParseException {
+String exec = "";
+String name = "";
+String path = "";
+
+// if args are provided, get args, only if in whitelist
+if (args.size() >= 1) {
+Object execObj = args.get(0);
+if (!(execObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) execObj).length() > 0) {
+exec = (String) execObj;
+}
+else {
+return null;
+}
+
+Object nameObj = args.get(1);
+if (!(nameObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) nameObj).length() > 0) {
+name = (String) nameObj;
+}
+else {
+return null;
+}
+
+if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+return null; //if not on whitelist
+}
+
+path = "/scripts" + name;
+try {
+File script = new File(path);
+if (!script.exists() || script.isDirectory()) {
+return null;
+}
+}
+catch (NullPointerException e)  {
+System.err.println("Error: " + e.toString());
+return null;
+}
+
+switch (exec) { //check if matches extensi

[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/439#discussion_r105325529
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java
 ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+   public static class ExecuteScript implements StellarFunction {
+
+private ThreadedStreamHandler inStream;
+private ThreadedStreamHandler errStream;
+private boolean isOnTheList = false;
+
+@Stellar(name="EXEC_SCRIPT",
+description = "Executes an external shell function via 
stellar.",
+params = {
+"exec - the executing cmd (ie. bash, sh, python)",
+"name - name of the script, located in /scripts " +
+"Do NOT include any special chars 
except(_), Do include file extension"
+},
+returns = "the return value of the function"
+)
+
+   @Override
+public Object apply(List args, Context context) throws 
ParseException {
+String exec = "";
+String name = "";
+String path = "";
+
+// if args are provided, get args, only if in whitelist
+if (args.size() >= 1) {
+Object execObj = args.get(0);
+if (!(execObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) execObj).length() > 0) {
+exec = (String) execObj;
+}
+else {
+return null;
+}
+
+Object nameObj = args.get(1);
+if (!(nameObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) nameObj).length() > 0) {
+name = (String) nameObj;
+}
+else {
+return null;
+}
+
+if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+return null; //if not on whitelist
+}
+
+path = "/scripts" + name;
+try {
+File script = new File(path);
+if (!script.exists() || script.isDirectory()) {
+return null;
+}
+}
+catch (NullPointerException e)  {
+System.err.println("Error: " + e.toString());
+return null;
+}
+
+switch (exec) { //check if matches extensi

[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/439#discussion_r105325302
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java
 ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+   public static class ExecuteScript implements StellarFunction {
+
+private ThreadedStreamHandler inStream;
+private ThreadedStreamHandler errStream;
+private boolean isOnTheList = false;
+
+@Stellar(name="EXEC_SCRIPT",
+description = "Executes an external shell function via 
stellar.",
+params = {
+"exec - the executing cmd (ie. bash, sh, python)",
+"name - name of the script, located in /scripts " +
+"Do NOT include any special chars 
except(_), Do include file extension"
+},
+returns = "the return value of the function"
+)
+
+   @Override
+public Object apply(List args, Context context) throws 
ParseException {
+String exec = "";
+String name = "";
+String path = "";
+
+// if args are provided, get args, only if in whitelist
+if (args.size() >= 1) {
+Object execObj = args.get(0);
+if (!(execObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) execObj).length() > 0) {
+exec = (String) execObj;
+}
+else {
+return null;
+}
+
+Object nameObj = args.get(1);
+if (!(nameObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) nameObj).length() > 0) {
+name = (String) nameObj;
+}
+else {
+return null;
+}
+
+if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+return null; //if not on whitelist
+}
+
+path = "/scripts" + name;
+try {
+File script = new File(path);
+if (!script.exists() || script.isDirectory()) {
+return null;
+}
+}
+catch (NullPointerException e)  {
+System.err.println("Error: " + e.toString());
+return null;
+}
+
+switch (exec) { //check if matches extensi

[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/439#discussion_r105325249
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java
 ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+   public static class ExecuteScript implements StellarFunction {
+
+private ThreadedStreamHandler inStream;
+private ThreadedStreamHandler errStream;
+private boolean isOnTheList = false;
+
+@Stellar(name="EXEC_SCRIPT",
+description = "Executes an external shell function via 
stellar.",
+params = {
+"exec - the executing cmd (ie. bash, sh, python)",
+"name - name of the script, located in /scripts " +
+"Do NOT include any special chars 
except(_), Do include file extension"
+},
+returns = "the return value of the function"
+)
+
+   @Override
+public Object apply(List args, Context context) throws 
ParseException {
+String exec = "";
+String name = "";
+String path = "";
+
+// if args are provided, get args, only if in whitelist
+if (args.size() >= 1) {
+Object execObj = args.get(0);
+if (!(execObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) execObj).length() > 0) {
+exec = (String) execObj;
+}
+else {
+return null;
+}
+
+Object nameObj = args.get(1);
+if (!(nameObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) nameObj).length() > 0) {
+name = (String) nameObj;
+}
+else {
+return null;
+}
+
+if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+return null; //if not on whitelist
+}
+
+path = "/scripts" + name;
+try {
+File script = new File(path);
+if (!script.exists() || script.isDirectory()) {
+return null;
+}
+}
+catch (NullPointerException e)  {
+System.err.println("Error: " + e.toString());
+return null;
+}
+
+switch (exec) { //check if matches extensi

[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/439#discussion_r105325103
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java
 ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+   public static class ExecuteScript implements StellarFunction {
+
+private ThreadedStreamHandler inStream;
+private ThreadedStreamHandler errStream;
+private boolean isOnTheList = false;
+
+@Stellar(name="EXEC_SCRIPT",
+description = "Executes an external shell function via 
stellar.",
+params = {
+"exec - the executing cmd (ie. bash, sh, python)",
+"name - name of the script, located in /scripts " +
+"Do NOT include any special chars 
except(_), Do include file extension"
+},
+returns = "the return value of the function"
+)
+
+   @Override
+public Object apply(List args, Context context) throws 
ParseException {
+String exec = "";
+String name = "";
+String path = "";
+
+// if args are provided, get args, only if in whitelist
+if (args.size() >= 1) {
+Object execObj = args.get(0);
+if (!(execObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) execObj).length() > 0) {
+exec = (String) execObj;
+}
+else {
+return null;
+}
+
+Object nameObj = args.get(1);
+if (!(nameObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) nameObj).length() > 0) {
+name = (String) nameObj;
+}
+else {
+return null;
+}
+
+if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+return null; //if not on whitelist
+}
+
+path = "/scripts" + name;
+try {
+File script = new File(path);
+if (!script.exists() || script.isDirectory()) {
+return null;
+}
+}
+catch (NullPointerException e)  {
+System.err.println("Error: " + e.toString());
+return null;
+}
+
+switch (exec) { //check if matches extensi

[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/439#discussion_r105324915
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java
 ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+   public static class ExecuteScript implements StellarFunction {
+
+private ThreadedStreamHandler inStream;
+private ThreadedStreamHandler errStream;
+private boolean isOnTheList = false;
+
+@Stellar(name="EXEC_SCRIPT",
+description = "Executes an external shell function via 
stellar.",
+params = {
+"exec - the executing cmd (ie. bash, sh, python)",
+"name - name of the script, located in /scripts " +
+"Do NOT include any special chars 
except(_), Do include file extension"
+},
+returns = "the return value of the function"
+)
+
+   @Override
+public Object apply(List args, Context context) throws 
ParseException {
+String exec = "";
+String name = "";
+String path = "";
+
+// if args are provided, get args, only if in whitelist
+if (args.size() >= 1) {
+Object execObj = args.get(0);
+if (!(execObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) execObj).length() > 0) {
+exec = (String) execObj;
+}
+else {
+return null;
+}
+
+Object nameObj = args.get(1);
+if (!(nameObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) nameObj).length() > 0) {
+name = (String) nameObj;
+}
+else {
+return null;
+}
+
+if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+return null; //if not on whitelist
+}
+
+path = "/scripts" + name;
--- End diff --

what file chooser dialog is that?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #405: METRON-641: Fixed kibana_master.py Python3 miss

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/405
  
I will later tonight or first thing tomorrow am


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
I am +1.  I have been working downstream and building off of this for a 
bit, and been able to get quick and full up with everything started.  I have 
some things that I would like to improve on, but like @dlyle65535 says, this is 
step 2 of many


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #474: METRON-758: HdfsServiceImplTest should sort fil...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/474
  
+1 Pulled into the failing branch -> issue resolved in travis


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
I'm fine with that.  Just being up front on the change is enough from now.  
Troubleshooting this may be a big user issue generator


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
you point on blueprints reminds me that this kind of obsoletes the 
'deploying metron on an ambari managed cluster' wiki page, and that page is 
very valuable


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
No, I don't have a problem with the ansible decision.  And I don't even 
mean for this feedback to hold up this PR.  I just think it is harder to 
troubleshoot and work with it since you don't see logs for a substantial 
portion of the process.  Since there isn't another person looking at developing 
on top of this vs. the resulting functionality, I wanted to bring that up.   We 
can log jiras to make it better or whatever.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
If someone is doing what the old workflow is ( just from the command line - 
not opening ambari until everything is done ) they will never see anything but 
the fail, please fix and try again.  I don't think we should have this be a 
'two parter', where the caller has to go hunting through different things in 
the vm and out of it to hunt things down.   The cadillac fix would be for a 
post build step that gathers the logs into a buildResult dir or something.  The 
maven output that is 'eaten', the *.txt files that the ambari ui refers to.

Also, if someone wanted to use automation with this, it will be a problem.

There is, to me, a change with this, from a modular build where we were 
doing multiple build steps, with each in and of itself being self documenting, 
to a monolithic build


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
I can offer a slightly different perspective on this PR, from the pov of 
someone working downstream ( i rebased on it ), and who has been working on 
adding things to the spec and install.

Many of my issues have been of my own doing ( trying and failing to use 
templates for spec's but not go all the way ), not doing everything in maven 
from the start since ansible is more flexible etc .

What I would like to bring up is the 'black box' nature of this build now.  
I don't see errors surface into the ansible log they way they would before.  

The build starts building, and while it is running all the maven commands 
there is no output, and no logging.   I have had to drop down into the rpm 
level and build there to see errors, or go into the vagrant image and hunt down 
logs to see problems with deployment ( a good tip is when the ansible log says 
it is deploying cluster, connect to ambari and monitor the ops, which tail the 
logs and give the file names, with are NOT var/log based ).

The point of this specific feedback is that we have added a very large 
amount of time to the vagrant up, during which many many things could go wrong, 
and none of them surface to the user.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
@dlyle65535 can you point to any documentation for the ambari api/functions 
that we are using?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
David, after fetch and merge of your latest I can no longer vagrant up 
full_dev

==> node1: Adding box 'new_base' (v0) for provider: virtualbox
node1: Downloading: 
/Users/dml/projects/metron-dlyle/metron-deployment/packer-build/builds/base-centos-6.7-2.1.20170303223924.git.33abe8cf13c347a2dfdece145a7b8c17f2a423c0_dirty.virtualbox.box
An error occurred while downloading the remote file. The error
message, if any, is reproduced below. Please fix this error and try
again.

Couldn't open file 
/Users/dml/projects/metron-dlyle/metron-deployment/packer-build/builds/base-centos-6.7-2.1.20170303223924.git.33abe8cf13c347a2dfdece145a7b8c17f2a423c0_dirty.virtualbox.box
➜  full-dev-platform git:(METRON-258-RPM) 




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
Well, if we don't want to put deployment things with the src, then ansible 
is a more flexible and easier to use tool for certain tasks too. But we will 
talk about it when I get it working


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
ok - when you review what I'm doing in the playbook/role we can talk about 
alternatives


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
Maybe we should add that line to the doc or create a script??


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
ansible-playbook -v -i "localhost," -c local playbooks/metron_build.yml

now it is running and i'll see about the errors



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
Do I have to create an inventory with my local machine name?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
 ansible-playbook -v playbooks/metron_build.yml  
Using 
/Users/ottofowler/src/apache/forks/incubator-metron/metron-deployment/ansible.cfg
 as config file
 [WARNING]: provided hosts list is empty, only localhost is available

 __ 
< PLAY >
 -- 
\   ^__^
 \  (oo)\___
(__)\   )\/\
||w |
|| ||

skipping: no hosts matched
  
< PLAY RECAP >
  
\   ^__^
 \  (oo)\___
(__)\   )\/\
||w |
|| ||



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
I need to run the playbook though. 
It doesn't match any hosts


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
can you share your command line?  Do you run it from /playbooks?  does you 
-i an inventory?  Did you copy or create an ansible.cfg?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
I want to run the playbook to just build the rpm's, not deploy.  So just 
metron_build


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
How are you running the playbook?  I cannot get it to execute


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #467: METRON-743: Sort the files when reading results...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/467
  
+1, and i'm now seeing this in my personal travis


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #468: METRON-744: Allow Stellar functions to b...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/468#discussion_r103708886
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/resolver/ClasspathFunctionResolver.java
 ---
@@ -65,24 +72,48 @@
  *
  */
 public class ClasspathFunctionResolver extends BaseFunctionResolver {
-
-  /**
-   * The key for a global property that defines one or more regular 
expressions
-   * that specify what should be included when searching for Stellar 
functions.
-   */
-  public static final String STELLAR_SEARCH_INCLUDES_KEY = 
"stellar.function.resolver.includes";
-
-  /**
+  public enum Config {
+STELLAR_VFS_PATHS("stellar.function.paths", ""),
+/**
+ * The key for a global property that defines one or more regular 
expressions
+ * that specify what should be included when searching for Stellar 
functions.
+ */
+STELLAR_SEARCH_INCLUDES_KEY("stellar.function.resolver.includes", ""),
--- End diff --

If we leave it for that purpose, let's rename it and document it as such so 
it is clear.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/468
  
Run the free up space script


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r103239678
  
--- Diff: metron-deployment/roles/java_jdk/defaults/main.yml ---
@@ -15,4 +15,4 @@
 #  limitations under the License.
 #
 ---
--- End diff --

Is there a reason not to go to the latest ( _121 ) if we are going to 
update?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r103238671
  
--- Diff: metron-deployment/roles/kibana/README.md ---
@@ -1,35 +0,0 @@
-Kibana 4
--- End diff --

The file I commented was the readme that documented how to add or modify 
the kibana page template


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r103235131
  
--- Diff: metron-deployment/roles/kibana/README.md ---
@@ -1,35 +0,0 @@
-Kibana 4
--- End diff --

Is this documented anywhere else?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #467: METRON-743: Sort the files when reading results...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/467
  
Thank you for clarifying, that makes sense.  Sorry to confuse the issue


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #467: METRON-743: Sort the files when reading results...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/467
  
There are two things here I *think*

1. The test assumptions and the failure on some platforms ( if this is 
indeed the problem )
2. What this means for the real use of the pcap query system

I think we can address the test first, and may have to discuss the 2.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #467: METRON-743: Sort the files when reading results...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/467
  
What are the performance penalties of doing this?

Part of the discussion in  
[HADOOP-12009](https://issues.apache.org/jira/browse/HADOOP-12009) that lead to 
the clarification of this behavior in the spec was about why they don't do this 
themselves for the performance penalty.

Could you get OOM errors or other issues running large queries that would 
have worked before?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #466: METRON-742: Generated code for profile window s...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/466
  
[INFO] 

[INFO] Reactor Summary:
[INFO] 
[INFO] Metron . SUCCESS [  
0.002 s]
[INFO] metron-analytics ... SUCCESS [  
0.000 s]
[INFO] metron-maas-common . SUCCESS [  
0.800 s]
[INFO] metron-platform  SUCCESS [  
0.001 s]
[INFO] metron-test-utilities .. SUCCESS [  
0.357 s]
[INFO] metron-integration-test  SUCCESS [  
2.171 s]
[INFO] metron-maas-service  SUCCESS [  
1.720 s]
[INFO] metron-common .. SUCCESS [ 
40.376 s]
[INFO] metron-statistics .. SUCCESS [01:03 
min]
[INFO] metron-hbase ... SUCCESS [01:11 
min]
[INFO] metron-profiler-common . SUCCESS [ 
10.194 s]
[INFO] metron-profiler-client . SUCCESS [01:30 
min]
[INFO] metron-profiler  SUCCESS [ 
29.689 s]
[INFO] metron-writer .. SUCCESS [ 
15.689 s]
[INFO] metron-enrichment .. SUCCESS [ 
49.701 s]
[INFO] metron-indexing  SUCCESS [ 
11.529 s]
[INFO] metron-solr  SUCCESS [ 
13.183 s]
[INFO] metron-pcap  SUCCESS [  
0.785 s]
[INFO] metron-parsers . SUCCESS [ 
48.027 s]
[INFO] metron-pcap-backend  SUCCESS [ 
29.235 s]
[INFO] metron-data-management . SUCCESS [01:11 
min]
[INFO] metron-api . SUCCESS [ 
26.331 s]
[INFO] metron-management .. SUCCESS [ 
39.091 s]
[INFO] elasticsearch-shaded ... SUCCESS [  
5.547 s]
[INFO] metron-elasticsearch ... SUCCESS [ 
41.378 s]
[INFO] metron-deployment .. SUCCESS [  
0.000 s]
[INFO] Metron Ambari Management Pack .. SUCCESS [  
0.148 s]
[INFO] metron-docker .. SUCCESS [  
2.782 s]
[INFO] 

[INFO] BUILD SUCCESS
[INFO] 

[INFO] Total time: 11:05 min
[INFO] Finished at: 2017-02-26T00:05:34-05:00
[INFO] Final Memory: 112M/2199M
[INFO] 

0:05 ottofowler@Winterfell 
/Users/ottofowler/src/apache/work/incubator-metron-pr-466
% git status
On branch pr-466
nothing to commit, working tree clean




+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #466: METRON-742: Generated code for profile window s...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/466
  
Does building or running the tests have anything to do with it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #466: METRON-742: Generated code for profile window s...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/466
  
So - the test is - build without changing anything after clone, and there 
should not be shown in git status right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #449: METRON-701 Triage Metrics Produced by the Profi...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/449
  
I'm sorry, no Nick I don't


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #449: METRON-701 Triage Metrics Produced by the Profi...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/449
  
I think routing from stellar within routing in storm is confusing.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/438
  
I think what we are saying is 1.  I think that is the best way to move 
forward.
Just make sure that this commit isn't merged into an RC for 0.3.1



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #462: METRON-734 Builds failing because of MaxMind DB...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/462
  
This build is not even starting in travis :(


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/438
  
I commented on 735 so it wins. #settled


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/438
  
let me know which one to comment on!



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/438
  
That makes sense, thanks for the explanation.
I am +1 for having SOLR specific flattening, although documenting well 
needs to be a priority.
Also, if we do this, please log a jira that the flattening in the json 
mapper needs to be optional by configuration


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/438
  
Nick - if you look at the JSONMapParser, Casey and I implemented a 
flattener that you would use for this.  So, all of our json stuff from that 
_was_ already flattened.  I am more confused about there being *non* flattened 
things going through actually.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #462: METRON-734 Builds failing because of MaxMind DB...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/462
  
+1, build and tests run (  mvn -q clean integration-test install && 
build_utils/verify_licenses.sh ), on machine that was failing every every time.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #461: METRON-733: Remove Geo database from ParserBolt

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/461
  
I am seeing this on my local branch, not the PR


On February 22, 2017 at 09:13:12, Nick Allen (notificati...@github.com)
wrote:

The build error does not seem related to your PR. I saw the same failure on
#450 <https://github.com/apache/incubator-metron/pull/450>.

The resource com.fasterxml:oss-parent:pom:28 really does not exist in Maven
Central, AFAIK. Hmm.

[ERROR] Failed to execute goal on project metron-enrichment: Could not
resolve dependencies
for project org.apache.metron:metron-enrichment:jar:0.3.1:
Failed to collect dependencies at com.maxmind.geoip2:geoip2:jar:2.8.0 ->
 com.maxmind.db:maxmind-db:jar:1.2.1 ->
com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT:
Failed to read artifact descriptor for
com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT:
Could not find artifact com.fasterxml:oss-parent:pom:28 in central
(http://repo.maven.apache.org/maven2) -> [Help 1]

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub

<https://github.com/apache/incubator-metron/pull/461#issuecomment-281679880>,
or mute the thread

<https://github.com/notifications/unsubscribe-auth/ABD1_98dtsK9jFWAsUhiM3fkRJZzHr_Kks5rfEJ3gaJpZM4MIH4d>
.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #461: METRON-733: Remove Geo database from ParserBolt

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/461
  
Does this mean you can get rid of the local geo database in the parsers 
resource dir?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #450: METRON-690: Create a DSL-based timestamp lookup...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/450
  
Honestly, I was thinking more about the 'kill everything' part, which seems 
to be repeated a lot


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #455: METRON-720 modify generate-md.sh to re-throw er...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/455
  
ok - i'm not building this branch, just what i've seen from other builds


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #455: METRON-720 modify generate-md.sh to re-throw er...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/455
  
re: "site.xml is now a fully auto-generated file"  that would be why it is 
showing in git as unstaged but modified?  
Is there a way we can stop that from happening?  If it is generated does it 
need to be in git?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #450: METRON-690: Create a DSL-based timestamp lookup...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/450
  
Not for nothing @cestella but those steps should just be a script if we are 
going to need them all the time


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #457: METRON-724 add [IN] to the documented ke...

[ottobackwards]

GitHub user ottobackwards opened a pull request:

https://github.com/apache/incubator-metron/pull/457

METRON-724 add [IN] to the documented keywords that require escaping

The keyword 'in' must be escaped to be used in Stellar, but is not 
documented as such.
This PR adds in the documentation and a unit test.
Verify:
- by running the tests
- by using the Stellar Console to evaluate an expression
- by building and viewing the documentation ( see below ) and the readme.md 
in github



Thank you for submitting a contribution to Apache Metron (Incubating).
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.
Please refer also to our [Build Verification 
guildlines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check
the following:

### For all changes:
- [ x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ n/a] Have you included steps to reproduce the behavior or problem that 
is being changed or addressed?
- [x ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x ] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:

```
mvn -q clean integration-test install && build_utils/verify_licenses.sh 
```

- [x ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [n/a ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [n/a ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
site-book/target/site/index.html.

```
cd site-book
bin/generate-md.sh
mvn site:site

```

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommened that [travis-ci](https://travis-ci.org) is set up for 
your personal repository such that your branches are built there before 
submitting a pull request.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/incubator-metron METRON-724

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/457.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #457


commit b90987bbeca346301a18aea2a7913a072fde4a3a
Author: Otto Fowler 
Date:   2017-02-17T17:06:58Z

METRON-724 add [IN] to the documented keywords that require escaping

add in unit test




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #456: METRON-721 Add github pull request template to ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/456
  
I put a reference to the verification doc at the top ' for smoke test guide 
' and then put a checkbox for running in vagrant - how's that?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #456: METRON-721 Add github pull request template to ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/456
  
So - why don't we come up with a cut down version of that, maybe restricted 
to trying quick or full dev?  or reference and link like we do with the full 
dev guide?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #456: METRON-721 Add github pull request templ...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/456#discussion_r101297815
  
--- Diff: .github/PULL_REQUEST_TEMPLATE.md ---
@@ -0,0 +1,40 @@
+Thank you for submitting a contribution to Apache Metron (Incubating).
+Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions
+
+In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check
+the following:
+
+### For all changes:
+- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel)
 
+
+- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
+
+- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
+
+
+### For code changes:
+- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
+- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
+- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
+
+```
+mvn -q clean integration-test install && build_utils/verify_licenses.sh 
+```
+
+- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
+- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
+
+### For documentation related changes:
+- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book?
+
+```
+cd site-book
+bin/generate-md.sh
+mvn site:site
+
+view changes via target/site/index.html
--- End diff --

done



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #456: METRON-721 Add github pull request template to ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/456
  
Thanks for the review!

I am not sure about verifying builds.  I don't think we should put 
everything that is in the Dev Guide or the Verification in the PR template, 
there is some balance to be had there.  I don't think we expect every PR to 
have validated the builds anyways.

I did make the jira change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #456: METRON-721 Add github pull request templ...

[ottobackwards]

GitHub user ottobackwards opened a pull request:

https://github.com/apache/incubator-metron/pull/456

METRON-721 Add github pull request template to metron

The idea for this template is to hit some high level things to remind 
submitters of while referencing the comprehensive development guidelines 
document.

[Github 
templates](https://help.github.com/articles/creating-a-pull-request-template-for-your-repository/)
 can only be picked up off of the main branch, so aside from checking the 
markdown formatting, I do not know what other testing can be done

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/incubator-metron METRON-721

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/456.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #456


commit 3efd8e9af2fd141da8bdd48b019ce2fcebb01911
Author: Otto Fowler 
Date:   2017-02-15T12:03:49Z

add github pull request template

the idea for this template is to hit some high level things to remind 
submitters of

while referencing the comprehensive development guidelines document

we don't want this to be too large a document




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #452: Removed MySQL from Enrichment Diagram

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/452
  
Please rename the PR to start with "METRON-715".  Then we can land this


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #454: METRON-716 site-book README.md

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/454
  
@mattf-horton are you ok on this after the revisions?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #454: METRON-716 site-book README.md

[ottobackwards]

GitHub user ottobackwards opened a pull request:

https://github.com/apache/incubator-metron/pull/454

METRON-716 site-book README.md

Initial draft of a readme for site-book documentation.

Questions:  should we list the tools used more explicitly?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/incubator-metron METRON-716

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/454.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #454


commit f56190441f4751bc2493b49b1495258ed0a25c5b
Author: Otto Fowler 
Date:   2017-02-13T20:54:26Z

initial draft of site-book README.md




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #453: METRON-694: Index Errors from Topologies

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/453
  
@cestella - how can we utilize the error indexing if we were going to say - 
output errors or warnings that there were deprecated stellar statements?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #453: METRON-694: Index Errors from Topologies

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/453
  
What comes to mind is the 'source' of an error.  Is this error wrong 
because METRON thinks it is invalid, or it it wrong because of some other 
configuration specific evaluation.  I don't think we do this now with stellar   
(   IF (BUSINESS_RULE(rule_valid_field,field1) == FALSE) DROP_ERROR()) however.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #453: METRON-694: Index Errors from Topologies

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/453
  
That makes complete sense, we should call that stuff out pre-review.  I 
think what we are seeing is that folks have some really good ideas and are 
willing to contribute to documentation on this project, we should take 
advantage.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #453: METRON-694: Index Errors from Topologies

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/453#discussion_r100820671
  
--- Diff: 
metron-platform/metron-indexing/src/main/config/zookeeper/indexing/error.json 
---
@@ -0,0 +1,17 @@
+{
+  "hdfs" : {
--- End diff --

error is too generic.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #453: METRON-694: Index Errors from Topologies

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/453#discussion_r100813285
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
 ---
@@ -37,6 +37,12 @@
 Metron apps indexed HDFS dir
 
 
--- End diff --

Should these HDSF directories more specifically pertain to topology 
processing?  As the other use cases came into being, just 'error' may be too 
generic


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #453: METRON-694: Index Errors from Topologies

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/453#discussion_r100817317
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/config/elasticsearch_error.properties
 ---
@@ -0,0 +1,69 @@
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
--- End diff --

This is duplicated in the MPACK isn't it?  Is there not a way to have this 
defined once and used in multiple places?   


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #453: METRON-694: Index Errors from Topologies

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/453#discussion_r100815478
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/error/MetronError.java
 ---
@@ -0,0 +1,200 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.error;
+
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.commons.lang.exception.ExceptionUtils;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.Constants.ErrorType;
+import org.json.simple.JSONObject;
+
+import java.net.InetAddress;
+import java.net.UnknownHostException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.metron.common.Constants.ErrorFields;
+
+public class MetronError {
+
+  private String message;
+  private Throwable throwable;
+  private String sensorType = "error";
+  private ErrorType errorType = ErrorType.DEFAULT_ERROR;
+  private Set errorFields;
+  private List rawMessages;
+
+  public MetronError withMessage(String message) {
+this.message = message;
+return this;
+  }
+
+  public MetronError withThrowable(Throwable throwable) {
+this.throwable = throwable;
+return this;
+  }
+
+  public MetronError withSensorType(String sensorType) {
+this.sensorType = sensorType;
+return this;
+  }
+
+  public MetronError withErrorType(ErrorType errorType) {
+this.errorType = errorType;
+return this;
+  }
+
+  public MetronError withErrorFields(Set errorFields) {
+this.errorFields = errorFields;
+return this;
+  }
+
+
+  public MetronError addRawMessage(Object rawMessage) {
+if (rawMessage != null) {
+  if (this.rawMessages == null) {
+this.rawMessages = new ArrayList<>();
+  }
+  this.rawMessages.add(rawMessage);
+}
+return this;
+  }
+
+  public MetronError withRawMessages(List rawMessages) {
+this.rawMessages = rawMessages;
+return this;
+  }
+
+  public Optional getThrowable() {
+return throwable != null ? Optional.of(throwable) : Optional.empty();
+  }
+
+  @SuppressWarnings({"unchecked"})
+  public JSONObject getJSONObject() {
+JSONObject errorMessage = new JSONObject();
+errorMessage.put(Constants.SENSOR_TYPE, "error");
+
+   /*
+ * Save full stack trace in object.
--- End diff --

Instead of having a method with multiple null checks
Maybe have a method that just calls other methods, and put the checks in 
there

handleThrowable(errorMessage);
handleRawMessages(errorMessage);
etc etc



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---