[jira] [Commented] (METRON-1053) Relocate Metron Docker

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099413#comment-16099413
 ] 

ASF GitHub Bot commented on METRON-1053:


Github user kylerichardson commented on the issue:

https://github.com/apache/metron/pull/659
  
Found a couple of issues when running from a clean source tree. A few quick 
mods to the pom and one Dockerfile. I'll submit a PR on your branch in the 
morning.


> Relocate Metron Docker
> --
>
> Key: METRON-1053
> URL: https://issues.apache.org/jira/browse/METRON-1053
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Nick Allen
>Priority: Minor
> Fix For: 0.4.1
>
>
> Having metron-docker at the top-level of the project seems to catch the 
> attention of new users.  Some then start using metron-docker to 
> explore/try-out/demo Metron.  
> The metron-docker code that we have is not well-suited for this purpose.  It 
> is only really useful for development.  It is not regularly tested and 
> maintained like our Vagrant environment.
> I am proposing that we move the top-level "metron-docker" directory to 
> another location to avoid confusing new users.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1005) Create Decodable Row Key for Profiler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099368#comment-16099368
 ] 

ASF GitHub Bot commented on METRON-1005:


Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129187853
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,402 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(PROFILER_SALT_DIVISOR.getDefault(Integer.class),
+PROFILER_PERIOD.getDefault(Long.class),
+
TimeUnit.valueOf(PROFILER_PERIOD_UNITS.getDefault(String.class)));
+  }
+
+  public 

[jira] [Commented] (METRON-1005) Create Decodable Row Key for Profiler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099365#comment-16099365
 ] 

ASF GitHub Bot commented on METRON-1005:


Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129186527
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,402 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(PROFILER_SALT_DIVISOR.getDefault(Integer.class),
+PROFILER_PERIOD.getDefault(Long.class),
+
TimeUnit.valueOf(PROFILER_PERIOD_UNITS.getDefault(String.class)));
+  }
+
+  public 

[jira] [Commented] (METRON-1005) Create Decodable Row Key for Profiler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099367#comment-16099367
 ] 

ASF GitHub Bot commented on METRON-1005:


Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129193892
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,382 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(1000, 15, TimeUnit.MINUTES);
+  }
+
+  public DecodableRowKeyBuilder(int saltDivisor, long duration, TimeUnit 
units) {
+this.saltDivisor = saltDivisor;
+this.periodDurationMillis = units.toMillis(duration);
+  }
+
+  /**
+   * Builds a list of row keys necessary to retrieve profile measurements 
over
+   * a time horizon.
+   *
+   * @param profile The name of the profile.
+   * @param entity The name of the entity.
+   * @param groups The group(s) used to sort 

[jira] [Commented] (METRON-1005) Create Decodable Row Key for Profiler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099366#comment-16099366
 ] 

ASF GitHub Bot commented on METRON-1005:


Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129190841
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,402 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(PROFILER_SALT_DIVISOR.getDefault(Integer.class),
+PROFILER_PERIOD.getDefault(Long.class),
+
TimeUnit.valueOf(PROFILER_PERIOD_UNITS.getDefault(String.class)));
+  }
+
+  public 

[jira] [Commented] (METRON-1060) Add performance timing logging to enrichment topology

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099148#comment-16099148
 ] 

ASF GitHub Bot commented on METRON-1060:


GitHub user mmiklavc opened a pull request:

https://github.com/apache/metron/pull/665

METRON-1060: Add performance timing logging to enrichment topology

https://issues.apache.org/jira/browse/METRON-1060

Adds performance logging timers to the enrichment topology. I've made this 
configurable so that the timers are separate from our standard logging. Any 
opinions on documenting this, ie should it be added to the enrichment docs? I 
also noticed some performance logging added to the StellarAdapter class, but we 
haven't added documentation for the "feature."

### Testing

Run full dev
```
cd metron/metron-deployment/vagrant/full-dev-platform
vagrant up
```

**Set the log levels**

Go to the storm UI - http://node1:8744 and select the enrichment topology. 
Scroll down to "Change Log Level" and enter the following loggers, level=DEBUG, 
timeout=5000 sec:

> org.apache.metron.enrichment.bolt.GenericEnrichmentBolt$Perf
> org.apache.metron.enrichment.bolt.SplitBolt$Perf
> org.apache.metron.enrichment.bolt.JoinBolt$Perf 

In the enrichment worker log you should see lines similar to the following 
indicating that performance logging has been enabled:
> 2017-07-21 20:53:42.673 o.a.s.d.worker [INFO] Adding config for: 
org.apache.metron.enrichment.bolt.SplitBolt$Perf with level: DEBUG
> 2017-07-21 20:54:14.747 o.a.s.d.worker [INFO] Adding config for: 
org.apache.metron.enrichment.bolt.JoinBolt$Perf with level: DEBUG
> 2017-07-21 20:54:39.914 o.a.s.d.worker [INFO] Adding config for: 
org.apache.metron.enrichment.bolt.GenericEnrichmentBolt$Perf with level: DEBUG

**Check the log output**

> You should see log entries similar to the following:
> 2017-07-24 21:13:05.171 o.a.m.e.b.GenericEnrichmentBolt$Perf [DEBUG] 
key=4fe7c88a-79bd-4889-ac79-ad7a0d494291, enrich type=host, enrich time (ms): 0
> 2017-07-24 21:13:05.171 o.a.m.e.b.GenericEnrichmentBolt$Perf [DEBUG] 
key=4fe7c88a-79bd-4889-ac79-ad7a0d494291, enrich type=geo, enrich time (ms): 0
> 2017-07-24 21:13:05.171 o.a.m.e.b.GenericEnrichmentBolt$Perf [DEBUG] 
key=4fe7c88a-79bd-4889-ac79-ad7a0d494291, enrich type=host, enrich time (ms): 0
> 2017-07-24 21:13:05.171 o.a.m.e.b.GenericEnrichmentBolt$Perf [DEBUG] 
key=4fe7c88a-79bd-4889-ac79-ad7a0d494291, enrich type=geo, enrich time (ms): 0
> 2017-07-24 21:13:05.171 o.a.m.e.b.GenericEnrichmentBolt$Perf [DEBUG] 
key=4fe7c88a-79bd-4889-ac79-ad7a0d494291, execute() time (ms): 0
> 2017-07-24 21:13:05.171 o.a.m.e.b.GenericEnrichmentBolt$Perf [DEBUG] 
key=4fe7c88a-79bd-4889-ac79-ad7a0d494291, execute() time (ms): 0

That should do it!

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your 

[jira] [Created] (METRON-1060) Add performance timing logging to enrichment topology

[Michael Miklavcic (JIRA)]

Michael Miklavcic created METRON-1060:
-

 Summary: Add performance timing logging to enrichment topology
 Key: METRON-1060
 URL: https://issues.apache.org/jira/browse/METRON-1060
 Project: Metron
  Issue Type: Improvement
Reporter: Michael Miklavcic
Assignee: Michael Miklavcic






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1056) Get field types from Elasticsearch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099136#comment-16099136
 ] 

ASF GitHub Bot commented on METRON-1056:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/662#discussion_r129156078
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/search/FieldType.java
 ---
@@ -0,0 +1,52 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.indexing.dao.search;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+public enum FieldType {
+  @JsonProperty("string")
+  STRING("string"),
+  @JsonProperty("ip")
+  IP("ip"),
+  @JsonProperty("integer")
+  INTEGER("integer"),
+  @JsonProperty("long")
+  LONG("long"),
+  @JsonProperty("date")
+  DATE("date"),
+  @JsonProperty("float")
+  FLOAT("float"),
+  @JsonProperty("double")
+  DOUBLE("double"),
+  @JsonProperty("boolean")
+  BOOLEAN("boolean"),
+  @JsonProperty("other")
--- End diff --

Just to be clear, I don't think this should hold up this PR, but that we in 
a general sense need to go into more detail.


> Get field types from Elasticsearch
> --
>
> Key: METRON-1056
> URL: https://issues.apache.org/jira/browse/METRON-1056
> Project: Metron
>  Issue Type: Improvement
>Reporter: Ryan Merriman
>
> We need a way to retrieve field types from Elasticsearch for a specified list 
> of indices.  This will also be exposed through our indexing DAO layer with 
> Elasticsearch being the initial implementation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1056) Get field types from Elasticsearch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099117#comment-16099117
 ] 

ASF GitHub Bot commented on METRON-1056:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/662#discussion_r129152107
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/search/FieldType.java
 ---
@@ -0,0 +1,52 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.indexing.dao.search;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+public enum FieldType {
+  @JsonProperty("string")
+  STRING("string"),
+  @JsonProperty("ip")
+  IP("ip"),
+  @JsonProperty("integer")
+  INTEGER("integer"),
+  @JsonProperty("long")
+  LONG("long"),
+  @JsonProperty("date")
+  DATE("date"),
+  @JsonProperty("float")
+  FLOAT("float"),
+  @JsonProperty("double")
+  DOUBLE("double"),
+  @JsonProperty("boolean")
+  BOOLEAN("boolean"),
+  @JsonProperty("other")
--- End diff --

At some point, if we are going to abstract these other systems, we need to 
document our abstractions.  I am not sure where though.  Like stellar, the rest 
api documentation prob. needs to be broken out with some better detail for 
specific entry points, including references to outside systems the user is 
expected to know.


> Get field types from Elasticsearch
> --
>
> Key: METRON-1056
> URL: https://issues.apache.org/jira/browse/METRON-1056
> Project: Metron
>  Issue Type: Improvement
>Reporter: Ryan Merriman
>
> We need a way to retrieve field types from Elasticsearch for a specified list 
> of indices.  This will also be exposed through our indexing DAO layer with 
> Elasticsearch being the initial implementation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1056) Get field types from Elasticsearch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099091#comment-16099091
 ] 

ASF GitHub Bot commented on METRON-1056:


Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/662#discussion_r129147554
  
--- Diff: 
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/IndexingDaoIntegrationTest.java
 ---
@@ -27,28 +28,32 @@
 import org.json.simple.parser.ParseException;
 import org.junit.*;
 
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
 import java.util.List;
+import java.util.Map;
 
--- End diff --

"location_point" is of type geo_field so it's falls into the "other" 
category.


> Get field types from Elasticsearch
> --
>
> Key: METRON-1056
> URL: https://issues.apache.org/jira/browse/METRON-1056
> Project: Metron
>  Issue Type: Improvement
>Reporter: Ryan Merriman
>
> We need a way to retrieve field types from Elasticsearch for a specified list 
> of indices.  This will also be exposed through our indexing DAO layer with 
> Elasticsearch being the initial implementation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1056) Get field types from Elasticsearch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099089#comment-16099089
 ] 

ASF GitHub Bot commented on METRON-1056:


Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/662#discussion_r129147257
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/search/FieldType.java
 ---
@@ -0,0 +1,52 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.indexing.dao.search;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+public enum FieldType {
+  @JsonProperty("string")
+  STRING("string"),
+  @JsonProperty("ip")
+  IP("ip"),
+  @JsonProperty("integer")
+  INTEGER("integer"),
+  @JsonProperty("long")
+  LONG("long"),
+  @JsonProperty("date")
+  DATE("date"),
+  @JsonProperty("float")
+  FLOAT("float"),
+  @JsonProperty("double")
+  DOUBLE("double"),
+  @JsonProperty("boolean")
+  BOOLEAN("boolean"),
+  @JsonProperty("other")
--- End diff --

"other" just means it's not one of the other simple types in the enum.  The 
Elasticsearch Java API returns the search result as a Map type 
which is also the return type of the controller so it's up to Jackson to 
serialize it.  We are not doing anything special to map values to types when 
returning search results.  So far I haven't seen Jackson have any problems with 
the types being returned.  From what I can tell Elasticsearch mostly sticks to 
simple types for the values.


> Get field types from Elasticsearch
> --
>
> Key: METRON-1056
> URL: https://issues.apache.org/jira/browse/METRON-1056
> Project: Metron
>  Issue Type: Improvement
>Reporter: Ryan Merriman
>
> We need a way to retrieve field types from Elasticsearch for a specified list 
> of indices.  This will also be exposed through our indexing DAO layer with 
> Elasticsearch being the initial implementation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1056) Get field types from Elasticsearch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099083#comment-16099083
 ] 

ASF GitHub Bot commented on METRON-1056:


Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/662#discussion_r129145507
  
--- Diff: 
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/SearchController.java
 ---
@@ -45,4 +49,18 @@
   ResponseEntity search(final @ApiParam(name = 
"searchRequest", value = "Search request", required = true) @RequestBody 
SearchRequest searchRequest) throws RestException {
 return new ResponseEntity<>(searchService.search(searchRequest), 
HttpStatus.OK);
   }
+
--- End diff --

It's a legitimate concern but I think synchronous is ok here because 
Elasticsearch is intended to provide results in real-time.  The http javascript 
APIs we use are asynchronous so it shouldn't make a difference in the UI.  If 
this were a batch operation (Hive, Map Reduce) I would agree that it should be 
an async call.


> Get field types from Elasticsearch
> --
>
> Key: METRON-1056
> URL: https://issues.apache.org/jira/browse/METRON-1056
> Project: Metron
>  Issue Type: Improvement
>Reporter: Ryan Merriman
>
> We need a way to retrieve field types from Elasticsearch for a specified list 
> of indices.  This will also be exposed through our indexing DAO layer with 
> Elasticsearch being the initial implementation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1056) Get field types from Elasticsearch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099072#comment-16099072
 ] 

ASF GitHub Bot commented on METRON-1056:


Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/662#discussion_r129143530
  
--- Diff: metron-interface/metron-rest/README.md ---
@@ -353,6 +355,20 @@ Request and Response objects are JSON formatted.  The 
JSON schemas are available
   * searchRequest - Search request
   * Returns:
 * 200 - Search results
+
--- End diff --

This isn't intended to be the source of truth for types that we support.  I 
would think any data type someone may need should be supported in some way.  
This is restricted to (mostly) simple types and is driven by the need to format 
simple values in different ways.  Anything else besides simple types is 
categorized as "other".  If at any point there is a need to include more types 
this can easily be extended.  

The Elasticsearch API gives you either a number or string (only string in 
the case of the REST API) so we need more information about the type in 
Elasticsearch to format it correctly.  Solr is similar.  I will take another 
pass at the documentation to make this more obvious.


> Get field types from Elasticsearch
> --
>
> Key: METRON-1056
> URL: https://issues.apache.org/jira/browse/METRON-1056
> Project: Metron
>  Issue Type: Improvement
>Reporter: Ryan Merriman
>
> We need a way to retrieve field types from Elasticsearch for a specified list 
> of indices.  This will also be exposed through our indexing DAO layer with 
> Elasticsearch being the initial implementation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1059) address checkstyle warning AvoidStarImport in metron-stellar

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099040#comment-16099040
 ] 

ASF GitHub Bot commented on METRON-1059:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/664
  
+1 Thanks for the contribution


> address checkstyle warning AvoidStarImport in metron-stellar
> 
>
> Key: METRON-1059
> URL: https://issues.apache.org/jira/browse/METRON-1059
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
> Environment: Mac OSX 10.12.5
>Reporter: Artem Ervits
>Priority: Trivial
>  Labels: newbie
> Fix For: Next + 1
>
>
> replace star notation with explicit imports



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1059) address checkstyle warning AvoidStarImport in metron-stellar

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099038#comment-16099038
 ] 

ASF GitHub Bot commented on METRON-1059:


Github user dbist commented on the issue:

https://github.com/apache/metron/pull/664
  
@nickwallen @ottobackwards separated by new line and placed them in 
separated blocks.


> address checkstyle warning AvoidStarImport in metron-stellar
> 
>
> Key: METRON-1059
> URL: https://issues.apache.org/jira/browse/METRON-1059
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
> Environment: Mac OSX 10.12.5
>Reporter: Artem Ervits
>Priority: Trivial
>  Labels: newbie
> Fix For: Next + 1
>
>
> replace star notation with explicit imports



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1059) address checkstyle warning AvoidStarImport in metron-stellar

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098967#comment-16098967
 ] 

ASF GitHub Bot commented on METRON-1059:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/664
  
I agree with you @ottobackwards .   We typically have a block of imports, 
then a blank line then a block of static imports with no wildcards.  

Our current development guidelines say to follow Sun/Oracle Java style 
guidelines.  For the Java style guidelines I cannot find enough details for 
imports and specifically static imports.  I know those standards are close-ish 
to Google's Java standards, which specifically has guidelines around this.


https://google.github.io/styleguide/javaguide.html#s3.3.3-import-ordering-and-spacing


> address checkstyle warning AvoidStarImport in metron-stellar
> 
>
> Key: METRON-1059
> URL: https://issues.apache.org/jira/browse/METRON-1059
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
> Environment: Mac OSX 10.12.5
>Reporter: Artem Ervits
>Priority: Trivial
>  Labels: newbie
> Fix For: Next + 1
>
>
> replace star notation with explicit imports



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1059) address checkstyle warning AvoidStarImport in metron-stellar

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098942#comment-16098942
 ] 

ASF GitHub Bot commented on METRON-1059:


Github user dbist commented on the issue:

https://github.com/apache/metron/pull/664
  
I would agree it would be better for readability, let me know and I'll 
generate another commit.


> address checkstyle warning AvoidStarImport in metron-stellar
> 
>
> Key: METRON-1059
> URL: https://issues.apache.org/jira/browse/METRON-1059
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
> Environment: Mac OSX 10.12.5
>Reporter: Artem Ervits
>Priority: Trivial
>  Labels: newbie
> Fix For: Next + 1
>
>
> replace star notation with explicit imports



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1059) address checkstyle warning AvoidStarImport in metron-stellar

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098937#comment-16098937
 ] 

ASF GitHub Bot commented on METRON-1059:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/664
  
That is better.  I would just like some confirmation from someone else ( 
@cestella, @nickwallen ).  I *think* that we want the static imports separate, 
and after the other imports.  That is what I've seen in other files.


> address checkstyle warning AvoidStarImport in metron-stellar
> 
>
> Key: METRON-1059
> URL: https://issues.apache.org/jira/browse/METRON-1059
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
> Environment: Mac OSX 10.12.5
>Reporter: Artem Ervits
>Priority: Trivial
>  Labels: newbie
> Fix For: Next + 1
>
>
> replace star notation with explicit imports



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1059) address checkstyle warning AvoidStarImport in metron-stellar

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098931#comment-16098931
 ] 

ASF GitHub Bot commented on METRON-1059:


Github user dbist commented on the issue:

https://github.com/apache/metron/pull/664
  
@ottobackwards organized the imports, thanks


> address checkstyle warning AvoidStarImport in metron-stellar
> 
>
> Key: METRON-1059
> URL: https://issues.apache.org/jira/browse/METRON-1059
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
> Environment: Mac OSX 10.12.5
>Reporter: Artem Ervits
>Priority: Trivial
>  Labels: newbie
> Fix For: Next + 1
>
>
> replace star notation with explicit imports



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (METRON-1059) address checkstyle warning AvoidStarImport in metron-stellar

[Artem Ervits (JIRA)]

Artem Ervits created METRON-1059:


 Summary: address checkstyle warning AvoidStarImport in 
metron-stellar
 Key: METRON-1059
 URL: https://issues.apache.org/jira/browse/METRON-1059
 Project: Metron
  Issue Type: Bug
Affects Versions: 0.4.1
 Environment: Mac OSX 10.12.5
Reporter: Artem Ervits
Priority: Trivial
 Fix For: Next + 1


replace star notation with explicit imports



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1056) Get field types from Elasticsearch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098508#comment-16098508
 ] 

ASF GitHub Bot commented on METRON-1056:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/662#discussion_r129052380
  
--- Diff: metron-interface/metron-rest/README.md ---
@@ -353,6 +355,20 @@ Request and Response objects are JSON formatted.  The 
JSON schemas are available
   * searchRequest - Search request
   * Returns:
 * 200 - Search results
+
--- End diff --

Such as complex types.

This comes through here, because we are creating our own Field Type Enums


> Get field types from Elasticsearch
> --
>
> Key: METRON-1056
> URL: https://issues.apache.org/jira/browse/METRON-1056
> Project: Metron
>  Issue Type: Improvement
>Reporter: Ryan Merriman
>
> We need a way to retrieve field types from Elasticsearch for a specified list 
> of indices.  This will also be exposed through our indexing DAO layer with 
> Elasticsearch being the initial implementation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1056) Get field types from Elasticsearch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098505#comment-16098505
 ] 

ASF GitHub Bot commented on METRON-1056:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/662#discussion_r129051676
  
--- Diff: metron-interface/metron-rest/README.md ---
@@ -353,6 +355,20 @@ Request and Response objects are JSON formatted.  The 
JSON schemas are available
   * searchRequest - Search request
   * Returns:
 * 200 - Search results
+
--- End diff --

This is not a problem for this PR, but rather the documentation in general. 
 The actual TYPES returned from the rest calls are not documented at all.


> Get field types from Elasticsearch
> --
>
> Key: METRON-1056
> URL: https://issues.apache.org/jira/browse/METRON-1056
> Project: Metron
>  Issue Type: Improvement
>Reporter: Ryan Merriman
>
> We need a way to retrieve field types from Elasticsearch for a specified list 
> of indices.  This will also be exposed through our indexing DAO layer with 
> Elasticsearch being the initial implementation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1056) Get field types from Elasticsearch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098503#comment-16098503
 ] 

ASF GitHub Bot commented on METRON-1056:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/662#discussion_r129054322
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/search/FieldType.java
 ---
@@ -0,0 +1,52 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.indexing.dao.search;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+public enum FieldType {
+  @JsonProperty("string")
+  STRING("string"),
+  @JsonProperty("ip")
+  IP("ip"),
+  @JsonProperty("integer")
+  INTEGER("integer"),
+  @JsonProperty("long")
+  LONG("long"),
+  @JsonProperty("date")
+  DATE("date"),
+  @JsonProperty("float")
+  FLOAT("float"),
+  @JsonProperty("double")
+  DOUBLE("double"),
+  @JsonProperty("boolean")
+  BOOLEAN("boolean"),
+  @JsonProperty("other")
--- End diff --

What does other mean?  Is it mapped to string in return values.. 


> Get field types from Elasticsearch
> --
>
> Key: METRON-1056
> URL: https://issues.apache.org/jira/browse/METRON-1056
> Project: Metron
>  Issue Type: Improvement
>Reporter: Ryan Merriman
>
> We need a way to retrieve field types from Elasticsearch for a specified list 
> of indices.  This will also be exposed through our indexing DAO layer with 
> Elasticsearch being the initial implementation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1056) Get field types from Elasticsearch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098507#comment-16098507
 ] 

ASF GitHub Bot commented on METRON-1056:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/662#discussion_r129055024
  
--- Diff: 
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/IndexingDaoIntegrationTest.java
 ---
@@ -27,28 +28,32 @@
 import org.json.simple.parser.ParseException;
 import org.junit.*;
 
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
 import java.util.List;
+import java.util.Map;
 
--- End diff --

How do we test unknown


> Get field types from Elasticsearch
> --
>
> Key: METRON-1056
> URL: https://issues.apache.org/jira/browse/METRON-1056
> Project: Metron
>  Issue Type: Improvement
>Reporter: Ryan Merriman
>
> We need a way to retrieve field types from Elasticsearch for a specified list 
> of indices.  This will also be exposed through our indexing DAO layer with 
> Elasticsearch being the initial implementation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1058) address checkstyle warnings for UnusedImports in metron-stellar

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098500#comment-16098500
 ] 

ASF GitHub Bot commented on METRON-1058:


GitHub user dbist opened a pull request:

https://github.com/apache/metron/pull/663

METRON-1058 address checkstyle warnings for UnusedImports in metron-stellar

## Contributor Comments
[Please place any comments here.  A description of the problem/enhancement, 
how to reproduce the issue, your testing methodology, etc.]


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dbist/metron METRON-1058

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/663.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #663


commit 0ffc362158966b6d3921e0f165503a3b9b0b8f50
Author: Artem Ervits 
Date:   2017-07-24T14:01:01Z

METRON-1058

commit 095cc94159c0a18490e46d8588cfd993574d1537
Author: Artem Ervits 
Date:   2017-07-24T14:36:22Z

METRON-1058




> address checkstyle warnings for UnusedImports in metron-stellar
> ---
>
> Key: METRON-1058
> URL: https://issues.apache.org/jira/browse/METRON-1058
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
> Environment: Mac OSX 10.12.5
> Maven 3.5.0
> JDK 1.8.0_141
>Reporter: Artem Ervits
>Priority: Minor
>  Labels: newbie
> Fix For: Next + 1
>
> Attachments: METRON-1058-0.patch
>
>
> addressing UnusedImports warning in checkstyle report for metron-stellar



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1053) Relocate Metron Docker

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098459#comment-16098459
 ] 

ASF GitHub Bot commented on METRON-1053:


Github user kylerichardson commented on the issue:

https://github.com/apache/metron/pull/659
  
Going to run this up now. I'll get my feedback in today.

On Mon, Jul 24, 2017 at 10:02 AM, JonZeolla 
wrote:

> I took a high level look when it first came out - no concerns. I'm mobile
> only until Thursday so don't hold this up based on me.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



> Relocate Metron Docker
> --
>
> Key: METRON-1053
> URL: https://issues.apache.org/jira/browse/METRON-1053
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Nick Allen
>Priority: Minor
> Fix For: 0.4.1
>
>
> Having metron-docker at the top-level of the project seems to catch the 
> attention of new users.  Some then start using metron-docker to 
> explore/try-out/demo Metron.  
> The metron-docker code that we have is not well-suited for this purpose.  It 
> is only really useful for development.  It is not regularly tested and 
> maintained like our Vagrant environment.
> I am proposing that we move the top-level "metron-docker" directory to 
> another location to avoid confusing new users.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1053) Relocate Metron Docker

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098458#comment-16098458
 ] 

ASF GitHub Bot commented on METRON-1053:


Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/659
  
I took a high level look when it first came out - no concerns.  I'm mobile 
only until Thursday so don't hold this up based on me.


> Relocate Metron Docker
> --
>
> Key: METRON-1053
> URL: https://issues.apache.org/jira/browse/METRON-1053
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Nick Allen
>Priority: Minor
> Fix For: 0.4.1
>
>
> Having metron-docker at the top-level of the project seems to catch the 
> attention of new users.  Some then start using metron-docker to 
> explore/try-out/demo Metron.  
> The metron-docker code that we have is not well-suited for this purpose.  It 
> is only really useful for development.  It is not regularly tested and 
> maintained like our Vagrant environment.
> I am proposing that we move the top-level "metron-docker" directory to 
> another location to avoid confusing new users.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1050) Improve Docs of `profiler.period.duration`

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098456#comment-16098456
 ] 

ASF GitHub Bot commented on METRON-1050:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/656
  
bump.  Can anyone offer a binding +1?  Would like to get this in.


> Improve Docs of `profiler.period.duration` 
> ---
>
> Key: METRON-1050
> URL: https://issues.apache.org/jira/browse/METRON-1050
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.1
>
>
> It is a common mistake for new user's of the Profiler, to alter the 
> Profiler's `profiler.period.duration` property, but not alter the 
> corresponding `profiler.client.period.duration` property.  The new user is 
> then unable to read the profile data as they would expect.
> More should be done, but this JIRA focuses on improving the documentation 
> around this common error experienced by new users.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1053) Relocate Metron Docker

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098454#comment-16098454
 ] 

ASF GitHub Bot commented on METRON-1053:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/659
  
@ottobackwards @JonZeolla @kylerichardson  You guys all chimed in on the 
original email chain.  I'd like to get your eyes on this. Does this PR work for 
you?


> Relocate Metron Docker
> --
>
> Key: METRON-1053
> URL: https://issues.apache.org/jira/browse/METRON-1053
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Nick Allen
>Priority: Minor
> Fix For: 0.4.1
>
>
> Having metron-docker at the top-level of the project seems to catch the 
> attention of new users.  Some then start using metron-docker to 
> explore/try-out/demo Metron.  
> The metron-docker code that we have is not well-suited for this purpose.  It 
> is only really useful for development.  It is not regularly tested and 
> maintained like our Vagrant environment.
> I am proposing that we move the top-level "metron-docker" directory to 
> another location to avoid confusing new users.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1056) Get field types from Elasticsearch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098449#comment-16098449
 ] 

ASF GitHub Bot commented on METRON-1056:


Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/662
  
Sure @ottobackwards.  There is a similar api for Solr that will return 
field types for a list of collections.  When we add Solr as an indexing option 
we will create a "SolrDao" class that will implement the 
"IndexDao.getColumnMetadata(List indices)" and 
"IndexDao.getCommonColumnMetadata(List indices)" methods.


> Get field types from Elasticsearch
> --
>
> Key: METRON-1056
> URL: https://issues.apache.org/jira/browse/METRON-1056
> Project: Metron
>  Issue Type: Improvement
>Reporter: Ryan Merriman
>
> We need a way to retrieve field types from Elasticsearch for a specified list 
> of indices.  This will also be exposed through our indexing DAO layer with 
> Elasticsearch being the initial implementation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1054) Ambari Mpack Attempts to Kill Topologies That Are Not Running

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098433#comment-16098433
 ] 

ASF GitHub Bot commented on METRON-1054:


Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/660


> Ambari Mpack Attempts to Kill Topologies That Are Not Running
> -
>
> Key: METRON-1054
> URL: https://issues.apache.org/jira/browse/METRON-1054
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.1
>
>
> The Ambari Metron MPack attempts to kill topologies that are not running.  It 
> also attempts to start topologies that are already running.  It reports these 
> conditions as errors.
> The MPack should not report errors when attempting to stop an already stopped 
> topology nor when starting an already started topology.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1054) Ambari Mpack Attempts to Kill Topologies That Are Not Running

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098333#comment-16098333
 ] 

ASF GitHub Bot commented on METRON-1054:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/660
  
+1, by inspection.  Thanks a lot for improving this!


> Ambari Mpack Attempts to Kill Topologies That Are Not Running
> -
>
> Key: METRON-1054
> URL: https://issues.apache.org/jira/browse/METRON-1054
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.1
>
>
> The Ambari Metron MPack attempts to kill topologies that are not running.  It 
> also attempts to start topologies that are already running.  It reports these 
> conditions as errors.
> The MPack should not report errors when attempting to stop an already stopped 
> topology nor when starting an already started topology.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1054) Ambari Mpack Attempts to Kill Topologies That Are Not Running

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098331#comment-16098331
 ] 

ASF GitHub Bot commented on METRON-1054:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/660
  
BTW - I ran this up and tested against my latest commit.  I followed the 
instructions in the PR description.


> Ambari Mpack Attempts to Kill Topologies That Are Not Running
> -
>
> Key: METRON-1054
> URL: https://issues.apache.org/jira/browse/METRON-1054
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.1
>
>
> The Ambari Metron MPack attempts to kill topologies that are not running.  It 
> also attempts to start topologies that are already running.  It reports these 
> conditions as errors.
> The MPack should not report errors when attempting to stop an already stopped 
> topology nor when starting an already started topology.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1054) Ambari Mpack Attempts to Kill Topologies That Are Not Running

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098329#comment-16098329
 ] 

ASF GitHub Bot commented on METRON-1054:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/660
  
@nickwallen No, unfortunately.  Nobody's sat down and figured out a proper 
test strategy, because it's absurdly awkward.  We probably should, to ease this 
sort of thing.


> Ambari Mpack Attempts to Kill Topologies That Are Not Running
> -
>
> Key: METRON-1054
> URL: https://issues.apache.org/jira/browse/METRON-1054
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.1
>
>
> The Ambari Metron MPack attempts to kill topologies that are not running.  It 
> also attempts to start topologies that are already running.  It reports these 
> conditions as errors.
> The MPack should not report errors when attempting to stop an already stopped 
> topology nor when starting an already started topology.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (METRON-1058) address checkstyle warnings for UnusedImports in metron-stellar

[Artem Ervits (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Ervits updated METRON-1058:
-
Attachment: METRON-1058-0.patch

> address checkstyle warnings for UnusedImports in metron-stellar
> ---
>
> Key: METRON-1058
> URL: https://issues.apache.org/jira/browse/METRON-1058
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
> Environment: Mac OSX 10.12.5
> Maven 3.5.0
> JDK 1.8.0_141
>Reporter: Artem Ervits
>Priority: Minor
>  Labels: newbie
> Fix For: Next + 1
>
> Attachments: METRON-1058-0.patch
>
>
> addressing UnusedImports warning in checkstyle report for metron-stellar



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1054) Ambari Mpack Attempts to Kill Topologies That Are Not Running

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098324#comment-16098324
 ] 

ASF GitHub Bot commented on METRON-1054:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/660
  
@justinleet There are no unit tests for the Ambari Python scripts, correct? 
 Or am I missing them?


> Ambari Mpack Attempts to Kill Topologies That Are Not Running
> -
>
> Key: METRON-1054
> URL: https://issues.apache.org/jira/browse/METRON-1054
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.1
>
>
> The Ambari Metron MPack attempts to kill topologies that are not running.  It 
> also attempts to start topologies that are already running.  It reports these 
> conditions as errors.
> The MPack should not report errors when attempting to stop an already stopped 
> topology nor when starting an already started topology.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)