[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
Resolved the merge conflicts from Master and addressed review comments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104569239
  
--- Diff: metron-deployment/roles/metron-builder/tasks/main.yml ---
@@ -15,4 +15,7 @@
 #  limitations under the License.
 #
 ---
-hdp_repo_def: 
"http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.5.0.0/hdp.repo";
+- name: Build Deployment Artifacts
+  local_action: shell cd {{ metron_build_dir }} && mvn clean package 
-DskipTests -P HDP-2.5.0.0,mpack,build-rpms
--- End diff --

Yes, separate JIRA works for me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104565691
  
--- Diff: metron-deployment/roles/metron-builder/tasks/main.yml ---
@@ -15,4 +15,7 @@
 #  limitations under the License.
 #
 ---
-hdp_repo_def: 
"http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.5.0.0/hdp.repo";
+- name: Build Deployment Artifacts
+  local_action: shell cd {{ metron_build_dir }} && mvn clean package 
-DskipTests -P HDP-2.5.0.0,mpack,build-rpms
--- End diff --

I think I'd rather check in Maven and fail better earlier. What would you 
think about handling that as a standalone Jira?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104565422
  
--- Diff: metron-deployment/roles/quick_dev/tasks/main.yml ---
@@ -15,23 +15,50 @@
 #  limitations under the License.
 #
 ---
-#
-# Workaround for Kafka not starting
-# Fire off async start followed by
-# Sync start -execution will pause until
-# final start completes.
-#
-- name: Start the ambari cluster - no wait
-  ambari_cluster_state:
+- name: Delete the Metron Components from Ambari
--- End diff --

Exactly. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104558938
  
--- Diff: metron-deployment/roles/metron-builder/tasks/main.yml ---
@@ -15,4 +15,7 @@
 #  limitations under the License.
 #
 ---
-hdp_repo_def: 
"http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.5.0.0/hdp.repo";
+- name: Build Deployment Artifacts
+  local_action: shell cd {{ metron_build_dir }} && mvn clean package 
-DskipTests -P HDP-2.5.0.0,mpack,build-rpms
--- End diff --

Maybe something like...
```
- name: Build Deployment Artifacts
   local_action: shell cd {{ metron_build_dir }} && mvn clean package 
-DskipTests -P HDP-2.5.0.0,mpack,build-rpms
   become: false
   run_once: true
   ignore_errors: yes
   register: results

- fails: msg="Failed to build Metron... Is Docker Running?  "
   when: results.rc != 0
```

Or could we splits the Metron build into multiple steps?  First do the `mvn 
clean package`.  Then build the RPMS in a second step (the step that adds the 
Docker dependency) , so we know more concretely when the issue is likely to be 
that Docker not installed or not running?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104558496
  
--- Diff: metron-deployment/roles/metron-builder/tasks/main.yml ---
@@ -15,4 +15,7 @@
 #  limitations under the License.
 #
 ---
-hdp_repo_def: 
"http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.5.0.0/hdp.repo";
+- name: Build Deployment Artifacts
+  local_action: shell cd {{ metron_build_dir }} && mvn clean package 
-DskipTests -P HDP-2.5.0.0,mpack,build-rpms
--- End diff --

This is a "nice to have", but I noticed is that if you run "Full Dev" 
`vagrant up` without Docker running, it fails at this step with an awfully long 
and convoluted error message.  

```
...
ed(Launcher.java:289)", "\tat 
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)", 
"\tat 
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)",
 "\tat 
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)", 
"[INFO] 
", 
"[INFO] Reactor Summary:", "[INFO] ", "[INFO] Metron 
. SUCCESS [  0.327 s]", "[INFO] 
metron-analytics ... SUCCESS [  0.002 s]", 
"[INFO] metron-maas-common . SUCCESS [  5.415 
s]", "[INFO] metron-platform  SUCCESS [  
0.001 s]", "[INFO] metron-test-utilities .. SUCCESS 
[  3.629 s]", "[INFO] metron-integration-test  
SUCCESS [  7.044 s]", "[INFO] metron-maas-service 
 SUCCESS [  3.743 s]", "[INFO] 
 metron-common .. SUCCESS [ 18.669 s]", 
"[INFO] metron-statistics .. SUCCESS [ 13.954 
s]", "[INFO] metron-hbase ... SUCCESS [  
2.471 s]", "[INFO] metron-profiler-common . SUCCESS 
[  0.754 s]", "[INFO] metron-profiler-client . 
SUCCESS [ 15.322 s]", "[INFO] metron-profiler 
 SUCCESS [ 32.045 s]", "[INFO] 
metron-writer .. SUCCESS [ 12.181 s]", 
"[INFO] metron-enrichment .. SUCCESS [ 39.843 
s]", "[INFO] metron-indexing  SUCCESS [ 
12.021 s]", "[INFO] metron-solr  
SUCCESS [ 14.621 s]", "[INFO] metron-pcap 
 SUCCESS [  1.248 s]", "[INFO] 
metron-parsers . SUCCESS [ 40.486 s]", 
"[INFO] m
 etron-pcap-backend  SUCCESS [ 22.573 s]", 
"[INFO] metron-data-management . SUCCESS [ 45.768 
s]", "[INFO] metron-api . SUCCESS [ 
20.498 s]", "[INFO] metron-management .. 
SUCCESS [  4.632 s]", "[INFO] elasticsearch-shaded 
... SUCCESS [  3.916 s]", "[INFO] 
metron-elasticsearch ... SUCCESS [ 44.956 s]", 
"[INFO] metron-deployment .. SUCCESS [  0.000 
s]", "[INFO] metron-rpm . FAILURE [ 
10.066 s]", "[INFO] Metron Ambari Management Pack .. 
SKIPPED", "[INFO] metron-docker .. 
SKIPPED", "[INFO] 
", 
"[INFO] BUILD FAILURE", "[INFO] 
", 
"[INFO] To
 tal time: 06:16 min", "[INFO] Finished at: 2017-03-06T19:09:39-05:00", "[INFO] 
Final Memory: 162M/1993M", "[INFO] 
", 
"[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.5.0:exec 
(docker-build) on project metron-rpm: Command execution failed. Process exited 
with an error: 1 (Exit value: 1) -> [Help 1]", "[ERROR] ", "[ERROR] To see the 
full stack trace of the errors, re-run Maven with the -e switch.", "[ERROR] 
Re-run Maven using the -X switch to enable full debug logging.", "[ERROR] ", 
"[ERROR] For more information about the errors and possible solutions, please 
read the following articles:", "[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException";, 
"[ERROR] ", "[ERROR] After correcting the problems, you can resume the build 
with the command", "[ERROR]   mvn  -rf :metron-rpm"], "warnings": []}
```

I think this might cause some headaches, especially for new folks.  
Ideally, we could add some kind of more intelligible error message.



---
If your project is set up for it, you can reply to this email and 

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104553128
  
--- Diff: metron-deployment/roles/quick_dev/tasks/main.yml ---
@@ -15,23 +15,50 @@
 #  limitations under the License.
 #
 ---
-#
-# Workaround for Kafka not starting
-# Fire off async start followed by
-# Sync start -execution will pause until
-# final start completes.
-#
-- name: Start the ambari cluster - no wait
-  ambari_cluster_state:
+- name: Delete the Metron Components from Ambari
--- End diff --

Ah, ok.  So our Atlas image has Metron pre-installed via the MPack.  But 
this role then uninstalls and reinstalls it so that we get the latest, right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104548141
  
--- Diff: metron-deployment/roles/kafka-broker/tasks/main.yml ---
@@ -1,41 +0,0 @@
-#
-#  Licensed to the Apache Software Foundation (ASF) under one or more
-#  contributor license agreements.  See the NOTICE file distributed with
-#  this work for additional information regarding copyright ownership.
-#  The ASF licenses this file to You under the Apache License, Version 2.0
-#  (the "License"); you may not use this file except in compliance with
-#  the License.  You may obtain a copy of the License at
-#
-#  http://www.apache.org/licenses/LICENSE-2.0
-#
-#  Unless required by applicable law or agreed to in writing, software
-#  distributed under the License is distributed on an "AS IS" BASIS,
-#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-#  See the License for the specific language governing permissions and
-#  limitations under the License.
-#

-- name: Retrieve HDP repository definition
-  get_url:
-url: "{{ hdp_repo_def }}"
-dest: /etc/yum.repos.d/hdp.repo
-mode: 0644
-
-- name: Install kafka
-  yum: name={{item}}
-  with_items:
-- java-1.8.0-openjdk
-- kafka
-- zookeeper-server
-
-- name: Create pid directories
-  file: path={{ item }} state=directory mode=0755
-  with_items:
-- /var/run/zookeeper
-- /var/run/kafka
-
-- name: Start zookeeper
-  shell: /usr/hdp/current/zookeeper-server/bin/zookeeper-server start
-
-- name: Start kafka
--- End diff --

Oops. I'll put it back with the next push.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104546536
  
--- Diff: metron-deployment/packaging/packer-build/.gitignore ---
@@ -1,6 +1,6 @@
 .bundle/
 iso
-*.box

+builds/base-centos-6.7-2.1.20170303223924.git.33abe8cf13c347a2dfdece145a7b8c17f2a423c0_dirty.virtualbox.box
--- End diff --

Urm, no. I must have dumped my copy buffer by mistake while IntelliJ was 
open. It'll be gone in the next changeset. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104546225
  
--- Diff: metron-deployment/vagrant/full-dev-platform/Vagrantfile ---
@@ -53,7 +53,7 @@ hosts = [{
 Vagrant.configure(2) do |config|
 
   # all hosts built on centos 6
-  config.vm.box = "bento/centos-6.7"
+  config.vm.box = "metron/centos_base"
--- End diff --

Exactly that for exactly that reason. That and I can control the swap space 
size now. I get a much more reliable (not perfect, but better) full dev 
experience with this image. Something that I wanted to do for quite some time, 
but I finally got it done because I was having trouble differentiating between 
errors I had introduced and pre-existing errors. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104545807
  
--- Diff: metron-deployment/roles/quick_dev/tasks/main.yml ---
@@ -15,23 +15,50 @@
 #  limitations under the License.
 #
 ---
-#
-# Workaround for Kafka not starting
-# Fire off async start followed by
-# Sync start -execution will pause until
-# final start completes.
-#
-- name: Start the ambari cluster - no wait
-  ambari_cluster_state:
+- name: Delete the Metron Components from Ambari
--- End diff --

It bridges the gap between the quick-dev image and a fully functioning 
instance. To wit:

- Removes the stale Metron that ships with the image from Ambari
- Removes the stale Metron RPMs
- Re-installs fresh Metron packages
- Starts the cluster

We only require these activities on a quick-dev image. If you think of 
Ansible as declarative, it's not that weird. We apply the quick-dev role to the 
quick-dev host.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104544895
  
--- Diff: metron-deployment/roles/kibana/README.md ---
@@ -1,35 +0,0 @@
-Kibana 4
-
-
-This role installs Kibana along with the default Metron Dashboard.
-
-### FAQ
-
- How do I change Metron's default dashboard?
--- End diff --

There's a Python 
[module](https://github.com/apache/incubator-metron/tree/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/4.5.1/package/scripts/dashboard)
 that I wrote as part of the MPack that does the work there. I have an 
outstanding item to write up it's basic usage, but can't find a good place in 
the code. Ideas?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104539908
  
--- Diff: metron-deployment/roles/kafka-broker/tasks/main.yml ---
@@ -1,41 +0,0 @@
-#
-#  Licensed to the Apache Software Foundation (ASF) under one or more
-#  contributor license agreements.  See the NOTICE file distributed with
-#  this work for additional information regarding copyright ownership.
-#  The ASF licenses this file to You under the Apache License, Version 2.0
-#  (the "License"); you may not use this file except in compliance with
-#  the License.  You may obtain a copy of the License at
-#
-#  http://www.apache.org/licenses/LICENSE-2.0
-#
-#  Unless required by applicable law or agreed to in writing, software
-#  distributed under the License is distributed on an "AS IS" BASIS,
-#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-#  See the License for the specific language governing permissions and
-#  limitations under the License.
-#

-- name: Retrieve HDP repository definition
-  get_url:
-url: "{{ hdp_repo_def }}"
-dest: /etc/yum.repos.d/hdp.repo
-mode: 0644
-
-- name: Install kafka
-  yum: name={{item}}
-  with_items:
-- java-1.8.0-openjdk
-- kafka
-- zookeeper-server
-
-- name: Create pid directories
-  file: path={{ item }} state=directory mode=0755
-  with_items:
-- /var/run/zookeeper
-- /var/run/kafka
-
-- name: Start zookeeper
-  shell: /usr/hdp/current/zookeeper-server/bin/zookeeper-server start
-
-- name: Start kafka
--- End diff --

The `kafka_broker` role is used by the `fastcapa-test-platform`.  Removing 
it will break that environment.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104540418
  
--- Diff: metron-deployment/roles/kibana/README.md ---
@@ -1,35 +0,0 @@
-Kibana 4
-
-
-This role installs Kibana along with the default Metron Dashboard.
-
-### FAQ
-
- How do I change Metron's default dashboard?
--- End diff --

How does the load of the Kibana Dashboard work now?  Do we not use 
`elasticdump` any longer?  Bravo, if you found a better way.

This README was intended to help describe how to make changes to the Metron 
Kibana Dashboard and get those changes into source control.  I just want to 
make sure this same sort of thing is either obvious now or described somewhere 
else.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104542365
  
--- Diff: metron-deployment/roles/quick_dev/tasks/main.yml ---
@@ -15,23 +15,50 @@
 #  limitations under the License.
 #
 ---
-#
-# Workaround for Kafka not starting
-# Fire off async start followed by
-# Sync start -execution will pause until
-# final start completes.
-#
-- name: Start the ambari cluster - no wait
-  ambari_cluster_state:
+- name: Delete the Metron Components from Ambari
--- End diff --

What does the `quick-dev` role do?  Curious why we have a role specific to 
one of our environments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104543437
  
--- Diff: metron-deployment/vagrant/full-dev-platform/Vagrantfile ---
@@ -53,7 +53,7 @@ hosts = [{
 Vagrant.configure(2) do |config|
 
   # all hosts built on centos 6
-  config.vm.box = "bento/centos-6.7"
+  config.vm.box = "metron/centos_base"
--- End diff --

How did our Atlas environment change?  It seems like we now create our one 
CentOS image?  

Seems like a good idea being that Bento's image could and did change 
without warning.  Do I build the CentOS image with the rest of the 
`packer-build` stuff?  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104537561
  
--- Diff: metron-deployment/packaging/packer-build/.gitignore ---
@@ -1,6 +1,6 @@
 .bundle/
 iso
-*.box

+builds/base-centos-6.7-2.1.20170303223924.git.33abe8cf13c347a2dfdece145a7b8c17f2a423c0_dirty.virtualbox.box
--- End diff --

Did you intend to make this change?  Seems oddly specific. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
I'm working on those merge conflicts right now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #453: METRON-694: Index Errors from Topologies

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/453


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
@JonZeolla - makes sense, I've pushed that change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
Not a showstopper for me, but maybe worth a mention in `Upgrading.md` that 
CentOS 6 shouldn't be used any longer.  Note that `Upgrading.md` gets a bit of 
a modification in METRON-752 (#470).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #453: METRON-694: Index Errors from Topologies

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/453
  
I will create an issue to track the possibility for looping.  

Other than that, this looks good.  +1 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
From @JonZeolla - "Looks like Docker is a dependency now which rules out 
CentOS 6 if we strictly follow the docker documentation.  Was that expected or 
would you assume people would opt for the workaround steps?"

It is, to integrate with the MPack we have to build RPMs and (right now 
anyway), so that means Docker. It was always implicit in this ticket, but I did 
try to underline it over the weekend. I wasn't aware that Docker and CentOS6 
were problematic. I used to use those two together a little while back, but it 
was forever ago in Dockertime.

Is that a showstopper for anyone?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #471: METRON-755 Update GitHub PR Template

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/incubator-metron/pull/471
  
Ok, moved it to the top.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #453: METRON-694: Index Errors from Topologies

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/incubator-metron/pull/453
  
Does anyone have any outstanding comments that they need addressed?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [GitHub] incubator-metron issue #471: METRON-755 Update GitHub PR Template

[Matt Foley]

I feel the introductory text for the PR should be at the top.  That’s the 
interesting part.  The template checklist is to make sure the necessary got 
done; it’s important, but should come second. IMHO.


On 3/6/17, 8:01 AM, "David Lyle"  wrote:

I have a weak preference for top comments.


On Mon, Mar 6, 2017 at 10:22 AM, JonZeolla  wrote:

> Github user JonZeolla commented on the issue:
>
> https://github.com/apache/incubator-metron/pull/471
>
> It seems that some people prefer to add their comments to the top
> instead of the bottom.  I have no preference.  Would anybody recommend we
> swap it?
>
>
> ---
> If your project is set up for it, you can reply to this email and have 
your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, 
please
> contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
> with INFRA.
> ---
>

Re: [DISCUSS][PROPOSAL] Acceptance Tests

[Michael Miklavcic]

Ok, yes I agree. In my experience with e2e/acceptance tests, they're best
kept general with an emphasis on verifying that all the plumbing works
together. So yes, there are definite edge cases I think we'll want to test
here, but I say that with the caveat that I think we should ideally cover
as many non-happy-path cases in unit and integration tests as possible. As
an example, I don't think it makes sense to cover most of the profiler
windowing DSL language edge cases in acceptance tests instead of or in
addition to unit/integration tests unless there is something specific to
the integration with a given an environment that we think could be
problematic.

M

On Mon, Mar 6, 2017 at 11:32 AM, Casey Stella  wrote:

> No, I'm saying that they shouldn't be restricted to real-world use-cases.
> The E2E tests I laid out weren't real-world, but they did exercise the
> components similar to real-world use-cases.  They should also be able to be
> able to tread outside of the happy-path for those use-cases.
>
> On Mon, Mar 6, 2017 at 6:30 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > "I don't think acceptance tests should loosely associate with real uses,
> > but they should
> > be free to delve into weird non-happy-pathways."
> >
> > Not following - are you saying they should *tightly* associate with real
> > uses and additonally include non-happy-path?
> >
> > On Fri, Mar 3, 2017 at 12:57 PM, Casey Stella 
> wrote:
> >
> > > It is absolutely not a naive question, Matt.  We don't have a lot (or
> > any)
> > > docs about our integration tests; it's more of a "follow the lead" type
> > of
> > > thing at the moment, but that should be rectified.
> > >
> > > The integration tests spin up and down infrastructure in-process, some
> of
> > > which are real and some of which are mock versions of the services.
> > These
> > > are good for catching some types of bugs, but often things sneak
> through,
> > > like:
> > >
> > >- Hbase and storm can't exist in the same JVM, so HBase is mocked in
> > >those cases.
> > >- The FileSystem that we get for Hadoop is the LocalRawFileSystem,
> not
> > >truly HDFS.  There are differences and we've run into
> > them..hilariously
> > > at
> > >times. ;)
> > >- Things done statically in a bolt are shared across all bolts
> because
> > >they all are threads in the same process
> > >
> > > It's good, it catches bugs, it lets us debug things easily, it runs
> with
> > > every single build automatically via travis.
> > > It's bad because it's awkward to get the dependencies isolated
> > sufficiently
> > > for all of these components to get them to play nice in the same JVM.
> > >
> > > Acceptance tests would be run against a real cluster, so they would:
> > >
> > >- run against real components, not testing or mock components
> > >- run against multiple nodes
> > >
> > > I can imagine a world where we can unify the two to a certain degree in
> > > many cases if we could spin up a docker version of Metron to run as
> part
> > of
> > > the build, but I think in the meantime, we should focus on providing
> > both.
> > >
> > > I suspect the reference application is possibly inspiring my
> suggestions
> > > here, but I think the main difference here is that the reference
> > > application is intended to be informational from a end-user
> perspective:
> > > it's detailing a use-case that users will understand.  I don't think
> > > acceptance tests should loosely associate with real uses, but they
> should
> > > be free to delve into weird non-happy-pathways.
> > >
> > > On Fri, Mar 3, 2017 at 2:16 PM, Matt Foley  wrote:
> > >
> > > > Automating stuff that now has to be done manually gets a big +1.
> > > >
> > > > But, Casey, could you please clarify the relationship between what
> you
> > > > plan to do and the current “integration test” framework?  Will this
> be
> > in
> > > > the form of additional integration tests? Or a different test
> > framework?
> > > > Can it be done in the integration test framework, rather than
> creating
> > > new
> > > > mechanism?
> > > >
> > > > BTW, if that’s a naïve question, forgive me, but I could find zero
> > > > documentation for the existing integration test capability, neither
> > wiki
> > > > pages nor READMEs nor Jiras.  If there are any docs, please point me
> at
> > > > them.  Or even archived email threads.
> > > >
> > > > There is also something called the “Reference Application”
> > > > https://cwiki.apache.org/confluence/display/METRON/
> > > > Metron+Reference+Application which sounds remarkably like what you
> > > > propose to automate.  Is there / can there / should there be a
> > > relationship?
> > > >
> > > > Thanks,
> > > > --Matt
> > > >
> > > > On 3/3/17, 7:40 AM, "Otto Fowler"  wrote:
> > > >
> > > > +1
> > > >
> > > > I agree with Justin’s points.
> > > >
> > > >
> > > > On March 3, 2017 at 08:41:37, Justin Leet (justinjl...@gmail.com
> )
> > > > wrote:
> > > >
> > > >

Re: [DISCUSS] Wiki use and migration of docs

[Michael Miklavcic]

Just to clarify the point about migrating from the wiki - you'd like to see
use cases, demos, cookbooks, etc. moved into the git repo? I'm +1 for this
as it allows us to version the examples with each commit and release. A
feature that is currently lacking and more difficult to accomplish on the
wiki. We could always link the wiki and/or Metron website to the latest
master branch for up to date examples. I'd like to give people more avenues
to access information than less, but definitely do not want to duplicate
docs.

I think it also makes sense to me to keep the examples closer to the code.
As a compromise between project root level vs package level, I think
keeping an examples folder within each Maven module seems reasonable to me.
Examples.md seems ok. We might even choose to create a folder for examples
and name the md file by feature. Or we could create sub-folders naming the
feature and drop examples.md files in each folder. I'd also like to include
the concept of "cookbook" examples here as well. I think there is overlap
with demos and use cases, but cookbook examples can often be more specific
to a fine grained task.

Mike


On Mon, Mar 6, 2017 at 7:02 AM, zeo...@gmail.com  wrote:

> bump
>
> On Sat, Feb 11, 2017 at 2:15 PM zeo...@gmail.com  wrote:
>
> > This morning I had an opportunity to watch the video from yesterday's
> > community demo, and there was some really good discussion towards the end
> > about documentation of examples that I wanted to follow up with.  For
> > future reference, here is
> > the recording of what I'm referring to - this is all as a follow-up to
> > Matt's great work via METRON-660
> > .
> >
> > I am looking for feedback on an idea for the future of Metron
> > documentation.  At a high level, I would like to migrate materials from
> the
> > wiki pages throughout the git repo and modify our documentation
> generation
> > scripts to key in on tutorials vs readmes.  Once we have agreement on
> this
> > I would be happy to handle any data migration and manipulation as
> necessary.
> >
> > More specifically, I would like to establish a convention for the names
> of
> > example or tutorial md files that we could then use when generating the
> > release documentation.  Say we use "examples.md", we could then generate
> > an examples/tutorials top level area in the site-docs without having to
> add
> > it into the git repo itself.  In addition, this lets the examples.md
> > files exist more closely to the code they are about, which seems to be
> the
> > preference of most people currently working on the project.
> >
> > A good example of this would be to break Casey's outlier analysis example
> >  metron-analytics/metron-statistics#outlier-analysis>
> >   into a new examples.md in the same directory.  I would think more
> > generalized examples/tutorials would exist in the root of the git repo.
> > I'm also game for arguments that we take another approach, such as
> making a
> > new top level folder in the repo for all examples/tutorials, but that
> would
> > be less preferred in my opinion.
> >
> > We could probably move the overview
> > ,
> > architecture
> >  >,
> > tutorials
> > , and
> > governance
> > 
> > wiki materials without much of an issue.  Pages like the tech talks
> >  and
> > community
> > 
> information probably
> > fit better in the Metron site
> >  area of
> > GitHub, and not as a md.  The items that I wouldn't be sure about
> migrating
> > are things like the user research
> >  or
> meeting
> > notes  >.
> > Is there still value in having these materials published?  Maybe we leave
> > them behind in the Wiki and use it as more of an archive store for
> > historical context?
> >
> > If I don't get any strong disagreement with this idea, I'm going to throw
> > together a first attempt.  I also opened a ticket for this - METRON-714
> > .
> >
> > Jon
> > --
> >
> > Jon
> >
> > Sent from my mobile device
> >
> --
>
> Jon
>
> Sent from my mobile device
>

Re: [DISCUSS][PROPOSAL] Acceptance Tests

[Casey Stella]

No, I'm saying that they shouldn't be restricted to real-world use-cases.
The E2E tests I laid out weren't real-world, but they did exercise the
components similar to real-world use-cases.  They should also be able to be
able to tread outside of the happy-path for those use-cases.

On Mon, Mar 6, 2017 at 6:30 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> "I don't think acceptance tests should loosely associate with real uses,
> but they should
> be free to delve into weird non-happy-pathways."
>
> Not following - are you saying they should *tightly* associate with real
> uses and additonally include non-happy-path?
>
> On Fri, Mar 3, 2017 at 12:57 PM, Casey Stella  wrote:
>
> > It is absolutely not a naive question, Matt.  We don't have a lot (or
> any)
> > docs about our integration tests; it's more of a "follow the lead" type
> of
> > thing at the moment, but that should be rectified.
> >
> > The integration tests spin up and down infrastructure in-process, some of
> > which are real and some of which are mock versions of the services.
> These
> > are good for catching some types of bugs, but often things sneak through,
> > like:
> >
> >- Hbase and storm can't exist in the same JVM, so HBase is mocked in
> >those cases.
> >- The FileSystem that we get for Hadoop is the LocalRawFileSystem, not
> >truly HDFS.  There are differences and we've run into
> them..hilariously
> > at
> >times. ;)
> >- Things done statically in a bolt are shared across all bolts because
> >they all are threads in the same process
> >
> > It's good, it catches bugs, it lets us debug things easily, it runs with
> > every single build automatically via travis.
> > It's bad because it's awkward to get the dependencies isolated
> sufficiently
> > for all of these components to get them to play nice in the same JVM.
> >
> > Acceptance tests would be run against a real cluster, so they would:
> >
> >- run against real components, not testing or mock components
> >- run against multiple nodes
> >
> > I can imagine a world where we can unify the two to a certain degree in
> > many cases if we could spin up a docker version of Metron to run as part
> of
> > the build, but I think in the meantime, we should focus on providing
> both.
> >
> > I suspect the reference application is possibly inspiring my suggestions
> > here, but I think the main difference here is that the reference
> > application is intended to be informational from a end-user perspective:
> > it's detailing a use-case that users will understand.  I don't think
> > acceptance tests should loosely associate with real uses, but they should
> > be free to delve into weird non-happy-pathways.
> >
> > On Fri, Mar 3, 2017 at 2:16 PM, Matt Foley  wrote:
> >
> > > Automating stuff that now has to be done manually gets a big +1.
> > >
> > > But, Casey, could you please clarify the relationship between what you
> > > plan to do and the current “integration test” framework?  Will this be
> in
> > > the form of additional integration tests? Or a different test
> framework?
> > > Can it be done in the integration test framework, rather than creating
> > new
> > > mechanism?
> > >
> > > BTW, if that’s a naïve question, forgive me, but I could find zero
> > > documentation for the existing integration test capability, neither
> wiki
> > > pages nor READMEs nor Jiras.  If there are any docs, please point me at
> > > them.  Or even archived email threads.
> > >
> > > There is also something called the “Reference Application”
> > > https://cwiki.apache.org/confluence/display/METRON/
> > > Metron+Reference+Application which sounds remarkably like what you
> > > propose to automate.  Is there / can there / should there be a
> > relationship?
> > >
> > > Thanks,
> > > --Matt
> > >
> > > On 3/3/17, 7:40 AM, "Otto Fowler"  wrote:
> > >
> > > +1
> > >
> > > I agree with Justin’s points.
> > >
> > >
> > > On March 3, 2017 at 08:41:37, Justin Leet (justinjl...@gmail.com)
> > > wrote:
> > >
> > > +1 to both. Having this would especially ease a lot of testing that
> > > hits
> > > multiple areas (which there is a fair amount of, given that we're
> > > building
> > > pretty quickly).
> > >
> > > I do want to point out that adding this type of thing makes the
> speed
> > > of
> > > our builds and tests more important, because they already take up a
> > > good
> > > amount of time. There are obviously tickets to optimize these
> things,
> > > but
> > > I would like to make sure we don't pile too much on to every
> testing
> > > cycle
> > > before a PR. Having said that, I think the testing proposed is
> > > absolutely
> > > valuable enough to go forward with.
> > >
> > > Justin
> > >
> > > On Fri, Mar 3, 2017 at 8:33 AM, Casey Stella 
> > > wrote:
> > >
> > > > I also propose, once this is done, that we modify the developer
> > > bylaws
> > > and
> > > > the github PR script to ensure t

Re: [DISCUSS][PROPOSAL] Acceptance Tests

[Michael Miklavcic]

"I don't think acceptance tests should loosely associate with real uses,
but they should
be free to delve into weird non-happy-pathways."

Not following - are you saying they should *tightly* associate with real
uses and additonally include non-happy-path?

On Fri, Mar 3, 2017 at 12:57 PM, Casey Stella  wrote:

> It is absolutely not a naive question, Matt.  We don't have a lot (or any)
> docs about our integration tests; it's more of a "follow the lead" type of
> thing at the moment, but that should be rectified.
>
> The integration tests spin up and down infrastructure in-process, some of
> which are real and some of which are mock versions of the services.  These
> are good for catching some types of bugs, but often things sneak through,
> like:
>
>- Hbase and storm can't exist in the same JVM, so HBase is mocked in
>those cases.
>- The FileSystem that we get for Hadoop is the LocalRawFileSystem, not
>truly HDFS.  There are differences and we've run into them..hilariously
> at
>times. ;)
>- Things done statically in a bolt are shared across all bolts because
>they all are threads in the same process
>
> It's good, it catches bugs, it lets us debug things easily, it runs with
> every single build automatically via travis.
> It's bad because it's awkward to get the dependencies isolated sufficiently
> for all of these components to get them to play nice in the same JVM.
>
> Acceptance tests would be run against a real cluster, so they would:
>
>- run against real components, not testing or mock components
>- run against multiple nodes
>
> I can imagine a world where we can unify the two to a certain degree in
> many cases if we could spin up a docker version of Metron to run as part of
> the build, but I think in the meantime, we should focus on providing both.
>
> I suspect the reference application is possibly inspiring my suggestions
> here, but I think the main difference here is that the reference
> application is intended to be informational from a end-user perspective:
> it's detailing a use-case that users will understand.  I don't think
> acceptance tests should loosely associate with real uses, but they should
> be free to delve into weird non-happy-pathways.
>
> On Fri, Mar 3, 2017 at 2:16 PM, Matt Foley  wrote:
>
> > Automating stuff that now has to be done manually gets a big +1.
> >
> > But, Casey, could you please clarify the relationship between what you
> > plan to do and the current “integration test” framework?  Will this be in
> > the form of additional integration tests? Or a different test framework?
> > Can it be done in the integration test framework, rather than creating
> new
> > mechanism?
> >
> > BTW, if that’s a naïve question, forgive me, but I could find zero
> > documentation for the existing integration test capability, neither wiki
> > pages nor READMEs nor Jiras.  If there are any docs, please point me at
> > them.  Or even archived email threads.
> >
> > There is also something called the “Reference Application”
> > https://cwiki.apache.org/confluence/display/METRON/
> > Metron+Reference+Application which sounds remarkably like what you
> > propose to automate.  Is there / can there / should there be a
> relationship?
> >
> > Thanks,
> > --Matt
> >
> > On 3/3/17, 7:40 AM, "Otto Fowler"  wrote:
> >
> > +1
> >
> > I agree with Justin’s points.
> >
> >
> > On March 3, 2017 at 08:41:37, Justin Leet (justinjl...@gmail.com)
> > wrote:
> >
> > +1 to both. Having this would especially ease a lot of testing that
> > hits
> > multiple areas (which there is a fair amount of, given that we're
> > building
> > pretty quickly).
> >
> > I do want to point out that adding this type of thing makes the speed
> > of
> > our builds and tests more important, because they already take up a
> > good
> > amount of time. There are obviously tickets to optimize these things,
> > but
> > I would like to make sure we don't pile too much on to every testing
> > cycle
> > before a PR. Having said that, I think the testing proposed is
> > absolutely
> > valuable enough to go forward with.
> >
> > Justin
> >
> > On Fri, Mar 3, 2017 at 8:33 AM, Casey Stella 
> > wrote:
> >
> > > I also propose, once this is done, that we modify the developer
> > bylaws
> > and
> > > the github PR script to ensure that PR authors:
> > >
> > > - Update the acceptance tests where appropriate
> > > - Run the tests as a smoketest
> > >
> > >
> > >
> > > On Fri, Mar 3, 2017 at 8:21 AM, Casey Stella 
> > wrote:
> > >
> > > > Hi All,
> > > >
> > > > After doing METRON-744, where I had to walk through a manual test
> > of
> > > every
> > > > place that Stellar touched, it occurred to me that we should
> script
> > this.
> > > > It also occurred to me that some scripts that are run by the PR
> > author
> > to
> > > > ensure no regressions and, eventually maybe

[GitHub] incubator-metron issue #471: METRON-755 Update GitHub PR Template

[kylerichardson]

Github user kylerichardson commented on the issue:

https://github.com/apache/incubator-metron/pull/471
  
I'd prefer contributor comments at the top.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
@dlyle65535 can you point to any documentation for the ambari api/functions 
that we are using?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
@ottobackwards - sure, I used 
[this](https://cwiki.apache.org/confluence/display/AMBARI/API+usage+scenarios%2C+troubleshooting%2C+and+other+FAQs)
 as a guide.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
This is ready for review. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #471: METRON-755 Update GitHub PR Template

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/471
  
I prefer top, but I don't really care that much.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [GitHub] incubator-metron issue #471: METRON-755 Update GitHub PR Template

[David Lyle]

I have a weak preference for top comments.


On Mon, Mar 6, 2017 at 10:22 AM, JonZeolla  wrote:

> Github user JonZeolla commented on the issue:
>
> https://github.com/apache/incubator-metron/pull/471
>
> It seems that some people prefer to add their comments to the top
> instead of the bottom.  I have no preference.  Would anybody recommend we
> swap it?
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
> with INFRA.
> ---
>

[GitHub] incubator-metron issue #471: METRON-755 Update GitHub PR Template

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/incubator-metron/pull/471
  
It seems that some people prefer to add their comments to the top instead 
of the bottom.  I have no preference.  Would anybody recommend we swap it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #473: METRON-712: Separate evaluation from parsing in...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/473
  
TODO: I still have yet to add some documentation around the 
microbenchmarking suite.
TODO: Run this up in vagrant.  I can't do so now as I am on hotel wifi, so 
it'll be this Thursday before I get a chance.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #473: METRON-712: Separate evaluation from par...

[cestella]

GitHub user cestella opened a pull request:

https://github.com/apache/incubator-metron/pull/473

METRON-712: Separate evaluation from parsing in Stellar

# Description
With the current implementation of Stellar, we cannot cache the parse tree 
and then apply it after the fact. It's just an artifact of how we do the 
parsing: we actually execute the statement as we parse rather than constructing 
an AST that can then be evaluated later given a message. Essentially what I'm 
proposing is that we build the equivalent of Pattern.compile() in Java except 
for Stellar.
We should for multiple reasons:
* code clarity - decoupling the stellar language from the generated parser 
code
* performance - saving lexing and parsing for every message.  Also, the 
resulting parse-stack may be much smaller than the somewhat complex.

In this PR, I have added a google cache that will cache the resulting 
compiled expression in `BaseStellarProcessor` for 10 minutes (by default).  I 
have also created a microbenchmarking suite and have evaluated this on a few 
representative expressions.
Results:
* `TO_UPPER('casey')`
  * Median ms before: `880.5`
  * Median ms after: `15`
  * Speedup: 58.6x faster
* `TO_LOWER(name)`
  * Median ms before: `497`
  * Median ms after: `3`
  * Speedup: 165.6x faster
* `1 + 2*(3 + int_num) / 10.0`
  * Median ms before: `676`
  * Median ms after: `4`
  * Speedup: 169x faster
* `1.5 + 2*(3 + double_num) / 10.0`
  * Median ms before: `634`
  * Median ms after: `1`
  * Speedup: 634x faster
* `if ('foo' in ['foo']) OR one == very_nearly_one then 'one' else 'two'`
  * Median ms before: `616`
  * Median ms after: `23`
  * Speedup: 26x faster
* `1 + 2*(3 + int_num) / 10.0`
  * Median ms before: `601`
  * Median ms after: `2`
  * Speedup: 300.5x faster
* `DOMAIN_TO_TLD(domain)`
  * Median ms before: `505`
  * Median ms after: `16`
  * Speedup: 32.5x
* `DOMAIN_REMOVE_SUBDOMAINS(domain)`
  * Median ms before: `496`
  * Median ms after: `11`
  * Speedup: 45x faster


# Testing Plan
Please refer to the METRON-744 
[PR](https://github.com/apache/incubator-metron/pull/468#issue-210707129) 
testing plan.

In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check
the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:

```
mvn -q clean integration-test install && build_utils/verify_licenses.sh 
```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
site-book/target/site/index.html.

```
cd site-book
bin/generate-md.sh
mvn site:site

```

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommened that [travis-ci](https://travis-ci.org) is set up for 
your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron stellar_optimize

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/473.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the fo

Re: [DISCUSS] Wiki use and migration of docs

[zeo...@gmail.com]

bump

On Sat, Feb 11, 2017 at 2:15 PM zeo...@gmail.com  wrote:

> This morning I had an opportunity to watch the video from yesterday's
> community demo, and there was some really good discussion towards the end
> about documentation of examples that I wanted to follow up with.  For
> future reference, here is
> the recording of what I'm referring to - this is all as a follow-up to
> Matt's great work via METRON-660
> .
>
> I am looking for feedback on an idea for the future of Metron
> documentation.  At a high level, I would like to migrate materials from the
> wiki pages throughout the git repo and modify our documentation generation
> scripts to key in on tutorials vs readmes.  Once we have agreement on this
> I would be happy to handle any data migration and manipulation as necessary.
>
> More specifically, I would like to establish a convention for the names of
> example or tutorial md files that we could then use when generating the
> release documentation.  Say we use "examples.md", we could then generate
> an examples/tutorials top level area in the site-docs without having to add
> it into the git repo itself.  In addition, this lets the examples.md
> files exist more closely to the code they are about, which seems to be the
> preference of most people currently working on the project.
>
> A good example of this would be to break Casey's outlier analysis example
> 
>   into a new examples.md in the same directory.  I would think more
> generalized examples/tutorials would exist in the root of the git repo.
> I'm also game for arguments that we take another approach, such as making a
> new top level folder in the repo for all examples/tutorials, but that would
> be less preferred in my opinion.
>
> We could probably move the overview
> ,
> architecture
> ,
> tutorials
> , and
> governance
> 
> wiki materials without much of an issue.  Pages like the tech talks
>  and
> community
>  
> information probably
> fit better in the Metron site
>  area of
> GitHub, and not as a md.  The items that I wouldn't be sure about migrating
> are things like the user research
>  or meeting
> notes .
> Is there still value in having these materials published?  Maybe we leave
> them behind in the Wiki and use it as more of an archive store for
> historical context?
>
> If I don't get any strong disagreement with this idea, I'm going to throw
> together a first attempt.  I also opened a ticket for this - METRON-714
> .
>
> Jon
> --
>
> Jon
>
> Sent from my mobile device
>
-- 

Jon

Sent from my mobile device

[GitHub] incubator-metron issue #175: METRON-243 Ensure templates are installed on al...

[liqk01]

Github user liqk01 commented on the issue:

https://github.com/apache/incubator-metron/pull/175
  
But i have single node only , and i donnot want to delete existing index 
,then how do i resolve it ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---