[jira] [Commented] (METRON-1552) Add gzip file validation check to the geo loader

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472853#comment-16472853
 ] 

ASF GitHub Bot commented on METRON-1552:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/1011#discussion_r187759693
  
--- Diff: 
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/geo/GeoEnrichmentLoader.java
 ---
@@ -167,26 +184,38 @@ protected void loadGeoIpDatabase(CommandLine cli) 
throws IOException {
 System.out.println("Successfully created and updated new GeoIP 
information");
   }
 
-  protected File downloadGeoFile(String urlStr, String tmpDir) {
+  protected File downloadGeoFile(String urlStr, String tmpDir, int  
numRetries) {
 File localFile = null;
-try {
-  URL url = new URL(urlStr);
-  localFile = new File(tmpDir + new File(url.getPath()).getName());
+int attempts = 0;
+boolean valid = false;
+while (attempts <= numRetries) {
+  try {
+URL url = new URL(urlStr);
+localFile = new File(tmpDir + new File(url.getPath()).getName());
 
-  System.out.println("Downloading " + url.toString() + " to " + 
localFile.getAbsolutePath());
-  if (localFile.exists() && !localFile.delete()) {
-System.err.println("File already exists locally and can't be 
deleted.  Please delete before continuing");
-System.exit(3);
+System.out.println("Downloading " + url.toString() + " to " + 
localFile.getAbsolutePath());
+if (localFile.exists() && !localFile.delete()) {
+  System.err.println(
+  "File already exists locally and can't be deleted.  Please 
delete before continuing");
+  System.exit(3);
+}
+FileUtils.copyURLToFile(url, localFile, 5000, 1);
+if (!CompressionStrategies.GZIP.test(localFile)) {
+  throw new IOException("Invalid Gzip file");
+}
+  } catch (MalformedURLException e) {
+System.err.println("Malformed URL - aborting: " + e);
+e.printStackTrace();
+System.exit(4);
+  } catch (IOException e) {
+System.err.println("Warning: Unable to copy remote GeoIP database 
to local file, attempt " + attempts + ": " + e);
+e.printStackTrace();
--- End diff --

D'oh, I retract the comment, this is to stderr.  My bad


> Add gzip file validation check to the geo loader
> 
>
> Key: METRON-1552
> URL: https://issues.apache.org/jira/browse/METRON-1552
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1011: METRON-1552: Add gzip file validation check to th...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/1011#discussion_r187759693
  
--- Diff: 
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/geo/GeoEnrichmentLoader.java
 ---
@@ -167,26 +184,38 @@ protected void loadGeoIpDatabase(CommandLine cli) 
throws IOException {
 System.out.println("Successfully created and updated new GeoIP 
information");
   }
 
-  protected File downloadGeoFile(String urlStr, String tmpDir) {
+  protected File downloadGeoFile(String urlStr, String tmpDir, int  
numRetries) {
 File localFile = null;
-try {
-  URL url = new URL(urlStr);
-  localFile = new File(tmpDir + new File(url.getPath()).getName());
+int attempts = 0;
+boolean valid = false;
+while (attempts <= numRetries) {
+  try {
+URL url = new URL(urlStr);
+localFile = new File(tmpDir + new File(url.getPath()).getName());
 
-  System.out.println("Downloading " + url.toString() + " to " + 
localFile.getAbsolutePath());
-  if (localFile.exists() && !localFile.delete()) {
-System.err.println("File already exists locally and can't be 
deleted.  Please delete before continuing");
-System.exit(3);
+System.out.println("Downloading " + url.toString() + " to " + 
localFile.getAbsolutePath());
+if (localFile.exists() && !localFile.delete()) {
+  System.err.println(
+  "File already exists locally and can't be deleted.  Please 
delete before continuing");
+  System.exit(3);
+}
+FileUtils.copyURLToFile(url, localFile, 5000, 1);
+if (!CompressionStrategies.GZIP.test(localFile)) {
+  throw new IOException("Invalid Gzip file");
+}
+  } catch (MalformedURLException e) {
+System.err.println("Malformed URL - aborting: " + e);
+e.printStackTrace();
+System.exit(4);
+  } catch (IOException e) {
+System.err.println("Warning: Unable to copy remote GeoIP database 
to local file, attempt " + attempts + ": " + e);
+e.printStackTrace();
--- End diff --

D'oh, I retract the comment, this is to stderr.  My bad


---

[GitHub] metron pull request #1011: METRON-1552: Add gzip file validation check to th...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/1011#discussion_r187759685
  
--- Diff: 
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/geo/GeoEnrichmentLoader.java
 ---
@@ -167,26 +184,38 @@ protected void loadGeoIpDatabase(CommandLine cli) 
throws IOException {
 System.out.println("Successfully created and updated new GeoIP 
information");
   }
 
-  protected File downloadGeoFile(String urlStr, String tmpDir) {
+  protected File downloadGeoFile(String urlStr, String tmpDir, int  
numRetries) {
 File localFile = null;
-try {
-  URL url = new URL(urlStr);
-  localFile = new File(tmpDir + new File(url.getPath()).getName());
+int attempts = 0;
+boolean valid = false;
+while (attempts <= numRetries) {
+  try {
+URL url = new URL(urlStr);
+localFile = new File(tmpDir + new File(url.getPath()).getName());
 
-  System.out.println("Downloading " + url.toString() + " to " + 
localFile.getAbsolutePath());
-  if (localFile.exists() && !localFile.delete()) {
-System.err.println("File already exists locally and can't be 
deleted.  Please delete before continuing");
-System.exit(3);
+System.out.println("Downloading " + url.toString() + " to " + 
localFile.getAbsolutePath());
+if (localFile.exists() && !localFile.delete()) {
+  System.err.println(
+  "File already exists locally and can't be deleted.  Please 
delete before continuing");
+  System.exit(3);
+}
+FileUtils.copyURLToFile(url, localFile, 5000, 1);
+if (!CompressionStrategies.GZIP.test(localFile)) {
+  throw new IOException("Invalid Gzip file");
+}
+  } catch (MalformedURLException e) {
+System.err.println("Malformed URL - aborting: " + e);
+e.printStackTrace();
--- End diff --

D'oh I retract the statement.  This is to stderr; my bad


---

[jira] [Commented] (METRON-1552) Add gzip file validation check to the geo loader

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472851#comment-16472851
 ] 

ASF GitHub Bot commented on METRON-1552:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/1011#discussion_r187759685
  
--- Diff: 
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/geo/GeoEnrichmentLoader.java
 ---
@@ -167,26 +184,38 @@ protected void loadGeoIpDatabase(CommandLine cli) 
throws IOException {
 System.out.println("Successfully created and updated new GeoIP 
information");
   }
 
-  protected File downloadGeoFile(String urlStr, String tmpDir) {
+  protected File downloadGeoFile(String urlStr, String tmpDir, int  
numRetries) {
 File localFile = null;
-try {
-  URL url = new URL(urlStr);
-  localFile = new File(tmpDir + new File(url.getPath()).getName());
+int attempts = 0;
+boolean valid = false;
+while (attempts <= numRetries) {
+  try {
+URL url = new URL(urlStr);
+localFile = new File(tmpDir + new File(url.getPath()).getName());
 
-  System.out.println("Downloading " + url.toString() + " to " + 
localFile.getAbsolutePath());
-  if (localFile.exists() && !localFile.delete()) {
-System.err.println("File already exists locally and can't be 
deleted.  Please delete before continuing");
-System.exit(3);
+System.out.println("Downloading " + url.toString() + " to " + 
localFile.getAbsolutePath());
+if (localFile.exists() && !localFile.delete()) {
+  System.err.println(
+  "File already exists locally and can't be deleted.  Please 
delete before continuing");
+  System.exit(3);
+}
+FileUtils.copyURLToFile(url, localFile, 5000, 1);
+if (!CompressionStrategies.GZIP.test(localFile)) {
+  throw new IOException("Invalid Gzip file");
+}
+  } catch (MalformedURLException e) {
+System.err.println("Malformed URL - aborting: " + e);
+e.printStackTrace();
--- End diff --

D'oh I retract the statement.  This is to stderr; my bad


> Add gzip file validation check to the geo loader
> 
>
> Key: METRON-1552
> URL: https://issues.apache.org/jira/browse/METRON-1552
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1552) Add gzip file validation check to the geo loader

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472828#comment-16472828
 ] 

ASF GitHub Bot commented on METRON-1552:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/1011#discussion_r187757704
  
--- Diff: 
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/geo/GeoEnrichmentLoader.java
 ---
@@ -167,26 +184,38 @@ protected void loadGeoIpDatabase(CommandLine cli) 
throws IOException {
 System.out.println("Successfully created and updated new GeoIP 
information");
   }
 
-  protected File downloadGeoFile(String urlStr, String tmpDir) {
+  protected File downloadGeoFile(String urlStr, String tmpDir, int  
numRetries) {
 File localFile = null;
-try {
-  URL url = new URL(urlStr);
-  localFile = new File(tmpDir + new File(url.getPath()).getName());
+int attempts = 0;
+boolean valid = false;
+while (attempts <= numRetries) {
+  try {
+URL url = new URL(urlStr);
+localFile = new File(tmpDir + new File(url.getPath()).getName());
 
-  System.out.println("Downloading " + url.toString() + " to " + 
localFile.getAbsolutePath());
-  if (localFile.exists() && !localFile.delete()) {
-System.err.println("File already exists locally and can't be 
deleted.  Please delete before continuing");
-System.exit(3);
+System.out.println("Downloading " + url.toString() + " to " + 
localFile.getAbsolutePath());
+if (localFile.exists() && !localFile.delete()) {
+  System.err.println(
+  "File already exists locally and can't be deleted.  Please 
delete before continuing");
+  System.exit(3);
+}
+FileUtils.copyURLToFile(url, localFile, 5000, 1);
+if (!CompressionStrategies.GZIP.test(localFile)) {
+  throw new IOException("Invalid Gzip file");
+}
+  } catch (MalformedURLException e) {
+System.err.println("Malformed URL - aborting: " + e);
+e.printStackTrace();
+System.exit(4);
+  } catch (IOException e) {
+System.err.println("Warning: Unable to copy remote GeoIP database 
to local file, attempt " + attempts + ": " + e);
+e.printStackTrace();
--- End diff --

same question, print to stderr?


> Add gzip file validation check to the geo loader
> 
>
> Key: METRON-1552
> URL: https://issues.apache.org/jira/browse/METRON-1552
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1552) Add gzip file validation check to the geo loader

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472827#comment-16472827
 ] 

ASF GitHub Bot commented on METRON-1552:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/1011#discussion_r187757683
  
--- Diff: 
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/geo/GeoEnrichmentLoader.java
 ---
@@ -167,26 +184,38 @@ protected void loadGeoIpDatabase(CommandLine cli) 
throws IOException {
 System.out.println("Successfully created and updated new GeoIP 
information");
   }
 
-  protected File downloadGeoFile(String urlStr, String tmpDir) {
+  protected File downloadGeoFile(String urlStr, String tmpDir, int  
numRetries) {
 File localFile = null;
-try {
-  URL url = new URL(urlStr);
-  localFile = new File(tmpDir + new File(url.getPath()).getName());
+int attempts = 0;
+boolean valid = false;
+while (attempts <= numRetries) {
+  try {
+URL url = new URL(urlStr);
+localFile = new File(tmpDir + new File(url.getPath()).getName());
 
-  System.out.println("Downloading " + url.toString() + " to " + 
localFile.getAbsolutePath());
-  if (localFile.exists() && !localFile.delete()) {
-System.err.println("File already exists locally and can't be 
deleted.  Please delete before continuing");
-System.exit(3);
+System.out.println("Downloading " + url.toString() + " to " + 
localFile.getAbsolutePath());
+if (localFile.exists() && !localFile.delete()) {
+  System.err.println(
+  "File already exists locally and can't be deleted.  Please 
delete before continuing");
+  System.exit(3);
+}
+FileUtils.copyURLToFile(url, localFile, 5000, 1);
+if (!CompressionStrategies.GZIP.test(localFile)) {
+  throw new IOException("Invalid Gzip file");
+}
+  } catch (MalformedURLException e) {
+System.err.println("Malformed URL - aborting: " + e);
+e.printStackTrace();
--- End diff --

Should we print the stack trace to stderr?


> Add gzip file validation check to the geo loader
> 
>
> Key: METRON-1552
> URL: https://issues.apache.org/jira/browse/METRON-1552
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1552) Add gzip file validation check to the geo loader

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472826#comment-16472826
 ] 

ASF GitHub Bot commented on METRON-1552:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/1011#discussion_r187757695
  
--- Diff: 
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/geo/GeoEnrichmentLoader.java
 ---
@@ -167,26 +184,38 @@ protected void loadGeoIpDatabase(CommandLine cli) 
throws IOException {
 System.out.println("Successfully created and updated new GeoIP 
information");
   }
 
-  protected File downloadGeoFile(String urlStr, String tmpDir) {
+  protected File downloadGeoFile(String urlStr, String tmpDir, int  
numRetries) {
 File localFile = null;
-try {
-  URL url = new URL(urlStr);
-  localFile = new File(tmpDir + new File(url.getPath()).getName());
+int attempts = 0;
+boolean valid = false;
+while (attempts <= numRetries) {
+  try {
+URL url = new URL(urlStr);
+localFile = new File(tmpDir + new File(url.getPath()).getName());
 
-  System.out.println("Downloading " + url.toString() + " to " + 
localFile.getAbsolutePath());
-  if (localFile.exists() && !localFile.delete()) {
-System.err.println("File already exists locally and can't be 
deleted.  Please delete before continuing");
-System.exit(3);
+System.out.println("Downloading " + url.toString() + " to " + 
localFile.getAbsolutePath());
+if (localFile.exists() && !localFile.delete()) {
+  System.err.println(
+  "File already exists locally and can't be deleted.  Please 
delete before continuing");
+  System.exit(3);
+}
+FileUtils.copyURLToFile(url, localFile, 5000, 1);
+if (!CompressionStrategies.GZIP.test(localFile)) {
+  throw new IOException("Invalid Gzip file");
+}
+  } catch (MalformedURLException e) {
+System.err.println("Malformed URL - aborting: " + e);
+e.printStackTrace();
+System.exit(4);
--- End diff --

Should we exit or throw an exception?


> Add gzip file validation check to the geo loader
> 
>
> Key: METRON-1552
> URL: https://issues.apache.org/jira/browse/METRON-1552
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1552) Add gzip file validation check to the geo loader

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472829#comment-16472829
 ] 

ASF GitHub Bot commented on METRON-1552:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/1011#discussion_r187757615
  
--- Diff: 
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/geo/GeoEnrichmentLoader.java
 ---
@@ -146,7 +156,14 @@ protected void loadGeoIpDatabase(CommandLine cli) 
throws IOException {
 System.out.println("Retrieving GeoLite2 archive");
 String url = GeoEnrichmentOptions.GEO_URL.get(cli, 
"http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz;);
 String tmpDir = GeoEnrichmentOptions.TMP_DIR.get(cli, "/tmp") + "/"; 
// Make sure there's a file separator at the end
-File localGeoFile = downloadGeoFile(url, tmpDir);
+int numRetries = 
Integer.parseInt(GeoEnrichmentOptions.RETRIES.get(cli, "2"));
--- End diff --

can we make "2" a constant?


> Add gzip file validation check to the geo loader
> 
>
> Key: METRON-1552
> URL: https://issues.apache.org/jira/browse/METRON-1552
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1011: METRON-1552: Add gzip file validation check to th...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/1011#discussion_r187757695
  
--- Diff: 
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/geo/GeoEnrichmentLoader.java
 ---
@@ -167,26 +184,38 @@ protected void loadGeoIpDatabase(CommandLine cli) 
throws IOException {
 System.out.println("Successfully created and updated new GeoIP 
information");
   }
 
-  protected File downloadGeoFile(String urlStr, String tmpDir) {
+  protected File downloadGeoFile(String urlStr, String tmpDir, int  
numRetries) {
 File localFile = null;
-try {
-  URL url = new URL(urlStr);
-  localFile = new File(tmpDir + new File(url.getPath()).getName());
+int attempts = 0;
+boolean valid = false;
+while (attempts <= numRetries) {
+  try {
+URL url = new URL(urlStr);
+localFile = new File(tmpDir + new File(url.getPath()).getName());
 
-  System.out.println("Downloading " + url.toString() + " to " + 
localFile.getAbsolutePath());
-  if (localFile.exists() && !localFile.delete()) {
-System.err.println("File already exists locally and can't be 
deleted.  Please delete before continuing");
-System.exit(3);
+System.out.println("Downloading " + url.toString() + " to " + 
localFile.getAbsolutePath());
+if (localFile.exists() && !localFile.delete()) {
+  System.err.println(
+  "File already exists locally and can't be deleted.  Please 
delete before continuing");
+  System.exit(3);
+}
+FileUtils.copyURLToFile(url, localFile, 5000, 1);
+if (!CompressionStrategies.GZIP.test(localFile)) {
+  throw new IOException("Invalid Gzip file");
+}
+  } catch (MalformedURLException e) {
+System.err.println("Malformed URL - aborting: " + e);
+e.printStackTrace();
+System.exit(4);
--- End diff --

Should we exit or throw an exception?


---

[GitHub] metron pull request #1011: METRON-1552: Add gzip file validation check to th...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/1011#discussion_r187757704
  
--- Diff: 
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/geo/GeoEnrichmentLoader.java
 ---
@@ -167,26 +184,38 @@ protected void loadGeoIpDatabase(CommandLine cli) 
throws IOException {
 System.out.println("Successfully created and updated new GeoIP 
information");
   }
 
-  protected File downloadGeoFile(String urlStr, String tmpDir) {
+  protected File downloadGeoFile(String urlStr, String tmpDir, int  
numRetries) {
 File localFile = null;
-try {
-  URL url = new URL(urlStr);
-  localFile = new File(tmpDir + new File(url.getPath()).getName());
+int attempts = 0;
+boolean valid = false;
+while (attempts <= numRetries) {
+  try {
+URL url = new URL(urlStr);
+localFile = new File(tmpDir + new File(url.getPath()).getName());
 
-  System.out.println("Downloading " + url.toString() + " to " + 
localFile.getAbsolutePath());
-  if (localFile.exists() && !localFile.delete()) {
-System.err.println("File already exists locally and can't be 
deleted.  Please delete before continuing");
-System.exit(3);
+System.out.println("Downloading " + url.toString() + " to " + 
localFile.getAbsolutePath());
+if (localFile.exists() && !localFile.delete()) {
+  System.err.println(
+  "File already exists locally and can't be deleted.  Please 
delete before continuing");
+  System.exit(3);
+}
+FileUtils.copyURLToFile(url, localFile, 5000, 1);
+if (!CompressionStrategies.GZIP.test(localFile)) {
+  throw new IOException("Invalid Gzip file");
+}
+  } catch (MalformedURLException e) {
+System.err.println("Malformed URL - aborting: " + e);
+e.printStackTrace();
+System.exit(4);
+  } catch (IOException e) {
+System.err.println("Warning: Unable to copy remote GeoIP database 
to local file, attempt " + attempts + ": " + e);
+e.printStackTrace();
--- End diff --

same question, print to stderr?


---

[GitHub] metron pull request #1011: METRON-1552: Add gzip file validation check to th...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/1011#discussion_r187757683
  
--- Diff: 
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/geo/GeoEnrichmentLoader.java
 ---
@@ -167,26 +184,38 @@ protected void loadGeoIpDatabase(CommandLine cli) 
throws IOException {
 System.out.println("Successfully created and updated new GeoIP 
information");
   }
 
-  protected File downloadGeoFile(String urlStr, String tmpDir) {
+  protected File downloadGeoFile(String urlStr, String tmpDir, int  
numRetries) {
 File localFile = null;
-try {
-  URL url = new URL(urlStr);
-  localFile = new File(tmpDir + new File(url.getPath()).getName());
+int attempts = 0;
+boolean valid = false;
+while (attempts <= numRetries) {
+  try {
+URL url = new URL(urlStr);
+localFile = new File(tmpDir + new File(url.getPath()).getName());
 
-  System.out.println("Downloading " + url.toString() + " to " + 
localFile.getAbsolutePath());
-  if (localFile.exists() && !localFile.delete()) {
-System.err.println("File already exists locally and can't be 
deleted.  Please delete before continuing");
-System.exit(3);
+System.out.println("Downloading " + url.toString() + " to " + 
localFile.getAbsolutePath());
+if (localFile.exists() && !localFile.delete()) {
+  System.err.println(
+  "File already exists locally and can't be deleted.  Please 
delete before continuing");
+  System.exit(3);
+}
+FileUtils.copyURLToFile(url, localFile, 5000, 1);
+if (!CompressionStrategies.GZIP.test(localFile)) {
+  throw new IOException("Invalid Gzip file");
+}
+  } catch (MalformedURLException e) {
+System.err.println("Malformed URL - aborting: " + e);
+e.printStackTrace();
--- End diff --

Should we print the stack trace to stderr?


---

[GitHub] metron pull request #1011: METRON-1552: Add gzip file validation check to th...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/1011#discussion_r187757615
  
--- Diff: 
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/geo/GeoEnrichmentLoader.java
 ---
@@ -146,7 +156,14 @@ protected void loadGeoIpDatabase(CommandLine cli) 
throws IOException {
 System.out.println("Retrieving GeoLite2 archive");
 String url = GeoEnrichmentOptions.GEO_URL.get(cli, 
"http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz;);
 String tmpDir = GeoEnrichmentOptions.TMP_DIR.get(cli, "/tmp") + "/"; 
// Make sure there's a file separator at the end
-File localGeoFile = downloadGeoFile(url, tmpDir);
+int numRetries = 
Integer.parseInt(GeoEnrichmentOptions.RETRIES.get(cli, "2"));
--- End diff --

can we make "2" a constant?


---

[jira] [Commented] (METRON-1552) Add gzip file validation check to the geo loader

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472823#comment-16472823
 ] 

ASF GitHub Bot commented on METRON-1552:


GitHub user mmiklavc opened a pull request:

https://github.com/apache/metron/pull/1011

METRON-1552: Add gzip file validation check to the geo loader

## Contributor Comments

https://issues.apache.org/jira/browse/METRON-1552

We don't have retries implemented in the loader, so network and other type 
of failures will cause the download to blow up. This adds default retries to 
the loader that can be overridden by the cli options passed to the loader.

I need to finish manual testing, but putting it out there. Also added a 
compression strategy class to metron-common to handle the validity check.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
n/a

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mmiklavc/metron add-geo-file-check

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/1011.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1011


commit 8421dea9f27bc899cd202801d5dc9c6d238dde0f
Author: Michael Miklavcic 
Date:   2018-05-12T00:14:42Z

Add check for Geo database file




> Add gzip file validation check to the geo loader
> 
>
> Key: METRON-1552
> URL: https://issues.apache.org/jira/browse/METRON-1552
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1011: METRON-1552: Add gzip file validation check to th...

[mmiklavc]

GitHub user mmiklavc opened a pull request:

https://github.com/apache/metron/pull/1011

METRON-1552: Add gzip file validation check to the geo loader

## Contributor Comments

https://issues.apache.org/jira/browse/METRON-1552

We don't have retries implemented in the loader, so network and other type 
of failures will cause the download to blow up. This adds default retries to 
the loader that can be overridden by the cli options passed to the loader.

I need to finish manual testing, but putting it out there. Also added a 
compression strategy class to metron-common to handle the validity check.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
n/a

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mmiklavc/metron add-geo-file-check

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/1011.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1011


commit 8421dea9f27bc899cd202801d5dc9c6d238dde0f
Author: Michael Miklavcic 
Date:   2018-05-12T00:14:42Z

Add check for Geo database file




---

[jira] [Created] (METRON-1552) Add gzip file validation check to the geo loader

[Michael Miklavcic (JIRA)]

Michael Miklavcic created METRON-1552:
-

 Summary: Add gzip file validation check to the geo loader
 Key: METRON-1552
 URL: https://issues.apache.org/jira/browse/METRON-1552
 Project: Metron
  Issue Type: Improvement
Reporter: Michael Miklavcic
Assignee: Michael Miklavcic






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1421) Create a SolrMetaAlertDao

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472670#comment-16472670
 ] 

ASF GitHub Bot commented on METRON-1421:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/970
  
Pulled in https://github.com/justinleet/metron/pull/18 from @merrimanr to 
address the reported bugs when metaalerts become empty.  Thanks for digging 
into the bug you found, it's super helpful!

The short story for anyone here, is that the latest commit prevents the 
removal of all child alerts from a metalert (which caused problems for Solr in 
particular).  Given that having an empty collection of alerts has no useful 
meaning (and there's no score and such), it's just disallowed entirely if a 
remove request will leave a metaalert empty.


> Create a SolrMetaAlertDao
> -
>
> Key: METRON-1421
> URL: https://issues.apache.org/jira/browse/METRON-1421
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
>
> Create an implementation of the MetaAlertDao for Solr. This will involve 
> implementing the various MetaAlertDao methods using the SolrJ library and 
> also providing a SolrMetaAlertIntegrationTest (similar to 
> ElasticsearchMetaAlertIntegrationTest).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #970: METRON-1421: Create a SolrMetaAlertDao

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/970
  
Pulled in https://github.com/justinleet/metron/pull/18 from @merrimanr to 
address the reported bugs when metaalerts become empty.  Thanks for digging 
into the bug you found, it's super helpful!

The short story for anyone here, is that the latest commit prevents the 
removal of all child alerts from a metalert (which caused problems for Solr in 
particular).  Given that having an empty collection of alerts has no useful 
meaning (and there's no score and such), it's just disallowed entirely if a 
remove request will leave a metaalert empty.


---

[jira] [Created] (METRON-1551) Profiler Should Not Use Java Serialization

[Nick Allen (JIRA)]

Nick Allen created METRON-1551:
--

 Summary: Profiler Should Not Use Java Serialization
 Key: METRON-1551
 URL: https://issues.apache.org/jira/browse/METRON-1551
 Project: Metron
  Issue Type: Bug
Reporter: Nick Allen


When running the Profiler in a topology where serialization occurs, the 
following error happens.  This can occur when the number of workers is greater 
than 1.

The topology should not be using Java serialization for serializing tuple 
values as this will negatively impact performance. 

{code}
2018-05-09 10:48:35.136 o.a.s.d.executor [ERROR] 
java.lang.RuntimeException: java.lang.RuntimeException: 
java.io.NotSerializableException: 
org.apache.metron.common.configuration.profiler.ProfileResult
 at 
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.disruptor$consume_loop_STAR_$fn__7183.invoke(disruptor.clj:83) 
~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) 
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
 at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.RuntimeException: java.io.NotSerializableException: 
org.apache.metron.common.configuration.profiler.ProfileResult
 at 
org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:41)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) 
~[kryo-3.0.3.jar:?]
 at 
com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
 ~[kryo-3.0.3.jar:?]
 at 
com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
 ~[kryo-3.0.3.jar:?]
 at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) 
~[kryo-3.0.3.jar:?]
 at 
org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 ... 6 more
Caused by: java.io.NotSerializableException: 
org.apache.metron.common.configuration.profiler.ProfileResult
 at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) 
~[?:1.8.0_162]
 at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) 
~[?:1.8.0_162]
 at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) 
~[?:1.8.0_162]
 at 
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) 
~[?:1.8.0_162]
 at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) 
~[?:1.8.0_162]
 at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) 
~[?:1.8.0_162]
 at 
org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:38)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) 
~[kryo-3.0.3.jar:?]
 at 
com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
 ~[kryo-3.0.3.jar:?]
 at 
com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
 ~[kryo-3.0.3.jar:?]
 at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) 
~[kryo-3.0.3.jar:?]
 at 
org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 

[jira] [Commented] (METRON-1549) Add empty object test to WriterBoltIntegrationTest implementation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472380#comment-16472380
 ] 

ASF GitHub Bot commented on METRON-1549:


Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/1009


> Add empty object test to WriterBoltIntegrationTest implementation
> -
>
> Key: METRON-1549
> URL: https://issues.apache.org/jira/browse/METRON-1549
> Project: Metron
>  Issue Type: Test
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
> Verify expected output when empty objects returned by custom parsers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1009: METRON-1549: Add empty object test to WriterBoltI...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/1009


---

[jira] [Commented] (METRON-1548) Alerts UI: Remove hardcoded source:type and other fields that may vary

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471944#comment-16471944
 ] 

ASF GitHub Bot commented on METRON-1548:


GitHub user sardell opened a pull request:

https://github.com/apache/metron/pull/1010

METRON-1548: Remove hardcoded source:type from Alerts UI

## Contributor Comments
Instead of 'source:type' being hard coded as a string in the Alerts UI, we 
now use a global-config service to retrieve whether a user's setup is using 
"source:type" or "source.type" syntax (e.g. elastic search or solr). The app 
default's to using "source:type" if no configuration is specified. If a user 
wants, they can update their configuration to use source.type by adding 
`"sourceType": "source.type"` to their configuration.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sardell/metron METRON-1548

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/1010.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1010


commit 1f6edee9a8a9a2b83072408a5089f1b95b35bb4b
Author: Shane Ardell 
Date:   2018-05-07T11:19:42Z

remove hard-coded source:type and replace with env variable

commit c37d23988a6466256512c7417d855fc15af0b065
Author: Shane Ardell 
Date:   2018-05-11T12:20:51Z

get config from API instead of a environment file

commit be59b8e7e8f5abfd1ab3edf21658695e4cd4e46e
Author: Shane Ardell 
Date:   2018-05-11T13:32:56Z

remove unused facet and group constants

commit ec68878b96d8831e03299d19c8daf7cef7f3104b
Author: Shane Ardell 
Date:   2018-05-11T13:34:41Z

use global config to set source type in update service

commit e8935077f779cbffcc35400014cf24d6bf0e2de5
Author: Shane Ardell 
Date:   2018-05-11T13:37:23Z

remove sourceType var from environment file




> Alerts UI: Remove hardcoded source:type and other fields that may vary
> 

[GitHub] metron pull request #1010: METRON-1548: Remove hardcoded source:type from Al...

[sardell]

GitHub user sardell opened a pull request:

https://github.com/apache/metron/pull/1010

METRON-1548: Remove hardcoded source:type from Alerts UI

## Contributor Comments
Instead of 'source:type' being hard coded as a string in the Alerts UI, we 
now use a global-config service to retrieve whether a user's setup is using 
"source:type" or "source.type" syntax (e.g. elastic search or solr). The app 
default's to using "source:type" if no configuration is specified. If a user 
wants, they can update their configuration to use source.type by adding 
`"sourceType": "source.type"` to their configuration.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sardell/metron METRON-1548

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/1010.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1010


commit 1f6edee9a8a9a2b83072408a5089f1b95b35bb4b
Author: Shane Ardell 
Date:   2018-05-07T11:19:42Z

remove hard-coded source:type and replace with env variable

commit c37d23988a6466256512c7417d855fc15af0b065
Author: Shane Ardell 
Date:   2018-05-11T12:20:51Z

get config from API instead of a environment file

commit be59b8e7e8f5abfd1ab3edf21658695e4cd4e46e
Author: Shane Ardell 
Date:   2018-05-11T13:32:56Z

remove unused facet and group constants

commit ec68878b96d8831e03299d19c8daf7cef7f3104b
Author: Shane Ardell 
Date:   2018-05-11T13:34:41Z

use global config to set source type in update service

commit e8935077f779cbffcc35400014cf24d6bf0e2de5
Author: Shane Ardell 
Date:   2018-05-11T13:37:23Z

remove sourceType var from environment file




---

[jira] [Commented] (METRON-1549) Add empty object test to WriterBoltIntegrationTest implementation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471852#comment-16471852
 ] 

ASF GitHub Bot commented on METRON-1549:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1009
  
+1


> Add empty object test to WriterBoltIntegrationTest implementation
> -
>
> Key: METRON-1549
> URL: https://issues.apache.org/jira/browse/METRON-1549
> Project: Metron
>  Issue Type: Test
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
> Verify expected output when empty objects returned by custom parsers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1009: METRON-1549: Add empty object test to WriterBoltIntegrat...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1009
  
+1


---

[jira] [Commented] (METRON-1549) Add empty object test to WriterBoltIntegrationTest implementation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471846#comment-16471846
 ] 

ASF GitHub Bot commented on METRON-1549:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1009
  
+1 This looks good.  Thanks!


> Add empty object test to WriterBoltIntegrationTest implementation
> -
>
> Key: METRON-1549
> URL: https://issues.apache.org/jira/browse/METRON-1549
> Project: Metron
>  Issue Type: Test
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
> Verify expected output when empty objects returned by custom parsers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1009: METRON-1549: Add empty object test to WriterBoltIntegrat...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1009
  
+1 This looks good.  Thanks!


---

[jira] [Commented] (METRON-1549) Add empty object test to WriterBoltIntegrationTest implementation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471790#comment-16471790
 ] 

ASF GitHub Bot commented on METRON-1549:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1009
  
I think this is good.  My work on the builder/Processor stuff was a decent 
attempt to reduce duplicate code in the integration tests without a total 
rethink.  I think what we see across the integration tests is so much 
commonality that it is apparent that some re-think could be useful to have more 
standardize test classes and use cases structured such that there is less 
overhead of duplicated code required.

As I have done some work on other projects and seen other approaches this 
has only become more clear.  commons-vfs for example runs the same tests 
against every provider, with each provider providing specialization of it's 
suites. 

It would seem that we should be able to have a more generic suite or 
testing harness for *n* topologies/topics working together, as a next step 
refactoring of my prior `Process` work.

I don't think this is the PR for that though


> Add empty object test to WriterBoltIntegrationTest implementation
> -
>
> Key: METRON-1549
> URL: https://issues.apache.org/jira/browse/METRON-1549
> Project: Metron
>  Issue Type: Test
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
> Verify expected output when empty objects returned by custom parsers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1009: METRON-1549: Add empty object test to WriterBoltIntegrat...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1009
  
I think this is good.  My work on the builder/Processor stuff was a decent 
attempt to reduce duplicate code in the integration tests without a total 
rethink.  I think what we see across the integration tests is so much 
commonality that it is apparent that some re-think could be useful to have more 
standardize test classes and use cases structured such that there is less 
overhead of duplicated code required.

As I have done some work on other projects and seen other approaches this 
has only become more clear.  commons-vfs for example runs the same tests 
against every provider, with each provider providing specialization of it's 
suites. 

It would seem that we should be able to have a more generic suite or 
testing harness for *n* topologies/topics working together, as a next step 
refactoring of my prior `Process` work.

I don't think this is the PR for that though


---