[jira] [Created] (METRON-2213) Documentation not deployed
Dale Richardson created METRON-2213: --- Summary: Documentation not deployed Key: METRON-2213 URL: https://issues.apache.org/jira/browse/METRON-2213 Project: Metron Issue Type: Bug Reporter: Dale Richardson [https://metron.apache.org/current-book/index.html#Navigating_the_Architecture] has a link to "parsers", (I am not sure the Navigating the architecture internal link is valid either). which links to [https://metron.apache.org/current-book/metron-platform/metron-parsers-common/index.html] This web page does not appear to be deployed. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [metron] mmiklavc commented on issue #1470: METRON-2193 Upgrade Enrichments for HBase 2.0.2
mmiklavc commented on issue #1470: METRON-2193 Upgrade Enrichments for HBase 2.0.2 URL: https://github.com/apache/metron/pull/1470#issuecomment-521073596 Still trying to wrap my head around this PR and the scope of changes. > Many of the changes here are not backwards compatible which prevents me from introducing them against master, unfortunately. HBase introduced the new API in earlier versions, prior to deprecation. I don't doubt the problem exists, but I'm having a hard time understanding what we depended on that has 2 incompatible analogs in the newer versions of the HBase API. Can you provide some concrete examples? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] mmiklavc commented on issue #1341: METRON-614: Eliminate use of the default Charset
mmiklavc commented on issue #1341: METRON-614: Eliminate use of the default Charset URL: https://github.com/apache/metron/pull/1341#issuecomment-521062716 That is correct @justinleet This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] nickwallen commented on issue #1470: METRON-2193 Upgrade Enrichments for HBase 2.0.2
nickwallen commented on issue #1470: METRON-2193 Upgrade Enrichments for HBase 2.0.2 URL: https://github.com/apache/metron/pull/1470#issuecomment-521045619 > Replacing deprecated API calls against master Many of the changes here are not backwards compatible which prevents me from introducing them against master, unfortunately. > This is also a decent time to, instead of replacing the mistake of passing around HTableInterface to create a key-value store abstraction which you can pass around instead that supports scan, get and put. For the work on this feature branch, the `HBaseClient` is that common abstraction. It was already in the code base and was used in a few different areas, so I leveraged that. As part of the upgrade, I've tried to port over as many functions as possible to use the `HBaseClient`. See #1456 for more context. There are still a few remaining places that don't directly use an `HBaseClient`, but instead pass around HBase abstractions like `Table` as they always have. There is just a limit to how much time I can spend on this upgrade. The ones that I have not ported to `HBaseClient` were ones where the manner in which they are tested did not force my hand in transitioning them to `HBaseClient`. For example, we have to integration test "streaming enrichments" against HBase, Kafka and Storm. With our current IT approach, these all have to run in memory. But we cannot run HBase in-memory with Storm and Kafka, which is why the integration tests in `master` actually use a "mock" HBase instance. In these cases since HBase cannot co-exist in memory, these classes were ported to `HBaseClient` and so ultimately use a `FakeHBaseClient`in the IT. For those that were not ported to `HbaseClient`, I'd like us to upgrade or deprecate those after this feature branch is complete. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] nickwallen commented on issue #1470: METRON-2193 Upgrade Enrichments for HBase 2.0.2
nickwallen commented on issue #1470: METRON-2193 Upgrade Enrichments for HBase 2.0.2 URL: https://github.com/apache/metron/pull/1470#issuecomment-521041113 > I will say, however, that I was surprised at the size of a single PR until I looked... Conflating so many things inside of a PR which is already inherently risky seems to increase the risk multiplicatively. It is also difficult to review, I'd say. I do agree with you @cestella that this PR is large and I very much prefer small, isolated PRs. The HDP-3.1 upgrade has been a huge effort so far and I've tried to break it down into small, reviewable PRs. Here are all the PRs that we've been able to land so far as part of the upgrade. * #1395 * #1397 * #1447 * #1448 * #1451 * #1454 * #1456 * #1457 * #1458 I tried to follow that similar pattern prior to opening this PR and ran into some problems. I attempted to open separate PRs for each functional area included here. Something along the lines of... * Enrichment Coprocessor * Legacy Adapters and Stellar Enrichment functions * Data management; TAXII, CSV loaders But since (1) the changes in the areas listed below are not backwards compatible (unlike all the other preceding PRs listed above) and (2) there are many interdependencies between these areas, I was not able to submit separate PRs that would actually compile. To help cut the fat and reduce the size of this PR, there are some changes here that I could try to undo or extract into separate PRs. These come to mind immediately 1. We have a public field accessed in `EnrichmentKey`. This was changed to getters and it ended up impacting a lot of files. 1. `LookupKV` -> `EnrichmentResult`. I actually tried to 'undo' this before submitting this PR and I ran into a problem. I can try to to tackle this again so I can at least describe why this might be needed, if not address it. 1. The deprecation of "Least Recently Used Pruner" (assuming that is acceptable to the community post-discuss) would have to come before this PR. > Rearchitecture/code abstraction rewriting against.. Do the items I listed above (1) public field access and (2) LookupKV -> EnrichmentResult cover what you mean by rearchitecture? Are there other items that you are thinking of that fall under this heading? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] justinleet commented on issue #1341: METRON-614: Eliminate use of the default Charset
justinleet commented on issue #1341: METRON-614: Eliminate use of the default Charset URL: https://github.com/apache/metron/pull/1341#issuecomment-521040026 @mmiklavc If they're using a system default that isn't UTF-8 (say ISO_8859_1), all the parsers that currently work would silently start using the new default of UTF-8 and no longer be able to parse the data. Let me know if this understanding is incorrect. For most, if not all, users this shouldn't be a concern, but I just want to make sure it's communicated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] cestella commented on a change in pull request #1470: METRON-2193 Upgrade Enrichments for HBase 2.0.2
cestella commented on a change in pull request #1470: METRON-2193 Upgrade
Enrichments for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1470#discussion_r313643143
##
File path:
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/bulk/LeastRecentlyUsedPruner.java
##
@@ -1,221 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.metron.dataloads.bulk;
-
-import com.google.common.base.Function;
-import com.google.common.base.Joiner;
-import org.apache.commons.cli.*;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hbase.HBaseConfiguration;
-import org.apache.hadoop.hbase.client.Scan;
-import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.mapreduce.Job;
-import org.apache.hadoop.util.GenericOptionsParser;
-import org.apache.metron.dataloads.hbase.mr.PrunerMapper;
-
-import javax.annotation.Nullable;
-import java.io.IOException;
-import java.text.DateFormat;
-import java.text.SimpleDateFormat;
-import java.util.Date;
-
-public class LeastRecentlyUsedPruner {
Review comment:
nice to see you too! I just responded.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] mmiklavc commented on issue #1341: METRON-614: Eliminate use of the default Charset
mmiklavc commented on issue #1341: METRON-614: Eliminate use of the default Charset URL: https://github.com/apache/metron/pull/1341#issuecomment-521034747 I assume you mean if they were using the system default, which is Java's default when a charset is not provided when converting from bytearray to String? Just to be clear, this is an upgrade concern based off the fundamental nature of this PR change as a whole, not specifically the specific parser configs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] cestella commented on issue #1470: METRON-2193 Upgrade Enrichments for HBase 2.0.2
cestella commented on issue #1470: METRON-2193 Upgrade Enrichments for HBase 2.0.2 URL: https://github.com/apache/metron/pull/1470#issuecomment-521033037 This PR piqued my interest. :) First off, I'm glad to see we're fixing the use of the deprecated `HTables` and HBase APIs, so that's fantastic. Thanks for the effort on this @nickwallen . I will say, however, that I was surprised at the size of a single PR until I looked. This PR seems to be a mix of: * Dependency changes for HBase 2.0.2 * Rearchitecture/code abstraction rewriting (e.g. your point 11) * Replacing the deprecated API calls * Deprecating features I have some concerns about this approach. Conflating so many things inside of a PR which is already inherently risky seems to increase the risk multiplicatively. It is also difficult to review, I'd say. I would suggest that these three separate concerns be split across 3 separate PRs: * Replacing deprecated API calls against `master` * This is also a decent time to, instead of replacing the mistake of passing around `HTableInterface` to create a key-value store abstraction which you can pass around instead that supports scan, get and put. * Dependency changes for HBase 2.0.2 against this feature branch * Deprecating features * As I said in a previous comment, only after a community discussion and I'd strongly suggest it be separate from the upgrade. * Rearchitecture/code abstraction rewriting against `master` *after* the HDP upgrade has landed. As I say, please don't take my feedback as an indication that I don't appreciate the work that went into this. There is much good here, but there is just..well..very much here. :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] justinleet commented on issue #1341: METRON-614: Eliminate use of the default Charset
justinleet commented on issue #1341: METRON-614: Eliminate use of the default Charset URL: https://github.com/apache/metron/pull/1341#issuecomment-521029586 @mmiklavc We should probably add a note to the Upgrading.md. There is a backwards compatibility concern when upgrading: When a current user's default Charset is not UTF_8, they'll need to update their parser configs to use the appropriate one, or parsing will run into issues. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] ottobackwards commented on a change in pull request #1470: METRON-2193 Upgrade Enrichments for HBase 2.0.2
ottobackwards commented on a change in pull request #1470: METRON-2193 Upgrade
Enrichments for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1470#discussion_r313635130
##
File path:
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/bulk/LeastRecentlyUsedPruner.java
##
@@ -1,221 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.metron.dataloads.bulk;
-
-import com.google.common.base.Function;
-import com.google.common.base.Joiner;
-import org.apache.commons.cli.*;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hbase.HBaseConfiguration;
-import org.apache.hadoop.hbase.client.Scan;
-import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.mapreduce.Job;
-import org.apache.hadoop.util.GenericOptionsParser;
-import org.apache.metron.dataloads.hbase.mr.PrunerMapper;
-
-import javax.annotation.Nullable;
-import java.io.IOException;
-import java.text.DateFormat;
-import java.text.SimpleDateFormat;
-import java.util.Date;
-
-public class LeastRecentlyUsedPruner {
Review comment:
@cestella can you answer the discuss around what this actually does? What
the goal was?
Nice to see you ;)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1470: METRON-2193 Upgrade Enrichments for HBase 2.0.2
nickwallen commented on a change in pull request #1470: METRON-2193 Upgrade
Enrichments for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1470#discussion_r313626182
##
File path:
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/bulk/LeastRecentlyUsedPruner.java
##
@@ -1,221 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.metron.dataloads.bulk;
-
-import com.google.common.base.Function;
-import com.google.common.base.Joiner;
-import org.apache.commons.cli.*;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hbase.HBaseConfiguration;
-import org.apache.hadoop.hbase.client.Scan;
-import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.mapreduce.Job;
-import org.apache.hadoop.util.GenericOptionsParser;
-import org.apache.metron.dataloads.hbase.mr.PrunerMapper;
-
-import javax.annotation.Nullable;
-import java.io.IOException;
-import java.text.DateFormat;
-import java.text.SimpleDateFormat;
-import java.util.Date;
-
-public class LeastRecentlyUsedPruner {
Review comment:
Agreed. I can open that discuss thread.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] ottobackwards commented on issue #1341: METRON-614: Eliminate use of the default Charset
ottobackwards commented on issue #1341: METRON-614: Eliminate use of the default Charset URL: https://github.com/apache/metron/pull/1341#issuecomment-521019388 Ran this up in full-dev, everything seems to be working. Ship it. +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] cestella commented on a change in pull request #1470: METRON-2193 Upgrade Enrichments for HBase 2.0.2
cestella commented on a change in pull request #1470: METRON-2193 Upgrade
Enrichments for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1470#discussion_r313617975
##
File path:
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/bulk/LeastRecentlyUsedPruner.java
##
@@ -1,221 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.metron.dataloads.bulk;
-
-import com.google.common.base.Function;
-import com.google.common.base.Joiner;
-import org.apache.commons.cli.*;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hbase.HBaseConfiguration;
-import org.apache.hadoop.hbase.client.Scan;
-import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.mapreduce.Job;
-import org.apache.hadoop.util.GenericOptionsParser;
-import org.apache.metron.dataloads.hbase.mr.PrunerMapper;
-
-import javax.annotation.Nullable;
-import java.io.IOException;
-import java.text.DateFormat;
-import java.text.SimpleDateFormat;
-import java.util.Date;
-
-public class LeastRecentlyUsedPruner {
Review comment:
I completely agree that deprecating a feature merits a community discussion.
I'd start a discuss thread about this whole PR if there are any breaking
changes.
For the record, I created the feature and would be in favor of deprecating
it, but only after a discussion.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] cstella-stripe commented on a change in pull request #1470: METRON-2193 Upgrade Enrichments for HBase 2.0.2
cstella-stripe commented on a change in pull request #1470: METRON-2193 Upgrade
Enrichments for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1470#discussion_r313616983
##
File path:
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/bulk/LeastRecentlyUsedPruner.java
##
@@ -1,221 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.metron.dataloads.bulk;
-
-import com.google.common.base.Function;
-import com.google.common.base.Joiner;
-import org.apache.commons.cli.*;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hbase.HBaseConfiguration;
-import org.apache.hadoop.hbase.client.Scan;
-import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.mapreduce.Job;
-import org.apache.hadoop.util.GenericOptionsParser;
-import org.apache.metron.dataloads.hbase.mr.PrunerMapper;
-
-import javax.annotation.Nullable;
-import java.io.IOException;
-import java.text.DateFormat;
-import java.text.SimpleDateFormat;
-import java.util.Date;
-
-public class LeastRecentlyUsedPruner {
Review comment:
I completely agree that deprecating a feature merits a community discussion.
I'd start a discuss thread about this whole PR if there are any breaking
changes.
For the record, I created the feature and would be in favor of deprecating
it, but only after a discussion.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] cstella-stripe commented on a change in pull request #1470: METRON-2193 Upgrade Enrichments for HBase 2.0.2
cstella-stripe commented on a change in pull request #1470: METRON-2193 Upgrade
Enrichments for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1470#discussion_r313616983
##
File path:
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/bulk/LeastRecentlyUsedPruner.java
##
@@ -1,221 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.metron.dataloads.bulk;
-
-import com.google.common.base.Function;
-import com.google.common.base.Joiner;
-import org.apache.commons.cli.*;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hbase.HBaseConfiguration;
-import org.apache.hadoop.hbase.client.Scan;
-import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.mapreduce.Job;
-import org.apache.hadoop.util.GenericOptionsParser;
-import org.apache.metron.dataloads.hbase.mr.PrunerMapper;
-
-import javax.annotation.Nullable;
-import java.io.IOException;
-import java.text.DateFormat;
-import java.text.SimpleDateFormat;
-import java.util.Date;
-
-public class LeastRecentlyUsedPruner {
Review comment:
I completely agree that deprecating a feature merits a community discussion.
I'd start a discuss thread about this whole PR if there are any breaking
changes.
For the record, I created the feature and would be in favor of deprecating
it, but only after a discussion.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] justinleet commented on issue #1341: METRON-614: Eliminate use of the default Charset
justinleet commented on issue #1341: METRON-614: Eliminate use of the default Charset URL: https://github.com/apache/metron/pull/1341#issuecomment-521011869 Merged. Adds some docs and the test fix. @mmiklavc Assuming the build succeeds, are we good for @nickwallen + @ottobackwards to follow up on their reviews? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] mmiklavc commented on a change in pull request #1470: METRON-2193 Upgrade Enrichments for HBase 2.0.2
mmiklavc commented on a change in pull request #1470: METRON-2193 Upgrade
Enrichments for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1470#discussion_r313615221
##
File path:
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/bulk/LeastRecentlyUsedPruner.java
##
@@ -1,221 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.metron.dataloads.bulk;
-
-import com.google.common.base.Function;
-import com.google.common.base.Joiner;
-import org.apache.commons.cli.*;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hbase.HBaseConfiguration;
-import org.apache.hadoop.hbase.client.Scan;
-import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.mapreduce.Job;
-import org.apache.hadoop.util.GenericOptionsParser;
-import org.apache.metron.dataloads.hbase.mr.PrunerMapper;
-
-import javax.annotation.Nullable;
-import java.io.IOException;
-import java.text.DateFormat;
-import java.text.SimpleDateFormat;
-import java.util.Date;
-
-public class LeastRecentlyUsedPruner {
Review comment:
> 10. Removes the LeastRecentlyUsedPruner. This logic is exposed to the user
in the script bin/threatintel_bulk_prune.sh. I had some difficulty getting the
integration test working and I do not believe this is worth the effort to
upgrade. I found almost no documentation around this functionality. I fully
expect to initiate more discussion around this. If there is a need for this, I
can work further on upgrading it.
I think deprecating an entire feature set probably requires a discussion
before we accept work that completely removes it. Typically when we've done
this in the past there's been at least a complementary, if not more robust,
alternative to the existing functionality.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] mmiklavc commented on issue #1341: METRON-614: Eliminate use of the default Charset
mmiklavc commented on issue #1341: METRON-614: Eliminate use of the default Charset URL: https://github.com/apache/metron/pull/1341#issuecomment-520998393 Well that's odd - apparently our version of Mockito doesn't call the provided concrete method for default interface methods. Submitting another fix now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] justinleet commented on issue #1341: METRON-614: Eliminate use of the default Charset
justinleet commented on issue #1341: METRON-614: Eliminate use of the default Charset URL: https://github.com/apache/metron/pull/1341#issuecomment-520965262 @mmiklavc Unit test looks like it fails because a mocked parser NPEs on the `getReadCharset`. Could just do a `when(broParser.getReadCharset()).thenReturn(StandardCharsets.UTF_8);` to fix the the issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] mmiklavc commented on issue #1474: METRON-2201: The description for the IS_IP method default behavior needs to corrected as per implementation
mmiklavc commented on issue #1474: METRON-2201: The description for the IS_IP method default behavior needs to corrected as per implementation URL: https://github.com/apache/metron/pull/1474#issuecomment-520930649 +1, thanks @MohanDV This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] mmiklavc commented on issue #1474: METRON-2201: The description for the IS_IP method default behavior needs to corrected as per implementation
mmiklavc commented on issue #1474: METRON-2201: The description for the IS_IP
method default behavior needs to corrected as per implementation
URL: https://github.com/apache/metron/pull/1474#issuecomment-520930529
For reference, the underlying implementation of this validator
https://commons.apache.org/proper/commons-validator/apidocs/src-html/org/apache/commons/validator/routines/InetAddressValidator.html
performs an "or" on IPV4/IPV6
```
077public boolean isValid(String inetAddress) {
078return isValidInet4Address(inetAddress) ||
isValidInet6Address(inetAddress);
079}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] ottobackwards commented on issue #1481: METRON-2212 Add debugging developer docs to hbase-server README
ottobackwards commented on issue #1481: METRON-2212 Add debugging developer docs to hbase-server README URL: https://github.com/apache/metron/pull/1481#issuecomment-520927977 +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] justinleet commented on issue #1341: METRON-614: Eliminate use of the default Charset
justinleet commented on issue #1341: METRON-614: Eliminate use of the default Charset URL: https://github.com/apache/metron/pull/1341#issuecomment-520927745 The PR from @mmiklavc (https://github.com/justinleet/metron/pull/22) got merged to address the couple `readCharset` issues. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] justinleet commented on a change in pull request #1341: METRON-614: Eliminate use of the default Charset
justinleet commented on a change in pull request #1341: METRON-614: Eliminate
use of the default Charset
URL: https://github.com/apache/metron/pull/1341#discussion_r313517446
##
File path:
metron-platform/metron-parsing/metron-parsers-common/src/main/java/org/apache/metron/parsers/interfaces/MessageParser.java
##
@@ -81,4 +81,11 @@
*/
boolean validate(T message);
+ /**
+ * Provides a hook to override the default charset parsers use to read data.
+ * @return Charset to use for for reading
+ */
+ default Charset getReadCharset() {
+return StandardCharsets.UTF_8;
Review comment:
Sounds good. I'd honestly just forgotten that conversation existed at this
point and missed it when I too quickly looked.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] mmiklavc opened a new pull request #1481: METRON-2212 Add debugging developer docs to hbase-server README
mmiklavc opened a new pull request #1481: METRON-2212 Add debugging developer
docs to hbase-server README
URL: https://github.com/apache/metron/pull/1481
## Contributor Comments
https://issues.apache.org/jira/browse/METRON-2212
Adds some detail to assist developers/devops in debugging HBase regionserver
issues. This is a very basic documentation update. I verified formatting with
mvn clean site. All commands were tested and verified in full dev.
## Pull Request Checklist
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be
created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- n/a Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- n/a Have you included steps or a guide to how the change may be verified
and tested manually?
- n/a Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```
- n/a Have you written or updated unit tests and or integration tests to
verify your changes?
- n/a If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- n/a Have you verified the basic functionality of the build by building and
running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in which
it is rendered by building and verifying the site-book? If not then run the
following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
- n/a Have you ensured that any documentation diagrams have been updated,
along with their source files, using [draw.io](https://www.draw.io/)? See
[Metron Development
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines)
for instructions.
Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for
your personal repository such that your branches are built there before
submitting a pull request.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (METRON-2212) Add debugging developer docs to hbase-server README
Michael Miklavcic created METRON-2212: - Summary: Add debugging developer docs to hbase-server README Key: METRON-2212 URL: https://issues.apache.org/jira/browse/METRON-2212 Project: Metron Issue Type: Improvement Reporter: Michael Miklavcic Assignee: Michael Miklavcic -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (METRON-2211) [UI] Alerts UI should optionally render timestamp in local time
Shane Ardell created METRON-2211: Summary: [UI] Alerts UI should optionally render timestamp in local time Key: METRON-2211 URL: https://issues.apache.org/jira/browse/METRON-2211 Project: Metron Issue Type: Improvement Reporter: Shane Ardell Currently, timestamps show in UTC time when the timestamp field is included as a column header. This could be confusing for a user seeing alerts "from the future" because their local time is behind UTC time. Even if there is no confusion, a user might prefer seeing alerts based on their local time vs UTC. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [metron] tiborm closed pull request #1480: METRON-2209: [UI] Showing loading indicator when querying alerts
tiborm closed pull request #1480: METRON-2209: [UI] Showing loading indicator when querying alerts URL: https://github.com/apache/metron/pull/1480 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
