[GitHub] nickwallen commented on issue #1325: METRON-1970 Add Metadata to Error Messages Generated During Parsing
nickwallen commented on issue #1325: METRON-1970 Add Metadata to Error Messages Generated During Parsing URL: https://github.com/apache/metron/pull/1325#issuecomment-460406276 I validated that Elasticsearch does indeed capture the metadata fields when the error message is indexed. You can see `metadata:metron:topic` in the screenshot using Kibana.  I will see where I can add some documentation around the interplay of metadata, errors, and Solr or Elasticsearch. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (METRON-1970) Add Metadata to Error Messages Generated During Parsing
[ https://issues.apache.org/jira/browse/METRON-1970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1970: --- Description: A user can choose to add metadata to each parsed message using the `readMetadata` parser option. The error messages that are generated when a parser error occurs, do not include this metadata. This means that the original metadata is effectively lost when an error occurs. The metadata should be persisted with the error message in Kafka. This allows a user to create their own automated retry mechanism using the error message stored in Kafka. was: A user can choose to add metadata to each parsed message using the `readMetadata` parser option. The error messages that are generated when a parser error occurs, do not include this metadata. This means that the metadata is lost when an error occurs. The metadata should be persisted with the error message. > Add Metadata to Error Messages Generated During Parsing > --- > > Key: METRON-1970 > URL: https://issues.apache.org/jira/browse/METRON-1970 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > A user can choose to add metadata to each parsed message using the > `readMetadata` parser option. The error messages that are generated when a > parser error occurs, do not include this metadata. This means that the > original metadata is effectively lost when an error occurs. > The metadata should be persisted with the error message in Kafka. This > allows a user to create their own automated retry mechanism using the error > message stored in Kafka. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1988) Latency Summarizer Will Not Start
[
https://issues.apache.org/jira/browse/METRON-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760186#comment-16760186
]
Nick Allen commented on METRON-1988:
The implementation exists in Enrichment.
{code:java}
[root@node1 0.7.1]# jar -tvf lib/metron-enrichment-0.7.1-uber.jar | grep
LatencySummarizer
6198 Mon Feb 04 13:56:02 UTC 2019
org/apache/metron/enrichment/cli/LatencySummarizer$LatencyStats.class
2170 Mon Feb 04 13:56:02 UTC 2019
org/apache/metron/enrichment/cli/LatencySummarizer$LatencyStats$1.class
747 Mon Feb 04 13:56:02 UTC 2019
org/apache/metron/enrichment/cli/LatencySummarizer$Pair.class
6803 Mon Feb 04 13:56:02 UTC 2019
org/apache/metron/enrichment/cli/LatencySummarizer.class{code}
Even if I update the script to use the Enrichment jar with something like
this...
{code:java}
source /etc/default/metron
export ENRICHMENT_JAR=`find $METRON_HOME/lib -name metron-enrichment-*.jar`
java $METRON_JVMFLAGS -cp $ENRICHMENT_JAR
org.apache.metron.enrichment.cli.LatencySummarizer "$@"{code}
The tool errors out without providing any useful information
{code:java}
[root@node1 0.7.1]# bin/latency_summarizer.sh
Exception in thread "main"
org.apache.metron.jackson.databind.JsonMappingException: No content to map due
to end-of-input
at [Source: ; line: 1, column: 0]
at
org.apache.metron.jackson.databind.JsonMappingException.from(JsonMappingException.java:255)
at
org.apache.metron.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3851)
at
org.apache.metron.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3792)
at
org.apache.metron.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2816)
at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:92)
at
org.apache.metron.enrichment.cli.LatencySummarizer.main(LatencySummarizer.java:183){code}
Help? Am I missing an CLI argument? I guess not.
{code:java}
[root@node1 0.7.1]# bin/latency_summarizer.sh -h
usage: latencysummarizer [-h] [-m]
-h,--help This screen
-m,--mean_only Print the mean only when we summarize{code}
> Latency Summarizer Will Not Start
> -
>
> Key: METRON-1988
> URL: https://issues.apache.org/jira/browse/METRON-1988
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Priority: Major
>
> {code:java}
> [root@node1 0.7.1]# source /etc/default/metron
> [root@node1 0.7.1]# bin/latency_summarizer.sh
> Error: Could not find or load main class
> org.apache.metron.enrichment.cli.LatencySummarizer{code}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (METRON-1988) Latency Summarizer Will Not Start
Nick Allen created METRON-1988:
--
Summary: Latency Summarizer Will Not Start
Key: METRON-1988
URL: https://issues.apache.org/jira/browse/METRON-1988
Project: Metron
Issue Type: Bug
Reporter: Nick Allen
{code:java}
[root@node1 0.7.1]# source /etc/default/metron
[root@node1 0.7.1]# bin/latency_summarizer.sh
Error: Could not find or load main class
org.apache.metron.enrichment.cli.LatencySummarizer{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] justinleet commented on issue #1325: METRON-1970 Add Metadata to Error Messages Generated During Parsing
justinleet commented on issue #1325: METRON-1970 Add Metadata to Error Messages Generated During Parsing URL: https://github.com/apache/metron/pull/1325#issuecomment-460393409 The validation I'd do is to at least test my assumption (dynamic mapping) for ES. And then I'd make sure any differences get documented. The documentation you mention (things like how to upgrade schemas to alleviate this sort of thing) would be great. Just to reiterate for anyone hopping in, I think this definitely fits the use case of getting things to Kafka (which helps out with things like replay use cases if a user decides to reconsume data). For the real time stores, my concern is that they behave differently. A) That ES's dynamic mapping means we do map this data and B) That Solr drops it completely. This could have implications on dashboarding. I'm okay with this PR just going through to Kafka if there's not a reasonable way to align ES and Solr here. It might be worth a follow-on ticket. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] nickwallen edited a comment on issue #1325: METRON-1970 Add Metadata to Error Messages Generated During Parsing
nickwallen edited a comment on issue #1325: METRON-1970 Add Metadata to Error Messages Generated During Parsing URL: https://github.com/apache/metron/pull/1325#issuecomment-460359027 > ... although it'd be nice to add the well understood fields. The only well-understood field would be topic which ends up looking like `metron.metadata.topic`. And that is assuming that the user doesn't change the prefix which by default is `metron.metadata`. No other fields are added auto-magically. All other metadata fields are custom fields defined by the user. What due diligence do you think I should do here? * Make sure the error metadata "topic" field is indexed properly with our out-of-the-box index templates in Elasticsearch? * Or since that prefix can be changed by the user, which would break any template I provide, maybe I should just add some docs describing how a user should update their index templates when using metadata? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (METRON-1987) Upgrade Alert UI to stable Bootstrap 4
[ https://issues.apache.org/jira/browse/METRON-1987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Ardell updated METRON-1987: - Description: Like we did for the Management UI, we should upgrade the version of Bootstrap that we are using to the latest stable version. We are currently on an alpha version which includes deprecated class names and values. > Upgrade Alert UI to stable Bootstrap 4 > -- > > Key: METRON-1987 > URL: https://issues.apache.org/jira/browse/METRON-1987 > Project: Metron > Issue Type: Bug >Reporter: Shane Ardell >Priority: Major > > Like we did for the Management UI, we should upgrade the version of Bootstrap > that we are using to the latest stable version. We are currently on an alpha > version which includes deprecated class names and values. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1987) Upgrade Alert UI to stable Bootstrap 4
Shane Ardell created METRON-1987: Summary: Upgrade Alert UI to stable Bootstrap 4 Key: METRON-1987 URL: https://issues.apache.org/jira/browse/METRON-1987 Project: Metron Issue Type: Bug Reporter: Shane Ardell -- This message was sent by Atlassian JIRA (v7.6.3#76005)
