[jira] [Commented] (METRON-2221) Notebook import fails through Zeppelin REST API
[
https://issues.apache.org/jira/browse/METRON-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911538#comment-16911538
]
Anand Subramanian commented on METRON-2221:
---
Few things to note:
* Importing the notebooks manually works fine.
* Importing the notebooks using 'curl' command works fine.
* This issue is seen only when importing notebooks using the python-requests
module.
Upon speaking with the Zeppelin dev, they are not very sure why the notebook
import is failing through python
The solution in order to get this working would be to swap out the python code
in
[here|https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py#L223]
with 'curl' commands.
> Notebook import fails through Zeppelin REST API
> ---
>
> Key: METRON-2221
> URL: https://issues.apache.org/jira/browse/METRON-2221
> Project: Metron
> Issue Type: Bug
>Reporter: Anand Subramanian
>Assignee: Anand Subramanian
>Priority: Major
>
> Steps to Reproduce:
> * Go to Ambari UI -> Metron -> Service Actions -> Zeppelin Notebook Import
> * Import fails. Ambari install logs show:
> {code:java}
> Traceback (most recent call last):
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.7.1.1.9.1.0/package/scripts/indexing_master.py",
> line 244, in
> Indexing().execute()
> File
> "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py",
> line 375, in execute
> method(env)
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.7.1.1.9.1.0/package/scripts/indexing_master.py",
> line 240, in zeppelin_notebook_import
> res = ses.post(zeppelin_import_url, files=zeppelin_notebook)
> File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 349, in
> post
> return self.request('POST', url, data=data, **kwargs)
> File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 288, in
> request
> resp = self.send(prep, stream=stream, timeout=timeout, verify=verify,
> cert=cert, proxies=proxies)
> File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 383, in
> send
> r = adapter.send(request, **kwargs)
> File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 206, in
> send
> raise ConnectionError(sockerr)
> requests.exceptions.ConnectionError: [Errno 104] Connection reset by peer
> {code}
>
> Zeppelin logs show the following error:
>
> {code:java}
> WARN [2019-04-22 16:01:21,547] ({qtp1327536153-15}
> ServletHandler.java[doHandle]:620) -
> javax.servlet.ServletException: com.google.gson.JsonSyntaxException:
> java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line
> 1 column 35
> at
> org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:489)
> at
> org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
> at
> org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
> at
> org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
> at
> org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
> at
> org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
> at
> org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
> at
> org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
> at
> org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
> at
> org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
> at
> org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
> at
> org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
> at
> org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
> at
> org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at
[jira] [Updated] (METRON-2221) Notebook import fails through Zeppelin REST API
[
https://issues.apache.org/jira/browse/METRON-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anand Subramanian updated METRON-2221:
--
Description:
Steps to Reproduce:
* Go to Ambari UI -> Metron -> Service Actions -> Zeppelin Notebook Import
* Import fails. Ambari install logs show:
{code:java}
Traceback (most recent call last):
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.7.1.1.9.1.0/package/scripts/indexing_master.py",
line 244, in
Indexing().execute()
File
"/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py",
line 375, in execute
method(env)
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.7.1.1.9.1.0/package/scripts/indexing_master.py",
line 240, in zeppelin_notebook_import
res = ses.post(zeppelin_import_url, files=zeppelin_notebook)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 349, in
post
return self.request('POST', url, data=data, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 288, in
request
resp = self.send(prep, stream=stream, timeout=timeout, verify=verify,
cert=cert, proxies=proxies)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 383, in
send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 206, in
send
raise ConnectionError(sockerr)
requests.exceptions.ConnectionError: [Errno 104] Connection reset by peer
{code}
Zeppelin logs show the following error:
{code:java}
WARN [2019-04-22 16:01:21,547] ({qtp1327536153-15}
ServletHandler.java[doHandle]:620) -
javax.servlet.ServletException: com.google.gson.JsonSyntaxException:
java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1
column 35
at
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:489)
at
org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
at
org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
at
org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at
org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at
org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at
org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at
org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
at
org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
at
org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at
org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at
org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
at
org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
at
org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.apache.zeppelin.server.CorsFilter.doFilter(CorsFilter.java:72)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at
[jira] [Created] (METRON-2221) Notebook import fails through Zeppelin REST API
Anand Subramanian created METRON-2221:
-
Summary: Notebook import fails through Zeppelin REST API
Key: METRON-2221
URL: https://issues.apache.org/jira/browse/METRON-2221
Project: Metron
Issue Type: Bug
Reporter: Anand Subramanian
Assignee: Anand Subramanian
Steps to Reproduce:
* Go to Ambari UI -> Metron -> Service Actions -> Zeppelin Notebook Import
* Import fails. Ambari install logs show:
{{}}
{{}}
{code:java}
Traceback (most recent call last):
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.7.1.1.9.1.0/package/scripts/indexing_master.py",
line 244, in
Indexing().execute()
File
"/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py",
line 375, in execute
method(env)
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.7.1.1.9.1.0/package/scripts/indexing_master.py",
line 240, in zeppelin_notebook_import
res = ses.post(zeppelin_import_url, files=zeppelin_notebook)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 349, in
post
return self.request('POST', url, data=data, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 288, in
request
resp = self.send(prep, stream=stream, timeout=timeout, verify=verify,
cert=cert, proxies=proxies)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 383, in
send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 206, in
send
raise ConnectionError(sockerr)
requests.exceptions.ConnectionError: [Errno 104] Connection reset by peer
{code}
{{}}
Zeppelin logs show the following error:
{code:java}
WARN [2019-04-22 16:01:21,547] ({qtp1327536153-15}
ServletHandler.java[doHandle]:620) -
javax.servlet.ServletException: com.google.gson.JsonSyntaxException:
java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1
column 35
at
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:489)
at
org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
at
org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
at
org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at
org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at
org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at
org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at
org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
at
org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
at
org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at
org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at
org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
at
org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
at
org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.apache.zeppelin.server.CorsFilter.doFilter(CorsFilter.java:72)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
[jira] [Created] (METRON-2167) Unable to filter with alert_status:NEW
Anand Subramanian created METRON-2167:
-
Summary: Unable to filter with alert_status:NEW
Key: METRON-2167
URL: https://issues.apache.org/jira/browse/METRON-2167
Project: Metron
Issue Type: Bug
Reporter: Anand Subramanian
The following query does not return any results when run through
/api/v1/search/search:
{code:java}
{ "indices": [], "facetFields": [], "query": "(alert_status:NEW OR
metron_alert.alert_status:NEW)", "from": 0, "size": 25 }{code}
However when I change the query to look for any of 'OPEN' or 'ESCALATE' or
'RESOLVE', it returns results.
It is also observed that sorting on the 'Alert Status' field is broken. Here's
an example:
{code:java}
{ "indices": [], "facetFields": [], "query": "*", "from": 0, "size": 25,
"sort": [ { "field": "alert_status", "sortOrder": "asc" } ] }{code}
{{}}yields the following error...
{code:java}
{ "responseCode": 500, "message": "Failed to execute search;
error='IllegalArgumentException: Fielddata is disabled on text fields by
default. Set fielddata=true on [__anonymous_text] in order to load fielddata in
memory by uninverting the inverted index. Note that this can however use
significant memory. Alternatively use a keyword field instead.',
search='{\"from\":0,\"size\":25,\"query\":{\"constant_score\":{\"filter\":{\"bool\":{\"must\":[{\"bool\":{\"should\":[{\"query_string\":{\"query\":\"*\",\"fields\":[],\"use_dis_max\":true,\"tie_breaker\":0.0,\"default_operator\":\"or\",\"auto_generate_phrase_queries\":false,\"max_determinized_states\":1,\"enable_position_increments\":true,\"fuzziness\":\"AUTO\",\"fuzzy_prefix_length\":0,\"fuzzy_max_expansions\":50,\"phrase_slop\":0,\"escape\":false,\"split_on_whitespace\":true,\"boost\":1.0}},{\"nested\":{\"query\":{\"query_string\":{\"query\":\"*\",\"fields\":[],\"use_dis_max\":true,\"tie_breaker\":0.0,\"default_operator\":\"or\",\"auto_generate_phrase_queries\":false,\"max_determinized_states\":1,\"enable_position_increments\":true,\"fuzziness\":\"AUTO\",\"fuzzy_prefix_length\":0,\"fuzzy_max_expansions\":50,\"phrase_slop\":0,\"escape\":false,\"split_on_whitespace\":true,\"boost\":1.0}},\"path\":\"metron_alert\",\"ignore_unmapped\":false,\"score_mode\":\"none\",\"boost\":1.0}}],\"disable_coord\":false,\"adjust_pure_negative\":true,\"boost\":1.0}},{\"bool\":{\"should\":[{\"term\":{\"status\":{\"value\":\"active\",\"boost\":1.0}}},{\"bool\":{\"must_not\":[{\"exists\":{\"field\":\"status\",\"boost\":1.0}}],\"disable_coord\":false,\"adjust_pure_negative\":true,\"boost\":1.0}}],\"disable_coord\":false,\"adjust_pure_negative\":true,\"boost\":1.0}}],\"must_not\":[{\"exists\":{\"field\":\"metaalerts\",\"boost\":1.0}}],\"disable_coord\":false,\"adjust_pure_negative\":true,\"boost\":1.0}},\"boost\":1.0}},\"_source\":{\"includes\":[],\"excludes\":[]},\"sort\":[{\"alert_status\":{\"order\":\"asc\",\"missing\":\"_first\",\"unmapped_type\":\"text\"}}],\"track_scores\":true,\"aggregations\":{\"source:type_count\":{\"terms\":{\"field\":\"source:type\",\"size\":10,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"_count\":\"desc\"},{\"_term\":\"asc\"}]}},\"ip_src_addr_count\":{\"terms\":{\"field\":\"ip_src_addr\",\"size\":10,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"_count\":\"desc\"},{\"_term\":\"asc\"}]}},\"ip_dst_addr_count\":{\"terms\":{\"field\":\"ip_dst_addr\",\"size\":10,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"_count\":\"desc\"},{\"_term\":\"asc\"}]}},\"enrichments:geo:ip_dst_addr:country_count\":{\"terms\":{\"field\":\"enrichments:geo:ip_dst_addr:country\",\"size\":10,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"_count\":\"desc\"},{\"_term\":\"asc\"}]'",
"fullMessage": "IllegalArgumentException: Fielddata is disabled on text fields
by default. Set fielddata=true on [__anonymous_text] in order to load fielddata
in memory by uninverting the inverted index. Note that this can however use
significant memory. Alternatively use a keyword field instead." }
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-2032) Create summary table having list of stellar functions in README
[ https://issues.apache.org/jira/browse/METRON-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anand Subramanian updated METRON-2032: -- Fix Version/s: 0.7.1 > Create summary table having list of stellar functions in README > --- > > Key: METRON-2032 > URL: https://issues.apache.org/jira/browse/METRON-2032 > Project: Metron > Issue Type: Task >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > Fix For: 0.7.1 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Create a summary table with hyperlinks to stellar functions under: > https://github.com/apache/metron/blob/master/metron-platform/metron-management/README.md#functions -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2046) IS_EMPTY stellar functions fails on empty map
[
https://issues.apache.org/jira/browse/METRON-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anand Subramanian updated METRON-2046:
--
Fix Version/s: 0.7.1
> IS_EMPTY stellar functions fails on empty map
> -
>
> Key: METRON-2046
> URL: https://issues.apache.org/jira/browse/METRON-2046
> Project: Metron
> Issue Type: Bug
>Reporter: Anand Subramanian
>Assignee: Anand Subramanian
>Priority: Major
> Fix For: 0.7.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The following case is not working as expected:
> {code:java}
> [Stellar]>>> val := GEO_GET("10.1.1.1")
> {}
> [Stellar]>>> IS_EMPTY( val)
> false{code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1962) Make entering JDBC details in REST config to be optional
[ https://issues.apache.org/jira/browse/METRON-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anand Subramanian updated METRON-1962: -- Fix Version/s: 0.7.1 > Make entering JDBC details in REST config to be optional > > > Key: METRON-1962 > URL: https://issues.apache.org/jira/browse/METRON-1962 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > Fix For: 0.7.1 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > In the case where user chooses to have LDAP enabled for REST authentication, > the REST config still asks for entering the following parameters as mandatory: > * Metron JDBC URL > * Metron JDBC Driver > * Metron JDBC username > * Metron JDBC password > * Metron JDBC platform > These should be optional so user can choose not to enter values for them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1955) Update metron SPEC file to include syslog 3164 parser
[ https://issues.apache.org/jira/browse/METRON-1955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anand Subramanian updated METRON-1955: -- Fix Version/s: 0.7.1 > Update metron SPEC file to include syslog 3164 parser > - > > Key: METRON-1955 > URL: https://issues.apache.org/jira/browse/METRON-1955 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > Fix For: 0.7.1 > > Time Spent: 1h > Remaining Estimate: 0h > > Update metron.spec file to include syslog 3164 parser -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2046) IS_EMPTY stellar functions fails on empty map
[
https://issues.apache.org/jira/browse/METRON-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anand Subramanian updated METRON-2046:
--
Description:
The following case is not working as expected:
{code:java}
[Stellar]>>> val := GEO_GET("10.1.1.1")
{}
[Stellar]>>> IS_EMPTY( val)
false{code}
was:
The following case is not working as expected:
{code:java}
[Stellar]>>> val := GEO_GET("10.1.1.1")
[Stellar]>>> IS_EMPTY( val)
false{code}
> IS_EMPTY stellar functions fails on empty map
> -
>
> Key: METRON-2046
> URL: https://issues.apache.org/jira/browse/METRON-2046
> Project: Metron
> Issue Type: Bug
>Reporter: Anand Subramanian
>Assignee: Anand Subramanian
>Priority: Major
>
> The following case is not working as expected:
> {code:java}
> [Stellar]>>> val := GEO_GET("10.1.1.1")
> {}
> [Stellar]>>> IS_EMPTY( val)
> false{code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (METRON-2046) IS_EMPTY stellar functions fails on empty map
Anand Subramanian created METRON-2046:
-
Summary: IS_EMPTY stellar functions fails on empty map
Key: METRON-2046
URL: https://issues.apache.org/jira/browse/METRON-2046
Project: Metron
Issue Type: Bug
Reporter: Anand Subramanian
Assignee: Anand Subramanian
The following case is not working as expected:
{code:java}
[Stellar]>>> val := GEO_GET("10.1.1.1")
[Stellar]>>> IS_EMPTY( val)
false{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-2040) `mvn clean site` fails to run
[
https://issues.apache.org/jira/browse/METRON-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796043#comment-16796043
]
Anand Subramanian commented on METRON-2040:
---
Thanks [~msmiklavcic] , it was my bad. I was trying to run the command from
outside the 'site-book' folder which was the reason for the failure. I checked
again and it is seen to work fine. Closing this issue as 'Done'.
> `mvn clean site` fails to run
> --
>
> Key: METRON-2040
> URL: https://issues.apache.org/jira/browse/METRON-2040
> Project: Metron
> Issue Type: Bug
>Reporter: Anand Subramanian
>Priority: Major
>
> Noticing from latest master that *mvn clean site* is failing with the
> following error:
>
> {code:java}
> ➜ metron git:(master) mvn clean site
> [INFO] Scanning for projects...
> [WARNING]
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.metron:metron-profiler-storm:jar:0.7.1
>
> [INFO]
>
> [INFO] Forking metron-analytics 0.7.1
> [INFO]
>
> [INFO]
> [INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-versions) @
> metron-analytics ---
> [INFO]
> [INFO] --- jacoco-maven-plugin:0.7.9:prepare-agent (default) @
> metron-analytics ---
> [INFO] argLine set to
> -javaagent:/Users/asubramanian/.m2/repository/org/jacoco/org.jacoco.agent/0.7.9/org.jacoco.agent-0.7.9-runtime.jar=destfile=/Users/asubramanian/Desktop/Metron/anand-metron-fork/metron-latest/metron/metron-analytics/target/jacoco.exec
> [INFO]
> [INFO]
>
> [INFO] Forking metron-maas-common 0.7.1
> [INFO]
>
> Downloading from HDPReleases:
> http://repo.hortonworks.com/content/repositories/releases/org/apache/metron/stellar-common/0.7.1/stellar-common-0.7.1.jar
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Metron 0.7.1 ... FAILURE [ 4.167 s]
> [INFO] metron-stellar . SKIPPED
> [INFO] stellar-common . SKIPPED
> [INFO] metron-analytics ... SKIPPED
> [INFO] metron-maas-common . SKIPPED
> [INFO] metron-platform SKIPPED
> [INFO] metron-zookeeper ... SKIPPED
> [INFO] metron-test-utilities .. SKIPPED
> [INFO] metron-integration-test SKIPPED
> [INFO] metron-maas-service SKIPPED
> [INFO] metron-common .. SKIPPED
> [INFO] metron-statistics .. SKIPPED
> [INFO] metron-writer .. SKIPPED
> [INFO] metron-storm-kafka-override SKIPPED
> [INFO] metron-storm-kafka . SKIPPED
> [INFO] metron-hbase ... SKIPPED
> [INFO] metron-profiler-common . SKIPPED
> [INFO] metron-profiler-client . SKIPPED
> [INFO] metron-profiler-storm .. SKIPPED
> [INFO] metron-profiler-spark .. SKIPPED
> [INFO] metron-profiler-repl ... SKIPPED
> [INFO] metron-hbase-client SKIPPED
> [INFO] metron-enrichment .. SKIPPED
> [INFO] metron-indexing SKIPPED
> [INFO] metron-solr SKIPPED
> [INFO] metron-job . SKIPPED
> [INFO] metron-pcap SKIPPED
> [INFO] metron-pcap-backend SKIPPED
> [INFO] metron-data-management . SKIPPED
> [INFO] metron-parsing . SKIPPED
> [INFO] metron-parsers-common .. SKIPPED
> [INFO] metron-parsers . SKIPPED
> [INFO] metron-management .. SKIPPED
> [INFO] elasticsearch-shaded ... SKIPPED
> [INFO] metron-elasticsearch ... SKIPPED
> [INFO] metron-parsing-storm ... SKIPPED
> [INFO] metron-deployment .. SKIPPED
> [INFO] Elasticsearch Ambari Management Pack 0.7.1.0 ... SKIPPED
> [INFO] Metron Ambari Management Pack 0.7.1.0
[jira] [Created] (METRON-2040) `mvn clean site` fails to run
Anand Subramanian created METRON-2040:
-
Summary: `mvn clean site` fails to run
Key: METRON-2040
URL: https://issues.apache.org/jira/browse/METRON-2040
Project: Metron
Issue Type: Bug
Reporter: Anand Subramanian
Noticing from latest master that *mvn clean site* is failing with the following
error:
{code:java}
➜ metron git:(master) mvn clean site
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for
org.apache.metron:metron-profiler-storm:jar:0.7.1
[INFO]
[INFO] Forking metron-analytics 0.7.1
[INFO]
[INFO]
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-versions) @
metron-analytics ---
[INFO]
[INFO] --- jacoco-maven-plugin:0.7.9:prepare-agent (default) @ metron-analytics
---
[INFO] argLine set to
-javaagent:/Users/asubramanian/.m2/repository/org/jacoco/org.jacoco.agent/0.7.9/org.jacoco.agent-0.7.9-runtime.jar=destfile=/Users/asubramanian/Desktop/Metron/anand-metron-fork/metron-latest/metron/metron-analytics/target/jacoco.exec
[INFO]
[INFO]
[INFO] Forking metron-maas-common 0.7.1
[INFO]
Downloading from HDPReleases:
http://repo.hortonworks.com/content/repositories/releases/org/apache/metron/stellar-common/0.7.1/stellar-common-0.7.1.jar
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Metron 0.7.1 ... FAILURE [ 4.167 s]
[INFO] metron-stellar . SKIPPED
[INFO] stellar-common . SKIPPED
[INFO] metron-analytics ... SKIPPED
[INFO] metron-maas-common . SKIPPED
[INFO] metron-platform SKIPPED
[INFO] metron-zookeeper ... SKIPPED
[INFO] metron-test-utilities .. SKIPPED
[INFO] metron-integration-test SKIPPED
[INFO] metron-maas-service SKIPPED
[INFO] metron-common .. SKIPPED
[INFO] metron-statistics .. SKIPPED
[INFO] metron-writer .. SKIPPED
[INFO] metron-storm-kafka-override SKIPPED
[INFO] metron-storm-kafka . SKIPPED
[INFO] metron-hbase ... SKIPPED
[INFO] metron-profiler-common . SKIPPED
[INFO] metron-profiler-client . SKIPPED
[INFO] metron-profiler-storm .. SKIPPED
[INFO] metron-profiler-spark .. SKIPPED
[INFO] metron-profiler-repl ... SKIPPED
[INFO] metron-hbase-client SKIPPED
[INFO] metron-enrichment .. SKIPPED
[INFO] metron-indexing SKIPPED
[INFO] metron-solr SKIPPED
[INFO] metron-job . SKIPPED
[INFO] metron-pcap SKIPPED
[INFO] metron-pcap-backend SKIPPED
[INFO] metron-data-management . SKIPPED
[INFO] metron-parsing . SKIPPED
[INFO] metron-parsers-common .. SKIPPED
[INFO] metron-parsers . SKIPPED
[INFO] metron-management .. SKIPPED
[INFO] elasticsearch-shaded ... SKIPPED
[INFO] metron-elasticsearch ... SKIPPED
[INFO] metron-parsing-storm ... SKIPPED
[INFO] metron-deployment .. SKIPPED
[INFO] Elasticsearch Ambari Management Pack 0.7.1.0 ... SKIPPED
[INFO] Metron Ambari Management Pack 0.7.1.0 .. SKIPPED
[INFO] metron-contrib . SKIPPED
[INFO] metron-docker .. SKIPPED
[INFO] metron-performance . SKIPPED
[INFO] metron-interface ... SKIPPED
[INFO] metron-config .. SKIPPED
[INFO] metron-alerts .. SKIPPED
[INFO] metron-rest-client . SKIPPED
[INFO] metron-rest SKIPPED
[INFO] site-book
[jira] [Created] (METRON-2032) Create summary table having list of stellar functions in README
Anand Subramanian created METRON-2032: - Summary: Create summary table having list of stellar functions in README Key: METRON-2032 URL: https://issues.apache.org/jira/browse/METRON-2032 Project: Metron Issue Type: Task Reporter: Anand Subramanian Assignee: Anand Subramanian Create a summary table with hyperlinks to stellar functions under: https://github.com/apache/metron/blob/master/metron-platform/metron-management/README.md#functions -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1962) Make entering JDBC details in REST config to be optional
[ https://issues.apache.org/jira/browse/METRON-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anand Subramanian updated METRON-1962: -- Description: In the case where user chooses to have LDAP enabled for REST authentication, the REST config still asks for entering the following parameters as mandatory: * Metron JDBC URL * Metron JDBC Driver * Metron JDBC username * Metron JDBC password * Metron JDBC platform These should be optional so user can choose not to enter values for them. was: In the case where user chooses to have LDAP enabled for REST authentication, the REST config still asks for entering the following parameters as mandatory: * Metron JDBC URL * Metron JDBC Driver * Metron JDBC username * Metron JDBC password These should be optional so user can choose not to enter values for them. > Make entering JDBC details in REST config to be optional > > > Key: METRON-1962 > URL: https://issues.apache.org/jira/browse/METRON-1962 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > In the case where user chooses to have LDAP enabled for REST authentication, > the REST config still asks for entering the following parameters as mandatory: > * Metron JDBC URL > * Metron JDBC Driver > * Metron JDBC username > * Metron JDBC password > * Metron JDBC platform > These should be optional so user can choose not to enter values for them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1962) Make entering JDBC details in REST config to be optional
[ https://issues.apache.org/jira/browse/METRON-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anand Subramanian updated METRON-1962: -- Description: In the case where user chooses to have LDAP enabled for REST authentication, the REST config still asks for entering the following parameters as mandatory: * Metron JDBC URL * Metron JDBC Driver * Metron JDBC username * Metron JDBC password These should be optional so user can choose not to enter values for them. was:When > Make entering JDBC details in REST config to be optional > > > Key: METRON-1962 > URL: https://issues.apache.org/jira/browse/METRON-1962 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > In the case where user chooses to have LDAP enabled for REST authentication, > the REST config still asks for entering the following parameters as mandatory: > * Metron JDBC URL > * Metron JDBC Driver > * Metron JDBC username > * Metron JDBC password > These should be optional so user can choose not to enter values for them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1955) Update metron SPEC file to include syslog 3164 parser
Anand Subramanian created METRON-1955: - Summary: Update metron SPEC file to include syslog 3164 parser Key: METRON-1955 URL: https://issues.apache.org/jira/browse/METRON-1955 Project: Metron Issue Type: Bug Reporter: Anand Subramanian Assignee: Anand Subramanian Update metron.spec file to include syslog 3164 parser -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1861) REST fails to start when LDAP enabled and 'Active Spring profiles' config is empty
[
https://issues.apache.org/jira/browse/METRON-1861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anand Subramanian updated METRON-1861:
--
Description:
When 'LDAP enabled' (in Configs -> Metron -> Security tab) is set to 'On' and
the 'Active Spring profiles' (in Configs -> Metron -> REST tab) is blank, REST
service fails to start with the following :
{code:java}
18/11/08 08:10:02 boot.SpringApplication: Application run
java.lang.IllegalArgumentException: Invalid profile []: must contain text
at
org.springframework.core.env.AbstractEnvironment.validateProfile(AbstractEnvironment.java:365)
at
org.springframework.core.env.AbstractEnvironment.setActiveProfiles(AbstractEnvironment.java:256)
at
org.springframework.core.env.AbstractEnvironment.doGetActiveProfiles(AbstractEnvironment.java:242)
at
org.springframework.core.env.AbstractEnvironment.getActiveProfiles(AbstractEnvironment.java:226)
at
org.springframework.boot.SpringApplication.configureProfiles(SpringApplication.java:522)
at
org.springframework.boot.SpringApplication.configureEnvironment(SpringApplication.java:479)
at
org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:357)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:317)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1255)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1243)
at
org.apache.metron.rest.MetronRestApplication.main(MetronRestApplication.java:36){code}
This is because the 'metron_spring_profiles_active' is set to the value ',ldap'
which is not valid.
was:
When 'LDAP enabled' (in Configs -> Metron -> Security tab) is set to 'On' and
the 'Active Spring profiles' (in Configs -> Metron -> REST tab) is blank, REST
service fails to start with the following error:
{code:java}
18/11/08 08:10:02 ERROR boot.SpringApplication: Application run failed
java.lang.IllegalArgumentException: Invalid profile []: must contain text
at
org.springframework.core.env.AbstractEnvironment.validateProfile(AbstractEnvironment.java:365)
at
org.springframework.core.env.AbstractEnvironment.setActiveProfiles(AbstractEnvironment.java:256)
at
org.springframework.core.env.AbstractEnvironment.doGetActiveProfiles(AbstractEnvironment.java:242)
at
org.springframework.core.env.AbstractEnvironment.getActiveProfiles(AbstractEnvironment.java:226)
at
org.springframework.boot.SpringApplication.configureProfiles(SpringApplication.java:522)
at
org.springframework.boot.SpringApplication.configureEnvironment(SpringApplication.java:479)
at
org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:357)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:317)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1255)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1243)
at
org.apache.metron.rest.MetronRestApplication.main(MetronRestApplication.java:36){code}
This is because the 'metron_spring_profiles_active' is set to the value ',ldap'
which is not valid.
> REST fails to start when LDAP enabled and 'Active Spring profiles' config is
> empty
> --
>
> Key: METRON-1861
> URL: https://issues.apache.org/jira/browse/METRON-1861
> Project: Metron
> Issue Type: Bug
>Reporter: Anand Subramanian
>Assignee: Anand Subramanian
>Priority: Major
>
> When 'LDAP enabled' (in Configs -> Metron -> Security tab) is set to 'On' and
> the 'Active Spring profiles' (in Configs -> Metron -> REST tab) is blank,
> REST service fails to start with the following :
>
> {code:java}
> 18/11/08 08:10:02 boot.SpringApplication: Application run
> java.lang.IllegalArgumentException: Invalid profile []: must contain text
> at
> org.springframework.core.env.AbstractEnvironment.validateProfile(AbstractEnvironment.java:365)
> at
> org.springframework.core.env.AbstractEnvironment.setActiveProfiles(AbstractEnvironment.java:256)
> at
> org.springframework.core.env.AbstractEnvironment.doGetActiveProfiles(AbstractEnvironment.java:242)
> at
> org.springframework.core.env.AbstractEnvironment.getActiveProfiles(AbstractEnvironment.java:226)
> at
> org.springframework.boot.SpringApplication.configureProfiles(SpringApplication.java:522)
> at
> org.springframework.boot.SpringApplication.configureEnvironment(SpringApplication.java:479)
> at
> org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:357)
> at org.springframework.boot.SpringApplication.run(SpringApplication.java:317)
> at
> org.springframework.boot.SpringApplication.run(SpringApplication.java:1255)
> at
>
[jira] [Updated] (METRON-1861) REST fails to start when LDAP enabled and 'Active Spring profiles' config is empty
[
https://issues.apache.org/jira/browse/METRON-1861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anand Subramanian updated METRON-1861:
--
Description:
When 'LDAP enabled' (in Configs -> Metron -> Security tab) is set to 'On' and
the 'Active Spring profiles' (in Configs -> Metron -> REST tab) is blank, REST
service fails to start with the following error:
{code:java}
18/11/08 08:10:02 ERROR boot.SpringApplication: Application run failed
java.lang.IllegalArgumentException: Invalid profile []: must contain text
at
org.springframework.core.env.AbstractEnvironment.validateProfile(AbstractEnvironment.java:365)
at
org.springframework.core.env.AbstractEnvironment.setActiveProfiles(AbstractEnvironment.java:256)
at
org.springframework.core.env.AbstractEnvironment.doGetActiveProfiles(AbstractEnvironment.java:242)
at
org.springframework.core.env.AbstractEnvironment.getActiveProfiles(AbstractEnvironment.java:226)
at
org.springframework.boot.SpringApplication.configureProfiles(SpringApplication.java:522)
at
org.springframework.boot.SpringApplication.configureEnvironment(SpringApplication.java:479)
at
org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:357)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:317)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1255)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1243)
at
org.apache.metron.rest.MetronRestApplication.main(MetronRestApplication.java:36){code}
This is because the 'metron_spring_profiles_active' is set to the value ',ldap'
which is not valid.
> REST fails to start when LDAP enabled and 'Active Spring profiles' config is
> empty
> --
>
> Key: METRON-1861
> URL: https://issues.apache.org/jira/browse/METRON-1861
> Project: Metron
> Issue Type: Bug
>Reporter: Anand Subramanian
>Assignee: Anand Subramanian
>Priority: Major
>
> When 'LDAP enabled' (in Configs -> Metron -> Security tab) is set to 'On' and
> the 'Active Spring profiles' (in Configs -> Metron -> REST tab) is blank,
> REST service fails to start with the following error:
>
> {code:java}
> 18/11/08 08:10:02 ERROR boot.SpringApplication: Application run failed
> java.lang.IllegalArgumentException: Invalid profile []: must contain text
> at
> org.springframework.core.env.AbstractEnvironment.validateProfile(AbstractEnvironment.java:365)
> at
> org.springframework.core.env.AbstractEnvironment.setActiveProfiles(AbstractEnvironment.java:256)
> at
> org.springframework.core.env.AbstractEnvironment.doGetActiveProfiles(AbstractEnvironment.java:242)
> at
> org.springframework.core.env.AbstractEnvironment.getActiveProfiles(AbstractEnvironment.java:226)
> at
> org.springframework.boot.SpringApplication.configureProfiles(SpringApplication.java:522)
> at
> org.springframework.boot.SpringApplication.configureEnvironment(SpringApplication.java:479)
> at
> org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:357)
> at org.springframework.boot.SpringApplication.run(SpringApplication.java:317)
> at
> org.springframework.boot.SpringApplication.run(SpringApplication.java:1255)
> at
> org.springframework.boot.SpringApplication.run(SpringApplication.java:1243)
> at
> org.apache.metron.rest.MetronRestApplication.main(MetronRestApplication.java:36){code}
>
> This is because the 'metron_spring_profiles_active' is set to the value
> ',ldap' which is not valid.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (METRON-1861) REST fails to start when LDAP enabled and 'Active Spring profiles' config is empty
Anand Subramanian created METRON-1861: - Summary: REST fails to start when LDAP enabled and 'Active Spring profiles' config is empty Key: METRON-1861 URL: https://issues.apache.org/jira/browse/METRON-1861 Project: Metron Issue Type: Bug Reporter: Anand Subramanian Assignee: Anand Subramanian -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1798) Add mpack support for parser aggregation
[ https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anand Subramanian updated METRON-1798: -- Fix Version/s: 0.6.1 > Add mpack support for parser aggregation > > > Key: METRON-1798 > URL: https://issues.apache.org/jira/browse/METRON-1798 > Project: Metron > Issue Type: Task >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > Fix For: 0.6.1 > > > Support spawning of storm topologies if a user specifies an aggregated parser > configuration at: > Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers" > > For example, specifying the following: > "bro,snort,yaf", "snort,yaf", yaf > should spawn an aggregated topology for first two, and a regular topology for > the 'yaf'. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1802) When switching parser topologies, some of the original parser topologies can fail to be shut down properly
[ https://issues.apache.org/jira/browse/METRON-1802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anand Subramanian updated METRON-1802: -- Description: This defect was reported by [~nickwallen] when testing PR[ #1215|[https://github.com/apache/metron/pull/1215].] Steps to reproduce: # On full-dev, start with default parsers; [bro,snort] # Change Metron Parsers setting to say ["bro"] under Services -> Metron -> Configs -> 'Parsers' tab -> 'Metron Parsers' field. Save configuration and restart required services to apply the config change. # Observe that the original "snort" topology is never shutdown. One would expect this to be shutdown. was: This defect was reported by [~nickwallen] when testing PR[ #1215|[https://github.com/apache/metron/pull/1215].] Steps to reproduce: # On full-dev, start with default parsers; [bro,snort] # Change Metron Parsers setting to say ["bro"] and save configuration # Restart the Parser topology to apply the config change. # Observe that the original "snort" topology is never shutdown. One would expect this to be shutdown. > When switching parser topologies, some of the original parser topologies can > fail to be shut down properly > -- > > Key: METRON-1802 > URL: https://issues.apache.org/jira/browse/METRON-1802 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Priority: Major > > This defect was reported by [~nickwallen] when testing PR[ > #1215|[https://github.com/apache/metron/pull/1215].] > Steps to reproduce: > # On full-dev, start with default parsers; [bro,snort] > # Change Metron Parsers setting to say ["bro"] under Services -> Metron -> > Configs -> 'Parsers' tab -> 'Metron Parsers' field. Save configuration and > restart required services to apply the config change. > # Observe that the original "snort" topology is never shutdown. One would > expect this to be shutdown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1802) When switching parser topologies, some of the original parser topologies can fail to be shut down properly
Anand Subramanian created METRON-1802: - Summary: When switching parser topologies, some of the original parser topologies can fail to be shut down properly Key: METRON-1802 URL: https://issues.apache.org/jira/browse/METRON-1802 Project: Metron Issue Type: Bug Reporter: Anand Subramanian This defect was reported by [~nickwallen] when testing PR[ #1215|[https://github.com/apache/metron/pull/1215].] Steps to reproduce: # On full-dev, start with default parsers; [bro,snort] # Change Metron Parsers setting to say ["bro"] and save configuration # Restart the Parser topology to apply the config change. # Observe that the original "snort" topology is never shutdown. One would expect this to be shutdown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1798) Add mpack support for parser aggregation
Anand Subramanian created METRON-1798: - Summary: Add mpack support for parser aggregation Key: METRON-1798 URL: https://issues.apache.org/jira/browse/METRON-1798 Project: Metron Issue Type: Task Reporter: Anand Subramanian Assignee: Anand Subramanian Support spawning of storm topologies if a user specifies an aggregated parser configuration at: Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers" For example, specifying the following: "bro,snort,yaf", "snort,yaf", yaf should spawn an aggregated topology for first two, and a regular topology for the 'yaf'. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1719) PCAP sensor should have its own service startup scripts
[ https://issues.apache.org/jira/browse/METRON-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566359#comment-16566359 ] Anand Subramanian commented on METRON-1719: --- Dupe of METRON-1709 > PCAP sensor should have its own service startup scripts > --- > > Key: METRON-1719 > URL: https://issues.apache.org/jira/browse/METRON-1719 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Assignee: Mohan >Priority: Major > > Presently, the PCAP sensor needs to be started from the command like from > $METRON_HOME/bin/start_pcap_topology.sh, unlike the other parsers which use a > different startup script. > PCAP sensor needs to have its own startup and service monitoring scripts, > similar to the other parsers. It needs to be able to be started/stopped from > Ambari and the Management UI like the other parsers. > IMHO, the stock startup status of the PCAP sensor on full-dev environment > should be set to 'Stopped' state, since keeping it turned ON would only > burden (the already burdened) full-dev deployment further. This point is > however open for discussion. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1719) PCAP sensor should have its own service startup scripts
Anand Subramanian created METRON-1719: - Summary: PCAP sensor should have its own service startup scripts Key: METRON-1719 URL: https://issues.apache.org/jira/browse/METRON-1719 Project: Metron Issue Type: Bug Reporter: Anand Subramanian Presently, the PCAP sensor needs to be started from the command like from $METRON_HOME/bin/start_pcap_topology.sh, unlike the other parsers which use a different startup script. PCAP sensor needs to have its own startup and service monitoring scripts, similar to the other parsers. It needs to be able to be started/stopped from Ambari and the Management UI like the other parsers. IMHO, the stock startup status of the PCAP sensor on full-dev environment should be set to 'Stopped' state, since keeping it turned ON would only burden (the already burdened) full-dev deployment further. This point is however open for discussion. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1695) Expose pcap properties through Ambari
[ https://issues.apache.org/jira/browse/METRON-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anand Subramanian updated METRON-1695: -- Description: Currently, the $METRON_HOME/config/pcap.properties file is hardcoded with the defaults. One has to hand edit the file before deploying the PCAP topology. These properties should be configurable via Ambari. (was: Presently, the $METRON_HOME/config/pcap.properties file is hardcoded with the defaults. One has to hand edit the file before deploying the PCAP topology. These properties should be configurable via Ambari.) > Expose pcap properties through Ambari > - > > Key: METRON-1695 > URL: https://issues.apache.org/jira/browse/METRON-1695 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Priority: Major > > Currently, the $METRON_HOME/config/pcap.properties file is hardcoded with the > defaults. One has to hand edit the file before deploying the PCAP topology. > These properties should be configurable via Ambari. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1695) Expose pcap properties through Ambari
[ https://issues.apache.org/jira/browse/METRON-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anand Subramanian reassigned METRON-1695: - Assignee: Anand Subramanian > Expose pcap properties through Ambari > - > > Key: METRON-1695 > URL: https://issues.apache.org/jira/browse/METRON-1695 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Currently, the $METRON_HOME/config/pcap.properties file is hardcoded with the > defaults. One has to hand edit the file before deploying the PCAP topology. > These properties should be configurable via Ambari. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1695) Expose pcap properties through Ambari
Anand Subramanian created METRON-1695: - Summary: Expose pcap properties through Ambari Key: METRON-1695 URL: https://issues.apache.org/jira/browse/METRON-1695 Project: Metron Issue Type: Bug Reporter: Anand Subramanian Presently, the $METRON_HOME/config/pcap.properties file is hardcoded with the defaults. One has to hand edit the file before deploying the PCAP topology. These properties should be configurable via Ambari. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1510) Update Metron website to include info about github update subscription
Anand Subramanian created METRON-1510: - Summary: Update Metron website to include info about github update subscription Key: METRON-1510 URL: https://issues.apache.org/jira/browse/METRON-1510 Project: Metron Issue Type: Bug Reporter: Anand Subramanian Per the discussion [here|https://lists.apache.org/thread.html/9a450a46701eb0924d431a042737b904ab344eb052f630b112a7e5a6@%3Cdev.metron.apache.org%3E], github updates have been moved to a separate issues list at https://lists.apache.org/list.html?issues@metron.apache.org The Metron website needs an update to reflect the same. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1484) Indexing Topologies Fail Partial Start in Ambari
[ https://issues.apache.org/jira/browse/METRON-1484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395310#comment-16395310 ] Anand Subramanian commented on METRON-1484: --- Are these errors in METRON-1451 seen in the ambar-agent.log? > Indexing Topologies Fail Partial Start in Ambari > > > Key: METRON-1484 > URL: https://issues.apache.org/jira/browse/METRON-1484 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Priority: Major > > If either topology is down, Ambari shows all of Indexing as dead. Clicking > start attempts to start them both and fails if either is still running. > Furthermore, it appears to retry 3 times before finally failing the command. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1386) Fix Metron Website Required Links
[ https://issues.apache.org/jira/browse/METRON-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anand Subramanian reassigned METRON-1386: - Assignee: Anand Subramanian > Fix Metron Website Required Links > - > > Key: METRON-1386 > URL: https://issues.apache.org/jira/browse/METRON-1386 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Assignee: Anand Subramanian >Priority: Major > > We seem to be missing a fair amount of the required links on our homepage. We > need to add them. > The requirements are listed at > https://www.apache.org/foundation/marks/pmcs#navigation > Reporting is available at: https://whimsy.apache.org/site/project/metron -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1457) Move ASF links to main page in the Metron website
Anand Subramanian created METRON-1457: - Summary: Move ASF links to main page in the Metron website Key: METRON-1457 URL: https://issues.apache.org/jira/browse/METRON-1457 Project: Metron Issue Type: Bug Reporter: Anand Subramanian Assignee: Anand Subramanian The latest changes that went out as a part of METRON-1386 ([https://github.com/apache/metron/pull/935]) still did not fix the requisites per: [https://whimsy.apache.org/site/project/metron] Apparently, whimsy looks for the relevant links in the main index.html page and not sub-links. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1386) Fix Metron Website Required Links
[ https://issues.apache.org/jira/browse/METRON-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360497#comment-16360497 ] Anand Subramanian commented on METRON-1386: --- I have created METRON-1456 to track the graphics changes. > Fix Metron Website Required Links > - > > Key: METRON-1386 > URL: https://issues.apache.org/jira/browse/METRON-1386 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Priority: Major > > We seem to be missing a fair amount of the required links on our homepage. We > need to add them. > The requirements are listed at > https://www.apache.org/foundation/marks/pmcs#navigation > Reporting is available at: https://whimsy.apache.org/site/project/metron -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1456) Metron logo needs to follow Apache standard policy
Anand Subramanian created METRON-1456:
-
Summary: Metron logo needs to follow Apache standard policy
Key: METRON-1456
URL: https://issues.apache.org/jira/browse/METRON-1456
Project: Metron
Issue Type: Bug
Reporter: Anand Subramanian
Metron logo needs to conform to Apache standards
[https://www.apache.org/foundation/marks/pmcs#graphics]
{quote}Logos are important to recognize as trademarks as well. For the
project's official logo (if it has one, and especially if it uses the ASF
feather), ensure that it includes a small "TM" symbol in the graphic or
immediately adjacent to it. For pages that include the project logo on them,
ensure you mention "... and the Project logo are trademarks..." in the
attribution.
{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (METRON-1451) On Centos full dev, Metron Indexing shows up as stopped
Anand Subramanian created METRON-1451:
-
Summary: On Centos full dev, Metron Indexing shows up as stopped
Key: METRON-1451
URL: https://issues.apache.org/jira/browse/METRON-1451
Project: Metron
Issue Type: Bug
Affects Versions: 0.4.3
Reporter: Anand Subramanian
Assignee: Anand Subramanian
Fix For: Next + 1
In Ambari, the Metron Indexing status is shown as 'stopped'. The agent log
shows the following error:
{code:java}
INFO 2018-02-05 22:21:39,990 PythonReflectiveExecutor.py:67 - Reflective
command failed with exception:
Traceback (most recent call last):
File
"/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py",
line 59, in run_file
imp.load_source('__main__', script)
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.4.3/package/scripts/indexing_master.py",
line 18, in
import requests
File "/usr/lib/python2.6/site-packages/requests/__init__.py", line 53, in
from .packages.urllib3.contrib import pyopenssl
File "/usr/lib/python2.6/site-packages/requests/packages/__init__.py", line
61, in load_module
if name in sys.modules:
AttributeError: 'NoneType' object has no attribute 'modules'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-993) Quick dev broken with latest master - services unable to start
[
https://issues.apache.org/jira/browse/METRON-993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344569#comment-16344569
]
Anand Subramanian commented on METRON-993:
--
Quick dev deployment is now obsolete.
> Quick dev broken with latest master - services unable to start
> --
>
> Key: METRON-993
> URL: https://issues.apache.org/jira/browse/METRON-993
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Anand Subramanian
>Priority: Major
>
> I am trying to bring up a quick dev environment using the latest bits from
> master and I noticed that metron services fails to come up. The below error
> is seen for Metron Enrichment in Ambari:
> {code}
> stderr:
> Traceback (most recent call last):
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.3.1/package/scripts/enrichment_master.py",
> line 84, in
> Enrichment().execute()
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 280, in execute
> method(env)
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.3.1/package/scripts/enrichment_master.py",
> line 51, in start
> metron_service.load_global_config(params)
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.3.1/package/scripts/metron_service.py",
> line 76, in load_global_config
> init_config()
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.3.1/package/scripts/metron_service.py",
> line 31, in init_config
> path=ambari_format("{java_home}/bin")
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py",
> line 155, in __init__
> self.env.run()
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 160, in run
> self.run_action(resource, action)
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 124, in run_action
> provider_action()
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
> line 273, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 293, in _call
> raise ExecutionFailed(err_msg, code, out, err)
> resource_management.core.exceptions.ExecutionFailed: Execution of
> '/usr/metron/0.3.1/bin/zk_load_configs.sh --mode PUSH -i
> /usr/metron/0.3.1/config/zookeeper -z node1:2181' returned 127. /bin/bash:
> /usr/metron/0.3.1/bin/zk_load_configs.sh: No such file or directory
> stdout:
> 2017-06-09 09:32:22,007 - The hadoop conf dir
> /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for
> version 2.5.3.0-37
> 2017-06-09 09:32:22,007 - Checking if need to create versioned conf dir
> /etc/hadoop/2.5.3.0-37/0
> 2017-06-09 09:32:22,009 - call[('ambari-python-wrap', '/usr/bin/conf-select',
> 'create-conf-dir', '--package', 'hadoop', '--stack-version', '2.5.3.0-37',
> '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False,
> 'stderr': -1}
> 2017-06-09 09:32:22,097 - call returned (1, '/etc/hadoop/2.5.3.0-37/0 exist
> already', '')
> 2017-06-09 09:32:22,098 - checked_call[('ambari-python-wrap',
> '/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop',
> '--stack-version', '2.5.3.0-37', '--conf-version', '0')] {'logoutput': False,
> 'sudo': True, 'quiet': False}
> 2017-06-09 09:32:22,304 - checked_call returned (0, '')
> 2017-06-09 09:32:22,306 - Ensuring that hadoop has the correct symlink
> structure
> 2017-06-09 09:32:22,306 - Using hadoop conf dir:
> /usr/hdp/current/hadoop-client/conf
> 2017-06-09 09:32:23,537 - The hadoop conf dir
> /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for
> version 2.5.3.0-37
> 2017-06-09 09:32:23,561 - Checking if need to create versioned conf dir
> /etc/hadoop/2.5.3.0-37/0
> 2017-06-09 09:32:23,562 - call[('ambari-python-wrap', '/usr/bin/conf-select',
> 'create-conf-dir', '--package', 'hadoop', '--stack-version', '2.5.3.0-37',
> '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False,
> 'stderr': -1}
> 2017-06-09 09:32:23,915 - call returned (1, '/etc/hadoop/2.5.3.0-37/0 exist
> already', '')
> 2017-06-09 09:32:23,918 - checked_call[('ambari-python-wrap',
> '/usr/bin/conf-select',
[jira] [Commented] (METRON-1399) ES 5.x requires additional configuration for OS' which use systemd
[
https://issues.apache.org/jira/browse/METRON-1399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344567#comment-16344567
]
Anand Subramanian commented on METRON-1399:
---
[~nickwallen] has addressed this issue as a part of METRON-1370
(https://github.com/apache/metron/pull/903).
> ES 5.x requires additional configuration for OS' which use systemd
> --
>
> Key: METRON-1399
> URL: https://issues.apache.org/jira/browse/METRON-1399
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.3
>Reporter: Anand Subramanian
>Assignee: Nick Allen
>Priority: Major
>
> This is with latest master, which has elasticsearch 5.6.2.
> On a Centos 7, it is seen that the Elasticsearch service fails to start with
> error as follows:
> {code}
> [2017-12-07T10:06:36,451][ERROR][o.e.b.Bootstrap ]
> [metron-1.openstacklocal] node validation exception
> [1] bootstrap checks failed
> [1]: memory locking requested for elasticsearch process but memory is not
> locked
> [2017-12-07T10:06:36,459][INFO ][o.e.n.Node ]
> [metron-1.openstacklocal] stopping ...
> [2017-12-07T10:06:36,496][INFO ][o.e.n.Node ]
> [metron-1.openstacklocal] stopped
> [2017-12-07T10:06:36,497][INFO ][o.e.n.Node ]
> [metron-1.openstacklocal] closing ...
> [2017-12-07T10:06:36,522][INFO ][o.e.n.Node ]
> [metron-1.openstacklocal] closed
> {code}
> There was no change in behavior after adding the entry
> "MAX_LOCKED_MEMORY=unlimited" in /etc/sysconfig/elasticsearch.
> I found that ES 5.x requires additional configuration for environments that
> use systemd (like CentOS 7). The following link explains the changes required:
> https://www.elastic.co/guide/en/elasticsearch/reference/master/setting-system-settings.html#systemd
> This hence needs to be done at install/configuration time for ES to come up
> properly on OS' that use 'systemd'.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (METRON-1399) ES 5.x requires additional configuration for OS' which use systemd
[
https://issues.apache.org/jira/browse/METRON-1399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anand Subramanian reassigned METRON-1399:
-
Assignee: Nick Allen
> ES 5.x requires additional configuration for OS' which use systemd
> --
>
> Key: METRON-1399
> URL: https://issues.apache.org/jira/browse/METRON-1399
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.3
>Reporter: Anand Subramanian
>Assignee: Nick Allen
>Priority: Major
>
> This is with latest master, which has elasticsearch 5.6.2.
> On a Centos 7, it is seen that the Elasticsearch service fails to start with
> error as follows:
> {code}
> [2017-12-07T10:06:36,451][ERROR][o.e.b.Bootstrap ]
> [metron-1.openstacklocal] node validation exception
> [1] bootstrap checks failed
> [1]: memory locking requested for elasticsearch process but memory is not
> locked
> [2017-12-07T10:06:36,459][INFO ][o.e.n.Node ]
> [metron-1.openstacklocal] stopping ...
> [2017-12-07T10:06:36,496][INFO ][o.e.n.Node ]
> [metron-1.openstacklocal] stopped
> [2017-12-07T10:06:36,497][INFO ][o.e.n.Node ]
> [metron-1.openstacklocal] closing ...
> [2017-12-07T10:06:36,522][INFO ][o.e.n.Node ]
> [metron-1.openstacklocal] closed
> {code}
> There was no change in behavior after adding the entry
> "MAX_LOCKED_MEMORY=unlimited" in /etc/sysconfig/elasticsearch.
> I found that ES 5.x requires additional configuration for environments that
> use systemd (like CentOS 7). The following link explains the changes required:
> https://www.elastic.co/guide/en/elasticsearch/reference/master/setting-system-settings.html#systemd
> This hence needs to be done at install/configuration time for ES to come up
> properly on OS' that use 'systemd'.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1369) squid events are not seen in Alerts UI
[
https://issues.apache.org/jira/browse/METRON-1369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344563#comment-16344563
]
Anand Subramanian commented on METRON-1369:
---
In order for the squid events to show up in the Alerts UI, there needs to be a
template added into Elasticsearch so that it gets applied dynamically as the
data flows in.
The following needs to be done before ingesting into squid sensor:
{code:java}
curl -XPUT 'http://node1:9200/_template/squid_index' -d '
{
"template": "squid_index*",
"mappings": {
"squid_doc": {
"dynamic_templates": [
{
"geo_location_point": {
"match": "enrichments:geo:*:location_point",
"match_mapping_type": "*",
"mapping": {
"type": "geo_point"
}
}
},
{
"geo_country": {
"match": "enrichments:geo:*:country",
"match_mapping_type": "*",
"mapping": {
"type": "keyword"
}
}
},
{
"geo_city": {
"match": "enrichments:geo:*:city",
"match_mapping_type": "*",
"mapping": {
"type": "keyword"
}
}
},
{
"geo_location_id": {
"match": "enrichments:geo:*:locID",
"match_mapping_type": "*",
"mapping": {
"type": "keyword"
}
}
},
{
"geo_dma_code": {
"match": "enrichments:geo:*:dmaCode",
"match_mapping_type": "*",
"mapping": {
"type": "keyword"
}
}
},
{
"geo_postal_code": {
"match": "enrichments:geo:*:postalCode",
"match_mapping_type": "*",
"mapping": {
"type": "keyword"
}
}
},
{
"geo_latitude": {
"match": "enrichments:geo:*:latitude",
"match_mapping_type": "*",
"mapping": {
"type": "float"
}
}
},
{
"geo_longitude": {
"match": "enrichments:geo:*:longitude",
"match_mapping_type": "*",
"mapping": {
"type": "float"
}
}
},
{
"timestamps": {
"match": "*:ts",
"match_mapping_type": "*",
"mapping": {
"type": "date",
"format": "epoch_millis"
}
}
},
{
"threat_triage_score": {
"mapping": {
"type": "float"
},
"match": "threat:triage:*score",
"match_mapping_type": "*"
}
},
{
"threat_triage_reason": {
"mapping": {
"type": "text",
"fielddata": "true"
},
"match": "threat:triage:rules:*:reason",
"match_mapping_type": "*"
}
},
{
"threat_triage_name": {
"mapping": {
"type": "text",
"fielddata": "true"
},
"match": "threat:triage:rules:*:name",
"match_mapping_type": "*"
}
}
],
"properties": {
"timestamp": {
"type": "date",
"format": "epoch_millis"
},
"source:type": {
"type": "keyword"
},
"ip_dst_addr": {
"type": "ip"
},
"ip_dst_port": {
"type": "integer"
},
"ip_src_addr": {
"type": "ip"
},
"ip_src_port": {
"type": "integer"
},
"alert": {
"type": "nested"
},
"guid": {
"type": "keyword"
}
}
}
}
}
'{code}
Validate that the template loaded as expected by running:
{code:java}
curl -XGET 'http://node1:9200/_template/squid_index?pretty'{code}
[~mmiklavcic] has updated the Wiki instructions here to reflect the same. Thank
you Mike!
[https://cwiki.apache.org/confluence/display/METRON/2016/04/25/Metron+Tutorial+-+Fundamentals+Part+1%3A+Creating+a+New+Telemetry]
> squid events are not seen in Alerts UI
> --
>
> Key: METRON-1369
> URL: https://issues.apache.org/jira/browse/METRON-1369
> Project: Metron
> Issue Type: Bug
>Reporter: Anand Subramanian
>Assignee: Nick Allen
>Priority: Major
>
> On a metron cluster, I have added the squid telemetry by following the [Wiki
> instructions|https://cwiki.apache.org/confluence/display/METRON/2016/05/02/Metron+Tutorial+-+Fundamentals+Part+4%3A+Pluggable+Threat+Intelligence].
> I am able to see the squid indices present in elasticsearch, but not able to
> see them in the alerts UI. On further checking, I found out that this is due
> to the missing nested "alert" field in the index mappings--which is required
> for the Alerts UI to pick up the
[jira] [Created] (METRON-1399) ES 5.x requires additional configuration for OS' which use systemd
Anand Subramanian created METRON-1399:
-
Summary: ES 5.x requires additional configuration for OS' which
use systemd
Key: METRON-1399
URL: https://issues.apache.org/jira/browse/METRON-1399
Project: Metron
Issue Type: Bug
Affects Versions: 0.4.3
Reporter: Anand Subramanian
This is with latest master, which has elasticsearch 5.6.2.
On a Centos 7, it is seen that the Elasticsearch service fails to start with
error as follows:
{code}
[2017-12-07T10:06:36,451][ERROR][o.e.b.Bootstrap ]
[metron-1.openstacklocal] node validation exception
[1] bootstrap checks failed
[1]: memory locking requested for elasticsearch process but memory is not locked
[2017-12-07T10:06:36,459][INFO ][o.e.n.Node ]
[metron-1.openstacklocal] stopping ...
[2017-12-07T10:06:36,496][INFO ][o.e.n.Node ]
[metron-1.openstacklocal] stopped
[2017-12-07T10:06:36,497][INFO ][o.e.n.Node ]
[metron-1.openstacklocal] closing ...
[2017-12-07T10:06:36,522][INFO ][o.e.n.Node ]
[metron-1.openstacklocal] closed
{code}
There was no change in behavior after adding the entry
"MAX_LOCKED_MEMORY=unlimited" in /etc/sysconfig/elasticsearch.
I found that ES 5.x requires additional configuration for environments that use
systemd (like CentOS 7). The following link explains the changes required:
https://www.elastic.co/guide/en/elasticsearch/reference/master/setting-system-settings.html#systemd
This hence needs to be done at install/configuration time for ES to come up
properly on OS' that use 'systemd'.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (METRON-1389) Zeppelin notebook import does not work with Ambari 2.6
Anand Subramanian created METRON-1389: - Summary: Zeppelin notebook import does not work with Ambari 2.6 Key: METRON-1389 URL: https://issues.apache.org/jira/browse/METRON-1389 Project: Metron Issue Type: Bug Affects Versions: 0.4.2 Reporter: Anand Subramanian Assignee: Anand Subramanian Fix For: Next + 1 This issue is seen with Ambari 2.6.0.0. Doing 'Metron -> Service Actions -> Zeppelin Notebook Import' shows that the notebooks have been imported, but when I go to the Zeppelin dashboard, I cannot find any of the notebooks. In recent versions of Ambari, [Shiro Authentication|https://zeppelin.apache.org/docs/0.6.0/security/shiroauthentication.html] is enabled which requires a user/password authentication in order to interact with Zeppelin. We need to incorporate this in order to be able to import metron Zeppelin notebooks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (METRON-1369) squid events are not seen in Alerts UI
Anand Subramanian created METRON-1369:
-
Summary: squid events are not seen in Alerts UI
Key: METRON-1369
URL: https://issues.apache.org/jira/browse/METRON-1369
Project: Metron
Issue Type: Bug
Reporter: Anand Subramanian
On a metron cluster, I have added the squid telemetry by following the [Wiki
instructions|https://cwiki.apache.org/confluence/display/METRON/2016/05/02/Metron+Tutorial+-+Fundamentals+Part+4%3A+Pluggable+Threat+Intelligence].
I am able to see the squid indices present in elasticsearch, but not able to
see them in the alerts UI. On further checking, I found out that this is due to
the missing nested "alert" field in the index mappings--which is required for
the Alerts UI to pick up the index.
This was confirmed in the rest logs as well.
{code}
17/12/18 10:39:40 ERROR dao.ElasticsearchRequestSubmitter: Shard search failure
[1/1]; reason=IllegalStateException: [nested] failed to find nested object
under path [alert], index=squid_index_2017.12.18.10, shard=0,
status=BAD_REQUEST, nodeId=NOjEpjy0RxCV1ltcqp4Erw
[squid_index_2017.12.18.10/a7-5TaroSRCHMrJnCWkYww] QueryShardException[failed
to create query: {
"constant_score" : {
"filter" : {
"bool" : {
"must" : [
{
"bool" : {
"should" : [
{
"query_string" : {
"query" : "*",
"fields" : [ ],
"use_dis_max" : true,
"tie_breaker" : 0.0,
"default_operator" : "or",
"auto_generate_phrase_queries" : false,
"max_determinized_states" : 1,
"enable_position_increments" : true,
"fuzziness" : "AUTO",
"fuzzy_prefix_length" : 0,
"fuzzy_max_expansions" : 50,
"phrase_slop" : 0,
"escape" : false,
"split_on_whitespace" : true,
"boost" : 1.0
}
},
{
"nested" : {
"query" : {
"query_string" : {
"query" : "*",
"fields" : [ ],
"use_dis_max" : true,
"tie_breaker" : 0.0,
"default_operator" : "or",
"auto_generate_phrase_queries" : false,
"max_determinized_states" : 1,
"enable_position_increments" : true,
"fuzziness" : "AUTO",
"fuzzy_prefix_length" : 0,
"fuzzy_max_expansions" : 50,
"phrase_slop" : 0,
"escape" : false,
"split_on_whitespace" : true,
"boost" : 1.0
}
},
"path" : "alert",
"ignore_unmapped" : false,
"score_mode" : "none",
"boost" : 1.0
}
}
],
"disable_coord" : false,
"adjust_pure_negative" : true,
"boost" : 1.0
}
},
{
"bool" : {
"should" : [
{
"term" : {
"status" : {
"value" : "active",
"boost" : 1.0
}
}
},
{
"bool" : {
"must_not" : [
{
"exists" : {
"field" : "status",
"boost" : 1.0
}
}
],
"disable_coord" : false,
"adjust_pure_negative" : true,
"boost" : 1.0
}
}
],
"disable_coord" : false,
"adjust_pure_negative" : true,
"boost" : 1.0
}
}
],
"must_not" : [
{
"exists" : {
"field" : "metaalerts",
"boost" : 1.0
}
}
],
"disable_coord" : false,
"adjust_pure_negative" : true,
"boost" : 1.0
}
},
"boost" : 1.0
}
}]; nested: IllegalStateException[[nested] failed to find nested object under
path [alert]];
at
org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:329)
at
org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:312)
at
[jira] [Created] (METRON-1335) Install metron-maas-service RPM as a part of the full-dev deployment
Anand Subramanian created METRON-1335: - Summary: Install metron-maas-service RPM as a part of the full-dev deployment Key: METRON-1335 URL: https://issues.apache.org/jira/browse/METRON-1335 Project: Metron Issue Type: Bug Reporter: Anand Subramanian Assignee: Anand Subramanian Currently, Metron MaaS is not installed by default on the full dev deployments. We copy the metron-maas-service RPM onto the /localrepo folder, but the RPM is not installed automatically. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (METRON-1328) Enhance platform-info.sh script to check if docker daemon is running
Anand Subramanian created METRON-1328: - Summary: Enhance platform-info.sh script to check if docker daemon is running Key: METRON-1328 URL: https://issues.apache.org/jira/browse/METRON-1328 Project: Metron Issue Type: Improvement Reporter: Anand Subramanian Assignee: Anand Subramanian For successful full-dev the docker daemon needs to be in running state. It is possible at times that docker is installed, but is not started. It would be good to add a check to the platform-info script to check for running status. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1327) Vagrant full-dev fails to build
[
https://issues.apache.org/jira/browse/METRON-1327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262459#comment-16262459
]
Anand Subramanian commented on METRON-1327:
---
Hi, can you paste the output of the running the script
"metron-deployment/scripts/platform-info.sh" ?
> Vagrant full-dev fails to build
> ---
>
> Key: METRON-1327
> URL: https://issues.apache.org/jira/browse/METRON-1327
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: Paweł Białasiewicz
>
> Vagrant full-dev fails with the following output:
> {code:java}
> "Downloaded:
> https://jcenter.bintray.com/org/springframework/spring-core/3.0.5.RELEASE/spring-core-3.0.5.RELEASE.pom
> (3 KB at 17.1 KB/sec)"
> "Downloading:
> http://clojars.org/repo/org/springframework/spring-expression/3.0.5.RELEASE/spring-expression-3.0.5.RELEASE.pom;
> "[INFO] "
> "[INFO] --- frontend-maven-plugin:1.3:npm (ng build) @ metron-config ---"
> "[INFO] npm not inheriting proxy config from Maven"
> "[INFO] Running 'npm run build' in
> /home/black/metron/metron-interface/metron-config"
> " "
> "Downloading:
> https://jcenter.bintray.com/org/springframework/spring-expression/3.0.5.RELEASE/spring-expression-3.0.5.RELEASE.pom;
> "1/2 KB "
> "2/2 KB "
> " "
> "Downloaded:
> https://jcenter.bintray.com/org/springframework/spring-expression/3.0.5.RELEASE/spring-expression-3.0.5.RELEASE.pom
> (2 KB at 9.7 KB/sec)"
> "Downloading:
> http://clojars.org/repo/org/springframework/spring-tx/3.2.6.RELEASE/spring-tx-3.2.6.RELEASE.pom;
> "[INFO] "
> "[INFO] > metron-management-ui@0.4.2 build
> /home/black/metron/metron-interface/metron-config"
> "[INFO] > ./node_modules/angular-cli/bin/ng build -prod"
> "[INFO] "
> " "
> "Downloading:
> https://jcenter.bintray.com/org/springframework/spring-tx/3.2.6.RELEASE/spring-tx-3.2.6.RELEASE.pom;
> "2 KB "
> "4 KB "
> " "
> "Downloaded:
> https://jcenter.bintray.com/org/springframework/spring-tx/3.2.6.RELEASE/spring-tx-3.2.6.RELEASE.pom
> (4 KB at 5.5 KB/sec)"
> "Downloading:
> http://clojars.org/repo/org/springframework/spring-webmvc/3.2.6.RELEASE/spring-webmvc-3.2.6.RELEASE.pom;
> " "
> "Downloading:
> https://jcenter.bintray.com/org/springframework/spring-webmvc/3.2.6.RELEASE/spring-webmvc-3.2.6.RELEASE.pom;
> "2 KB "
> "6 KB "
> "8 KB "
> "9 KB "
> " "
> "Downloaded:
> https://jcenter.bintray.com/org/springframework/spring-webmvc/3.2.6.RELEASE/spring-webmvc-3.2.6.RELEASE.pom
> (9 KB at 18.0 KB/sec)"
> "Downloading:
> http://clojars.org/repo/org/springframework/spring-web/3.2.6.RELEASE/spring-web-3.2.6.RELEASE.pom;
> "[INFO] Cannot find module 'tough-cookie'"
> "[INFO] Error: Cannot find module 'tough-cookie'"
> "[INFO] at Function.Module._resolveFilename (module.js:440:15)"
> "[INFO] at Function.Module._load (module.js:388:25)"
> "[INFO] at Module.require (module.js:468:17)"
> "[INFO] at require (internal/module.js:20:19)"
> "[INFO] at Object.
> (/home/black/metron/metron-interface/metron-config/node_modules/request/lib/cookies.js:3:13)"
> "[INFO] at Module._compile (module.js:541:32)"
> "[INFO] at Object.Module._extensions..js (module.js:550:10)"
> "[INFO] at Module.load (module.js:458:32)"
> "[INFO] at tryModuleLoad (module.js:417:12)"
> "[INFO] at Function.Module._load (module.js:409:3)"
> "[INFO] at Module.require (module.js:468:17)"
> "[INFO] at require (internal/module.js:20:19)"
> "[INFO] at Object.
> (/home/black/metron/metron-interface/metron-config/node_modules/request/index.js:18:15)"
> "[INFO] at Module._compile (module.js:541:32)"
> "[INFO] at Object.Module._extensions..js (module.js:550:10)"
> "[INFO] at Module.load (module.js:458:32)"
> "[INFO] at tryModuleLoad (module.js:417:12)"
> "[INFO] at Function.Module._load (module.js:409:3)"
> "[INFO] at Module.require (module.js:468:17)"
> "[INFO] at require (internal/module.js:20:19)"
> "[INFO] at Leek._enqueue
> (/home/black/metron/metron-interface/metron-config/node_modules/leek/lib/leek.js:60:30)"
> "[INFO] at Leek.track
> (/home/black/metron/metron-interface/metron-config/node_modules/leek/lib/leek.js:87:15)"
> "[INFO] at Class.Command.validateAndRun
> (/home/black/metron/metron-interface/metron-config/node_modules/angular-cli/lib/models/command.js:119:18)"
> "[INFO] at
> /home/black/metron/metron-interface/metron-config/node_modules/angular-cli/lib/cli/cli.js:86:22"
> "[INFO] at tryCatch
> (/home/black/metron/metron-interface/metron-config/node_modules/rsvp/dist/lib/rsvp/-internal.js:198:12)"
> "[INFO] at invokeCallback
> (/home/black/metron/metron-interface/metron-config/node_modules/rsvp/dist/lib/rsvp/-internal.js:211:13)"
> "[INFO] at
>
[jira] [Created] (METRON-1326) Metron deploy with Kerberos fails on Ambari 2.5 during ES service stop
Anand Subramanian created METRON-1326:
-
Summary: Metron deploy with Kerberos fails on Ambari 2.5 during ES
service stop
Key: METRON-1326
URL: https://issues.apache.org/jira/browse/METRON-1326
Project: Metron
Issue Type: Bug
Environment: 12 node VM cluster running CentOS 7
Reporter: Anand Subramanian
I am noticing that Metron deploy is failing when enabling Kerberos on a 12-node
VM cluster managed by Ambari 2.5.2.
The error is seen during the "Stop Services" step while kerberizing for
Elasticsearch Master and Elasticsearch Data Node services.
I confirmed that the same deployment goes through fine for Ambari 2.4.2
version. I am able to setup the Kerberized cluster fine.
For Ambari 2.4, for the "Elasticsearch Data Node Stop" step, we stop the slave,
and do not check on the status of the service after the 'service stop' command
was issued. But with Ambari 2.5, we attempt to check the status after the
service stop command was issued.
*In Ambari 2.4*
{code}
stdout:
Stop the Slave
2017-11-07 10:21:27,755 - Execute['service elasticsearch stop'] {}
Command completed successfully!
{code}
*In Ambari 2.5*
{code}
Stop the Slave
2017-11-07 10:12:48,481 - Execute['service elasticsearch stop'] {}
2017-11-07 10:12:48,599 - Waiting for actual component stop
Status of the Slave
2017-11-07 10:12:48,600 - Execute['service elasticsearch status'] {}
Command failed after 1 tries
{code}
Apparently the status command is returning a result with error code 3, which
the ambari agent is not liking and hence calling the step as a failure.
I am not sure entirely if this is something to be handled by Metron or by
Ambari. Please feel free to close this defect in case this is deemed out of
scope of Metron.
Here is the full error log from the UI
{code}
stderr:
Traceback (most recent call last):
File
"/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic_slave.py",
line 71, in
Elasticsearch().execute()
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 332, in execute
self.execute_prefix_function(self.command_name, 'after', env)
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 350, in execute_prefix_function
method(env)
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 398, in after_stop
status_method(env)
File
"/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic_slave.py",
line 59, in status
Execute(status_cmd)
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py",
line 166, in __init__
self.env.run()
File
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
line 160, in run
self.run_action(resource, action)
File
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
line 124, in run_action
provider_action()
File
"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
line 262, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
line 102, in checked_call
tries=tries, try_sleep=try_sleep,
timeout_kill_strategy=timeout_kill_strategy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
line 303, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'service
elasticsearch status' returned 3. ● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled;
vendor preset: disabled)
Active: inactive (dead)
Docs: http://www.elastic.co
Nov 07 10:12:47 metron-12 elasticsearch[25937]: [2017-11-07 10:12:47,340][INFO
][cluster.service ] [metron-12.openstacklocal] removed
{{metron-9.openstacklocal}{lTJDzEA6Sp6_6ryTY8XSJQ}{172.22.97.19}{172.22.97.19:9300}{master=false},},
reason:
zen-disco-node_left({metron-9.openstacklocal}{lTJDzEA6Sp6_6ryTY8XSJQ}{172.22.97.19}{172.22.97.19:9300}{master=false})
Nov 07 10:12:47 metron-12 elasticsearch[25937]: [2017-11-07 10:12:47,466][INFO
][cluster.service ] [metron-12.openstacklocal] removed
{{metron-8.openstacklocal}{Q7pgb5LLSj-oHMxld-DFfw}{172.22.97.188}{172.22.97.188:9300}{master=false},},
reason:
zen-disco-node_left({metron-8.openstacklocal}{Q7pgb5LLSj-oHMxld-DFfw}{172.22.97.188}{172.22.97.188:9300}{master=false})
Nov 07 10:12:47 metron-12
[jira] [Created] (METRON-1317) 401 Unauthorized seen when logout from Management UI
Anand Subramanian created METRON-1317:
-
Summary: 401 Unauthorized seen when logout from Management UI
Key: METRON-1317
URL: https://issues.apache.org/jira/browse/METRON-1317
Project: Metron
Issue Type: Bug
Reporter: Anand Subramanian
Attachments: 401-Unauthorized-ManagementUI.png
When you click on the 'Logout' link in the Management UI, it returns to the
login screen, and shows an error saying 'Session has expired'. From the dev
tools console, we can see a '401 Unauthorized' with the following error:
{code}
main.a736a6b4ad8978bebe79.bundle.js:2124 GET http://node1:4200/api/v1/user 401
(Unauthorized)
n @ main.a736a6b4ad8978bebe79.bundle.js:2124
t.scheduleTask @ main.a736a6b4ad8978bebe79.bundle.js:2131
n.scheduleMacroTask @ main.a736a6b4ad8978bebe79.bundle.js:2131
(anonymous) @ main.a736a6b4ad8978bebe79.bundle.js:2124
send @ VM982:3
(anonymous) @ main.a736a6b4ad8978bebe79.bundle.js:1373
t.subscribe @ main.a736a6b4ad8978bebe79.bundle.js:15
t.init @ main.a736a6b4ad8978bebe79.bundle.js:260
t @ main.a736a6b4ad8978bebe79.bundle.js:260
get @ t.ngfactory.js:267
tInjector.getInternal @ t.ngfactory.js:447
e.get @ main.a736a6b4ad8978bebe79.bundle.js:1268
_View_t_Host0.createInternal @ t_Host.ngfactory.js:15
t.create @ main.a736a6b4ad8978bebe79.bundle.js:1900
t.create @ main.a736a6b4ad8978bebe79.bundle.js:1240
e.bootstrap @ main.a736a6b4ad8978bebe79.bundle.js:715
(anonymous) @ main.a736a6b4ad8978bebe79.bundle.js:715
e._moduleDoBootstrap @ main.a736a6b4ad8978bebe79.bundle.js:715
(anonymous) @ main.a736a6b4ad8978bebe79.bundle.js:715
t.invoke @ main.a736a6b4ad8978bebe79.bundle.js:2131
onInvoke @ main.a736a6b4ad8978bebe79.bundle.js:1942
t.invoke @ main.a736a6b4ad8978bebe79.bundle.js:2131
n.run @ main.a736a6b4ad8978bebe79.bundle.js:2131
(anonymous) @ main.a736a6b4ad8978bebe79.bundle.js:2131
t.invokeTask @ main.a736a6b4ad8978bebe79.bundle.js:2131
onInvokeTask @ main.a736a6b4ad8978bebe79.bundle.js:1942
t.invokeTask @ main.a736a6b4ad8978bebe79.bundle.js:2131
n.runTask @ main.a736a6b4ad8978bebe79.bundle.js:2131
o @ main.a736a6b4ad8978bebe79.bundle.js:2131
Promise resolved (async)
n @ main.a736a6b4ad8978bebe79.bundle.js:2131
r @ main.a736a6b4ad8978bebe79.bundle.js:2131
t.scheduleTask @ main.a736a6b4ad8978bebe79.bundle.js:2131
n.scheduleMicroTask @ main.a736a6b4ad8978bebe79.bundle.js:2131
f @ main.a736a6b4ad8978bebe79.bundle.js:2131
t.then @ main.a736a6b4ad8978bebe79.bundle.js:2131
t._compileComponents @ main.a736a6b4ad8978bebe79.bundle.js:1086
t._compileModuleAndComponents @ main.a736a6b4ad8978bebe79.bundle.js:1086
t.compileModuleAsync @ main.a736a6b4ad8978bebe79.bundle.js:1086
e._bootstrapModuleWithZone @ main.a736a6b4ad8978bebe79.bundle.js:715
e.bootstrapModule @ main.a736a6b4ad8978bebe79.bundle.js:715
(anonymous) @ main.a736a6b4ad8978bebe79.bundle.js:1508
r @ inline.js:18
(anonymous) @ main.a736a6b4ad8978bebe79.bundle.js:2131
r @ inline.js:18
window.webpackJsonp @ inline.js:18
(anonymous) @ main.a736a6b4ad8978bebe79.bundle.js:1
DevTools failed to parse SourceMap: http://node1:4200/bootstrap.css.map
:4200/login:1 This page includes a password or credit card input in a
non-secure context. A warning has been added to the URL bar. For more
information, see https://goo.gl/zmWq3m.
{code}
See screenshot attached.
Note that this behavior is not seen with the Alerts UI, when the same steps are
repeated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-353) Monit unable to start mysqld after successful single node vagrant deployment
[
https://issues.apache.org/jira/browse/METRON-353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226621#comment-16226621
]
Anand Subramanian commented on METRON-353:
--
This defect is not valid anymore since we do not depend on mysql for Geo
enrichment.
> Monit unable to start mysqld after successful single node vagrant deployment
>
>
> Key: METRON-353
> URL: https://issues.apache.org/jira/browse/METRON-353
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.2.1BETA
> Environment: Quick dev setup installation using vagrant on Macbook
> Pro.
>Reporter: Anand Subramanian
> Labels: metronqe
> Attachments: messages, messages-20160725, output-platform-info.rtf
>
>
> This is a single node quick-dev deployment from HEAD using vagrant on a
> Macbook Pro. The platform-info output is attached.
> I noticed that monit was not able to start despite trying multiple times and
> I had to start it manually post-deployment.
> Note that post-installation, I am able to manually start the mysqld service
> and it comes up fine.
> Here are the relevant lines from the /var/log/messages on the vagrant node.
> {code}
> Jul 25 10:30:11 node1 ansible-command: Invoked with warn=True executable=None
> chdir=None _raw_params=monit start mysql removes=None creates=None
> _uses_shell=True
>
> Jul 25 10:30:21 node1 ansible-command: Invoked with warn=True executable=None
> chdir=None _raw_params=monit start mysql removes=None creates=None
> _uses_shell=True
> Jul 25 10:30:32 node1 ansible-command: Invoked with warn=True executable=None
> chdir=None _raw_params=monit start mysql removes=None creates=None
> _uses_shell=True
> Jul 25 10:30:42 node1 ansible-command: Invoked with warn=True executable=None
> chdir=None _raw_params=monit start mysql removes=None creates=None
> _uses_shell=True
> Jul 25 10:30:52 node1 ansible-command: Invoked with warn=True executable=None
> chdir=None _raw_params=monit start mysql removes=None creates=None
> _uses_shell=True
> Jul 25 10:31:02 node1 ansible-command: Invoked with warn=True executable=None
> chdir=None _raw_params=monit start mysql removes=None creates=None
> _uses_shell=True
>
> {code}
> From the /var/log/mysqld.log, I could see that there was a shutdown initiated
> after monit tried restarting bunch of times:
> {code}
> 2016-07-25T10:30:55.152359Z 0 [Note] Giving 0 client threads a chance to die
> gracefully
> 2016-07-25T10:30:55.152397Z 0 [Note] Shutting down slave threads
> 2016-07-25T10:30:55.152405Z 0 [Note] Forcefully disconnecting 0 remaining
> clients
> 2016-07-25T10:30:55.152471Z 0 [Note] Event Scheduler: Purging the queue. 0
> events
> 2016-07-25T10:30:55.153536Z 0 [Note] Binlog end
> 2016-07-25T10:30:55.154549Z 0 [Note] Shutting down plugin 'validate_password'
> 2016-07-25T10:30:55.154573Z 0 [Note] Shutting down plugin 'ngram'
> 2016-07-25T10:30:55.154577Z 0 [Note] Shutting down plugin 'ARCHIVE'
> 2016-07-25T10:30:55.154581Z 0 [Note] Shutting down plugin 'partition'
> 2016-07-25T10:30:55.154584Z 0 [Note] Shutting down plugin 'BLACKHOLE'
> 2016-07-25T10:30:55.154588Z 0 [Note] Shutting down plugin 'CSV'
> 2016-07-25T10:30:55.154601Z 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
> 2016-07-25T10:30:55.154621Z 0 [Note] Shutting down plugin 'MyISAM'
> 2016-07-25T10:30:55.154631Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
> 2016-07-25T10:30:55.154635Z 0 [Note] Shutting down plugin
> 'INNODB_SYS_DATAFILES'
> 2016-07-25T10:30:55.154638Z 0 [Note] Shutting down plugin
> 'INNODB_SYS_TABLESPACES'
> 2016-07-25T10:30:55.154640Z 0 [Note] Shutting down plugin
> ‘INNODB_SYS_FOREIGN_COLS'
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1178) kinit + authorize on all nodes running topologies for multi-node metron deployments
[
https://issues.apache.org/jira/browse/METRON-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226617#comment-16226617
]
Anand Subramanian commented on METRON-1178:
---
This issue is no longer seen now. It looks like the set of fixes made by Nick
around this area resolved this as well. Closing defect.
> kinit + authorize on all nodes running topologies for multi-node metron
> deployments
>
>
> Key: METRON-1178
> URL: https://issues.apache.org/jira/browse/METRON-1178
> Project: Metron
> Issue Type: Bug
>Reporter: Anand Subramanian
>
> In a 12-node deployment, it is seen post-kerberization that some of the
> topologies fail to start with exceptions thrown like one pasted below. This
> is because the node does not have the proper credentials, since kinit has not
> occurred on this host.
> {code}
> 2017-09-11 16:04:46.923 o.a.h.i.Client [WARN] Exception encountered while
> connecting to the server : javax.security.sasl.SaslException: GSS ini
> tiate failed [Caused by GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos tgt)]
> 2017-09-11 16:04:46.930 o.a.m.p.GrokParser [ERROR] Failed on local exception:
> java.io.IOException: javax.security.sasl.SaslException: GSS initi
> ate failed [Caused by GSSException: No valid credentials provided (Mechanism
> level: Failed to find any Kerberos tgt)]; Host Details : local hos
> t is: "metron-2/xx.xx.xx.xx; destination host is:
> "metron-2.openstacklocal":8020;
> java.io.IOException: Failed on local exception: java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSExcep
> tion: No valid credentials provided (Mechanism level: Failed to find any
> Kerberos tgt)]; Host Details : local host is: "nat-r7-lqys-metron-2/17
> 2.22.104.43"; destination host is: "nat-r7-lqys-metron-2.openstacklocal":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
> ~[stormjar.jar:?]
> at org.apache.hadoop.ipc.Client.call(Client.java:1480) ~[stormjar.jar:?]
> at org.apache.hadoop.ipc.Client.call(Client.java:1407) ~[stormjar.jar:?]
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> ~[stormjar.jar:?]
> at com.sun.proxy.$Proxy45.getFileInfo(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
> ~[stor
> mjar.jar:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_141]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_141]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> ~[stormjar.jar:?]
> at com.sun.proxy.$Proxy46.getFileInfo(Unknown Source) ~[?:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> ~[stormjar.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
> ~[stormjar.jar:?]
> at
> org.apache.metron.parsers.GrokParser.openInputStream(GrokParser.java:83)
> ~[stormjar.jar:?]
> at org.apache.metron.parsers.GrokParser.init(GrokParser.java:94)
> [stormjar.jar:?]
> at
> org.apache.metron.parsers.bolt.ParserBolt.prepare(ParserBolt.java:108)
> [stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__6573$fn__6586.invoke(executor.clj:798)
> [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:482)
> [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
> Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS
> initiate failed [Caused by GSSException: No valid credentials provided (
> Mechanism level: Failed to find any Kerberos tgt)]
>
[jira] [Created] (METRON-1283) Install Elasticsearch template as a part of the mpack startup scripts
Anand Subramanian created METRON-1283: - Summary: Install Elasticsearch template as a part of the mpack startup scripts Key: METRON-1283 URL: https://issues.apache.org/jira/browse/METRON-1283 Project: Metron Issue Type: Bug Reporter: Anand Subramanian Assignee: Anand Subramanian For a Metron multi-node deployment using mpack, the Elasticsearch template is required to be installed manually post-setup. These templates are required for the proper working of, for e.g. the Alerts UI. In the event that these templates are not installed, and if data is ingested, these would not be shown in the Alerts UI, since there would be missing fields without the template files (E.g. snort alert indices are not displayed in the Alerts UI, since it is missing the "alerts" field from the mapping). In such a case, one needs to install the templates, delete all indices for the given parser and re-ingest data again into the parser for it to appear in the Alerts UI. Further, the indices from all the parsers will have to be deleted and re-ingested again which could be a tedious job in the event that this step was missed out by chance. I have also seen other ill-effects from having stale indices for parsers that was created before template install. While documenting the template installation is a good practice, nothing would more failsafe than installing the template as a part of the mpack startup scripts itself. Note that this issue would not be seen on vagrant deployments, since the 'load_web_templates' role would trigger the installation automatically. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (METRON-1263) Start Alerts UI service after Metron REST
Anand Subramanian created METRON-1263: - Summary: Start Alerts UI service after Metron REST Key: METRON-1263 URL: https://issues.apache.org/jira/browse/METRON-1263 Project: Metron Issue Type: Bug Reporter: Anand Subramanian Assignee: Anand Subramanian The Metron Alerts UI service should be started after the Metron REST service has started, similar to the Metron Management UI service. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (METRON-1178) kinit + authorize on all nodes running topologies for multi-node metron deployments
[
https://issues.apache.org/jira/browse/METRON-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anand Subramanian updated METRON-1178:
--
Description:
In a 12-node deployment, it is seen post-kerberization that some of the
topologies fail to start with exceptions thrown like one pasted below. This is
because the node does not have the proper credentials, since kinit has not
occurred on this host.
{code}
2017-09-11 16:04:46.923 o.a.h.i.Client [WARN] Exception encountered while
connecting to the server : javax.security.sasl.SaslException: GSS ini
tiate failed [Caused by GSSException: No valid credentials provided (Mechanism
level: Failed to find any Kerberos tgt)]
2017-09-11 16:04:46.930 o.a.m.p.GrokParser [ERROR] Failed on local exception:
java.io.IOException: javax.security.sasl.SaslException: GSS initi
ate failed [Caused by GSSException: No valid credentials provided (Mechanism
level: Failed to find any Kerberos tgt)]; Host Details : local hos
t is: "metron-2/xx.xx.xx.xx; destination host is:
"metron-2.openstacklocal":8020;
java.io.IOException: Failed on local exception: java.io.IOException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSExcep
tion: No valid credentials provided (Mechanism level: Failed to find any
Kerberos tgt)]; Host Details : local host is: "nat-r7-lqys-metron-2/17
2.22.104.43"; destination host is: "nat-r7-lqys-metron-2.openstacklocal":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
~[stormjar.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1480) ~[stormjar.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1407) ~[stormjar.jar:?]
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
~[stormjar.jar:?]
at com.sun.proxy.$Proxy45.getFileInfo(Unknown Source) ~[?:?]
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
~[stor
mjar.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_141]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_141]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_141]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
~[stormjar.jar:?]
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
~[stormjar.jar:?]
at com.sun.proxy.$Proxy46.getFileInfo(Unknown Source) ~[?:?]
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116)
~[stormjar.jar:?]
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
~[stormjar.jar:?]
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
~[stormjar.jar:?]
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
~[stormjar.jar:?]
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
~[stormjar.jar:?]
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
~[stormjar.jar:?]
at
org.apache.metron.parsers.GrokParser.openInputStream(GrokParser.java:83)
~[stormjar.jar:?]
at org.apache.metron.parsers.GrokParser.init(GrokParser.java:94)
[stormjar.jar:?]
at
org.apache.metron.parsers.bolt.ParserBolt.prepare(ParserBolt.java:108)
[stormjar.jar:?]
at
org.apache.storm.daemon.executor$fn__6573$fn__6586.invoke(executor.clj:798)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:482)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate
failed [Caused by GSSException: No valid credentials provided (
Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:682)
~[stormjar.jar:?]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_141]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_141]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
~[stormjar.jar:?]
at
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:645)
~[stormjar.jar:?]
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:732)
~[stormjar.jar:?]
at
[jira] [Updated] (METRON-1178) kinit + authorize on all nodes running topologies for multi-node metron deployments
[
https://issues.apache.org/jira/browse/METRON-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anand Subramanian updated METRON-1178:
--
Summary: kinit + authorize on all nodes running topologies for multi-node
metron deployments (was: kinit on all nodes running topologies for multi-node
metron deployments )
> kinit + authorize on all nodes running topologies for multi-node metron
> deployments
>
>
> Key: METRON-1178
> URL: https://issues.apache.org/jira/browse/METRON-1178
> Project: Metron
> Issue Type: Bug
>Reporter: Anand Subramanian
>
> In a 12-node deployment, it is seen post-kerberization that some of the
> topologies fail to start with exceptions thrown like one pasted below. This
> is because the node does not have the proper credentials, since kinit has not
> occurred on this host.
> {code}
> 2017-09-11 16:04:46.923 o.a.h.i.Client [WARN] Exception encountered while
> connecting to the server : javax.security.sasl.SaslException: GSS ini
> tiate failed [Caused by GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos tgt)]
> 2017-09-11 16:04:46.930 o.a.m.p.GrokParser [ERROR] Failed on local exception:
> java.io.IOException: javax.security.sasl.SaslException: GSS initi
> ate failed [Caused by GSSException: No valid credentials provided (Mechanism
> level: Failed to find any Kerberos tgt)]; Host Details : local hos
> t is: "metron-2/xx.xx.xx.xx; destination host is:
> "metron-2.openstacklocal":8020;
> java.io.IOException: Failed on local exception: java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSExcep
> tion: No valid credentials provided (Mechanism level: Failed to find any
> Kerberos tgt)]; Host Details : local host is: "nat-r7-lqys-metron-2/17
> 2.22.104.43"; destination host is: "nat-r7-lqys-metron-2.openstacklocal":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
> ~[stormjar.jar:?]
> at org.apache.hadoop.ipc.Client.call(Client.java:1480) ~[stormjar.jar:?]
> at org.apache.hadoop.ipc.Client.call(Client.java:1407) ~[stormjar.jar:?]
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> ~[stormjar.jar:?]
> at com.sun.proxy.$Proxy45.getFileInfo(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
> ~[stor
> mjar.jar:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_141]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_141]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> ~[stormjar.jar:?]
> at com.sun.proxy.$Proxy46.getFileInfo(Unknown Source) ~[?:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> ~[stormjar.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
> ~[stormjar.jar:?]
> at
> org.apache.metron.parsers.GrokParser.openInputStream(GrokParser.java:83)
> ~[stormjar.jar:?]
> at org.apache.metron.parsers.GrokParser.init(GrokParser.java:94)
> [stormjar.jar:?]
> at
> org.apache.metron.parsers.bolt.ParserBolt.prepare(ParserBolt.java:108)
> [stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__6573$fn__6586.invoke(executor.clj:798)
> [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:482)
> [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
> Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS
> initiate failed [Caused by GSSException: No valid credentials provided (
> Mechanism level: Failed to find any Kerberos
[jira] [Created] (METRON-1177) Stale running topologies seen post-kerberization and cause exceptions
Anand Subramanian created METRON-1177:
-
Summary: Stale running topologies seen post-kerberization and
cause exceptions
Key: METRON-1177
URL: https://issues.apache.org/jira/browse/METRON-1177
Project: Metron
Issue Type: Bug
Reporter: Anand Subramanian
*Steps to Reproduce*
1. Install Ambari on 12 node cluster
2. Install Metron using MPack
3. Metron topologies are now running as user 'storm'
4. Setup standalone KDC on Ambari node by following these
[steps|https://github.com/apache/metron/blob/master/metron-deployment/Kerberos-manual-setup.md#setup-a-kdc]
5. Initiate "kerberize" cluster using Ambari
6. Ambari triggers "Stop all services"
7. In the event of a race condition, the REST service shuts down (which
prevents from being able to shutdown metron topologies)
8. Metron topology shut down occurs, but Metron Mpack cannot stop topologies
because REST already stopped. Metron ignores this condition and continues on
assuming the topologies are down
9. Storm service shuts down
10. Cluster kerberization occurs
11. Ambari triggers "Start all services"
12. Storm service starts.
13. Since the Metron topologies were running when Storm was shutdown, these
same topologies are attempted to be started again by Storm, but are declared as
"already running"
14. All other services start successfully
15. Now, if we attempt to run a test with the stale topologies, following
exceptions are seen:
{code}
AuthorizationException(msg:getTopologyInfo on topology snort is not authorized)
at
org.apache.storm.generated.Nimbus$getTopologyInfo_result$getTopologyInfo_resultStandardScheme.read(Nimbus.java:36971)
at
org.apache.storm.generated.Nimbus$getTopologyInfo_result$getTopologyInfo_resultStandardScheme.read(Nimbus.java:36939)
at
org.apache.storm.generated.Nimbus$getTopologyInfo_result.read(Nimbus.java:36870)
at
org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at
org.apache.storm.generated.Nimbus$Client.recv_getTopologyInfo(Nimbus.java:1182)
at
org.apache.storm.generated.Nimbus$Client.getTopologyInfo(Nimbus.java:1169)
at
org.hw.metron.lib.storm.StormClient.isTopologyRunning(StormClient.java:150)
at
org.hw.metron.lib.storm.StormClient.startTopology(StormClient.java:68)
at
functional.SnortTopologyTest.snortElasticsearchIndexTest(SnortTopologyTest.java:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:86)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:643)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:820)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1128)
at
org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
at org.testng.TestRunner.privateRun(TestRunner.java:782)
at org.testng.TestRunner.run(TestRunner.java:632)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:366)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:319)
at org.testng.SuiteRunner.run(SuiteRunner.java:268)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1244)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1169)
at org.testng.TestNG.run(TestNG.java:1064)
at
org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:295)
at
org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:84)
at
org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:90)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-872) "Metron Enrichment Start" Fails in Quick Dev
[
https://issues.apache.org/jira/browse/METRON-872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153505#comment-16153505
]
Anand Subramanian commented on METRON-872:
--
[~nickwallen] I agree with you about taking down quick dev, since it has been
broken for sometime now.
> "Metron Enrichment Start" Fails in Quick Dev
>
>
> Key: METRON-872
> URL: https://issues.apache.org/jira/browse/METRON-872
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>
> h3. Problem
> The "Quick Dev" deployment fails to complete deployment successfully for me.
> Ambari fails to launch the Enrichment topology with this error.
> {code}
> resource_management.core.exceptions.ExecutionFailed: Execution of
> '/usr/metron/0.3.1/bin/zk_load_configs.sh --mode PUSH -i
> /usr/metron/0.3.1/config/zookeeper -z node1:2181' returned 127. /bin/bash:
> /usr/metron/0.3.1/bin/zk_load_configs.sh: No such file or directory
> {code}
> This is confusing to me because we should be on 0.4.0, so why is it
> referencing 0.3.1?
> h3. Steps to Reproduce
> First, I had to `install` the Metron libs (which I don't remember being a
> requirement before.) Then I was able to `vagrant up`.
> {code}
> cd incubator-metron
> mvn clean install -DskipTests -T 1C
> cd metron-deployment/vagrant/quick-dev-platform
> vagrant up
> {code}
> h3. Additional Clues
> When I login to the "Quick Dev" VM, I see that there are some 0.4.0 libs and
> some 0.3.1 libs.
> {code}
> [root@node1 metron]# ls -l /usr/metron/0.3.1/lib/
> total 167800
> -rw-r--r--. 1 root root 85332259 Apr 3 21:05 metron-api-0.3.1.jar
> -rw-r--r--. 1 root root 86490682 Mar 30 13:09 metron-pcap-backend-0.3.1.jar
> [root@node1 metron]# ls -l /usr/metron/0.4.0/lib/
> total 463428
> -rw-r--r--. 1 root root 48425098 Apr 21 13:45 metron-common-0.4.0.jar
> -rw-r--r--. 1 root root 124020704 Apr 21 13:50
> metron-data-management-0.4.0.jar
> -rw-r--r--. 1 root root 114754821 Apr 21 13:50
> metron-elasticsearch-0.4.0-uber.jar
> -rw-r--r--. 1 root root 93280994 Apr 21 13:48
> metron-enrichment-0.4.0-uber.jar
> -rw-r--r--. 1 root root 94059479 Apr 21 13:50 metron-parsers-0.4.0-uber.jar
> {code}
> Where did those 0.3.1 libs come from? When I look in my local Maven
> repository on my build machine, I don't see any 0.3.1 libs at all, only 0.4.0
> {code}
> $ find ~/.m2/repository/org/apache/metron/ -name "0.3.1"
> $ find ~/.m2/repository/org/apache/metron/ -name "0.4.0"
> /Users/nallen/.m2/repository/org/apache/metron//elasticsearch-shaded/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//Metron/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-analytics/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-api/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-common/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-config/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-data-management/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-deployment/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-docker/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-elasticsearch/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-enrichment/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-hbase/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-indexing/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-integration-test/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-interface/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-maas-common/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-maas-service/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-management/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-parsers/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-pcap/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-pcap-backend/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-platform/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-profiler/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-profiler-client/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-profiler-common/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-rest/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-rest-client/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-solr/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-statistics/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-storm-kafka/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-test-utilities/0.4.0
> /Users/nallen/.m2/repository/org/apache/metron//metron-writer/0.4.0
> {code}
> I seem to have the latest "Quick Dev" box
[jira] [Created] (METRON-1146) Add ability to parse JSON string into JSONObject for stellar
Anand Subramanian created METRON-1146:
-
Summary: Add ability to parse JSON string into JSONObject for
stellar
Key: METRON-1146
URL: https://issues.apache.org/jira/browse/METRON-1146
Project: Metron
Issue Type: Bug
Reporter: Anand Subramanian
Assignee: Anand Subramanian
Use jackson to parse a given JSON string and return a JSONObject on the
stellar. For example:
{code}
MAP_GET( 'bar', PARSE_JSON_STRING( '{ "foo" : 1, "bar" : 2}' )
{code}
would give the output 2
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (METRON-1093) Start/stop/restart topologies use Storm CLI
[ https://issues.apache.org/jira/browse/METRON-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149019#comment-16149019 ] Anand Subramanian edited comment on METRON-1093 at 8/31/17 2:03 PM: Due to this issue, some of the storm topologies end up running post-kerberos. Because of this, some of the topologies would throw authentication errors, since they are stale and haven't been killed/redeployed. Indices might not be generated if the indexing topology were not restarted for e.g. Another side effect also seen is that Management UI reports the status of the sensors incorrectly. The sensors would show up as stopped, while the actual status is running. was (Author: anandsubbu): Due to this issue, some of the storm topologies end up as still running post-kerberos. Because of this, some of the topologies would throw authentication errors, since they are stale and haven't been killed/redeployed. Another side effect also seen is that Management UI reports the status of the sensors incorrectly. The sensors would show up as stopped, while the actual status is running. > Start/stop/restart topologies use Storm CLI > --- > > Key: METRON-1093 > URL: https://issues.apache.org/jira/browse/METRON-1093 > Project: Metron > Issue Type: Improvement >Reporter: David M. Lyle > > Currently both Ambari status check and Ambari Start/Stop/Restart topologies > use Storm REST to get the status of the topologies. This can lead to an > inaccurate status when the topologies are running and the Storm UI is not. > During an Ambari restart, the topologies wouldn't be restarted when this > occurs. > Ambari needs status check to be quick, so continue to use Storm REST. Modify > start/stop/restart to get the running topologies via the Storm CLI. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1093) Start/stop/restart topologies use Storm CLI
[ https://issues.apache.org/jira/browse/METRON-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149019#comment-16149019 ] Anand Subramanian commented on METRON-1093: --- Due to this issue, some of the storm topologies end up as still running post-kerberos. Because of this, some of the topologies would throw authentication errors, since they are stale and haven't been killed/redeployed. Another side effect also seen is that Management UI reports the status of the sensors incorrectly. The sensors would show up as stopped, while the actual status is running. > Start/stop/restart topologies use Storm CLI > --- > > Key: METRON-1093 > URL: https://issues.apache.org/jira/browse/METRON-1093 > Project: Metron > Issue Type: Improvement >Reporter: David M. Lyle > > Currently both Ambari status check and Ambari Start/Stop/Restart topologies > use Storm REST to get the status of the topologies. This can lead to an > inaccurate status when the topologies are running and the Storm UI is not. > During an Ambari restart, the topologies wouldn't be restarted when this > occurs. > Ambari needs status check to be quick, so continue to use Storm REST. Modify > start/stop/restart to get the running topologies via the Storm CLI. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (METRON-1141) NameError: metronProfilerHost is not defined in stack_advisor.py
[
https://issues.apache.org/jira/browse/METRON-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anand Subramanian updated METRON-1141:
--
Summary: NameError: metronProfilerHost is not defined in stack_advisor.py
(was: Fix metronProfilerHost name in stack_advisor.py)
> NameError: metronProfilerHost is not defined in stack_advisor.py
>
>
> Key: METRON-1141
> URL: https://issues.apache.org/jira/browse/METRON-1141
> Project: Metron
> Issue Type: Bug
>Reporter: Anand Subramanian
>Assignee: Anand Subramanian
>
> With the most recent changes from METRON-1122, when trying to deploy a Metron
> stack using Ambari + Mpack, the wizard hangs at step #5 "Assign Masters"
> page. The logs show the following error message:
> {code}
> Error occured in stack advisor.
> Error details: global name 'metronProfilerHost' is not defined
> 30 Aug 2017 06:12:50,285 INFO [ambari-client-thread-27]
> StackAdvisorRunner:71 - advisor script stderr: Traceback (most recent
> call last):
> File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 158,
> in
> main(sys.argv)
> File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 106,
> in main
> result = stackAdvisor.validateComponentLayout(services, hosts)
> File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
> line 529, in validateComponentLayout
> validationItems = self.getComponentLayoutValidations(services, hosts)
> File
> "/var/lib/ambari-server/resources/scripts/./../stacks/HDP/2.5/services/stack_advisor.py",
> line 93, in getComponentLayoutValidations
> parentItems = super(HDP25StackAdvisor,
> self).getComponentLayoutValidations(services, hosts)
> File
> "/var/lib/ambari-server/resources/scripts/./../stacks/HDP/2.3/services/stack_advisor.py",
> line 40, in getComponentLayoutValidations
> parentItems = super(HDP23StackAdvisor,
> self).getComponentLayoutValidations(services, hosts)
> File
> "/var/lib/ambari-server/resources/scripts/./../stacks/HDP/2.0.6/services/stack_advisor.py",
> line 41, in getComponentLayoutValidations
> items = super(HDP206StackAdvisor,
> self).getComponentLayoutValidations(services, hosts)
> File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
> line 550, in getComponentLayoutValidations
>
> items.extend(serviceAdvisor.getServiceComponentLayoutValidations(services,
> hosts))
> File
> "/var/lib/ambari-server/resources/common-services/METRON/0.4.1.1.3.0.0/service_advisor.py",
> line 80, in getServiceComponentLayoutValidations
> if metronParsersHost != metronProfilerHost:
> NameError: global name 'metronProfilerHost' is not defined
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (METRON-1141) Fix metronProfilerHost name in stack_advisor.py
Anand Subramanian created METRON-1141:
-
Summary: Fix metronProfilerHost name in stack_advisor.py
Key: METRON-1141
URL: https://issues.apache.org/jira/browse/METRON-1141
Project: Metron
Issue Type: Bug
Reporter: Anand Subramanian
Assignee: Anand Subramanian
With the most recent changes from METRON-1122, when trying to deploy a Metron
stack using Ambari + Mpack, the wizard hangs at step #5 "Assign Masters" page.
The logs show the following error message:
{code}
Error occured in stack advisor.
Error details: global name 'metronProfilerHost' is not defined
30 Aug 2017 06:12:50,285 INFO [ambari-client-thread-27] StackAdvisorRunner:71
- advisor script stderr: Traceback (most recent call last):
File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 158,
in
main(sys.argv)
File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 106,
in main
result = stackAdvisor.validateComponentLayout(services, hosts)
File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
line 529, in validateComponentLayout
validationItems = self.getComponentLayoutValidations(services, hosts)
File
"/var/lib/ambari-server/resources/scripts/./../stacks/HDP/2.5/services/stack_advisor.py",
line 93, in getComponentLayoutValidations
parentItems = super(HDP25StackAdvisor,
self).getComponentLayoutValidations(services, hosts)
File
"/var/lib/ambari-server/resources/scripts/./../stacks/HDP/2.3/services/stack_advisor.py",
line 40, in getComponentLayoutValidations
parentItems = super(HDP23StackAdvisor,
self).getComponentLayoutValidations(services, hosts)
File
"/var/lib/ambari-server/resources/scripts/./../stacks/HDP/2.0.6/services/stack_advisor.py",
line 41, in getComponentLayoutValidations
items = super(HDP206StackAdvisor,
self).getComponentLayoutValidations(services, hosts)
File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
line 550, in getComponentLayoutValidations
items.extend(serviceAdvisor.getServiceComponentLayoutValidations(services,
hosts))
File
"/var/lib/ambari-server/resources/common-services/METRON/0.4.1.1.3.0.0/service_advisor.py",
line 80, in getServiceComponentLayoutValidations
if metronParsersHost != metronProfilerHost:
NameError: global name 'metronProfilerHost' is not defined
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (METRON-1128) MAAS_GET_ENDPOINT - Unable to resolve function error seen for squid example
Anand Subramanian created METRON-1128:
-
Summary: MAAS_GET_ENDPOINT - Unable to resolve function error seen
for squid example
Key: METRON-1128
URL: https://issues.apache.org/jira/browse/METRON-1128
Project: Metron
Issue Type: Bug
Reporter: Anand Subramanian
Assignee: Casey Stella
While attempting to run the the DGA MaaS
[example|https://github.com/apache/metron/tree/master/metron-analytics/metron-maas-service#example]
with squid, a ParseException is seen saying it could not resolve the
'MAAS_GET_ENDPOINT' function. Full exception is pasted at the bottom of the
description.
I confirmed that the metron-management jar is present under $METRON_HOME/lib.
Also, the MAAS functions are listed when I checked on the stellar REPL.
[Stellar]>>> ?MAAS_
?MAAS_GET_ENDPOINT ?MAAS_MODEL_APPLY
I have tried to kill and start the squid topology, but it did not resolve the
issue.
Full exception:
{code}
017-08-23 09:11:01.113 o.a.s.d.executor [ERROR]
java.lang.IllegalStateException: Unable to process transformation:
MAP_GET('is_malicious', MAAS_MODEL_APPLY(MAAS_GET_ENDPOINT('dga'), {'host' :
domain_without_subdomains})) for is_malicious because Unable to resolve
function named 'MAAS_GET_ENDPOINT'. Valid functions are
IS_DATE,ZIP,IS_DOMAIN,MONTH,FORMAT,TRIM,SQRT,WEEK_OF_MONTH,JOIN,MAP_GET,LOG10,FLOOR,TO_INTEGER,YEAR,OUTLIER_MAD_SCORE,WEEK_OF_YEAR,PROTOCOL_TO_NAME,GET_FIRST,FILL_LEFT,STATS_MAX,CHOP,DECODE,PROFILE_FIXED,DOMAIN_REMOVE_SUBDOMAINS,STATS_SUM_SQUARES,REGEXP_GROUP_VAL,STATS_INIT,ENRICHMENT_GET,STATS_SD,HLLP_INIT,GEO_GET,GET,STATS_COUNT,IS_INTEGER,DAY_OF_WEEK,MAP,ENCODE,IS_EMPTY,DAY_OF_YEAR,MAP_EXISTS,TO_LOWER,STATS_SKEWNESS,ENRICHMENT_EXISTS,DAY_OF_MONTH,LENGTH,IN_SUBNET,SPLIT,PROFILER_APPLY,ABS,STATS_VARIANCE,STATS_ADD,PROFILER_FLUSH,TO_UPPER,OUTLIER_MAD_STATE_MERGE,TO_EPOCH_TIMESTAMP,BLOOM_EXISTS,GET_SUPPORTED_ENCODINGS,COS,BLOOM_INIT,URL_TO_PATH,OUTLIER_MAD_ADD,URL_TO_PROTOCOL,TO_FLOAT,STATS_SUM_LOGS,IS_IP,TO_LONG,STATS_MEAN,PROFILE_GET,CHOMP,TAN,LN,LIST_ADD,PROFILER_INIT,ENDS_WITH,DOMAIN_TO_TLD,COUNT_MATCHES,BYTEARRAY_MATCHER,TO_STRING,HLLP_MERGE,HLLP_CARDINALITY,APPEND_IF_MISSING,BLOOM_MERGE,STARTS_WITH,STATS_KURTOSIS,DOMAIN_REMOVE_TLD,LOG2,SYSTEM_ENV_GET,CEILING,HLLP_ADD,SIN,STATS_BIN,REDUCE,IS_EMAIL,REGEXP_MATCH,PROFILE_WINDOW,URL_TO_PORT,ROUND,GET_LAST,FILL_RIGHT,STATS_POPULATION_VARIANCE,SYSTEM_PROPERTY_GET,IS_ENCODING,FILTER,BIN,BLOOM_ADD,STATS_QUADRATIC_MEAN,STATS_GEOMETRIC_MEAN,STATS_MIN,PREPEND_IF_MISSING,URL_TO_HOST,ZIP_LONGEST,IS_URL,STATS_SUM,STATS_MERGE,STATS_PERCENTILE,EXP,STRING_ENTROPY,TO_DOUBLE
at
org.apache.metron.common.field.transformation.StellarTransformation.map(StellarTransformation.java:61)
~[stormjar.jar:?]
at
org.apache.metron.common.configuration.FieldTransformer.transform(FieldTransformer.java:108)
~[stormjar.jar:?]
at
org.apache.metron.common.configuration.FieldTransformer.transformAndUpdate(FieldTransformer.java:120)
~[stormjar.jar:?]
at
org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:190)
[stormjar.jar:?]
at
org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734)
[storm-core-1.0.1.2.5.6.0-40.jar:1.0.1.2.5.6.0-40]
at
org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
[storm-core-1.0.1.2.5.6.0-40.jar:1.0.1.2.5.6.0-40]
at
org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
[storm-core-1.0.1.2.5.6.0-40.jar:1.0.1.2.5.6.0-40]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
[storm-core-1.0.1.2.5.6.0-40.jar:1.0.1.2.5.6.0-40]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
[storm-core-1.0.1.2.5.6.0-40.jar:1.0.1.2.5.6.0-40]
at
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
[storm-core-1.0.1.2.5.6.0-40.jar:1.0.1.2.5.6.0-40]
at
org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
[storm-core-1.0.1.2.5.6.0-40.jar:1.0.1.2.5.6.0-40]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
[storm-core-1.0.1.2.5.6.0-40.jar:1.0.1.2.5.6.0-40]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
Caused by: org.apache.metron.stellar.dsl.ParseException: Unable to resolve
function named 'MAAS_GET_ENDPOINT'. Valid functions are
[jira] [Created] (METRON-1112) Metron Ambari deployment failure on CentOS7
Anand Subramanian created METRON-1112:
-
Summary: Metron Ambari deployment failure on CentOS7
Key: METRON-1112
URL: https://issues.apache.org/jira/browse/METRON-1112
Project: Metron
Issue Type: Bug
Affects Versions: 0.4.1
Reporter: Anand Subramanian
Due to a recent issue with installation of nodejs
(https://bugs.centos.org/view.php?id=13669), metron deployments on CentOS7 will
fail.
A temporary workaround is to do the following on the nodes that will have
metron:
{code}
rpm -ivh
https://kojipkgs.fedoraproject.org//packages/http-parser/2.7.1/3.el7/x86_64/http-parser-2.7.1-3.el7.x86_64.rpm
&& yum -y install nodejs
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (METRON-1078) Metron Indexing fails to stop during "Stop All Services"
Anand Subramanian created METRON-1078:
-
Summary: Metron Indexing fails to stop during "Stop All Services"
Key: METRON-1078
URL: https://issues.apache.org/jira/browse/METRON-1078
Project: Metron
Issue Type: Bug
Affects Versions: 0.4.1
Reporter: Anand Subramanian
Fix For: Next + 1
During the kerberization of metron stack, it is observed that the Metron
Indexing service is unable to stop just after the Kerberos Service checks are
completed.
The error in the ambari wizard shows the following:
{code}
raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
resource_management.core.exceptions.ExecutionFailed: Execution of 'curl
--max-time 3 http://metron-8.openstacklocal:8744/api/v1/topology/summary
1>/tmp/tmpRifTJh 2>/tmp/tmpM4Km1g' returned 7. % Total% Received % Xferd
Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
0 00 00 0 0 0 --:--:-- --:--:-- --:--:--
0curl: (7) Failed connect to metron-8.openstacklocal:8744; Connection refused
{code}
This is possibly because the storm service could have possibly shut down before
the metron indexing service; hence the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-895) Ambari "Metron Enrichment Start" Fails - The TGT found is not renewable
[
https://issues.apache.org/jira/browse/METRON-895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064980#comment-16064980
]
Anand Subramanian commented on METRON-895:
--
Marking bug as resolved, since the above comment from Nick works, and it is
already in place.
> Ambari "Metron Enrichment Start" Fails - The TGT found is not renewable
> ---
>
> Key: METRON-895
> URL: https://issues.apache.org/jira/browse/METRON-895
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>
> After Kerberizing a cluster in Ambari with Metron already installed, I am
> unable to launch the enrichment topology. It complains that the "The TGT
> found is not renewable".
> I am running on CentOS 7 and is likely specific to this version of CentOS.
> {code}
> [root@y113 ~]# cat /etc/centos-release
> CentOS Linux release 7.2.1511 (Core)
> {code}
> {code}
> Traceback (most recent call last):
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.4.0/package/scripts/enrichment_master.py",
> line 113, in
> Enrichment().execute()
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 280, in execute
> method(env)
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.4.0/package/scripts/enrichment_master.py",
> line 74, in start
> commands.start_enrichment_topology()
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.4.0/package/scripts/enrichment_commands.py",
> line 146, in start_enrichment_topology
> user=self.__params.metron_user)
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py",
> line 155, in __init__
> self.env.run()
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 160, in run
> self.run_action(resource, action)
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 124, in run_action
> provider_action()
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
> line 273, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 293, in _call
> raise ExecutionFailed(err_msg, code, out, err)
> resource_management.core.exceptions.ExecutionFailed: Execution of
> '/usr/metron/0.4.0/bin/start_enrichment_topology.sh
> -s enrichment -z
> y113.l42scl.hortonworks.com:2181,y114.l42scl.hortonworks.com:2181,y115.l42scl.hortonworks.com:2181'
> returned 1. Running: /usr/jdk64/jdk1.8.0_77/bin/java -server -Ddaemon.name=
> -Dstorm.options= -Dstorm.home=/usr/hdp/2.5.3.0-37/storm
> -Dstorm.log.dir=/var/log/storm
> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file=
> -cp
> /usr/hdp/2.5.3.0-37/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.5.3.0-37/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/storm-rename-hack-1.0.1.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/asm-5.0.3.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.5.3.0-37/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.5.3.0-37/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.5.3.0-37/storm/lib/zookeeper.jar:/usr/hdp/2.5.3.0-37/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.5.3.0-37/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.5.3.0-37/storm/lib/storm-core-1.0.1.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/objenesis-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/ambari-metrics-storm-sink.jar
> org.apache.storm.daemon.ClientJarTransformerRunner
> org.apache.storm.hack.StormShadeTransformer
> /usr/metron/0.4.0/lib/metron-enrichment-0.4.0-uber.jar
> /tmp/0bea323c2aba11e781e900351a9eb24a.jar
> Running: /usr/jdk64/jdk1.8.0_77/bin/java -client -Ddaemon.name=
> -Dstorm.options= -Dstorm.home=/usr/hdp/2.5.3.0-37/storm
> -Dstorm.log.dir=/var/log/storm
> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file=
> -cp
>
[jira] [Created] (METRON-949) Change from error to warning message for Metron REST in Ambari service_advisor
Anand Subramanian created METRON-949: Summary: Change from error to warning message for Metron REST in Ambari service_advisor Key: METRON-949 URL: https://issues.apache.org/jira/browse/METRON-949 Project: Metron Issue Type: Bug Reporter: Anand Subramanian Assignee: Anand Subramanian Priority: Minor Attachments: Metron-REST-service_check.png In the "Assign Masters" page of the Ambari deployment wizard, there is a check for the Metron REST host to have storm supervisor installed. This is currently displayed as an Error message However in the flow of the wizard, the actual storm supervisor selection can only be done in the subsequent screen. So, this message should be changed into a Warning instead (exclamation inside a triangle). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-897) Failed to Start Elasticsearch Deployed with MPack - SettingsException
[
https://issues.apache.org/jira/browse/METRON-897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989678#comment-15989678
]
Anand Subramanian commented on METRON-897:
--
Hi [~nickwallen], I did the change on the nodes themselves, not through Ambari.
> Failed to Start Elasticsearch Deployed with MPack - SettingsException
> -
>
> Key: METRON-897
> URL: https://issues.apache.org/jira/browse/METRON-897
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>
> {code}
> SettingsException[Failed to load settings from [elasticsearch.yml]]
> {code}
> When deploying Elasticsearch with the Ambari MPack, it fails to start. This
> is in an environment with 1 master node and 2 data nodes all running on 3
> separate hosts. The exception is this.
> {code}
> Apr 26 16:45:41 y113 systemd: Starting Elasticsearch...
> Apr 26 16:45:41 y113 systemd: Started Elasticsearch.
> Apr 26 16:45:41 y113 elasticsearch: Exception in thread "main"
> SettingsException[Failed to load settings from [elasticsearch.yml]]; nested:
> ParserException[while parsing a block mapping
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 2, column 1:
> Apr 26 16:45:41 y113 elasticsearch: cluster:
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: expected , but found FlowEntry
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 67, column 26:
> Apr 26 16:45:41 y113 elasticsearch: network.host: "_lo:ipv4_","_eth0:ipv4_"
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: ];
> Apr 26 16:45:41 y113 elasticsearch: Likely root cause: while parsing a block
> mapping
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 2, column 1:
> Apr 26 16:45:41 y113 elasticsearch: cluster:
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: expected , but found FlowEntry
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 67, column 26:
> Apr 26 16:45:41 y113 elasticsearch: network.host: "_lo:ipv4_","_eth0:ipv4_"
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: at
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:570)
> Apr 26 16:45:41 y113 elasticsearch: at
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)
> Apr 26 16:45:41 y113 elasticsearch: at
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:168)
> Apr 26 16:45:41 y113 elasticsearch: at
> com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:342)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:53)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.serializeObject(XContentSettingsLoader.java:99)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:67)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:45)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.common.settings.loader.YamlSettingsLoader.load(YamlSettingsLoader.java:46)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1080)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1067)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:88)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
> Apr 26 16:45:41 y113 elasticsearch: Refer to the log for complete error
> details.
> Apr 26 16:45:41 y113 systemd: elasticsearch.service: main process exited,
> code=exited, status=1/FAILURE
> Apr 26 16:45:41 y113 systemd: Unit elasticsearch.service entered failed state.
> Apr 26 16:45:41 y113 systemd: elasticsearch.service failed.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (METRON-897) Failed to Start Elasticsearch Deployed with MPack - SettingsException
[
https://issues.apache.org/jira/browse/METRON-897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988571#comment-15988571
]
Anand Subramanian edited comment on METRON-897 at 4/28/17 10:08 AM:
I faced the same issue on an openstack deployment. In my case, adding the
square brackets did not help on the ES master node, however adding the IP
address of the node running ES master helped to bring up the service.
Adding square brackets worked on the data nodes though.
was (Author: anandsubbu):
I faced the same issue on an openstack deployment. In my case, adding the
square brackets did not help, however adding the IP address of the node running
ES master helped to bring up the service.
> Failed to Start Elasticsearch Deployed with MPack - SettingsException
> -
>
> Key: METRON-897
> URL: https://issues.apache.org/jira/browse/METRON-897
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>
> {code}
> SettingsException[Failed to load settings from [elasticsearch.yml]]
> {code}
> When deploying Elasticsearch with the Ambari MPack, it fails to start. This
> is in an environment with 1 master node and 2 data nodes all running on 3
> separate hosts. The exception is this.
> {code}
> Apr 26 16:45:41 y113 systemd: Starting Elasticsearch...
> Apr 26 16:45:41 y113 systemd: Started Elasticsearch.
> Apr 26 16:45:41 y113 elasticsearch: Exception in thread "main"
> SettingsException[Failed to load settings from [elasticsearch.yml]]; nested:
> ParserException[while parsing a block mapping
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 2, column 1:
> Apr 26 16:45:41 y113 elasticsearch: cluster:
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: expected , but found FlowEntry
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 67, column 26:
> Apr 26 16:45:41 y113 elasticsearch: network.host: "_lo:ipv4_","_eth0:ipv4_"
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: ];
> Apr 26 16:45:41 y113 elasticsearch: Likely root cause: while parsing a block
> mapping
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 2, column 1:
> Apr 26 16:45:41 y113 elasticsearch: cluster:
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: expected , but found FlowEntry
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 67, column 26:
> Apr 26 16:45:41 y113 elasticsearch: network.host: "_lo:ipv4_","_eth0:ipv4_"
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: at
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:570)
> Apr 26 16:45:41 y113 elasticsearch: at
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)
> Apr 26 16:45:41 y113 elasticsearch: at
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:168)
> Apr 26 16:45:41 y113 elasticsearch: at
> com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:342)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:53)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.serializeObject(XContentSettingsLoader.java:99)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:67)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:45)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.common.settings.loader.YamlSettingsLoader.load(YamlSettingsLoader.java:46)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1080)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1067)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:88)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
> Apr 26 16:45:41 y113 elasticsearch: Refer to the log for complete error
> details.
> Apr 26 16:45:41 y113 systemd: elasticsearch.service: main process exited,
> code=exited, status=1/FAILURE
> Apr 26 16:45:41 y113 systemd: Unit
