Anil,
Do you have any examples you can share about the use of profiler jars in your
Java code?
Thanks,
Stéphane
From: Anil Donthireddy [mailto:anil.donthire...@sstech.us]
Sent: Wednesday, April 24, 2019 19:25
To: DAVY Stephane OBS/CSO
Cc: user@metron.apache.org
Subject: RE: Various questions
Hello all,
As we heavily use Elasticsearch in our company, with some support from Elastic
company, I'd like to share with about index and template. Here is the starting
template I use:
{
"": {
"template": "_index_*",
"settings": {
"index": {
"number_of_shards": "1",
Try querying for that record in the REPL with PROFILE_GET and then using
MAP_GET on it. It will likely work as you expect.
value := PROFILE_GET("test", ...)
MAP_GET("sum_rcvd_bytes", value)
On Thu, Apr 25, 2019 at 2:38 AM wrote:
> Hello Nick,
>
>
>
> Thanks for your answer. Well, I don’t kno
I realize that I’ve missed a part of the story regarding shards. The good size
for shards is around 40~50GB. So, if your index grows up to 200 or 300GB, you
of course need to increase the number of shards to come back around this size.
This is also why I’d suggest to have .MM.dd in the “Elas
REST_GET is not included in that version. You will need a more recent
release.
On Wed, Apr 24, 2019 at 4:04 PM Anil Donthireddy
wrote:
> I see the version as “0.5.1.1.6.0.0” in the ambari components list for
> metron.
>
>
>
> *From:* Michael Miklavcic [mailto:michael.miklav...@gmail.com]
> *Se
Well, it seems that I have another issue right now:
[Stellar]>>> PROFILE_GET('simple_count','22.0.35.5', PROFILE_FIXED(30,
'MINUTES'))
[!] Unable to parse: PROFILE_GET('simple_count','22.0.35.5', PROFILE_FIXED(30,
'MINUTES')) due to: Unable to access table: profiler
It looks like a permission i
Hello,
I'm trying to load some JSON data which has the following structure (this is a
sample):
{
"_index": "indexing",
"_type": "Event",
"_id": "AWAkTAefYn0uCUpkHmCy",
"_score": 1,
"_source": {
"dst": "127.0.0.1",
"devTimeEpoch": "151243734",
"dstPort": "0",
"srcPor
Raw message in this case assumes that the raw message is a String embedded
in the json field that you supply, not a nested json object, so it is
looking for
“_source” : “some other embedded string of some format like syslog in json”
There are other message strategies, but I’m not sure they would
So, the enveloped message doesn’t support getting an already parsed json
object from the enveloped json, we would have to do some work to support
this, Even if we _could_ wrangle it in there now, from what I can see we
would still have to serialize to bytes to pass to the actual parser and
that
Seems like this would a good additional strategy, something like
ENVELOPE_PARSED? Any thoughts on a good name?
On Thu, 25 Apr 2019 at 16:20, Otto Fowler wrote:
> So, the enveloped message doesn’t support getting an already parsed json
> object from the enveloped json, we would have to do some w
I’m not sure about the name, I’m more thinking about the case.
I’m not sure this is an enveloped issue, or a new feature for the json map
parser ( or if you could do it with the jsonMap parser and JSONPath )
On April 25, 2019 at 11:23:25, Simon Elliston Ball (
si...@simonellistonball.com) wrote:
(1) Did you launch the REPL with the -z option?
(2) What user are you running the REPL as?
(3) Can you scan the table using the 'hbase shell'? Something like this...
echo "scan 'profiler'" | hbase shell
(4) Can you show the full session from launching the REPL, running the
PROFILE_GET, and
Also, our support for nested, unflattened json isn’t great to begin with.
Stephane, can you state your use case?
Do you want to get _source only to transform it? or do you want to use
source as the message and discard the top level fields? other?
On April 25, 2019 at 11:31:36, Otto Fowler (
> Otto: I’m not sure this is an enveloped issue, or a new feature for the
json map parser
This is not an issue with JSONMapParser. This is an issue with the
"enveloping" mechanism, prior to when the JSONMapParser gets the message.
The entire message has been parsed as a JSON object including the
Hello,
Actually, I want to keep only the _source part. The full story is that these
data are a dump from another Elasticsearch cluster. After reading this:
https://metron.apache.org/current-book/metron-platform/metron-parsers/ParserChaining.html,
I thought I could do the same with JSON. In this
> Stephane: How can I debug this?
We created the PARSER* functions to help debug issues like this.
Unfortunately, it does not work in this case because the bug is in the
Enveloping, not in the Parser itself. This is gap in the PARSER_*
functionality that I think we should address to make debuggin
OK, I finally found the problem when pasting the whole error stack in the mail:
Caused by: java.lang.RuntimeException: Unexpected version format: 11.0.3
The first java in my path is java 11. When switching to Java 8 it worked
correctly
Stéphane
From: Nick Allen [mailto:n...@nickallen.org]
Sen
The issue I think would be with the transformations not working or being
applicable no?
On April 25, 2019 at 12:19:29, Nick Allen (n...@nickallen.org) wrote:
> Stephane: How can I debug this?
We created the PARSER* functions to help debug issues like this.
Unfortunately, it does not work in thi
Wait, are we sure that's the case? Generally speaking, messages coming
into the parser which contains the envelope strategy has a _source field
which is a string, which this isn't (it's JSON).
For instance, the format expected is:
{
"_index": "indexing",
"_type": "Event",
"_id": "AWAkTAe
On April 25, 2019 at 12:05:45, Nick Allen (n...@nickallen.org) wrote:
> Otto: I’m not sure this is an enveloped issue, or a new feature for the
json map parser
This is not an issue with JSONMapParser. This is an issue with the
"enveloping" mechanism, prior to when the JSONMapParser gets the mess
Alternatively, you might also be able to compile it and bundle it yourself
per the custom Stellar function docs:
1.
https://github.com/apache/metron/blob/master/metron-stellar/stellar-common/3rdPartyStellar.md
(it
appears this link has possibly been orphaned since I don't see it
refer
Thanks for sharing Stephane! Just an fyi, we do also recommend setting the
type mapping for strings like you've shown, as indicated here -
https://github.com/apache/metron/tree/master/metron-platform/metron-elasticsearch#type-mappings
.
On Thu, Apr 25, 2019 at 6:33 AM wrote:
> I realize that I’v
22 matches
Mail list logo
