You need to first make sure delete.topic.enable is set to true. Make sure
there is no producer and consumer registered to the topic. (Stop all storm
topologies if you can and make sure all of them are stopped entirely.)
Then, you can use kafka topic cli to delete the topic by using --delete
[topic_
Hi All,
The recent version of Metron ships with unified enrichment topology. It
looks like this topology does come with the detailed timestamp fields at it
was covered in the previous enrichment architecture. What are the timestamp
fields available with the new topology? Given we have been using s
Hi All,
I was wondering if it is possible to filter out some fields from landing
indexer only (Elasticsearch) from Metron side. Either in enrichment or
indexing topology. I want to store all the parsed/enriched fields in HDFS,
but I want to filter some of them for Elasticsearch. Cleary, we can de
Hi Jack,
Good to see you here. Would it help if you can introduce a signature
for every event and then try to filter based on the signature? a duplicate
message might be related to the source or at least once guarantee of Storm.
Cheers,
Ali
On Fri, Feb 23, 2018 at 3:14 PM, Jack Burgess
wrote:
Any example would be appreciated.
On 9 Feb. 2018 15:37, "Ali Nazemian" wrote:
> Match is not available in our version. What about MAP_GET? How can I use
> it for this matter? I couldn't understand what the solution is.
>
> On Fri, Feb 9, 2018 at 3:02 PM,
ement in there I
> expect. See the match statement at https://github.com/apache/
> metron/blob/master/metron-stellar/stellar-common/README.md under core
> functions (it’s relatively new)
>
> Simon
>
> Sent from my iPhone
>
> On 9 Feb 2018, at 03:55, Ali Nazemian wrote:
&
Hi All,
I was wondering how we can address if statement in the config section to
have a different mapping in certain conditions. The following syntax is not
acceptable.
{
"parserClassName": "org.apache.metron.parsers.asa.BasicAsaParser",
"filterClassName": null,
"sensorTopic": "test-asa",
"writer
metron.enrichment.stellar.ObjectGet
>> that was purpose-built to retrieve files from HDFS. If you wanted to
>> retrieve a configuration from HDFS that would be a good example (if you
>> can't just use that functions directly).
>>
>> On Fri, Feb 2, 2018 at 8:50 AM A
t; global config option.
>
> Simon
>
> On 2 Feb 2018, at 13:42, Ali Nazemian wrote:
>
> Does it mean every time the function gets called it will load the config,
> but if I use the global one it will only read it one time and it will be
> available in memory?
>
> On 2 F
use the Stellar version, and in fact that’s the
> general direction the project is heading. We haven’t quite deprecated the
> plain HBase Bolt… but Stellar is definitely the preferred option.
>
> Simon
>
> > On 2 Feb 2018, at 07:10, Ali Nazemian wrote:
> >
> > Hi All,
&g
, since most instances of
> stellar apply global config to their context.
>
> Simon
>
>
> On 2 Feb 2018, at 07:14, Ali Nazemian wrote:
>
> Will be any problem if the Stellar function we want to implement need to
> load an external config file?
>
> Cheers,
> A
Will be any problem if the Stellar function we want to implement need to
load an external config file?
Cheers,
Ali
On Thu, Jan 18, 2018 at 4:58 PM, Ali Nazemian wrote:
> Thanks, All.
>
> Yes, Nick. It is highly related to our use case and the way that we are
> going to enrich
Hi All,
Is there any performance difference between HBase enrichment and Stellar
enrichment? We have an HBase enrichment that we need to have a customised
key for it. HBase enrichment doesn't give us the full flexibility of using
any logic for a Key generation, so I was wondering whether there wil
t;
>
> On Wed, Jan 17, 2018 at 7:38 AM Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
> Have you looked at the recent TLSH functions in Stellar? We already have
> that for similarity preserving hashes.
>
>
>
> Simon
>
>
>
>
> On 1
ed, Jan 17, 2018 at 6:29 AM Ali Nazemian
> wrote:
>
>> Thanks, Simon. We have already got a script to deal with classpath
>> management for the parsers. We should be able to use it for this extension
>> as well.
>>
>> Yeah, I agree. It will be much easier to defi
e dev list is probably the notion of
> defining stellar functions in stellar, which would be a much simpler
> solution than custom java functions if you can already express you logic in
> stellar.
>
> Simon
>
>
> On 17 Jan 2018, at 10:37, Ali Nazemian wrote:
>
> Hi Simon,
>
r example. Is that what
> you’re looking for? Maybe some sort of syntax to create a named stellar
> function similar to the way we create lambdas?
>
> Simon
>
> > On 17 Jan 2018, at 07:25, Ali Nazemian wrote:
> >
> > Hi all,
> >
> > Is there any way that
Hi all,
Is there any way that we can define a function that can be used rather than
duplicating a logic multiple times?
Cheers,
Ali
Hi,
We have a very bizarre situation with one of our platforms. Our problem is
we have about 6-7 mins extra latency on our platform. We have noticed there
is about 60k-70k total lag on the indexing consumer of the indexing topic.
This lag neither decrease nor increase! The bizarre situation is thi
>> I am not sure why the stream is closed. But, I have opened
>> https://issues.apache.org/jira/browse/METRON-1153, because we should
>> verify the stream before attempting to write.
>>
>>
>> On September 3, 2017 at 21:28:16, Ali Nazemian (alinazem...@gmail.com)
&
, I have opened
> https://issues.apache.org/jira/browse/METRON-1153, because we should
> verify the stream before attempting to write.
>
>
> On September 3, 2017 at 21:28:16, Ali Nazemian (alinazem...@gmail.com)
> wrote:
>
>
> Hi all,
>
> We have run into an issue o
Hi all,
We have run into an issue on Indexing topology on the HDFS bolt recently.
We are using HDFS TDE for encryption at rest and it is working properly for
2-3 days. After that, we can see the following exception frequently on HDFS
writer bolt and the throughput of this topology drops significan
cal file on the Metron master node, as long as that file
> exists prior to Ambari's attempt to use it.
>
> Let me know if that solves the problem; I haven't taken a look at that
> stuff in a little bit, so I may have to dig a bit deeper if that doesn't
> resolve it.
>
Hi,
Recently we have blocked internet connection to one of our platforms. After
we had restarted Enrichment topology, we found out that topology cannot
start anymore and it keeps throwing the following exception.
2017-07-28 04:41:38.816 o.a.c.f.r.c.TreeCache [ERROR]
java.lang.IllegalStateExcepti
ly going to kill your disks at any reasonable
> scale.
>
> Simon
>
> > On 14 Jul 2017, at 10:31, Ali Nazemian wrote:
> >
> > Hi,
> >
> > I am investigating different tuning aspects, and I was wondering how I
> can change the policy of Elasticsearch index
Hi,
I am investigating different tuning aspects, and I was wondering how I can
change the policy of Elasticsearch indexing. Currently, as a default
behaviour, events are stored in separate indices hourly. How can I change
this behaviour? Is this a hard-coded design or I can change it through
confi
ike the score aggregated?
>
> On Thu, Jun 22, 2017 at 8:07 PM, Ali Nazemian
> wrote:
>
>> Thanks, Casey and Nick. Is there any way that we can somehow overcome
>> this requirement with the current features? Exclude MAAS.
>>
>> On Thu, Jun 22, 2017 at 11:42 PM,
Stella wrote:
>
>> That's correct that it's the last step. Honestly, the threat triage
>> functions were added prior to Stellar really being a thing. We should
>> allow arbitrary stellar statements in there rather than a fixed approach,
>> so it's
Hi all,
I know there are four different Treat Triage aggregation functions we can
use for the case of triggering multiple rules. These functions are "max',
"min", "mean", "positive mean". I was wondering whether there is any way I
can implement the following logic with the Treat Triage functions f
be nice if our enrichment cache mechanism reported hit/miss
>> stats or something.
>>
>>
>> On June 19, 2017 at 09:58:25, Ali Nazemian (alinazem...@gmail.com) wrote:
>>
>> I have already increased the cache value. However, clearly, the HBase
>> enrichment i
same clientside caching as
>>> the Hbase bolt?
>>>
>>> Simon
>>>
>>> On 19 Jun 2017, at 06:21, Casey Stella wrote:
>>>
>>> In order to do that, the easiest thing to do is to create a stellar
>>> function to load and do in-me
Hi all,
We are using Metron HBase enrichment for a few use cases, but we have
noticed the achievable throughput is not very great. I was wondering
whether there is a way to load the external enrichment data in-memory and
use it with normal Stellar enrichments. In our use cases, the number of
rows
; Hortonworks for HDP. You probably need to look mostly at the ports
> specified in your config files.
>
> I would suggest that you treat all HDP nodes as able to talk to each other
> across all ports - but limit anything which talks to those nodes. That is a
> lot easier.
o those nodes. That is a
> lot easier.
>
>
> On 30 May 2017 at 10:49, Ali Nazemian wrote:
>
>> Hi all,
>>
>> For deploying Metron in production, we need to specify all of the port
>> and protocols connectivities. I was wondering how Metron components
it before giving up.
>
> I would be happy to share what I have, or help maintain something more
> complete if it exists somewhere that I'm not aware of.
>
> Jon
>
> On Tue, May 30, 2017, 5:49 AM Ali Nazemian wrote:
>
>> Hi all,
>>
>> For deploying Metron
Hi all,
For deploying Metron in production, we need to specify all of the port and
protocols connectivities. I was wondering how Metron components connected
to each other. Is there any document available regarding the ports and
connectivities of Metron components?
Regards,
Ali
"tenant_name+device_type+default_device".
>
> Yes, you can. You've got if/else, JOIN, IS_EMPTY, and others that should
> make implementing this logic pretty easy.
>
>
>
>
> On Tue, May 23, 2017 at 10:34 PM, Ali Nazemian
> wrote:
>
>> Hi,
>>
>
Hi,
I was wondering how I can manage Stellar syntax to be aligned with the
following structure for the HBase enrichment:
HBase_row_key: tenant_name+device_type+device_name
At the high-level, I need to create a separate field via a post-parse
Stellar function to be a concatenation of tenan_name,
ion. There is a PR out for this currently:
> https://github.com/apache/metron/pull/584
>
> Casey
>
> On Tue, May 16, 2017 at 4:26 AM, Ali Nazemian
> wrote:
>
>> I am still facing this issue and couldn't manage to fix it. I would be
>> really grateful If somebody c
I am still facing this issue and couldn't manage to fix it. I would be
really grateful If somebody can help me.
Thanks,
Ali
On Sun, May 14, 2017 at 1:58 PM, Ali Nazemian wrote:
> I was wrong. I think I couldn't increase the timeout value for Kafka spout
> properly. Therefore,
e I didn't have
this issue with the previous version?
On Sun, May 14, 2017 at 3:00 AM, Ali Nazemian wrote:
> Hi,
>
> I have installed the new version of HCP recently. I can see that the
> following error has appeared in Storm UI at Kafka spout sectio
Hi,
I have installed the new version of HCP recently. I can see that the
following error has appeared in Storm UI at Kafka spout section related to
Parser topologies:
org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be
completed since the group has already rebalanced and ass
through disabling Storm reliability!!
Another wired fact is I have this problem only for the enrichments and
indexing topologies. All of the parsers are fine!
On Sun, Apr 23, 2017 at 12:39 AM, Ali Nazemian
wrote:
> In response to your question for decreasing the value of spout pending, no
>
In response to your question for decreasing the value of spout pending, no
even with the value of 10 failure ratio was the same. However, throughput
dropped significantly.
On Sun, Apr 23, 2017 at 12:27 AM, Ali Nazemian
wrote:
> I have noticed if I decrease the parallelism for spouts the fail
spout pending config lower do you get to a point with no
> errors (at obvious consequences to throughput)? Also how many ackers are
> you running?
>
> On Sat, Apr 22, 2017 at 00:50 Ali Nazemian wrote:
>
>> I have disabled the reliability retry by setting the number of
>>
that.
On Sat, Apr 22, 2017 at 2:36 PM, Ali Nazemian wrote:
> Is the following fact rings any bell?
>
> There is no failure at the bolt level acknowledgement, but from the
> topology status, the rate of failure is very high! This is the same
> scenario for both indexing and enrich
Is the following fact rings any bell?
There is no failure at the bolt level acknowledgement, but from the
topology status, the rate of failure is very high! This is the same
scenario for both indexing and enrichment topologies.
On Sat, Apr 22, 2017 at 2:29 PM, Ali Nazemian wrote:
> The va
at's your storm configuration for topology.max.spout.pending? If it's
> not set, then try setting it to 1000 and bouncing the topologies.
>
> On Fri, Apr 21, 2017 at 12:54 PM, Ali Nazemian
> wrote:
>
>> No, nothing ...
>>
>> On Sat, Apr 22, 2017 at 2:46 AM, Casey Stella wrot
No, nothing ...
On Sat, Apr 22, 2017 at 2:46 AM, Casey Stella wrote:
> Anything going on in the kafka broker logs?
>
> On Fri, Apr 21, 2017 at 12:24 PM, Ali Nazemian
> wrote:
>
>> Although this is a test platform with a way less spec than production, it
>> should
atencies are pretty high. I think what's happening is
> that the tuples aren't being acked fast enough and are timing out. How
> taxed is your ES box? Can you drop the batch size down to maybe 100 and
> see what happens?
>
> On Fri, Apr 21, 2017 at 12:05 PM, Ali Nazemi
0:53 AM, Casey Stella
>> wrote:
>>
>>> Could I see a little more of that screen? Specifically what the bolts
>>> look like.
>>>
>>> On Fri, Apr 21, 2017 at 11:51 AM, Ali Nazemian
>>> wrote:
>>>
>>>> Please find the st
Please find the storm-UI screenshot as follows.
http://imgur.com/FhIrGFd
On Sat, Apr 22, 2017 at 1:41 AM, Ali Nazemian wrote:
> Hi Casey,
>
> - topology.message.timeout: It was 30s at first. I have increased it to
> 300s, no changes!
> - It is a very basic geo-enrichment and
errors in the
>> logs. Would you mind sending over a screenshot of the indexing topology
>> from the storm UI? You might not be able to paste the image on the mailing
>> list, so maybe an imgur link would be in order.
>>
>> Thanks,
>>
>> Casey
>>
>> O
are
> several catch blocks across the different topologies that transform errors
> into json objects and forward them on to the indexing topology. If you're
> not seeing anything in the worker logs it's likely the errors were captured
> there instead.
>
> Ryan
>
> On Fri,
No everything is fine at the log level. Also, when I checked resource
consumption at the workers, there had been plenty resources still available!
On Fri, Apr 21, 2017 at 10:04 PM, Casey Stella wrote:
> Seeing anything in the storm logs for the workers?
>
> On Fri, Apr 21, 2017 at
Hi all,
After I tried to tune the Metron performance I have noticed the rate of
failure for the indexing/enrichment topologies are very high (about 95%).
However, I can see the messages in Elasticsearch. I have tried to increase
the timeout value for the acknowledgement. It didn't fix the problem.
md#notes-on-performance-tuning>
>
> Jon
>
> On Thu, Apr 20, 2017 at 8:45 AM Ali Nazemian
> wrote:
>
>> Hi all,
>>
>> I was wondering what the best practice would be in terms of defining a
>> right value for the number of workers and executors as well as
Hi all,
I was wondering what the best practice would be in terms of defining a
right value for the number of workers and executors as well as right value
for spout and bolt parallelisation? What about the number of partitions for
"indexing", "enrichments" and device parsers Kafka topics?
I have s
59 matches
Mail list logo
