You need to first make sure delete.topic.enable is set to true. Make sure
there is no producer and consumer registered to the topic. (Stop all storm
topologies if you can and make sure all of them are stopped entirely.)
Then, you can use kafka topic cli to delete the topic by using --delete
Hi All,
The recent version of Metron ships with unified enrichment topology. It
looks like this topology does come with the detailed timestamp fields at it
was covered in the previous enrichment architecture. What are the timestamp
fields available with the new topology? Given we have been using
Hi Jack,
Good to see you here. Would it help if you can introduce a signature
for every event and then try to filter based on the signature? a duplicate
message might be related to the source or at least once guarantee of Storm.
Cheers,
Ali
On Fri, Feb 23, 2018 at 3:14 PM, Jack Burgess
Hi All,
I was wondering how we can address if statement in the config section to
have a different mapping in certain conditions. The following syntax is not
acceptable.
{
"parserClassName": "org.apache.metron.parsers.asa.BasicAsaParser",
"filterClassName": null,
"sensorTopic": "test-asa",
ET function defined in
>> org.apache.metron.enrichment.stellar.ObjectGet
>> that was purpose-built to retrieve files from HDFS. If you wanted to
>> retrieve a configuration from HDFS that would be a good example (if you
>> can't just use that functions directly).
>>
ss, but most likely, yes. Hence
> global config option.
>
> Simon
>
> On 2 Feb 2018, at 13:42, Ali Nazemian <alinazem...@gmail.com> wrote:
>
> Does it mean every time the function gets called it will load the config,
> but if I use the global one it will only read it one ti
ng,
> so you should be able to use the Stellar version, and in fact that’s the
> general direction the project is heading. We haven’t quite deprecated the
> plain HBase Bolt… but Stellar is definitely the preferred option.
>
> Simon
>
> > On 2 Feb 2018, at 07:10, Ali Nazemian &l
al config for example and refer to that, since most instances of
> stellar apply global config to their context.
>
> Simon
>
>
> On 2 Feb 2018, at 07:14, Ali Nazemian <alinazem...@gmail.com> wrote:
>
> Will be any problem if the Stellar function we want to implement need to
Will be any problem if the Stellar function we want to implement need to
load an external config file?
Cheers,
Ali
On Thu, Jan 18, 2018 at 4:58 PM, Ali Nazemian <alinazem...@gmail.com> wrote:
> Thanks, All.
>
> Yes, Nick. It is highly related to our use case and the way that
Hi All,
Is there any performance difference between HBase enrichment and Stellar
enrichment? We have an HBase enrichment that we need to have a customised
key for it. HBase enrichment doesn't give us the full flexibility of using
any logic for a Key generation, so I was wondering whether there
ike this in Stellar with named functions.
>
>
>
> Best
>
>
>
>
>
>
>
>
>
> On Wed, Jan 17, 2018 at 7:38 AM Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
> Have you looked at the recent TLSH functions in Stellar? We already hav
gt;
>
>
>
> On Wed, Jan 17, 2018 at 6:29 AM Ali Nazemian <alinazem...@gmail.com>
> wrote:
>
>> Thanks, Simon. We have already got a script to deal with classpath
>> management for the parsers. We should be able to use it for this extension
>> as well.
>
mple. Is that what
> you’re looking for? Maybe some sort of syntax to create a named stellar
> function similar to the way we create lambdas?
>
> Simon
>
> > On 17 Jan 2018, at 07:25, Ali Nazemian <alinazem...@gmail.com> wrote:
> >
> > Hi all,
> >
>
Hi all,
Is there any way that we can define a function that can be used rather than
duplicating a logic multiple times?
Cheers,
Ali
Hi,
We have a very bizarre situation with one of our platforms. Our problem is
we have about 6-7 mins extra latency on our platform. We have noticed there
is about 60k-70k total lag on the indexing consumer of the indexing topic.
This lag neither decrease nor increase! The bizarre situation is
Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
>> I am not sure why the stream is closed. But, I have opened
>> https://issues.apache.org/jira/browse/METRON-1153, because we should
>> verify the stream before attempting to write.
>>
>>
>> On Sept
sure why the stream is closed. But, I have opened
> https://issues.apache.org/jira/browse/METRON-1153, because we should
> verify the stream before attempting to write.
>
>
> On September 3, 2017 at 21:28:16, Ali Nazemian (alinazem...@gmail.com)
> wrote:
>
>
> Hi all,
>
Hi all,
We have run into an issue on Indexing topology on the HDFS bolt recently.
We are using HDFS TDE for encryption at rest and it is working properly for
2-3 days. After that, we can see the following exception frequently on HDFS
writer bolt and the throughput of this topology drops
Hi,
Recently we have blocked internet connection to one of our platforms. After
we had restarted Enrichment topology, we found out that topology cannot
start anymore and it keeps throwing the following exception.
2017-07-28 04:41:38.816 o.a.c.f.r.c.TreeCache [ERROR]
Hi,
I am investigating different tuning aspects, and I was wondering how I can
change the policy of Elasticsearch indexing. Currently, as a default
behaviour, events are stored in separate indices hourly. How can I change
this behaviour? Is this a hard-coded design or I can change it through
u'd like the score aggregated?
>
> On Thu, Jun 22, 2017 at 8:07 PM, Ali Nazemian <alinazem...@gmail.com>
> wrote:
>
>> Thanks, Casey and Nick. Is there any way that we can somehow overcome
>> this requirement with the current features? Exclude MAAS.
>>
>> On T
Hi all,
I know there are four different Treat Triage aggregation functions we can
use for the case of triggering multiple rules. These functions are "max',
"min", "mean", "positive mean". I was wondering whether there is any way I
can implement the following logic with the Treat Triage functions
chment component could just be optimized.
>> It would be nice if our enrichment cache mechanism reported hit/miss
>> stats or something.
>>
>>
>> On June 19, 2017 at 09:58:25, Ali Nazemian (alinazem...@gmail.com) wrote:
>>
>> I have already increased the ca
kup.
>>> Does the stellar enrichment function not use the same clientside caching as
>>> the Hbase bolt?
>>>
>>> Simon
>>>
>>> On 19 Jun 2017, at 06:21, Casey Stella <ceste...@gmail.com> wrote:
>>>
>>> In order to do that, t
quot; lookup
> for "tenant_name+device_type+default_device".
>
> Yes, you can. You've got if/else, JOIN, IS_EMPTY, and others that should
> make implementing this logic pretty easy.
>
>
>
>
> On Tue, May 23, 2017 at 10:34 PM, Ali Nazemian <alinazem...@gmail.com>
is through disabling Storm reliability!!
Another wired fact is I have this problem only for the enrichments and
indexing topologies. All of the parsers are fine!
On Sun, Apr 23, 2017 at 12:39 AM, Ali Nazemian <alinazem...@gmail.com>
wrote:
> In response to your question for decreasing the value
In response to your question for decreasing the value of spout pending, no
even with the value of 10 failure ratio was the same. However, throughput
dropped significantly.
On Sun, Apr 23, 2017 at 12:27 AM, Ali Nazemian <alinazem...@gmail.com>
wrote:
> I have noticed if I decrease the pa
Please find the storm-UI screenshot as follows.
http://imgur.com/FhIrGFd
On Sat, Apr 22, 2017 at 1:41 AM, Ali Nazemian <alinazem...@gmail.com> wrote:
> Hi Casey,
>
> - topology.message.timeout: It was 30s at first. I have increased it to
> 300s, no changes!
> - I
Hi all,
After I tried to tune the Metron performance I have noticed the rate of
failure for the indexing/enrichment topologies are very high (about 95%).
However, I can see the messages in Elasticsearch. I have tried to increase
the timeout value for the acknowledgement. It didn't fix the
30 matches
Mail list logo
