Re: SysLog using CEF Parser (RSysLogs)

[Farrukh Naveed Anjum]

Any suggestion how to fix that ?

On Mon, Jan 22, 2018 at 9:01 PM, Farrukh Naveed Anjum <
anjum.farr...@gmail.com> wrote:

> Hi Simon,
>
> Thanks for replying yes, these are indexing bolt errors. I am basically
> trying to forward RSyslog via Nifi. It comes down all the way till indexing
> bolts causes error.
>
> My purpose of using Generic CEF Parser is so that it accumolate SysLog ? I
> did not give him any format, just created a CEF Parsers in Metron
> Management UI. Do I need to give some kind of pattern too ? Or it can
> figure out default syslog pattern ? Kindly guide
>
> By the way following is the indexing bolt error
>
>
>   at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) 
> [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> 2018-01-16 02:34:16.543 o.a.s.d.executor [ERROR]
> java.lang.Exception: WARNING: Default and (likely) unoptimized writer config 
> used for hdfs writer and sensor profiler
>   at 
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:234)
>  [stormjar.jar:?]
>   at 
> org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) 
> [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> 2018-01-16 02:34:16.543 o.a.s.d.executor [ERROR]
> java.lang.Exception: WARNING: Default and (likely) unoptimized writer config 
> used for elasticsearch writer and sensor profiler
>   at 
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:234)
>  [stormjar.jar:?]
>   at 
> org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) 
> [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> 2018-01-16 02:34:16.547 o.a.s.d.executor [ERROR]
> java.lang.Exception: WARNING: Default and (likely) unoptimized writer config 
> used for hdfs writer and sensor profiler
>   at 
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:234)
>  [stormjar.jar:?]
>   at 
> org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.j

Re: SysLog using CEF Parser (RSysLogs)

[Farrukh Naveed Anjum]

Hi Simon,

Thanks for replying yes, these are indexing bolt errors. I am basically
trying to forward RSyslog via Nifi. It comes down all the way till indexing
bolts causes error.

My purpose of using Generic CEF Parser is so that it accumolate SysLog ? I
did not give him any format, just created a CEF Parsers in Metron
Management UI. Do I need to give some kind of pattern too ? Or it can
figure out default syslog pattern ? Kindly guide

By the way following is the indexing bolt error


at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
2018-01-16 02:34:16.543 o.a.s.d.executor [ERROR]
java.lang.Exception: WARNING: Default and (likely) unoptimized writer
config used for hdfs writer and sensor profiler
at 
org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:234)
[stormjar.jar:?]
at 
org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at 
org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at 
org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at 
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at 
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at 
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at 
org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
2018-01-16 02:34:16.543 o.a.s.d.executor [ERROR]
java.lang.Exception: WARNING: Default and (likely) unoptimized writer
config used for elasticsearch writer and sensor profiler
at 
org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:234)
[stormjar.jar:?]
at 
org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at 
org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at 
org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at 
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at 
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at 
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at 
org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
2018-01-16 02:34:16.547 o.a.s.d.executor [ERROR]
java.lang.Exception: WARNING: Default and (likely) unoptimized writer
config used for hdfs writer and sensor profiler
at 
org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:234)
[stormjar.jar:?]
at 
org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at 
org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at 
org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at 
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at 
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at 
org.apache.storm.disruptor$consume_batch_when_availabl

Re: SysLog using CEF Parser (RSysLogs)

[Otto Fowler]

If it reaches the Indexing topology it is not a Parser problem, in almost
all cases.



On January 22, 2018 at 03:24:35, Farrukh Naveed Anjum (
anjum.farr...@gmail.com) wrote:

Yes its Strom Indexing Bolt that is halting it. Any one working on CEF
Parser (Can Syslog work with it like RSyslog). We are stuck at that point.

Please see the above error and suggest

On Mon, Jan 22, 2018 at 1:10 PM, Gaurav Bapat  wrote:

> Hi,
>
> Even I am stuck with the same, and dont know how to solve the issue.
>
> Looks like this is a parsing error
>
> On 22 January 2018 at 13:00, Farrukh Naveed Anjum  > wrote:
>
>> Hi,
>>
>> I am trying to Ingest syslog using CEF Parser it is not creating any
>> Elastic Search Index based on.
>>
>> Any suggestion how can I achieve it ?
>>
>>
>>
>>
>> --
>> With Regards
>> Farrukh Naveed Anjum
>>
>
>


--
With Regards
Farrukh Naveed Anjum

Re: SysLog using CEF Parser (RSysLogs)

[Simon Elliston Ball]

Are there any errors in the logs for the indexing bolt? I would expect the 
errors are probably at the elastic ingest point, and probably caused by an 
incorrect elastic template for the CEF data. 

Simon

> On 22 Jan 2018, at 08:24, Farrukh Naveed Anjum  
> wrote:
> 
> Yes its Strom Indexing Bolt that is halting it. Any one working on CEF Parser 
> (Can Syslog work with it like RSyslog). We are stuck at that point.
> 
> Please see the above error and suggest
> 
> On Mon, Jan 22, 2018 at 1:10 PM, Gaurav Bapat  > wrote:
> Hi,
> 
> Even I am stuck with the same, and dont know how to solve the issue.
> 
> Looks like this is a parsing error
> 
> On 22 January 2018 at 13:00, Farrukh Naveed Anjum  > wrote:
> Hi,
> 
> I am trying to Ingest syslog using CEF Parser it is not creating any Elastic 
> Search Index based on. 
> 
> Any suggestion how can I achieve it ?
> 
> 
> 
> 
> -- 
> With Regards
> Farrukh Naveed Anjum
> 
> 
> 
> 
> -- 
> With Regards
> Farrukh Naveed Anjum

Re: SysLog using CEF Parser (RSysLogs)

[Farrukh Naveed Anjum]

Yes its Strom Indexing Bolt that is halting it. Any one working on CEF
Parser (Can Syslog work with it like RSyslog). We are stuck at that point.

Please see the above error and suggest

On Mon, Jan 22, 2018 at 1:10 PM, Gaurav Bapat  wrote:

> Hi,
>
> Even I am stuck with the same, and dont know how to solve the issue.
>
> Looks like this is a parsing error
>
> On 22 January 2018 at 13:00, Farrukh Naveed Anjum  > wrote:
>
>> Hi,
>>
>> I am trying to Ingest syslog using CEF Parser it is not creating any
>> Elastic Search Index based on.
>>
>> Any suggestion how can I achieve it ?
>>
>>
>>
>>
>> --
>> With Regards
>> Farrukh Naveed Anjum
>>
>
>


-- 
With Regards
Farrukh Naveed Anjum

Re: SysLog using CEF Parser (RSysLogs)

[Gaurav Bapat]

Hi,

Even I am stuck with the same, and dont know how to solve the issue.

Looks like this is a parsing error

On 22 January 2018 at 13:00, Farrukh Naveed Anjum 
wrote:

> Hi,
>
> I am trying to Ingest syslog using CEF Parser it is not creating any
> Elastic Search Index based on.
>
> Any suggestion how can I achieve it ?
>
>
>
>
> --
> With Regards
> Farrukh Naveed Anjum
>

SysLog using CEF Parser (RSysLogs)

[Farrukh Naveed Anjum]

Hi,

I am trying to Ingest syslog using CEF Parser it is not creating any
Elastic Search Index based on.

Any suggestion how can I achieve it ?




-- 
With Regards
Farrukh Naveed Anjum