Re: [heka] heka wedged

[Eric LEMOINE]

> Buffering does help, yes. When buffering is in use, the shared pack from
the global pool is recycled immediately after the message is written to the
disk buffer. Each plugin that uses disk buffering uses its own packs
internally that don't intermingle with the global packs at all.

That makes sense. Thanks for confirming.
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

Re: [heka] heka wedged

[Rob Miller]

On 04/04/2016 05:32 AM, Eric LEMOINE wrote:

On Wed, Mar 30, 2016 at 9:01 PM, Rob Miller  wrote:
> Heka can get wedged when there's a deadlock in the pipeline, usually related
> to pack exhaustion. For example, a filter might block because it's waiting
> for an empty pack with which to inject a new message, but there are no packs
> available because they're all tied up in the input channels for the filter.
> Tweaking the channel and pool sizes is usually helpful here.


Wondering if using buffering on filters could be a solution… I guess
that depends on when packs are recycled when buffering is used.

Buffering does help, yes. When buffering is in use, the shared pack from the 
global pool is recycled immediately after the message is written to the disk 
buffer. Each plugin that uses disk buffering uses its own packs internally that 
don't intermingle with the global packs at all.

-r



___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

Re: [heka] heka wedged

[Eric LEMOINE]

On Wed, Mar 30, 2016 at 9:01 PM, Rob Miller  wrote:
> Heka can get wedged when there's a deadlock in the pipeline, usually related
> to pack exhaustion. For example, a filter might block because it's waiting
> for an empty pack with which to inject a new message, but there are no packs
> available because they're all tied up in the input channels for the filter.
> Tweaking the channel and pool sizes is usually helpful here.


Wondering if using buffering on filters could be a solution… I guess
that depends on when packs are recycled when buffering is used.
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

Re: [heka] heka wedged

[Rob Miller]

The defaults of pool size = 100 and chan size = 30 are good for many cases. If 
you need to increase the pool size, that's fine, but don't make the channel 
size as large. I'd try reducing my channel size to 50.

-r

On 3/30/16 4:27 PM, Ramin Ali Dousti wrote:

Hi Rob, thanks for the response/info.

Yes, as you might remember I implemented your heartbeat suggestion
months ago and am sending heartbeats of a chain of hekad's to influx and
am using kapacitor's deadman watch to catch the lack of heartbeat and
it's working perfectly fine.

The problem I'm trying to solve is to eliminate the manual intervention
of USR1/USR2 exercise; hence my question about any possible reasons for
wedgedness. This is my current channel/pool setting (I know this doesn't
mean much without knowing what is passing through the pipeline):

[hekad]
maxprocs = 4
base_dir = "/local/heka/run"
pid_file = "/var/run/heka-edge/heka-edge.pid"
poolsize = 500
plugin_chansize = 500

Do you have any heuristics about the values I should be using?

Ramin


On Wed, Mar 30, 2016 at 3:01 PM, Rob Miller mailto:rmil...@mozilla.com>> wrote:

Heka can get wedged when there's a deadlock in the pipeline, usually
related to pack exhaustion. For example, a filter might block
because it's waiting for an empty pack with which to inject a new
message, but there are no packs available because they're all tied
up in the input channels for the filter. Tweaking the channel and
pool sizes is usually helpful here.

There are no verbosity levels to Heka's logging, sorry. When Heka is
wedged, you can get a dump to stdout of the current internal state
of the pipeline by sending SIGUSR1 to the Heka process. You can get
the dump, followed by serializing all of the sandboxes and exiting
the process by sending SIGUSR2.

You can monitor for wedgedness by setting up a filter to emit a
message every N seconds, and then setting up an output to catch
those messages. You could write them to a file and monitor that the
file is growing, or send them to a downstream listener that will
notice if the heartbeats stop coming.

-r


On 03/30/2016 11:13 AM, Ramin Ali Dousti wrote:

Hi Timur,

1) wedged:
In my case, my hekad gets wedged for no reason (or I think i cannot
pinpoint as to why)

2) logging
Yes, I could do a LogOutput (which I'm doing when I'm
troubleshooting)
but I was looking for some kind of application logging with
different
severity so that i could up or down the severity and get to see
less or
more log entries about what was going on. I'm not talking about the
actual messages that are passing through the system. For example
when a
TcpInput drops a connection you will see a line indicating that, I'd
like to see a similar thing about "important events" within the
system.
For example "I've just become wedged because ...".

Thanks for the answers though.

Best,
Ramin

On Wed, Mar 30, 2016 at 7:34 AM, Timur Batyrshin
mailto:ert...@gmail.com>
>> wrote:

 Hi Ramin,

 When heka is running it emits HeapAlloc, HeapSys and few other
 metrics which you can graph or alert based on them.
 I've seen heka freezing only when I send TERM signal to it
and some
 Lua plugin fails to stop correctly or something like that.
 May be someone else will give you insights on other conditions.

 For checking how heka is processing I know 3 ways:
 * dashboard

(http://hekad.readthedocs.org/en/v0.10.0/config/outputs/dashboard.html)
 -- it doesn't display individual messages but displays
number of
 messages processed, queues, lags, etc
 * if you send messages over TcpOutput you can enable
buffering and
 use "heka-cat" command to browse/tail protobuf logs in
/var/cache/heka
 * the most usable for me way is creating additional output
like the
 following:

 [debug_encoder]
 type="RstEncoder"

 [LogOutput]
 encoder="debug_encoder"
 message_matcher="TRUE"

 This makes heka dump all messages to stdout. If you store
that in
 logs hope you rotate them based on size reached.


 Best regards,
 Timur

 On Wed, Mar 30, 2016 at 2:22 AM, Ramin Ali Dousti
mailto:dou...@gmail.com>
 >> wrote:

 Hi,

 I have a nagging question. I read in the documentations
and I
 also experienced it myself that hekad might become
"wedged" and
 th

Re: [heka] heka wedged

[Ramin Ali Dousti]

Hi Rob, thanks for the response/info.

Yes, as you might remember I implemented your heartbeat suggestion months
ago and am sending heartbeats of a chain of hekad's to influx and am using
kapacitor's deadman watch to catch the lack of heartbeat and it's working
perfectly fine.

The problem I'm trying to solve is to eliminate the manual intervention of
USR1/USR2 exercise; hence my question about any possible reasons for
wedgedness. This is my current channel/pool setting (I know this doesn't
mean much without knowing what is passing through the pipeline):

[hekad]
maxprocs = 4
base_dir = "/local/heka/run"
pid_file = "/var/run/heka-edge/heka-edge.pid"
poolsize = 500
plugin_chansize = 500

Do you have any heuristics about the values I should be using?

Ramin


On Wed, Mar 30, 2016 at 3:01 PM, Rob Miller  wrote:

> Heka can get wedged when there's a deadlock in the pipeline, usually
> related to pack exhaustion. For example, a filter might block because it's
> waiting for an empty pack with which to inject a new message, but there are
> no packs available because they're all tied up in the input channels for
> the filter. Tweaking the channel and pool sizes is usually helpful here.
>
> There are no verbosity levels to Heka's logging, sorry. When Heka is
> wedged, you can get a dump to stdout of the current internal state of the
> pipeline by sending SIGUSR1 to the Heka process. You can get the dump,
> followed by serializing all of the sandboxes and exiting the process by
> sending SIGUSR2.
>
> You can monitor for wedgedness by setting up a filter to emit a message
> every N seconds, and then setting up an output to catch those messages. You
> could write them to a file and monitor that the file is growing, or send
> them to a downstream listener that will notice if the heartbeats stop
> coming.
>
> -r
>
>
> On 03/30/2016 11:13 AM, Ramin Ali Dousti wrote:
>
>> Hi Timur,
>>
>> 1) wedged:
>> In my case, my hekad gets wedged for no reason (or I think i cannot
>> pinpoint as to why)
>>
>> 2) logging
>> Yes, I could do a LogOutput (which I'm doing when I'm troubleshooting)
>> but I was looking for some kind of application logging with different
>> severity so that i could up or down the severity and get to see less or
>> more log entries about what was going on. I'm not talking about the
>> actual messages that are passing through the system. For example when a
>> TcpInput drops a connection you will see a line indicating that, I'd
>> like to see a similar thing about "important events" within the system.
>> For example "I've just become wedged because ...".
>>
>> Thanks for the answers though.
>>
>> Best,
>> Ramin
>>
>> On Wed, Mar 30, 2016 at 7:34 AM, Timur Batyrshin > > wrote:
>>
>> Hi Ramin,
>>
>> When heka is running it emits HeapAlloc, HeapSys and few other
>> metrics which you can graph or alert based on them.
>> I've seen heka freezing only when I send TERM signal to it and some
>> Lua plugin fails to stop correctly or something like that.
>> May be someone else will give you insights on other conditions.
>>
>> For checking how heka is processing I know 3 ways:
>> * dashboard
>> (
>> http://hekad.readthedocs.org/en/v0.10.0/config/outputs/dashboard.html)
>> -- it doesn't display individual messages but displays number of
>> messages processed, queues, lags, etc
>> * if you send messages over TcpOutput you can enable buffering and
>> use "heka-cat" command to browse/tail protobuf logs in /var/cache/heka
>> * the most usable for me way is creating additional output like the
>> following:
>>
>> [debug_encoder]
>> type="RstEncoder"
>>
>> [LogOutput]
>> encoder="debug_encoder"
>> message_matcher="TRUE"
>>
>> This makes heka dump all messages to stdout. If you store that in
>> logs hope you rotate them based on size reached.
>>
>>
>> Best regards,
>> Timur
>>
>> On Wed, Mar 30, 2016 at 2:22 AM, Ramin Ali Dousti > > wrote:
>>
>> Hi,
>>
>> I have a nagging question. I read in the documentations and I
>> also experienced it myself that hekad might become "wedged" and
>> there would be no activity within the system while the process
>> seems up from the outside. My question is what are the
>> conditions that this might happen? The reason I ask is that my
>> hekad instances might be running for weeks with no problem but
>> they could end up "wedged" for no obvious reason. Knowing what
>> gets them into this mode, might help me prevent that situation.
>>
>> Also, hekad seems a very quiet process log-wise. How can I have
>> it log "important" information about its doing?
>>
>> I really appreciate any insight.
>>
>> --
>> Ramin
>>
>> ___
>> Heka mailing list
>> Heka@mozilla.org 

Re: [heka] heka wedged

[Rob Miller]

Heka can get wedged when there's a deadlock in the pipeline, usually related to 
pack exhaustion. For example, a filter might block because it's waiting for an 
empty pack with which to inject a new message, but there are no packs available 
because they're all tied up in the input channels for the filter. Tweaking the 
channel and pool sizes is usually helpful here.

There are no verbosity levels to Heka's logging, sorry. When Heka is wedged, 
you can get a dump to stdout of the current internal state of the pipeline by 
sending SIGUSR1 to the Heka process. You can get the dump, followed by 
serializing all of the sandboxes and exiting the process by sending SIGUSR2.

You can monitor for wedgedness by setting up a filter to emit a message every N 
seconds, and then setting up an output to catch those messages. You could write 
them to a file and monitor that the file is growing, or send them to a 
downstream listener that will notice if the heartbeats stop coming.

-r


On 03/30/2016 11:13 AM, Ramin Ali Dousti wrote:

Hi Timur,

1) wedged:
In my case, my hekad gets wedged for no reason (or I think i cannot
pinpoint as to why)

2) logging
Yes, I could do a LogOutput (which I'm doing when I'm troubleshooting)
but I was looking for some kind of application logging with different
severity so that i could up or down the severity and get to see less or
more log entries about what was going on. I'm not talking about the
actual messages that are passing through the system. For example when a
TcpInput drops a connection you will see a line indicating that, I'd
like to see a similar thing about "important events" within the system.
For example "I've just become wedged because ...".

Thanks for the answers though.

Best,
Ramin

On Wed, Mar 30, 2016 at 7:34 AM, Timur Batyrshin mailto:ert...@gmail.com>> wrote:

Hi Ramin,

When heka is running it emits HeapAlloc, HeapSys and few other
metrics which you can graph or alert based on them.
I've seen heka freezing only when I send TERM signal to it and some
Lua plugin fails to stop correctly or something like that.
May be someone else will give you insights on other conditions.

For checking how heka is processing I know 3 ways:
* dashboard
(http://hekad.readthedocs.org/en/v0.10.0/config/outputs/dashboard.html)
-- it doesn't display individual messages but displays number of
messages processed, queues, lags, etc
* if you send messages over TcpOutput you can enable buffering and
use "heka-cat" command to browse/tail protobuf logs in /var/cache/heka
* the most usable for me way is creating additional output like the
following:

[debug_encoder]
type="RstEncoder"

[LogOutput]
encoder="debug_encoder"
message_matcher="TRUE"

This makes heka dump all messages to stdout. If you store that in
logs hope you rotate them based on size reached.


Best regards,
Timur

On Wed, Mar 30, 2016 at 2:22 AM, Ramin Ali Dousti mailto:dou...@gmail.com>> wrote:

Hi,

I have a nagging question. I read in the documentations and I
also experienced it myself that hekad might become "wedged" and
there would be no activity within the system while the process
seems up from the outside. My question is what are the
conditions that this might happen? The reason I ask is that my
hekad instances might be running for weeks with no problem but
they could end up "wedged" for no obvious reason. Knowing what
gets them into this mode, might help me prevent that situation.

Also, hekad seems a very quiet process log-wise. How can I have
it log "important" information about its doing?

I really appreciate any insight.

--
Ramin

___
Heka mailing list
Heka@mozilla.org 
https://mail.mozilla.org/listinfo/heka





--
Ramin


___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka



___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

Re: [heka] heka wedged

[Ramin Ali Dousti]

Hi Timur,

1) wedged:
In my case, my hekad gets wedged for no reason (or I think i cannot
pinpoint as to why)

2) logging
Yes, I could do a LogOutput (which I'm doing when I'm troubleshooting) but
I was looking for some kind of application logging with different severity
so that i could up or down the severity and get to see less or more log
entries about what was going on. I'm not talking about the actual messages
that are passing through the system. For example when a TcpInput drops a
connection you will see a line indicating that, I'd like to see a similar
thing about "important events" within the system. For example "I've just
become wedged because ...".

Thanks for the answers though.

Best,
Ramin

On Wed, Mar 30, 2016 at 7:34 AM, Timur Batyrshin  wrote:

> Hi Ramin,
>
> When heka is running it emits HeapAlloc, HeapSys and few other metrics
> which you can graph or alert based on them.
> I've seen heka freezing only when I send TERM signal to it and some Lua
> plugin fails to stop correctly or something like that.
> May be someone else will give you insights on other conditions.
>
> For checking how heka is processing I know 3 ways:
> * dashboard (
> http://hekad.readthedocs.org/en/v0.10.0/config/outputs/dashboard.html) --
> it doesn't display individual messages but displays number of messages
> processed, queues, lags, etc
> * if you send messages over TcpOutput you can enable buffering and use
> "heka-cat" command to browse/tail protobuf logs in /var/cache/heka
> * the most usable for me way is creating additional output like the
> following:
>
> [debug_encoder]
> type="RstEncoder"
>
> [LogOutput]
> encoder="debug_encoder"
> message_matcher="TRUE"
>
> This makes heka dump all messages to stdout. If you store that in logs
> hope you rotate them based on size reached.
>
>
> Best regards,
> Timur
>
> On Wed, Mar 30, 2016 at 2:22 AM, Ramin Ali Dousti 
> wrote:
>
>> Hi,
>>
>> I have a nagging question. I read in the documentations and I also
>> experienced it myself that hekad might become "wedged" and there would be
>> no activity within the system while the process seems up from the outside.
>> My question is what are the conditions that this might happen? The reason I
>> ask is that my hekad instances might be running for weeks with no problem
>> but they could end up "wedged" for no obvious reason. Knowing what gets
>> them into this mode, might help me prevent that situation.
>>
>> Also, hekad seems a very quiet process log-wise. How can I have it log
>> "important" information about its doing?
>>
>> I really appreciate any insight.
>>
>> --
>> Ramin
>>
>> ___
>> Heka mailing list
>> Heka@mozilla.org
>> https://mail.mozilla.org/listinfo/heka
>>
>>
>


-- 
Ramin
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

Re: [heka] heka wedged

[Timur Batyrshin]

Hi Ramin,

When heka is running it emits HeapAlloc, HeapSys and few other metrics
which you can graph or alert based on them.
I've seen heka freezing only when I send TERM signal to it and some Lua
plugin fails to stop correctly or something like that.
May be someone else will give you insights on other conditions.

For checking how heka is processing I know 3 ways:
* dashboard (
http://hekad.readthedocs.org/en/v0.10.0/config/outputs/dashboard.html) --
it doesn't display individual messages but displays number of messages
processed, queues, lags, etc
* if you send messages over TcpOutput you can enable buffering and use
"heka-cat" command to browse/tail protobuf logs in /var/cache/heka
* the most usable for me way is creating additional output like the
following:

[debug_encoder]
type="RstEncoder"

[LogOutput]
encoder="debug_encoder"
message_matcher="TRUE"

This makes heka dump all messages to stdout. If you store that in logs hope
you rotate them based on size reached.


Best regards,
Timur

On Wed, Mar 30, 2016 at 2:22 AM, Ramin Ali Dousti  wrote:

> Hi,
>
> I have a nagging question. I read in the documentations and I also
> experienced it myself that hekad might become "wedged" and there would be
> no activity within the system while the process seems up from the outside.
> My question is what are the conditions that this might happen? The reason I
> ask is that my hekad instances might be running for weeks with no problem
> but they could end up "wedged" for no obvious reason. Knowing what gets
> them into this mode, might help me prevent that situation.
>
> Also, hekad seems a very quiet process log-wise. How can I have it log
> "important" information about its doing?
>
> I really appreciate any insight.
>
> --
> Ramin
>
> ___
> Heka mailing list
> Heka@mozilla.org
> https://mail.mozilla.org/listinfo/heka
>
>
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

[heka] heka wedged

[Ramin Ali Dousti]

Hi,

I have a nagging question. I read in the documentations and I also
experienced it myself that hekad might become "wedged" and there would be
no activity within the system while the process seems up from the outside.
My question is what are the conditions that this might happen? The reason I
ask is that my hekad instances might be running for weeks with no problem
but they could end up "wedged" for no obvious reason. Knowing what gets
them into this mode, might help me prevent that situation.

Also, hekad seems a very quiet process log-wise. How can I have it log
"important" information about its doing?

I really appreciate any insight.

-- 
Ramin
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka