[jira] [Commented] (SOLR-12454) tweak Overseer leadership transition related logging
[ https://issues.apache.org/jira/browse/SOLR-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16547029#comment-16547029 ] Christine Poerschke commented on SOLR-12454: Returned to this ticket after some time and as per above have gone ahead and committed the patch -- as always though, further comments, suggestions, revisions etc. welcome -- thank you. > tweak Overseer leadership transition related logging > > > Key: SOLR-12454 > URL: https://issues.apache.org/jira/browse/SOLR-12454 > Project: Solr > Issue Type: Task >Reporter: Christine Poerschke >Assignee: Christine Poerschke >Priority: Minor > Fix For: master (8.0), 7.5 > > Attachments: SOLR-12454-overseer-scenario-run.patch, > SOLR-12454.patch, SOLR-12454.patch, overseer-scenario-run.sh > > > Proposed patch to follow. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12454) tweak Overseer leadership transition related logging
[ https://issues.apache.org/jira/browse/SOLR-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16547026#comment-16547026 ] ASF subversion and git services commented on SOLR-12454: Commit df08f56420738aa38b8db011ef956b3b488b86c5 in lucene-solr's branch refs/heads/branch_7x from [~cpoerschke] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=df08f56 ] SOLR-12454: Tweak Overseer leadership transition related logging for easier troubleshooting. > tweak Overseer leadership transition related logging > > > Key: SOLR-12454 > URL: https://issues.apache.org/jira/browse/SOLR-12454 > Project: Solr > Issue Type: Task >Reporter: Christine Poerschke >Assignee: Christine Poerschke >Priority: Minor > Attachments: SOLR-12454-overseer-scenario-run.patch, > SOLR-12454.patch, SOLR-12454.patch, overseer-scenario-run.sh > > > Proposed patch to follow. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12454) tweak Overseer leadership transition related logging
[ https://issues.apache.org/jira/browse/SOLR-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16547001#comment-16547001 ] ASF subversion and git services commented on SOLR-12454: Commit e9c6effd07f5bd14ce87c8cea6ced64a33d8d6b0 in lucene-solr's branch refs/heads/master from [~cpoerschke] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e9c6eff ] SOLR-12454: Tweak Overseer leadership transition related logging for easier troubleshooting. > tweak Overseer leadership transition related logging > > > Key: SOLR-12454 > URL: https://issues.apache.org/jira/browse/SOLR-12454 > Project: Solr > Issue Type: Task >Reporter: Christine Poerschke >Assignee: Christine Poerschke >Priority: Minor > Attachments: SOLR-12454-overseer-scenario-run.patch, > SOLR-12454.patch, SOLR-12454.patch, overseer-scenario-run.sh > > > Proposed patch to follow. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12454) tweak Overseer leadership transition related logging
[
https://issues.apache.org/jira/browse/SOLR-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505118#comment-16505118
]
Christine Poerschke commented on SOLR-12454:
Attached revised SOLR-12454.patch and (not intended to be committed) patch with
script to relatively easily explore different overseer scenarios e.g.
spontaneous disappearance (here deliberate deletion) of the
{{/overseer_elect/leader}} node.
> tweak Overseer leadership transition related logging
>
>
> Key: SOLR-12454
> URL: https://issues.apache.org/jira/browse/SOLR-12454
> Project: Solr
> Issue Type: Task
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Attachments: SOLR-12454-overseer-scenario-run.patch,
> SOLR-12454.patch, SOLR-12454.patch, overseer-scenario-run.sh
>
>
> Proposed patch to follow.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12454) tweak Overseer leadership transition related logging
[
https://issues.apache.org/jira/browse/SOLR-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504841#comment-16504841
]
Christine Poerschke commented on SOLR-12454:
bq. ... {{Overseer (id=null) closing}} logging emitted at the _start_ of
apparently _every_ solr node seems confusing?
Okay, so this is at the start of _almost_ every solr node excluding the node
that wins the overseer election. SOLR-12464 opened for this specifically
separately for clarity.
> tweak Overseer leadership transition related logging
>
>
> Key: SOLR-12454
> URL: https://issues.apache.org/jira/browse/SOLR-12454
> Project: Solr
> Issue Type: Task
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Attachments: SOLR-12454.patch, overseer-scenario-run.sh
>
>
> Proposed patch to follow.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12454) tweak Overseer leadership transition related logging
[
https://issues.apache.org/jira/browse/SOLR-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503886#comment-16503886
]
Christine Poerschke commented on SOLR-12454:
{quote}... If you can isolate the added logging to log events that will only
occur during transition ...
{quote}
Yes, that is the intention – the logging is intended to assist not hinder and
ideally to assist out-of-the-box when someone hits overseer issues for the
first time e.g. due to (external) zookeeper hiccups of some sort.
In lieu of integration or regression tests, I just attached the
{{overseer-scenario-run.sh}} script I wrote for my own use to more easily
explore the scenarios i.e. {{download and unpack zookeeper, start external
zookeeper, start solr cloud, trigger overseer transition, stop solr, stop
zookeeper}} – that kind of thing.
+My next intended step+ is to consider both the proposed {{info}} as well as
the existing {{info}} logging for {{Overseer.java}} e.g. the {{Overseer
(id=null) closing}} logging emitted at the _start_ of apparently _every_ solr
node seems confusing?
{quote}... don't want another of those log lines spamming the startup sequence
or popping up every second just to say "I'm alive". ...
{quote}
I agree. Spammy logging can hinder log interpretation and troubleshooting.
{quote}... Also see related SOLR-12456 which I just filed ...
{quote}
Good find, thanks!
> tweak Overseer leadership transition related logging
>
>
> Key: SOLR-12454
> URL: https://issues.apache.org/jira/browse/SOLR-12454
> Project: Solr
> Issue Type: Task
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Attachments: SOLR-12454.patch, overseer-scenario-run.sh
>
>
> Proposed patch to follow.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12454) tweak Overseer leadership transition related logging
[
https://issues.apache.org/jira/browse/SOLR-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503400#comment-16503400
]
Jan Høydahl commented on SOLR-12454:
By dynamic I meant issuing the HTTP command in order to raise certain logging
to DEBUG level across the cluster. I'm not sure if overseer debugging is
something so common that we should bog down everyone with those low-level logs?
Anyone who had Overseer issues once or twice and want to catch the next one
could of course configure log4j2.xml if they don't want to run verbose ({{-v}})
for all of Solr.
That said, I have not analysed the proposed patch in detail. If you can isolate
the added logging to log events that will only occur during transition then it
may be ok. But I don't want another of those log lines spamming the startup
sequence or popping up every second just to say "I'm alive". Also see related
SOLR-12456 which I just filed to try to get rid of spammy DEBUG-level logging
from the overseer.
> tweak Overseer leadership transition related logging
>
>
> Key: SOLR-12454
> URL: https://issues.apache.org/jira/browse/SOLR-12454
> Project: Solr
> Issue Type: Task
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Attachments: SOLR-12454.patch
>
>
> Proposed patch to follow.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12454) tweak Overseer leadership transition related logging
[ https://issues.apache.org/jira/browse/SOLR-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502988#comment-16502988 ] Daniel Collins commented on SOLR-12454: --- The nature of Overseer problems we have seen, is that by the time we realize we have lost the overseer, there is nothing more to do other than bounce to regain a new one. What we need is logging before/during the loss of overseer. [~janhoy], by on demand, did you mean dynamically after the fact, or just we should have a custom logging configuration which would set those particular debug statements to log permanently in our setup? The former wouldn't work (see above), but I that latter might be something we could look at. I'm curious how others deal with these kinds of issues, or do they fortunately not have them :). > tweak Overseer leadership transition related logging > > > Key: SOLR-12454 > URL: https://issues.apache.org/jira/browse/SOLR-12454 > Project: Solr > Issue Type: Task >Reporter: Christine Poerschke >Assignee: Christine Poerschke >Priority: Minor > Attachments: SOLR-12454.patch > > > Proposed patch to follow. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12454) tweak Overseer leadership transition related logging
[ https://issues.apache.org/jira/browse/SOLR-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502549#comment-16502549 ] Jan Høydahl commented on SOLR-12454: Can you give background for why such debugging cannot be done by raising the log level for the Overseer to DEBUG on demand? That's what we have debug levels for, not :) > tweak Overseer leadership transition related logging > > > Key: SOLR-12454 > URL: https://issues.apache.org/jira/browse/SOLR-12454 > Project: Solr > Issue Type: Task >Reporter: Christine Poerschke >Assignee: Christine Poerschke >Priority: Minor > Attachments: SOLR-12454.patch > > > Proposed patch to follow. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12454) tweak Overseer leadership transition related logging
[
https://issues.apache.org/jira/browse/SOLR-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502339#comment-16502339
]
Christine Poerschke commented on SOLR-12454:
Attached proposed patch which is intended to make it easier to troubleshoot and
debug Overseer leadership transition related problems.
cc/fyi [~romseygeek] and [~janhoy] w.r.t. the earlier SOLR-5563 changes. This
here would revert a small number of those info-to-debug changes but the changes
are very selective and targetted at code paths that should not happen on a
regular basis (e.g. {{am_i_leader unclear}}) or that happen infrequently i.e.
overseer leadership transition. What do you think?
> tweak Overseer leadership transition related logging
>
>
> Key: SOLR-12454
> URL: https://issues.apache.org/jira/browse/SOLR-12454
> Project: Solr
> Issue Type: Task
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Attachments: SOLR-12454.patch
>
>
> Proposed patch to follow.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
