I think you are attributing too much importance to some data that can hardly be considered information let me try to mention some points that I think are valid here. 1) Not every message is valuable, you state that every message we throw away makes it harder to debug, but certainly a message like "downloading N%" is useless, you can record the start of the download and failure/end but the steps intermediate are quite useless. We can argue later which messages satisfy this criteria, but I am completely sure that some do. 2) Not filling the history buffer with superfluos messages will help here, although I do agree we should find a more elegant deletion criteria (time sounds right) while not loosing sight of size (at this point no one has a record of the actual cost of storing these things in terms of space therefore we cannot make decisions based on the scale we want to support.
Regarding "making the charm decide" I agree its something we might not want to do, I would actually not export this to the charm, I would just use it internally, since we are opinionated we can very well decide what goes there. Adding more status information will help adding observability, not having a flag for internal ephemeral statuses strikes me a bit as deleting with the left hand what you just wrote with the right one (saying might not translate well from Spanish) Finally, can this be a potential shoot in our own foot tool? yes, but so can almost any other part of our code base, this is something juju will use to report information to the user therefore we control it and, if we are not careful we will shot ourselves but, then again, if we are not careful with almost any part of the code we will do so too. Cheers On Thu, Mar 17, 2016 at 6:51 AM, William Reade <william.re...@canonical.com> wrote: I see this as a combination of two problems: > > 1) We're spamming the end user with "whatever's in the status-history > collection" rather than presenting a digest tuned for their needs. > 2) Important messages get thrown away way too early, because we don't know > which messages are important. > > I think the pocket/transient/expiry solutions boil down to "let's make the > charmer decide what's important", and I don't think that will help. The > charmer is only sending those messages *because she believes they're > important*; even if we had "perfect" trimming heuristics for the end user, > we do the *charmer* a disservice by leaving them no record of what their > charm actually did. > > And, more generally: *every* message we throw away makes it hard to > correctly analyse any older message. This applies within a given entity's > domain, but also across entities: if you're trying to understand the > interactions between 2 units, but one of those units is generating many > more messages, you'll have 200 messages to inspect; but the 100 for the > faster unit will only cover (say) the last 30 for the slower one, leaving > 70 slow-unit messages that can't be correlated with the other unit's > actions. At best, those messages are redundant; at worst, they're actively > misleading. > > So: I do not believe that any approach that can be summed up as "let's > throw away *more* messages" is going to help either. We need to fix (2) so > that we have raw status data that extends reasonably far back in time; and > then we need to fix (1) so that we usefully precis that data for the user > (...and! leave a path that makes the raw data observable, for the cases > where our heuristics are unhelpful). > > Cheers > William > > PS re: UX of asking for N entries... I can see end-user stories for > timespans, and for "the last N *significant* changes". What's the scenario > where a user wants to see exactly 50 message atoms? > > On Thu, Mar 17, 2016 at 6:30 AM, John Meinel <j...@arbash-meinel.com> > wrote: > >> >> >> On Thu, Mar 17, 2016 at 8:41 AM, Ian Booth <ian.bo...@canonical.com> >> wrote: >> >>> >>> Machines, services and units all now support recording status history. >>> Two >>> issues have come up: >>> >>> 1. https://bugs.launchpad.net/juju-core/+bug/1530840 >>> >>> For units, especially in steady state, status history is spammed with >>> update-status hook invocations which can obscure the hooks we really >>> care about >>> >>> 2. https://bugs.launchpad.net/juju-core/+bug/1557918 >>> >>> We now have the concept of recording a machine provisioning status. This >>> is >>> great because it gives observability to what is happening as a node is >>> being >>> allocated in the cloud. With LXD, this feature has been used to give >>> visibility >>> to progress of the image downloads (finally, yay). But what happens is >>> that the >>> machine status history gets filled with lots of "Downloading x%" type >>> messages. >>> >>> We have a pruner which caps the history to 100 entries per entity. But >>> we need a >>> way to deal with the spam, and what is displayed when the user asks for >>> juju >>> status-history. >>> >>> Options to solve bug 1 >>> >>> A. >>> Filter out duplicate status entries when presenting to the user. eg say >>> "update-status (x43)". This still allows the circular buffer for that >>> entity to >>> fill with "spam" though. We could make the circular buffer size much >>> larger. But >>> there's still the issue of UX where a user ask for the X most recent >>> entries. >>> What do we give them? The X most recent de-duped entries? >>> >>> B. >>> If the we go to record history and the current previous entry is the >>> same as >>> what we are about to record, just update the timestamp. For update >>> status, my >>> view is we don't really care how many times the hook was run, but rather >>> when >>> was the last time it ran. >>> >> >> The problem is that it isn't the same as the "last" message. Going to the >> original paste: >> >> TIME TYPE STATUS MESSAGE >> 26 Dec 2015 13:51:59Z agent idle >> 26 Dec 2015 13:56:57Z agent executing running update-status hook >> 26 Dec 2015 13:56:59Z agent idle >> 26 Dec 2015 14:01:57Z agent executing running update-status hook >> 26 Dec 2015 14:01:59Z agent idle >> >> Which means there is an "running update-status" *and* a "idle" message. >> So we can't just say "is the last message == this message". It would have >> to look deeper in history, and how deep should we be looking? what happens >> if a given charm does one more "status-set" during its update-status hook >> to set the status of the unit to "still happy". Then we would have 3. >> (agent executing, unit happy, agent idle) >> >> >>> Options to solve bug 2 >>> >>> A. >>> Allow a flag when setting status to say "this status value is transient" >>> and so >>> it is recorded in status but not logged in history. >>> >>> B. >>> Do not record machine provisioning status in history. It could be argued >>> this >>> info is more or less transient and once the machine comes up, we don't >>> care so >>> much about it anymore. It was introduced to give observability to machine >>> allocation. >>> >> >> Isn't this the same as (A)? We need a way to say that *this* message >> should be showed but not saved forever. Or are you saying that until a >> machine comes up as "running" we shouldn't save any of the messages? I >> don't think we want that, because when provisioning fails you want to know >> what steps were achieved. >> >> >>> >>> Any other options? >>> Opinions on preferred solutions? >>> >>> I really want to get this fixed before Juju 2.0 >>> >> >> We could do a "log level" rather than just "transient or not", and that >> would decide what would get displayed by default. (so you can ask for >> 'update-status' messages but they wouldn't be shown by default). The >> problem is that we want to keep status messages pruned at a sane level and >> with 2 updates for every 'update-status' call history of 100 is only >> 100/2*5/60 ~ 4hours of history. If something interesting happened >> yesterday, you're SOL. >> >> What if we added a "interesting lifetime" to status messages. So the >> status-set could indicate how long the message would be preserved? >> "update-status" and "idle" could be flagged as preserved for only 1 hour, >> and "dowloading %" could be flagged at say 5 minutes. Too complicated? It >> certainly complicates the pruner (not terribly, when we record them we just >> record an expire time that is indexed and the pruner just removes >> everything that is over its expiry time.) >> >> Alternatively we could have some sort of UUID for messages to indicate >> that "this message is actually similar to other messages with this UUID" >> and we prune them based on that. (UUIDs get flagged with a different number >> of messages to keep than the global 100 for otherwise untagged messages.) >> >> "Transient" is the easiest to understand, but doesn't really solve bug #1. >> >> If we think of the "UUID" version as something like a named "status >> pocket" maybe its actually tasteful. You'd have the "global" pocket that >> has our default 100 most-recent-messages, and then you can create any new >> pocket that has a default of say 10 messages. So you would be doing: >> status-set --pocket hook-execution update-status >> status-set --pocket download Downloading X% done >> >> That also lets charms do nice things at hook execution time when they're >> downloading large resources, without spamming the status-history log. >> >> It does complicate the model.... >> >> John >> =:-> >> >> >> >> -- >> Juju-dev mailing list >> Juju-dev@lists.ubuntu.com >> Modify settings or unsubscribe at: >> https://lists.ubuntu.com/mailman/listinfo/juju-dev >> >> > > -- > Juju-dev mailing list > Juju-dev@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju-dev > >
-- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
