741550557 <[email protected]> 于2019年12月9日周一 下午9:42写道：
> Thank for your reply, the issues you mentioned are very critical and > meaningful. > There I will answer what you mentioned. Sorry, I'm not good at comment > mode, so I use different colors and “ “ prefix to QA. > > > As we already have designed limit mechanism at backend and agent > side(according to your design), also the number would not be big(10 most > likely), we just need a list to storage the trace-id(s) > > > If just need a list to storage trace-id(s), so how can I map to the > thread? I hope to use the map to quickly find thread info from trace-id. > How can I get thread-stack information from your way? Could you please > help elaborate? > Why do you need to do that? You just save a list of thread ids which should do thread dump, or remove some thread id from them when the trace id is finished. This is easy to do this by doing a loop search in the list. Right? Thread-stack is in the list, they are stored in an element. Also, they are in a list too. I think you were thinking the same all stack in a single map? That will cause a very dangerous memory and GC risk. > > > Could you explain the (2), what do you mean `stop`? I think if your > sampling mechanism should include the sampling duration. > > > As far as the communication between the sniffer and the OAP server, I hope > the sniffer only needs to obtain the thread-monitor task that needs to be > monitored at this time. The termination condition can be stopped by the > sniffer or the OAP server. > If It’s just an OAP server notification, it may be more complicated. Cause > OAP server need record sniffer has received the current command, and > sniffer is not stable, such as sniffer has shutdown when receiving the > command, at this time, no thread information I have been collected. > > > I think that the active calculation termination by the OAP server can make > the monitoring more controllable, of course, the client can also actively > report the end. > I think it’s necessary to provide a protection mechanism for the sniffer > side, and it can be released quickly when the business peak period or the > probe suddenly occupies a lot of CPU / memory resources. Therefore, the > function of stopping monitoring can be provided in the UI interface, so > that the sniffer can recover. > Sampling duration is required, but only as a default termination > thread-monitor condition. > But you should know, in the real case, the thread dump monitor is a sampling mechanism, you are even hard to know where they are happening. Then you have to send the stop notification to every instance. Even you could send the notification, but could you explain how you know to stop? The scenario is, you are facing an issue, which trace and metrics can't explain, so you active thread dump, right? At the same time, you want to stop? CPU and memory resources should be guaranteed by design level, such as 1. Limited thread dump task for one service. 2. Limited thread dump traces in the certain time window. For example, the OAP backend/UI would say, you only could 1. Set 3 thread dump commands in the same time window. 2. Every command will require the sampling thread dump number should be less than 5 traces. At the same time, in order to make this sampling works, only active sampling thread dump after the trace executed more than 200ms(value is an example only). 3. Thread dump could be sent to the backend duration sampling to reduce the memory cache. 4. Thread dump period should not less than 5ms, recommend 20ms 5. How depth the thread dump should do We need a very detailed design, above are just my thoughts, in order to share the idea, the safe of the agent should not be by UI button. Otherwise, your online system will be very dangerous, which is not the design goal of SkyWalking. > > > The sampling period depends on how you are going to visualize it. > > > Yes, I agree. I hope can provide a select/input let trace count and time > windows can be configurable in UI. Of course, this is my current idea, and > if there have other plains, I will adopt it. > > > Highly doubt about this, reduce the memory, maybe, only reduce if the > codes > are running the loop or facing lock issue. But if it is neither of these > two, they are different. > Also, please consider the CPU cost of the comparison of the stack. You > need > a performance benchmark to verify if you want this. > > > I didn’t understand that first sentence. In my personal experience, most > of the cases are blocking in the lock(socket/local) and running loop. I > have not imagined any other cases? > For the second sentence, I think I can add a thread-stack-element field to > storage the top-level element of last stack information. When get stack > information next time, I can compare the current top-level element that is > the same with that field. > I do this mainly to reduce duplicate thread-stack information form taking > up too much memory space, this is a way to optimizing memory space. It can > consider remove it, or do you have a better memory-saving solution? After > all, memory and CPU resources are very valuable in the sniffer. > I know you mean about reducing the memory, but do you consider how much CPU you will cost do a full thread dump comparison? The thread dump could easily be hundreds of lines in Java. I mean this is a tradeoff, CPU or memory. If you are just using limited memory, before you could send the snapshot to backend while collecting new, even could save into the disk(if really necessary). In my experience, compress is always very high risk in the agent, if you want to do that, you need a benchmark test to improve that, this CPU cost is small enough. > > > The trace number and time window should be configurable, that is I mean > more complex. Inthe current SamplingServcie, only n traces per 3 seconds. > But here, it is a dynamic rule. > > > I expect that it can be configured at the UI level for special trace count > and time windows as I said above. > For SamplingService, my previous tech design was not rigorous enough, and > there were indeed problems. > Maybe we need to extend a new SamplingService, build a mapping base on > endpoint-id and AtomicInteger. > For `first 5 traces of this endpoint in the next 5 mins`, just need to > increment it. > For sampling, maybe use another schedule task to reset AtomicInteger value. > You could avoid map, by using ArrayList with RangeAtomicInteger(SkyWalking provides that) to let the trace context to get the slot. Also, you are considering `active sampling after trace execution time more than xxx ms`, you should add remove mechanism during runtime. Anyway, try your best to avoid using Map, especially this map could be changed in the runtime. > > > I think at least should be a level one new page called configuration or > command page, which could set up the multiple sampling rule and visualize > the existing tasks and related sampling data. > > > I think it’s necessary to add a new page to the configuration > thread-monitor task, I think the specific UI display should be designed in > detail. > For example, what I expected is similar to the trace page. The left side > displays the configuration, and the right side quickly displays the related > trace list. When clicked, it quickly links to the trace page and displays > the sidebox display. > I ’m not good at this. Do you have any good plans? > UI is the thing that is hard to discuss by text, so I am pretty sure, we need some demo(could not be the codes, that is I mean drew by a tool) It is OK to show a trace with thread dumps on another page, even better linking to your task ID. But this kind of abstract description is hard to continue, no details I mean. > And I feel that the two of us have a different understanding of the > configuration object. I think it is more of a task than a command. I don't > know which way is better? > I suddenly thought of a problem. I think that some real problems are often > triggered at a specific period, such as a fixed business peak period, and > we cannot guarantee that the user will operate on the UI. > So should the task mechanism be adopted to ensure that it can be run at a > specific period? > This makes sense to me, and it is a just enhance feature. It is just a start time sampling rule. > > > We don't have separated thread monitor view table, how about we add an > icon > at the segment list, and add icon at the first span of this segment in > trace detail view? > I think the latter one should be an entrance of the thread view. > > > I think it's a good idea. The link I mentioned in one of the answers > above, I think it is also a convenient entry point. > The switch button I mentioned earlier is only a data filtering item in the > query of the trace list and does not need a separate table UI. > As you intend to have a separated page for thread sampling, it is OK to > > > If you have some visualization idea, drawn by any tool you like supporting > comment, we could discuss it there. In my mind, we should support > visualize > the thread dump stack through the time windows, and support aggregate them > by choosing the continued stack snapshots on the time window. > > > I think we should find a front-end who is better at discussing together > because this depends on how the front-end UI can be displayed. > BTW: I can provide code for the OAP server and sniffer, and the frontend > may need to look for help in the community alone. Hope that any front-end > friends can participate in the topic discussion. > Once you have the demo, I could loop our UI committers in for UI side development. But UI committers may not be familiar with thread dump context story. We need to resolve that first. Let's start up a demo, such as some slides on Google doc? > > > > > The above is my answer to all the questions, and I look forward to your > reply at any time. As more and more discussions took place, the details > became more and more complete. This is good. > Everyone is welcome to discuss together if you have any questions or good > ideas, please let me know. > I think we could move the discussion to the design doc as the next step. Please use this https://docs.google.com/document/d/1rxMf1WN3PaFaZp7r8JmtwfdkmjLTcFW_ETAZv5FIU-s/edit# Trite the design including 1. Key features 2. Protocol 3. Work mechanism 4. UI design, prototype and anything you think important before writing codes. This is SkyWalking CLI design doc, you could use it as a reference. https://docs.google.com/document/d/1WBnRNF0ABxaSdBZo6Gv2hMzCQzj04YAePUdOyLWHWew/edit# > > > 原始邮件 > 发件人:Sheng [email protected] > 收件人:[email protected] > 发送时间:2019年12月9日(周一) 10:50 > 主题:Re: A proposal for Skywalking(thread monitor) > > > Hi Thanks for writing this proposal with a detailed design. My comments > are inline. 741550557 [email protected] 于2019年12月8日周日 下午11:22写道： Thanks > for your reply, I have carefully read these issues you mentioned, and > these issues mentioned are very meaningful and critical. I will give you > technical details about the issues you mentioned below. I find these > issues are related, so I will explain them in different dimensions. use > a different protocol to transmission trace and thread-stack: 1. add a > boolean field in segment data, to record has thread monitored. and is good > for filter monitored trace in UI. 2. add new BootService, storage Map to > record relate trace-id and trace-stack information. As we already have > designed limit mechanism at backend and agent side(according to your > design), also the number would not be big(10 most likely), we just need a > list to storage the trace-id(s) 3. listen > TracingContextListener#afterFinished if the current segment has thread > monitored, mark current trace-id don’t need to monitor anymore. (Cause if > for-each the step 2 map, the remove operation will fail and throw > exception). 4. when thread-monitor main thread running, It will for-each > step 2 map and check is it don’t need monitor anymore, I will put data > into new data carrier. 5. generate new thread-monitor gRPC protocol to > send data from the data carrier. The agent side design seems pretty good. > the server receives thread-stack logic: 1. storage stack-stack > informations and trace-id/segment-id relations on a different table. 2. > check thread-monitor is need to be stop on receiving data or schedule. > Could you explain the (2), what do you mean `stop`? I think if your > sampling mechanism should include the sampling duration. reduce CPU and > memory in sniffer: 1. through the configuration of thread monitoring in > the UI, you can configure the performance loss. For example, set the > monitoring level: fast monitoring (100ms), medium speed monitoring > (500ms), slow speed monitoring (1000ms). The sampling period depends on > how you are going to visualize it. 2. add new integer field on per > thread-stack, if current thread-stack last element same as last time, > don’t need storage, just increment it. I think it will save a lot of > memory space. Highly doubt about this, reduce the memory, maybe, only > reduce if the codes are running the loop or facing lock issue. But if it is > neither of these two, they are different. Also, please consider the CPU > cost of the comparison of the stack. You need a performance benchmark to > verify if you want this. 3. create new VM args to setting thread-monitor > pool size, It dependence on user, maybe default 3? (this can be discussed > later) I think UI limit is enough. 3 seems good to me. 4. limit > thread-stack-element size to 100, I think it can resolve most of the > scenes already. It also can create a new VM args if need. multiple > sampling methods can choose :(just my current thoughts, can add more) 1. > base on current client SamplingServcie, extra a new factor holder to > increment, and reset on schedule. Yours may be a little more complex than > the current SamplingServcie, right? Based on the next rule. 2. `first 5 > traces of this endpoint in the next 5 mins`, it a good idea. My > understanding is that within a few minutes, each instance can send a > specified number of traces. The trace number and time window should be > configurable, that is I mean more complex. Inthe current SamplingServcie, > only n traces per 3 seconds. But here, it is a dynamic rule. UI settings > and sniffer perception: 1. create a new button on the dashboard page, It > can create or stop a thread-monitor. It can be dynamic load thread-monitor > status when reselecting endpoint. I think at least should be a level one > new page called configuration or command page, which could set up the > multiple sampling rule and visualize the existing tasks and related > sampling data. 2. sniffer creates a new scheduled task to check the > current service has need monitor endpoint each 5 seconds. (I see current > sniffer has command functions, feel that principle is the same as the > scheduler) Seems reasonable. thread-monitor on the UI:(That’s just my > initial thoughts, I think there will have a better way to show) 1. When > switch to the trace page, I think we need to add a new switch button to > filter thread-monitor trace. 2. make a new thread-monitor icon on the same > segment. It means it has thread-stack information. We don't have > separated thread monitor view table, how about we add an icon at the > segment list, and add icon at the first span of this segment in trace > detail view? I think the latter one should be an entrance of the thread > view. 3. show on the information sidebox when the user clicks the > thread-monitor segment(any span). create a new tab, like the log tab. If > you have some visualization idea, drawn by any tool you like supporting > comment, we could discuss it there. In my mind, we should support visualize > the thread dump stack through the time windows, and support aggregate them > by choosing the continued stack snapshots on the time window. They're > just a description of my current implementation details for thread-monitor > if these seem to work. I can do some time planning for these tasks. Sorry, > my English is not very well, hope you can understand. Maybe these seem to > have some problem, any good idea or suggestion are welcome. Very > appreciated you to lead this new direction. It is a long term task but > should be interesting. :) Good work, carry on. 原始邮件 发件人:Sheng > [email protected] 收件人:[email protected] > 发送时间:2019年12月8日(周日) 08:31 主题:Re: A proposal for Skywalking(thread > monitor) First of all, thanks for your proposal. Thread monitoring is > super important for application performance. So basically, I agree with > this proposal. But for tech details, I think we need more discussion in > the following ways 1. Do you want to add thread status to the trace? If > so, why don't consider this as a UI level join? Because we could know > thread id in the trace when we create a span, right? Then we have all the > thread dump(if), we could ask UI to query specific thread context based > on timestamp and thread number(s). 2. For thread dump, I don't know > whether you do the performance evaluation for this OP. From my > experiences, `get all need thread monitor segment every 100 milliseconds` > is a very high cost in your application and agent. So, you may need to be > careful about doing this. 3. Endpoint related thread dump with some > sampling mechanisms makes more sense to me. And this should be activated > by UI. We should only provide a conditional thread dump sampling > mechanism, such as `first 5 traces of this endpoint in the next 5 mins`. > Jian Tan I think DaoCloud also has customized this feature in your > internal SkyWalking. Could you share what you do? Sheng Wu 吴晟 Twitter, > wusheng1108 741550557 [email protected] 于2019年12月8日周日 上午12:14写道： Hello > everyone, I would like to share a new feature with skywalking, called > “thread monitor”. Background When our company used skywalking to APM > earlier, we found that many traces did not have enough span to fill up, > doubting whether there were some third-party frameworks that we didn't > enhance or programmers API usage errors such as java CountDown number is 3 > but there are only 2 countdowns. So we decide to write a new feature to > monitor executing trace thread stack, then we can get more information on > the trace, quick known what’s happening on that trace. Structure > Agent(thread monitor) — gRPC protocol — OAP Server(Storage) — > Skywalking-Rocketbot-UI More detail OAP Server: 1. Storage witch traces > need to monitor(i suggest storage on the endpoint, add new boolean field > named needThreadMonitor) 2. Provide GraphQL API to change endpoint monitor > status. 3. Monitor Trace parse, storage thread stack if the segment has > any thread info. Skywalking-Rocketbot-UI: 1. Add a new switch button on > the dashboard, It can read or modify endpoint status. 2. It will show > every thread stack on click trace detail. Agent: 1. setup two new > BootService: 1) find any need thread monitor endpoint in current service, > start on a new schedule take and works on each minute. 2) start new > schedule task to get all need thread monitor segment each 100 > milliseconds, and put a new thread dump task to a global thread > pool(fixed, count number default 3). 2. check endpoint need thread monitor > on create entry/local span(TracingConext#createEntry/LocalSpan). If need, > It will be marked and put into thread monitor map. 3. when TraceingContext > finishes, It will get thread has monitored, and send all thread stack to > server. Finally, I don’t know it is a good idea to get more information on > trace? If you have any good ideas or suggestions on this, please let me > know. Mrpro
