[
https://issues.apache.org/jira/browse/IMPALA-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17235818#comment-17235818
]
Fifteen edited comment on IMPALA-9865 at 11/20/20, 1:46 AM:
------------------------------------------------------------
Hi, we have just encountered a similar situation, but I think our approach is
highly imperfect:
# With MT_DOP set to 64;
# A single profile can be very large. For the record, 1.4GB profile downloaded
from ClouderaManager
# With growing number of queries, ServiceMonitor halt constantly. CPU usage
can reach 90%, we believe it's busy with unserializing huge profiles.
# We found that the longest part is 'Detailed Fragments', which is less
interesting than `Averaged Fragment` part.
# So we modified on `Coordinator.cc`, remove 'Detailed Fragment' and kept
'Averaged Fragment'
# Now CM seems to work well with 'Slimed Profiles', each profile's size is now
XX KBs
!image-2020-11-20-09-25-08-082.png!
was (Author: fifteencai):
We have just encountered a similar situation, but I think our approach is
highly imperfect:
# With MT_DOP set to 64;
# A single profile can be very large. For the record, 1.4GB profile downloaded
from ClouderaManager
# With growing number of queries, ServiceMonitor halt constantly. CPU usage
can reach 90%, we believe it's busy with unserializing huge profiles.
# We found that the longest part is 'Detailed Fragments', which is less
interesting than `Averaged Fragment` part.
# So we modified on `Coordinator.cc`, remove 'Detailed Fragment' and kept
'Averaged Fragment'
# Now CM seems to work well with 'Slimed Profiles', each profile's size is now
XX KBs
!image-2020-11-20-09-25-08-082.png!
> Utility to pretty-print thrift profiles at various levels
> ---------------------------------------------------------
>
> Key: IMPALA-9865
> URL: https://issues.apache.org/jira/browse/IMPALA-9865
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Major
> Attachments: image-2020-11-20-09-25-08-082.png
>
>
> The prototyping work in IMPALA-9382 revealed some hard trade-offs between
> having a full-fidelity text profile and readability.
> We want to have a text profile with less information by default so that it is
> more readable, and rely on the thrift profile for more detailed debugging.
> This would be easier if we provided a utility that can pretty-print a thrift
> profile at different levels of detail.
> This JIRA is to reduce the default level of pretty-printing for aggregated
> profiles, but provide a utility that can dump both the basic and full
> versions. My thought is to start off with the same 4 levels as explain, but
> maybe only implement 2 levels to start off with - basic and extended.
> The utility should be able to handle the same cases as
> bin/parse-thrift-profile.py (profile log and single profile in a file) and
> maybe print only a specified query from a profile log. We can use the
> DeserializeFromArchiveString() method that was removed in IMPALA-9381, then
> pretty-print the deserialised profile.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
