CPU Utilization
Hello: This is more of a general question. Our application uses Derby in the in-memory mode only. When the application is configured to use complex queries, such configuration causes CPU utilization (on the Linux server) to go as high as 300% or even higher. Using simpler queries don't seem to lead to such high utilization. Is there any way to control or lower CPU utilization by Derby? Thank you, Pavel. Jefferies archives and monitors outgoing and incoming e-mail. The contents of this email, including any attachments, are confidential to the ordinary user of the email address to which it was addressed. If you are not the addressee of this email you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. This email may be produced at the request of regulators or in connection with civil litigation. Jefferies accepts no liability for any errors or omissions arising as a result of transmission. Use by other than intended recipients is prohibited. In the United Kingdom, Jefferies operates as Jefferies International Limited; registered in England: no. 1978621; registered office: Vintners Place, 68 Upper Thames Street, London EC4V 3BJ. Jefferies International Limited is authorised and regulated by the Financial Services Authority.
Re: CPU Utilization
On 07.01.2013 20:51, Pavel Bortnovskiy wrote: Hello: This is more of a general question. Our application uses Derby in the in-memory mode only. When the application is configured to use complex queries, such configuration causes CPU utilization (on the Linux server) to go as high as 300% or even higher. Using simpler queries don't seem to lead to such high utilization. Is there any way to control or lower CPU utilization by Derby? Hi Pavel, I don't believe there is such a mechanism in place, at least not specifically for the in-memory back end. Since this is in-memory, the latency associated with page operations is very low. Do you know if Derby is using the CPU for something useful (i.e. processing queries effectively), or is the CPU spent on wasteful activities? One potential wasteful activity is moving data back and forth from the page cache (page cache too small?). Is this a highly concurrent scenario? Regards, -- Kristian Thank you, Pavel. Jefferies archives and monitors outgoing and incoming e-mail. The contents of this email, including any attachments, are confidential to the ordinary user of the email address to which it was addressed. If you are not the addressee of this email you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. This email may be produced at the request of regulators or in connection with civil litigation. Jefferies accepts no liability for any errors or omissions arising as a result of transmission. Use by other than intended recipients is prohibited. In the United Kingdom, Jefferies operates as Jefferies International Limited; registered in England: no. 1978621; registered office: Vintners Place, 68 Upper Thames Street, London EC4V 3BJ. Jefferies International Limited is authorised and regulated by the Financial Services Authority.
RE: CPU Utilization
Thank you, Kristian, for your quick response. How can I diagnose this and determine what Derby is doing during the periods of high CPU utilization? As far as I can determine from watching the log and the "top" simultaneously, it seems that utilization peeks when data are accessed. In other words, the query itself seems to run very fast, but accessing the data (with the JDBC's ResultSet) seems to peek the CPU usage. P. From: Kristian Waagan [mailto:kristian.waa...@oracle.com] Sent: Wednesday, January 09, 2013 2:42 PM To: derby-user@db.apache.org Subject: Re: CPU Utilization On 07.01.2013 20:51, Pavel Bortnovskiy wrote: Hello: This is more of a general question. Our application uses Derby in the in-memory mode only. When the application is configured to use complex queries, such configuration causes CPU utilization (on the Linux server) to go as high as 300% or even higher. Using simpler queries don't seem to lead to such high utilization. Is there any way to control or lower CPU utilization by Derby? Hi Pavel, I don't believe there is such a mechanism in place, at least not specifically for the in-memory back end. Since this is in-memory, the latency associated with page operations is very low. Do you know if Derby is using the CPU for something useful (i.e. processing queries effectively), or is the CPU spent on wasteful activities? One potential wasteful activity is moving data back and forth from the page cache (page cache too small?). Is this a highly concurrent scenario? Regards, -- Kristian Thank you, Pavel. Jefferies archives and monitors outgoing and incoming e-mail. The contents of this email, including any attachments, are confidential to the ordinary user of the email address to which it was addressed. If you are not the addressee of this email you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. This email may be produced at the request of regulators or in connection with civil litigation. Jefferies accepts no liability for any errors or omissions arising as a result of transmission. Use by other than intended recipients is prohibited. In the United Kingdom, Jefferies operates as Jefferies International Limited; registered in England: no. 1978621; registered office: Vintners Place, 68 Upper Thames Street, London EC4V 3BJ. Jefferies International Limited is authorised and regulated by the Financial Services Authority. Jefferies archives and monitors outgoing and incoming e-mail. The contents of this email, including any attachments, are confidential to the ordinary user of the email address to which it was addressed. If you are not the addressee of this email you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. This email may be produced at the request of regulators or in connection with civil litigation. Jefferies accepts no liability for any errors or omissions arising as a result of transmission. Use by other than intended recipients is prohibited. In the United Kingdom, Jefferies operates as Jefferies International Limited; registered in England: no. 1978621; registered office: Vintners Place, 68 Upper Thames Street, London EC4V 3BJ. Jefferies International Limited is authorised and regulated by the Financial Services Authority.
Re: CPU Utilization
On 09.01.2013 20:45, Pavel Bortnovskiy wrote: Thank you, Kristian, for your quick response. How can I diagnose this and determine what Derby is doing during the periods of high CPU utilization? I'm not quite sure how to do this easily. Some suggestions: o Use jmap to dump the heap histogram. While easy to do, getting useful information from the histogram generally requires knowledge about Derby's internals. This may or may not result in clues about what's going on. o Use your favorite Java profiler/sampler to see where the processing time is being spent. If you don't have a favorite tool for this, you might get away with using VisualVM, NetBeans, or Eclipse. In some cases it can be useful to run HPROF directly. As far as I can determine from watching the log and the "top" simultaneously, it seems that utilization peeks when data are accessed. In other words, the query itself seems to run very fast, but accessing the data (with the JDBC's ResultSet) seems to peek the CPU usage. What you describe above is typical, since the actual processing happens when you start accessing the data (i.e. typically through ResultSet.next()). Depending on the query, it can for instance start reading and sorting data when you invoke next the first time. You might want to check the query plans of your queries, or at least inspect them visually, to understand if they would read all the data of a table. As usual there are more questions than answers here: o how big is your dataset? o how big is your page cache? o do you have enough space on the heap? (excessive GC can cause high CPU utilization) o what's the degree of concurrency? (single vs multiple connections executing queries, I assume concurrent access but in theory it's possible to see high CPU usage otherwise too) o what do your queries look like? -- Kristian P. *From:*Kristian Waagan [mailto:kristian.waa...@oracle.com] *Sent:* Wednesday, January 09, 2013 2:42 PM *To:* derby-user@db.apache.org *Subject:* Re: CPU Utilization On 07.01.2013 20:51, Pavel Bortnovskiy wrote: Hello: This is more of a general question. Our application uses Derby in the in-memory mode only. When the application is configured to use complex queries, such configuration causes CPU utilization (on the Linux server) to go as high as 300% or even higher. Using simpler queries don't seem to lead to such high utilization. Is there any way to control or lower CPU utilization by Derby? Hi Pavel, I don't believe there is such a mechanism in place, at least not specifically for the in-memory back end. Since this is in-memory, the latency associated with page operations is very low. Do you know if Derby is using the CPU for something useful (i.e. processing queries effectively), or is the CPU spent on wasteful activities? One potential wasteful activity is moving data back and forth from the page cache (page cache too small?). Is this a highly concurrent scenario? Regards, -- Kristian Thank you, Pavel. Jefferies archives and monitors outgoing and incoming e-mail. The contents of this email, including any attachments, are confidential to the ordinary user of the email address to which it was addressed. If you are not the addressee of this email you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. This email may be produced at the request of regulators or in connection with civil litigation. Jefferies accepts no liability for any errors or omissions arising as a result of transmission. Use by other than intended recipients is prohibited. In the United Kingdom, Jefferies operates as Jefferies International Limited; registered in England: no. 1978621; registered office: Vintners Place, 68 Upper Thames Street, London EC4V 3BJ. Jefferies International Limited is authorised and regulated by the Financial Services Authority. Jefferies archives and monitors outgoing and incoming e-mail. The contents of this email, including any attachments, are confidential to the ordinary user of the email address to which it was addressed. If you are not the addressee of this email you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. This email may be produced at the request of regulators or in connection with civil litigation. Jefferies accepts no liability for any errors or omissions arising as a result of transmission. Use by other than intended recipients is prohibited. In the United Kingdom, Jefferies operates as Jefferies International Limited;
