[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #1 from Benoît Robin --- Seemingly regardless of the number of operations to catgorize, CPU surge is between 25-35s. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #2 from Benoît Robin --- Also remarked there is no network activity at all during the CPU surge. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #3 from Benoît Robin --- Removing history reduces file size to 3.5 MB, but doesn't change anything to the CPU surge. I also remarked that, when opening the file, there was a brief (5s) network peak, corresponding to the file read, then the CPU climbed to one core 100% and stayed there for the usual 30-some seconds. Still no network activity during CPU surge. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #4 from Benoît Robin --- Copied the file to a local disk and tested. Just the same CPU surge as before, so my conclusion is now that NETWORK IS NOT GUILTY ! Seems my first observation of a second-long surge was wrong... It's a memory from 3 weeks ago, so don't take it into account... Unfortunately, I now have no box left on the old Skrooge version to check with my save of the old file before migration. Btw, I found the old version number : it was 1.9.3 ! -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 Stephane MANKOWSKI changed: What|Removed |Added Ever confirmed|0 |1 Status|UNCONFIRMED |ASSIGNED Assignee|guillaume.deb...@gmail.com |steph...@mankowski.fr --- Comment #5 from Stephane MANKOWSKI --- Hi, Could you try this from a console? export SKGTRACEPERFO=1 skrooge Play your scenario except the action causing the issue. Click on Tools/Restart Profiling Play the last action causing the issue. Click on Tools/Open Profiling ==> A page has been opened. Copy and paste the content here. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #6 from Stephane MANKOWSKI --- Did you do the test? -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #7 from Stephane MANKOWSKI --- Did you do the test? I need your help to be able to correct this potential issue. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #8 from Christoph Feck --- If you can provide the information requested in comment #5, please add it. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 Christoph Feck changed: What|Removed |Added Status|ASSIGNED|NEEDSINFO Resolution|--- |WAITINGFORINFO --- Comment #9 from Christoph Feck --- To further investigate this issue, KDE developers need the information requested in comment #5. If you can provide it, or need help with finding that information, please add a comment. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #10 from Benoît Robin --- Hi Stephane, just read your comments, I'll run the test when I got time, tomorrow evening at best or by next week-end at worst. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #11 from Benoît Robin --- Created attachment 107224 --> https://bugs.kde.org/attachment.cgi?id=107224&action=edit Profiling when the issue occurs Scenario : - Console : export SKGTRACEPERFO=1, - Launch Skrooge, - Open my accounting file : shows around 35s of CPU max, - Go to any account, - Tools > Restart Profiling, - Change the category on an operation : shows around 35s of CPU max, - Tools > Open Profiling, - Copy and Paste in 2017-08-12-Profiling-Report.csv file. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #12 from Benoît Robin --- Hi Stephane, hi Christoph, Just attached the profiling information. Hope it'll help you to do Skuld job on the bug ^-^ Best regards, Benoît 'Mutos' Robin -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 Stephane MANKOWSKI changed: What|Removed |Added Status|NEEDSINFO |ASSIGNED Resolution|WAITINGFORINFO |--- --- Comment #13 from Stephane MANKOWSKI --- Hi, This helps a lot. Indeed, we can see that line: virtual SKGAdviceList SKGPayeePlugin::advice(const QStringList&) ;1; 32888.54614 ; 32888.54614 ; 32888.54614 ; 32888.54614 ; 32888.54614 ; 32888.54614 It means that the computation of the advices around "Payee" took 32000 ms (32s). This is really interesting. First: You certainly have a dashboard opened when you do a modification on operation. On this dashboard, you have the "Advice" widget. This widget takes time. So you have 2 workarounds: 1- You can close your dashboard when you do a modification but you will still have bad performances when you will open it again. 2- You can remove the "Advice" widget from the dashboard. Second: I published today a new version of Skrooge (2.9.0). On this version, the dashboard is not refreshed when this is not the displayed page. This should be better for you. It means that you don't have to close the dashboard when you do a modification on operations but you will still have bad performances when you will switch on dashboard. Last: This could be interesting to find why the "Payee" advice has so bad performances. Could you do that? export SKGTRACESQL=1 run skrooge, play you scenario and send me the traces (you should see the SQL query causing issue) Could you give me the number of payees you have in your document too? Regards. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #14 from Stephane MANKOWSKI --- One last question: Are you french? I am. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #15 from Stephane MANKOWSKI --- On last comment: In 2.9.0, I did the correction Better performance in advice "Some payees seem to be identical". This new version should be good for you. To install it: https://launchpad.net/~s-mankowski/+archive/ubuntu/ppa-kf5 -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #16 from Benoît Robin --- Hi Stephane, Thanks for the fix and advice ! This morning, I remarked a skrooge update passed and thought you might have found the issue ^-^ I don't know what is a payee, so I can't answer your question about their number. I'll search that. Accounting for me is just importing my bank's files, marking operations with categories and from time to time issuing a report of category summaries. But indeed I have a dashboard with the "advices" widget. I don't need that widget anyway, but it was there, so I kept it. I'll remove it and that's all... Anyway, I'll run the same scenario with the up-trace and we'll see what comes out of it... -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #17 from Benoît Robin --- Hi Stéphane, And yes, I am French ^-^ As for the test, as the update to 2.9.0 passed on my main computer, I now just can't get the bug anymore. I tried to refresh the advices widget, even turned on all advices, but to no avail. Oh, and as for payees, in French it should be what is called a "tiers", if I understand well. I never bothered to look at those, they're not even displayed in my custom operation format. I'll try on another computer, which didn't get the update, and I give you what I found. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #18 from Benoît Robin --- Created attachment 107245 --> https://bugs.kde.org/attachment.cgi?id=107245&action=edit Report of Profiling + SQL Hi Stéphane, I doubled up with the previous trace and the file, which is copied from the console, contains both trace and SQL. This is probably redundant. >From your diagnostic, I guess the statement that causes the issue is the following : ##executeSqliteOrder:SELECT EXISTS(SELECT 1 FROM payee WHERE EXISTS (SELECT 1 FROM payee p2 WHERE p2.id<>payee.id AND upper(p2.t_name)=upper(payee.t_name) AND p2.t_name<>payee.t_name)) TIME=27582 ms, (with fetch):27582 ms It was executed twice, which matches with my observing two CPU max-outs, one at startup and one when I updated the operation's category. Hope that could help you improve that operation. It seems a quite simple brute-force statement on all the base and I'm not SQL-wise enough to know if such a seemingly simple request can be improved ^-^ Benoît 'Mutos' Robin -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #19 from Benoît Robin --- Ah, and I'll keep 2.8.1 on the laptop until you tell me you don't need further information... -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #20 from Stephane MANKOWSKI --- Hi Benoit, Thank you I have the confirmation of my analysis. You can install Skrooge 2.9.0. You should not have the issue. Could you confirm here after the update? Regards. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #21 from Benoît Robin --- Hi Stéphane, I confirm 2.9.0, which is already installed onmy desktop machine, doesn't have the issue anymore. I'll install it on the laptop. Merci de ton aide ^-^ Btw, who can close the case ? -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 Stephane MANKOWSKI changed: What|Removed |Added Resolution|--- |FIXED Status|ASSIGNED|RESOLVED --- Comment #22 from Stephane MANKOWSKI --- Already fixed 2.9.0. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #8 from Christoph Feck --- If you can provide the information requested in comment #5, please add it. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 Christoph Feck changed: What|Removed |Added Status|ASSIGNED|NEEDSINFO Resolution|--- |WAITINGFORINFO --- Comment #9 from Christoph Feck --- To further investigate this issue, KDE developers need the information requested in comment #5. If you can provide it, or need help with finding that information, please add a comment. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #10 from Benoît Robin --- Hi Stephane, just read your comments, I'll run the test when I got time, tomorrow evening at best or by next week-end at worst. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #11 from Benoît Robin --- Created attachment 107224 --> https://bugs.kde.org/attachment.cgi?id=107224&action=edit Profiling when the issue occurs Scenario : - Console : export SKGTRACEPERFO=1, - Launch Skrooge, - Open my accounting file : shows around 35s of CPU max, - Go to any account, - Tools > Restart Profiling, - Change the category on an operation : shows around 35s of CPU max, - Tools > Open Profiling, - Copy and Paste in 2017-08-12-Profiling-Report.csv file. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #12 from Benoît Robin --- Hi Stephane, hi Christoph, Just attached the profiling information. Hope it'll help you to do Skuld job on the bug ^-^ Best regards, Benoît 'Mutos' Robin -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 Stephane MANKOWSKI changed: What|Removed |Added Status|NEEDSINFO |ASSIGNED Resolution|WAITINGFORINFO |--- --- Comment #13 from Stephane MANKOWSKI --- Hi, This helps a lot. Indeed, we can see that line: virtual SKGAdviceList SKGPayeePlugin::advice(const QStringList&) ;1; 32888.54614 ; 32888.54614 ; 32888.54614 ; 32888.54614 ; 32888.54614 ; 32888.54614 It means that the computation of the advices around "Payee" took 32000 ms (32s). This is really interesting. First: You certainly have a dashboard opened when you do a modification on operation. On this dashboard, you have the "Advice" widget. This widget takes time. So you have 2 workarounds: 1- You can close your dashboard when you do a modification but you will still have bad performances when you will open it again. 2- You can remove the "Advice" widget from the dashboard. Second: I published today a new version of Skrooge (2.9.0). On this version, the dashboard is not refreshed when this is not the displayed page. This should be better for you. It means that you don't have to close the dashboard when you do a modification on operations but you will still have bad performances when you will switch on dashboard. Last: This could be interesting to find why the "Payee" advice has so bad performances. Could you do that? export SKGTRACESQL=1 run skrooge, play you scenario and send me the traces (you should see the SQL query causing issue) Could you give me the number of payees you have in your document too? Regards. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #14 from Stephane MANKOWSKI --- One last question: Are you french? I am. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #15 from Stephane MANKOWSKI --- On last comment: In 2.9.0, I did the correction Better performance in advice "Some payees seem to be identical". This new version should be good for you. To install it: https://launchpad.net/~s-mankowski/+archive/ubuntu/ppa-kf5 -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #16 from Benoît Robin --- Hi Stephane, Thanks for the fix and advice ! This morning, I remarked a skrooge update passed and thought you might have found the issue ^-^ I don't know what is a payee, so I can't answer your question about their number. I'll search that. Accounting for me is just importing my bank's files, marking operations with categories and from time to time issuing a report of category summaries. But indeed I have a dashboard with the "advices" widget. I don't need that widget anyway, but it was there, so I kept it. I'll remove it and that's all... Anyway, I'll run the same scenario with the up-trace and we'll see what comes out of it... -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #17 from Benoît Robin --- Hi Stéphane, And yes, I am French ^-^ As for the test, as the update to 2.9.0 passed on my main computer, I now just can't get the bug anymore. I tried to refresh the advices widget, even turned on all advices, but to no avail. Oh, and as for payees, in French it should be what is called a "tiers", if I understand well. I never bothered to look at those, they're not even displayed in my custom operation format. I'll try on another computer, which didn't get the update, and I give you what I found. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #18 from Benoît Robin --- Created attachment 107245 --> https://bugs.kde.org/attachment.cgi?id=107245&action=edit Report of Profiling + SQL Hi Stéphane, I doubled up with the previous trace and the file, which is copied from the console, contains both trace and SQL. This is probably redundant. >From your diagnostic, I guess the statement that causes the issue is the following : ##executeSqliteOrder:SELECT EXISTS(SELECT 1 FROM payee WHERE EXISTS (SELECT 1 FROM payee p2 WHERE p2.id<>payee.id AND upper(p2.t_name)=upper(payee.t_name) AND p2.t_name<>payee.t_name)) TIME=27582 ms, (with fetch):27582 ms It was executed twice, which matches with my observing two CPU max-outs, one at startup and one when I updated the operation's category. Hope that could help you improve that operation. It seems a quite simple brute-force statement on all the base and I'm not SQL-wise enough to know if such a seemingly simple request can be improved ^-^ Benoît 'Mutos' Robin -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #19 from Benoît Robin --- Ah, and I'll keep 2.8.1 on the laptop until you tell me you don't need further information... -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #20 from Stephane MANKOWSKI --- Hi Benoit, Thank you I have the confirmation of my analysis. You can install Skrooge 2.9.0. You should not have the issue. Could you confirm here after the update? Regards. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #21 from Benoît Robin --- Hi Stéphane, I confirm 2.9.0, which is already installed onmy desktop machine, doesn't have the issue anymore. I'll install it on the laptop. Merci de ton aide ^-^ Btw, who can close the case ? -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 Stephane MANKOWSKI changed: What|Removed |Added Resolution|--- |FIXED Status|ASSIGNED|RESOLVED --- Comment #22 from Stephane MANKOWSKI --- Already fixed 2.9.0. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #1 from Benoît Robin --- Seemingly regardless of the number of operations to catgorize, CPU surge is between 25-35s. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #2 from Benoît Robin --- Also remarked there is no network activity at all during the CPU surge. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #3 from Benoît Robin --- Removing history reduces file size to 3.5 MB, but doesn't change anything to the CPU surge. I also remarked that, when opening the file, there was a brief (5s) network peak, corresponding to the file read, then the CPU climbed to one core 100% and stayed there for the usual 30-some seconds. Still no network activity during CPU surge. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #4 from Benoît Robin --- Copied the file to a local disk and tested. Just the same CPU surge as before, so my conclusion is now that NETWORK IS NOT GUILTY ! Seems my first observation of a second-long surge was wrong... It's a memory from 3 weeks ago, so don't take it into account... Unfortunately, I now have no box left on the old Skrooge version to check with my save of the old file before migration. Btw, I found the old version number : it was 1.9.3 ! -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 Stephane MANKOWSKI changed: What|Removed |Added Ever confirmed|0 |1 Status|UNCONFIRMED |ASSIGNED Assignee|guillaume.deb...@gmail.com |steph...@mankowski.fr --- Comment #5 from Stephane MANKOWSKI --- Hi, Could you try this from a console? export SKGTRACEPERFO=1 skrooge Play your scenario except the action causing the issue. Click on Tools/Restart Profiling Play the last action causing the issue. Click on Tools/Open Profiling ==> A page has been opened. Copy and paste the content here. -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #6 from Stephane MANKOWSKI --- Did you do the test? -- You are receiving this mail because: You are watching all bug changes.
[skrooge] [Bug 382152] Skrooge over network data maxes CPU on one core during several seconds when setting an operation's category
https://bugs.kde.org/show_bug.cgi?id=382152 --- Comment #7 from Stephane MANKOWSKI --- Did you do the test? I need your help to be able to correct this potential issue. -- You are receiving this mail because: You are watching all bug changes.
