[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Jaime Torres]

https://bugs.kde.org/show_bug.cgi?id=384561

--- Comment #22 from Jaime Torres  ---
Another recently discovered solution to avoid thousands of stat(/etc/timezone)
is to set the environment variable TZ, "export TZ=:/etc/localtime"
seen in https://news.ycombinator.com/item?id=13697555

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Christoph Feck]

https://bugs.kde.org/show_bug.cgi?id=384561

--- Comment #21 from Christoph Feck  ---
Let me add that we rather prefer people reporting duplicates instead of not
reporting issues just because they fear they could report a bug that is already
known. The work involved to mark a duplicate is neglegible compared to work for
any other bugs.

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Nate Graham]

https://bugs.kde.org/show_bug.cgi?id=384561

--- Comment #20 from Nate Graham  ---
In fact, I just wrote a new page for how to file a good bug:
https://community.kde.org/Get_Involved/Bug_Reporting

We are *very* interested in making life easier for people who want to get
involved in bug triaging.

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Mahendra Tallur]

https://bugs.kde.org/show_bug.cgi?id=384561

Mahendra Tallur  changed:

   What|Removed |Added

 CC||ma...@free.fr

--- Comment #19 from Mahendra Tallur  ---
Hi, thanks so much for the fix ! I was affected too but always thought the
issue was the USB stack or something else.

Regarding Nate, Massimo & Jaime's exchange : I must say, as a user and
occasional reporter, the bugtracker is a bit overwhelming. Despite my being
quite careful, I filed dozens of duplicates, which made me feel very bad. It
gave me the feeling of wasting the other volunteers' time. 

Some other times, I think I didn't file against the right component and the
bugreport was kinda lost.

So I'm wondering 
1) is there something in the bugtracker that is intrinsically misleading and 
adds work to triagers & devs ?
2) or should we help "educate" the users such as me so they file fewer reports,
fewer duplicates and target them properly ? 
3) the current "usability & productivity" campaign makes me extremely
enthusiastic. Could it set the groundwork for a long-term meta-organization of
the triaging / fixing work ?

Thanks again so much to everyone involved. Exciting times :-)

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Massimo Callegari]

https://bugs.kde.org/show_bug.cgi?id=384561

--- Comment #18 from Massimo Callegari  ---
Hey guys, I have repeated both the initial tests: cp vs Dolphin

KDE Plasma 5.12, KDE Frameworks 5.43.0, Qt 5.9.3, Dolphin 17.12.2

cp took 7 minutes, 31 seconds
Dolphin took 9 minutes, 57 seconds

Excellent result !

@Jaime Torres, thanks a lot again for looking into this. I'm pretty sure many
users will appreciate this performance gain as much as I do !

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Jaime Torres]

https://bugs.kde.org/show_bug.cgi?id=384561

--- Comment #17 from Jaime Torres  ---
Git commit 8d73867b3d4339ec026f92c4c2e89c265249aeb2 by Jaime Torres.
Committed on 07/02/2018 at 18:05.
Pushed by jtamate into branch 'master'.

Don't stat(/etc/localtime) between read() and write() copying files

Summary:

Unfortunately, QDateTime::currentDateTime() checks /etc/localtime
each time it is called.
Chaning to QElapsedTime, no check of /etc/localtime.
Reproducing bug 384561, the strace of file.so was something like:
read(), stat(/etc/localtime), stat(/etc/localtime),
stat(/etc/localtime), stat(/etc/localtime), stat(/etc/localtime),
write(), read() ..
Now it is: read(), write()
It also reduces the cpu in io/wait around 10% in a debug build.

Test Plan:
kio tests work as before
desktop: works in dolphin

Reviewers: #frameworks, fvogt, dfaure

Reviewed By: dfaure

Subscribers: fvogt, ngraham

Tags: #frameworks

Differential Revision: https://phabricator.kde.org/D9983

M  +14   -13   src/core/slavebase.cpp

https://commits.kde.org/kio/8d73867b3d4339ec026f92c4c2e89c265249aeb2

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Nate Graham]

https://bugs.kde.org/show_bug.cgi?id=384561

--- Comment #16 from Nate Graham  ---
Yes, unifying duplicates is a big job. I've done a lot of bug screening in my
time, in Bugzilla as well as others, and I doubt hunting for duplicates can be
done by anyone other than a human being.

In general, KDE suffers from a lack of what you might call corporate
professionalism: we don't have many paid programmers or any paid bug screeners;
we don't have project managers to keep projects on task; we don't categorize
bugs based on priority; etc. There are certain projects like Plasmashell that
are run closer to this model, but a lot of KDE is run like a volunteer
organization because, well, that's what it is!

My personal opinion is that a bit more discipline would really help us, if we
can apply it without making the project un-fun to work on, which would defeat
the purpose by repelling volunteers. That's mostly a cultural matter--getting
people to screen bugs for projects they work on and use the Priority field--but
I think having at least one overall release manager would be hugely beneficial,
as that person would have a high level overview of the whole community and
could help keep things on track, which ultimately increases the quality of our
products and generates more excitement and volunteer contribution.

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Jaime Torres]

https://bugs.kde.org/show_bug.cgi?id=384561

--- Comment #15 from Jaime Torres  ---
Thank you for your report and for your help making KDE software great.

Bugzilla, as far as I know, does have priorities: by project, by component, by
user votes, by number of duplicates, by keywords, by tags, by date, ...

The main problem is that there are lots of duplicated bugs that right now only
humans mark as such, even the previous search as you type a new bug is
insufficient (I know because I've reported duplicated bugs myself), and this
manual task consumes a lot of time.
With the improvements of the so called deep learning, is there any free
software to search for duplicated reports that is as good as humans or better?

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Nate Graham]

https://bugs.kde.org/show_bug.cgi?id=384561

--- Comment #14 from Nate Graham  ---
We get 30-40 new bugs every day. Since you are a developer, you know how
important bug screening is. So why don't you help us screen the bugs? That's
what I do (in addition to submitting patches and organizing a KDE-wide effort
to address longstanding or high-impact bugs, like this one). I do it with a
smile, too, because negativity is destructive.

This isn't a commercial company. People donate their time because they love the
project. Filing good bug reports is very important--and we appreciate that very
much! But as you've observed, we have more than 27,000 bugs and we need help to
triage them. With your development expertise, I'm sure you could submit some
patches, too! And if you need a hand setting up a KDE development environment,
I'd be happy to help; just shoot me an email.

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Dr . Chapatin]

https://bugs.kde.org/show_bug.cgi?id=384561

Dr. Chapatin  changed:

   What|Removed |Added

 CC||bugsefor...@gmx.com

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Massimo Callegari]

https://bugs.kde.org/show_bug.cgi?id=384561

--- Comment #13 from Massimo Callegari  ---
First and foremost, @Jaime Torres: thanks a lot for looking into this !

> @Massimo: It would be nice as feedback, if the upcoming KDE Frameworks v5.43
> fixes this for you - just to be sure your problem is truly fixed :-)

Will surely do.

I tried to avoid flaming this thread, but now I'd like to clarify my comment
#3.
I never asked for a fix in 3 months. I am a (Qt) developer too and I know how
this works.
However, as a developer I try to quickly identify the real issues from the fake
ones. In a few days/weeks I try at least to reproduce the issue and give it a
priority.
This way I know where I have to concentrate my efforts. Moreover if I'm working
on something, I tend to forget everything else, so "marking" an issue with a
priority helps my memory months later.

What I complained about is that in 3 months no one changed the status of this
issue to 'confirmed', meaning (to me) no one tried to reproduce it.
This is the best way to loose an issue report that can help KDE to get better
among other 27000+.

And because of this, I found Nate's comment quite ridiculous:
> There are currently 27,062 open bugs across all of KDE[1]. This seems like an 
> important one, to be sure, but it competes with 27,061 others.

This means there is no priority assignment and all those 27062 bugs have the
same weight.

As for me fixing the issue...no way !
I already contributed to KDE many years ago (Kate's Symbol Viewer plugin is
mine) and I know how difficult is even to prepare the KDE build environment.
Plus, I am already swamped with my project issues, and I believe I already did
my job -as user- by dedicating time to open this issue report as much
accurately as possible.

I am frustrated, yes, because I use KDE since 15 years and as a user it hurts
me to see how Plasma is still not in shape, even after a dozen of minors.
I see changelogs of new versions saying "new breeze theme, new panel design,
etc" and it makes me angry when I know there are core issues to be fixed first.

In conclusion, no harsh from my side. I know for sure there are excellent
developers behind KDE that dedicate an awful amount of their personal time to
this project.
I only wish there could be a better priority system in place.

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Jaime Torres]

https://bugs.kde.org/show_bug.cgi?id=384561

--- Comment #12 from Jaime Torres  ---
I just want you to notice that not every developer has a privative filesystem
available, and it took me "only" two weeks to investigate and reduce the time
spent copying files, and is not over yet.
The other bug, that I noticed sometimes reproducing this one, is even harder to
fix.

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Jonathan Verner]

https://bugs.kde.org/show_bug.cgi?id=384561

Jonathan Verner  changed:

   What|Removed |Added

 CC||jonathan.ver...@gmail.com

--- Comment #11 from Jonathan Verner  ---
(In reply to Alexander Nestorov from comment #10)
> "3 months and nobody cares" you say? HAHAHAHAHA
> 
> https://bugs.kde.org/show_bug.cgi?id=342056
> 
> 3 years :)

Yes, I know it feels like a long time, but please see Nate's comment above. A
simple thank you would go a long way... Try to imagine what it feels like to be
a developer, who invests a nontrivial amount of time into fixing this bug only
to get a snarky "it took 3 years" comment.

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Alexander Nestorov]

https://bugs.kde.org/show_bug.cgi?id=384561

Alexander Nestorov  changed:

   What|Removed |Added

 CC||alexander...@gmail.com

--- Comment #10 from Alexander Nestorov  ---
"3 months and nobody cares" you say? HAHAHAHAHA

https://bugs.kde.org/show_bug.cgi?id=342056

3 years :)

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Dominik Haumann]

https://bugs.kde.org/show_bug.cgi?id=384561

Dominik Haumann  changed:

   What|Removed |Added

   Version Fixed In||KDE Frameworks 5.43

--- Comment #9 from Dominik Haumann  ---
@Massimo: It would be nice as feedback, if the upcoming KDE Frameworks v5.43
fixes this for you - just to be sure your problem is truly fixed :-)

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Jaime Torres]

https://bugs.kde.org/show_bug.cgi?id=384561

Jaime Torres  changed:

   What|Removed |Added

 Resolution|--- |FIXED
 Status|CONFIRMED   |RESOLVED
  Latest Commit||https://commits.kde.org/kjo
   ||bwidgets/240a9c1f01d0f11638
   ||be61e0869a494c7f3ab697

--- Comment #8 from Jaime Torres  ---
Git commit 240a9c1f01d0f11638be61e0869a494c7f3ab697 by Jaime Torres.
Committed on 15/01/2018 at 18:39.
Pushed by jtamate into branch 'master'.

Code de-duplication in byteSize(double size)

Summary:
This code is used to pretty print the amount of data copied.
Replace this duplicated code by calling formatByteSize from KFormat.

Test Plan:

Reviewers: #frameworks, mwolff

Reviewed By: mwolff

Subscribers: ngraham, cfeck, mwolff, broulik

Tags: #frameworks

Differential Revision: https://phabricator.kde.org/D9770

M  +3-25   src/kjobtrackerformatters.cpp

https://commits.kde.org/kjobwidgets/240a9c1f01d0f11638be61e0869a494c7f3ab697

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Jaime Torres]

https://bugs.kde.org/show_bug.cgi?id=384561

--- Comment #7 from Jaime Torres  ---
Git commit f7e00b40a6d35fbfe536cdb99d6f22b643676e5e by Jaime Torres.
Committed on 14/01/2018 at 12:39.
Pushed by jtamate into branch 'master'.

Don't stat(/etc/localtime) between read() and write() copying files

Summary:

Unfortunately, QDateTime::currentDateTime() checks /etc/localtime
each time it is called.
Chaning to QElapsedTime, no check of /etc/localtime.
Reproducing bug 384561, the strace of file.so was something like:
read(), stat(/etc/localtime), stat(/etc/localtime),
stat(/etc/localtime), stat(/etc/localtime), stat(/etc/localtime),
write(), read() ..
Now it is: read(), write()
It also reduces the cpu in io/wait around 10% in a debug build.

Reviewers: #frameworks, dfaure

Reviewed By: dfaure

Subscribers: broulik, ngraham, #dolphin

Tags: #frameworks

Differential Revision: https://phabricator.kde.org/D9844

M  +13   -12   src/core/slavebase.cpp

https://commits.kde.org/kio/f7e00b40a6d35fbfe536cdb99d6f22b643676e5e

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Marijo]

https://bugs.kde.org/show_bug.cgi?id=384561

Marijo  changed:

   What|Removed |Added

 CC||mustacmar...@hotmail.de

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Jaime Torres]

https://bugs.kde.org/show_bug.cgi?id=384561

Jaime Torres  changed:

   What|Removed |Added

 Ever confirmed|0   |1
 Status|UNCONFIRMED |CONFIRMED

--- Comment #6 from Jaime Torres  ---
With iotop or ksysguard and plots about I/O, there are huge differences.
with cp/mv, no process is waiting for io, with dolphin plasma spends a lot of
time waiting for io (column io> value 99%).
with cp/mv, the i/o plots are more or less constant at some i/o speed , but
with dolphin, at the beginning there are "mountains" and later on only peaks.

In the code:
* the cp/mv copy buffer is 128Kib allocated in the heap and is memory aligned
* the current buffer used by dolphin is 32Kib in the stack and is not
guaranteed to be memory aligned.

I've tried changing the buffer to the heap, with 128Kib, and aligned and it
improves a little bit the problem at the beginning, but after a while the
performance drops and the peaks are shown again in ksysguard.

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Jaime Torres]

https://bugs.kde.org/show_bug.cgi?id=384561

Jaime Torres  changed:

   What|Removed |Added

 CC||jtam...@gmail.com

--- Comment #5 from Jaime Torres  ---
The main difference between cp/mv and file.so (copying/moving from dolphin) to
a hdd ntfs partition, as strace tells, is that:
  cp/mv only does this loop: 
  * read from one file and write to another.
  while file.so does this loop:
  * read from one file, write to another file and stat /etc/localtime 5 times.

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Nate Graham]

https://bugs.kde.org/show_bug.cgi?id=384561

Nate Graham  changed:

   What|Removed |Added

   Keywords||usability

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Nate Graham]

https://bugs.kde.org/show_bug.cgi?id=384561

--- Comment #4 from Nate Graham  ---
I understand your frustration, but aiming verbal abuse at the people who you
are expecting to perform work for free on an item of your choice isn't the best
way to get it done. There are currently 27,062 open bugs across all of KDE[1].
This seems like an important one, to be sure, but it competes with 27,061
others.

If you want to get this done faster, you're always welcome to submit a patch
using http://phabricator.kde.org/. If you don't feel up to the task, perhaps
you could locate someone else willing and able to have a look? Or polish your
own C++ and Qt skills. 4 months ago I had never written a line or c++ in my
life, yet by today I've had dozens of patches accepted:
https://phabricator.kde.org/people/commits/11806/

But please understand that FOSS projects like this one (And GNOME too, to be
fair) are always limited for resources. The absolute best way to improve this
situation is to lend some resources of your own to the effort. Possible avenues
include submitting patches, screening bugs, donating financially[2], etc. But
posting anger and negativity in bug reports works against your desire because
it makes people *less* likely to do the work. I am an unpaid volunteer. I am
choosing to screen bugs for the benefit of others instead of playing with my
kids right now on this lovely Saturday morning. Do you think it makes me want
to screen bugs more or less when I wake up to an email notification that people
like me "don't actually listen to their software users" and that the project I
choose to work on for no money "is degrading into something unusable" ?

Don't let cynicism and negativity drag you down. Nothing improves without
effort. There's a huge amount to do here--climb aboard, and we'd be happy to
have you!

https://www.kde.org/get-involved
https://www.kde.org/develop


[1] https://bugs.kde.org/weekly-bug-summary.cgi
[2] https://www.kde.org/donations

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Massimo Callegari]

https://bugs.kde.org/show_bug.cgi?id=384561

--- Comment #3 from Massimo Callegari  ---
Almost 3 months and nobody cares about this.
No comments, no status change.
No wonder why KDE is so buggy when developers don't actually listen to their
software users.
I use KDE since 15 years and instead of improving, KDE is degrading into
something unusable.

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Nate Graham]

https://bugs.kde.org/show_bug.cgi?id=384561

Nate Graham  changed:

   What|Removed |Added

 CC||pointedst...@zoho.com

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Simon Andric]

https://bugs.kde.org/show_bug.cgi?id=384561

Simon Andric  changed:

   What|Removed |Added

 CC||simonandr...@gmail.com

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Elvis Angelaccio]

https://bugs.kde.org/show_bug.cgi?id=384561

Elvis Angelaccio  changed:

   What|Removed |Added

Version|17.08.0 |unspecified
Product|dolphin |frameworks-kio
 CC||kdelibs-b...@kde.org
   Assignee|dolphin-bugs-n...@kde.org   |fa...@kde.org
  Component|general |general

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-kio] [Bug 384561] Dolphin copy operation is 4,5x slower than cp

[Elvis Angelaccio]

https://bugs.kde.org/show_bug.cgi?id=384561

Elvis Angelaccio  changed:

   What|Removed |Added

Version|unspecified |5.37.0

-- 
You are receiving this mail because:
You are watching all bug changes.