[Bug 541937] Re: copying files to an usb pandrive is slow and slows down in time (Lucid Lynx)
*** This bug is a duplicate of bug 197762 *** https://bugs.launchpad.net/bugs/197762 No, no, no! Please, can we not tell people to dogpile onto bug #197762? That bug is so dogpiled as to be useless. We need to have separate bugs for separate performance problems, instead of assuming that all reported performance problems with USB drives should be directed to a bug that multiple people have been trying to shut down as there's no intelligent life here, Scotty, which was opened over two years ago, and for which there are at least three separate and distinct problems, and probably more, all horribly mixed together. Can someone official in Ubuntu step up and say whether or not this horrible practice should be continued? I'm seriously thinking about writing a blog entry, Why Launchpad is terminally broken, and why all upstream devs should ignore it as a waste of their time, and pointing to #197762, and misguided attempts to get people to dogpile onto this misbegotten bug, as a exhibit number #1 as to why kernel developers are ignoring Launchpad and why it is more of a menance than a help to upstream developers. If canonical isn't providing upstream development assistance, and they're not providing enough talented people to do bug triage, and they are claiming that the value they are bringing to upstream projects is lots of users and lots of bug reports, this is a prime example of why This Isn't Working For Us. -- copying files to an usb pandrive is slow and slows down in time (Lucid Lynx) https://bugs.launchpad.net/bugs/541937 You received this bug notification because you are a member of Ubuntu Desktop Bugs, which is a bug assignee (via bug 197762). -- desktop-bugs mailing list desktop-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/desktop-bugs
[Bug 197762] Re: file transfers on USB flash key (pendrive) or USB HDD are slowing down with time
Comment which I added to bug #541937: No, no, no! Please, can we not tell people to dogpile onto bug #197762? That bug is so dogpiled as to be useless. We need to have separate bugs for separate performance problems, instead of assuming that all reported performance problems with USB drives should be directed to a bug that multiple people have been trying to shut down as there's no intelligent life here, Scotty, which was opened over two years ago, and for which there are at least three separate and distinct problems, and probably more, all horribly mixed together. Can someone official in Ubuntu step up and say whether or not this horrible practice should be continued? I'm seriously thinking about writing a blog entry, Why Launchpad is terminally broken, and why all upstream devs should ignore it as a waste of their time, and pointing to #197762, and misguided attempts to get people to dogpile onto this misbegotten bug, as a exhibit number #1 as to why kernel developers are ignoring Launchpad and why it is more of a menance than a help to upstream developers. If canonical isn't providing upstream development assistance, and they're not providing enough talented people to do bug triage, and they are claiming that the value they are bringing to upstream projects is lots of users and lots of bug reports, this is a prime example of why This Isn't Working For Us. -- file transfers on USB flash key (pendrive) or USB HDD are slowing down with time https://bugs.launchpad.net/bugs/197762 You received this bug notification because you are a member of Ubuntu Desktop Bugs, which is a bug assignee. -- desktop-bugs mailing list desktop-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/desktop-bugs
Re: [Bug 197762] Re: file transfers on USB flash key (pendrive) or USB HDD are slowing down with time
On Tue, Apr 20, 2010 at 09:06:13PM -, Jeremy Foshee wrote: Ted, I couldn't agree more. I'm not sure I fall into what you would call someone official in Ubuntu, but I am the Kernel Bug Triager. My stance on these is to have reporters, with any deviation from the reported bug they think is affecting them, file a new bug while subscribing to the bug they find similar. In this fashion, they are able to test fixes against their reported issue for the other reported bugs. The difficulty is in the push to consolidate that seems to be afflicting some bug tragers. This is difficult to address, as I am sure you can imagine. So let me suggest some technological fixes first --- not that they will completely solve the problem, but Launchpad desperately needs them. The first is I think we need to have a way for someone to put a statement at the very top of the Launchpad bug. Either right before or right after the bug description. This would be a place for someone to put a message such as for people who think that they have this problem, please see this wiki page first, which will describe how to do more bug-specific diagnosis --- i.e., how to collect various bits of diagnostic data and then to suggest which of several subsidary bugs that they can check the this bug affects me too box. One of the major problem is that Launchpad is !...@#@! slow once it has more than several hundred comments, and if people try to put helpful information in the two hundred and seventieth comment, most users never bother to read that far. So you really need a way to put information to users who are first getting redirected to the bug that they will actually *see*. The other thing that you really need is a way for a bug to be closed. A bug which is closed is one which can't be used as a duplicate, and for which no one but an admin is allowed to post further comments, or change the state of the bug, or add some other package as being affected by a bug. It's basically a way of saying, This bug is a EPA Superfund toxic waste site, and please open a new bug if you think you have a similar problem. Combined with the first feature, where someone can place a message explaining why a bug is closed, and what people should do, would be a big, big help. Another thing that might be useful (and which would also require some number of triagers to have privilege bits; you can't give it out to everyone), is some way to mark certain comments as being irrelevant. If the bug is really about USB1 vs. USB2 detection, the last thing you need is for someone to say, me too! I have a HDD performance problem when using XFS. When we have a vast number of non-technical users, we desperately need a way to moderate out the crap comments. Maybe for political reasons they are displayed by default, but if an upstream developer is going to viewing the bug, they really will want a way to hit a button and get rid of the irrelevant comments. (Maybe there would be a moderation reason where the triager can say, unrelated problem; please open a new bug report with full details of your hardware and software configuration before marking a comment with the crap-and-should-be-hidden flag.) Other thing which bug triagers should do is be more ready to change the bug title. Bug titles which talk about random performance problem practically *invite* dogpiling. If the problem has been localized to a USB1 vs USB2 detection problem in the driver layer, then change the bug description title to say that. Otherwise, people will see performance probem, and say, hey, the gross symptoms match mine, this must be the place for me to tail about how horrible Ubuntu is and how everyone should switch to Windows instead. Another problem is that right now, more often than not, the vast majority of the bugs that I see are wrongly assigned wrong package. I've lost package of the time an init scripts or Plymouth bug is wrongly assigned to e2fsprogs. Or when a bug in dpkg is wrongly assigned to libss. If users aren't going to get it right, maybe they don't get assign a package name until they pass a test that shows they really can get it right. Or maybe certain people can be put on a list so that they don't get notified via e-mail until a triager verifies that a bug is actually real, and not crap. Right now, the ratio of valid bug e-mails coming from Launchpad to invalid ones is easily 50 to 1 if not higher. If we can't fix it, that's fine. I'll write my blog posting about why Launchpad is worthless, and I'll desubscribe myself from all Launchpad reports, and I'll move on - Ted -- file transfers on USB flash key (pendrive) or USB HDD are slowing down with time https://bugs.launchpad.net/bugs/197762 You received this bug notification because you are a member of Ubuntu Desktop Bugs, which is a bug assignee. -- desktop-bugs mailing list desktop-bugs@lists.ubuntu.com
[Bug 197762] Re: file transfers on USB flash key (pendrive) or USB HDD are slowing down with time
@allisterbrizan, Please, do the investigation, and then open a new bug!!! Your issue with dd clones is completely different from kwstas tsob's problem in #256 which had to do with source file fragmentation because of bittorrent. You see? Both are performance problems, and both are complete different!! And discussing them in the same bug report in Launchpad is completely unproductive!!! So the issue is not writing off performance problems; just this specific Launchpad bug, which is clearly degenerated beyond any hope of help. (It doesn't help that launchpad is very slow when there are over 80 comments in the bug report.) If you want to have any hope of anyone paying attention, OPEN A NEW BUG. -- file transfers on USB flash key (pendrive) or USB HDD are slowing down with time https://bugs.launchpad.net/bugs/197762 You received this bug notification because you are a member of Ubuntu Desktop Bugs, which is a bug assignee. -- desktop-bugs mailing list desktop-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/desktop-bugs
[Bug 197762] Re: file transfers on USB flash key (pendrive) or USB HDD are slowing down with time
i had similar issues but also when i was trying to copy a file(around 50-700mb or larger) from my local hdd to another folder of the same hdd!!! so what i found out was that this was happening only in files that i had downloaded with transmission! What file system were you using on your hdd, and what version of Ubuntu are you using? This isn't surprising, because bittorrent downloads tend to write the file in random order (depending on what is made available from the bt seeders and the bt peers) and in multiple threads. Using ext4 and a relatively modern kernel, it won't be too bad, because ext4 has better allocation algorithms than ext3. It's best though, if the bt client uses fallocate(), which is I'll bet what qbittorrent is doing. So this is a useful bit of information, but I suspect some users have different causes for their problems, which is one of the reasons why most kernel developers have refused to use Launchpad. Too many ubuntu users are glomming onto a bug report, and they post different performance problems all in a single bug report, without bothering to give full information about their hardware, software versions, what file system they are using, etc. And then on top of that Launchpad is slow as a dog when there it gets too long. Which is why this bug is marked invalid and why I beg people, if they have performance problems, to open new bugs and to give full details about their hardware configuration, what version of Ubuntu they are running, the exact specifics of the USB device, etc. Otherwise this can be a great place for people to vent, but most people who aren't paid to pay attention to dysfunctional bug forums will simply say, Beam me up Scotty, there's no intelligent life here. -- file transfers on USB flash key (pendrive) or USB HDD are slowing down with time https://bugs.launchpad.net/bugs/197762 You received this bug notification because you are a member of Ubuntu Desktop Bugs, which is a bug assignee. -- desktop-bugs mailing list desktop-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/desktop-bugs
[Bug 197762] Re: file transfers on USB disk are very slow
Botond, Again, I would ask that you take a look at the procedures found here: https://help.ubuntu.com/community/DiskPerformance ... and then open another bug, complete with details of kernel version, dstat -D dev when doing a dd if=/dev/zero of=dev bs=32k, and then the output of dmesg | grep usb so we can see the usb-related messages, etc. I just did my own testing, using a 2.6.32-git6 based kernel (so it has a lot of freshly merged ext4 improvements), and what I can see when copying a large file to an Aptiva 2GB USB stick (purchased from Microcenter; identifies as a Sandisk device via lsusb -v), is a slowdown when doing a cp bigfile /mnt, when copying to ext2 and ext3, sometimes to a crawl (4k/second according to dstat), but when a I do dd if=/dev/zero of=/dev/sdc, I see 6MB/sec. I see the same 6MB/sec while mke2fs is zeroing out the inode table, and if I copy the large file using ext4, I see 6MB/sec (very rarely it drops down to 3-5MB/sec, but only for a second). When the same USB stick is formatted using vfat, I see a wide range of write bandwidths, from 3-6MB/sec, with an average of maybe 4-5 MB/sec. My hardware is a Lenovo T400 with 4GB of memory, and the writes to the disk start showing up with dstat after at most a 1 second after I hit the return key to initiate the cp from the shell prompt. So as you can see, these are very different results than what you have gotten. I get very similar (although perhaps slightly degraded) results as far as write speends are concerned when I mount the ext2 or ext3 file system using ext4 by the way. And I can consistently replicate these results, which indicates that at least on my system, raw sequential writes (i.e., via dd if=/dev/zero ...) are 6MB/sec, and using the ext4 file system driver gets me close the same raw sequential write rate, regardless of whether the file system is formatted using ext2, ext3, or ext4 (although things are a bit worse when extents are disabled). I am getting perhaps 75-80% of the raw write speed when using vfat to a USB stick, consistently over writing a 1GB file to a 2GB USB stick. With ext2 and ext3, I lost patience before the file copy completed, but after 10 seconds or so with ext2, and perhaps 30 seconds with ext3, the write speeds as reported by dstat -D had dropped down to 4-8 kb/sec, and it would stay there for a few minutes, and then suddenly pop back up to around 3-4 mb/sec, only to later (after some time had passed) drop back down to 4-8 kb/sec. The bottom line is that everyone is reporting slightly different symptoms, and so opening separate bug reports, with detailed information is the only hope of figuring out what is going on. I am going to posit that some people are losing because of bad interactions between the kernel and their USB controller (which should show up when they see bad results with raw writes to the device). Others may be seeing interesting interactions between the write order and size of the writes by a particular file system driver and their particular USB stick. Depending on the sophistication of the flash controller on the USB stick, the USB stick may handle small (4k) writes much less efficiently than large (64k to 128k) writes, especially if the writes are random access as opposed to sequential in nature. Note that the requirements for writing large jpegs in a digital camera (what many of the really cheap flash controllers were originally optimized for) are quite different from those needed for a general-purpose file system. I don't have time to much more in the way of detailed analysis, unfortunately (especially since at least with my USB stick and ext4, things were hunky-dory :-). If I had the time I would next try a different collection of USB sticks, and different kernels, including the default karmic kernel, to see if my results stayed the same while varying one variable at a time. That is, doing an exhaustive series of tests following the scientific method, which hopefully most folks learned in their high school science classes. If someone does have the time to do this work, I'd suggest opening a new bug (since Launchpad is really slow with large bugs, and there's a lot of noise in this bug already), and after you do this large series of tests, with the information I've suggested here and in the Wiki, post a pointer to the new bug in this bug report, and I'll promise to take a look at it. -- file transfers on USB disk are very slow https://bugs.launchpad.net/bugs/197762 You received this bug notification because you are a member of Ubuntu Desktop Bugs, which is a bug assignee. -- desktop-bugs mailing list desktop-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/desktop-bugs
[Bug 197762] Re: file transfers on USB disk are very slow
On Tue, Jul 14, 2009 at 08:45:56PM -0700, Ron Brogden wrote: Theodore, if I may ask you - are you just another random Linux user here or are you actually officially associated with Ubuntu in a support role? I see that you are associated with IBM, have written some file system utilities and appear to be connected with kernel development but am unsure your role here with respects to this bug. I'm a linux kernel developer, and yes, I'm the maintaining of the ext4 filesystem and e2fsprogs. I started paying attention to this problem because someone sent a note to the LKML (Linux Kernel Mailiung List) complaining that kernel developers don't pay attention to Ubuntu bugs. Keep in mind most Ubuntu users get Ubuntu for free, and pay no support fees; so it's not surprising that you will see very few Ubuntu support personnel paying attention to bugs like this one. So at some level, the Ubuntu user community and Canonical has a choice to make. If they are willing to let users just whine and whine and whine and _not_ help us by running experiments and opening separate bugs for each problem, and give full information (and we will provide them with explicit instructions on how to run those experiments), fine. We can just go back to ignoring Ubuntu launchpad as being totally, completely, worthless for actually fixing bugs. It can just be a place toxic sewer pit for users to whine, and if Canonical wants to pay someone to try to extract useful information (lots of luck; theres very little here) after the fact, they can go right ahead. We'll just file Ubuntu into the there's no intelligent life here category, and ignore pleas for help on LKML when people ask developers to pay attention to Launchpad bugs. Alternatively, if the goal is to have one part of the community helping another, then the users have to be helpful. That means doing research before filing bugs, and filing separate bugs for separate issues. Launchpad fails miserably when there are more than 80 comments; it's slow and painful to use. So I am trying to see if we can salvage the Launchpad culture so it can be useful, but ultimately, maybe it's a losing battle, and we should give up. Obviously most folks have no idea why their USB transfers are slow but they know something is up. It should be no surprise that they post unclear bug reports since there is no real way for them to tell whether their file manager, the kernel or some other bit of software is the cause. All they know is something is wrong and for those that dual boot, it does not happen with Windows (not my situation but it is far from uncommon). Of course this is frustrating for the bug fixer but then the response should be a request for hardware that displays the problem at the very least (which I believe has been offered here). The problem is that we need to separate out the reports. When something is as vague as disk transfers are slow, there's a extremely high probability that there may be multiple things going on. Some people might be having genuine hardware problems; others might be having filesystem issues. Eric's (epv's) problem *might* be the classic ext3/fsync stalled write problem --- I can't tell, because _he_ hasn't filed a separate bug report and given information about his problem. Do you really expect a car mechanic to fix all problem descriptions of the form my car is hard to start simultaneously? They all have the same symptoms, so *obviously* they must have same root cause. NOT. Since you are apparently a file system expert, how about writing a quick utility that gives out the details you think would be pertinent for assessing file transfer speeds over USB? This could then be used as a benchmark and hopefully show once and for all where the bottleneck is occuring. Presumably hacking the cp source (or dd or whatever) to add in some timing information is not a huge undertaking. The 'dd' program already provides this information. For example, like this: sudo dd if=/dev/sdXX of=/dev/null bs=32k count=32k But we have people whining on this bug about how they are GUI persons only, and don't have time to do any research on the bug. If all they are doing is whining, then maybe they should really go to Windows or MacOS. Or they should get a paid support contract from some Linux distribution where someone can be paid to hold their hands. There's No Such Thing As A Free Lunch, you know. If you are not associated with Ubuntu support and are not actually offering help then why are you polluting this bug report further than it already is? This bug report is already hopeless; the original bug report was against Ubuntu 8.04, over a year ago and two releases ago. Which is why I suggest people who are really serious about solving the problem open fresh bug reports, with full and complete detail. They should not assume that just because they have a similar problem, the hardware information submitted by someone else 12
Re: [Bug 197762] Re: file transfers on USB disk are very slow
On Wed, Jul 15, 2009 at 11:33:07AM -, kulight wrote: I've just remembered why the user base of Linux is so small I do actually expect my mechanic to find and fix what's wrong with my car when I just tell him it won't start And I don’t care if it's the engine the battery or the radiator You have to understand that most users have no idea where the problem is coming from or if its two separate ones They are just using the system and see the problems Ah, but (a) you pay your mechanic, and (b) you bring your car in, and don't expect him to fix it over the phone or over an web board. There are plenty of services for which you can get pay-for-service for Linux. For the right price, I've been flown into New York City to work at a large financial institution or even into a classified machine room at a major defense contractor. But the difference with Ubuntu Launchpad is that no one pays any money and they somehow expect gold-plated support. Sorry, for free support, you have to expect a certain amount of self service. It's still a way better deal than you will get anywhere else --- but if you wan the quality of service you get when you drive the car to your dealer, then you will have to (a) pay the bug fixer, and (b) let the bug fixer have direct access to your machine. -- file transfers on USB disk are very slow https://bugs.launchpad.net/bugs/197762 You received this bug notification because you are a member of Ubuntu Desktop Bugs, which is a bug assignee. -- desktop-bugs mailing list desktop-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/desktop-bugs
[Bug 197762] Re: file transfers on USB disk are very slow
Eric, The problem is that your problem may be very different from other peoples' problem. First of all, in your test, you are continually dirtying the page cache with cp /dev/zero /mnt/bigfile.This brings in filesystem effects, and VM effects, and so it's hard to tell what is actually going on. Without a lot of instrumentation, and knowing exactly the state of memory, and when the filesystem might be doing a commit, it's hard to say exactly what might be going on. You also haven't given us the kernel bootup messages (dmesg output after the system boots), as well as the type of device that you are using, the lsusb output, etc., etc., etc.You haven't told us how much memory there is in the system. And certainly the test which you did (cp /dev/zero /media/disk/bigfile ); sleep 10 ; time echo foo /media/disk/test is not one that we can replicate on Windows, so the assertion that something is wrong is also not something which is immediately obvious given the very limited amount of information you have given. Maybe it's *somewhere* in this bug report, bug this is why it is really not useful to mingle multiple users' problems in one bug report. If you want to open a separate bug report, give lots of details, and be willing to work with people who can suggest specific, repeatable experiments that examine one aspect of the system at a time, then we can try to see if there's something unusual. If you just want to whine about how horrible Ubuntu or Linux is, then keep on doing what you're doing here. It's not going to lead to anything useful though. But if you are really interested in being constructive, we need your help in giving us something useful in terms of enough information so we can find the root cause of a problem, and then try to solve it. -- file transfers on USB disk are very slow https://bugs.launchpad.net/bugs/197762 You received this bug notification because you are a member of Ubuntu Desktop Bugs, which is a bug assignee. -- desktop-bugs mailing list desktop-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/desktop-bugs
[Bug 197762] Re: file transfers on USB disk are very slow
jamesnmandy, For someone who 'does not have the time to file a proper bug report', you seem to have plenty of time to write notes complaining about the bug. Seriously --- if you aren't willing to help determine the problem --- and you like GUI solutions --- maybe you should just go back to Windows. You may find that you have even less responsiveness to your bug reports, unless you pay $$$ to Microsoft, but hey, if it makes you happier, you should switch. The problem could be with Nautilus (the gui file manager); or it could be with the device driver; or it could be with the ext3 filesystem issue where some process was triggering fsync calls. If you aren't willing to figure out why it's no fast enough for you, and you aren't willing to lift your little finger to help us figure out why; then please, don't complain --- you're clearly unhappy, and perhaps moving to Vista will make you happy. And at the end of the day, it's better for you to be happy than to have you bitter and complaining --- really, trust me, in the end you'll be happier to stop kvetching about Linux. It's simply not productive for anyone, least of all yourself. On the other hand, if you are willing to learn a few things, and spend a bit of time, then maybe using Linux will be more rewarding for you --- and again, in the long time you will probably be happier. -- file transfers on USB disk are very slow https://bugs.launchpad.net/bugs/197762 You received this bug notification because you are a member of Ubuntu Desktop Bugs, which is a bug assignee. -- desktop-bugs mailing list desktop-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/desktop-bugs
[Bug 197762] Re: file transfers on USB disk are very slow
Night64: No, you're using usb-storage device driver for both the pendrive and the ipod.The messages that you quoted as coming from the ipod just means that the usb-storage module had gotten unloaded, and when it was loaded as a module, it prints those lines to the kernel log. The fact that things were fast going to the ipod and variable when you write to the your USB flash device tends to indicate that the problem is specific to the flash device; it might just be that the flash device is sometimes slower because writing to flash can be slower sometimes, and it might be exacerbated by the FAT filesystem, which is never known as a speed demon. Note that this might not be true for other users on this bug report --- which is why I've suggested everyone file separate bug reports. People using Ubuntu 8.10 might possibly have been using the Low Performance USB storage driver, which Canonical inexplicably configured into older Ubuntu kernels. Anyone using that will definitely see massive performance problems --- and this has been known for ages. (Another reason why many kernel developers aren't happy trying to clear up Ubuntu-specific bugs) In any case, this kernel configuration bug doesn't seem to be a problem for Ubuntu 9.04; libusual.ko doesn't appear to be built any more, which is a Very Good Thing. Some users from a year ago who might have been using the low performance USB driver might have a very different problem than yours --- which is why it's better for each user to open separate bug reports. Remember that most of the people with the experience and knowledge to help are volunteers. If Ubuntu users continue to make life difficult, kernel developers will simply stop volunteering to wade through the cesspit of Ubuntu launchpad bugs -- file transfers on USB disk are very slow https://bugs.launchpad.net/bugs/197762 You received this bug notification because you are a member of Ubuntu Desktop Bugs, which is a bug assignee. -- desktop-bugs mailing list desktop-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/desktop-bugs
Re: [Bug 197762] Re: file transfers on USB disk are very slow
On Wed, May 20, 2009 at 08:48:34PM -, Night64 wrote: Thanks for the info, Ts'o. But your comment about libusual caught my eye. In Ubuntu 9.04 this module still shows up, right? [81601.434729] usbcore: registered new interface driver libusual Oops, sorry, that's what I get for not checking the driver names more closely. libusual is mostly harmless; it's the mapping function that arbitrates between ub and usb-storage drivers. The USB low performance device driver is ub.ko, normally found in, kernel/drivers/block/ub.ko: % find /lib/modules -name ub.ko /lib/modules/2.6.27-9-generic/kernel/drivers/block/ub.ko /lib/modules/2.6.28-11-generic/kernel/drivers/block/ub.ko You really, really, *really* want to avoid using the low performance USB driver at all costs. Fortunately at least with modern kernels libusual defaults to trying to prefer the usb-storage driver. I'm not sure of the time frame, but a while ago, some distributions were defaulting to using the ub driver, and *boy* was that a disaster. The ub driver is useful in embedded devices where you might not want to have the full USB stack (udev, hal, etc.), but I'm not sure why it anyone would want it to use it on a normal system Simon's observation is quite right, some flash devices are just slow. Note that 1x flash devices will only write at 150k/s. A 16x device will write at 2.4 MB/s. And this is assuming the filesystem doesn't get in the way. This is the speed rating used by Compact Flash or Secure Digital (CF or SD) cards. USB sticks generally don't have speed ratings at all, and depending on what you've purchased, they could be quite slow indeed. Note that with SD cards, sometimes the high speeds advertised (i.e., 150x) may not apply at all unless you have special hardware --- or sometimes the manufacturers are just lying. But given that there do exist 1x flash devices that only write at 150k/s should be a warning that some flash devices Are Just Slow. Hence my request that people who are really interested in debugging this should use USB hard drives, and not USB flash devices --- there's no way for us on the other side to know whether you are using a high quality or low-quality flash device, and so the speed problem could just be normal operation. In general, sometimes small writes will seem to go fast, because they get cached in memory. But just because the write operation seems to have completed doesn't mean that it's really done, or that it's safe to remove the flash device. So when you see an LED on a USB stick flashing, that's a sign that it's still writing out. If you write a very large file, eventually there's no more memory available, and so now the slow write speed becomes visible to the user. So this is why I'm becoming more and more convinced that we should just nuke this bug from orbit and start over. Naive users who don't understand this, can often see normal system performance, and report a me too!, and this just gums up the works. There ***may*** be a real bug hiding in here somewhere, by some users. Or it culd be a configuration bug; or a hardware bug. But there are too many users who are saying, I'm seeing this too!, when in fact they may just be seeing normal system behaviour. - Ted -- file transfers on USB disk are very slow https://bugs.launchpad.net/bugs/197762 You received this bug notification because you are a member of Ubuntu Desktop Bugs, which is a bug assignee. -- desktop-bugs mailing list desktop-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/desktop-bugs
[Bug 197762] Re: file transfers on USB disk are very slow
Ulrich --- if you have time to try to do more tests, may I gently suggest that you start a new Launchpad bug?Drop a pointer from in this launchpad bug number to the new one, but let's do all of the experimentation on a new launchpad bug. There are a couple of reasons for this: (1) Launchpad has horrific scalability problems, and it's painfully slow to find new entries on this Launchpad entry, since it has so many comments already. (2) It's not clear everyone on this bug is sufferring from the same bug. They may have similar symptoms, but that doesn't mean it is the same bug. It's actually **highly** confusing to have many people glomming on with a me, too!! comment, giving a very tiny amount of information, but not actually giving a full description of what they are seeing. In fact, the tiny amounts of data may be horribly misleading, because they may be seeing a different bug. (3) As a result, many kernel developers on LKML don't spend a lot of time on Launchpad bugs; the bug reporting is ___so___ ___bad___ that it's just too frustrating and time-consuming for them. And unfortunately, Canonical has a fairly limited kernel team, and it takes them a lot of time try to synthesize a good bug report from a lousy one. In fact, someone recently posted this bug report on LKML, and what it mostly got was a vent from someone complaining how useless most Ubuntu bugs were. (Well, it also got me interested enough to actually take a look, but I've gotten inured to how lousy Ubuntu bug reporting infrastructure is for large bugs, as well as mostly uninformative bug reports.) So if we do have one or more people who have the time and energy to do some real bug reporting (which takes real work), may suggest that each person open a separate bug, and each bug, give (a) full details about what kernel version, including whether it is a Ubuntu kernel or a mainline kernel, (b) full details about your hardware version, including which disks are hooked up which way (i.e., eSATA, USB, SATA, PATA, etc.), and (c) exhaustive details about how you ran your test case, whether or not is repeatable, etc. Also note that it is an instance of Launchpad bug #197762, and that anybody else who wants to glom on with a me, too is kindly asked to put that remark in the #197762 cesspit^H^H^H^H^H^H^H^Htop-level bug, and not in the new bug report, which we can hopefully keep specific to your situation. I would again suggest that you try doing multiple tests: 1) Copying one file from one disk to another 2) Copying from /dev/zero to file on the disk 3) Copying from /dev/zero to the raw disk device (yes, this will blow away your filesystem; hopefully you have scratch devices) The ideal bug reporter will have multiple devices that can be attached via different hardware interfaces (i.e., eSATA, USB, maybe a spare SATA port), so we can try to rule out hardware attachment problems. If someone is willing to do all of that, I'm more than willing to try to drill down as far as I can; although if it turns out to be a USB problem, I'll need to call in help from a USB developer but with a well constructed bug report, that shouldn't be a problem; I should be able to overcome LKML prejudices about lousy Launchpad bugs (no really, this one is different; we've kept it clean and professional, and there's plenty of information --- and we have a technically cluefull user who is willing to work with you and run experiments until the problem can be solved) More generally, maybe someone with the time can start a Wiki page on debugging filesystem performance problems, maybe on the Ubuntu wiki somewhere, or on ext4.wiki.kernel.org.There are plenty of tools like that people can use for debugging performance problems, beyond just dstat. Other tools include blktrace, sar, iostat, vmstat, and many others. Launchpad is really a lousy place to have these discussions; a wiki or a forum topic is really much better. -- file transfers on USB disk are very slow https://bugs.launchpad.net/bugs/197762 You received this bug notification because you are a member of Ubuntu Desktop Bugs, which is a bug assignee. -- desktop-bugs mailing list desktop-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/desktop-bugs
[Bug 197762] Re: file transfers on USB disk are very slow
For people who are seeing this --- try booting the kernel with the mem= command line option and see if this makes a difference. For example if you have 4 gigs of memory, try using mem=1G; if you have a gig of memory, try using mem=512M. Does this change how big the file needs to be before you start seeing the slowdown? -- file transfers on USB disk are very slow https://bugs.launchpad.net/bugs/197762 You received this bug notification because you are a member of Ubuntu Desktop Bugs, which is a bug assignee. -- desktop-bugs mailing list desktop-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/desktop-bugs
[Bug 197762] Re: file transfers on USB disk are very slow
Something else to try; I would suggest using dstat with the -D option (i.e, dstat -D sda,sdb) or iostat 1 (although I find dstat to be a bit more user friendly) to measure I/O transfer rates. I would recommend using dd if=/dev/zero of=/media/disk/test.img bs=32k to measure the transfer rates, and I would *not* recommend using a flash based device because they can show a lot of variance all by themselves.So please, use some kind of HDD. Speaking personally, I've been doing a rather huge amounts of copying of files back and forth between disks, using ext4, in the 30-40 gigabyte range, and I've not noticed anything like this. This could be filesystem related, or related to the page writeback algorithms, and so something else to try would be to try dd'ing to a raw disk, and see if you see the slowdown. For me, using a 2.6.30-rc5 and the ext4 filesystem, I've done many, *many* copies of data using rsync -axH to copy over entire filesystems (for doing things like testing fsck speedups between ext4 and ext3) and I've noticed a problem. I normally have a window open running dstat -D sda,sdb, so I can get dynamic output like this once a second: total-cpu-usage --dsk/sdb-dsk/sdc-- -net/total- ---paging-- ---system-- usr sys idl wai hiq siq| read writ: read writ| recv send| in out | int csw 19 26 42 11 0 2| 016M: 15M0 | 0 0 | 0 0 |2081 9176 20 24 41 13 0 2| 016M: 15M0 | 0 0 | 0 0 |2092 9287 18 24 42 14 0 1| 012M: 14M0 | 0 0 | 0 0 |2003 8987 Note that filesystem activity can make a huge difference. When copying files around, there's enough seeks going on that in practice I can only read or write 16 megabytes/second. If I do a dd if=/dev/zero of=/scratch/zero.img bs=32k, I get this: total-cpu-usage --dsk/sdb-- -net/total- ---paging-- ---system-- usr sys idl wai hiq siq| read writ| recv send| in out | int csw 3 10 57 28 0 1| 057M| 0 0 | 0 0 | 746 1268 4 27 43 25 0 2|4096B 58M| 0 0 | 0 0 |1065 951 4 21 44 30 0 1| 060M| 0 0 | 0 0 | 701 396 5 23 50 20 0 2| 061M| 0 0 | 0 0 | 718 288 Here's what I get if I do dd if=/dev/zero of=/dev/closure/scratch bs=32k total-cpu-usage --dsk/sdb-- -net/total- ---paging-- ---system-- usr sys idl wai hiq siq| read writ| recv send| in out | int csw 11 13 47 27 0 1| 061M| 0 0 | 0 0 | 740 1327 3 15 49 32 0 1| 061M| 0 0 | 0 0 | 522 249 2 10 50 37 0 1| 061M| 0 0 | 0 0 | 476 246 6 14 49 30 0 1| 061M| 0 0 | 0 0 | 544 260 3 12 49 35 0 1| 061M| 60B 60B| 0 0 | 558 260 And here's what I get if I format /dev/closure/scratch as ext3 instead of ext4, and then do dd if=/dev/zero of=/scratch/zero.img bs=32k: usr sys idl wai hiq siq| read writ| recv send| in out | int csw 5 19 17 57 0 1|4096B 47M| 0 0 | 0 0 | 637 503 6 15 0 77 0 1| 052M| 0 0 | 0 0 | 608 611 5 1 0 94 0 1| 051M| 0 0 | 0 0 | 385 450 So when people complain about slow disk performance, you really need to control variables. All I can say is that on a Thinkpad X61s, using SATA disks, USB disks, and SATA disks over a SATA/PATA/SATA bridge (horrific hardware kludge in when you use disks in the Ultrabay slot --- don't ask) I've not seen the problem are complaining about. Note that if you are doing copies when there are other programs going on in the background, including programs like firefox triggering off fsync()'s this can also affect performance numbers. So what's really needed are people who are willing to spend a lot of time drilling down into why the performance is dropping, and by controlling variables; I recommend using a scratch filesystems and scratch disks that you don't mind reformatting so you can do multiple repeatable experiments, and to do it on a system that isn't running anything else in the background. -- file transfers on USB disk are very slow https://bugs.launchpad.net/bugs/197762 You received this bug notification because you are a member of Ubuntu Desktop Bugs, which is a bug assignee. -- desktop-bugs mailing list desktop-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/desktop-bugs
