serial/ftdi_sio byte loss / performance regression
Hi
I have noticed that the ftdi_sio serial driver in recent kernel versions
has very bad performance when used through the Python's serial library.
As a test case I have a custom device that will send a continuous block
of 5k characters once every few seconds over a RS-232 line (115200 baud)
to an Olimex programmer (based on FT2232C, also tried one with FT2232H).
Programmer is connected to a Linux system where a simple Python script
reads the device:
import serial
comm = serial.Serial("/dev/ttyUSB0", 115200)
while True:
line = comm.readline()
With kernels before 3.7.0 the script reads uncorrupted data while using
newer kernels (including 3.9.4) the Python script sees heavy byte loss.
"top" shows an 95% "idle" CPU. Only very slow transmissions (on the
order of tens of bytes per second) will come through uncorrupted.
Using git-bisect, I have found the commit that introduced this problem:
6f602912c9d0c84c2edbd446dd9f72660b701605
usb: serial: ftdi_sio: Add missing chars_in_buffer function
This might also be related with the unusual way Python serial library
reads the device. It uses select() with no timeout and single byte
read()s in a loop. strace output:
select(4, [3], [], [], NULL)= 1 (in [3])
read(3, "D", 1) = 1
select(4, [3], [], [], NULL)= 1 (in [3])
read(3, "E", 1) = 1
...
With sufficiently large read()s the byte loss can be eliminated.
With the commit above, each select() now causes an additional round trip
over USB to read the state of the hardware buffer. It's possible that
constant status querying triggers some bug in the hardware or the query
is simply too slow and causes overflows in the hardware buffer.
Thanks for your help
Tomaž
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: serial/ftdi_sio byte loss / performance regression
On Thu, Jun 06, 2013 at 10:50:36AM +0200, Tomaž Šolc wrote:
> Hi
>
> I have noticed that the ftdi_sio serial driver in recent kernel versions
> has very bad performance when used through the Python's serial library.
>
> As a test case I have a custom device that will send a continuous block
> of 5k characters once every few seconds over a RS-232 line (115200 baud)
> to an Olimex programmer (based on FT2232C, also tried one with FT2232H).
>
> Programmer is connected to a Linux system where a simple Python script
> reads the device:
>
> import serial
> comm = serial.Serial("/dev/ttyUSB0", 115200)
> while True:
> line = comm.readline()
>
> With kernels before 3.7.0 the script reads uncorrupted data while using
> newer kernels (including 3.9.4) the Python script sees heavy byte loss.
> "top" shows an 95% "idle" CPU. Only very slow transmissions (on the
> order of tens of bytes per second) will come through uncorrupted.
>
> Using git-bisect, I have found the commit that introduced this problem:
>
> 6f602912c9d0c84c2edbd446dd9f72660b701605
> usb: serial: ftdi_sio: Add missing chars_in_buffer function
>
> This might also be related with the unusual way Python serial library
> reads the device. It uses select() with no timeout and single byte
> read()s in a loop. strace output:
>
> select(4, [3], [], [], NULL)= 1 (in [3])
> read(3, "D", 1) = 1
> select(4, [3], [], [], NULL)= 1 (in [3])
> read(3, "E", 1) = 1
> ...
>
> With sufficiently large read()s the byte loss can be eliminated.
>
> With the commit above, each select() now causes an additional round trip
> over USB to read the state of the hardware buffer. It's possible that
> constant status querying triggers some bug in the hardware or the query
> is simply too slow and causes overflows in the hardware buffer.
You're absolutely right. This is a known issue (the select overhead)
that was just recently fixed by commit a37025b5c7 ("USB: ftdi_sio: fix
chars_in_buffer overhead") in v3.10-rc3. Care to give v3.10-rc4 a try?
Greg, perhaps we should consider backporting the wait-until-sent
patches (i.e. 0693196fe..4746b6c6e)?
Thanks,
Johan
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: serial/ftdi_sio byte loss / performance regression
On Thu, Jun 06, 2013 at 11:58:56AM +0200, Johan Hovold wrote:
> On Thu, Jun 06, 2013 at 10:50:36AM +0200, Tomaž Šolc wrote:
> > Hi
> >
> > I have noticed that the ftdi_sio serial driver in recent kernel versions
> > has very bad performance when used through the Python's serial library.
> >
> > As a test case I have a custom device that will send a continuous block
> > of 5k characters once every few seconds over a RS-232 line (115200 baud)
> > to an Olimex programmer (based on FT2232C, also tried one with FT2232H).
> >
> > Programmer is connected to a Linux system where a simple Python script
> > reads the device:
> >
> > import serial
> > comm = serial.Serial("/dev/ttyUSB0", 115200)
> > while True:
> > line = comm.readline()
> >
> > With kernels before 3.7.0 the script reads uncorrupted data while using
> > newer kernels (including 3.9.4) the Python script sees heavy byte loss.
> > "top" shows an 95% "idle" CPU. Only very slow transmissions (on the
> > order of tens of bytes per second) will come through uncorrupted.
> >
> > Using git-bisect, I have found the commit that introduced this problem:
> >
> > 6f602912c9d0c84c2edbd446dd9f72660b701605
> > usb: serial: ftdi_sio: Add missing chars_in_buffer function
> >
> > This might also be related with the unusual way Python serial library
> > reads the device. It uses select() with no timeout and single byte
> > read()s in a loop. strace output:
> >
> > select(4, [3], [], [], NULL)= 1 (in [3])
> > read(3, "D", 1) = 1
> > select(4, [3], [], [], NULL)= 1 (in [3])
> > read(3, "E", 1) = 1
> > ...
> >
> > With sufficiently large read()s the byte loss can be eliminated.
> >
> > With the commit above, each select() now causes an additional round trip
> > over USB to read the state of the hardware buffer. It's possible that
> > constant status querying triggers some bug in the hardware or the query
> > is simply too slow and causes overflows in the hardware buffer.
>
> You're absolutely right. This is a known issue (the select overhead)
> that was just recently fixed by commit a37025b5c7 ("USB: ftdi_sio: fix
> chars_in_buffer overhead") in v3.10-rc3. Care to give v3.10-rc4 a try?
>
> Greg, perhaps we should consider backporting the wait-until-sent
> patches (i.e. 0693196fe..4746b6c6e)?
Yes, that's a good idea, I'll do that for the next round of stable
updates, after this next release tomorrow.
thanks,
greg k-h
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: serial/ftdi_sio byte loss / performance regression
On 06/06/2013 11:58 AM, Johan Hovold wrote:
> On Thu, Jun 06, 2013 at 10:50:36AM +0200, Tomaž Šolc wrote:
>> I have noticed that the ftdi_sio serial driver in recent kernel versions
>> has very bad performance when used through the Python's serial library.
>>
>> As a test case I have a custom device that will send a continuous block
>> of 5k characters once every few seconds over a RS-232 line (115200 baud)
>> to an Olimex programmer (based on FT2232C, also tried one with FT2232H).
>>
>> Programmer is connected to a Linux system where a simple Python script
>> reads the device:
>>
>> import serial
>> comm = serial.Serial("/dev/ttyUSB0", 115200)
>> while True:
>> line = comm.readline()
>>
>> With kernels before 3.7.0 the script reads uncorrupted data while using
>> newer kernels (including 3.9.4) the Python script sees heavy byte loss.
>> "top" shows an 95% "idle" CPU. Only very slow transmissions (on the
>> order of tens of bytes per second) will come through uncorrupted.
>>
>> Using git-bisect, I have found the commit that introduced this problem:
>>
>> 6f602912c9d0c84c2edbd446dd9f72660b701605
>> usb: serial: ftdi_sio: Add missing chars_in_buffer function
>>
>> This might also be related with the unusual way Python serial library
>> reads the device. It uses select() with no timeout and single byte
>> read()s in a loop. strace output:
>>
>> select(4, [3], [], [], NULL)= 1 (in [3])
>> read(3, "D", 1) = 1
>> select(4, [3], [], [], NULL)= 1 (in [3])
>> read(3, "E", 1) = 1
>> ...
>>
>> With sufficiently large read()s the byte loss can be eliminated.
>>
>> With the commit above, each select() now causes an additional round trip
>> over USB to read the state of the hardware buffer. It's possible that
>> constant status querying triggers some bug in the hardware or the query
>> is simply too slow and causes overflows in the hardware buffer.
>
> You're absolutely right. This is a known issue (the select overhead)
> that was just recently fixed by commit a37025b5c7 ("USB: ftdi_sio: fix
> chars_in_buffer overhead") in v3.10-rc3. Care to give v3.10-rc4 a try?
Johan, thanks for the prompt reply. I have tried v3.10-rc4 and indeed
the issue is fixed.
Best regards
Tomaž
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: serial/ftdi_sio byte loss / performance regression
On Thu, Jun 06, 2013 at 11:21:54AM -0700, Greg KH wrote:
> On Thu, Jun 06, 2013 at 11:58:56AM +0200, Johan Hovold wrote:
> > On Thu, Jun 06, 2013 at 10:50:36AM +0200, Tomaž Šolc wrote:
> > > Hi
> > >
> > > I have noticed that the ftdi_sio serial driver in recent kernel versions
> > > has very bad performance when used through the Python's serial library.
> > >
> > > As a test case I have a custom device that will send a continuous block
> > > of 5k characters once every few seconds over a RS-232 line (115200 baud)
> > > to an Olimex programmer (based on FT2232C, also tried one with FT2232H).
> > >
> > > Programmer is connected to a Linux system where a simple Python script
> > > reads the device:
> > >
> > > import serial
> > > comm = serial.Serial("/dev/ttyUSB0", 115200)
> > > while True:
> > > line = comm.readline()
> > >
> > > With kernels before 3.7.0 the script reads uncorrupted data while using
> > > newer kernels (including 3.9.4) the Python script sees heavy byte loss.
> > > "top" shows an 95% "idle" CPU. Only very slow transmissions (on the
> > > order of tens of bytes per second) will come through uncorrupted.
> > >
> > > Using git-bisect, I have found the commit that introduced this problem:
> > >
> > > 6f602912c9d0c84c2edbd446dd9f72660b701605
> > > usb: serial: ftdi_sio: Add missing chars_in_buffer function
> > >
> > > This might also be related with the unusual way Python serial library
> > > reads the device. It uses select() with no timeout and single byte
> > > read()s in a loop. strace output:
> > >
> > > select(4, [3], [], [], NULL)= 1 (in [3])
> > > read(3, "D", 1) = 1
> > > select(4, [3], [], [], NULL)= 1 (in [3])
> > > read(3, "E", 1) = 1
> > > ...
> > >
> > > With sufficiently large read()s the byte loss can be eliminated.
> > >
> > > With the commit above, each select() now causes an additional round trip
> > > over USB to read the state of the hardware buffer. It's possible that
> > > constant status querying triggers some bug in the hardware or the query
> > > is simply too slow and causes overflows in the hardware buffer.
> >
> > You're absolutely right. This is a known issue (the select overhead)
> > that was just recently fixed by commit a37025b5c7 ("USB: ftdi_sio: fix
> > chars_in_buffer overhead") in v3.10-rc3. Care to give v3.10-rc4 a try?
> >
> > Greg, perhaps we should consider backporting the wait-until-sent
> > patches (i.e. 0693196fe..4746b6c6e)?
>
> Yes, that's a good idea, I'll do that for the next round of stable
> updates, after this next release tomorrow.
I applied these, plus a few others in order to get them all to apply and
build properly.
But the last patch, 4746b6c6e, didn't apply, and I don't think we really
need it for the 3.9 kernel, do we?
thanks,
greg k-h
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: serial/ftdi_sio byte loss / performance regression
On Mon, Jun 10, 2013 at 02:39:06PM -0700, Greg KH wrote: > On Thu, Jun 06, 2013 at 11:21:54AM -0700, Greg KH wrote: > > On Thu, Jun 06, 2013 at 11:58:56AM +0200, Johan Hovold wrote: > > > Greg, perhaps we should consider backporting the wait-until-sent > > > patches (i.e. 0693196fe..4746b6c6e)? > > > > Yes, that's a good idea, I'll do that for the next round of stable > > updates, after this next release tomorrow. > > I applied these, plus a few others in order to get them all to apply and > build properly. > > But the last patch, 4746b6c6e, didn't apply, and I don't think we really > need it for the 3.9 kernel, do we? No, you're right. It's merely a clean up and slight optimisation as no chars_in_buffer needs the disconnect mutex after the other changes. Not sure why it wouldn't apply, though. I was a bit sloppy when I gave the commit refs above -- it should have been 0693196^..b16634a More specifically: 4746b6c USB: serial: clean up chars_in_buffer This one is not strictly necessary, but should apply to v3.9. ff93b19 USB: ti_usb_3410_5052: fix chars_in_buffer overhead This should not be applied to v3.9 as the problem it fixes wasn't introduced until v3.10-rc1. It seems you had to add two other patches to get this one to apply. They should be dropped as well. b16634a USB: io_ti: fix chars_in_buffer overhead Needed in 3.9. a37025b USB: ftdi_sio: fix chars_in_buffer overhead c413364 USB: ftdi_sio: clean up get_modem_status dcf0105 USB: serial: add generic wait_until_sent implementation These three are also needed in 3.9 (and also 3.8). 0693196 USB: serial: add wait_until_sent operation Needed in 3.9 (and 3.8). (You noticed my refspec did not include it and added it.) Sorry about the confusion. Johan -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: serial/ftdi_sio byte loss / performance regression
On Tue, Jun 11, 2013 at 12:43:25PM +0200, Johan Hovold wrote: > On Mon, Jun 10, 2013 at 02:39:06PM -0700, Greg KH wrote: > > On Thu, Jun 06, 2013 at 11:21:54AM -0700, Greg KH wrote: > > > On Thu, Jun 06, 2013 at 11:58:56AM +0200, Johan Hovold wrote: > > > > > Greg, perhaps we should consider backporting the wait-until-sent > > > > patches (i.e. 0693196fe..4746b6c6e)? > > > > > > Yes, that's a good idea, I'll do that for the next round of stable > > > updates, after this next release tomorrow. > > > > I applied these, plus a few others in order to get them all to apply and > > build properly. > > > > But the last patch, 4746b6c6e, didn't apply, and I don't think we really > > need it for the 3.9 kernel, do we? > > No, you're right. It's merely a clean up and slight optimisation as no > chars_in_buffer needs the disconnect mutex after the other changes. Not > sure why it wouldn't apply, though. > > I was a bit sloppy when I gave the commit refs above -- it should have > been > > 0693196^..b16634a > > More specifically: > > 4746b6c USB: serial: clean up chars_in_buffer > > This one is not strictly necessary, but should apply to v3.9. It doesn't apply to 3.9 at all, so it being not really necessary, I'll not include it :) > ff93b19 USB: ti_usb_3410_5052: fix chars_in_buffer overhead > > This should not be applied to v3.9 as the problem it fixes > wasn't introduced until v3.10-rc1. > > It seems you had to add two other patches to get this one to > apply. They should be dropped as well. Ok, I've dropped all 3 of these now. > b16634a USB: io_ti: fix chars_in_buffer overhead > > Needed in 3.9. Good, I have that one. > a37025b USB: ftdi_sio: fix chars_in_buffer overhead > c413364 USB: ftdi_sio: clean up get_modem_status > dcf0105 USB: serial: add generic wait_until_sent implementation > > These three are also needed in 3.9 (and also 3.8). I have these three. 3.8-stable is dead as far as I'm concerned, so I can't do anything about it :) > 0693196 USB: serial: add wait_until_sent operation > > Needed in 3.9 (and 3.8). (You noticed my refspec did not include > it and added it.) Yes, other things were breaking without it. > Sorry about the confusion. No worries, I've dropped the three patches listed above, so we should be all good to go. I'll do a 3.9.6-rc1 release today to get these all out to people to test. thanks, greg k-h -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: serial/ftdi_sio byte loss / performance regression
> > a37025b USB: ftdi_sio: fix chars_in_buffer overhead > > c413364 USB: ftdi_sio: clean up get_modem_status > > dcf0105 USB: serial: add generic wait_until_sent implementation > > > > These three are also needed in 3.9 (and also 3.8). > > I have these three. 3.8-stable is dead as far as I'm concerned, so I > can't do anything about it :) Yeah, I know. I just put it in parenthesis for reference. Feels like every kernel version is being picked up by someone these days. :) Johan -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
