CPU Cache and busdma usage in USB

2009-06-23 Thread Piotr Zięcik
While bringing up EHCI (8-CURRENT, new USB stack) on ARM machine we have found cache-related problem in the USB stack. The usb_pc_cpu_flush() and usb_pc_cpu_invalidate() functions are used to flush/invalidate CPU caches in various places in USB code. Internally, the functions are implemented u

Re: CPU Cache and busdma usage in USB

2009-06-23 Thread Alexandr Rybalko
On Tue, 23 Jun 2009 10:35:42 +0200 Piotr Zięcik wrote: >> While bringing up EHCI (8-CURRENT, new USB stack) on ARM machine we have >> found cache-related problem in the USB stack. >> >> The usb_pc_cpu_flush() and usb_pc_cpu_invalidate() functions are used to >> flush/invalidate CPU caches in

Re: CPU Cache and busdma usage in USB

2009-06-23 Thread Hans Petter Selasky
On Tuesday 23 June 2009 11:11:29 Alexandr Rybalko wrote: > On Tue, 23 Jun 2009 10:35:42 +0200 > > Piotr Zięcik wrote: > >> While bringing up EHCI (8-CURRENT, new USB stack) on ARM machine we > >> have found cache-related problem in the USB stack. > >> > >> The usb_pc_cpu_flush() and usb_pc_cpu_in

Re: CPU Cache and busdma usage in USB

2009-06-23 Thread M. Warner Losh
In message: <200906231912.20741.hsela...@c2i.net> Hans Petter Selasky writes: : On Tuesday 23 June 2009 11:11:29 Alexandr Rybalko wrote: : > On Tue, 23 Jun 2009 10:35:42 +0200 : > : > Piotr Zięcik wrote: : > >> While bringing up EHCI (8-CURRENT, new USB stack) on ARM machine we : > >

Re: CPU Cache and busdma usage in USB

2009-06-24 Thread Piotr Zięcik
> : > >> My question is about general idea of bus_dma usage for cache > : > >> operations. In my opinion we should not rely on bus_dmamap_sync() > : > >> behaviour as this function may do different things on different > : > >> architectures. This not always works as expected, which is clearly > :

Re: CPU Cache and busdma usage in USB

2009-06-28 Thread Hans Petter Selasky
Hi Piotr and Rafal, Your patch is not fully correct. It will break support for x86 and more when bounce pages are uses. Let's get the definitions right: man busdma BUS_DMASYNC_PREREADPerform any synchronization required prior to an update o

Re: CPU Cache and busdma usage in USB

2009-06-29 Thread Piotr Zięcik
Sunday 28 June 2009 11:54:40 Hans Petter Selasky napisał(a): > Hi Piotr and Rafal, > > Your patch is not fully correct. It will break support for x86 and more > when bounce pages are uses. > > Let's get the definitions right: > > man busdma > (...) > > My view: > > XXX_PREXXX functions should be us

Re: CPU Cache and busdma usage in USB

2009-06-29 Thread Hans Petter Selasky
Hi Piotr and Rafal, On Monday 29 June 2009 11:11:20 Piotr Zięcik wrote: > Sunday 28 June 2009 11:54:40 Hans Petter Selasky napisał(a): > > Hi Piotr and Rafal, > > > > Your patch is not fully correct. It will break support for x86 and more > > when bounce pages are uses. > > > > Let's get the defi

Re: CPU Cache and busdma usage in USB

2009-06-29 Thread Piotr Zięcik
Monday 29 June 2009 11:55:11 Hans Petter Selasky napisał(a): > Hi Piotr and Rafal, > > I'm not saying everything is OK, I just don't agree that the problem is in > USB. > > If you look at: > > http://fxr.watson.org/fxr/source/i386/i386/busdma_machdep.c#L931 > > Then you see the XXX_PREXXX means cop

Re: CPU Cache and busdma usage in USB

2009-06-29 Thread Hans Petter Selasky
Hi, On Monday 29 June 2009 12:49:43 Piotr Zięcik wrote: > Monday 29 June 2009 11:55:11 Hans Petter Selasky napisał(a): > > Hi Piotr and Rafal, > > Look into ehci_check_transfer() function > (http://fxr.watson.org/fxr/source/dev/usb/controller/ehci.c#L1294) > > usb_pc_cpu_invalidate() [bus_dmamap_

Re: CPU Cache and busdma usage in USB

2009-06-29 Thread Stanislav Sedov
On Mon, 29 Jun 2009 13:37:41 +0200 Hans Petter Selasky mentioned: > USB is currently _updating_ (!!) the PAGE offset part of "vaddr". If > cpu_dcache_inv_range() is called with an address not starting at the cache > line what will the cpu_dcache_inv_range() do? Will it skip to the next cache >

Re: CPU Cache and busdma usage in USB

2009-06-29 Thread Hans Petter Selasky
On Monday 29 June 2009 14:10:11 Stanislav Sedov wrote: > On Mon, 29 Jun 2009 13:37:41 +0200 > > Hans Petter Selasky mentioned: > > USB is currently _updating_ (!!) the PAGE offset part of "vaddr". If > > cpu_dcache_inv_range() is called with an address not starting at the > > cache line what will

Re: CPU Cache and busdma usage in USB

2009-06-29 Thread Piotr Zięcik
Monday 29 June 2009 13:37:41 Hans Petter Selasky napisał(a): > > Look into ehci_check_transfer() function > > (http://fxr.watson.org/fxr/source/dev/usb/controller/ehci.c#L1294) > > > > usb_pc_cpu_invalidate() [bus_dmamap_sync()] is not used in this function > > correcly. It is not paired with usb_

Re: CPU Cache and busdma usage in USB

2009-06-29 Thread Hans Petter Selasky
On Monday 29 June 2009 14:16:13 Piotr Zięcik wrote: > Look for bus_dmamap_sync() sync description, last sentence: > > If read and write operations are not preceded and followed by the > appropriate synchronization operations, behavior is undefined. > I don't think this is a problem. I do flush r

Re: CPU Cache and busdma usage in USB

2009-06-29 Thread Piotr Zięcik
Monday 29 June 2009 13:37:41 Hans Petter Selasky napisał(a): > if (op & BUS_DMASYNC_POSTREAD) { > if (bpage->vaddr_nocache == 0) { > cpu_dcache_inv_range(bpage->vaddr, > bpage->datacount); >

Re: CPU Cache and busdma usage in USB

2009-06-29 Thread Hans Petter Selasky
On Tuesday 23 June 2009 10:35:42 Piotr Zięcik wrote: > --- a/sys/dev/usb/usb_busdma.c > +++ b/sys/dev/usb/usb_busdma.c > @@ -658,8 +658,7 @@ usb_pc_cpu_invalidate(struct usb_page_cache *pc) > /* nothing has been loaded into this page cache! */ > return; > } >

Re: CPU Cache and busdma usage in USB

2009-06-29 Thread Stanislav Sedov
On Mon, 29 Jun 2009 14:14:45 +0200 Hans Petter Selasky mentioned: > On Monday 29 June 2009 14:10:11 Stanislav Sedov wrote: > > On Mon, 29 Jun 2009 13:37:41 +0200 > > > > Hans Petter Selasky mentioned: > > > USB is currently _updating_ (!!) the PAGE offset part of "vaddr". If > > > cpu_dcache_inv

Re: CPU Cache and busdma usage in USB

2009-06-29 Thread Hans Petter Selasky
On Monday 29 June 2009 17:06:37 Mark Tinguely wrote: > > What has to be changed in busdma_machdep.c for ARM/MIPS so that the > > problem= is=20 > > resolved. I think there are something missing there and not in USB! > > > > =2D-HPS > > I unsuccessfully looked for the start of this thread. Which

Re: CPU Cache and busdma usage in USB

2009-06-29 Thread Mark Tinguely
> > What has to be changed in busdma_machdep.c for ARM/MIPS so that the problem= > is=20 > resolved. I think there are something missing there and not in USB! > > =2D-HPS I unsuccessfully looked for the start of this thread. Which version of FreeBSD are you using? --Mark T. _

Re: CPU Cache and busdma usage in USB

2009-06-30 Thread Piotr Zięcik
Monday 29 June 2009 15:16:56 Hans Petter Selasky napisał(a): > > You want to change the flags passed to bus_dmamap_sync() so that the > flush/invalidate mapping gets right. I understand why your patch makes it > work. That's not the problem. > > In "src/sys/arm/arm/busdma_machdep.c" there is a func

Re: CPU Cache and busdma usage in USB

2009-06-30 Thread Hans Petter Selasky
On Tuesday 30 June 2009 10:37:48 Piotr Zięcik wrote: > Monday 29 June 2009 15:16:56 Hans Petter Selasky napisał(a): > > You want to change the flags passed to bus_dmamap_sync() so that the > > flush/invalidate mapping gets right. I understand why your patch makes it > > work. That's not the problem

Re: CPU Cache and busdma usage in USB

2009-07-05 Thread Hans Petter Selasky
On Tuesday 30 June 2009 11:28:23 Hans Petter Selasky wrote: > On Tuesday 30 June 2009 10:37:48 Piotr Zięcik wrote: > > Monday 29 June 2009 15:16:56 Hans Petter Selasky napisał(a): > > > You want to change the flags passed to bus_dmamap_sync() so that the > > > flush/invalidate mapping gets right. I

Re: CPU Cache and busdma usage in USB

2009-07-07 Thread Piotr Zięcik
> > I did not find time yet to test on my AT91RM9200 board. I hope to do a test > this week. Did you at semihalf find anything? > I had Checked USB behaviour on PowerPC without hardware cache coherency. The problem also exists here and patch helps. I also generated some logs which you asked for:

Re: CPU Cache and busdma usage in USB

2009-07-07 Thread Hans Petter Selasky
On Tuesday 07 July 2009 15:50:28 Piotr Zięcik wrote: > > I did not find time yet to test on my AT91RM9200 board. I hope to do a > > test this week. Did you at semihalf find anything? > Hi, > I had Checked USB behaviour on PowerPC without hardware cache coherency. > The problem also exists here an

Re: CPU Cache and busdma usage in USB

2009-07-08 Thread Hans Petter Selasky
On Wednesday 08 July 2009 10:58:17 Rafal Jaworowski wrote: > On 2009-07-07, at 18:46, Hans Petter Selasky wrote: > >> I had Checked USB behaviour on PowerPC without hardware cache > >> coherency. > >> The problem also exists here and patch helps. > > > > In my source code view the busdma sync funct

Re: CPU Cache and busdma usage in USB

2009-07-08 Thread Rafal Jaworowski
On 2009-07-07, at 18:46, Hans Petter Selasky wrote: I had Checked USB behaviour on PowerPC without hardware cache coherency. The problem also exists here and patch helps. In my source code view the busdma sync function is empty for power- pc. Your patch should have no effect at all for sy

Re: CPU Cache and busdma usage in USB

2009-07-08 Thread Piotr Zięcik
Wednesday 08 July 2009 11:03:43 Hans Petter Selasky napisał(a): > > And what about my patch suggestion in my previous e-mail having the same > subject. Does it work? > I have tested it and it does not work. By the way Writeback before Writeback Invalidate did not change cache behaviour too much.

Re: CPU Cache and busdma usage in USB

2009-07-08 Thread Sebastian Huber
Hi! The general approach for systems with no hardware cache coherence protocol should be: Memory to device DMA transfer of a buffer: 1. write all modified cache lines of the buffer back to memory (cache flush for buffer) 2. start DMA transfer Device to memory DMA transfer of a buffer: 1. inval

Re: CPU Cache and busdma usage in USB

2009-07-08 Thread Hans Petter Selasky
Hi, On Wednesday 08 July 2009 12:16:39 Piotr Zięcik wrote: > Wednesday 08 July 2009 11:03:43 Hans Petter Selasky napisał(a): > > And what about my patch suggestion in my previous e-mail having the same > > subject. Does it work? > > I have tested it and it does not work. By the way Writeback befor

Re: CPU Cache and busdma usage in USB

2009-07-08 Thread M. Warner Losh
In message: <200907081103.45388.hsela...@c2i.net> Hans Petter Selasky writes: : On Wednesday 08 July 2009 10:58:17 Rafal Jaworowski wrote: : > On 2009-07-07, at 18:46, Hans Petter Selasky wrote: : > >> I had Checked USB behaviour on PowerPC without hardware cache : > >> coherency. : >

Re: CPU Cache and busdma usage in USB

2009-07-09 Thread Piotr Zięcik
Wednesday 08 July 2009 12:50:56 Hans Petter Selasky napisał(a): > By flush you mean write from CPU cache to RAM, right. And nothing else? You > don't mean discard CPU cache by "flush" ??? Yes. By flush I mean write all valid and modified cache lines to RAM. > > However looking into logs which I h

Re: CPU Cache and busdma usage in USB

2009-07-09 Thread Piotr Zięcik
Wednesday 08 July 2009 12:30:46 Sebastian Huber napisał(a): > Device to memory DMA transfer of a buffer: > (...) > 3. all accesses to the buffer via the cache must wait until the DMA has > finished And this is the problem in my oppinion - USB stack does not wait. -- Best Regards. Piotr Zięcik __

Re: CPU Cache and busdma usage in USB

2009-07-09 Thread Hans Petter Selasky
Hi Piotr, On Thursday 09 July 2009 17:21:00 Piotr Zięcik wrote: > Wednesday 08 July 2009 12:50:56 Hans Petter Selasky napisał(a): > > By flush you mean write from CPU cache to RAM, right. And nothing else? > > You don't mean discard CPU cache by "flush" ??? > > Yes. By flush I mean write all valid

Re: CPU Cache and busdma usage in USB

2009-07-09 Thread Hans Petter Selasky
On Thursday 09 July 2009 18:34:40 Hans Petter Selasky wrote: > > Full log with backtraces is here: > > http://people.freebsd.org/~raj/logs/usb-cache.log > > 1) My analysis: Only the data areas are being flushed/invalidated. I also see that the flushing/invalidating is being done correctly. --HPS

Re: CPU Cache and busdma usage in USB

2009-07-14 Thread Piotr Zięcik
> > 1) My analysis: Only the data areas are being flushed/invalidated. No > transfer descriptors are flushed/invalidated. I see no cache operations > happening on any DMA control structures, even though there are calls from > EHCI to xxx_pc_flush() and xxx_pc_invalidate(). > This is 100% correct i

Re: CPU Cache and busdma usage in USB

2009-07-14 Thread Hans Petter Selasky
On Tuesday 14 July 2009 10:31:10 Piotr Zięcik wrote: > > 1) My analysis: Only the data areas are being flushed/invalidated. No > > transfer descriptors are flushed/invalidated. I see no cache operations > > happening on any DMA control structures, even though there are calls from > > EHCI to xxx_pc

Re: CPU Cache and busdma usage in USB

2009-07-14 Thread Rafal Jaworowski
On 2009-07-14, at 10:36, Hans Petter Selasky wrote: On Tuesday 14 July 2009 10:31:10 Piotr Zięcik wrote: 1) My analysis: Only the data areas are being flushed/invalidated. No transfer descriptors are flushed/invalidated. I see no cache operations happening on any DMA control structures, eve

Re: CPU Cache and busdma usage in USB

2009-07-14 Thread Rafal Jaworowski
On 2009-07-14, at 18:58, Marcel Moolenaar wrote: On Jul 14, 2009, at 2:21 AM, Rafal Jaworowski wrote: On 2009-07-14, at 10:36, Hans Petter Selasky wrote: On Tuesday 14 July 2009 10:31:10 Piotr Zięcik wrote: 1) My analysis: Only the data areas are being flushed/ invalidated. No transfer de

Re: CPU Cache and busdma usage in USB

2009-07-14 Thread Marcel Moolenaar
On Jul 14, 2009, at 2:21 AM, Rafal Jaworowski wrote: On 2009-07-14, at 10:36, Hans Petter Selasky wrote: On Tuesday 14 July 2009 10:31:10 Piotr Zięcik wrote: 1) My analysis: Only the data areas are being flushed/ invalidated. No transfer descriptors are flushed/invalidated. I see no cache

Re: CPU Cache and busdma usage in USB

2009-07-16 Thread Hans Petter Selasky
On Tuesday 14 July 2009 19:20:22 Rafal Jaworowski wrote: > >> Please note these problems should be considered as a showstopper > >> for the release since USB is currently broken on at least three ARM > >> platforms in the tree (Marvell). Hi, First of all I'm not able to boot into userland on my

Re: CPU Cache and busdma usage in USB

2009-07-17 Thread Rafal Jaworowski
On 2009-07-16, at 21:56, Hans Petter Selasky wrote: On Tuesday 14 July 2009 19:20:22 Rafal Jaworowski wrote: Please note these problems should be considered as a showstopper for the release since USB is currently broken on at least three ARM platforms in the tree (Marvell). Hi, First of al

Re: CPU Cache and busdma usage in USB

2009-07-17 Thread Bernd Walter
On Thu, Jul 16, 2009 at 09:56:04PM +0200, Hans Petter Selasky wrote: > On Tuesday 14 July 2009 19:20:22 Rafal Jaworowski wrote: > > > >> Please note these problems should be considered as a showstopper > > >> for the release since USB is currently broken on at least three ARM > > >> platforms in t

Re: CPU Cache and busdma usage in USB

2009-07-17 Thread Hans Petter Selasky
On Friday 17 July 2009 12:23:57 Rafal Jaworowski wrote: > On 2009-07-16, at 21:56, Hans Petter Selasky wrote: > > On Tuesday 14 July 2009 19:20:22 Rafal Jaworowski wrote: > Please note these problems should be considered as a showstopper > for the release since USB is currently broken on

Re: CPU Cache and busdma usage in USB

2009-07-17 Thread Olivier Houchard
On Fri, Jul 17, 2009 at 12:23:57PM +0200, Rafal Jaworowski wrote: > > >With the following patch in arm/cpufunc.c my UMASS device gets > >detected at > >boot time (KB9202B ARM board). Else I get a USB request timeout > >followed by a > >panic. But data transfers of more than 16KByte are likely