Re: [PATCH] x86-64: disable the GART before allocate aperture
> Yes, kdump kernel is generally <4GB . Won't I require IOMMU while I am > copying the high memory contents in second kernel (lets say 16 GB of memory > and destination device is not capable of addressing anything more than 4G > for DMA operation)? It cannot happen; you don't support mmap on vmcore. Also even if it was possible it would likely work for networking/block IO because they support bouncing using their own mechanism. When you read() from vmcore to write the data out you will always use the CPU to copy and then do IO from the copied data. > > If it's not then swiotlb will also not work because it won't get > > any memory <4GB. > > Will using IOMMU instead of swiotlb give noticeable perfomance boost. The difference is not that dramatic and less and less IO devices need it anyways. And if you're <4GB you will never need it. -Andi - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] x86-64: disable the GART before allocate aperture
On Sat, Jun 23, 2007 at 02:14:01AM +0200, Andi Kleen wrote: > On Saturday 23 June 2007 00:19:51 Alan Cox wrote: > > > YH do you think you can look at simply reserving a portion of the iommu? > > > And having the kexec on panic kernel use the reserved portion? > > > > How about simply reserving all of it for the base kernel and using soft > > iommu for the panic kernel, its hardly high performance criticial at this > > point. > > The kdump kernel should be normally all <4GB anyways. You won't > need any IOMMU for its IO unless you O_DIRECT/sendfile out of /proc/kcore. > Just don't do that (but I suspect it won't work anyways) > Hi Andi, Yes, kdump kernel is generally <4GB . Won't I require IOMMU while I am copying the high memory contents in second kernel (lets say 16 GB of memory and destination device is not capable of addressing anything more than 4G for DMA operation)? > If it's not then swiotlb will also not work because it won't get > any memory <4GB. > Will using IOMMU instead of swiotlb give noticeable perfomance boost. One of the goals is to automatically save the crash dump as soon as possible and boot back into production kernel to reduce system down time and reduce availability. Right now I am trvelling. After OLS will try to come up with a patch for reserving a couple of IOMMU entries for kdump purposes. Only issue is that any boot time reservation requirements make it impossible to enable a feature at run time without rebooting the system. Thanks Vivek - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] x86-64: disable the GART before allocate aperture
On Fri, Jun 22, 2007 at 05:27:50PM -0700, Yinghai Lu wrote: > Andi Kleen wrote: > >On Saturday 23 June 2007 00:19:51 Alan Cox wrote: > >The kdump kernel should be normally all <4GB anyways. You won't > >need any IOMMU for its IO unless you O_DIRECT/sendfile out of /proc/kcore. > >Just don't do that (but I suspect it won't work anyways) > > > >If it's not then swiotlb will also not work because it won't get > >any memory <4GB. > > > >But I doubt this was YH's problem - the panic kernel memory > >is always reserved and there shouldn't be any ongoing DMAs in this > >area anyways. And what happens outside the kdump kernel shouldn't matter. > > > >I suspect he rather saw problems with non kdump kexec where we > >can just shut down the GART properly beforehand. > > current I only test kexec only. So clean shut GART in first kernel will > help. > > where is hook for shutdown? add one in dma_ops? I was wondering the same thing. I think it should hook into the standard device model. It definitely should *not* be in the dma_ops (it has nothing to do with DMA...) - if needed for this or something else we should add a struct iommu_ops which includes a dma_ops member. Cheers, Muli - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] x86-64: disable the GART before allocate aperture
> But I doubt this was YH's problem - the panic kernel memory > is always reserved and there shouldn't be any ongoing DMAs in this > area anyways. And what happens outside the kdump kernel shouldn't matter. In the kdump case it looks like there is still DMA going on through the GART on some systems when kdump zaps it. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] x86-64: disable the GART before allocate aperture
> current I only test kexec only. So clean shut GART in first kernel will help. > > where is hook for shutdown? add one in dma_ops? The low level code could just register its own shutdown handler. No need to go through dma_ops I think. -Andi - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] x86-64: disable the GART before allocate aperture
Andi Kleen wrote: On Saturday 23 June 2007 00:19:51 Alan Cox wrote: The kdump kernel should be normally all <4GB anyways. You won't need any IOMMU for its IO unless you O_DIRECT/sendfile out of /proc/kcore. Just don't do that (but I suspect it won't work anyways) If it's not then swiotlb will also not work because it won't get any memory <4GB. But I doubt this was YH's problem - the panic kernel memory is always reserved and there shouldn't be any ongoing DMAs in this area anyways. And what happens outside the kdump kernel shouldn't matter. I suspect he rather saw problems with non kdump kexec where we can just shut down the GART properly beforehand. current I only test kexec only. So clean shut GART in first kernel will help. where is hook for shutdown? add one in dma_ops? YH - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] x86-64: disable the GART before allocate aperture
On Saturday 23 June 2007 00:19:51 Alan Cox wrote: > > YH do you think you can look at simply reserving a portion of the iommu? > > And having the kexec on panic kernel use the reserved portion? > > How about simply reserving all of it for the base kernel and using soft > iommu for the panic kernel, its hardly high performance criticial at this > point. The kdump kernel should be normally all <4GB anyways. You won't need any IOMMU for its IO unless you O_DIRECT/sendfile out of /proc/kcore. Just don't do that (but I suspect it won't work anyways) If it's not then swiotlb will also not work because it won't get any memory <4GB. But I doubt this was YH's problem - the panic kernel memory is always reserved and there shouldn't be any ongoing DMAs in this area anyways. And what happens outside the kdump kernel shouldn't matter. I suspect he rather saw problems with non kdump kexec where we can just shut down the GART properly beforehand. -Andi - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] x86-64: disable the GART before allocate aperture
Alan Cox <[EMAIL PROTECTED]> writes: > On Fri, 22 Jun 2007 15:57:08 -0700 > Yinghai Lu <[EMAIL PROTECTED]> wrote: > >> Alan Cox wrote: >> >> > Don't disable it, just don't touch it or any of its mappings. Leave it >> > *alone*, and use swiotlb. That'll maximise the ability to recover stuff >> > from the kexec kernel (since for one you may want to dump the gart when a >> > 3d app goes kerblam) >> >> How about LinuxBIOS + Kernel ===> Final kernel path? >> someday EFI(PEI) + Kernel ===> Final kernel path need that too. >> >> or the normal kexec path still could have clean shutdown. > > The kexec path for kdump should be swiotlb and leave the GART alone as > you are dumping as much state as you can and leaving stuff as is when > possible. The new-kernel case you shut everything down so you can shut > down the GART in the old kernel and re-initialise it in the new one Agreed. Eric - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] x86-64: disable the GART before allocate aperture
On Fri, 22 Jun 2007 15:57:08 -0700 Yinghai Lu <[EMAIL PROTECTED]> wrote: > Alan Cox wrote: > > > Don't disable it, just don't touch it or any of its mappings. Leave it > > *alone*, and use swiotlb. That'll maximise the ability to recover stuff > > from the kexec kernel (since for one you may want to dump the gart when a > > 3d app goes kerblam) > > How about LinuxBIOS + Kernel ===> Final kernel path? > someday EFI(PEI) + Kernel ===> Final kernel path need that too. > > or the normal kexec path still could have clean shutdown. The kexec path for kdump should be swiotlb and leave the GART alone as you are dumping as much state as you can and leaving stuff as is when possible. The new-kernel case you shut everything down so you can shut down the GART in the old kernel and re-initialise it in the new one Alan - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] x86-64: disable the GART before allocate aperture
Alan Cox wrote: Don't disable it, just don't touch it or any of its mappings. Leave it *alone*, and use swiotlb. That'll maximise the ability to recover stuff from the kexec kernel (since for one you may want to dump the gart when a 3d app goes kerblam) How about LinuxBIOS + Kernel ===> Final kernel path? someday EFI(PEI) + Kernel ===> Final kernel path need that too. or the normal kexec path still could have clean shutdown. YH - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] x86-64: disable the GART before allocate aperture
On Fri, 22 Jun 2007 15:43:00 -0700 Yinghai Lu <[EMAIL PROTECTED]> wrote: > Eric W. Biederman wrote: > > The original design came from thinking about systems where using the iommu > > was mandatory. I think we almost always reserve memory below 1G for the > > kexec > > on panic kernel so it really shouldn't be an issue in that case. Except > > we need to pass an option to force not using the iommu. I don't think > > noiommu or swiotlb is going to make any real difference. > > > > So I'm totally in favor of turning off features if we don't need them and we > > don't take a tremendous performance hit. (People get grumpy when writing > > all of memory to disk takes completely unreasonable amounts of time). > > > So you prefer to > add diable_gart in shutdown or suspend func and let kexec to use swiotlb > comand line? Don't disable it, just don't touch it or any of its mappings. Leave it *alone*, and use swiotlb. That'll maximise the ability to recover stuff from the kexec kernel (since for one you may want to dump the gart when a 3d app goes kerblam) Alan - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] x86-64: disable the GART before allocate aperture
Eric W. Biederman wrote: The original design came from thinking about systems where using the iommu was mandatory. I think we almost always reserve memory below 1G for the kexec on panic kernel so it really shouldn't be an issue in that case. Except we need to pass an option to force not using the iommu. I don't think noiommu or swiotlb is going to make any real difference. So I'm totally in favor of turning off features if we don't need them and we don't take a tremendous performance hit. (People get grumpy when writing all of memory to disk takes completely unreasonable amounts of time). So you prefer to add diable_gart in shutdown or suspend func and let kexec to use swiotlb comand line? YH - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] x86-64: disable the GART before allocate aperture
Alan Cox <[EMAIL PROTECTED]> writes: >> YH do you think you can look at simply reserving a portion of the iommu? >> And having the kexec on panic kernel use the reserved portion? > > How about simply reserving all of it for the base kernel and using soft > iommu for the panic kernel, its hardly high performance criticial at this > point. The original design came from thinking about systems where using the iommu was mandatory. I think we almost always reserve memory below 1G for the kexec on panic kernel so it really shouldn't be an issue in that case. Except we need to pass an option to force not using the iommu. I don't think noiommu or swiotlb is going to make any real difference. So I'm totally in favor of turning off features if we don't need them and we don't take a tremendous performance hit. (People get grumpy when writing all of memory to disk takes completely unreasonable amounts of time). Eric - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] x86-64: disable the GART before allocate aperture
> YH do you think you can look at simply reserving a portion of the iommu? > And having the kexec on panic kernel use the reserved portion? How about simply reserving all of it for the base kernel and using soft iommu for the panic kernel, its hardly high performance criticial at this point. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] x86-64: disable the GART before allocate aperture
On 6/22/07, Eric W. Biederman <[EMAIL PROTECTED]> wrote: Alan Cox <[EMAIL PROTECTED]> writes: For a normal kexec we should shut everything down before the kernel transition so it should not be an issue. YH do you think you can look at simply reserving a portion of the iommu? And having the kexec on panic kernel use the reserved portion? two copy region: one for first kernel, and one for second kernel? it should work. first kernel is using [64M, 128M), and the second will get assign to [64M,128M) again. when it try to memset to clear that region it will cause restart. in that region, only first 256K can not touched, even read. rest could be accessed. YH - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] x86-64: disable the GART before allocate aperture
On Fri, Jun 22, 2007 at 03:32:53PM -0600, Eric W. Biederman wrote: > Alan Cox <[EMAIL PROTECTED]> writes: > > > You've got mapped live gart pages from the previous kernel. Even > > if you disable the gart before a memset you may well have the > > video card using gart translations and possibly live IOMMU > > mappings for devices using it via bus mastering - and those will > > cause you MCE exceptions with a corrupt cpu context flag (ie not > > nicely recoverable). > > The original plan (which we have not followed up on). Was to > reserve a chunk of any iommu for the kexec on panic kernel. Then to > just have the second kernel use that unused chunk. This is how we > treat the normal memory space and it seems a nice and simple > approach to this kind of problem. > > For a normal kexec we should shut everything down before the kernel > transition so it should not be an issue. > > YH do you think you can look at simply reserving a portion of the > iommu? And having the kexec on panic kernel use the reserved > portion? How would this work with an isolation capable IOMMU which has different address spaces for different devices? (e.g., Calgary which is in mainline, Intel's VT-d which is coming soon, etc). Cheers, Muli - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] x86-64: disable the GART before allocate aperture
Andi Kleen <[EMAIL PROTECTED]> writes: > On Friday 22 June 2007 22:33, Alan Cox wrote: >> You've got mapped live gart pages from the previous kernel. Even if you >> disable the gart before a memset > > It's probably too late then. It could also interfere with other operations. > If anything the GART should be disabled during kexec shutdown. Perhaps we > just > need a suitable suspend function that does that. Eric, any preferences? Well it would be a shutdown method not a suspend method. The authors of the suspend code thought sharing code with the reboot and the rmmod case didn't make sense. For a normal kexec into another kernel I think this makes sense. >> you may well have the video card using >> gart translations and possibly live IOMMU mappings for devices using it >> via bus mastering - and those will cause you MCE exceptions with a >> corrupt cpu context flag (ie not nicely recoverable). > > We disable those machine checks on K8 because they're not fully reliable. For the kexec on panic not shutting the hardware down if at all possible is the ideal. There I think we need a mode to not touch a chunk of the iommu and reserve it for the kexec on panic kernel (or perhaps just use the swiotlb code if we need bounce buffers at all). Eric - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] x86-64: disable the GART before allocate aperture
Andi Kleen wrote: It's probably too late then. It could also interfere with other operations. If anything the GART should be disabled during kexec shutdown. Perhaps we just need a suitable suspend function that does that. Eric, any preferences? how about kdump? do we have chance to call that suspend func? YH - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] x86-64: disable the GART before allocate aperture
Alan Cox <[EMAIL PROTECTED]> writes: > You've got mapped live gart pages from the previous kernel. Even if you > disable the gart before a memset you may well have the video card using > gart translations and possibly live IOMMU mappings for devices using it > via bus mastering - and those will cause you MCE exceptions with a > corrupt cpu context flag (ie not nicely recoverable). The original plan (which we have not followed up on). Was to reserve a chunk of any iommu for the kexec on panic kernel. Then to just have the second kernel use that unused chunk. This is how we treat the normal memory space and it seems a nice and simple approach to this kind of problem. For a normal kexec we should shut everything down before the kernel transition so it should not be an issue. YH do you think you can look at simply reserving a portion of the iommu? And having the kexec on panic kernel use the reserved portion? Eric - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] x86-64: disable the GART before allocate aperture
On Friday 22 June 2007 22:33, Alan Cox wrote: > On Fri, 22 Jun 2007 12:31:24 -0700 > > Muli Ben-Yehuda <[EMAIL PROTECTED]> wrote: > > On Fri, Jun 22, 2007 at 12:19:15PM -0700, Yinghai Lu wrote: > > > [PATCH] x86-64: disable the GART before allocate aperture > > > > > > For K8 system: 4G RAM with memory hole remapping enabled, or more > > > than 4G RAM installed. when mem is allocated for GART, it will do > > > the memset for clear. and for kexec case, the first kernel already > > > enable that, the memset in second kernel will cause the system > > > restart. So disable that at first before we try to allocate mem for > > > it. > > > > Why does the memset in the second kernel cause a system restart? > > You've got mapped live gart pages from the previous kernel. Even if you > disable the gart before a memset It's probably too late then. It could also interfere with other operations. If anything the GART should be disabled during kexec shutdown. Perhaps we just need a suitable suspend function that does that. Eric, any preferences? > you may well have the video card using > gart translations and possibly live IOMMU mappings for devices using it > via bus mastering - and those will cause you MCE exceptions with a > corrupt cpu context flag (ie not nicely recoverable). We disable those machine checks on K8 because they're not fully reliable. -Andi - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] x86-64: disable the GART before allocate aperture
On Fri, 22 Jun 2007 12:31:24 -0700 Muli Ben-Yehuda <[EMAIL PROTECTED]> wrote: > On Fri, Jun 22, 2007 at 12:19:15PM -0700, Yinghai Lu wrote: > > [PATCH] x86-64: disable the GART before allocate aperture > > > > For K8 system: 4G RAM with memory hole remapping enabled, or more > > than 4G RAM installed. when mem is allocated for GART, it will do > > the memset for clear. and for kexec case, the first kernel already > > enable that, the memset in second kernel will cause the system > > restart. So disable that at first before we try to allocate mem for > > it. > > Why does the memset in the second kernel cause a system restart? You've got mapped live gart pages from the previous kernel. Even if you disable the gart before a memset you may well have the video card using gart translations and possibly live IOMMU mappings for devices using it via bus mastering - and those will cause you MCE exceptions with a corrupt cpu context flag (ie not nicely recoverable). Alan - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] x86-64: disable the GART before allocate aperture
Muli Ben-Yehuda wrote: On Fri, Jun 22, 2007 at 12:19:15PM -0700, Yinghai Lu wrote: [PATCH] x86-64: disable the GART before allocate aperture For K8 system: 4G RAM with memory hole remapping enabled, or more than 4G RAM installed. when mem is allocated for GART, it will do the memset for clear. and for kexec case, the first kernel already enable that, the memset in second kernel will cause the system restart. So disable that at first before we try to allocate mem for it. Why does the memset in the second kernel cause a system restart? GART is enabled by first kernel, and some driver is using that for DMA. YH - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] x86-64: disable the GART before allocate aperture
On 6/22/07, Muli Ben-Yehuda <[EMAIL PROTECTED]> wrote: On Fri, Jun 22, 2007 at 12:19:15PM -0700, Yinghai Lu wrote: > [PATCH] x86-64: disable the GART before allocate aperture > > For K8 system: 4G RAM with memory hole remapping enabled, or more > than 4G RAM installed. when mem is allocated for GART, it will do > the memset for clear. and for kexec case, the first kernel already > enable that, the memset in second kernel will cause the system > restart. So disable that at first before we try to allocate mem for > it. Why does the memset in the second kernel cause a system restart? the mem for aperture is still used by GART. or the translation is still enabled. YH - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] x86-64: disable the GART before allocate aperture
On Fri, Jun 22, 2007 at 12:19:15PM -0700, Yinghai Lu wrote: > [PATCH] x86-64: disable the GART before allocate aperture > > For K8 system: 4G RAM with memory hole remapping enabled, or more > than 4G RAM installed. when mem is allocated for GART, it will do > the memset for clear. and for kexec case, the first kernel already > enable that, the memset in second kernel will cause the system > restart. So disable that at first before we try to allocate mem for > it. Why does the memset in the second kernel cause a system restart? Cheers, Muli - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
