Re: Resizeable PCI BAR support V5
Hello Christian, after (long ;-)) vacation, injured wife (bad lumbago/luckily NO disc prolapse) on 2cond day @ our target, our daughter's 12th birthday, school start for both kids and mostly dad work I'm back... Do you have a V6 handy. Will do my fingers dirty if no Intel guy beats me at it. Greetings, Dieter Am 30.06.2017 14:55, schrieb Christian König: Hi Dieter, thanks a lot for testing that. But I think my poor little FUJITSU PRIMERGY TX150 S7, Xeon X3470 (Nehalem), PCIe 2.0, 24 GB is to old for this stuff... Well, actually you only need to figure out how to enable a PCIe window above the 4GB limit. Could be that the BIOS supports this with the ACPI tables (totally unlikely) or you could try to dig up the Northbridge documentation for this CPU from Intel and use my patch for the AMD CPUs as blueprint how to do this on an Intel CPU as well. Fact is you GFX hardware is perfectly capable of doing this, it's just that the BIOS/Motherboard didn't enabled a PCIe window per default to avoid problems with 32bit OSes. Regards, Christian. Am 30.06.2017 um 01:51 schrieb Dieter Nützel: Hello Christian, I've running this since you've sent it on-top of amd-staging-4.11. But I think my poor little FUJITSU PRIMERGY TX150 S7, Xeon X3470 (Nehalem), PCIe 2.0, 24 GB is to old for this stuff... [1.066475] pci :05:00.0: VF(n) BAR0 space: [mem 0x-0x0003 64bit] (contains BAR0 for 16 VFs) [1.066489] pci :05:00.0: VF(n) BAR2 space: [mem 0x-0x003f 64bit] (contains BAR2 for 16 VFs) [1.121656] pci :00:1c.0: BAR 15: assigned [mem 0x8000-0x801f 64bit pref] [1.121659] pci :00:1c.6: BAR 15: assigned [mem 0x8020-0x803f 64bit pref] [1.121662] pci :01:00.0: BAR 6: assigned [mem 0xb012-0xb013 pref] [1.121681] pci :05:00.0: BAR 6: assigned [mem 0xb028-0xb02f pref] [1.121683] pci :05:00.0: BAR 9: no space for [mem size 0x0040 64bit] [1.121684] pci :05:00.0: BAR 9: failed to assign [mem size 0x0040 64bit] [1.121685] pci :05:00.0: BAR 7: no space for [mem size 0x0004 64bit] [1.121687] pci :05:00.0: BAR 7: failed to assign [mem size 0x0004 64bit] [3.874180] amdgpu :01:00.0: BAR 0: releasing [mem 0xc000-0xcfff 64bit pref] [3.874182] amdgpu :01:00.0: BAR 2: releasing [mem 0xb040-0xb05f 64bit pref] [3.874198] pcieport :00:03.0: BAR 15: releasing [mem 0xb040-0xcfff 64bit pref] [3.874215] pcieport :00:03.0: BAR 15: no space for [mem size 0x3 64bit pref] [3.874217] pcieport :00:03.0: BAR 15: failed to assign [mem size 0x3 64bit pref] [3.874221] amdgpu :01:00.0: BAR 0: no space for [mem size 0x2 64bit pref] [3.874223] amdgpu :01:00.0: BAR 0: failed to assign [mem size 0x2 64bit pref] [3.874226] amdgpu :01:00.0: BAR 2: no space for [mem size 0x0020 64bit pref] [3.874227] amdgpu :01:00.0: BAR 2: failed to assign [mem size 0x0020 64bit pref] [3.874258] [drm] Not enough PCI address space for a large BAR. [3.874261] amdgpu :01:00.0: BAR 0: assigned [mem 0xc000-0xcfff 64bit pref] [3.874269] amdgpu :01:00.0: BAR 2: assigned [mem 0xb040-0xb05f 64bit pref] [3.874288] [drm] Detected VRAM RAM=8192M, BAR=256M Anyway rebase for current amd-staging-4.11 needed. Find attached dmesg-amd-staging-4.11-1.g7262353-default+.log.xz Greetings, Dieter Am 09.06.2017 10:59, schrieb Christian König: Hi everyone, This is the fith incarnation of this set of patches. It enables device drivers to resize and most likely also relocate the PCI BAR of devices they manage to allow the CPU to access all of the device local memory at once. This is very useful for GFX device drivers where the default PCI BAR is only about 256MB in size for compatibility reasons, but the device easily have multiple gigabyte of local memory. Some changes since V4: 1. Rebased on 4.11. 2. added the rb from Andy Shevchenko to patches which look complete now. 3. Move releasing the BAR and reallocating it on error to the driver side. 4. Add amdgpu support for GMC V6 hardware generation as well. Please review and/or comment, Christian. ___ dri-devel mailing list dri-de...@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: Resizeable PCI BAR support V5
Hi Dieter, thanks a lot for testing that. But I think my poor little FUJITSU PRIMERGY TX150 S7, Xeon X3470 (Nehalem), PCIe 2.0, 24 GB is to old for this stuff... Well, actually you only need to figure out how to enable a PCIe window above the 4GB limit. Could be that the BIOS supports this with the ACPI tables (totally unlikely) or you could try to dig up the Northbridge documentation for this CPU from Intel and use my patch for the AMD CPUs as blueprint how to do this on an Intel CPU as well. Fact is you GFX hardware is perfectly capable of doing this, it's just that the BIOS/Motherboard didn't enabled a PCIe window per default to avoid problems with 32bit OSes. Regards, Christian. Am 30.06.2017 um 01:51 schrieb Dieter Nützel: Hello Christian, I've running this since you've sent it on-top of amd-staging-4.11. But I think my poor little FUJITSU PRIMERGY TX150 S7, Xeon X3470 (Nehalem), PCIe 2.0, 24 GB is to old for this stuff... [1.066475] pci :05:00.0: VF(n) BAR0 space: [mem 0x-0x0003 64bit] (contains BAR0 for 16 VFs) [1.066489] pci :05:00.0: VF(n) BAR2 space: [mem 0x-0x003f 64bit] (contains BAR2 for 16 VFs) [1.121656] pci :00:1c.0: BAR 15: assigned [mem 0x8000-0x801f 64bit pref] [1.121659] pci :00:1c.6: BAR 15: assigned [mem 0x8020-0x803f 64bit pref] [1.121662] pci :01:00.0: BAR 6: assigned [mem 0xb012-0xb013 pref] [1.121681] pci :05:00.0: BAR 6: assigned [mem 0xb028-0xb02f pref] [1.121683] pci :05:00.0: BAR 9: no space for [mem size 0x0040 64bit] [1.121684] pci :05:00.0: BAR 9: failed to assign [mem size 0x0040 64bit] [1.121685] pci :05:00.0: BAR 7: no space for [mem size 0x0004 64bit] [1.121687] pci :05:00.0: BAR 7: failed to assign [mem size 0x0004 64bit] [3.874180] amdgpu :01:00.0: BAR 0: releasing [mem 0xc000-0xcfff 64bit pref] [3.874182] amdgpu :01:00.0: BAR 2: releasing [mem 0xb040-0xb05f 64bit pref] [3.874198] pcieport :00:03.0: BAR 15: releasing [mem 0xb040-0xcfff 64bit pref] [3.874215] pcieport :00:03.0: BAR 15: no space for [mem size 0x3 64bit pref] [3.874217] pcieport :00:03.0: BAR 15: failed to assign [mem size 0x3 64bit pref] [3.874221] amdgpu :01:00.0: BAR 0: no space for [mem size 0x2 64bit pref] [3.874223] amdgpu :01:00.0: BAR 0: failed to assign [mem size 0x2 64bit pref] [3.874226] amdgpu :01:00.0: BAR 2: no space for [mem size 0x0020 64bit pref] [3.874227] amdgpu :01:00.0: BAR 2: failed to assign [mem size 0x0020 64bit pref] [3.874258] [drm] Not enough PCI address space for a large BAR. [3.874261] amdgpu :01:00.0: BAR 0: assigned [mem 0xc000-0xcfff 64bit pref] [3.874269] amdgpu :01:00.0: BAR 2: assigned [mem 0xb040-0xb05f 64bit pref] [3.874288] [drm] Detected VRAM RAM=8192M, BAR=256M Anyway rebase for current amd-staging-4.11 needed. Find attached dmesg-amd-staging-4.11-1.g7262353-default+.log.xz Greetings, Dieter Am 09.06.2017 10:59, schrieb Christian König: Hi everyone, This is the fith incarnation of this set of patches. It enables device drivers to resize and most likely also relocate the PCI BAR of devices they manage to allow the CPU to access all of the device local memory at once. This is very useful for GFX device drivers where the default PCI BAR is only about 256MB in size for compatibility reasons, but the device easily have multiple gigabyte of local memory. Some changes since V4: 1. Rebased on 4.11. 2. added the rb from Andy Shevchenko to patches which look complete now. 3. Move releasing the BAR and reallocating it on error to the driver side. 4. Add amdgpu support for GMC V6 hardware generation as well. Please review and/or comment, Christian. ___ dri-devel mailing list dri-de...@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: Resizeable PCI BAR support V5
Hello Christian, I've running this since you've sent it on-top of amd-staging-4.11. But I think my poor little FUJITSU PRIMERGY TX150 S7, Xeon X3470 (Nehalem), PCIe 2.0, 24 GB is to old for this stuff... [1.066475] pci :05:00.0: VF(n) BAR0 space: [mem 0x-0x0003 64bit] (contains BAR0 for 16 VFs) [1.066489] pci :05:00.0: VF(n) BAR2 space: [mem 0x-0x003f 64bit] (contains BAR2 for 16 VFs) [1.121656] pci :00:1c.0: BAR 15: assigned [mem 0x8000-0x801f 64bit pref] [1.121659] pci :00:1c.6: BAR 15: assigned [mem 0x8020-0x803f 64bit pref] [1.121662] pci :01:00.0: BAR 6: assigned [mem 0xb012-0xb013 pref] [1.121681] pci :05:00.0: BAR 6: assigned [mem 0xb028-0xb02f pref] [1.121683] pci :05:00.0: BAR 9: no space for [mem size 0x0040 64bit] [1.121684] pci :05:00.0: BAR 9: failed to assign [mem size 0x0040 64bit] [1.121685] pci :05:00.0: BAR 7: no space for [mem size 0x0004 64bit] [1.121687] pci :05:00.0: BAR 7: failed to assign [mem size 0x0004 64bit] [3.874180] amdgpu :01:00.0: BAR 0: releasing [mem 0xc000-0xcfff 64bit pref] [3.874182] amdgpu :01:00.0: BAR 2: releasing [mem 0xb040-0xb05f 64bit pref] [3.874198] pcieport :00:03.0: BAR 15: releasing [mem 0xb040-0xcfff 64bit pref] [3.874215] pcieport :00:03.0: BAR 15: no space for [mem size 0x3 64bit pref] [3.874217] pcieport :00:03.0: BAR 15: failed to assign [mem size 0x3 64bit pref] [3.874221] amdgpu :01:00.0: BAR 0: no space for [mem size 0x2 64bit pref] [3.874223] amdgpu :01:00.0: BAR 0: failed to assign [mem size 0x2 64bit pref] [3.874226] amdgpu :01:00.0: BAR 2: no space for [mem size 0x0020 64bit pref] [3.874227] amdgpu :01:00.0: BAR 2: failed to assign [mem size 0x0020 64bit pref] [3.874258] [drm] Not enough PCI address space for a large BAR. [3.874261] amdgpu :01:00.0: BAR 0: assigned [mem 0xc000-0xcfff 64bit pref] [3.874269] amdgpu :01:00.0: BAR 2: assigned [mem 0xb040-0xb05f 64bit pref] [3.874288] [drm] Detected VRAM RAM=8192M, BAR=256M Anyway rebase for current amd-staging-4.11 needed. Find attached dmesg-amd-staging-4.11-1.g7262353-default+.log.xz Greetings, Dieter Am 09.06.2017 10:59, schrieb Christian König: Hi everyone, This is the fith incarnation of this set of patches. It enables device drivers to resize and most likely also relocate the PCI BAR of devices they manage to allow the CPU to access all of the device local memory at once. This is very useful for GFX device drivers where the default PCI BAR is only about 256MB in size for compatibility reasons, but the device easily have multiple gigabyte of local memory. Some changes since V4: 1. Rebased on 4.11. 2. added the rb from Andy Shevchenko to patches which look complete now. 3. Move releasing the BAR and reallocating it on error to the driver side. 4. Add amdgpu support for GMC V6 hardware generation as well. Please review and/or comment, Christian. ___ dri-devel mailing list dri-de...@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel dmesg-amd-staging-4.11-1.g7262353-default+.log.xz Description: application/xz
Re: Resizeable PCI BAR support V5
On Fri, Jun 09, 2017 at 10:59:41AM +0200, Christian König wrote: > Hi everyone, > > This is the fith incarnation of this set of patches. It enables device > drivers to resize and most likely also relocate the PCI BAR of devices > they manage to allow the CPU to access all of the device local memory at once. I think this is really v7, isn't it? I see a v6 posted May 9. > This is very useful for GFX device drivers where the default PCI BAR is only > about 256MB in size for compatibility reasons, but the device easily have > multiple gigabyte of local memory. > > Some changes since V4: > 1. Rebased on 4.11. I apply patches to topic branches based on -rc1. I did apply these by hand, but it's easier if they apply cleanly to -rc1. > 2. added the rb from Andy Shevchenko to patches which look complete now. > 3. Move releasing the BAR and reallocating it on error to the driver side. > 4. Add amdgpu support for GMC V6 hardware generation as well. > > Please review and/or comment, > Christian. >
Re: Resizeable PCI BAR support v5
On Thu, May 4, 2017 at 12:31 PM, Christian König wrote: > Hi everyone, > > this is the fifth incarnation of this set of patches. It enables device > drivers to resize and most likely also relocate the PCI BAR of devices > they manage to allow the CPU to access all of the device local memory at once. > > This is very useful for GFX device drivers where the default PCI BAR is only > about 256MB in size for compatibility reasons, but the device easily have > multiple gigabyte of local memory. > > Noteable changed compared to v4: > 1. Patch #1 is new and a minor cleanup to the PCI code. > 2. Some more helpers added to patch #2 > 3. We now print a note to syslog before releasing resources. > 4. Add defines for registers and bits used to add new root hub window. > > I've addressed mostly every review comment I've got so far and it looks like > we > are getting closer to landing this. Any more suggestions/hints how we could > improve the patchset would be very welcome. > Thanks for an update. I'm going to comment individual patches. Just for the future, please use versioning in the patches themselves (hint: -vX to git-format-patch). -- With Best Regards, Andy Shevchenko
