Re: Detecting NUMA per pmem
Hi All We encountered a strange issue using pmem emulation. We configure emulated pmem devices in our system by adding these params to the boot command: memmap=32G!224G memmap=32G!480G . PMEM devices look OK. This issue is specific to 4.14 kernel AND centos 7.5, other combinations follow exactly the same flow but don't crash. Happens when ndctl converts an /dev/pmem0 to /dev/dax0.0 command: ndctl create-namespace -f -e namespace0.0 --type=pmem --mode=dax This crashes the kernel in udev when trying to free page in do_munmap() syscall. And hangs. echo namespace0.0 > /sys/devices/platform/e820_pmem/ndbus0/region0/namespace0.0/driver/unbind Calltrace(): ndctl_unbind() ndctl_namespace_disable() ndctl_namespace_disable_invalidate() ndctl_namespace_disable_safe() namespace_destroy() namespace_reconfig() do_xaction_namespace() When we use for example ubuntu 16.04 with this kernel version it does not happen. When we use centos7.5 and kernel version 4.9 is also does not happen. When working with actual NVDIMMs this does not happen Did you encounter such an issue? Why does the kernel think that this area is mapped and who might be mapping it? If no one maps it why does the kernel has indication that these pages are mapped? Any help would be highly appreciated. Thanks Oren Berman On 10 January 2018 at 09:41, Oren Berman wrote: > Hi > Thanks for your answer > If we do memremap on the physical address > Of the nvram from within the kernel to get a new virtual address mapping > will it lock the mapping? > Can this be also a workaround? > Oren > > נשלח מה-iPhone שלי > > ב-10 בינו׳ 2018, בשעה 18:38, Dan Williams > כתב/ה: > > On Wed, Jan 10, 2018 at 7:23 AM, Oren Berman > wrote: > > Now to all of the forum > > > Hi Dan > > > Thanks we are going to try this. > > > Can you explain why this can cause this issue - is the NVDIMM memory space > > also being randomized? > > Is it done during runtime? > > > > Yes, kaslr randomizes the direct map. We have seen problems with it in > the past relative to setting up pmem mappings. We fixed one such bug > with this commit: > > fc5f9d5f151c x86/mm: Fix boot crash caused by incorrect loop count > calculation in sync_global_pgds() > > ...but it appears we may have another bug in this area. > > ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: Detecting NUMA per pmem
Hi Ross Thanks for the speedy reply. I am also adding the public list to this thread as you suggested. We have tried to dump the SPA table and this is what we get: /* * Intel ACPI Component Architecture * AML/ASL+ Disassembler version 20160108-64 * Copyright (c) 2000 - 2016 Intel Corporation * * Disassembly of NFIT, Sun Oct 22 10:46:19 2017 * * ACPI Data Table [NFIT] * * Format: [HexOffset DecimalOffset ByteLength] FieldName : FieldValue */ [000h 4]Signature : "NFIT"[NVDIMM Firmware Interface Table] [004h 0004 4] Table Length : 0028 [008h 0008 1] Revision : 01 [009h 0009 1] Checksum : B2 [00Ah 0010 6] Oem ID : "SUPERM" [010h 0016 8] Oem Table ID : "SMCI--MB" [018h 0024 4] Oem Revision : 0001 [01Ch 0028 4] Asl Compiler ID : " " [020h 0032 4]Asl Compiler Revision : 0001 [024h 0036 4] Reserved : Raw Table Data: Length 40 (0x28) : 4E 46 49 54 28 00 00 00 01 B2 53 55 50 45 52 4D // NFIT(.SUPERM 0010: 53 4D 43 49 2D 2D 4D 42 01 00 00 00 01 00 00 00 // SMCI--MB 0020: 01 00 00 00 00 00 00 00 As you can see the memory region info is missing. This specific check was done on a supermicro server. We also performed a bios update but the results were the same. As said before ,the pmem devices are detected correctly and we verified that they correspond to different numa nodes using the PCM utility.However, linux still reports both pmem devices to be on the same numa - Numa 0. If this information is missing, why pmem devices and address ranges are still detected correctly? Is there another table that we need to check? I also ran dmidecode and the NVDIMMs are being listed (we tested with netlist NVDIMMs). I can also see the bank locator showing P0 and P1 which I think indicates the numa. Here is an example: Handle 0x002D, DMI type 17, 40 bytes Memory Device Array Handle: 0x002A Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 16384 MB Form Factor: DIMM Set: None Locator: P1-DIMMA3 Bank Locator: P0_Node0_Channel0_Dimm2 Type: DDR4 Type Detail: Synchronous Speed: 2400 MHz Manufacturer: Netlist Serial Number: 66F50006 Asset Tag: P1-DIMMA3_AssetTag (date:16/42) Part Number: NV3A74SBT20-000 Rank: 1 Configured Clock Speed: 1600 MHz Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Handle 0x003B, DMI type 17, 40 bytes Memory Device Array Handle: 0x0038 Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 16384 MB Form Factor: DIMM Set: None Locator: P2-DIMME3 Bank Locator: P1_Node1_Channel0_Dimm2 Type: DDR4 Type Detail: Synchronous Speed: 2400 MHz Manufacturer: Netlist Serial Number: 66B50010 Asset Tag: P2-DIMME3_AssetTag (date:16/42) Part Number: NV3A74SBT20-000 Rank: 1 Configured Clock Speed: 1600 MHz Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Did you encounter such a a case? We would appreciate any insight you might have. BR Oren Berman On 20 October 2017 at 19:22, Ross Zwisler wrote: > On Thu, Oct 19, 2017 at 06:12:24PM +0300, Oren Berman wrote: > >Hi Ross > >My name is Oren Berman and I am a senior developer at lightbitslabs. > >We are working with NDIMMs but we encountered a problem that the > kernel > > does not seem to detect the numa id per PMEM device. > >It always reports numa 0 although we have NVDIMM devices on both > nodes. > >We checked that it always returns 0 from sysfs and also from > retrieving > >the device of pmem in the kernel and calling dev_to_node. > >The result is always 0 for both pmem0 and pmem1. > >In order to make sure that indeed both numa sockets are used we ran > >intel's pcm utlity. We verified that writing to pmem 0 increases > socket 0 > >utilization and writing to pmem1 increases socket 1 utilization so > the hw > >works properly. > >Only the detection seems to be invalid. > >Did you encounter such a problem? > >We are using kernel version 4.9 - are you aware of any fix for this > issue > >or workaround that we can use. > >Are we missing something? > >Thanks for any help you can give us. > >BR > >Oren Berman > > Hi Oren, > > My first guess is that your platform isn't properly filling out the > "proximity > domain" field in the NFIT SPA table. > > See section 5.2.25.2 in ACPI 6.2: > http://uefi.org/sites/default/files/resources/ACPI_6_2.pdf > > Here's how to check that: > > # cd /tmp > # cp /sys/firmware/acpi/tables/NFIT . > # iasl NFIT > > Intel ACPI Component Architecture > ASL+ Optimizing Compiler version 20160831-64 > Copyright (c) 2000 - 2016 Intel Corporation > > Binary file appears to be a valid ACPI table, disassembling > Input file NFIT, Length 0xE0 (224) bytes > ACPI: NFIT 0x
Re: Detecting NUMA per pmem
On Sun, Oct 22, 2017 at 4:33 AM, Oren Berman wrote: > Hi Ross > > Thanks for the speedy reply. I am also adding the public list to this > thread as you suggested. > > We have tried to dump the SPA table and this is what we get: > > /* > * Intel ACPI Component Architecture > * AML/ASL+ Disassembler version 20160108-64 > * Copyright (c) 2000 - 2016 Intel Corporation > * > * Disassembly of NFIT, Sun Oct 22 10:46:19 2017 > * > * ACPI Data Table [NFIT] > * > * Format: [HexOffset DecimalOffset ByteLength] FieldName : FieldValue > */ > > [000h 4]Signature : "NFIT"[NVDIMM Firmware > Interface Table] > [004h 0004 4] Table Length : 0028 > [008h 0008 1] Revision : 01 > [009h 0009 1] Checksum : B2 > [00Ah 0010 6] Oem ID : "SUPERM" > [010h 0016 8] Oem Table ID : "SMCI--MB" > [018h 0024 4] Oem Revision : 0001 > [01Ch 0028 4] Asl Compiler ID : " " > [020h 0032 4]Asl Compiler Revision : 0001 > > [024h 0036 4] Reserved : > > Raw Table Data: Length 40 (0x28) > > : 4E 46 49 54 28 00 00 00 01 B2 53 55 50 45 52 4D // NFIT(.SUPERM > 0010: 53 4D 43 49 2D 2D 4D 42 01 00 00 00 01 00 00 00 // SMCI--MB > 0020: 01 00 00 00 00 00 00 00 > > As you can see the memory region info is missing. > > This specific check was done on a supermicro server. > We also performed a bios update but the results were the same. > > As said before ,the pmem devices are detected correctly and we verified > that they correspond to different numa nodes using the PCM utility.However, > linux still reports both pmem devices to be on the same numa - Numa 0. > > If this information is missing, why pmem devices and address ranges are > still detected correctly? I suspect your BIOS might be using E820-type-12 to describe the pmem ranges which is not compliant with the ACPI specification and would need a BIOS change. > Is there another table that we need to check? You can dump /proc/iomem. If it shows "Persistent Memory (legacy)" then the BIOS is using the E820-type-12 description scheme which does not include NUMA information. ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: Detecting NUMA per pmem
Hi I have a question regrading NVDIMM detection. When we are working with NVDIMM of type 12 it is detected by the linux in legacy mode and we can accesses it as pmem or dax device. we have an e820 bios. When we are using a type 7 NVDIMM it is reported by the linux as persistence type 7 memory but there is no pmem or dax device created. Linux Kernel identifies this memory in the e820 table but it does not trigger nvdimm probe for it. Do you know what could be the cause? Is their a workaround for that? Can it still be treated as legacy mode so we can access it through pmem/dax device? BR Oren Berman On 22 October 2017 at 16:52, Dan Williams wrote: > On Sun, Oct 22, 2017 at 4:33 AM, Oren Berman > wrote: > > Hi Ross > > > > Thanks for the speedy reply. I am also adding the public list to this > > thread as you suggested. > > > > We have tried to dump the SPA table and this is what we get: > > > > /* > > * Intel ACPI Component Architecture > > * AML/ASL+ Disassembler version 20160108-64 > > * Copyright (c) 2000 - 2016 Intel Corporation > > * > > * Disassembly of NFIT, Sun Oct 22 10:46:19 2017 > > * > > * ACPI Data Table [NFIT] > > * > > * Format: [HexOffset DecimalOffset ByteLength] FieldName : FieldValue > > */ > > > > [000h 4]Signature : "NFIT"[NVDIMM Firmware > > Interface Table] > > [004h 0004 4] Table Length : 0028 > > [008h 0008 1] Revision : 01 > > [009h 0009 1] Checksum : B2 > > [00Ah 0010 6] Oem ID : "SUPERM" > > [010h 0016 8] Oem Table ID : "SMCI--MB" > > [018h 0024 4] Oem Revision : 0001 > > [01Ch 0028 4] Asl Compiler ID : " " > > [020h 0032 4]Asl Compiler Revision : 0001 > > > > [024h 0036 4] Reserved : > > > > Raw Table Data: Length 40 (0x28) > > > > : 4E 46 49 54 28 00 00 00 01 B2 53 55 50 45 52 4D // > NFIT(.SUPERM > > 0010: 53 4D 43 49 2D 2D 4D 42 01 00 00 00 01 00 00 00 // > SMCI--MB > > 0020: 01 00 00 00 00 00 00 00 > > > > As you can see the memory region info is missing. > > > > This specific check was done on a supermicro server. > > We also performed a bios update but the results were the same. > > > > As said before ,the pmem devices are detected correctly and we verified > > that they correspond to different numa nodes using the PCM > utility.However, > > linux still reports both pmem devices to be on the same numa - Numa 0. > > > > If this information is missing, why pmem devices and address ranges are > > still detected correctly? > > I suspect your BIOS might be using E820-type-12 to describe the pmem > ranges which is not compliant with the ACPI specification and would > need a BIOS change. > > > Is there another table that we need to check? > > You can dump /proc/iomem. If it shows "Persistent Memory (legacy)" > then the BIOS is using the E820-type-12 description scheme which does > not include NUMA information. > ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: Detecting NUMA per pmem
[sent from my phone, forgive formatting] Your BIOS would need to put SPA range entries in the ACPI NFIT. The problem with legacy pmem ranges in the e820 table is that it omits critical details like battery status and whether the platform supports flushing memory controller buffers at power loss (ADR). The NFIT can also reliably communicate NUMA information for NVDIMMs that e820 does not. On Wednesday, December 27, 2017, Oren Berman wrote: > Hi > > I have a question regrading NVDIMM detection. > > When we are working with NVDIMM of type 12 it is detected by the linux in > legacy mode and we can > accesses it as pmem or dax device. we have an e820 bios. > > When we are using a type 7 NVDIMM it is reported by the linux as > persistence type 7 memory but there is no pmem or dax device created. > Linux Kernel identifies this memory in the e820 table but it does not > trigger nvdimm probe for it. > Do you know what could be the cause? Is their a workaround for that? > Can it still be treated as legacy mode so we can access it through pmem/dax > device? > > BR > Oren Berman > > On 22 October 2017 at 16:52, Dan Williams > wrote: > > > On Sun, Oct 22, 2017 at 4:33 AM, Oren Berman > > wrote: > > > Hi Ross > > > > > > Thanks for the speedy reply. I am also adding the public list to this > > > thread as you suggested. > > > > > > We have tried to dump the SPA table and this is what we get: > > > > > > /* > > > * Intel ACPI Component Architecture > > > * AML/ASL+ Disassembler version 20160108-64 > > > * Copyright (c) 2000 - 2016 Intel Corporation > > > * > > > * Disassembly of NFIT, Sun Oct 22 10:46:19 2017 > > > * > > > * ACPI Data Table [NFIT] > > > * > > > * Format: [HexOffset DecimalOffset ByteLength] FieldName : FieldValue > > > */ > > > > > > [000h 4]Signature : "NFIT"[NVDIMM > Firmware > > > Interface Table] > > > [004h 0004 4] Table Length : 0028 > > > [008h 0008 1] Revision : 01 > > > [009h 0009 1] Checksum : B2 > > > [00Ah 0010 6] Oem ID : "SUPERM" > > > [010h 0016 8] Oem Table ID : "SMCI--MB" > > > [018h 0024 4] Oem Revision : 0001 > > > [01Ch 0028 4] Asl Compiler ID : " " > > > [020h 0032 4]Asl Compiler Revision : 0001 > > > > > > [024h 0036 4] Reserved : > > > > > > Raw Table Data: Length 40 (0x28) > > > > > > : 4E 46 49 54 28 00 00 00 01 B2 53 55 50 45 52 4D // > > NFIT(.SUPERM > > > 0010: 53 4D 43 49 2D 2D 4D 42 01 00 00 00 01 00 00 00 // > > SMCI--MB > > > 0020: 01 00 00 00 00 00 00 00 > > > > > > As you can see the memory region info is missing. > > > > > > This specific check was done on a supermicro server. > > > We also performed a bios update but the results were the same. > > > > > > As said before ,the pmem devices are detected correctly and we verified > > > that they correspond to different numa nodes using the PCM > > utility.However, > > > linux still reports both pmem devices to be on the same numa - Numa 0. > > > > > > If this information is missing, why pmem devices and address ranges are > > > still detected correctly? > > > > I suspect your BIOS might be using E820-type-12 to describe the pmem > > ranges which is not compliant with the ACPI specification and would > > need a BIOS change. > > > > > Is there another table that we need to check? > > > > You can dump /proc/iomem. If it shows "Persistent Memory (legacy)" > > then the BIOS is using the E820-type-12 description scheme which does > > not include NUMA information. > > > ___ > Linux-nvdimm mailing list > Linux-nvdimm@lists.01.org > https://lists.01.org/mailman/listinfo/linux-nvdimm > ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: Detecting NUMA per pmem
Thanks Dan. I understand the shortcomings of using legacy mode but currently my problem is that TYPE 12 is detected and I can use dax even in legacy mode but for some reason type 7 is not. Is there a way to force it be treated as legacy as well. The reason I am asking is that I am not sure I can change my bios and I know at least that type 12 NVDIMM is working for me. BR Oren On 28 December 2017 at 11:14, Dan Williams wrote: > [sent from my phone, forgive formatting] > > Your BIOS would need to put SPA range entries in the ACPI NFIT. The > problem with legacy pmem ranges in the e820 table is that it omits critical > details like battery status and whether the platform supports flushing > memory controller buffers at power loss (ADR). > > The NFIT can also reliably communicate NUMA information for NVDIMMs that > e820 does not. > > On Wednesday, December 27, 2017, Oren Berman > wrote: > >> Hi >> >> I have a question regrading NVDIMM detection. >> >> When we are working with NVDIMM of type 12 it is detected by the linux in >> legacy mode and we can >> accesses it as pmem or dax device. we have an e820 bios. >> >> When we are using a type 7 NVDIMM it is reported by the linux as >> persistence type 7 memory but there is no pmem or dax device created. >> Linux Kernel identifies this memory in the e820 table but it does not >> trigger nvdimm probe for it. >> Do you know what could be the cause? Is their a workaround for that? >> Can it still be treated as legacy mode so we can access it through >> pmem/dax >> device? >> >> BR >> Oren Berman >> >> On 22 October 2017 at 16:52, Dan Williams >> wrote: >> >> > On Sun, Oct 22, 2017 at 4:33 AM, Oren Berman >> > wrote: >> > > Hi Ross >> > > >> > > Thanks for the speedy reply. I am also adding the public list to this >> > > thread as you suggested. >> > > >> > > We have tried to dump the SPA table and this is what we get: >> > > >> > > /* >> > > * Intel ACPI Component Architecture >> > > * AML/ASL+ Disassembler version 20160108-64 >> > > * Copyright (c) 2000 - 2016 Intel Corporation >> > > * >> > > * Disassembly of NFIT, Sun Oct 22 10:46:19 2017 >> > > * >> > > * ACPI Data Table [NFIT] >> > > * >> > > * Format: [HexOffset DecimalOffset ByteLength] FieldName : >> FieldValue >> > > */ >> > > >> > > [000h 4]Signature : "NFIT"[NVDIMM >> Firmware >> > > Interface Table] >> > > [004h 0004 4] Table Length : 0028 >> > > [008h 0008 1] Revision : 01 >> > > [009h 0009 1] Checksum : B2 >> > > [00Ah 0010 6] Oem ID : "SUPERM" >> > > [010h 0016 8] Oem Table ID : "SMCI--MB" >> > > [018h 0024 4] Oem Revision : 0001 >> > > [01Ch 0028 4] Asl Compiler ID : " " >> > > [020h 0032 4]Asl Compiler Revision : 0001 >> > > >> > > [024h 0036 4] Reserved : >> > > >> > > Raw Table Data: Length 40 (0x28) >> > > >> > > : 4E 46 49 54 28 00 00 00 01 B2 53 55 50 45 52 4D // >> > NFIT(.SUPERM >> > > 0010: 53 4D 43 49 2D 2D 4D 42 01 00 00 00 01 00 00 00 // >> > SMCI--MB >> > > 0020: 01 00 00 00 00 00 00 00 >> > > >> > > As you can see the memory region info is missing. >> > > >> > > This specific check was done on a supermicro server. >> > > We also performed a bios update but the results were the same. >> > > >> > > As said before ,the pmem devices are detected correctly and we >> verified >> > > that they correspond to different numa nodes using the PCM >> > utility.However, >> > > linux still reports both pmem devices to be on the same numa - Numa >> 0. >> > > >> > > If this information is missing, why pmem devices and address ranges >> are >> > > still detected correctly? >> > >> > I suspect your BIOS might be using E820-type-12 to describe the pmem >> > ranges which is not compliant with the ACPI specification and would >> > need a BIOS change. >> > >> > > Is there another table that we need to check? >> > >> > You can dump /proc/iomem. If it shows "Persistent Memory (legacy)" >> > then the BIOS is using the E820-type-12 description scheme which does >> > not include NUMA information. >> > >> ___ >> Linux-nvdimm mailing list >> Linux-nvdimm@lists.01.org >> https://lists.01.org/mailman/listinfo/linux-nvdimm >> > ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: Detecting NUMA per pmem
Type-7 only tells the kernel to reserve the memory range. NFIT carves that reservation into pmem devices. Type-12 skips the reservation and creates a pmem device directly. There is no workaround if the platform only has a BIOS that produces a type-12 range. On Thursday, December 28, 2017, Oren Berman wrote: > Thanks Dan. > I understand the shortcomings of using legacy mode but currently my problem > is that TYPE 12 is detected and I can use dax even in legacy mode but for > some reason type 7 is not. Is there a way to force it be treated as legacy > as well. > The reason I am asking is that I am not sure I can change my bios and I > know at least that type 12 NVDIMM is working for me. > > BR > Oren > > > > On 28 December 2017 at 11:14, Dan Williams > wrote: > > > [sent from my phone, forgive formatting] > > > > Your BIOS would need to put SPA range entries in the ACPI NFIT. The > > problem with legacy pmem ranges in the e820 table is that it omits > critical > > details like battery status and whether the platform supports flushing > > memory controller buffers at power loss (ADR). > > > > The NFIT can also reliably communicate NUMA information for NVDIMMs that > > e820 does not. > > > > On Wednesday, December 27, 2017, Oren Berman > > wrote: > > > >> Hi > >> > >> I have a question regrading NVDIMM detection. > >> > >> When we are working with NVDIMM of type 12 it is detected by the linux > in > >> legacy mode and we can > >> accesses it as pmem or dax device. we have an e820 bios. > >> > >> When we are using a type 7 NVDIMM it is reported by the linux as > >> persistence type 7 memory but there is no pmem or dax device created. > >> Linux Kernel identifies this memory in the e820 table but it does not > >> trigger nvdimm probe for it. > >> Do you know what could be the cause? Is their a workaround for that? > >> Can it still be treated as legacy mode so we can access it through > >> pmem/dax > >> device? > >> > >> BR > >> Oren Berman > >> > >> On 22 October 2017 at 16:52, Dan Williams > >> wrote: > >> > >> > On Sun, Oct 22, 2017 at 4:33 AM, Oren Berman > >> > wrote: > >> > > Hi Ross > >> > > > >> > > Thanks for the speedy reply. I am also adding the public list to > this > >> > > thread as you suggested. > >> > > > >> > > We have tried to dump the SPA table and this is what we get: > >> > > > >> > > /* > >> > > * Intel ACPI Component Architecture > >> > > * AML/ASL+ Disassembler version 20160108-64 > >> > > * Copyright (c) 2000 - 2016 Intel Corporation > >> > > * > >> > > * Disassembly of NFIT, Sun Oct 22 10:46:19 2017 > >> > > * > >> > > * ACPI Data Table [NFIT] > >> > > * > >> > > * Format: [HexOffset DecimalOffset ByteLength] FieldName : > >> FieldValue > >> > > */ > >> > > > >> > > [000h 4]Signature : "NFIT"[NVDIMM > >> Firmware > >> > > Interface Table] > >> > > [004h 0004 4] Table Length : 0028 > >> > > [008h 0008 1] Revision : 01 > >> > > [009h 0009 1] Checksum : B2 > >> > > [00Ah 0010 6] Oem ID : "SUPERM" > >> > > [010h 0016 8] Oem Table ID : "SMCI--MB" > >> > > [018h 0024 4] Oem Revision : 0001 > >> > > [01Ch 0028 4] Asl Compiler ID : " " > >> > > [020h 0032 4]Asl Compiler Revision : 0001 > >> > > > >> > > [024h 0036 4] Reserved : > >> > > > >> > > Raw Table Data: Length 40 (0x28) > >> > > > >> > > : 4E 46 49 54 28 00 00 00 01 B2 53 55 50 45 52 4D // > >> > NFIT(.SUPERM > >> > > 0010: 53 4D 43 49 2D 2D 4D 42 01 00 00 00 01 00 00 00 // > >> > SMCI--MB > >> > > 0020: 01 00 00 00 00 00 00 00 > >> > > > >> > > As you can see the memory region info is missing. > >> > > > >> > > This specific check was done on a supermicro server. > >> > > We also performed a bios update but the results were the same. > >> > > > >> > > As said before ,the pmem devices are detected correctly and we > >> verified > >> > > that they correspond to different numa nodes using the PCM > >> > utility.However, > >> > > linux still reports both pmem devices to be on the same numa - Numa > >> 0. > >> > > > >> > > If this information is missing, why pmem devices and address ranges > >> are > >> > > still detected correctly? > >> > > >> > I suspect your BIOS might be using E820-type-12 to describe the pmem > >> > ranges which is not compliant with the ACPI specification and would > >> > need a BIOS change. > >> > > >> > > Is there another table that we need to check? > >> > > >> > You can dump /proc/iomem. If it shows "Persistent Memory (legacy)" > >> > then the BIOS is using the E820-type-12 description scheme which does > >> > not include NUMA information. > >> > > >> ___ > >> Linux-nvdimm mailing list > >> Linux-nvdimm@lists.01.org > >> https://lists.01.org/mailman/listinfo/linux-nvdimm > >> > > > __
Re: Detecting NUMA per pmem
You can try to force a legacy pmem device with memmap=XX!YY kernel parameter. On Thu, Dec 28, 2017 at 8:16 PM, Dan Williams wrote: > Type-7 only tells the kernel to reserve the memory range. NFIT carves that > reservation into pmem devices. Type-12 skips the reservation and creates a > pmem device directly. There is no workaround if the platform only has a > BIOS that produces a type-12 range. > > > On Thursday, December 28, 2017, Oren Berman > wrote: > > > Thanks Dan. > > I understand the shortcomings of using legacy mode but currently my > problem > > is that TYPE 12 is detected and I can use dax even in legacy mode but for > > some reason type 7 is not. Is there a way to force it be treated as > legacy > > as well. > > The reason I am asking is that I am not sure I can change my bios and I > > know at least that type 12 NVDIMM is working for me. > > > > BR > > Oren > > > > > > > > On 28 December 2017 at 11:14, Dan Williams > > wrote: > > > > > [sent from my phone, forgive formatting] > > > > > > Your BIOS would need to put SPA range entries in the ACPI NFIT. The > > > problem with legacy pmem ranges in the e820 table is that it omits > > critical > > > details like battery status and whether the platform supports flushing > > > memory controller buffers at power loss (ADR). > > > > > > The NFIT can also reliably communicate NUMA information for NVDIMMs > that > > > e820 does not. > > > > > > On Wednesday, December 27, 2017, Oren Berman > > > wrote: > > > > > >> Hi > > >> > > >> I have a question regrading NVDIMM detection. > > >> > > >> When we are working with NVDIMM of type 12 it is detected by the linux > > in > > >> legacy mode and we can > > >> accesses it as pmem or dax device. we have an e820 bios. > > >> > > >> When we are using a type 7 NVDIMM it is reported by the linux as > > >> persistence type 7 memory but there is no pmem or dax device created. > > >> Linux Kernel identifies this memory in the e820 table but it does not > > >> trigger nvdimm probe for it. > > >> Do you know what could be the cause? Is their a workaround for that? > > >> Can it still be treated as legacy mode so we can access it through > > >> pmem/dax > > >> device? > > >> > > >> BR > > >> Oren Berman > > >> > > >> On 22 October 2017 at 16:52, Dan Williams > > >> wrote: > > >> > > >> > On Sun, Oct 22, 2017 at 4:33 AM, Oren Berman < > o...@lightbitslabs.com> > > >> > wrote: > > >> > > Hi Ross > > >> > > > > >> > > Thanks for the speedy reply. I am also adding the public list to > > this > > >> > > thread as you suggested. > > >> > > > > >> > > We have tried to dump the SPA table and this is what we get: > > >> > > > > >> > > /* > > >> > > * Intel ACPI Component Architecture > > >> > > * AML/ASL+ Disassembler version 20160108-64 > > >> > > * Copyright (c) 2000 - 2016 Intel Corporation > > >> > > * > > >> > > * Disassembly of NFIT, Sun Oct 22 10:46:19 2017 > > >> > > * > > >> > > * ACPI Data Table [NFIT] > > >> > > * > > >> > > * Format: [HexOffset DecimalOffset ByteLength] FieldName : > > >> FieldValue > > >> > > */ > > >> > > > > >> > > [000h 4]Signature : "NFIT"[NVDIMM > > >> Firmware > > >> > > Interface Table] > > >> > > [004h 0004 4] Table Length : 0028 > > >> > > [008h 0008 1] Revision : 01 > > >> > > [009h 0009 1] Checksum : B2 > > >> > > [00Ah 0010 6] Oem ID : "SUPERM" > > >> > > [010h 0016 8] Oem Table ID : "SMCI--MB" > > >> > > [018h 0024 4] Oem Revision : 0001 > > >> > > [01Ch 0028 4] Asl Compiler ID : " " > > >> > > [020h 0032 4]Asl Compiler Revision : 0001 > > >> > > > > >> > > [024h 0036 4] Reserved : > > >> > > > > >> > > Raw Table Data: Length 40 (0x28) > > >> > > > > >> > > : 4E 46 49 54 28 00 00 00 01 B2 53 55 50 45 52 4D // > > >> > NFIT(.SUPERM > > >> > > 0010: 53 4D 43 49 2D 2D 4D 42 01 00 00 00 01 00 00 00 // > > >> > SMCI--MB > > >> > > 0020: 01 00 00 00 00 00 00 00 > > >> > > > > >> > > As you can see the memory region info is missing. > > >> > > > > >> > > This specific check was done on a supermicro server. > > >> > > We also performed a bios update but the results were the same. > > >> > > > > >> > > As said before ,the pmem devices are detected correctly and we > > >> verified > > >> > > that they correspond to different numa nodes using the PCM > > >> > utility.However, > > >> > > linux still reports both pmem devices to be on the same numa - > Numa > > >> 0. > > >> > > > > >> > > If this information is missing, why pmem devices and address > ranges > > >> are > > >> > > still detected correctly? > > >> > > > >> > I suspect your BIOS might be using E820-type-12 to describe the pmem > > >> > ranges which is not compliant with the ACPI specification and would > > >> > need a BIOS change. > > >> > > > >> > > Is there another table that we
Re: Detecting NUMA per pmem
Hi I would like to know if you encountered such a problem. We are accessing the nvram as memory from withing the kernel. By mapping dax device and reading its mapping we can know the physical address of the nvram. As a result we can access this address range in the kernel by calling phys_to_virt. This is working in most case but we saw some issue that after reboot, when trying to read the info saved on the nvram before the power off, one kernel thread was able to read from this range but another kernel thread got page fault. This is not recreated very easily and we need run many reboot sequences to get this failure again. Are you aware of any mapping issues of nvram to kernel space? Thanks for any suggestions you might have. BR Oren On 31 December 2017 at 10:23, Yigal Korman wrote: > You can try to force a legacy pmem device with memmap=XX!YY kernel > parameter. > > On Thu, Dec 28, 2017 at 8:16 PM, Dan Williams > wrote: > >> Type-7 only tells the kernel to reserve the memory range. NFIT carves that >> reservation into pmem devices. Type-12 skips the reservation and creates a >> pmem device directly. There is no workaround if the platform only has a >> BIOS that produces a type-12 range. >> >> >> On Thursday, December 28, 2017, Oren Berman >> wrote: >> >> > Thanks Dan. >> > I understand the shortcomings of using legacy mode but currently my >> problem >> > is that TYPE 12 is detected and I can use dax even in legacy mode but >> for >> > some reason type 7 is not. Is there a way to force it be treated as >> legacy >> > as well. >> > The reason I am asking is that I am not sure I can change my bios and I >> > know at least that type 12 NVDIMM is working for me. >> > >> > BR >> > Oren >> > >> > >> > >> > On 28 December 2017 at 11:14, Dan Williams >> > wrote: >> > >> > > [sent from my phone, forgive formatting] >> > > >> > > Your BIOS would need to put SPA range entries in the ACPI NFIT. The >> > > problem with legacy pmem ranges in the e820 table is that it omits >> > critical >> > > details like battery status and whether the platform supports flushing >> > > memory controller buffers at power loss (ADR). >> > > >> > > The NFIT can also reliably communicate NUMA information for NVDIMMs >> that >> > > e820 does not. >> > > >> > > On Wednesday, December 27, 2017, Oren Berman >> > > wrote: >> > > >> > >> Hi >> > >> >> > >> I have a question regrading NVDIMM detection. >> > >> >> > >> When we are working with NVDIMM of type 12 it is detected by the >> linux >> > in >> > >> legacy mode and we can >> > >> accesses it as pmem or dax device. we have an e820 bios. >> > >> >> > >> When we are using a type 7 NVDIMM it is reported by the linux as >> > >> persistence type 7 memory but there is no pmem or dax device created. >> > >> Linux Kernel identifies this memory in the e820 table but it does not >> > >> trigger nvdimm probe for it. >> > >> Do you know what could be the cause? Is their a workaround for that? >> > >> Can it still be treated as legacy mode so we can access it through >> > >> pmem/dax >> > >> device? >> > >> >> > >> BR >> > >> Oren Berman >> > >> >> > >> On 22 October 2017 at 16:52, Dan Williams >> > >> wrote: >> > >> >> > >> > On Sun, Oct 22, 2017 at 4:33 AM, Oren Berman < >> o...@lightbitslabs.com> >> > >> > wrote: >> > >> > > Hi Ross >> > >> > > >> > >> > > Thanks for the speedy reply. I am also adding the public list to >> > this >> > >> > > thread as you suggested. >> > >> > > >> > >> > > We have tried to dump the SPA table and this is what we get: >> > >> > > >> > >> > > /* >> > >> > > * Intel ACPI Component Architecture >> > >> > > * AML/ASL+ Disassembler version 20160108-64 >> > >> > > * Copyright (c) 2000 - 2016 Intel Corporation >> > >> > > * >> > >> > > * Disassembly of NFIT, Sun Oct 22 10:46:19 2017 >> > >> > > * >> > >> > > * ACPI Data Table [NFIT] >> > >> > > * >> > >> > > * Format: [HexOffset DecimalOffset ByteLength] FieldName : >> > >> FieldValue >> > >> > > */ >> > >> > > >> > >> > > [000h 4]Signature : "NFIT"[NVDIMM >> > >> Firmware >> > >> > > Interface Table] >> > >> > > [004h 0004 4] Table Length : 0028 >> > >> > > [008h 0008 1] Revision : 01 >> > >> > > [009h 0009 1] Checksum : B2 >> > >> > > [00Ah 0010 6] Oem ID : "SUPERM" >> > >> > > [010h 0016 8] Oem Table ID : "SMCI--MB" >> > >> > > [018h 0024 4] Oem Revision : 0001 >> > >> > > [01Ch 0028 4] Asl Compiler ID : " " >> > >> > > [020h 0032 4]Asl Compiler Revision : 0001 >> > >> > > >> > >> > > [024h 0036 4] Reserved : >> > >> > > >> > >> > > Raw Table Data: Length 40 (0x28) >> > >> > > >> > >> > > : 4E 46 49 54 28 00 00 00 01 B2 53 55 50 45 52 4D // >> > >> > NFIT(.SUPERM >> > >> > > 0010: 53 4D 43 49 2D 2D 4D 42 01 00 00 00 01 00 00 00 // >> > >> > SMCI--MB >> > >> >
Re: Detecting NUMA per pmem
On Tue, Jan 9, 2018 at 2:25 PM, Oren Berman wrote: > Hi > > I would like to know if you encountered such a problem. > > We are accessing the nvram as memory from withing the kernel. > By mapping dax device and reading its mapping we can know the physical > address of the nvram. > As a result we can access this address range in the kernel by calling > phys_to_virt. > This is working in most case but we saw some issue that after reboot, when > trying to read the info saved > on the nvram before the power off, one kernel thread was able to read > from this range but another kernel thread got page fault. > > This is not recreated very easily and we need run many reboot sequences to > get this failure again. > Are you aware of any mapping issues of nvram to kernel space? When are you using phys_to_virt()? That will only return a valid virtual address as long as the driver is loaded. It sounds like you may be losing a race with the driver setting up or tearing down the mappings. ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: Detecting NUMA per pmem
Hi Dan Which driver are you referring to? If it is the dax driver than it is always loaded - we see /dev/dax0. If you refer to the user space application which called the mmap on the dax device then this application is not running anymore. We used this application to get the virtual address mapping(doing mmap on dax) and then by going over the proc pagemap we got the physical address. After that the application terminates and we pass this physical address to our kernel thread . Then from the kernel thread we access this range by using phys_to_virt (we know the physical so we convert it virtual). As far as I know once in kernel space all address range should be mapped to the kernel page tables in 64 bit architecture ofcourse, thus accessible using phys_to_virt. Is this a wrong assumption when dealing with NVRAM? If I know the physical address of the nvram isn't it accessible from the kernel using the simple conversion of phys_to_virt? Thanks Oren On 10 January 2018 at 01:05, Dan Williams wrote: > On Tue, Jan 9, 2018 at 2:25 PM, Oren Berman > wrote: > > Hi > > > > I would like to know if you encountered such a problem. > > > > We are accessing the nvram as memory from withing the kernel. > > By mapping dax device and reading its mapping we can know the physical > > address of the nvram. > > As a result we can access this address range in the kernel by calling > > phys_to_virt. > > This is working in most case but we saw some issue that after reboot, > when > > trying to read the info saved > > on the nvram before the power off, one kernel thread was able to read > > from this range but another kernel thread got page fault. > > > > This is not recreated very easily and we need run many reboot sequences > to > > get this failure again. > > Are you aware of any mapping issues of nvram to kernel space? > > When are you using phys_to_virt()? That will only return a valid > virtual address as long as the driver is loaded. It sounds like you > may be losing a race with the driver setting up or tearing down the > mappings. > ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: Detecting NUMA per pmem
Hi A few more updates: a) This issue does not happen a lot I need to do a lot of power cycles for this to happen. b) We use 2 numa nodes but for some reason it happens only on the second numa. c) We added a debug feature that when page fault occurs we will trigger a thread from command line in the kernel(through configfs) that will read from any physical address that we give it. When we give it the faulty address - the thread succeeds reading from it with no page fault. We verified that it accessed the same virtual address that caused the page fault. BR Oren On 10 January 2018 at 09:21, Oren Berman wrote: > Hi Dan > > Which driver are you referring to? > If it is the dax driver than it is always loaded - we see /dev/dax0. > If you refer to the user space application which called the mmap on the > dax device then this application is not running anymore. > We used this application to get the virtual address mapping(doing mmap on > dax) and then by going over the proc pagemap we got the physical address. > After that the application terminates and we pass this physical address to > our kernel thread . > Then from the kernel thread we access this range by using phys_to_virt (we > know the physical so we convert it virtual). > As far as I know once in kernel space all address range should be mapped > to the kernel page tables in 64 bit architecture ofcourse, > thus accessible using phys_to_virt. > Is this a wrong assumption when dealing with NVRAM? > If I know the physical address of the nvram isn't it accessible from the > kernel using the simple conversion of phys_to_virt? > > Thanks > Oren > > > > > On 10 January 2018 at 01:05, Dan Williams > wrote: > >> On Tue, Jan 9, 2018 at 2:25 PM, Oren Berman >> wrote: >> > Hi >> > >> > I would like to know if you encountered such a problem. >> > >> > We are accessing the nvram as memory from withing the kernel. >> > By mapping dax device and reading its mapping we can know the physical >> > address of the nvram. >> > As a result we can access this address range in the kernel by calling >> > phys_to_virt. >> > This is working in most case but we saw some issue that after reboot, >> when >> > trying to read the info saved >> > on the nvram before the power off, one kernel thread was able to read >> > from this range but another kernel thread got page fault. >> > >> > This is not recreated very easily and we need run many reboot sequences >> to >> > get this failure again. >> > Are you aware of any mapping issues of nvram to kernel space? >> >> When are you using phys_to_virt()? That will only return a valid >> virtual address as long as the driver is loaded. It sounds like you >> may be losing a race with the driver setting up or tearing down the >> mappings. >> > > ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: Detecting NUMA per pmem
On Wed, Jan 10, 2018 at 5:13 AM, Oren Berman wrote: > Hi > > A few more updates: > > a) This issue does not happen a lot I need to do a lot of power cycles for > this to happen. > b) We use 2 numa nodes but for some reason it happens only on the second > numa. > c) We added a debug feature that when page fault occurs we will trigger a > thread from command line in the kernel(through configfs) that will read > from > any physical address that we give it. When we give it the faulty address - > the thread succeeds reading from it with no page fault. > We verified that it accessed the same virtual address that caused the page > fault. Does the problem go away if you specify: nokaslr ...on the kernel command line? ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: Detecting NUMA per pmem
Now to all of the forum Hi Dan Thanks we are going to try this. Can you explain why this can cause this issue - is the NVDIMM memory space also being randomized? Is it done during runtime? BR Oren On 10 January 2018 at 16:51, Dan Williams wrote: > On Wed, Jan 10, 2018 at 5:13 AM, Oren Berman > wrote: > > Hi > > > > A few more updates: > > > > a) This issue does not happen a lot I need to do a lot of power cycles > for > > this to happen. > > b) We use 2 numa nodes but for some reason it happens only on the second > > numa. > > c) We added a debug feature that when page fault occurs we will trigger a > > thread from command line in the kernel(through configfs) that will read > > from > > any physical address that we give it. When we give it the faulty address > - > > the thread succeeds reading from it with no page fault. > > We verified that it accessed the same virtual address that caused the > page > > fault. > > Does the problem go away if you specify: > > nokaslr > > ...on the kernel command line? > ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: Detecting NUMA per pmem
On Wed, Jan 10, 2018 at 7:23 AM, Oren Berman wrote: > Now to all of the forum > > Hi Dan > > Thanks we are going to try this. > > Can you explain why this can cause this issue - is the NVDIMM memory space > also being randomized? > Is it done during runtime? > Yes, kaslr randomizes the direct map. We have seen problems with it in the past relative to setting up pmem mappings. We fixed one such bug with this commit: fc5f9d5f151c x86/mm: Fix boot crash caused by incorrect loop count calculation in sync_global_pgds() ...but it appears we may have another bug in this area. ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: Detecting NUMA per pmem
Hi Thanks for your answer If we do memremap on the physical address Of the nvram from within the kernel to get a new virtual address mapping will it lock the mapping? Can this be also a workaround? Oren נשלח מה-iPhone שלי ב-10 בינו׳ 2018, בשעה 18:38, Dan Williams כתב/ה: >> On Wed, Jan 10, 2018 at 7:23 AM, Oren Berman wrote: >> Now to all of the forum >> >> Hi Dan >> >> Thanks we are going to try this. >> >> Can you explain why this can cause this issue - is the NVDIMM memory space >> also being randomized? >> Is it done during runtime? >> > > Yes, kaslr randomizes the direct map. We have seen problems with it in > the past relative to setting up pmem mappings. We fixed one such bug > with this commit: > > fc5f9d5f151c x86/mm: Fix boot crash caused by incorrect loop count > calculation in sync_global_pgds() > > ...but it appears we may have another bug in this area. ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm