Re: [edk2] SPI Flash Corruption
Thanks guys for your answers. I will open a bug in bugzilla and attach the binary. *Is there any data remains within 0x4 - 0x44000 region? * Yes there is Data between 0x4 and 0x44000. it's after 44 that you can only see FFs in the affected units. The bios we are using is based on the MinnowBoard Max 0.93 ( it's pretty old). It would be very helpful for me to know when the FTW was added, this way I can test with the Bios after that change. Samah On Fri, Sep 21, 2018 at 12:53 PM Andrew Fish wrote: > From a design point of view VPD == Vital Product Data. The idea behind VPD > was to be a place to store platform unique information generally programmed > in the factory. So things like serial number, system UUID, mac address, > etc. Usually VPD is programmed in the factory and never updated, thus it is > a good idea to put the VPD data in its own FLASH block, and always keep > that block locked. It is not uncommon for a FLASH update utility to not > update that block when the FD is updated. > > Thanks, > > Andrew Fish > > > On Sep 21, 2018, at 2:26 AM, Wei, David wrote: > > > > More comments: > > > > The NV Variable region starts from 0x4. Is there any data remains > within 0x4 - 0x44000 region? Could you dump the flash image and share > it with us, and also file a bug in https://bugzilla.tianocore.org as > Jiewen mentioned? > > > > It occurred to me that on some old version of Minnowboard Max BIOS, the > NV variable reclaiming process would take a long time ,so that inpatient > user may think the system is stuck and cut the power. This will break the > NV variable region. And in old version of Minnowboard Max BIOS, FTW driver > is not added for PEI stage, so system may not recover if PEI stage depends > on NV variable. > > > > Newer version of Minnowboard Max BIOS re-configures SPI flash clock to > make the NV Variable reclaiming process more faster, and also adds FTW for > PEI stage. I will check which version of Minnowboard Max BIOS has added > this fix. > > > > Thanks, > > David Wei > > > > Intel SSG/STO/UEFI BIOS > > > > > > -Original Message- > > From: edk2-devel [mailto:edk2-devel-boun...@lists.01.org] On Behalf Of > Yao, Jiewen > > Sent: Friday, September 21, 2018 7:44 AM > > To: Samah Mansour > > Cc: edk2-devel@lists.01.org; ler...@redhat.com > > Subject: Re: [edk2] SPI Flash Corruption > > Importance: High > > > > thank you, Samah. > > Would you please file a tracker in edkii bugzilla ? > > > > The term VPD might lead confusion here. > > Ideally VPD region is independent with UEFI variable region. It is a > special region to hold PCD with VPD type. > > I just look at the code. The open source minnowmax puts variable region > in the VPD region. As such there is discussion about variable atomicity. > But the variable atomicity cannot guarantee the integrity of FV header. > Additional check need to be done in platform FVB driver. > > > > If you can add a detailed reproducing step in the bugzilla, it will be > helpful for us to understand the problem. > > > > thank you! > > Yao, Jiewen > > > > > >> 在 2018年9月20日，下午11:47，Samah Mansour 写道： > >> > >> Hi Laszlo, > >> Thanks for your reply. > >> Actually what I see is that VPD (Vital Product Area between addresses > >> 44000->47DFF0 ) is completely wiped which causes the failure to boot! > >> Without the VPD unit cannot boot. > >> I will take a look at the white paper. > >> It would be helpful to know what's the impact of disabling the ability > of > >> the firmware to write those non volatile variables to flash. > >> > >> Samah > >> > >> > >>> On Thu, Sep 20, 2018 at 9:48 AM Laszlo Ersek > wrote: > >>> > >>>> On 09/19/18 16:26, Samah Mansour wrote: > >>>> Hello, > >>>> > >>>> > >>>> Our product uses a Baytrail with Minnowboard Max bios firmware ( > version > >>>> 0.93). Every now and then we see SPI flash corruption due to power > cuts > >>>> while the unit is booting which causes the unit not to boot anymore. > >>> After > >>>> investigation we noticed that the VPD area is all FFs (address > >>>> 44000->47DFF0). > >>>> > >>>> > >>>> We have noticed that the Bios while booting writes to the flash from > >>>> several places in the code, which is if interrupted most probably is > >>>> causing the corruption. > >>>&g
Re: [edk2] SPI Flash Corruption
From a design point of view VPD == Vital Product Data. The idea behind VPD was to be a place to store platform unique information generally programmed in the factory. So things like serial number, system UUID, mac address, etc. Usually VPD is programmed in the factory and never updated, thus it is a good idea to put the VPD data in its own FLASH block, and always keep that block locked. It is not uncommon for a FLASH update utility to not update that block when the FD is updated. Thanks, Andrew Fish > On Sep 21, 2018, at 2:26 AM, Wei, David wrote: > > More comments: > > The NV Variable region starts from 0x4. Is there any data remains within > 0x4 - 0x44000 region? Could you dump the flash image and share it with > us, and also file a bug in https://bugzilla.tianocore.org as Jiewen > mentioned? > > It occurred to me that on some old version of Minnowboard Max BIOS, the NV > variable reclaiming process would take a long time ,so that inpatient user > may think the system is stuck and cut the power. This will break the NV > variable region. And in old version of Minnowboard Max BIOS, FTW driver is > not added for PEI stage, so system may not recover if PEI stage depends on NV > variable. > > Newer version of Minnowboard Max BIOS re-configures SPI flash clock to make > the NV Variable reclaiming process more faster, and also adds FTW for PEI > stage. I will check which version of Minnowboard Max BIOS has added this fix. > > Thanks, > David Wei > > Intel SSG/STO/UEFI BIOS > > > -Original Message- > From: edk2-devel [mailto:edk2-devel-boun...@lists.01.org] On Behalf Of Yao, > Jiewen > Sent: Friday, September 21, 2018 7:44 AM > To: Samah Mansour > Cc: edk2-devel@lists.01.org; ler...@redhat.com > Subject: Re: [edk2] SPI Flash Corruption > Importance: High > > thank you, Samah. > Would you please file a tracker in edkii bugzilla ? > > The term VPD might lead confusion here. > Ideally VPD region is independent with UEFI variable region. It is a special > region to hold PCD with VPD type. > I just look at the code. The open source minnowmax puts variable region in > the VPD region. As such there is discussion about variable atomicity. But the > variable atomicity cannot guarantee the integrity of FV header. Additional > check need to be done in platform FVB driver. > > If you can add a detailed reproducing step in the bugzilla, it will be > helpful for us to understand the problem. > > thank you! > Yao, Jiewen > > >> 在 2018年9月20日，下午11:47，Samah Mansour 写道： >> >> Hi Laszlo, >> Thanks for your reply. >> Actually what I see is that VPD (Vital Product Area between addresses >> 44000->47DFF0 ) is completely wiped which causes the failure to boot! >> Without the VPD unit cannot boot. >> I will take a look at the white paper. >> It would be helpful to know what's the impact of disabling the ability of >> the firmware to write those non volatile variables to flash. >> >> Samah >> >> >>> On Thu, Sep 20, 2018 at 9:48 AM Laszlo Ersek wrote: >>> >>>> On 09/19/18 16:26, Samah Mansour wrote: >>>> Hello, >>>> >>>> >>>> Our product uses a Baytrail with Minnowboard Max bios firmware ( version >>>> 0.93). Every now and then we see SPI flash corruption due to power cuts >>>> while the unit is booting which causes the unit not to boot anymore. >>> After >>>> investigation we noticed that the VPD area is all FFs (address >>>> 44000->47DFF0). >>>> >>>> >>>> We have noticed that the Bios while booting writes to the flash from >>>> several places in the code, which is if interrupted most probably is >>>> causing the corruption. >>>> >>>> >>>> Why is the bios writing all these configurations to flash while booting, >>> is >>>> it to optimize boot time? is it ok if we disable the bios writing to >>> flash >>>> completely to protect ourselves from corruption? >>> >>> The firmware is at liberty to write various non-volatile UEFI variables >>> during boot. Some of those variables are standardized, some others may >>> be specific to UEFI drivers (with correspondingly private namespace >>> GUIDs for the variables). >>> >>> Power loss during flash write (and resultant flash corruption) is >>> expected. My understanding is that the Fault Tolerant Write protocol / >>> driver, sitting between the FVB (firmware volume block, i.e
Re: [edk2] SPI Flash Corruption
More comments: The NV Variable region starts from 0x4. Is there any data remains within 0x4 - 0x44000 region? Could you dump the flash image and share it with us, and also file a bug in https://bugzilla.tianocore.org as Jiewen mentioned? It occurred to me that on some old version of Minnowboard Max BIOS, the NV variable reclaiming process would take a long time ,so that inpatient user may think the system is stuck and cut the power. This will break the NV variable region. And in old version of Minnowboard Max BIOS, FTW driver is not added for PEI stage, so system may not recover if PEI stage depends on NV variable. Newer version of Minnowboard Max BIOS re-configures SPI flash clock to make the NV Variable reclaiming process more faster, and also adds FTW for PEI stage. I will check which version of Minnowboard Max BIOS has added this fix. Thanks, David Wei Intel SSG/STO/UEFI BIOS -Original Message- From: edk2-devel [mailto:edk2-devel-boun...@lists.01.org] On Behalf Of Yao, Jiewen Sent: Friday, September 21, 2018 7:44 AM To: Samah Mansour Cc: edk2-devel@lists.01.org; ler...@redhat.com Subject: Re: [edk2] SPI Flash Corruption Importance: High thank you, Samah. Would you please file a tracker in edkii bugzilla ? The term VPD might lead confusion here. Ideally VPD region is independent with UEFI variable region. It is a special region to hold PCD with VPD type. I just look at the code. The open source minnowmax puts variable region in the VPD region. As such there is discussion about variable atomicity. But the variable atomicity cannot guarantee the integrity of FV header. Additional check need to be done in platform FVB driver. If you can add a detailed reproducing step in the bugzilla, it will be helpful for us to understand the problem. thank you! Yao, Jiewen > 在 2018年9月20日，下午11:47，Samah Mansour 写道： > > Hi Laszlo, > Thanks for your reply. > Actually what I see is that VPD (Vital Product Area between addresses > 44000->47DFF0 ) is completely wiped which causes the failure to boot! > Without the VPD unit cannot boot. > I will take a look at the white paper. > It would be helpful to know what's the impact of disabling the ability of > the firmware to write those non volatile variables to flash. > > Samah > > >> On Thu, Sep 20, 2018 at 9:48 AM Laszlo Ersek wrote: >> >>> On 09/19/18 16:26, Samah Mansour wrote: >>> Hello, >>> >>> >>> Our product uses a Baytrail with Minnowboard Max bios firmware ( version >>> 0.93). Every now and then we see SPI flash corruption due to power cuts >>> while the unit is booting which causes the unit not to boot anymore. >> After >>> investigation we noticed that the VPD area is all FFs (address >>> 44000->47DFF0). >>> >>> >>> We have noticed that the Bios while booting writes to the flash from >>> several places in the code, which is if interrupted most probably is >>> causing the corruption. >>> >>> >>> Why is the bios writing all these configurations to flash while booting, >> is >>> it to optimize boot time? is it ok if we disable the bios writing to >> flash >>> completely to protect ourselves from corruption? >> >> The firmware is at liberty to write various non-volatile UEFI variables >> during boot. Some of those variables are standardized, some others may >> be specific to UEFI drivers (with correspondingly private namespace >> GUIDs for the variables). >> >> Power loss during flash write (and resultant flash corruption) is >> expected. My understanding is that the Fault Tolerant Write protocol / >> driver, sitting between the FVB (firmware volume block, i.e. flash) >> protocol / driver, and the variable write protocol / driver, implements >> a kind of journaling. It is described in the Intel whitepaper >> >> A Tour Beyond BIOS >> Implementing UEFI Authenticated Variables in SMM with EDKII >> September 2015 >> >> My expectation has been that the platform should recover from >> interrupted writes. That is, for a single given UEFI variable, you >> should either see "before" or "after" status, never "middle". (The >> whitepaper says that "Individual variable atomicity" is maintained even >> through a failed "reclaim", with the help of FTW.) >> >> If multiple variables should be in sync with each other, that's a >> different question. If the variables are not in sync, I think "failure >> to boot" may be a reasonable outcome. But, "failure to boot" means a lot >> of things, and I hope one should be
Re: [edk2] SPI Flash Corruption
thank you, Samah. Would you please file a tracker in edkii bugzilla ? The term VPD might lead confusion here. Ideally VPD region is independent with UEFI variable region. It is a special region to hold PCD with VPD type. I just look at the code. The open source minnowmax puts variable region in the VPD region. As such there is discussion about variable atomicity. But the variable atomicity cannot guarantee the integrity of FV header. Additional check need to be done in platform FVB driver. If you can add a detailed reproducing step in the bugzilla, it will be helpful for us to understand the problem. thank you! Yao, Jiewen > 在 2018年9月20日，下午11:47，Samah Mansour 写道： > > Hi Laszlo, > Thanks for your reply. > Actually what I see is that VPD (Vital Product Area between addresses > 44000->47DFF0 ) is completely wiped which causes the failure to boot! > Without the VPD unit cannot boot. > I will take a look at the white paper. > It would be helpful to know what's the impact of disabling the ability of > the firmware to write those non volatile variables to flash. > > Samah > > >> On Thu, Sep 20, 2018 at 9:48 AM Laszlo Ersek wrote: >> >>> On 09/19/18 16:26, Samah Mansour wrote: >>> Hello, >>> >>> >>> Our product uses a Baytrail with Minnowboard Max bios firmware ( version >>> 0.93). Every now and then we see SPI flash corruption due to power cuts >>> while the unit is booting which causes the unit not to boot anymore. >> After >>> investigation we noticed that the VPD area is all FFs (address >>> 44000->47DFF0). >>> >>> >>> We have noticed that the Bios while booting writes to the flash from >>> several places in the code, which is if interrupted most probably is >>> causing the corruption. >>> >>> >>> Why is the bios writing all these configurations to flash while booting, >> is >>> it to optimize boot time? is it ok if we disable the bios writing to >> flash >>> completely to protect ourselves from corruption? >> >> The firmware is at liberty to write various non-volatile UEFI variables >> during boot. Some of those variables are standardized, some others may >> be specific to UEFI drivers (with correspondingly private namespace >> GUIDs for the variables). >> >> Power loss during flash write (and resultant flash corruption) is >> expected. My understanding is that the Fault Tolerant Write protocol / >> driver, sitting between the FVB (firmware volume block, i.e. flash) >> protocol / driver, and the variable write protocol / driver, implements >> a kind of journaling. It is described in the Intel whitepaper >> >> A Tour Beyond BIOS >> Implementing UEFI Authenticated Variables in SMM with EDKII >> September 2015 >> >> My expectation has been that the platform should recover from >> interrupted writes. That is, for a single given UEFI variable, you >> should either see "before" or "after" status, never "middle". (The >> whitepaper says that "Individual variable atomicity" is maintained even >> through a failed "reclaim", with the help of FTW.) >> >> If multiple variables should be in sync with each other, that's a >> different question. If the variables are not in sync, I think "failure >> to boot" may be a reasonable outcome. But, "failure to boot" means a lot >> of things, and I hope one should be at least dropped to the setup >> utility or the shell. Are you seeing an actual crash? >> >> Laszlo >> > ___ > edk2-devel mailing list > edk2-devel@lists.01.org > https://lists.01.org/mailman/listinfo/edk2-devel ___ edk2-devel mailing list edk2-devel@lists.01.org https://lists.01.org/mailman/listinfo/edk2-devel
Re: [edk2] SPI Flash Corruption
Hi Laszlo, Thanks for your reply. Actually what I see is that VPD (Vital Product Area between addresses 44000->47DFF0 ) is completely wiped which causes the failure to boot! Without the VPD unit cannot boot. I will take a look at the white paper. It would be helpful to know what's the impact of disabling the ability of the firmware to write those non volatile variables to flash. Samah On Thu, Sep 20, 2018 at 9:48 AM Laszlo Ersek wrote: > On 09/19/18 16:26, Samah Mansour wrote: > > Hello, > > > > > > Our product uses a Baytrail with Minnowboard Max bios firmware ( version > > 0.93). Every now and then we see SPI flash corruption due to power cuts > > while the unit is booting which causes the unit not to boot anymore. > After > > investigation we noticed that the VPD area is all FFs (address > > 44000->47DFF0). > > > > > > We have noticed that the Bios while booting writes to the flash from > > several places in the code, which is if interrupted most probably is > > causing the corruption. > > > > > > Why is the bios writing all these configurations to flash while booting, > is > > it to optimize boot time? is it ok if we disable the bios writing to > flash > > completely to protect ourselves from corruption? > > The firmware is at liberty to write various non-volatile UEFI variables > during boot. Some of those variables are standardized, some others may > be specific to UEFI drivers (with correspondingly private namespace > GUIDs for the variables). > > Power loss during flash write (and resultant flash corruption) is > expected. My understanding is that the Fault Tolerant Write protocol / > driver, sitting between the FVB (firmware volume block, i.e. flash) > protocol / driver, and the variable write protocol / driver, implements > a kind of journaling. It is described in the Intel whitepaper > > A Tour Beyond BIOS > Implementing UEFI Authenticated Variables in SMM with EDKII > September 2015 > > My expectation has been that the platform should recover from > interrupted writes. That is, for a single given UEFI variable, you > should either see "before" or "after" status, never "middle". (The > whitepaper says that "Individual variable atomicity" is maintained even > through a failed "reclaim", with the help of FTW.) > > If multiple variables should be in sync with each other, that's a > different question. If the variables are not in sync, I think "failure > to boot" may be a reasonable outcome. But, "failure to boot" means a lot > of things, and I hope one should be at least dropped to the setup > utility or the shell. Are you seeing an actual crash? > > Laszlo > ___ edk2-devel mailing list edk2-devel@lists.01.org https://lists.01.org/mailman/listinfo/edk2-devel
Re: [edk2] SPI Flash Corruption
On 09/19/18 16:26, Samah Mansour wrote: > Hello, > > > Our product uses a Baytrail with Minnowboard Max bios firmware ( version > 0.93). Every now and then we see SPI flash corruption due to power cuts > while the unit is booting which causes the unit not to boot anymore. After > investigation we noticed that the VPD area is all FFs (address > 44000->47DFF0). > > > We have noticed that the Bios while booting writes to the flash from > several places in the code, which is if interrupted most probably is > causing the corruption. > > > Why is the bios writing all these configurations to flash while booting, is > it to optimize boot time? is it ok if we disable the bios writing to flash > completely to protect ourselves from corruption? The firmware is at liberty to write various non-volatile UEFI variables during boot. Some of those variables are standardized, some others may be specific to UEFI drivers (with correspondingly private namespace GUIDs for the variables). Power loss during flash write (and resultant flash corruption) is expected. My understanding is that the Fault Tolerant Write protocol / driver, sitting between the FVB (firmware volume block, i.e. flash) protocol / driver, and the variable write protocol / driver, implements a kind of journaling. It is described in the Intel whitepaper A Tour Beyond BIOS Implementing UEFI Authenticated Variables in SMM with EDKII September 2015 My expectation has been that the platform should recover from interrupted writes. That is, for a single given UEFI variable, you should either see "before" or "after" status, never "middle". (The whitepaper says that "Individual variable atomicity" is maintained even through a failed "reclaim", with the help of FTW.) If multiple variables should be in sync with each other, that's a different question. If the variables are not in sync, I think "failure to boot" may be a reasonable outcome. But, "failure to boot" means a lot of things, and I hope one should be at least dropped to the setup utility or the shell. Are you seeing an actual crash? Laszlo ___ edk2-devel mailing list edk2-devel@lists.01.org https://lists.01.org/mailman/listinfo/edk2-devel
[edk2] SPI Flash Corruption
Hello, Our product uses a Baytrail with Minnowboard Max bios firmware ( version 0.93). Every now and then we see SPI flash corruption due to power cuts while the unit is booting which causes the unit not to boot anymore. After investigation we noticed that the VPD area is all FFs (address 44000->47DFF0). We have noticed that the Bios while booting writes to the flash from several places in the code, which is if interrupted most probably is causing the corruption. Why is the bios writing all these configurations to flash while booting, is it to optimize boot time? is it ok if we disable the bios writing to flash completely to protect ourselves from corruption? Samah ___ edk2-devel mailing list edk2-devel@lists.01.org https://lists.01.org/mailman/listinfo/edk2-devel
