RE: Disabling L1 D-cache and side effects
Ben, You are right. After I corrected copy_page and __copy_tofrom_user functions Linux booted normally. Thanks a lot for the suggestions. Thanks, Marri -Original Message- From: Benjamin Herrenschmidt [mailto:[EMAIL PROTECTED] Sent: Tuesday, September 30, 2008 3:56 PM To: Tirumala Reddy Marri Cc: Olof Johansson; linuxppc-dev@ozlabs.org Subject: RE: Disabling L1 D-cache and side effects On Tue, 2008-09-30 at 15:26 -0700, Tirumala Reddy Marri wrote: > Ben, > I got to bring up Linux on one of the 440 processors with out L1 > dcache to do some bench marking and compare with L1 d-cache enabled. > > I am avoiding any references to dcbz ,dcbt and dcbst . Also the TLB's > are created with cache inhibited. I looked at lwarx/stwcx description, > there seem to be no dependency on L1 cache. Ok. Well, they are generally implemented at the L2 level but maybe not on 440, architecturally, they must be used on cacheable memory but it's possible that 440 being not SMP coherent, the actual implementation of those is too dumb to care. > I don't see any critical exceptions or traps. All I see is /init/bin > failing to execute because data is corrupted. Have you properly replaced dcbz with multiple stores ? I did some bring up work internally on some stuff where dcbz wasn't quite there yet and one pitfall to be careful is that if you force-enable the alternate CONFIG_8xx implementation in the various copy & memset routines in arch/powerpc/lib, you also need to fix those implementations to copy or clear 32 bytes instead of just 16, as 8xx has 16 byte cache lines. Typically failing to do so causes things like memset to fail to properly clear things such as page tables and thus random crap occurs. Cheers, Ben. > Thanks, > Marri > > -Original Message- > From: Benjamin Herrenschmidt [mailto:[EMAIL PROTECTED] > Sent: Tuesday, September 30, 2008 2:31 PM > To: Tirumala Reddy Marri > Cc: Olof Johansson; linuxppc-dev@ozlabs.org > Subject: RE: Disabling L1 D-cache and side effects > > On Tue, 2008-09-30 at 09:57 -0700, Tirumala Reddy Marri wrote: > > Ben, > > Thanks for the response. I am wondering how user space would get > > affected by absence of L1 Dcache. > > You didn't answer my question :-) > > Well, as I said, things like lwarx/stwcx not working, dcbz taking > alignment exceptions, etc... > > Ben. > > > Thanks, > > Marri > > > > -Original Message- > > From: Benjamin Herrenschmidt [mailto:[EMAIL PROTECTED] > > Sent: Tuesday, September 30, 2008 12:16 AM > > To: Tirumala Reddy Marri > > Cc: Olof Johansson; linuxppc-dev@ozlabs.org > > Subject: RE: Disabling L1 D-cache and side effects > > > > On Mon, 2008-09-29 at 14:38 -0700, Tirumala Reddy Marri wrote: > > > Could you please point me to the which does the Critical error > > > (Machine > > > Check) recovery. BTW I am successful booting the Linux until > > > rootfs is > > > > > being mounted. It fails to mount the Linux saying that blocks are > > > corrupted in file system. I had to modify lots of initial bring up > > > code to disable D-cache and make sure all TLB's are cache inhibited. > > > Ando also made sure none of the misc_32.S , entry_32.S and head.S > > > makes any references to d-cache. > > > > Why the heck are you doing that btw ? AFAIK, as Olof says, things > > like > > > atomic operations will not work, dcbz neither etc... it's likely > > that even if you manage to plaster around all of this in the kernel, > > whatever userspace code you'll try to run in userspace will blow up > too... > > > > Cheers, > > Ben. ___ Linuxppc-dev mailing list Linuxppc-dev@ozlabs.org https://ozlabs.org/mailman/listinfo/linuxppc-dev
RE: Disabling L1 D-cache and side effects
On Tue, 2008-09-30 at 15:26 -0700, Tirumala Reddy Marri wrote: > Ben, > I got to bring up Linux on one of the 440 processors with out L1 > dcache to do some bench marking and compare with L1 d-cache enabled. > > I am avoiding any references to dcbz ,dcbt and dcbst . Also the TLB's > are created with cache inhibited. I looked at lwarx/stwcx description, > there seem to be no dependency on L1 cache. Ok. Well, they are generally implemented at the L2 level but maybe not on 440, architecturally, they must be used on cacheable memory but it's possible that 440 being not SMP coherent, the actual implementation of those is too dumb to care. > I don't see any critical exceptions or traps. All I see is /init/bin > failing to execute because data is corrupted. Have you properly replaced dcbz with multiple stores ? I did some bring up work internally on some stuff where dcbz wasn't quite there yet and one pitfall to be careful is that if you force-enable the alternate CONFIG_8xx implementation in the various copy & memset routines in arch/powerpc/lib, you also need to fix those implementations to copy or clear 32 bytes instead of just 16, as 8xx has 16 byte cache lines. Typically failing to do so causes things like memset to fail to properly clear things such as page tables and thus random crap occurs. Cheers, Ben. > Thanks, > Marri > > -Original Message- > From: Benjamin Herrenschmidt [mailto:[EMAIL PROTECTED] > Sent: Tuesday, September 30, 2008 2:31 PM > To: Tirumala Reddy Marri > Cc: Olof Johansson; linuxppc-dev@ozlabs.org > Subject: RE: Disabling L1 D-cache and side effects > > On Tue, 2008-09-30 at 09:57 -0700, Tirumala Reddy Marri wrote: > > Ben, > > Thanks for the response. I am wondering how user space would get > > affected by absence of L1 Dcache. > > You didn't answer my question :-) > > Well, as I said, things like lwarx/stwcx not working, dcbz taking > alignment exceptions, etc... > > Ben. > > > Thanks, > > Marri > > > > -Original Message- > > From: Benjamin Herrenschmidt [mailto:[EMAIL PROTECTED] > > Sent: Tuesday, September 30, 2008 12:16 AM > > To: Tirumala Reddy Marri > > Cc: Olof Johansson; linuxppc-dev@ozlabs.org > > Subject: RE: Disabling L1 D-cache and side effects > > > > On Mon, 2008-09-29 at 14:38 -0700, Tirumala Reddy Marri wrote: > > > Could you please point me to the which does the Critical error > > > (Machine > > > Check) recovery. BTW I am successful booting the Linux until rootfs > > > is > > > > > being mounted. It fails to mount the Linux saying that blocks are > > > corrupted in file system. I had to modify lots of initial bring up > > > code to disable D-cache and make sure all TLB's are cache inhibited. > > > Ando also made sure none of the misc_32.S , entry_32.S and head.S > > > makes any references to d-cache. > > > > Why the heck are you doing that btw ? AFAIK, as Olof says, things like > > > atomic operations will not work, dcbz neither etc... it's likely that > > even if you manage to plaster around all of this in the kernel, > > whatever userspace code you'll try to run in userspace will blow up > too... > > > > Cheers, > > Ben. ___ Linuxppc-dev mailing list Linuxppc-dev@ozlabs.org https://ozlabs.org/mailman/listinfo/linuxppc-dev
RE: Disabling L1 D-cache and side effects
Ben, I got to bring up Linux on one of the 440 processors with out L1 dcache to do some bench marking and compare with L1 d-cache enabled. I am avoiding any references to dcbz ,dcbt and dcbst . Also the TLB's are created with cache inhibited. I looked at lwarx/stwcx description, there seem to be no dependency on L1 cache. I don't see any critical exceptions or traps. All I see is /init/bin failing to execute because data is corrupted. Thanks, Marri -Original Message- From: Benjamin Herrenschmidt [mailto:[EMAIL PROTECTED] Sent: Tuesday, September 30, 2008 2:31 PM To: Tirumala Reddy Marri Cc: Olof Johansson; linuxppc-dev@ozlabs.org Subject: RE: Disabling L1 D-cache and side effects On Tue, 2008-09-30 at 09:57 -0700, Tirumala Reddy Marri wrote: > Ben, > Thanks for the response. I am wondering how user space would get > affected by absence of L1 Dcache. You didn't answer my question :-) Well, as I said, things like lwarx/stwcx not working, dcbz taking alignment exceptions, etc... Ben. > Thanks, > Marri > > -Original Message- > From: Benjamin Herrenschmidt [mailto:[EMAIL PROTECTED] > Sent: Tuesday, September 30, 2008 12:16 AM > To: Tirumala Reddy Marri > Cc: Olof Johansson; linuxppc-dev@ozlabs.org > Subject: RE: Disabling L1 D-cache and side effects > > On Mon, 2008-09-29 at 14:38 -0700, Tirumala Reddy Marri wrote: > > Could you please point me to the which does the Critical error > > (Machine > > Check) recovery. BTW I am successful booting the Linux until rootfs > > is > > > being mounted. It fails to mount the Linux saying that blocks are > > corrupted in file system. I had to modify lots of initial bring up > > code to disable D-cache and make sure all TLB's are cache inhibited. > > Ando also made sure none of the misc_32.S , entry_32.S and head.S > > makes any references to d-cache. > > Why the heck are you doing that btw ? AFAIK, as Olof says, things like > atomic operations will not work, dcbz neither etc... it's likely that > even if you manage to plaster around all of this in the kernel, > whatever userspace code you'll try to run in userspace will blow up too... > > Cheers, > Ben. ___ Linuxppc-dev mailing list Linuxppc-dev@ozlabs.org https://ozlabs.org/mailman/listinfo/linuxppc-dev
RE: Disabling L1 D-cache and side effects
On Tue, 2008-09-30 at 09:57 -0700, Tirumala Reddy Marri wrote: > Ben, > Thanks for the response. I am wondering how user space would get > affected by absence of L1 Dcache. You didn't answer my question :-) Well, as I said, things like lwarx/stwcx not working, dcbz taking alignment exceptions, etc... Ben. > Thanks, > Marri > > -Original Message- > From: Benjamin Herrenschmidt [mailto:[EMAIL PROTECTED] > Sent: Tuesday, September 30, 2008 12:16 AM > To: Tirumala Reddy Marri > Cc: Olof Johansson; linuxppc-dev@ozlabs.org > Subject: RE: Disabling L1 D-cache and side effects > > On Mon, 2008-09-29 at 14:38 -0700, Tirumala Reddy Marri wrote: > > Could you please point me to the which does the Critical error > > (Machine > > Check) recovery. BTW I am successful booting the Linux until rootfs is > > > being mounted. It fails to mount the Linux saying that blocks are > > corrupted in file system. I had to modify lots of initial bring up > > code to disable D-cache and make sure all TLB's are cache inhibited. > > Ando also made sure none of the misc_32.S , entry_32.S and head.S > > makes any references to d-cache. > > Why the heck are you doing that btw ? AFAIK, as Olof says, things like > atomic operations will not work, dcbz neither etc... it's likely that > even if you manage to plaster around all of this in the kernel, whatever > userspace code you'll try to run in userspace will blow up too... > > Cheers, > Ben. ___ Linuxppc-dev mailing list Linuxppc-dev@ozlabs.org https://ozlabs.org/mailman/listinfo/linuxppc-dev
RE: Disabling L1 D-cache and side effects
Ben, Thanks for the response. I am wondering how user space would get affected by absence of L1 Dcache. Thanks, Marri -Original Message- From: Benjamin Herrenschmidt [mailto:[EMAIL PROTECTED] Sent: Tuesday, September 30, 2008 12:16 AM To: Tirumala Reddy Marri Cc: Olof Johansson; linuxppc-dev@ozlabs.org Subject: RE: Disabling L1 D-cache and side effects On Mon, 2008-09-29 at 14:38 -0700, Tirumala Reddy Marri wrote: > Could you please point me to the which does the Critical error > (Machine > Check) recovery. BTW I am successful booting the Linux until rootfs is > being mounted. It fails to mount the Linux saying that blocks are > corrupted in file system. I had to modify lots of initial bring up > code to disable D-cache and make sure all TLB's are cache inhibited. > Ando also made sure none of the misc_32.S , entry_32.S and head.S > makes any references to d-cache. Why the heck are you doing that btw ? AFAIK, as Olof says, things like atomic operations will not work, dcbz neither etc... it's likely that even if you manage to plaster around all of this in the kernel, whatever userspace code you'll try to run in userspace will blow up too... Cheers, Ben. ___ Linuxppc-dev mailing list Linuxppc-dev@ozlabs.org https://ozlabs.org/mailman/listinfo/linuxppc-dev
RE: Disabling L1 D-cache and side effects
On Mon, 2008-09-29 at 14:38 -0700, Tirumala Reddy Marri wrote: > Could you please point me to the which does the Critical error (Machine > Check) recovery. BTW I am successful booting the Linux until rootfs is > being mounted. It fails to mount the Linux saying that blocks are > corrupted in file system. I had to modify lots of initial bring up code > to disable D-cache and make sure all TLB's are cache inhibited. Ando > also made sure none of the misc_32.S , entry_32.S and head.S makes any > references to d-cache. Why the heck are you doing that btw ? AFAIK, as Olof says, things like atomic operations will not work, dcbz neither etc... it's likely that even if you manage to plaster around all of this in the kernel, whatever userspace code you'll try to run in userspace will blow up too... Cheers, Ben. ___ Linuxppc-dev mailing list Linuxppc-dev@ozlabs.org https://ozlabs.org/mailman/listinfo/linuxppc-dev
RE: Disabling L1 D-cache and side effects
Hi Olof, I see code in arch/powerpc/kernel/traps.c . But only cbe_machine_check_handler() is the only function assigned to ppc_md.machine_check_exception. Which is also not doing any recovery. It just dumps the registers and return 0. Which would cause system Panic. Could you please point me to the which does the Critical error (Machine Check) recovery. BTW I am successful booting the Linux until rootfs is being mounted. It fails to mount the Linux saying that blocks are corrupted in file system. I had to modify lots of initial bring up code to disable D-cache and make sure all TLB's are cache inhibited. Ando also made sure none of the misc_32.S , entry_32.S and head.S makes any references to d-cache. Thanks, Marri -Original Message- From: Olof Johansson [mailto:[EMAIL PROTECTED] Sent: Monday, September 29, 2008 2:14 PM To: Tirumala Reddy Marri Cc: linuxppc-dev@ozlabs.org Subject: Re: Disabling L1 D-cache and side effects On Mon, Sep 29, 2008 at 02:00:06PM -0700, Tirumala Reddy Marri wrote: > Olof, > Thanks for the response. > > Is there a piece of code in Linux which does the Machine check > recovery and continue normal execution ? Yes, there is. Several pieces, actually. -Olof ___ Linuxppc-dev mailing list Linuxppc-dev@ozlabs.org https://ozlabs.org/mailman/listinfo/linuxppc-dev
Re: Disabling L1 D-cache and side effects
On Mon, Sep 29, 2008 at 02:00:06PM -0700, Tirumala Reddy Marri wrote: > Olof, > Thanks for the response. > > Is there a piece of code in Linux which does the Machine check > recovery and continue normal execution ? Yes, there is. Several pieces, actually. -Olof ___ Linuxppc-dev mailing list Linuxppc-dev@ozlabs.org https://ozlabs.org/mailman/listinfo/linuxppc-dev
RE: Disabling L1 D-cache and side effects
Olof, Thanks for the response. Is there a piece of code in Linux which does the Machine check recovery and continue normal execution ? Thanks and Regards, Marri -Original Message- From: Olof Johansson [mailto:[EMAIL PROTECTED] Sent: Monday, September 29, 2008 11:05 AM To: Tirumala Reddy Marri Cc: linuxppc-dev@ozlabs.org Subject: Re: Disabling L1 D-cache and side effects On Mon, Sep 29, 2008 at 10:05:41AM -0700, Tirumala Reddy Marri wrote: > > Hi, > I had to bring up a PPC based SOC with L1 dcache disabled. I did > that and tried to boot Linux using RAMDISK/NFS mount. In RAMDISK I see > the file system errors. In case of NFS mount I see error saying failed > to load ld.so library. > > Could you guys please share thoughts what are the different side > effects might be causing this. There are a number of things you have to be careful about when you disable caches. Depending on your implementation, you likely can't use lwarx/stwcx, cache ops will not work, etc. Also, are you 100% sure that this is caused only by disabling L1D, and not by any other problems with your silicon? If you're doing early bringup of you'll have a whole lot of debug work in front of you, and this mailing list is probably not the best place to bring your homework. -Olof ___ Linuxppc-dev mailing list Linuxppc-dev@ozlabs.org https://ozlabs.org/mailman/listinfo/linuxppc-dev
Re: Disabling L1 D-cache and side effects
On Mon, Sep 29, 2008 at 10:05:41AM -0700, Tirumala Reddy Marri wrote: > > Hi, > I had to bring up a PPC based SOC with L1 dcache disabled. I did that > and tried to boot Linux using RAMDISK/NFS mount. In RAMDISK I see the > file system errors. In case of NFS mount I see error saying failed to > load ld.so library. > > Could you guys please share thoughts what are the different side > effects might be causing this. There are a number of things you have to be careful about when you disable caches. Depending on your implementation, you likely can't use lwarx/stwcx, cache ops will not work, etc. Also, are you 100% sure that this is caused only by disabling L1D, and not by any other problems with your silicon? If you're doing early bringup of you'll have a whole lot of debug work in front of you, and this mailing list is probably not the best place to bring your homework. -Olof ___ Linuxppc-dev mailing list Linuxppc-dev@ozlabs.org https://ozlabs.org/mailman/listinfo/linuxppc-dev
