So actually load instruction gets executed twice causing the assertion to fail on the second time.
7694139490000: system.switch_cpus.iew.lsq.thread0: Doing memory access for inst [sn:15059405] PC (0xffffffff810ed626=>0xffffffff810ed62a).(1=>2) 7694139490000: system.switch_cpus.iew.lsq.thread0: Load [sn:15059405] not executed from fault 7694139490000: system.switch_cpus.iew.lsq.thread0: 1- Setting [sn:15059405] as executed (I added this message to track when LSQ instructions are set as executed) I believe this instruction should then be committed and removed from the LSQ before before executed again, however, this does not happen. Instead it gets executed again before being removed and then comes the assertion failure that it has already executed. I see that it gets sent to commit 7694139490000: system.switch_cpus.iew: Sending instructions to commit, [sn:15059405] PC (0xffffffff810ed626=>0xffffffff810ed62a).(1=>2). but it never actually gets to commit and removed from LSQ. On Mon, Sep 9, 2019 at 3:01 PM Pouya Fotouhi <pfoto...@ucdavis.edu> wrote: > You can try dumping Exec trace for the last few million ticks and see what > is going on in your LSQ and why you have load instruction that is not > executed. > > Best, > > On Mon, Sep 9, 2019 at 11:28 AM Shehab Elsayed <shehaby...@gmail.com> > wrote: > >> I am not sure that prefetch_nta is the problem. For different runs the >> simulation would fail after different periods after printing the prefetch >> _nta warning message. Also, from what I have seen in different posts it >> seems that this warning has been around for a while. >> >> I tried compiling my hello world program with -march=athlon64 alone and >> together with -O0 and the the same problem happens. >> >> Also, the I am building my benchmark on the disk image directly using >> qemu and the gcc on the image is versio 5.4.0 >> >> >> >> On Sun, Sep 8, 2019 at 4:14 PM Pouya Fotouhi <pfoto...@ucdavis.edu> >> wrote: >> >>> Hi Shehab, >>> >>> Good, that's "progress"! >>> My guess off the top of my head is that you used a "more recent" >>> compiler (compared to what other gem5 users tend to use), and thus some >>> instructions are being generated that were not critical to the execution of >>> applications other users had so far (and that's mostly why those >>> instructions are not yet implemented). I think you have two options: >>> >>> 1. You can try implementing prefetch_nta, and possibly ignore the >>> non-temporal hint (i.e. implement it as a cacheable prefetch). You can >>> start by looking at the implementation of other prefetch instruction we >>> have in gem5 (basically you can do the same :) ). >>> 2. Try compiling your application (I think we are still talking >>> about the hello world, right?), and target an older architecture (you can >>> do as extreme as march=athlon64) with less optimizations involved to >>> avoid >>> these performance-optimizations (reducing cache pollution in this >>> particular case) that your compiler is trying to apply. >>> >>> My suggestion is to go with the first one, since running real >>> applications compiled for an older architecture with less optimization on a >>> "newer" system is the equivalent of not using "parts/features" of your >>> system (e.g. SIMD units, direct prefetch, etc), which would (possibly) >>> directly impact any study you are working on. >>> >>> Best, >>> >>> On Sat, Sep 7, 2019 at 8:27 PM Shehab Elsayed <shehaby...@gmail.com> >>> wrote: >>> >>>> I am sorry for the late update. I tried running with MESI_Two_Level but >>>> the simulation ends with this error. >>>> >>>> warn: instruction 'prefetch_nta' unimplemented >>>> >>>> gem5.opt: build/X86_MESI_Two_Level/cpu/o3/lsq_unit.hh:621: Fault >>>> LSQUnit<Impl>::read(LSQUnit<Impl>::LSQRequest*, int) [with Impl = >>>> O3CPUImpl; Fault = std::shared_ptr<FaultBase>; LSQUnit<Impl>::LSQRequest = >>>> L >>>> SQ<O3CPUImpl>::LSQRequest]: Assertion `!load_inst->isExecuted()' failed. >>>> >>>> Which I believe has something to do with a recent update since I don't >>>> remember seeing it before. And this error happens even for just 2 cores and >>>> 2 threads. >>>> >>>> On Fri, Sep 6, 2019 at 3:16 PM Pouya Fotouhi <pfoto...@ucdavis.edu> >>>> wrote: >>>> >>>>> Hi Shehab, >>>>> >>>>> As Jason pointed out, I won’t be surprised if you are having issues >>>>> with classic caches running workloads that rely on locking mechanisms. >>>>> Your >>>>> pthread implementation is possibly using some synchronization variables >>>>> which requires cache coherence to maintain its correctness, and classic >>>>> caches (at least for now) doesn’t support that. >>>>> >>>>> Switch to ruby caches (I suggest MESI Two Level to begin with), and >>>>> given your kernel version you should be getting stable behavior from gem5. >>>>> >>>>> Best, >>>>> >>>>> On Fri, Sep 6, 2019 at 11:47 AM Jason Lowe-Power <ja...@lowepower.com> >>>>> wrote: >>>>> >>>>>> Hi Shehab, >>>>>> >>>>>> IIRC, there are some issues when using classic caches + x86 + >>>>>> multiple cores on full system mode. I suggest using Ruby (MESI_two_level >>>>>> or >>>>>> MOESI_hammer) for FS simulations. >>>>>> >>>>>> Jason >>>>>> >>>>>> On Fri, Sep 6, 2019 at 11:24 AM Shehab Elsayed <shehaby...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> My latest experiments are with the classical memory system, but I >>>>>>> remember trying Ruby and it was not different. I am using kernel 4.8.13 >>>>>>> and >>>>>>> ubuntu-16.04.1-server-amd64 disk image. I am using Pthreads for my Hello >>>>>>> World program. >>>>>>> >>>>>>> >>>>>>> On Fri, Sep 6, 2019 at 1:13 PM Pouya Fotouhi <pfoto...@ucdavis.edu> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Shehab, >>>>>>>> >>>>>>>> Can you confirm a few details about the configuration you are >>>>>>>> using? Are you using classic caches or Ruby? What is the kernel >>>>>>>> version and >>>>>>>> disk image you are using? What is the implementation of your >>>>>>>> "multithreaded >>>>>>>> hello world" (are you using OMP)? >>>>>>>> >>>>>>>> Best, >>>>>>>> >>>>>>>> On Fri, Sep 6, 2019 at 8:58 AM Shehab Elsayed <shehaby...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> First of all, thanks for your replies, Ryan and Jason. >>>>>>>>> >>>>>>>>> I have already pulled the latest changes by Pouya and the problem >>>>>>>>> still persists. >>>>>>>>> >>>>>>>>> As for checkpointing, I was originally doing exactly what Jason >>>>>>>>> mentioned and ran into the same problem. I then switched to not >>>>>>>>> checkpointing just to avoid any problems that might be caused by >>>>>>>>> checkpointing (if any). My plan was to go back to checkpointing >>>>>>>>> after proving that it works without it. >>>>>>>>> >>>>>>>>> However, the problem doesn't seem to be related to KVM as linux >>>>>>>>> boots reliable every time. The problem happens after the benchmarks >>>>>>>>> starts >>>>>>>>> execution and it seems to be happening only when running multiple >>>>>>>>> cores >>>>>>>>> (>=4). My latest experiments with a single core and 8 threads for the >>>>>>>>> benchmark seem to be working fine. But once I increase the number of >>>>>>>>> simulated cores problems happen. >>>>>>>>> >>>>>>>>> Also, I have posted a link to the repo I am using to run my tests >>>>>>>>> in a previous message. I have also added 2 debug traces with the Exec >>>>>>>>> flag >>>>>>>>> for a working and non-working examples. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Sep 6, 2019 at 11:28 AM Jason Lowe-Power < >>>>>>>>> ja...@lowepower.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Shehab, >>>>>>>>>> >>>>>>>>>> One quick note: There is *no way* to have deterministic behavior >>>>>>>>>> when running with KVM. Since you are using the hardware, the >>>>>>>>>> underlying >>>>>>>>>> host OS will influence the execution path of the workload. >>>>>>>>>> >>>>>>>>>> To try to narrow down the bug you're seeing, you can try to take >>>>>>>>>> a checkpoint after booting with KVM. Then, the execution from the >>>>>>>>>> checkpoint should be deterministic since it is 100% in gem5. >>>>>>>>>> >>>>>>>>>> BTW, I doubt you can run the KVM CPU in a VM since this would >>>>>>>>>> require your hardware and the VM to support nested virtualization. >>>>>>>>>> There >>>>>>>>>> *is* support for this in the Linux kernel, but I don't think it's >>>>>>>>>> been >>>>>>>>>> widely deployed outside of specific cloud environments. >>>>>>>>>> >>>>>>>>>> One other note: Pouya has pushed some changes which implement >>>>>>>>>> some x86 instructions that were causing issues for him. You can try >>>>>>>>>> with >>>>>>>>>> the current gem5 mainline to see if that helps. >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Jason >>>>>>>>>> >>>>>>>>>> On Fri, Sep 6, 2019 at 8:22 AM Shehab Elsayed < >>>>>>>>>> shehaby...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> That's interesting. Are you using Full System as well? I don't >>>>>>>>>>> think FS behavior is supposed to be so dependent on the host >>>>>>>>>>> environment! >>>>>>>>>>> >>>>>>>>>>> On Fri, Sep 6, 2019 at 11:16 AM Gambord, Ryan < >>>>>>>>>>> gambo...@oregonstate.edu> wrote: >>>>>>>>>>> >>>>>>>>>>>> I have found that gem5 behavior is sensitive to the execution >>>>>>>>>>>> environment. I now run gem5 inside an ubuntu vm on qemu and have >>>>>>>>>>>> had much >>>>>>>>>>>> more consistent results. I haven't tried running kvm gem5 inside a >>>>>>>>>>>> kvm qemu >>>>>>>>>>>> vm, so not sure how that works, but might be worth trying. >>>>>>>>>>>> >>>>>>>>>>>> Ryan >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Sep 6, 2019, 08:07 Shehab Elsayed <shehaby...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I was wondering if anyone is running into the same problem or >>>>>>>>>>>>> if anyone has any suggestions on how to proceed with debugging >>>>>>>>>>>>> this problem. >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Jul 29, 2019 at 4:57 PM Shehab Elsayed < >>>>>>>>>>>>> shehaby...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Sorry for the spam. I just forgot to mention that the system >>>>>>>>>>>>>> configuration I am using is mainly from >>>>>>>>>>>>>> https://github.com/darchr/gem5/tree/jason/kvm-testing/configs/myconfigs. >>>>>>>>>>>>>> <https://github.com/darchr/gem5/tree/jason/kvm-testing/configs/myconfigs> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Shehab Y. Elsayed, MSc. >>>>>>>>>>>>>> PhD Student >>>>>>>>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer >>>>>>>>>>>>>> Engineering >>>>>>>>>>>>>> University of Toronto >>>>>>>>>>>>>> E-mail: shehaby...@gmail.com >>>>>>>>>>>>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Jul 29, 2019 at 4:08 PM Shehab Elsayed < >>>>>>>>>>>>>> shehaby...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I have set up a repo with gem5 that demonstrates the >>>>>>>>>>>>>>> problem. The repo includes the latest version of gem5 from >>>>>>>>>>>>>>> gem5's github >>>>>>>>>>>>>>> repo with a few patches applied to get KVM working together >>>>>>>>>>>>>>> with the kernel >>>>>>>>>>>>>>> binary and disk image I am using. You can get the repo at >>>>>>>>>>>>>>> https://github.com/ShehabElsayed/gem5_debug.git. >>>>>>>>>>>>>>> <https://github.com/ShehabElsayed/gem5_debug.git> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These steps should reproduce the problem: >>>>>>>>>>>>>>> 1- scons build/X86/gem5.opt >>>>>>>>>>>>>>> 2- ./scripts/get_fs_stuff.sh >>>>>>>>>>>>>>> 3- ./scripts/run_fs.sh 8 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I have also included sample m5term outputs for both a 2 >>>>>>>>>>>>>>> thread run (m5out_2t) and an 8 thread run (m5out_8t) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Any help is really appreciated. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Jul 23, 2019 at 11:01 AM Shehab Elsayed < >>>>>>>>>>>>>>> shehaby...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> When I enable the Exec debug flag I can see that it seems >>>>>>>>>>>>>>>> to be stuck in a spin lock (queued_spin_lock_slowpath) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Fri, Jul 19, 2019 at 5:28 PM Shehab Elsayed < >>>>>>>>>>>>>>>> shehaby...@gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hello All, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I have a gem5 X86 full system set up that starts with KVM >>>>>>>>>>>>>>>>> cores and then switches to O3 cores once the benchmark >>>>>>>>>>>>>>>>> reaches the region >>>>>>>>>>>>>>>>> of interest. Right now I am testing with a simple >>>>>>>>>>>>>>>>> multithreaded hello world benchmark. Sometimes the >>>>>>>>>>>>>>>>> benchmark completes successfully while others gem5 just seems >>>>>>>>>>>>>>>>> to hang after >>>>>>>>>>>>>>>>> starting the benchmark. I believe it is still executing some >>>>>>>>>>>>>>>>> instructions >>>>>>>>>>>>>>>>> but without making any progress. The chance of this behavior ( >>>>>>>>>>>>>>>>> indeterminism) happening increases as the number of >>>>>>>>>>>>>>>>> simulated cores or the number of threads created by the >>>>>>>>>>>>>>>>> benchmark increases. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Any ideas what might be the reason for this or how I can >>>>>>>>>>>>>>>>> start debugging this problem? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Note: I have tried the patch in https://gem5-review. >>>>>>>>>>>>>>>>> googlesource.com/c/public/gem5/+/19568 but the problem >>>>>>>>>>>>>>>>> persists. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> gem5-users mailing list >>>>>>>>>>>>> gem5-users@gem5.org >>>>>>>>>>>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> gem5-users mailing list >>>>>>>>>>>> gem5-users@gem5.org >>>>>>>>>>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> gem5-users mailing list >>>>>>>>>>> gem5-users@gem5.org >>>>>>>>>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> gem5-users mailing list >>>>>>>>>> gem5-users@gem5.org >>>>>>>>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> gem5-users mailing list >>>>>>>>> gem5-users@gem5.org >>>>>>>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Pouya Fotouhi >>>>>>>> PhD Candidate >>>>>>>> Department of Electrical and Computer Engineering >>>>>>>> University of California, Davis >>>>>>>> _______________________________________________ >>>>>>>> gem5-users mailing list >>>>>>>> gem5-users@gem5.org >>>>>>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>>>>> >>>>>>> _______________________________________________ >>>>>>> gem5-users mailing list >>>>>>> gem5-users@gem5.org >>>>>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>>>> >>>>>> _______________________________________________ >>>>>> gem5-users mailing list >>>>>> gem5-users@gem5.org >>>>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>>> >>>>> -- >>>>> Pouya Fotouhi >>>>> PhD Candidate >>>>> Department of Electrical and Computer Engineering >>>>> University of California, Davis >>>>> _______________________________________________ >>>>> gem5-users mailing list >>>>> gem5-users@gem5.org >>>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>> >>>> _______________________________________________ >>>> gem5-users mailing list >>>> gem5-users@gem5.org >>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>> >>> >>> >>> -- >>> Pouya Fotouhi >>> PhD Candidate >>> Department of Electrical and Computer Engineering >>> University of California, Davis >>> _______________________________________________ >>> gem5-users mailing list >>> gem5-users@gem5.org >>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >> >> _______________________________________________ >> gem5-users mailing list >> gem5-users@gem5.org >> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users > > > > -- > Pouya Fotouhi > PhD Candidate > Department of Electrical and Computer Engineering > University of California, Davis > _______________________________________________ > gem5-users mailing list > gem5-users@gem5.org > http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users
_______________________________________________ gem5-users mailing list gem5-users@gem5.org http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users
