Hi Shehab, Good, that's "progress"! My guess off the top of my head is that you used a "more recent" compiler (compared to what other gem5 users tend to use), and thus some instructions are being generated that were not critical to the execution of applications other users had so far (and that's mostly why those instructions are not yet implemented). I think you have two options:
1. You can try implementing prefetch_nta, and possibly ignore the non-temporal hint (i.e. implement it as a cacheable prefetch). You can start by looking at the implementation of other prefetch instruction we have in gem5 (basically you can do the same :) ). 2. Try compiling your application (I think we are still talking about the hello world, right?), and target an older architecture (you can do as extreme as march=athlon64) with less optimizations involved to avoid these performance-optimizations (reducing cache pollution in this particular case) that your compiler is trying to apply. My suggestion is to go with the first one, since running real applications compiled for an older architecture with less optimization on a "newer" system is the equivalent of not using "parts/features" of your system (e.g. SIMD units, direct prefetch, etc), which would (possibly) directly impact any study you are working on. Best, On Sat, Sep 7, 2019 at 8:27 PM Shehab Elsayed <shehaby...@gmail.com> wrote: > I am sorry for the late update. I tried running with MESI_Two_Level but > the simulation ends with this error. > > warn: instruction 'prefetch_nta' unimplemented > > gem5.opt: build/X86_MESI_Two_Level/cpu/o3/lsq_unit.hh:621: Fault > LSQUnit<Impl>::read(LSQUnit<Impl>::LSQRequest*, int) [with Impl = > O3CPUImpl; Fault = std::shared_ptr<FaultBase>; LSQUnit<Impl>::LSQRequest = L > SQ<O3CPUImpl>::LSQRequest]: Assertion `!load_inst->isExecuted()' failed. > > Which I believe has something to do with a recent update since I don't > remember seeing it before. And this error happens even for just 2 cores and > 2 threads. > > On Fri, Sep 6, 2019 at 3:16 PM Pouya Fotouhi <pfoto...@ucdavis.edu> wrote: > >> Hi Shehab, >> >> As Jason pointed out, I won’t be surprised if you are having issues with >> classic caches running workloads that rely on locking mechanisms. Your >> pthread implementation is possibly using some synchronization variables >> which requires cache coherence to maintain its correctness, and classic >> caches (at least for now) doesn’t support that. >> >> Switch to ruby caches (I suggest MESI Two Level to begin with), and given >> your kernel version you should be getting stable behavior from gem5. >> >> Best, >> >> On Fri, Sep 6, 2019 at 11:47 AM Jason Lowe-Power <ja...@lowepower.com> >> wrote: >> >>> Hi Shehab, >>> >>> IIRC, there are some issues when using classic caches + x86 + multiple >>> cores on full system mode. I suggest using Ruby (MESI_two_level or >>> MOESI_hammer) for FS simulations. >>> >>> Jason >>> >>> On Fri, Sep 6, 2019 at 11:24 AM Shehab Elsayed <shehaby...@gmail.com> >>> wrote: >>> >>>> My latest experiments are with the classical memory system, but I >>>> remember trying Ruby and it was not different. I am using kernel 4.8.13 and >>>> ubuntu-16.04.1-server-amd64 disk image. I am using Pthreads for my Hello >>>> World program. >>>> >>>> >>>> On Fri, Sep 6, 2019 at 1:13 PM Pouya Fotouhi <pfoto...@ucdavis.edu> >>>> wrote: >>>> >>>>> Hi Shehab, >>>>> >>>>> Can you confirm a few details about the configuration you are using? >>>>> Are you using classic caches or Ruby? What is the kernel version and disk >>>>> image you are using? What is the implementation of your "multithreaded >>>>> hello world" (are you using OMP)? >>>>> >>>>> Best, >>>>> >>>>> On Fri, Sep 6, 2019 at 8:58 AM Shehab Elsayed <shehaby...@gmail.com> >>>>> wrote: >>>>> >>>>>> First of all, thanks for your replies, Ryan and Jason. >>>>>> >>>>>> I have already pulled the latest changes by Pouya and the problem >>>>>> still persists. >>>>>> >>>>>> As for checkpointing, I was originally doing exactly what Jason >>>>>> mentioned and ran into the same problem. I then switched to not >>>>>> checkpointing just to avoid any problems that might be caused by >>>>>> checkpointing (if any). My plan was to go back to checkpointing >>>>>> after proving that it works without it. >>>>>> >>>>>> However, the problem doesn't seem to be related to KVM as linux >>>>>> boots reliable every time. The problem happens after the benchmarks >>>>>> starts >>>>>> execution and it seems to be happening only when running multiple cores >>>>>> (>=4). My latest experiments with a single core and 8 threads for the >>>>>> benchmark seem to be working fine. But once I increase the number of >>>>>> simulated cores problems happen. >>>>>> >>>>>> Also, I have posted a link to the repo I am using to run my tests in >>>>>> a previous message. I have also added 2 debug traces with the Exec flag >>>>>> for >>>>>> a working and non-working examples. >>>>>> >>>>>> >>>>>> On Fri, Sep 6, 2019 at 11:28 AM Jason Lowe-Power <ja...@lowepower.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Shehab, >>>>>>> >>>>>>> One quick note: There is *no way* to have deterministic behavior >>>>>>> when running with KVM. Since you are using the hardware, the underlying >>>>>>> host OS will influence the execution path of the workload. >>>>>>> >>>>>>> To try to narrow down the bug you're seeing, you can try to take a >>>>>>> checkpoint after booting with KVM. Then, the execution from the >>>>>>> checkpoint >>>>>>> should be deterministic since it is 100% in gem5. >>>>>>> >>>>>>> BTW, I doubt you can run the KVM CPU in a VM since this would >>>>>>> require your hardware and the VM to support nested virtualization. There >>>>>>> *is* support for this in the Linux kernel, but I don't think it's been >>>>>>> widely deployed outside of specific cloud environments. >>>>>>> >>>>>>> One other note: Pouya has pushed some changes which implement some >>>>>>> x86 instructions that were causing issues for him. You can try with the >>>>>>> current gem5 mainline to see if that helps. >>>>>>> >>>>>>> Cheers, >>>>>>> Jason >>>>>>> >>>>>>> On Fri, Sep 6, 2019 at 8:22 AM Shehab Elsayed <shehaby...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> That's interesting. Are you using Full System as well? I don't >>>>>>>> think FS behavior is supposed to be so dependent on the host >>>>>>>> environment! >>>>>>>> >>>>>>>> On Fri, Sep 6, 2019 at 11:16 AM Gambord, Ryan < >>>>>>>> gambo...@oregonstate.edu> wrote: >>>>>>>> >>>>>>>>> I have found that gem5 behavior is sensitive to the execution >>>>>>>>> environment. I now run gem5 inside an ubuntu vm on qemu and have had >>>>>>>>> much >>>>>>>>> more consistent results. I haven't tried running kvm gem5 inside a >>>>>>>>> kvm qemu >>>>>>>>> vm, so not sure how that works, but might be worth trying. >>>>>>>>> >>>>>>>>> Ryan >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Sep 6, 2019, 08:07 Shehab Elsayed <shehaby...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> I was wondering if anyone is running into the same problem or if >>>>>>>>>> anyone has any suggestions on how to proceed with debugging this >>>>>>>>>> problem. >>>>>>>>>> >>>>>>>>>> On Mon, Jul 29, 2019 at 4:57 PM Shehab Elsayed < >>>>>>>>>> shehaby...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Sorry for the spam. I just forgot to mention that the system >>>>>>>>>>> configuration I am using is mainly from >>>>>>>>>>> https://github.com/darchr/gem5/tree/jason/kvm-testing/configs/myconfigs. >>>>>>>>>>> <https://github.com/darchr/gem5/tree/jason/kvm-testing/configs/myconfigs> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Shehab Y. Elsayed, MSc. >>>>>>>>>>> PhD Student >>>>>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer >>>>>>>>>>> Engineering >>>>>>>>>>> University of Toronto >>>>>>>>>>> E-mail: shehaby...@gmail.com >>>>>>>>>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Jul 29, 2019 at 4:08 PM Shehab Elsayed < >>>>>>>>>>> shehaby...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> I have set up a repo with gem5 that demonstrates the problem. >>>>>>>>>>>> The repo includes the latest version of gem5 from gem5's github >>>>>>>>>>>> repo with a >>>>>>>>>>>> few patches applied to get KVM working together with the kernel >>>>>>>>>>>> binary and >>>>>>>>>>>> disk image I am using. You can get the repo at >>>>>>>>>>>> https://github.com/ShehabElsayed/gem5_debug.git. >>>>>>>>>>>> <https://github.com/ShehabElsayed/gem5_debug.git> >>>>>>>>>>>> >>>>>>>>>>>> These steps should reproduce the problem: >>>>>>>>>>>> 1- scons build/X86/gem5.opt >>>>>>>>>>>> 2- ./scripts/get_fs_stuff.sh >>>>>>>>>>>> 3- ./scripts/run_fs.sh 8 >>>>>>>>>>>> >>>>>>>>>>>> I have also included sample m5term outputs for both a 2 thread >>>>>>>>>>>> run (m5out_2t) and an 8 thread run (m5out_8t) >>>>>>>>>>>> >>>>>>>>>>>> Any help is really appreciated. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Jul 23, 2019 at 11:01 AM Shehab Elsayed < >>>>>>>>>>>> shehaby...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> When I enable the Exec debug flag I can see that it seems to >>>>>>>>>>>>> be stuck in a spin lock (queued_spin_lock_slowpath) >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Jul 19, 2019 at 5:28 PM Shehab Elsayed < >>>>>>>>>>>>> shehaby...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hello All, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I have a gem5 X86 full system set up that starts with KVM >>>>>>>>>>>>>> cores and then switches to O3 cores once the benchmark reaches >>>>>>>>>>>>>> the region >>>>>>>>>>>>>> of interest. Right now I am testing with a simple >>>>>>>>>>>>>> multithreaded hello world benchmark. Sometimes the benchmark >>>>>>>>>>>>>> completes successfully while others gem5 just seems to hang >>>>>>>>>>>>>> after starting >>>>>>>>>>>>>> the benchmark. I believe it is still executing some instructions >>>>>>>>>>>>>> but >>>>>>>>>>>>>> without making any progress. The chance of this behavior ( >>>>>>>>>>>>>> indeterminism) happening increases as the number of >>>>>>>>>>>>>> simulated cores or the number of threads created by the >>>>>>>>>>>>>> benchmark increases. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Any ideas what might be the reason for this or how I can >>>>>>>>>>>>>> start debugging this problem? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Note: I have tried the patch in https://gem5-review. >>>>>>>>>>>>>> googlesource.com/c/public/gem5/+/19568 but the problem >>>>>>>>>>>>>> persists. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>> gem5-users mailing list >>>>>>>>>> gem5-users@gem5.org >>>>>>>>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> gem5-users mailing list >>>>>>>>> gem5-users@gem5.org >>>>>>>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> gem5-users mailing list >>>>>>>> gem5-users@gem5.org >>>>>>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>>>>> >>>>>>> _______________________________________________ >>>>>>> gem5-users mailing list >>>>>>> gem5-users@gem5.org >>>>>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>>>> >>>>>> _______________________________________________ >>>>>> gem5-users mailing list >>>>>> gem5-users@gem5.org >>>>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>>> >>>>> >>>>> >>>>> -- >>>>> Pouya Fotouhi >>>>> PhD Candidate >>>>> Department of Electrical and Computer Engineering >>>>> University of California, Davis >>>>> _______________________________________________ >>>>> gem5-users mailing list >>>>> gem5-users@gem5.org >>>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>> >>>> _______________________________________________ >>>> gem5-users mailing list >>>> gem5-users@gem5.org >>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>> >>> _______________________________________________ >>> gem5-users mailing list >>> gem5-users@gem5.org >>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >> >> -- >> Pouya Fotouhi >> PhD Candidate >> Department of Electrical and Computer Engineering >> University of California, Davis >> _______________________________________________ >> gem5-users mailing list >> gem5-users@gem5.org >> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users > > _______________________________________________ > gem5-users mailing list > gem5-users@gem5.org > http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users -- Pouya Fotouhi PhD Candidate Department of Electrical and Computer Engineering University of California, Davis
_______________________________________________ gem5-users mailing list gem5-users@gem5.org http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users
