Hi Shan,
My schedule is very full today, but I should have some time to look into
this more tomorrow.
Can you email me the following (off the mailing list, so we don't spam
everybody):
1) The first few thousand lines of the Iface debug output.
2) A Simics checkpoint of your app immediately before you load flexus.
You can create this by issuing the write-configuration command. I am only
interested in the text file, not the image files.
3) A simics checkpoint after flexus has been loaded and run for a few
thousand cycles. Again, I only need the text file, no images.
Hopefully, this output will indicate why Flexus is not getting
instructions from all the CPUs, or suggest a further debugging strategy.
Thanks,
-Tom Wenisch
On Tue, 1 Nov 2005, shan wrote:
> Hi Tom,
> I still didn't figure this out. I have turned on the iface debugging, but
> I didn't see anything abnormal during the initialize state. I am not sure
> what to check in those two initialization functionss. I am just sure that
> the doInitialize() in InorderSimicsFeeder/InorderSimicsFeederImpl.cpp did go
> through all the four processors.
> Actually I have output:
> 17 <ExecuteImpl.cpp:285> (execute[0]) {0}- EX initalized
> 18 <ExecuteImpl.cpp:288> (execute[0]) {0}- Sequential Consistency Enabled
> 19 <ExecuteImpl.cpp:285> (execute[1]) {0}- EX initalized
> 20 <ExecuteImpl.cpp:288> (execute[1]) {0}- Sequential Consistency Enabled
> 21 <ExecuteImpl.cpp:285> (execute[2]) {0}- EX initalized
> 22 <ExecuteImpl.cpp:288> (execute[2]) {0}- Sequential Consistency Enabled
> 23 <ExecuteImpl.cpp:285> (execute[3]) {0}- EX initalized
> 24 <ExecuteImpl.cpp:288> (execute[3]) {0}- Sequential Consistency Enabled
> Does this mean executors are correctly connected?
>
> execute[0,1,3] never appears afterwards.
> Actually, I also have " L1i[0,1,3]" in the trace during the first 100~200
> step, but they disappear afterwards. Do you think maybe this is some
> scheduling configuration wrong?
>
> do you have some clue about this? Or do you have some suggestion, what type
> of debugging information will can to diagnose this?
>
> I also tried to run a multiple thread program after I comment that assertion
> on flexus.cc line 330. it runs, but it seems that only one thread is
> executing. Since this is a set-unset-flag application, the execution just
> keeps going on and hasn't finished after tens of minutes (I am not sure this
> is caused by slow simulating or because the 2nd thread never scheduled and
> thus the flag is never set). Do you think this imply a scheduling
> configuration problem?
>
> Thanks very much
> Shan
> -----Original Message-----
> From: Thomas Wenisch [mailto:[email protected]]
> Sent: Friday, October 28, 2005 8:18 PM
> To: shan
> Cc: 'SimFlex software support'
> Subject: Re: FW: [Simflex] x86 CMP configuration
>
>
>
> On Thu, 27 Oct 2005, shan wrote:
>
> >
> > I am trying to run the x86 CMP module. I am not sure if what I did is
> > correct.
> >
> > I used the simics enterprise-4p.simics macro and didn't change any
> > configuration on the flexus side (should I change any configuration at the
> > flexus side?)
> >
> > When I load the module, it shows:
> >
> > 1 <startup.cpp:107> {0}- Initializing Flexus.
>
> ...
>
> > 4 <InorderSimicsFeederImpl.cpp:338> (feeder[<undefined>]) {0}- Connecting:
> > cpu0
> >
> > 5 <InorderSimicsFeederImpl.cpp:338> (feeder[<undefined>]) {0}- Connecting:
> > cpu1
> >
> > 6 <InorderSimicsFeederImpl.cpp:338> (feeder[<undefined>]) {0}- Connecting:
> > cpu2
> >
> > 7 <InorderSimicsFeederImpl.cpp:338> (feeder[<undefined>]) {0}- Connecting:
> > cpu3
> >
>
> > Why above feeder index is 'undefined', does it imply I made some mistake
> in
> > configuration?
>
> The <undefined> notation is harmless. It indicates that this particular
> debug message does not have a {FlexusIdx} field associated with it,
> because it is in a piece of code that is not associated with a particular
> node. All debug messages actually consist of a bunch of fields, like the
> sequence number, file, line, source component, etc. You can filter,
> redirect, and reformat debug output based on these fields. Take a look at
> debug.cfg to get an idea of how this works.
>
> >
> >
> > Then I type 'continue'. After a while, the screen shows:
> >
> > 25 <MemoryMapImpl.cpp:319> {0}- Assigned 0 pages.
> >
> > 26 <flexus.cpp:240> {0}- Timestamp: 2005-Oct-25 19:49:10
> >
> > 27 <flexus.cpp:326> {108288}- Watchdog timer expired. No progress by CPU
> 0
> > for 100215cycles
> >
> > 28 <flexus.cpp:330> (<undefined>[<undefined>]) {108288}- Assertion failed:
> > ((!(theWatchdogCounts[i] < theWatchdogTimeout + 10))) : Watchdog timer
> > expired. No progress by CPU 0 for 100215cycles
> >
> > *** Simics getting shaky, switching to 'safe' mode.
> >
> > *** Simics (main thread) received an abort signal, probably an assertion.
> >
> > <Simics is running in 'safe' mode>
>
> > I checked the assertion at flexus.cpp line 330. It seems that if the cpu
> > idles for some time, the assertion would fail. Yes? I temporarily removed
> > the assertion and continues. And get following debug output:
> >
>
> This assertion indicates that one of the CPUs has not issued an
> instruction or completed a memory request for 100k cycles. This can only
> come about if there is a deadlock in a Flexus coherence protocol (very
> unlikely, as we have run trillions of cycles since we last found a
> deadlock bug, but perhaps there is something x86 specific that we are
> missing), or for some reason Flexus has not connected itself properly to
> your 4 CPUs. Given that you only see output from execute[2], the latter
> is very likely.
>
> I suggest you turn up the debug output to confirm that none of the CPUs
> are doing anything. Issue the command "flexus.set-debug-severity iface"
> before starting the simulation. You should get lots more output, from
> which you can figure out if any of the other nodes are active.
>
> The relevant pieces of code to study to figure out this bug are
> the init() method in InorderSimicsFeeder/SimicsTracer.hpp and
> doInitialize() in InorderSimicsFeeder/InorderSimicsFeederImpl.cpp. If
> there is a problem with hooking up to Simics, it will be in one of these
> two places.
>
> >
> >
> > 52 <ExecuteImpl.cpp:341> (execute[2]) {250546}- EX received Reply
> > MemoryMessage[Store Reply]: Addr:0xp:00ff3ffac Size:8 Core: 0
> >
> > 53 <ExecuteImpl.cpp:1287> (execute[2]) {250548}- EX Issuing memory
> request:
> > MemoryMessage[Store Request]: Addr:0xp:00ff3ffa8 Size:8 Core: 0
> >
> >
> >
> > Again, I fear there is something wrong. The reason is, based on the debug
> > output only execute[2] is doing work, however, all MemoroyMessage shows
> > 'core 0'. Is this something abnormal?
>
> Core 0 does not neccessarily indicate anything is wrong, because the core
> field is not set in the execute component, and would still be zero at the
> point this debug message is generated.
>
>
> > :-) sorry for the whole bunch of questions :-).
> >
> > Thanks very much
> >
> > Shan
> >
> >
>
> Regards,
> -Tom Wenisch
> Computer Architecture Lab
> Carnegie Mellon University
>
>
From penglu01 at hotmail.com Wed Nov 2 18:58:24 2005
From: penglu01 at hotmail.com (lu peng)
List-Post: [email protected]
Date: Wed Nov 2 19:12:15 2005
Subject: [Simflex] Debugging message
Message-ID: <[email protected]>
An HTML attachment was scrubbed...
URL:
http://sos.ece.cmu.edu/pipermail/simflex/attachments/20051102/d4f007a3/attachment.html
From twenisch at ece.cmu.edu Thu Nov 3 01:12:24 2005
From: twenisch at ece.cmu.edu (Thomas Wenisch)
List-Post: [email protected]
Date: Thu Nov 3 01:11:42 2005
Subject: [Simflex] Re: Debugging message
In-Reply-To: <[email protected]>
References: <[email protected]>
Message-ID: <pine.lnx.4.53l-ece.cmu.edu.0511030107380.32...@dalmore.ece.cmu.edu>
Hi Lu,
Trace severity debug output is compiled into your simulator by default
(Iface, Trace, Dev, and Crit are enabled at compiled time by default, Verb
and VVerb are not) and should appear when you set the debug system to
veryverbose. Are you seeing any other debug output (e.g., all the stuff
that Flexus generates by default when starting the simulation)? Are you
certain your DBG_ statement is executing? Does it appear if you change it
to a printf/cout? Does the message appear if you change it from Trace to
Dev severity?
Regards,
-Tom Wenisch
On Wed, 2 Nov 2005, lu peng wrote:
> Hi Tom,
>
> How are you? I am trying to output some debug message by DBG_(Trace, (...))
> statement. However, I can't find my output message in the debug.out even I
> use very
> verbose mode by set 'flexus.set-debug veryverbose". Could you tell the reason?
>
> Thanks a lot,
> Lu
>
>
From twenisch at ece.cmu.edu Sat Nov 5 18:43:19 2005
From: twenisch at ece.cmu.edu (Thomas Wenisch)
List-Post: [email protected]
Date: Sat Nov 5 18:42:51 2005
Subject: FW: [Simflex] x86 CMP configuration
In-Reply-To: <[email protected]>
References: <[email protected]>
Message-ID: <pine.lnx.4.53l-ece.cmu.edu.0511051826320.29...@dalmore.ece.cmu.edu>
Hi Shan,
I have investigated the traces you sent.
Throughout the entire trace you sent, all 4 CPUs are completely idle.
That is, the only instruction that gets passed from Simics to Flexus is a
HLT instruction. Linux uses the HLT instruction in the idle loop. Flexus
dutifully executes the HLT instructions (at IPC 1), and accumulates
"stall" cycles to indicate the number of cycles a CPU is halted.
Flexus then tries to return this stall time to Simics, indicating how mich
time Flexus believes has elasped. However, Simics does not permit stalls
to be returned for a HLT instruction. So, Flexus accumulates the stalls,
waiting to apply them later, once Simics allows it.
The trace you sent ends while this halting is still going on. However, I
can speculate as to what happens next.
Eventually, a timer interrupt comes along on one (or all) of the CPUs, and
they wake up to do something interesting. In the mean time, Flexus has
accumulated potentially hundreds of thousands of stall cycles. As soon as
the first instruction in the timer interrupt is fetched, Flexus applies
all the accumulated stall time. This instruction is delayed a *very long*
time. So long, that Flexus figures that something must be wrong and the
system is deadlocked. Hence, the assertion.
The problem here is the location of your Simics checkpoint. Your
checkpoint is in the OS idle loop, and potentially very far away from
where the useful work is that you want to measure. Even if this did not
lead to the untimely death of the simulator, starting simulations from
this checkpoint would still be bad, because you may be waiting for 10's of
minutes of simulation for the CPUs to get out of the idle loop.
You want to take a Simics checkpoint just before the part of your
application that you are actually interested in measuring. In
applications that you write (i.e., where you have the source code), you
can do this by adding "magic breakpoint" instructions to your code, which
you can use to stop simulation in Simics.
I suggest you add a "startup sequence" to your application that waits at a
barrier until all four CPUs are fired up (i.e., spawns threads,
initializes variables, etc.), and then triggers a magic breakpoint. You
can wait for this breakpoint in Simics, and then save the Simics
checkpoint you will use to start your Flexus simulations.
Look up magic breakpoints in the Simics documentation for more info on how
to do this.
Regards,
-Tom Wenisch
On Thu, 3 Nov 2005, shan wrote:
> It seems that all four cores are stuck with the same first instruction. They
> have never moved on.
< snip history, attachments >
