Re: [gem5-users] DRAM memory access latency
Hi Prathap, The crossbar has a given throughput and latency, but I think the default is 1 Ghz, and 128-bit (16 byte) wide data patch, with a single cycle overhead (64 bytes every 5 ns). If that is indeed a limit, then you can always increase the crossbar clock or width. Note that if you have very bursty reads for example you can easily build up a backlog. If the crossbar is not the issue, then perhaps the master port that is the destination for the response is causing the problem? Andreas From: Prathap Kolakkampadath mailto:kvprat...@gmail.com>> Date: Tuesday, 11 November 2014 03:41 To: Andreas Hansson mailto:andreas.hans...@arm.com>> Cc: gem5 users mailing list mailto:gem5-users@gem5.org>> Subject: Re: [gem5-users] DRAM memory access latency Hello Andreas, >> waiting in the port until the crossbar can accept it Is this because of Bus Contention? In that case, is there a way to reduce this latency by changing any parameters in gem5? Thanks, Prathap On Thu, Nov 6, 2014 at 2:30 PM, Andreas Hansson mailto:andreas.hans...@arm.com>> wrote: Hi Prathap, I suspect the answer to the mysterious 50 ns is due to the responses being sent back using a so called “queued port” in gem5. Thus, from the memory controller’s point of view the packet is all done, but is now waiting in the port until the crossbar can accept it. This queue can hold a number of packets if there has been a burst of responses that are trickling through the crossbar on their way back. You can always run with some debug flags to verify this (XBar, DRAM, PacketQueue etc). Coincidentally I have been working on a patch to remove this “invisible” queue and should hopefully have this on the review board shortly. Andreas From: Prathap Kolakkampadath mailto:kvprat...@gmail.com>> Date: Thursday, November 6, 2014 at 5:47 PM To: Andreas Hansson mailto:andreas.hans...@arm.com>> Cc: gem5 users mailing list mailto:gem5-users@gem5.org>> Subject: Re: [gem5-users] DRAM memory access latency Hello Andreas, Thanks for your reply. Ok. I got that the memory access latency indeed includes the queueing latency. And for the read/write request that miss the buffer has a static latency of Static frontend latency + Static backend latency. To summarize, the test i run is a latency benchmark which is a pointer chasing test(only one request at a time) , generate reads to a specific DRAM bank (Bank partitioned).This test is running on cpu0 of 4 cpu arm_detailed running at 1GHZ frequency with 1MB shared L2 cache and single channel LPDDR3 x32 DRAM. The bank used by cpu0 is not shared between other cpu's. Test statistics: system.mem_ctrls.avgQLat 43816.35 # Average queueing delay per DRAM burst system.mem_ctrls.avgBusLat5000.00 # Average bus latency per DRAM burst system.mem_ctrls.avgMemAccLat63816.35 # Average memory access latency per DRAM burst system.mem_ctrls.avgRdQLen 2.00 # Average read queue length when enqueuing system.mem_ctrls.avgGap 136814.25 # Average gap between requests system.l2.ReadReq_avg_miss_latency::switch_cpus0.data 114767.654811 # average ReadReq miss latency Based on above test statistics: avgMemAccLat is 63ns, which i presume the sum of tRP(15ns)+tRCD(15ns) +tCL(15ns)+static latency(20ns). Is this breakup correct? However the l2.ReadReq_avg_miss_atency is 114ns which is ~50 ns more than the avgMemAccLat. I couldn't figure out the components contributing to this 50ns latency. Your thoughts on this is much appreciated. Regards, Prathap On Thu, Nov 6, 2014 at 3:03 AM, Andreas Hansson mailto:andreas.hans...@arm.com>> wrote: Hi Prathap, The avgMemAccLat does indeed include any queueing latency. For the precise components included in the various latencies I would suggest checking the source code. Note that the controller is not just accounting for the static (and dynamic) DRAM latency, but also the static controller pipeline latency (and dynamic queueing latency). The controller static latency is two parameters that are by default also adding a few 10’s of nanoseconds. Let me know if you need more help breaking out the various components. Andreas From: Prathap Kolakkampadath via gem5-users mailto:gem5-users@gem5.org>> Reply-To: Prathap Kolakkampadath mailto:kvprat...@gmail.com>>, gem5 users mailing list mailto:gem5-users@gem5.org>> Date: Wednesday, 5 November 2014 05:36 To: Tao Zhang mailto:tao.zhang.0...@gmail.com>>, gem5 users mailing list mailto:gem5-users@gem5.org>>, Amin Farmahini mailto:amin...@gmail.com>> Subject: Re: [gem5-users] DRAM memory access latency Hi Tao,Amin, According to gem5 source, MemAccLat is the time difference between the packet enters i
Re: [gem5-users] DRAM memory access latency
Hello Andreas, >> waiting in the port until the crossbar can accept it Is this because of Bus Contention? In that case, is there a way to reduce this latency by changing any parameters in gem5? Thanks, Prathap On Thu, Nov 6, 2014 at 2:30 PM, Andreas Hansson wrote: > Hi Prathap, > > I suspect the answer to the mysterious 50 ns is due to the responses > being sent back using a so called “queued port” in gem5. Thus, from the > memory controller’s point of view the packet is all done, but is now > waiting in the port until the crossbar can accept it. This queue can hold a > number of packets if there has been a burst of responses that are trickling > through the crossbar on their way back. > > You can always run with some debug flags to verify this (XBar, DRAM, > PacketQueue etc). > > Coincidentally I have been working on a patch to remove this “invisible” > queue and should hopefully have this on the review board shortly. > > Andreas > > From: Prathap Kolakkampadath > Date: Thursday, November 6, 2014 at 5:47 PM > To: Andreas Hansson > Cc: gem5 users mailing list > > Subject: Re: [gem5-users] DRAM memory access latency > > Hello Andreas, > > Thanks for your reply. > > > Ok. I got that the memory access latency indeed includes the queueing > latency. And for the read/write request that miss the buffer has a static > latency of Static frontend latency + Static backend latency. > > > To summarize, the test i run is a latency benchmark which is a pointer > chasing test(only one request at a time) , generate reads to a specific > DRAM bank (Bank partitioned).This test is running on cpu0 of 4 cpu > arm_detailed running at 1GHZ frequency with 1MB shared L2 cache and single > channel LPDDR3 x32 DRAM. The bank used by cpu0 is not shared between other > cpu's. > > Test statistics: > > system.mem_ctrls.avgQLat >43816.35 # Average queueing delay per > DRAM burst > system.mem_ctrls.avgBusLat > 5000.00 # Average bus latency per DRAM burst > system.mem_ctrls.avgMemAccLat > 63816.35 # Average memory access latency per DRAM > burst > system.mem_ctrls.avgRdQLen > 2.00 # Average read queue length when enqueuing > system.mem_ctrls.avgGap > 136814.25 # Average gap between requests > system.l2.ReadReq_avg_miss_latency::switch_cpus0.data > 114767.654811 # average ReadReq miss latency > > Based on above test statistics: > > avgMemAccLat is 63ns, which i presume the sum of tRP(15ns)+tRCD(15ns) > +tCL(15ns)+static latency(20ns). > Is this breakup correct? > > However the l2.ReadReq_avg_miss_atency is 114ns which is ~50 ns more > than the avgMemAccLat. I couldn't figure out the components contributing to > this 50ns latency. Your thoughts on this is much appreciated. > > Regards, > Prathap > > > > > On Thu, Nov 6, 2014 at 3:03 AM, Andreas Hansson > wrote: > >> Hi Prathap, >> >> The avgMemAccLat does indeed include any queueing latency. For the >> precise components included in the various latencies I would suggest >> checking the source code. >> >> Note that the controller is not just accounting for the static (and >> dynamic) DRAM latency, but also the static controller pipeline latency (and >> dynamic queueing latency). The controller static latency is two parameters >> that are by default also adding a few 10’s of nanoseconds. >> >> Let me know if you need more help breaking out the various components. >> >> Andreas >> >> From: Prathap Kolakkampadath via gem5-users >> Reply-To: Prathap Kolakkampadath , gem5 users >> mailing list >> Date: Wednesday, 5 November 2014 05:36 >> To: Tao Zhang , gem5 users mailing list < >> gem5-users@gem5.org>, Amin Farmahini >> Subject: Re: [gem5-users] DRAM memory access latency >> >> Hi Tao,Amin, >> >> According to gem5 source, MemAccLat is the time difference between the >> packet enters in the controller and packet leaves the controller. I presume >> this added with BusLatency and static backend latency should match with >> system.l2.ReadReq_avg_miss_latency. However i see a difference of approx >> 50ns. >> >> >> As mentioned above if MemAccLat is the time a packet spends in memory >> controller, then it should include the queuing latency too. In that case >> the value of avgQLat looks suspicious. Is the avgQlat part of >> avgMemAccLat? >> >> Thanks, >> Prathap >> >> >> >> On Tue, Nov 4, 2014 at 3
Re: [gem5-users] DRAM memory access latency
Thanks for your reply. I will try to verify this and also get back to you with results once i run with your patch. Regards, Prathap On Thu, Nov 6, 2014 at 2:30 PM, Andreas Hansson wrote: > Hi Prathap, > > I suspect the answer to the mysterious 50 ns is due to the responses > being sent back using a so called “queued port” in gem5. Thus, from the > memory controller’s point of view the packet is all done, but is now > waiting in the port until the crossbar can accept it. This queue can hold a > number of packets if there has been a burst of responses that are trickling > through the crossbar on their way back. > > You can always run with some debug flags to verify this (XBar, DRAM, > PacketQueue etc). > > Coincidentally I have been working on a patch to remove this “invisible” > queue and should hopefully have this on the review board shortly. > > Andreas > > From: Prathap Kolakkampadath > Date: Thursday, November 6, 2014 at 5:47 PM > To: Andreas Hansson > Cc: gem5 users mailing list > > Subject: Re: [gem5-users] DRAM memory access latency > > Hello Andreas, > > Thanks for your reply. > > > Ok. I got that the memory access latency indeed includes the queueing > latency. And for the read/write request that miss the buffer has a static > latency of Static frontend latency + Static backend latency. > > > To summarize, the test i run is a latency benchmark which is a pointer > chasing test(only one request at a time) , generate reads to a specific > DRAM bank (Bank partitioned).This test is running on cpu0 of 4 cpu > arm_detailed running at 1GHZ frequency with 1MB shared L2 cache and single > channel LPDDR3 x32 DRAM. The bank used by cpu0 is not shared between other > cpu's. > > Test statistics: > > system.mem_ctrls.avgQLat >43816.35 # Average queueing delay per > DRAM burst > system.mem_ctrls.avgBusLat > 5000.00 # Average bus latency per DRAM burst > system.mem_ctrls.avgMemAccLat > 63816.35 # Average memory access latency per DRAM > burst > system.mem_ctrls.avgRdQLen > 2.00 # Average read queue length when enqueuing > system.mem_ctrls.avgGap > 136814.25 # Average gap between requests > system.l2.ReadReq_avg_miss_latency::switch_cpus0.data > 114767.654811 # average ReadReq miss latency > > Based on above test statistics: > > avgMemAccLat is 63ns, which i presume the sum of tRP(15ns)+tRCD(15ns) > +tCL(15ns)+static latency(20ns). > Is this breakup correct? > > However the l2.ReadReq_avg_miss_atency is 114ns which is ~50 ns more > than the avgMemAccLat. I couldn't figure out the components contributing to > this 50ns latency. Your thoughts on this is much appreciated. > > Regards, > Prathap > > > > > On Thu, Nov 6, 2014 at 3:03 AM, Andreas Hansson > wrote: > >> Hi Prathap, >> >> The avgMemAccLat does indeed include any queueing latency. For the >> precise components included in the various latencies I would suggest >> checking the source code. >> >> Note that the controller is not just accounting for the static (and >> dynamic) DRAM latency, but also the static controller pipeline latency (and >> dynamic queueing latency). The controller static latency is two parameters >> that are by default also adding a few 10’s of nanoseconds. >> >> Let me know if you need more help breaking out the various components. >> >> Andreas >> >> From: Prathap Kolakkampadath via gem5-users >> Reply-To: Prathap Kolakkampadath , gem5 users >> mailing list >> Date: Wednesday, 5 November 2014 05:36 >> To: Tao Zhang , gem5 users mailing list < >> gem5-users@gem5.org>, Amin Farmahini >> Subject: Re: [gem5-users] DRAM memory access latency >> >> Hi Tao,Amin, >> >> According to gem5 source, MemAccLat is the time difference between the >> packet enters in the controller and packet leaves the controller. I presume >> this added with BusLatency and static backend latency should match with >> system.l2.ReadReq_avg_miss_latency. However i see a difference of approx >> 50ns. >> >> >> As mentioned above if MemAccLat is the time a packet spends in memory >> controller, then it should include the queuing latency too. In that case >> the value of avgQLat looks suspicious. Is the avgQlat part of >> avgMemAccLat? >> >> Thanks, >> Prathap >> >> >> >> On Tue, Nov 4, 2014 at 3:11 PM, Tao Zhang >> wrote: >> >>> From the stats, I'd like to use
Re: [gem5-users] DRAM memory access latency
Hi Prathap, I suspect the answer to the mysterious 50 ns is due to the responses being sent back using a so called “queued port” in gem5. Thus, from the memory controller’s point of view the packet is all done, but is now waiting in the port until the crossbar can accept it. This queue can hold a number of packets if there has been a burst of responses that are trickling through the crossbar on their way back. You can always run with some debug flags to verify this (XBar, DRAM, PacketQueue etc). Coincidentally I have been working on a patch to remove this “invisible” queue and should hopefully have this on the review board shortly. Andreas From: Prathap Kolakkampadath mailto:kvprat...@gmail.com>> Date: Thursday, November 6, 2014 at 5:47 PM To: Andreas Hansson mailto:andreas.hans...@arm.com>> Cc: gem5 users mailing list mailto:gem5-users@gem5.org>> Subject: Re: [gem5-users] DRAM memory access latency Hello Andreas, Thanks for your reply. Ok. I got that the memory access latency indeed includes the queueing latency. And for the read/write request that miss the buffer has a static latency of Static frontend latency + Static backend latency. To summarize, the test i run is a latency benchmark which is a pointer chasing test(only one request at a time) , generate reads to a specific DRAM bank (Bank partitioned).This test is running on cpu0 of 4 cpu arm_detailed running at 1GHZ frequency with 1MB shared L2 cache and single channel LPDDR3 x32 DRAM. The bank used by cpu0 is not shared between other cpu's. Test statistics: system.mem_ctrls.avgQLat 43816.35 # Average queueing delay per DRAM burst system.mem_ctrls.avgBusLat5000.00 # Average bus latency per DRAM burst system.mem_ctrls.avgMemAccLat63816.35 # Average memory access latency per DRAM burst system.mem_ctrls.avgRdQLen 2.00 # Average read queue length when enqueuing system.mem_ctrls.avgGap 136814.25 # Average gap between requests system.l2.ReadReq_avg_miss_latency::switch_cpus0.data 114767.654811 # average ReadReq miss latency Based on above test statistics: avgMemAccLat is 63ns, which i presume the sum of tRP(15ns)+tRCD(15ns) +tCL(15ns)+static latency(20ns). Is this breakup correct? However the l2.ReadReq_avg_miss_atency is 114ns which is ~50 ns more than the avgMemAccLat. I couldn't figure out the components contributing to this 50ns latency. Your thoughts on this is much appreciated. Regards, Prathap On Thu, Nov 6, 2014 at 3:03 AM, Andreas Hansson mailto:andreas.hans...@arm.com>> wrote: Hi Prathap, The avgMemAccLat does indeed include any queueing latency. For the precise components included in the various latencies I would suggest checking the source code. Note that the controller is not just accounting for the static (and dynamic) DRAM latency, but also the static controller pipeline latency (and dynamic queueing latency). The controller static latency is two parameters that are by default also adding a few 10’s of nanoseconds. Let me know if you need more help breaking out the various components. Andreas From: Prathap Kolakkampadath via gem5-users mailto:gem5-users@gem5.org>> Reply-To: Prathap Kolakkampadath mailto:kvprat...@gmail.com>>, gem5 users mailing list mailto:gem5-users@gem5.org>> Date: Wednesday, 5 November 2014 05:36 To: Tao Zhang mailto:tao.zhang.0...@gmail.com>>, gem5 users mailing list mailto:gem5-users@gem5.org>>, Amin Farmahini mailto:amin...@gmail.com>> Subject: Re: [gem5-users] DRAM memory access latency Hi Tao,Amin, According to gem5 source, MemAccLat is the time difference between the packet enters in the controller and packet leaves the controller. I presume this added with BusLatency and static backend latency should match with system.l2.ReadReq_avg_miss_latency. However i see a difference of approx 50ns. As mentioned above if MemAccLat is the time a packet spends in memory controller, then it should include the queuing latency too. In that case the value of avgQLat looks suspicious. Is the avgQlat part of avgMemAccLat? Thanks, Prathap On Tue, Nov 4, 2014 at 3:11 PM, Tao Zhang mailto:tao.zhang.0...@gmail.com>> wrote: >From the stats, I'd like to use system.mem_ctrls.avgMemAccLat as the overall >average memory latency. It is 63.816ns, which is very close to 60ns as you >calculated. I guess the extra 3.816ns is due to the refresh penalty. -Tao On Tue, Nov 4, 2014 at 12:10 PM, Prathap Kolakkampadath mailto:kvprat...@gmail.com>> wrote: Hi Toa, Amin, Thanks for your reply. To discard interbank interference and queueing delay, i have partitioned the banks so that the latency benchmark has exclusive access to a bank. Also latenc
Re: [gem5-users] DRAM memory access latency
Hello Andreas, Thanks for your reply. Ok. I got that the memory access latency indeed includes the queueing latency. And for the read/write request that miss the buffer has a static latency of Static frontend latency + Static backend latency. To summarize, the test i run is a latency benchmark which is a pointer chasing test(only one request at a time) , generate reads to a specific DRAM bank (Bank partitioned).This test is running on cpu0 of 4 cpu arm_detailed running at 1GHZ frequency with 1MB shared L2 cache and single channel LPDDR3 x32 DRAM. The bank used by cpu0 is not shared between other cpu's. Test statistics: system.mem_ctrls.avgQLat 43816.35 # Average queueing delay per DRAM burst system.mem_ctrls.avgBusLat5000.00 # Average bus latency per DRAM burst system.mem_ctrls.avgMemAccLat63816.35 # Average memory access latency per DRAM burst system.mem_ctrls.avgRdQLen 2.00 # Average read queue length when enqueuing system.mem_ctrls.avgGap 136814.25 # Average gap between requests system.l2.ReadReq_avg_miss_latency::switch_cpus0.data 114767.654811 # average ReadReq miss latency Based on above test statistics: avgMemAccLat is 63ns, which i presume the sum of tRP(15ns)+tRCD(15ns) +tCL(15ns)+static latency(20ns). Is this breakup correct? However the l2.ReadReq_avg_miss_atency is 114ns which is ~50 ns more than the avgMemAccLat. I couldn't figure out the components contributing to this 50ns latency. Your thoughts on this is much appreciated. Regards, Prathap On Thu, Nov 6, 2014 at 3:03 AM, Andreas Hansson wrote: > Hi Prathap, > > The avgMemAccLat does indeed include any queueing latency. For the > precise components included in the various latencies I would suggest > checking the source code. > > Note that the controller is not just accounting for the static (and > dynamic) DRAM latency, but also the static controller pipeline latency (and > dynamic queueing latency). The controller static latency is two parameters > that are by default also adding a few 10’s of nanoseconds. > > Let me know if you need more help breaking out the various components. > > Andreas > > From: Prathap Kolakkampadath via gem5-users > Reply-To: Prathap Kolakkampadath , gem5 users > mailing list > Date: Wednesday, 5 November 2014 05:36 > To: Tao Zhang , gem5 users mailing list < > gem5-users@gem5.org>, Amin Farmahini > Subject: Re: [gem5-users] DRAM memory access latency > > Hi Tao,Amin, > > According to gem5 source, MemAccLat is the time difference between the > packet enters in the controller and packet leaves the controller. I presume > this added with BusLatency and static backend latency should match with > system.l2.ReadReq_avg_miss_latency. However i see a difference of approx > 50ns. > > > As mentioned above if MemAccLat is the time a packet spends in memory > controller, then it should include the queuing latency too. In that case > the value of avgQLat looks suspicious. Is the avgQlat part of > avgMemAccLat? > > Thanks, > Prathap > > > > On Tue, Nov 4, 2014 at 3:11 PM, Tao Zhang > wrote: > >> From the stats, I'd like to use system.mem_ctrls.avgMemAccLat as the >> overall average memory latency. It is 63.816ns, which is very close to 60ns >> as you calculated. I guess the extra 3.816ns is due to the refresh penalty. >> >> -Tao >> >> On Tue, Nov 4, 2014 at 12:10 PM, Prathap Kolakkampadath < >> kvprat...@gmail.com> wrote: >> >>> Hi Toa, Amin, >>> >>> >>> Thanks for your reply. >>> >>> To discard interbank interference and queueing delay, i have >>> partitioned the banks so that the latency benchmark has exclusive access to >>> a bank. Also latency benchmark is a pointer chasing benchmark, which will >>> generate a single read request at a time. >>> >>> >>> stats.txt says this: >>> >>> system.mem_ctrls.avgQLat >>> 43816.35 # Average queueing delay per DRAM burst >>> system.mem_ctrls.avgBusLat >>> 5000.00 # Average bus latency per DRAM burst >>> system.mem_ctrls.avgMemAccLat >>> 63816.35 # Average memory access latency per DRAM >>> burst >>> system.mem_ctrls.avgRdQLen >>> 2.00 # Average read queue length when enqueuing >>> system.mem_ctrls.avgGap >>> 136814.25 # Average gap between requests >>> system.l2.ReadReq_avg_miss_latency::switch_cpus0.data >>> 114767.654811 # average ReadReq m
Re: [gem5-users] DRAM memory access latency
Hi Prathap, The avgMemAccLat does indeed include any queueing latency. For the precise components included in the various latencies I would suggest checking the source code. Note that the controller is not just accounting for the static (and dynamic) DRAM latency, but also the static controller pipeline latency (and dynamic queueing latency). The controller static latency is two parameters that are by default also adding a few 10’s of nanoseconds. Let me know if you need more help breaking out the various components. Andreas From: Prathap Kolakkampadath via gem5-users mailto:gem5-users@gem5.org>> Reply-To: Prathap Kolakkampadath mailto:kvprat...@gmail.com>>, gem5 users mailing list mailto:gem5-users@gem5.org>> Date: Wednesday, 5 November 2014 05:36 To: Tao Zhang mailto:tao.zhang.0...@gmail.com>>, gem5 users mailing list mailto:gem5-users@gem5.org>>, Amin Farmahini mailto:amin...@gmail.com>> Subject: Re: [gem5-users] DRAM memory access latency Hi Tao,Amin, According to gem5 source, MemAccLat is the time difference between the packet enters in the controller and packet leaves the controller. I presume this added with BusLatency and static backend latency should match with system.l2.ReadReq_avg_miss_latency. However i see a difference of approx 50ns. As mentioned above if MemAccLat is the time a packet spends in memory controller, then it should include the queuing latency too. In that case the value of avgQLat looks suspicious. Is the avgQlat part of avgMemAccLat? Thanks, Prathap On Tue, Nov 4, 2014 at 3:11 PM, Tao Zhang mailto:tao.zhang.0...@gmail.com>> wrote: >From the stats, I'd like to use system.mem_ctrls.avgMemAccLat as the overall >average memory latency. It is 63.816ns, which is very close to 60ns as you >calculated. I guess the extra 3.816ns is due to the refresh penalty. -Tao On Tue, Nov 4, 2014 at 12:10 PM, Prathap Kolakkampadath mailto:kvprat...@gmail.com>> wrote: Hi Toa, Amin, Thanks for your reply. To discard interbank interference and queueing delay, i have partitioned the banks so that the latency benchmark has exclusive access to a bank. Also latency benchmark is a pointer chasing benchmark, which will generate a single read request at a time. stats.txt says this: system.mem_ctrls.avgQLat 43816.35 # Average queueing delay per DRAM burst system.mem_ctrls.avgBusLat5000.00 # Average bus latency per DRAM burst system.mem_ctrls.avgMemAccLat63816.35 # Average memory access latency per DRAM burst system.mem_ctrls.avgRdQLen 2.00 # Average read queue length when enqueuing system.mem_ctrls.avgGap 136814.25 # Average gap between requests system.l2.ReadReq_avg_miss_latency::switch_cpus0.data 114767.654811 # average ReadReq miss latency The average Gap between requests is equal to the L2 latency + DRAM Latency for this test. Also avgRdQLen is 2 because cache line size is 64 and DRAM interface is x32. Is the final latency sum of avgQLat + avgBusLat + avgMemAccLat ? Also when avgRdQLen is 2, i am not sure what amounts to high queueing latency? Regards, Prathap On Tue, Nov 4, 2014 at 1:38 PM, Amin Farmahini mailto:amin...@gmail.com>> wrote: Prathap, You are probably missing DRAM queuing latency (major reason) and other on-chip latencies (such as bus latency) if any. Thanks, Amin On Tue, Nov 4, 2014 at 1:28 PM, Prathap Kolakkampadath via gem5-users mailto:gem5-users@gem5.org>> wrote: Hello Users, I am measuring DRAM worst case memory access latency(tRP+tRCD +tCL+tBURST) using a latency benchmark on arm_detailed(1Ghz) with 1MB shared L2 cache and LPDDR3 x32 DRAM. According to DRAM timing parameters, tRP = '15ns, tRCD = '15ns', tCL = '15ns', tBURST = '5ns'. Latency measured by the benchmark on cache hit is 22 ns and on cache miss is 132ns. Which means DRAM memory access latency ~ 110ns. However according to calculation it should be tRP+tRCD+tCL+tBurst+static_backend_latency(10ns) = 60ns. The latency what i observe is almost 50ns higher than what it is supposed to be. Is there anything which I am missing? Do any one know what else could add to the DRAM memory access latency? Thanks, Prathap ___ gem5-users mailing list gem5-users@gem5.org<mailto:gem5-users@gem5.org> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users -- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank yo
Re: [gem5-users] DRAM memory access latency
Hi Tao,Amin, According to gem5 source, MemAccLat is the time difference between the packet enters in the controller and packet leaves the controller. I presume this added with BusLatency and static backend latency should match with system.l2.ReadReq_avg_miss_latency. However i see a difference of approx 50ns. As mentioned above if MemAccLat is the time a packet spends in memory controller, then it should include the queuing latency too. In that case the value of avgQLat looks suspicious. Is the avgQlat part of avgMemAccLat? Thanks, Prathap On Tue, Nov 4, 2014 at 3:11 PM, Tao Zhang wrote: > From the stats, I'd like to use system.mem_ctrls.avgMemAccLat as the > overall average memory latency. It is 63.816ns, which is very close to 60ns > as you calculated. I guess the extra 3.816ns is due to the refresh penalty. > > -Tao > > On Tue, Nov 4, 2014 at 12:10 PM, Prathap Kolakkampadath < > kvprat...@gmail.com> wrote: > >> Hi Toa, Amin, >> >> >> Thanks for your reply. >> >> To discard interbank interference and queueing delay, i have partitioned >> the banks so that the latency benchmark has exclusive access to a bank. >> Also latency benchmark is a pointer chasing benchmark, which will generate >> a single read request at a time. >> >> >> stats.txt says this: >> >> system.mem_ctrls.avgQLat >> 43816.35 # Average queueing delay per DRAM burst >> system.mem_ctrls.avgBusLat >> 5000.00 # Average bus latency per DRAM burst >> system.mem_ctrls.avgMemAccLat >> 63816.35 # Average memory access latency per DRAM >> burst >> system.mem_ctrls.avgRdQLen >> 2.00 # Average read queue length when enqueuing >> system.mem_ctrls.avgGap >> 136814.25 # Average gap between requests >> system.l2.ReadReq_avg_miss_latency::switch_cpus0.data >> 114767.654811 # average ReadReq miss latency >> >> The average Gap between requests is equal to the L2 latency + DRAM >> Latency for this test. Also avgRdQLen is 2 because cache line size is 64 >> and DRAM interface is x32. >> >> Is the final latency sum of avgQLat + avgBusLat + avgMemAccLat ? >> Also when avgRdQLen is 2, i am not sure what amounts to high queueing >> latency? >> >> Regards, >> Prathap >> >> >> >> On Tue, Nov 4, 2014 at 1:38 PM, Amin Farmahini wrote: >> >>> Prathap, >>> >>> You are probably missing DRAM queuing latency (major reason) and other >>> on-chip latencies (such as bus latency) if any. >>> >>> Thanks, >>> Amin >>> >>> On Tue, Nov 4, 2014 at 1:28 PM, Prathap Kolakkampadath via gem5-users < >>> gem5-users@gem5.org> wrote: >>> Hello Users, I am measuring DRAM worst case memory access latency(tRP+tRCD +tCL+tBURST) using a latency benchmark on arm_detailed(1Ghz) with 1MB shared L2 cache and LPDDR3 x32 DRAM. According to DRAM timing parameters, tRP = '15ns, tRCD = '15ns', tCL = '15ns', tBURST = '5ns'. Latency measured by the benchmark on cache hit is 22 ns and on cache miss is 132ns. Which means DRAM memory access latency ~ 110ns. However according to calculation it should be tRP+tRCD+tCL+tBurst+static_backend_latency(10ns) = 60ns. The latency what i observe is almost 50ns higher than what it is supposed to be. Is there anything which I am missing? Do any one know what else could add to the DRAM memory access latency? Thanks, Prathap ___ gem5-users mailing list gem5-users@gem5.org http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>> >>> >> > ___ gem5-users mailing list gem5-users@gem5.org http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users
Re: [gem5-users] DRAM memory access latency
Hi Toa, Amin, Thanks for your reply. To discard interbank interference and queueing delay, i have partitioned the banks so that the latency benchmark has exclusive access to a bank. Also latency benchmark is a pointer chasing benchmark, which will generate a single read request at a time. stats.txt says this: system.mem_ctrls.avgQLat 43816.35 # Average queueing delay per DRAM burst system.mem_ctrls.avgBusLat5000.00 # Average bus latency per DRAM burst system.mem_ctrls.avgMemAccLat63816.35 # Average memory access latency per DRAM burst system.mem_ctrls.avgRdQLen 2.00 # Average read queue length when enqueuing system.mem_ctrls.avgGap 136814.25 # Average gap between requests system.l2.ReadReq_avg_miss_latency::switch_cpus0.data 114767.654811 # average ReadReq miss latency The average Gap between requests is equal to the L2 latency + DRAM Latency for this test. Also avgRdQLen is 2 because cache line size is 64 and DRAM interface is x32. Is the final latency sum of avgQLat + avgBusLat + avgMemAccLat ? Also when avgRdQLen is 2, i am not sure what amounts to high queueing latency? Regards, Prathap On Tue, Nov 4, 2014 at 1:38 PM, Amin Farmahini wrote: > Prathap, > > You are probably missing DRAM queuing latency (major reason) and other > on-chip latencies (such as bus latency) if any. > > Thanks, > Amin > > On Tue, Nov 4, 2014 at 1:28 PM, Prathap Kolakkampadath via gem5-users < > gem5-users@gem5.org> wrote: > >> Hello Users, >> >> I am measuring DRAM worst case memory access latency(tRP+tRCD >> +tCL+tBURST) using a latency benchmark on arm_detailed(1Ghz) with 1MB >> shared L2 cache and LPDDR3 x32 DRAM. >> >> According to DRAM timing parameters, tRP = '15ns, tRCD = '15ns', tCL = >> '15ns', tBURST = '5ns'. Latency measured by the benchmark on cache hit is >> 22 ns and on cache miss is 132ns. Which means DRAM memory access latency ~ >> 110ns. However according to calculation it should be >> tRP+tRCD+tCL+tBurst+static_backend_latency(10ns) = 60ns. >> >> >> The latency what i observe is almost 50ns higher than what it is supposed >> to be. Is there anything which I am missing? Do any one know what else >> could add to the DRAM memory access latency? >> >> Thanks, >> Prathap >> >> >> ___ >> gem5-users mailing list >> gem5-users@gem5.org >> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >> > > ___ gem5-users mailing list gem5-users@gem5.org http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users
Re: [gem5-users] DRAM memory access latency
Prathap, You are probably missing DRAM queuing latency (major reason) and other on-chip latencies (such as bus latency) if any. Thanks, Amin On Tue, Nov 4, 2014 at 1:28 PM, Prathap Kolakkampadath via gem5-users < gem5-users@gem5.org> wrote: > Hello Users, > > I am measuring DRAM worst case memory access latency(tRP+tRCD +tCL+tBURST) > using a latency benchmark on arm_detailed(1Ghz) with 1MB shared L2 cache > and LPDDR3 x32 DRAM. > > According to DRAM timing parameters, tRP = '15ns, tRCD = '15ns', tCL = > '15ns', tBURST = '5ns'. Latency measured by the benchmark on cache hit is > 22 ns and on cache miss is 132ns. Which means DRAM memory access latency ~ > 110ns. However according to calculation it should be > tRP+tRCD+tCL+tBurst+static_backend_latency(10ns) = 60ns. > > > The latency what i observe is almost 50ns higher than what it is supposed > to be. Is there anything which I am missing? Do any one know what else > could add to the DRAM memory access latency? > > Thanks, > Prathap > > > ___ > gem5-users mailing list > gem5-users@gem5.org > http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users > ___ gem5-users mailing list gem5-users@gem5.org http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users
Re: [gem5-users] DRAM memory access latency
Hi Parthap, the latency tRP+tRCD+tCL+tBURST is only the static access latency for DRAM. In memory subsystem, there is also dynamic queuing delay due to the memory controller scheduling (reordering) and resource availability (bank conflict, refresh, other timing constraints like tFAW, tRRD, tWTR) . Therefore, the average request latency is more than the theoretic static latency. This is why there are many papers talking about the request scheduling and refresh relax to make possible optimization. -Tao On Tue, Nov 4, 2014 at 11:28 AM, Prathap Kolakkampadath via gem5-users < gem5-users@gem5.org> wrote: > Hello Users, > > I am measuring DRAM worst case memory access latency(tRP+tRCD +tCL+tBURST) > using a latency benchmark on arm_detailed(1Ghz) with 1MB shared L2 cache > and LPDDR3 x32 DRAM. > > According to DRAM timing parameters, tRP = '15ns, tRCD = '15ns', tCL = > '15ns', tBURST = '5ns'. Latency measured by the benchmark on cache hit is > 22 ns and on cache miss is 132ns. Which means DRAM memory access latency ~ > 110ns. However according to calculation it should be > tRP+tRCD+tCL+tBurst+static_backend_latency(10ns) = 60ns. > > > The latency what i observe is almost 50ns higher than what it is supposed > to be. Is there anything which I am missing? Do any one know what else > could add to the DRAM memory access latency? > > Thanks, > Prathap > > > ___ > gem5-users mailing list > gem5-users@gem5.org > http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users > ___ gem5-users mailing list gem5-users@gem5.org http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users
[gem5-users] DRAM memory access latency
Hello Users, I am measuring DRAM worst case memory access latency(tRP+tRCD +tCL+tBURST) using a latency benchmark on arm_detailed(1Ghz) with 1MB shared L2 cache and LPDDR3 x32 DRAM. According to DRAM timing parameters, tRP = '15ns, tRCD = '15ns', tCL = '15ns', tBURST = '5ns'. Latency measured by the benchmark on cache hit is 22 ns and on cache miss is 132ns. Which means DRAM memory access latency ~ 110ns. However according to calculation it should be tRP+tRCD+tCL+tBurst+static_backend_latency(10ns) = 60ns. The latency what i observe is almost 50ns higher than what it is supposed to be. Is there anything which I am missing? Do any one know what else could add to the DRAM memory access latency? Thanks, Prathap ___ gem5-users mailing list gem5-users@gem5.org http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users
