Hi JD,
Please see my response below
On 5/12/2015 1:16 PM, Ou Jia Dong wrote:
> Dear Mahesh,
> Please see my comments. Thanks!
> [AVM] This shouldn't happen, if your application configured
> `sectionCreationAttributes.expirationTime` with the absolute time
> Example : sectionCreationAttributes.expirationTime =
> (SA_TIME_ONE_MINUTE +(time((time_t*)0) * 1000000000));
> _We tried this. However, there are two issues:_
> _1. The memory continue to grow for update operation_
_[AVM] _This should be observed even with less number section say 100 ,
so please confirm that this behavior is observable
irrelevant of number of section .
> _2. The update operation failed after 1 minute. _
> _It seems the created section is deleted after one minute and thus
> does not meet our requirement (created section stays forever until we
> instruct to delete it)_
_[AVM] _is this behaviour is observed in case of system running out of
memory ?
can you please provide the erro return value of Ckpt API
failed after 1 minute.
-AVM
> Br,
> JD
> -----Original Message-----
> From: A V Mahesh [mailto:[email protected]]
> Sent: Tuesday, May 12, 2015 3:32 PM
> To: Ou Jia Dong; Yao Cheng LIANG; [email protected];
> Jesse Y. AI; Xiao Dong WANG
> Cc: Mui Ka Ho Alex
> Subject: Re: [users] ckptnd memory leak(very serious)
> Hi Ted,
> On 5/12/2015 12:52 PM, Ou Jia Dong wrote:
> > Once we completed the creation phase, the system memory shall not grow for
> > the pure update operation.
> >
> > Is our expectation correct?
> [AVM] Yes expectation is correct.
> >
> > Currently, we observed that the system memory continue to grow rapidly even
> > for pure update operation until it use
> up all system memory and terminated..
> [AVM] This shouldn't happen, if your application configured
> `sectionCreationAttributes.expirationTime` with the absolute time
> Example : sectionCreationAttributes.expirationTime =
> (SA_TIME_ONE_MINUTE +(time((time_t*)0) * 1000000000));
> We tried this. However, there are two issues:
> 1. The memory continue to grow for update operation
> 2. The update operation failed after 1 minute.
> It seems the created section is deleted after one minute and thus does
> not meet our requirement (created section stays forever until we
> instruct to delete it)
> I will run/check your application and will get back to you.
> -AVM
> On 5/12/2015 12:52 PM, Ou Jia Dong wrote:
> > Dear Mahesh,
> >
> > Thank you very much for your prompt help. We still have problem. Please
> > allow me to explain our scenario.
> >
> > Our design has the following requirement.
> >
> > Create certain number of sections (for example, 50000 sections). We need
> > to keep it forever.
> > After the sections created, we will continue to perform "update" operation
> > on it.
> >
> > What we do expect is the system memory usage will continue to grow during
> > section creation phase. Once we completed
> the creation phase, the system memory shall not grow for the pure
> update operation.
> >
> > Is our expectation correct? If not, is there any way to accomplish this?
> >
> > Currently, we observed that the system memory continue to grow rapidly even
> > for pure update operation until it use
> up all system memory and terminated..
> >
> >
> > Thanks!
> >
> > Br,
> > JD
> >
> >
> >
> > -----Original Message-----
> > From: Yao Cheng LIANG
> > Sent: Tuesday, May 12, 2015 3:08 PM
> > To: Yao Cheng LIANG; 'A V Mahesh';
> >[email protected]
> <mailto:[email protected]>; Jesse Y. AI; Xiao Dong WANG
> > Cc: Mui Ka Ho Alex; Ou Jia Dong
> > Subject: RE: [users] ckptnd memory leak(very serious)
> >
> >
> >
> > -----Original Message-----
> > From: Yao Cheng LIANG [mailto:[email protected]]
> > Sent: Tuesday, May 12, 2015 3:03 PM
> > To: 'A V Mahesh';[email protected]
> <mailto:[email protected]>
> > Subject: Re: [users] ckptnd memory leak(very serious)
> >
> > If you run my program, you will find the issue. It is very easy to
> > reproduce.
> >
> > Ted
> >
> > -----Original Message-----
> > From: Yao Cheng LIANG
> > Sent: Tuesday, May 12, 2015 3:01 PM
> > To: 'A V Mahesh';[email protected]
> <mailto:[email protected]>
> > Cc: Yao Cheng LIANG
> > Subject: RE: [users] ckptnd memory leak(very serious)
> >
> > If you go through my testing code, you will find that I create some
> > sections and then overwrite it, but the system
> memory keeps growing until it uses up all system memory. I can not
> understand why it related to expiration timer.
> >
> > Ted
> >
> > -----Original Message-----
> > From: A V Mahesh [mailto:[email protected]]
> > Sent: Tuesday, May 12, 2015 2:53 PM
> > To: Yao Cheng LIANG;[email protected]
> <mailto:[email protected]>
> > Subject: Re: [users] ckptnd memory leak(very serious)
> >
> > Section have expiration time so we need to run a timer for each
> > section based on application configured
> > `sectionCreationAttributes.expirationTime` and this timer memory will
> > be freed once the section/checkpoint Deleted/Expires
> >
> > -AVM
> >
> > On 5/12/2015 12:17 PM, Yao Cheng LIANG wrote:
> >> So you mean this is normal? Could you explain to us why this happen?
> >>
> >> Ted
> >>
> >> -----Original Message-----
> >> From: A V Mahesh [mailto:[email protected]]
> >> Sent: Tuesday, May 12, 2015 2:31 PM
> >> To: Yao Cheng LIANG;[email protected]
> <mailto:[email protected]>
> >> Subject: Re: [users] ckptnd memory leak(very serious)
> >>
> >>
> >> Each section will be associated with a timer , till the
> >> Section/checkpoint Deleted/Expires the cpsnd memory will grow ( in
> >> your case 500,0000 sections ) ,
> >>
> >> -AVM
> >>
> >> On 5/12/2015 11:42 AM, Yao Cheng LIANG wrote:
> >>> Seems it does not help. After change retention time as you advised, the
> >>> memory usage continues growing as
> before.
> >>>
> >>> You can simple reproduce the issue by running the app and view output
> >>> from "top -p pid_of_ckptnd". The
> residential memory grows very quickly. And valgrind has reported
> memory leak.
> >>>
> >>> Ted
> >>>
> >>> -----Original Message-----
> >>> From: Yao Cheng LIANG
> >>> Sent: Tuesday, May 12, 2015 1:55 PM
> >>> To: 'A V Mahesh';[email protected]
> <mailto:[email protected]>
> >>> Cc: Yao Cheng LIANG
> >>> Subject: RE: [users] ckptnd memory leak(very serious)
> >>>
> >>> But we really want to keep these data even if there is no one open it. Is
> >>> there an issue?
> >>>
> >>> Ted
> >>>
> >>> -----Original Message-----
> >>> From: A V Mahesh [mailto:[email protected]]
> >>> Sent: Tuesday, May 12, 2015 1:41 PM
> >>> To:[email protected]
> <mailto:[email protected]>
> >>> Subject: Re: [users] ckptnd memory leak(very serious)
> >>>
> >>> Hi Ted,
> >>>
> >>> I got the problem .
> >>>
> >>> The CPND memory keeps increasing because ofimproper configuration values
> >>> of `expirationTime` of
> `SaCkptSectionCreationAttributes` by `cpsv_test_app` application.
> >>>
> >>> cpsv expects `expirationTime` the absolute time , but your
> >>> `cpsv_test_app` application is passing relative time , as an impact
> >>> the section leap times expiration time is being configured with
> >>> huge value , so they are NOT being kept in free pool , so we are
> >>> seeing additional increase in CPND memoryfor each run of
> >>> `ckpt_test` ( please check section
> >>> 3.4.3.2 SaCkptSectionCreationAttributesT ofCKPT Specification).
> >>>
> >>> so please change your application as below and verify the CPND memory
> >>> after `cpsv_test_app` runs and memory
> leak should not be observed.
> >>>
> >>> //sectionCreationAttributes.expirationTime = 3600000000000ll; /* One
> >>> Hour */
> >>>
> >>> sectionCreationAttributes.expirationTime = (SA_TIME_ONE_MINUTE +(
> >>> time((time_t*)0) * 1000000000));
> >>>
> >>> Their is a ticket for Ckpt sample correction :
> >>>https://sourceforge.net/p/opensaf/tickets/1270/
> >>>
> >>>
> >>> -AVM
> >>>
> >>>
> >>> On 5/12/2015 10:58 AM, Yao Cheng LIANG wrote:
> >>>> All,
> >>>>
> >>>> Please see attached test program. It is modified based on chpt_demo.
> >>>>
> >>>> Thanks.
> >>>>
> >>>> Ted
> >>>>
> >>>> -----Original Message-----
> >>>> From: Ingvar Bergström [mailto:[email protected]]
> >>>> Sent: Tuesday, May 12, 2015 1:04 PM
> >>>> To: Yao Cheng LIANG;[email protected]
> <mailto:[email protected]>
> >>>> Subject: RE: ckptnd memory leak(very serious)
> >>>>
> >>>> Hi,
> >>>> I can create an OpenSAF ticket as a place holder for your findings.
> >>>> To make things easier for any person who want to dig into this:
> >>>> -Can you attach source code to the ckperf binary so it is possible to
> >>>> see the use case?
> >>>> also
> >>>> -Can you attach the source code for "I created 500,0000 sections
> >>>> first and then overwrite it with some data" use case, which I
> >>>> assume was the test case for the attached valgrind reports (or is
> >>>> this the ckperf binary used on two controllers started with "ckperf
> >>>> 1" and "ckperf
> >>>> 0"??)
> >>>>
> >>>> BR
> >>>> Ingvar
> >>>>
> >>>> -----Original Message-----
> >>>> From: Yao Cheng LIANG [mailto:[email protected]]
> >>>> Sent: den 12 maj 2015 03:43
> >>>> To: Yao Cheng LIANG;[email protected]
> <mailto:[email protected]>
> >>>> Subject: Re: [users] ckptnd memory leak(very serious)
> >>>>
> >>>> The issue is very easy to reproduce. I am running 4.6 with two
> >>>> controllers on RHEL 6.5 x86_64, and if you
> run the attached app on one node with command line "ckperf 1" on one
> node and "ckperf 0" on another. You will observe ckptnd memory
> increased very rapidly with cmd "top -p pid_of_ckptnd".
> >>>>
> >>>> BTW, I am running with TIPC.
> >>>>
> >>>> Ted
> >>>>
> >>>> -----Original Message-----
> >>>> From: Yao Cheng LIANG [mailto:[email protected]]
> >>>> Sent: Tuesday, May 12, 2015 5:47 AM
> >>>> To:[email protected]
> <mailto:[email protected]>
> >>>> Subject: [users] Fwd: ckptnd memory leak(very serious)
> >>>>
> >>>> Resend
> >>>>
> >>>>
> >>>> Sent from Samsung Mobile
> >>>>
> >>>>
> >>>> -------- Original message --------
> >>>> From: Yao Cheng LIANG
> >>>> Date:2015/05/11 5:48 PM (GMT+08:00)
> >>>> To:[email protected]
> <mailto:[email protected]>
> >>>> Cc: Yao Cheng LIANG
> >>>> Subject: ckptnd memory leak(very serious)
> >>>>
> >>>> All,
> >>>>
> >>>> I did some tests recently on ckptnd. I created 500,0000 sections first
> >>>> and then overwrite it with some data.
> And found serious memory leak. Please see attached file for valgrind
> report.
> >>>>
> >>>> Thanks.
> >>>>
> >>>> Ted
> >>>>
> >>>>
> >>>> -------------------------------------------------------------------
> >>>> -
> >>>> -
> >>>> -
> >>>> -------- One dashboard for servers and applications across
> >>>> Physical-Virtual-Cloud Widest out-of-the-box monitoring support
> >>>> with
> >>>> 50+ applications Performance metrics, stats and reports that give
> >>>> 50+ you
> >>>> Actionable Insights Deep dive visibility with transaction tracing
> >>>> using APM Insight.
> >>>>http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Opensaf-users mailing list
> >>>>[email protected]
> <mailto:[email protected]>
> >>>>https://lists.sourceforge.net/lists/listinfo/opensaf-users
> >>> --------------------------------------------------------------------
> >>> -
> >>> -
> >>> -------- One dashboard for servers and applications across
> >>> Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+
> >>> applications Performance metrics, stats
> and reports that give you Actionable Insights Deep dive visibility
> with transaction tracing using APM Insight.
> >>>http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
> >>> _______________________________________________
> >>> Opensaf-users mailing list
> >>>[email protected]
> <mailto:[email protected]>
> >>>https://lists.sourceforge.net/lists/listinfo/opensaf-users
> >
> > ----------------------------------------------------------------------
> > -------- One dashboard for servers and applications across
> > Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+
> > applications Performance metrics, stats
> and reports that give you Actionable Insights Deep dive visibility
> with transaction tracing using APM Insight.
> >http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
> > _______________________________________________
> > Opensaf-users mailing list
> >[email protected]
> <mailto:[email protected]>
> >https://lists.sourceforge.net/lists/listinfo/opensaf-users
------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Opensaf-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-users
