Creating Archive forms and ARS stability
Hi Folks, Environments: 1) ARS / ITSM 7.6.04 p4 Linux Oracle (customer dev machine - ie larger than the average VM) 2) ARS / ITSM 7.6.04 p0 Windows 2003 x64, MS SQL VM 2 cores (4 cpu to the OS) 8 Gb 3) ARS / ITSM 8.0.0 p0 Linux Oracle VM 2 cores (4 cpu to the OS) 8 Gb I create archive forms in an automated way one time but for many forms (over 100 in a full run). The forms I do this for are all Regular forms: not Joins or other forms types. In the API it is a simple argument passed to ARSetSchema on the non-archive schema. I find each creation takes 5 to 7 minutes, takes a huge amount of memory on the server - most of which is not released when completed, and drives the server CPU quite high to calm down again when completed. More concern is the fact that ARS crashes occasionally. For example, once after a customer built 4 forms on his development machine the server crashed building the 5th. After it restarted (automatically - the common fix these days for bad code in an application) a rerun of the job caused no server crashes at all for a subsequent 29ish forms. (Just running the creation against the Incident forms). On my VMs this also happens on rareish occasions. If I had to put a number on it, it would average once per 40ish forms - but this is a guestimate at best. It is run once and only once per VM instance and I tend to run it against all requests rather than say just Incident. Because of snapshots, it is easy enough to rerun this job. Obviously, causing a crash is never good, and documentation reading this process can cause server crashes is not a good thing J I also always restart the server process once the forms are built to clean up the memory - also not a good thing to tell customers: that after building their archive forms, their server should be restarted to clean up leaked memory J Of course it is hard to replicate this using Dev Studio as one must create a single archive form, wait the requisite 5 or so minutes, then manually create the next one. My process creates one, waits the requisite time, and then creates the next - right away (after say a few milliseconds). I have not tried this yet on 8.1 but intend to do so shortly. A further irritation is that the BMC documentation indicates that Merge writes into these forms are permitted. Yet I get the error saying that writes into these forms is not permitted - with all variations of the Merge options. I have not tested this on a lot of releases but it is true in the 8.0.0 unpatched release. I simply disassociate the archive forms from their original forms to get around this. Just curious. Has anyone dealt with this? Or communicated with BMC on this? I suppose I should raise a BMC ticket but I generally do not do this unless a customer has asked (and paid) me to do so J Ben Chernys Senior Software Architect logoSthInc-sm Canada / Deutschland Mobile: +49 171 380 2329GMT + 1 + [ DST ] Email:mailto:Ben.Chernys_AT_softwaretoolhouse.com Ben.Chernys_AT_softwaretoolhouse.com Web: http://www.softwaretoolhouse.com/ www.softwaretoolhouse.com We are a BMC Technology Alliance Partner. Check out Software Tool House's free Diary Editor and out Freebies Section for ITSM 7.6.04 and 8.0.0 Fields spreadsheets. Meta-Update, our premium ARS Data tool, lets you automate your imports, migrations, in no time at all, without programming, without staging forms, without merge workflow. http://www.softwaretoolhouse.com/ http://www.softwaretoolhouse.com/ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years image002.jpg smime.p7s Description: S/MIME cryptographic signature
Re: Creating Archive forms and ARS stability
Hi Ben, Thats interesting. Here are some thoughts: Is there always a crash after 4? Is it the same 4 forms? e.g. If you just created 4 other sample forms and used those do you get the same crash after 4? Do you think its something to do with form number 4? e.g. it didnt like diary fields or something like that? Or maybe it's due to the weight of data on form 4? When it crashes is there a core on the Linux box or is there a stack trace in the arerror.log? Are you seeing the copycache, in the thread logging, complete before you try the next form? You could probably fire up 5 dev studios and get them ready to save and try replicating it to a certain degree. Or actually maybe a quick driver script? Kind regards Danny Single Sign On (SSO) for the BMC Remedy AR System and ITSM http://www.javasystemsolutions.com/jss/ssoplugin Hi Folks, Environments: 1) ARS / ITSM 7.6.04 p4 Linux Oracle (customer dev machine - ie larger than the average VM) 2) ARS / ITSM 7.6.04 p0 Windows 2003 x64, MS SQL VM 2 cores (4 cpu to the OS) 8 Gb 3) ARS / ITSM 8.0.0 p0 Linux Oracle VM 2 cores (4 cpu to the OS) 8 Gb I create archive forms in an automated way one time but for many forms (over 100 in a full run). The forms I do this for are all Regular forms: not Joins or other forms types. In the API it is a simple argument passed to ARSetSchema on the non-archive schema. I find each creation takes 5 to 7 minutes, takes a huge amount of memory on the server - most of which is not released when completed, and drives the server CPU quite high to calm down again when completed. More concern is the fact that ARS crashes occasionally. For example, once after a customer built 4 forms on his development machine the server crashed building the 5th. After it restarted (automatically - the common fix these days for bad code in an application) a rerun of the job caused no server crashes at all for a subsequent 29ish forms. (Just running the creation against the Incident forms). On my VMs this also happens on rareish occasions. If I had to put a number on it, it would average once per 40ish forms - but this is a guestimate at best. It is run once and only once per VM instance and I tend to run it against all requests rather than say just Incident. Because of snapshots, it is easy enough to rerun this job. Obviously, causing a crash is never good, and documentation reading this process can cause server crashes is not a good thing J I also always restart the server process once the forms are built to clean up the memory - also not a good thing to tell customers: that after building their archive forms, their server should be restarted to clean up leaked memory J Of course it is hard to replicate this using Dev Studio as one must create a single archive form, wait the requisite 5 or so minutes, then manually create the next one. My process creates one, waits the requisite time, and then creates the next - right away (after say a few milliseconds). I have not tried this yet on 8.1 but intend to do so shortly. A further irritation is that the BMC documentation indicates that Merge writes into these forms are permitted. Yet I get the error saying that writes into these forms is not permitted - with all variations of the Merge options. I have not tested this on a lot of releases but it is true in the 8.0.0 unpatched release. I simply disassociate the archive forms from their original forms to get around this. Just curious. Has anyone dealt with this? Or communicated with BMC on this? I suppose I should raise a BMC ticket but I generally do not do this unless a customer has asked (and paid) me to do so J Ben Chernys Senior Software Architect logoSthInc-sm Canada / Deutschland Mobile: +49 171 380 2329GMT + 1 + [ DST ] Email:mailto:Ben.Chernys_AT_softwaretoolhouse.com Ben.Chernys_AT_softwaretoolhouse.com Web: http://www.softwaretoolhouse.com/ www.softwaretoolhouse.com We are a BMC Technology Alliance Partner. Check out Software Tool House's free Diary Editor and out Freebies Section for ITSM 7.6.04 and 8.0.0 Fields spreadsheets. Meta-Update, our premium ARS Data tool, lets you automate your imports, migrations, in no time at all, without programming, without staging forms, without merge workflow. http://www.softwaretoolhouse.com/ http://www.softwaretoolhouse.com/ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Creating Archive forms and ARS stability
Hi Danny, I haven't done much investigation. I expect there is a core but it will be relatively useless without source. I haven’t run arserver under any of the tracing utilities. I do run with server tracing on but truth to tell, I haven't inspected them. As I said, it is on random forms (rather 'arbitrary'),and arbitrary numbers of forms, etc. As I said, sometimes it blows after a few dozen forms. Other times it doesn't blow up at all. Data should not matter. It is doing nothing with data - it is only building the form. I don't test it often 'cause once the forms are created, there is no need to do so again. It's just that I ran it on a customer's dev box this morning and that is where the number 4 came out. Interestingly, the archive form was there on restart (so it may well have been on cleanup; perhaps a double free or free of a null, or some such thing). I could do more investigation but then I'd be working for BMC - and not getting paid, and I am swamped right now. For now, I will simply document the warnings etc. Just that it's quite ugly to have to do that. I did have to once before, on an old release and it took 3 or 4 major releases 'till that problem was fixed. I finally removed that warning and the work-around code :) Thanks Ben -Original Message- From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Danny Kellett Sent: May-22-13 11:33 To: arslist@ARSLIST.ORG Subject: Re: Creating Archive forms and ARS stability Hi Ben, Thats interesting. Here are some thoughts: Is there always a crash after 4? Is it the same 4 forms? e.g. If you just created 4 other sample forms and used those do you get the same crash after 4? Do you think its something to do with form number 4? e.g. it didnt like diary fields or something like that? Or maybe it's due to the weight of data on form 4? When it crashes is there a core on the Linux box or is there a stack trace in the arerror.log? Are you seeing the copycache, in the thread logging, complete before you try the next form? You could probably fire up 5 dev studios and get them ready to save and try replicating it to a certain degree. Or actually maybe a quick driver script? Kind regards Danny Single Sign On (SSO) for the BMC Remedy AR System and ITSM http://www.javasystemsolutions.com/jss/ssoplugin Hi Folks, Environments: 1) ARS / ITSM 7.6.04 p4 Linux Oracle (customer dev machine - ie larger than the average VM) 2) ARS / ITSM 7.6.04 p0 Windows 2003 x64, MS SQL VM 2 cores (4 cpu to the OS) 8 Gb 3) ARS / ITSM 8.0.0 p0 Linux Oracle VM 2 cores (4 cpu to the OS) 8 Gb I create archive forms in an automated way one time but for many forms (over 100 in a full run). The forms I do this for are all Regular forms: not Joins or other forms types. In the API it is a simple argument passed to ARSetSchema on the non-archive schema. I find each creation takes 5 to 7 minutes, takes a huge amount of memory on the server - most of which is not released when completed, and drives the server CPU quite high to calm down again when completed. More concern is the fact that ARS crashes occasionally. For example, once after a customer built 4 forms on his development machine the server crashed building the 5th. After it restarted (automatically - the common fix these days for bad code in an application) a rerun of the job caused no server crashes at all for a subsequent 29ish forms. (Just running the creation against the Incident forms). On my VMs this also happens on rareish occasions. If I had to put a number on it, it would average once per 40ish forms - but this is a guestimate at best. It is run once and only once per VM instance and I tend to run it against all requests rather than say just Incident. Because of snapshots, it is easy enough to rerun this job. Obviously, causing a crash is never good, and documentation reading this process can cause server crashes is not a good thing J I also always restart the server process once the forms are built to clean up the memory - also not a good thing to tell customers: that after building their archive forms, their server should be restarted to clean up leaked memory J Of course it is hard to replicate this using Dev Studio as one must create a single archive form, wait the requisite 5 or so minutes, then manually create the next one. My process creates one, waits the requisite time, and then creates the next - right away (after say a few milliseconds). I have not tried this yet on 8.1 but intend to do so shortly. A further irritation is that the BMC documentation indicates that Merge writes into these forms are permitted. Yet I get the error saying that writes into these forms is not permitted - with all variations of the Merge options. I have not tested this on a lot of releases
Re: Creating Archive forms and ARS stability
Hi Ben, The core actually can help. You will find the function names recognisable to us that work with the api. I was thinking you might see a a createfield then then something like a malloc error on null etc. I have found a few bugs in my time with that. Work a look. Data should not matter. My thinking was if there are locks due to copycaches and the AR Server wasnt handling it correctly. I am assuming this is all happening on 390600. Good luck anyway. Regards Danny Single Sign On (SSO) for the BMC Remedy AR System and ITSM http://www.javasystemsolutions.com/jss/ssoplugin Hi Danny, I haven't done much investigation. I expect there is a core but it will be relatively useless without source. I haven’t run arserver under any of the tracing utilities. I do run with server tracing on but truth to tell, I haven't inspected them. As I said, it is on random forms (rather 'arbitrary'),and arbitrary numbers of forms, etc. As I said, sometimes it blows after a few dozen forms. Other times it doesn't blow up at all. Data should not matter. It is doing nothing with data - it is only building the form. I don't test it often 'cause once the forms are created, there is no need to do so again. It's just that I ran it on a customer's dev box this morning and that is where the number 4 came out. Interestingly, the archive form was there on restart (so it may well have been on cleanup; perhaps a double free or free of a null, or some such thing). I could do more investigation but then I'd be working for BMC - and not getting paid, and I am swamped right now. For now, I will simply document the warnings etc. Just that it's quite ugly to have to do that. I did have to once before, on an old release and it took 3 or 4 major releases 'till that problem was fixed. I finally removed that warning and the work-around code :) Thanks Ben -Original Message- From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Danny Kellett Sent: May-22-13 11:33 To: arslist@ARSLIST.ORG Subject: Re: Creating Archive forms and ARS stability Hi Ben, Thats interesting. Here are some thoughts: Is there always a crash after 4? Is it the same 4 forms? e.g. If you just created 4 other sample forms and used those do you get the same crash after 4? Do you think its something to do with form number 4? e.g. it didnt like diary fields or something like that? Or maybe it's due to the weight of data on form 4? When it crashes is there a core on the Linux box or is there a stack trace in the arerror.log? Are you seeing the copycache, in the thread logging, complete before you try the next form? You could probably fire up 5 dev studios and get them ready to save and try replicating it to a certain degree. Or actually maybe a quick driver script? Kind regards Danny Single Sign On (SSO) for the BMC Remedy AR System and ITSM http://www.javasystemsolutions.com/jss/ssoplugin Hi Folks, Environments: 1) ARS / ITSM 7.6.04 p4 Linux Oracle (customer dev machine - ie larger than the average VM) 2) ARS / ITSM 7.6.04 p0 Windows 2003 x64, MS SQL VM 2 cores (4 cpu to the OS) 8 Gb 3) ARS / ITSM 8.0.0 p0 Linux Oracle VM 2 cores (4 cpu to the OS) 8 Gb I create archive forms in an automated way one time but for many forms (over 100 in a full run). The forms I do this for are all Regular forms: not Joins or other forms types. In the API it is a simple argument passed to ARSetSchema on the non-archive schema. I find each creation takes 5 to 7 minutes, takes a huge amount of memory on the server - most of which is not released when completed, and drives the server CPU quite high to calm down again when completed. More concern is the fact that ARS crashes occasionally. For example, once after a customer built 4 forms on his development machine the server crashed building the 5th. After it restarted (automatically - the common fix these days for bad code in an application) a rerun of the job caused no server crashes at all for a subsequent 29ish forms. (Just running the creation against the Incident forms). On my VMs this also happens on rareish occasions. If I had to put a number on it, it would average once per 40ish forms - but this is a guestimate at best. It is run once and only once per VM instance and I tend to run it against all requests rather than say just Incident. Because of snapshots, it is easy enough to rerun this job. Obviously, causing a crash is never good, and documentation reading this process can cause server crashes is not a good thing J I also always restart the server process once the forms are built to clean up the memory - also not a good thing to tell customers: that after building their archive forms, their server should be restarted to clean up leaked memory J Of course it is hard to replicate this using Dev Studio as one must
