Re: Wierd behavior with Java API - ar-server
The file is read in bit, get the same time if I using SQL also... Seems like the context.endBulkEntryTransaction() does not get recorded in the AR_SERVER_STAT_API_TIME server statistic option. The statistic are the same before and after the call has been initiated Need to hunt down more examples... Regards, Jarl 2009/8/17 Lyle Taylor tayl...@ldschurch.org: While you're at it, it may help to add some more detail here. Is the basic structure of your program like this: Timing Point A) Begin Bulk Transaction (optional) Timing Point B) LOOP - Create Entry Timing Point C) Commit Bulk Transaction (optional) Timing Point D) There are three basic parts, with four points to use when measuring time. If that matches your program's basic structure, then it's important to know where you're measuring your times. If you're simply measuring the time elapsed between points A and D (essentially, the entire execution time), that may not be granular enough to isolate differences. Based on your last e-mail, it sounds like you might be interested in the time it takes to get from point B to C (processing the entire loop of 10,000 records) with and without a bulk transaction and with either a regular or a display-only form. Is that correct? One other item to consider is how you're reading in your csv file. Are you reading it in all at once and then processing the records, or are you reading it in line by line as you call the Create Entry functions? If the latter, you'll probably get more consistent or accurate timing results if you read the entire file into memory and then process the records rather than reading line by line as you go, because other disk operations may interfere with reading the file efficiently, and no two runs will be alike. Lyle -Original Message- From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Jarl Grøneng Sent: Monday, August 17, 2009 2:36 PM To: arslist@ARSLIST.ORG Subject: Re: Wierd behavior with Java API - ar-server I'll set up some more automated tests tomorrow, its too late for this kind of things over here :-) -- Jarl 2009/8/17 Lyle Taylor tayl...@ldschurch.org: If you run each of the scenarios multiple times and average the times, do you still see a 25% (or 20%, depending on how you figure it) difference between 1 2 and 3 4? Lyle -Original Message- From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Jarl Grøneng Sent: Monday, August 17, 2009 2:01 PM To: arslist@ARSLIST.ORG Subject: Re: Wierd behavior with Java API - ar-server 2009/8/17 Lyle Taylor tayl...@ldschurch.org: Jarl, Your comment that the difference between 1 and 2 is due to one commit to the database is incorrect. That end is the same - the difference is in the network latency due to 1 vs 10,000 API calls across the network. That's the exact same difference that you see between 3 4. Did a new test-run, and the commit in example 2), with bulk transaction took approx 18 seconds. 2009-08-17 21:52:50,593 DEBUG [main] ARImport (ARImport.java:365) - Start bulk commit 2009-08-17 21:53:08,203 DEBUG [main] ARImport (ARImport.java:372) - End bulk create, sending all requests to AR Server The network latency should be very small since the client, server and database is on the same machine (as I wrote in my first email) It is 25% difference between 3-4 and 1-2 (16 seconds vs 20 seconds) -- Jarl ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: Where the Answers Are NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: Where the Answers Are ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: Where the Answers Are ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: Where the Answers Are ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: Where the Answers Are
Wierd behavior with Java API - ar-server
I have a JavaAPI program that reads a csv file, and import the data into a form 1) Import 10.000 records into a regular form, takes appox 70 seconds 2) Import 10.000 records into a regular form with bulkimport enabled , takes appox 50 seconds: The time gap between 1 and 2 is mainly one commit to the database. 3) Import 10.000 records into a display only form, takes appox 20 seconds 4) Import 10.000 records into a display only forrm with bulkimport enabled, takes appox 4 seconds. Why should it be that large difference between 3 and 4? AR server 7.5 patch 2 Oracle 10 (all figures on arserver on my laptop) -- Jarl ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: Where the Answers Are
Re: Wierd behavior with Java API - ar-server
Jarl, Let's hear it for LATENCY. The network kind (ok, there is a bit of savings on basic overhead with user checking and parameter marshalling and fewer TCP transactions (although they are different sizes) and other things like this because of fewer API calls). Total volume of data across the wire is the same, but it is many small packets (10,000 small) or fewer larger packets ( larger with being how many things are in the bulk operation you are doing). In your first example, the difference is about 20 seconds. In your second example, the difference is about 16 seconds. The difference between case 1 and case 2 is whether there is a write to the DB involved. Well, the DB write and commit is the majority of the time in these cases. In the bulk case, it is about 46 seconds of the time 4 seconds vs. 50 seconds. Interestingly, in the non bulk case, it is about 50 seconds of the time 20 seconds vs. 70 seconds. Now, with this, it looks like there is 4 seconds related to DB difference and 16 that is common and not related to DB difference. I would guess that the 4 second DB difference is small because of there being no data in the tables and probably small records and no other work so the extra commits you are saving (not saving anything from an insert or workflow perspective so it is just the multiple transaction aspect) are relatively small in this case. It is still 20% savings! The time savings on the network are relatively constant in this mix -- although if you have high network latency, you are saving even more. For example, say there were chunks of 100 operations in the bulk operations. That would mean 100 chunks. If you were talking about .25 second latency, you are talking about 100 * .25 or 25 seconds vs. 1 * .25 or 2500 seconds in time for the savings on the network (notice how any overhead of the API calls is totally swamped by network time when there is latency). Again, fixed, but just a bigger number than the above. It is important to look at what the savings is in situations like this and where the savings is coming from. I cannot gaurantee that the numbers are all in the locations I note here, but the concept of what the bulk calls do and save and where the savings are (db vs. network) are correct. Also, it shows why the numbers that look odd at first glance are much more reasonable and make more sense. You are just swamped here with network and overhead issues that are taking the majority of the time while the DB side is efficient and giving you gain, but a lower percentage of the gain. I hope this helps explain what is likely occurring and how to interpret your data results. Doug Mueller -Original Message- From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Jarl Grøneng Sent: Monday, August 17, 2009 8:16 AM To: arslist@ARSLIST.ORG Subject: Wierd behavior with Java API - ar-server I have a JavaAPI program that reads a csv file, and import the data into a form 1) Import 10.000 records into a regular form, takes appox 70 seconds 2) Import 10.000 records into a regular form with bulkimport enabled , takes appox 50 seconds: The time gap between 1 and 2 is mainly one commit to the database. 3) Import 10.000 records into a display only form, takes appox 20 seconds 4) Import 10.000 records into a display only forrm with bulkimport enabled, takes appox 4 seconds. Why should it be that large difference between 3 and 4? AR server 7.5 patch 2 Oracle 10 (all figures on arserver on my laptop) -- Jarl ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: Where the Answers Are ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: Where the Answers Are
Re: Wierd behavior with Java API - ar-server
Doug, Thanks for you long answer. (I think I have wast some of your time here...) I may have explain the issue a bit wrong... When I wrote bulkimport, I ment using the beginBulkEntryTransaction() option. So in example 3), I create 10.000 entries in a display only form. This should not do any commit in the database, it should do no database transaction at all (regards the logs, it does not do anything in the database) In example 4) I using the beginBulkEntryTransaction() when creating entries. This also should do none database transaction (is does one, Commit) However, regards the internal logic inside ar-server, example 3) and 4) should be quite equal. Still the non bulk transaction takes 5 times longer than the bulk transaction. Regards, Jarl 2009/8/17 Mueller, Doug doug_muel...@bmc.com: Jarl, Let's hear it for LATENCY. The network kind (ok, there is a bit of savings on basic overhead with user checking and parameter marshalling and fewer TCP transactions (although they are different sizes) and other things like this because of fewer API calls). Total volume of data across the wire is the same, but it is many small packets (10,000 small) or fewer larger packets ( larger with being how many things are in the bulk operation you are doing). In your first example, the difference is about 20 seconds. In your second example, the difference is about 16 seconds. The difference between case 1 and case 2 is whether there is a write to the DB involved. Well, the DB write and commit is the majority of the time in these cases. In the bulk case, it is about 46 seconds of the time 4 seconds vs. 50 seconds. Interestingly, in the non bulk case, it is about 50 seconds of the time 20 seconds vs. 70 seconds. Now, with this, it looks like there is 4 seconds related to DB difference and 16 that is common and not related to DB difference. I would guess that the 4 second DB difference is small because of there being no data in the tables and probably small records and no other work so the extra commits you are saving (not saving anything from an insert or workflow perspective so it is just the multiple transaction aspect) are relatively small in this case. It is still 20% savings! The time savings on the network are relatively constant in this mix -- although if you have high network latency, you are saving even more. For example, say there were chunks of 100 operations in the bulk operations. That would mean 100 chunks. If you were talking about .25 second latency, you are talking about 100 * .25 or 25 seconds vs. 1 * .25 or 2500 seconds in time for the savings on the network (notice how any overhead of the API calls is totally swamped by network time when there is latency). Again, fixed, but just a bigger number than the above. It is important to look at what the savings is in situations like this and where the savings is coming from. I cannot gaurantee that the numbers are all in the locations I note here, but the concept of what the bulk calls do and save and where the savings are (db vs. network) are correct. Also, it shows why the numbers that look odd at first glance are much more reasonable and make more sense. You are just swamped here with network and overhead issues that are taking the majority of the time while the DB side is efficient and giving you gain, but a lower percentage of the gain. I hope this helps explain what is likely occurring and how to interpret your data results. Doug Mueller -Original Message- From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Jarl Grøneng Sent: Monday, August 17, 2009 8:16 AM To: arslist@ARSLIST.ORG Subject: Wierd behavior with Java API - ar-server I have a JavaAPI program that reads a csv file, and import the data into a form 1) Import 10.000 records into a regular form, takes appox 70 seconds 2) Import 10.000 records into a regular form with bulkimport enabled , takes appox 50 seconds: The time gap between 1 and 2 is mainly one commit to the database. 3) Import 10.000 records into a display only form, takes appox 20 seconds 4) Import 10.000 records into a display only forrm with bulkimport enabled, takes appox 4 seconds. Why should it be that large difference between 3 and 4? AR server 7.5 patch 2 Oracle 10 (all figures on arserver on my laptop) -- Jarl ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: Where the Answers Are ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: Where the Answers Are ___ UNSUBSCRIBE or access ARSlist Archives
Re: Wierd behavior with Java API - ar-server
Jarl, Well, there is something you don't know about what is really going on that may help explain why my explaination answers your question You are making the same number of API calls (well, in the bulk case you are making 2 more for the begin and commit). HOWEVER, what I wasn't clear about is it is API calls that go across the network that are at issue here not actual API calls. This is because some API calls don't go across the network. In your scenario, what is happening In the example for #3, you issue 10,000 CreateEntry calls. This generates 10,000 operations across the network, one per CreateEntry call. In the example for #4, you issue 1 beginBulk, 10,000 CreateEntry, and 1 commitBulk calls. This generates 1 (ONE) operation across the network. Notice the major difference of 10,000 operations across the network vs. the 1 operation across the network in the two scenarios. The beginBulk operation just starts queueing up the operations you want to perform until it gets the commitBulk (or endBulk or whatever the name is) and when it gets that, it sends all the pending commands across the network to the server. Now, there is 1 BIG set of data flowing vs. 10,000 little sets of data flowing across the network for these scenarios. This is the major difference in operation. Within the server, the operations are the same with each operation being processed. And, in the case of your display only form, there is no DB operation so it is just the running through the list of 10,000 operations, any data validation, any filter processing, and that is it. So, the difference of 4 seconds vs. 20 seconds is the diffence of 10,000 individual small interactions between your program and the server and the 1 big interaction between your program and the server. With this understanding of what is really going on within your API calls (and why you may want to do this in chunks of 100 or 1000 rather than letting the bulk call include 100s of thousands to control memory sizes on both client and server with the large pending list) help to now match with my explaination of what is going on and why the perceived difference in timing is not really a difference? Doug Mueller -Original Message- From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Jarl Grøneng Sent: Monday, August 17, 2009 11:38 AM To: arslist@ARSLIST.ORG Subject: Re: Wierd behavior with Java API - ar-server Doug, Thanks for you long answer. (I think I have wast some of your time here...) I may have explain the issue a bit wrong... When I wrote bulkimport, I ment using the beginBulkEntryTransaction() option. So in example 3), I create 10.000 entries in a display only form. This should not do any commit in the database, it should do no database transaction at all (regards the logs, it does not do anything in the database) In example 4) I using the beginBulkEntryTransaction() when creating entries. This also should do none database transaction (is does one, Commit) However, regards the internal logic inside ar-server, example 3) and 4) should be quite equal. Still the non bulk transaction takes 5 times longer than the bulk transaction. Regards, Jarl 2009/8/17 Mueller, Doug doug_muel...@bmc.com: Jarl, Let's hear it for LATENCY. The network kind (ok, there is a bit of savings on basic overhead with user checking and parameter marshalling and fewer TCP transactions (although they are different sizes) and other things like this because of fewer API calls). Total volume of data across the wire is the same, but it is many small packets (10,000 small) or fewer larger packets ( larger with being how many things are in the bulk operation you are doing). In your first example, the difference is about 20 seconds. In your second example, the difference is about 16 seconds. The difference between case 1 and case 2 is whether there is a write to the DB involved. Well, the DB write and commit is the majority of the time in these cases. In the bulk case, it is about 46 seconds of the time 4 seconds vs. 50 seconds. Interestingly, in the non bulk case, it is about 50 seconds of the time 20 seconds vs. 70 seconds. Now, with this, it looks like there is 4 seconds related to DB difference and 16 that is common and not related to DB difference. I would guess that the 4 second DB difference is small because of there being no data in the tables and probably small records and no other work so the extra commits you are saving (not saving anything from an insert or workflow perspective so it is just the multiple transaction aspect) are relatively small in this case. It is still 20% savings! The time savings on the network are relatively constant in this mix -- although if you have high network latency, you are saving even more. For example, say there were chunks of 100 operations in the bulk operations. That would mean 100 chunks. If you were talking
Re: Wierd behavior with Java API - ar-server
Doug, Your explanation and answer are probably correct since you are the master of AR System :-) (use the source Luke) But I find the figures to far off what I would expect and gonna spend some more time to pinpoint where the small amout of time goes Regards, Jarl 2009/8/17 Mueller, Doug doug_muel...@bmc.com: Jarl, Well, there is something you don't know about what is really going on that may help explain why my explaination answers your question You are making the same number of API calls (well, in the bulk case you are making 2 more for the begin and commit). HOWEVER, what I wasn't clear about is it is API calls that go across the network that are at issue here not actual API calls. This is because some API calls don't go across the network. In your scenario, what is happening In the example for #3, you issue 10,000 CreateEntry calls. This generates 10,000 operations across the network, one per CreateEntry call. In the example for #4, you issue 1 beginBulk, 10,000 CreateEntry, and 1 commitBulk calls. This generates 1 (ONE) operation across the network. Notice the major difference of 10,000 operations across the network vs. the 1 operation across the network in the two scenarios. The beginBulk operation just starts queueing up the operations you want to perform until it gets the commitBulk (or endBulk or whatever the name is) and when it gets that, it sends all the pending commands across the network to the server. Now, there is 1 BIG set of data flowing vs. 10,000 little sets of data flowing across the network for these scenarios. This is the major difference in operation. Within the server, the operations are the same with each operation being processed. And, in the case of your display only form, there is no DB operation so it is just the running through the list of 10,000 operations, any data validation, any filter processing, and that is it. So, the difference of 4 seconds vs. 20 seconds is the diffence of 10,000 individual small interactions between your program and the server and the 1 big interaction between your program and the server. With this understanding of what is really going on within your API calls (and why you may want to do this in chunks of 100 or 1000 rather than letting the bulk call include 100s of thousands to control memory sizes on both client and server with the large pending list) help to now match with my explaination of what is going on and why the perceived difference in timing is not really a difference? Doug Mueller -Original Message- From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Jarl Grøneng Sent: Monday, August 17, 2009 11:38 AM To: arslist@ARSLIST.ORG Subject: Re: Wierd behavior with Java API - ar-server Doug, Thanks for you long answer. (I think I have wast some of your time here...) I may have explain the issue a bit wrong... When I wrote bulkimport, I ment using the beginBulkEntryTransaction() option. So in example 3), I create 10.000 entries in a display only form. This should not do any commit in the database, it should do no database transaction at all (regards the logs, it does not do anything in the database) In example 4) I using the beginBulkEntryTransaction() when creating entries. This also should do none database transaction (is does one, Commit) However, regards the internal logic inside ar-server, example 3) and 4) should be quite equal. Still the non bulk transaction takes 5 times longer than the bulk transaction. Regards, Jarl 2009/8/17 Mueller, Doug doug_muel...@bmc.com: Jarl, Let's hear it for LATENCY. The network kind (ok, there is a bit of savings on basic overhead with user checking and parameter marshalling and fewer TCP transactions (although they are different sizes) and other things like this because of fewer API calls). Total volume of data across the wire is the same, but it is many small packets (10,000 small) or fewer larger packets ( larger with being how many things are in the bulk operation you are doing). In your first example, the difference is about 20 seconds. In your second example, the difference is about 16 seconds. The difference between case 1 and case 2 is whether there is a write to the DB involved. Well, the DB write and commit is the majority of the time in these cases. In the bulk case, it is about 46 seconds of the time 4 seconds vs. 50 seconds. Interestingly, in the non bulk case, it is about 50 seconds of the time 20 seconds vs. 70 seconds. Now, with this, it looks like there is 4 seconds related to DB difference and 16 that is common and not related to DB difference. I would guess that the 4 second DB difference is small because of there being no data in the tables and probably small records and no other work so the extra commits you are saving (not saving anything from an insert or workflow perspective so
Re: Wierd behavior with Java API - ar-server
Jarl, I'm not sure what you're looking for, the difference between 3 and 4 is pretty much equivalent to the difference between 1 and 2. You've got differences of 16 and 20 seconds between 3 4 and 1 2, and these are adequately accounted for with Doug's explanation of 1 vs. many API calls across the network. Your comment that the difference between 1 and 2 is due to one commit to the database is incorrect. That end is the same - the difference is in the network latency due to 1 vs 10,000 API calls across the network. That's the exact same difference that you see between 3 4. The two different scenarios (DO form vs. regular form) are behaving essentially exactly the same. It looks like there really is no difference between the two except that your first scenario involves database commits which adds 50 seconds to the overall processing time that you don't get with the DO form. Lyle -Original Message- From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Jarl Grøneng Sent: Monday, August 17, 2009 1:28 PM To: arslist@ARSLIST.ORG Subject: Re: Wierd behavior with Java API - ar-server Doug, Your explanation and answer are probably correct since you are the master of AR System :-) (use the source Luke) But I find the figures to far off what I would expect and gonna spend some more time to pinpoint where the small amout of time goes Regards, Jarl 2009/8/17 Mueller, Doug doug_muel...@bmc.com: Jarl, Well, there is something you don't know about what is really going on that may help explain why my explaination answers your question You are making the same number of API calls (well, in the bulk case you are making 2 more for the begin and commit). HOWEVER, what I wasn't clear about is it is API calls that go across the network that are at issue here not actual API calls. This is because some API calls don't go across the network. In your scenario, what is happening In the example for #3, you issue 10,000 CreateEntry calls. This generates 10,000 operations across the network, one per CreateEntry call. In the example for #4, you issue 1 beginBulk, 10,000 CreateEntry, and 1 commitBulk calls. This generates 1 (ONE) operation across the network. Notice the major difference of 10,000 operations across the network vs. the 1 operation across the network in the two scenarios. The beginBulk operation just starts queueing up the operations you want to perform until it gets the commitBulk (or endBulk or whatever the name is) and when it gets that, it sends all the pending commands across the network to the server. Now, there is 1 BIG set of data flowing vs. 10,000 little sets of data flowing across the network for these scenarios. This is the major difference in operation. Within the server, the operations are the same with each operation being processed. And, in the case of your display only form, there is no DB operation so it is just the running through the list of 10,000 operations, any data validation, any filter processing, and that is it. So, the difference of 4 seconds vs. 20 seconds is the diffence of 10,000 individual small interactions between your program and the server and the 1 big interaction between your program and the server. With this understanding of what is really going on within your API calls (and why you may want to do this in chunks of 100 or 1000 rather than letting the bulk call include 100s of thousands to control memory sizes on both client and server with the large pending list) help to now match with my explaination of what is going on and why the perceived difference in timing is not really a difference? Doug Mueller -Original Message- From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Jarl Grøneng Sent: Monday, August 17, 2009 11:38 AM To: arslist@ARSLIST.ORG Subject: Re: Wierd behavior with Java API - ar-server Doug, Thanks for you long answer. (I think I have wast some of your time here...) I may have explain the issue a bit wrong... When I wrote bulkimport, I ment using the beginBulkEntryTransaction() option. So in example 3), I create 10.000 entries in a display only form. This should not do any commit in the database, it should do no database transaction at all (regards the logs, it does not do anything in the database) In example 4) I using the beginBulkEntryTransaction() when creating entries. This also should do none database transaction (is does one, Commit) However, regards the internal logic inside ar-server, example 3) and 4) should be quite equal. Still the non bulk transaction takes 5 times longer than the bulk transaction. Regards, Jarl 2009/8/17 Mueller, Doug doug_muel...@bmc.com: Jarl, Let's hear it for LATENCY. The network kind (ok, there is a bit of savings on basic overhead with user checking and parameter marshalling and fewer TCP
Re: Wierd behavior with Java API - ar-server
2009/8/17 Lyle Taylor tayl...@ldschurch.org: Jarl, Your comment that the difference between 1 and 2 is due to one commit to the database is incorrect. That end is the same - the difference is in the network latency due to 1 vs 10,000 API calls across the network. That's the exact same difference that you see between 3 4. Did a new test-run, and the commit in example 2), with bulk transaction took approx 18 seconds. 2009-08-17 21:52:50,593 DEBUG [main] ARImport (ARImport.java:365) - Start bulk commit 2009-08-17 21:53:08,203 DEBUG [main] ARImport (ARImport.java:372) - End bulk create, sending all requests to AR Server The network latency should be very small since the client, server and database is on the same machine (as I wrote in my first email) It is 25% difference between 3-4 and 1-2 (16 seconds vs 20 seconds) -- Jarl ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: Where the Answers Are
Re: Wierd behavior with Java API - ar-server
If you run each of the scenarios multiple times and average the times, do you still see a 25% (or 20%, depending on how you figure it) difference between 1 2 and 3 4? Lyle -Original Message- From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Jarl Grøneng Sent: Monday, August 17, 2009 2:01 PM To: arslist@ARSLIST.ORG Subject: Re: Wierd behavior with Java API - ar-server 2009/8/17 Lyle Taylor tayl...@ldschurch.org: Jarl, Your comment that the difference between 1 and 2 is due to one commit to the database is incorrect. That end is the same - the difference is in the network latency due to 1 vs 10,000 API calls across the network. That's the exact same difference that you see between 3 4. Did a new test-run, and the commit in example 2), with bulk transaction took approx 18 seconds. 2009-08-17 21:52:50,593 DEBUG [main] ARImport (ARImport.java:365) - Start bulk commit 2009-08-17 21:53:08,203 DEBUG [main] ARImport (ARImport.java:372) - End bulk create, sending all requests to AR Server The network latency should be very small since the client, server and database is on the same machine (as I wrote in my first email) It is 25% difference between 3-4 and 1-2 (16 seconds vs 20 seconds) -- Jarl ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: Where the Answers Are NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: Where the Answers Are
Re: Wierd behavior with Java API - ar-server
I'll set up some more automated tests tomorrow, its too late for this kind of things over here :-) -- Jarl 2009/8/17 Lyle Taylor tayl...@ldschurch.org: If you run each of the scenarios multiple times and average the times, do you still see a 25% (or 20%, depending on how you figure it) difference between 1 2 and 3 4? Lyle -Original Message- From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Jarl Grøneng Sent: Monday, August 17, 2009 2:01 PM To: arslist@ARSLIST.ORG Subject: Re: Wierd behavior with Java API - ar-server 2009/8/17 Lyle Taylor tayl...@ldschurch.org: Jarl, Your comment that the difference between 1 and 2 is due to one commit to the database is incorrect. That end is the same - the difference is in the network latency due to 1 vs 10,000 API calls across the network. That's the exact same difference that you see between 3 4. Did a new test-run, and the commit in example 2), with bulk transaction took approx 18 seconds. 2009-08-17 21:52:50,593 DEBUG [main] ARImport (ARImport.java:365) - Start bulk commit 2009-08-17 21:53:08,203 DEBUG [main] ARImport (ARImport.java:372) - End bulk create, sending all requests to AR Server The network latency should be very small since the client, server and database is on the same machine (as I wrote in my first email) It is 25% difference between 3-4 and 1-2 (16 seconds vs 20 seconds) -- Jarl ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: Where the Answers Are NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: Where the Answers Are ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: Where the Answers Are
Re: Wierd behavior with Java API - ar-server
While you're at it, it may help to add some more detail here. Is the basic structure of your program like this: Timing Point A) Begin Bulk Transaction (optional) Timing Point B) LOOP - Create Entry Timing Point C) Commit Bulk Transaction (optional) Timing Point D) There are three basic parts, with four points to use when measuring time. If that matches your program's basic structure, then it's important to know where you're measuring your times. If you're simply measuring the time elapsed between points A and D (essentially, the entire execution time), that may not be granular enough to isolate differences. Based on your last e-mail, it sounds like you might be interested in the time it takes to get from point B to C (processing the entire loop of 10,000 records) with and without a bulk transaction and with either a regular or a display-only form. Is that correct? One other item to consider is how you're reading in your csv file. Are you reading it in all at once and then processing the records, or are you reading it in line by line as you call the Create Entry functions? If the latter, you'll probably get more consistent or accurate timing results if you read the entire file into memory and then process the records rather than reading line by line as you go, because other disk operations may interfere with reading the file efficiently, and no two runs will be alike. Lyle -Original Message- From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Jarl Grøneng Sent: Monday, August 17, 2009 2:36 PM To: arslist@ARSLIST.ORG Subject: Re: Wierd behavior with Java API - ar-server I'll set up some more automated tests tomorrow, its too late for this kind of things over here :-) -- Jarl 2009/8/17 Lyle Taylor tayl...@ldschurch.org: If you run each of the scenarios multiple times and average the times, do you still see a 25% (or 20%, depending on how you figure it) difference between 1 2 and 3 4? Lyle -Original Message- From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Jarl Grøneng Sent: Monday, August 17, 2009 2:01 PM To: arslist@ARSLIST.ORG Subject: Re: Wierd behavior with Java API - ar-server 2009/8/17 Lyle Taylor tayl...@ldschurch.org: Jarl, Your comment that the difference between 1 and 2 is due to one commit to the database is incorrect. That end is the same - the difference is in the network latency due to 1 vs 10,000 API calls across the network. That's the exact same difference that you see between 3 4. Did a new test-run, and the commit in example 2), with bulk transaction took approx 18 seconds. 2009-08-17 21:52:50,593 DEBUG [main] ARImport (ARImport.java:365) - Start bulk commit 2009-08-17 21:53:08,203 DEBUG [main] ARImport (ARImport.java:372) - End bulk create, sending all requests to AR Server The network latency should be very small since the client, server and database is on the same machine (as I wrote in my first email) It is 25% difference between 3-4 and 1-2 (16 seconds vs 20 seconds) -- Jarl ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: Where the Answers Are NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: Where the Answers Are ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: Where the Answers Are ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: Where the Answers Are
