Jarl, I'm not sure what you're looking for, the difference between 3 and 4 is pretty much equivalent to the difference between 1 and 2. You've got differences of 16 and 20 seconds between 3 & 4 and 1 & 2, and these are adequately accounted for with Doug's explanation of 1 vs. many API calls across the network.
Your comment that the difference between 1 and 2 is due to one commit to the database is incorrect. That end is the same - the difference is in the network latency due to 1 vs 10,000 API calls across the network. That's the exact same difference that you see between 3 & 4. The two different scenarios (DO form vs. regular form) are behaving essentially exactly the same. It looks like there really is no difference between the two except that your first scenario involves database commits which adds 50 seconds to the overall processing time that you don't get with the DO form. Lyle -----Original Message----- From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Jarl Grøneng Sent: Monday, August 17, 2009 1:28 PM To: arslist@ARSLIST.ORG Subject: Re: Wierd behavior with Java API -> ar-server Doug, Your explanation and answer are probably correct since you are the master of AR System :-) (use the source Luke....) But I find the figures to far off what I would expect and gonna spend some more time to pinpoint where the small amout of time goes.... Regards, Jarl 2009/8/17 Mueller, Doug <doug_muel...@bmc.com>: > Jarl, > > Well, there is something you don't know about what is really going on that > may help explain why my explaination answers your question.... > > You are making the same number of API calls (well, in the bulk case you are > making 2 more for the begin and commit). HOWEVER, what I wasn't clear about > is it is API calls that go across the network that are at issue here not > actual API calls. This is because some API calls don't go across the network. > > In your scenario, what is happening.... > > In the example for #3, you issue 10,000 CreateEntry calls. This generates > 10,000 operations across the network, one per CreateEntry call. > > In the example for #4, you issue 1 beginBulk, 10,000 CreateEntry, and 1 > commitBulk calls. This generates 1 (ONE) operation across the network. > > Notice the major difference of 10,000 operations across the network vs. the > 1 operation across the network in the two scenarios. > > The beginBulk operation just starts queueing up the operations you want to > perform until it gets the commitBulk (or endBulk or whatever the name is) and > when it gets that, it sends all the pending commands across the network to the > server. > > Now, there is 1 BIG set of data flowing vs. 10,000 little sets of data flowing > across the network for these scenarios. > > This is the major difference in operation. Within the server, the operations > are the same with each operation being processed. And, in the case of your > display only form, there is no DB operation so it is just the running through > the list of 10,000 operations, any data validation, any filter processing, and > that is it. > > So, the difference of 4 seconds vs. 20 seconds is the diffence of 10,000 > individual small interactions between your program and the server and the 1 > big interaction between your program and the server. > > With this understanding of what is really going on within your API calls (and > why you may want to do this in chunks of 100 or 1000 rather than letting the > bulk call include 100s of thousands to control memory sizes on both client and > server with the large pending list) help to now match with my explaination of > what is going on and why the perceived difference in timing is not really a > difference? > > Doug Mueller > > -----Original Message----- > From: Action Request System discussion list(ARSList) > [mailto:arsl...@arslist.org] On Behalf Of Jarl Grøneng > Sent: Monday, August 17, 2009 11:38 AM > To: arslist@ARSLIST.ORG > Subject: Re: Wierd behavior with Java API -> ar-server > > Doug, > > Thanks for you long answer. (I think I have wast some of your time here...) > > I may have explain the issue a bit wrong... When I wrote bulkimport, I > ment using the beginBulkEntryTransaction() option. > > So in example 3), I create 10.000 entries in a display only form. This > should not do any commit in the database, it should do no database > transaction at all (regards the logs, it does not do anything in the > database) In example 4) I using the beginBulkEntryTransaction() when > creating entries. This also should do none database transaction (is > does one, Commit) > > However, regards the internal logic inside ar-server, example 3) and > 4) should be quite equal. Still the non bulk transaction takes 5 times > longer than the bulk transaction. > > Regards, > Jarl > > > > > 2009/8/17 Mueller, Doug <doug_muel...@bmc.com>: >> Jarl, >> >> Let's hear it for LATENCY. The network kind.... (ok, there is a bit of >> savings on basic overhead with user checking and parameter marshalling and >> fewer TCP transactions (although they are different sizes) and other things >> like this because of fewer API calls). Total volume of data across the wire >> is the same, but it is many small packets (10,000 small) or fewer larger >> packets (xxxx larger with xxxx being how many things are in the bulk >> operation >> you are doing). >> >> In your first example, the difference is about 20 seconds. >> In your second example, the difference is about 16 seconds. >> >> The difference between case 1 and case 2 is whether there is a write to the >> DB involved. Well, the DB write and commit is the majority of the time in >> these cases. In the bulk case, it is about 46 seconds of the time 4 seconds >> vs. 50 seconds. Interestingly, in the non bulk case, it is about 50 seconds >> of the time 20 seconds vs. 70 seconds. >> >> Now, with this, it looks like there is 4 seconds related to DB difference and >> 16 that is common and not related to DB difference. >> >> I would guess that the 4 second DB difference is small because of there being >> no data in the tables and probably small records and no other work so the >> extra commits you are saving (not saving anything from an insert or workflow >> perspective so it is just the multiple transaction aspect) are relatively >> small in this case. It is still 20% savings! >> >> The time savings on the network are relatively constant in this mix -- >> although >> if you have high network latency, you are saving even more. For example, >> say there were chunks of 100 operations in the bulk operations. That would >> mean 100 chunks. If you were talking about .25 second latency, you are >> talking >> about 100 * .25 or 25 seconds vs. 10000 * .25 or 2500 seconds in time for >> the savings on the network (notice how any overhead of the API calls is >> totally >> swamped by network time when there is latency). Again, fixed, but just a >> bigger number than the above. >> >> It is important to look at what the savings is in situations like this and >> where the savings is coming from. I cannot gaurantee that the numbers are >> all >> in the locations I note here, but the concept of what the bulk calls do and >> save and where the savings are (db vs. network) are correct. Also, it shows >> why the numbers that look odd at first glance are much more reasonable and >> make more sense. >> >> You are just swamped here with network and overhead issues that are taking >> the >> majority of the time while the DB side is efficient and giving you gain, but >> a lower percentage of the gain. >> >> I hope this helps explain what is likely occurring and how to interpret your >> data results. >> >> Doug Mueller >> >> -----Original Message----- >> From: Action Request System discussion list(ARSList) >> [mailto:arsl...@arslist.org] On Behalf Of Jarl Grøneng >> Sent: Monday, August 17, 2009 8:16 AM >> To: arslist@ARSLIST.ORG >> Subject: Wierd behavior with Java API -> ar-server >> >> I have a JavaAPI program that reads a csv file, and import the data into a >> form >> >> 1) Import 10.000 records into a regular form, takes appox 70 seconds >> 2) Import 10.000 records into a regular form with bulkimport enabled , >> takes appox 50 seconds: >> >> The time gap between 1 and 2 is mainly one commit to the database. >> >> >> 3) Import 10.000 records into a display only form, takes appox 20 seconds >> 4) Import 10.000 records into a display only forrm with bulkimport >> enabled, takes appox 4 seconds. >> >> >> Why should it be that large difference between 3 and 4? >> >> >> AR server 7.5 patch 2 >> Oracle 10 >> >> (all figures on arserver on my laptop) >> >> -- >> Jarl >> >> _______________________________________________________________________________ >> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org >> Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers Are" >> >> _______________________________________________________________________________ >> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org >> Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers Are" >> > > _______________________________________________________________________________ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers Are" > > _______________________________________________________________________________ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers Are" > _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers Are" NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers Are"
