Hi guys, Thank you very much for discussion, which was started by Adrian. I should likely create separate thread not to abuse his original post. Anyway...
Unfortunately I do not understand FASTSCAN interface. I know that there is JBASE_DISTRIB_FASTSCAN environment variable. It is already set to 1 in our config. I am not sure however of full meaning of the variable. Jim explained partly in the past and I could not understand difference / benefits from normal file traversal. I should also check my work mail - perhaps I have explanation and simply do not remember anymore. T24 R06 is currently doing something like that (worker thread concept): 1. Allocate JOB.LIST.x file name (name completely unpredictable - can be JOB.LIST.1 or JOB.LIST.7 or something else if job is really old) 2. Store ids/transaction data to be processed. Single recordid may contain from 1 to X transaction data inside. 3. Each agent: SELECT joblist SAMPLE number process transaction/transaction(s) that are not being processed (locked) again SELECT joblist SAMPLE number.... and so on, and so on... Above pattern could be changed to avoid unnecessary SELECT joblist SAMPLE number. Say agent could keep his own, local list of items (SELECT joblist SAMPLE number result): a) if item not existent remove it from locally kept list (already processed by other agent) b) if item locked keep it still on the list and count (being processed, but agent may fatal out for any reason (DEADLOCK eg.)) c) if item not locked - process and remove from the list d) if list empty or count<number - SELECT joblist SAMPLE number 5000 as number should be fine (should be fine for 500 agents 10 servers each) Unnecessary SELECTs should be avoided. JOBLIST files can not be marked I would prefer also that JOB.LIST names are unique and PREDICTABLE! JOB.LIST file is though thing for jBASE, because of right sizing and variable data volume. You know however that there are big jobs with more transactions and smaller jobs with less transactions - but you can not size joblists well! Currently joblist files needs to be sized (more or less) the same way, which is not perfect. AFAIK JR is not available for jBASE 4.1. Last thing (jBASE 4.1) We see that SELECT joblist SAMPLE number (traversing file) causes a lot of locks to be taken (each group is binary locked). I will try to paste truss output. That is normal, right? FASTSCAN could change it? (but joblist files are SELECTed and updated at the same time!) NB. Jim I saw your (very old) post - http://groups.google.com/group/jBASE/browse_thread/thread/6133536abb4c30e7/0d4703621957b4d7?hl=en&lnk=gst&q=file+system#0d4703621957b4d7. Perhaphs you would like to give us trial of you "J5" filetype? :) I will try to give you more information about (P7) problems we face. Kind regards Pawel Dnia 17-08-2011 o godz. 18:35 Jim Idle napisał(a): > Actually, the locking behavior you suggest will only be the case so long > as the select is in hash order and not a SSELECT. A SSELECT will cause > different batch processes to access the same groups simultaneously almost > immediately, with a probability related to the number of groups (assuming > well sized) and the number of batch processes. > > Also don't forget that batch inputs will be processed better if you make > the files read only and do not delete anything until a cleanup at the end > (of all processing). Knowing that there is no problem with read/update > locking, J4 files will not take the locks (unless someone changed this in > jBASE 5 for instance). Also, you should look in to FASTSCAN as application > programmers. > > Jim > > > -----Original Message----- > > From: jbase@googlegroups.com [mailto:jbase@googlegroups.com] On Behalf > > Of John Watson > > Sent: Wednesday, August 17, 2011 2:59 AM > > To: jBASE > > Subject: Re: jbase problems show-items-locks > > > > Hi Pawel, > > > > You have said > > > > 'In our case high 'lock collision' ratio on group locks is killing > > performance. JOB.LIST.x files suffer from multiple, concurrent 'SELECTs > > SAMPLE' (~ full scan) and multiple, concurrent READUs/WRITEs' > > > > I'm not sure what release you use at the moment but I would only expect > > to see the above problem towards the end of a job when a high number of > > agents are being used - at the start of the job a list is created and > > each agent takes 1/number of agents * total number of records as the > > agent work load - there should be no duplication of records across > > agents at this initial stage (but of course there could be a > > duplication of groups) > > > > When this workload is consumed a full select will take place on the > > JOB.LIST file and hence we can have more issues with collisions on > > groups, records processed by other agents etc. > > > > Are you sure that the problem is actually with the JOB.LIST table and > > not locks on BATCH.STATUS or BATCH - there have been problems in this > > area in much older releases and they should now be addressed. > > > > If the issue is with group locks on the JOB.LIST file and the overhead > > really is substantial then perhaps there is a logical argument to > > oversize the file to minimise the number of records in a group - to be > > worthwhile the increase in select time would have to outweigh the time > > lost to group locks. > > > > Cheers, > > > > John. -- Please read the posting guidelines at: http://groups.google.com/group/jBASE/web/Posting%20Guidelines IMPORTANT: Type T24: at the start of the subject line for questions specific to Globus/T24 To post, send email to jBASE@googlegroups.com To unsubscribe, send email to jbase-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/jBASE?hl=en
