Hi Pawel, The BATCH.JOB.,CONTROL works nearly as you describe;
The select JOB.LIST SAMPLE is simply to see if there is work remaining, it does not create the workload for an agent; instead a UNIQIE chunk of the JOB.LIST is taken for processing. This can be seen in the COMO where it says; * Processing - is the total number of records in the JOB.LIST at that time * Offset - is the starting position in the list of records that this agent will process INT(((Total Number to Process / Number of Sessions) * My session Number) - (Total Number to Process / Number of Sessions)) * Number of Sessions - the number of agents currently contributing to this job. The records processed by the agent will be the section of the job list starting at the offset and will contain (processing/number of sessions) records - this effectively means that each agent does process a specific workload so no need to hold a sperate file for each agent. This only happens if there are 10*the number of agents worth of records to process, in this case the whole list is selected by each agent. Other than the SAMPLE issue the mechanism works more or less as you describe. You are quite right that sometimes a row in the job list file may contain a number of explicit IDs - this is a mechanism to group particular kinds of data and avoud locking. If you really want to prove this then create a sample set of routines where the record routine simply prints the ID to be processed - you will see there is no conflict until near the end of the job (less than the required number of records to make the splitting worthwhile) The JOB.LIST file number is the lowest unused number found from checking the F.LOCKING table - if you have orphan records in here then the job list files could be high. There are times when a particular job could get different job files from one run to the next - this is if different BATCH records share the same stage and hence the jobs can run concurrently. HTH John. On Aug 18, 1:01 am, "Pawel (privately)" <pprivat...@wp.pl> wrote: > Hi guys, > > Thank you very much for discussion, which was started by Adrian. I should > likely create separate thread not to abuse his original post. Anyway... > > Unfortunately I do not understand FASTSCAN interface. I know that there is > JBASE_DISTRIB_FASTSCAN environment variable. It is already set to 1 in our > config. I am not sure however of full meaning of the variable. Jim explained > partly in the past and I could not understand difference / benefits from > normal file traversal. I should also check my work mail - perhaps I have > explanation and simply do not remember anymore. > > T24 R06 is currently doing something like that (worker thread concept): > 1. Allocate JOB.LIST.x file name (name completely unpredictable - can be > JOB.LIST.1 or JOB.LIST.7 or something else if job is really old) > 2. Store ids/transaction data to be processed. Single recordid may contain > from 1 to X transaction data inside. > 3. Each agent: > SELECT joblist SAMPLE number > process transaction/transaction(s) that are not being processed (locked) > again SELECT joblist SAMPLE number.... > and so on, and so on... > > Above pattern could be changed to avoid unnecessary SELECT joblist SAMPLE > number. Say agent could keep his own, local list of items (SELECT joblist > SAMPLE number result): > a) if item not existent remove it from locally kept list (already processed > by other agent) > b) if item locked keep it still on the list and count (being processed, but > agent may fatal out for any reason (DEADLOCK eg.)) > c) if item not locked - process and remove from the list > d) if list empty or count<number - SELECT joblist SAMPLE number > 5000 as number should be fine (should be fine for 500 agents 10 servers each) > Unnecessary SELECTs should be avoided. JOBLIST files can not be marked > > I would prefer also that JOB.LIST names are unique and PREDICTABLE! JOB.LIST > file is though thing for jBASE, because of right sizing and variable data > volume. You know however that there are big jobs with more transactions and > smaller jobs with less transactions - but you can not size joblists well! > Currently joblist files needs to be sized (more or less) the same way, which > is not perfect. AFAIK JR is not available for jBASE 4.1. > > Last thing (jBASE 4.1) > We see that SELECT joblist SAMPLE number (traversing file) causes a lot of > locks to be taken (each group is binary locked). I will try to paste truss > output. That is normal, right? FASTSCAN could change it? (but joblist files > are SELECTed and updated at the same time!) > > NB. Jim I saw your (very old) post > -http://groups.google.com/group/jBASE/browse_thread/thread/6133536abb4.... > Perhaphs you would like to give us trial of you "J5" filetype? :) > > I will try to give you more information about (P7) problems we face. > > Kind regards > Pawel > > Dnia 17-08-2011 o godz. 18:35 Jim Idle napisa (a): > > > > > Actually, the locking behavior you suggest will only be the case so long > > as the select is in hash order and not a SSELECT. A SSELECT will cause > > different batch processes to access the same groups simultaneously almost > > immediately, with a probability related to the number of groups (assuming > > well sized) and the number of batch processes. > > > Also don't forget that batch inputs will be processed better if you make > > the files read only and do not delete anything until a cleanup at the end > > (of all processing). Knowing that there is no problem with read/update > > locking, J4 files will not take the locks (unless someone changed this in > > jBASE 5 for instance). Also, you should look in to FASTSCAN as application > > programmers. > > > Jim > > > > -----Original Message----- > > > From: jbase@googlegroups.com [mailto:jbase@googlegroups.com] On Behalf > > > Of John Watson > > > Sent: Wednesday, August 17, 2011 2:59 AM > > > To: jBASE > > > Subject: Re: jbase problems show-items-locks > > > > Hi Pawel, > > > > You have said > > > > 'In our case high 'lock collision' ratio on group locks is killing > > > performance. JOB.LIST.x files suffer from multiple, concurrent 'SELECTs > > > SAMPLE' (~ full scan) and multiple, concurrent READUs/WRITEs' > > > > I'm not sure what release you use at the moment but I would only expect > > > to see the above problem towards the end of a job when a high number of > > > agents are being used - at the start of the job a list is created and > > > each agent takes 1/number of agents * total number of records as the > > > agent work load - there should be no duplication of records across > > > agents at this initial stage (but of course there could be a > > > duplication of groups) > > > > When this workload is consumed a full select will take place on the > > > JOB.LIST file and hence we can have more issues with collisions on > > > groups, records processed by other agents etc. > > > > Are you sure that the problem is actually with the JOB.LIST table and > > > not locks on BATCH.STATUS or BATCH - there have been problems in this > > > area in much older releases and they should now be addressed. > > > > If the issue is with group locks on the JOB.LIST file and the overhead > > > really is substantial then perhaps there is a logical argument to > > > oversize the file to minimise the number of records in a group - to be > > > worthwhile the increase in select time would have to outweigh the time > > > lost to group locks. > > > > Cheers, > > > > John.- Hide quoted text - > > - Show quoted text - -- Please read the posting guidelines at: http://groups.google.com/group/jBASE/web/Posting%20Guidelines IMPORTANT: Type T24: at the start of the subject line for questions specific to Globus/T24 To post, send email to jBASE@googlegroups.com To unsubscribe, send email to jbase-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/jBASE?hl=en
