Only outside of U2 using UniObjects can you achieve any type of parallel activity. We have through UniObjects got 80 processes working from a single Eclipse session through the use of threads in Java.
UniObjects creates individual uvapi_slave or udapi_slave for each of these processes but the system or in this case the udapi_server or uvadpi_server cannot handle as many threads as we would like. We never ran out of memory on our 8GB Windows 2008R2 Server nor did SSD 120GB drive fail to keep up with the 80 ANALYZE.FILES or the 80 RESIZE commands we were issuing on from our XLr8Resizer product within Eclipse. The only way we got this working was to set the retries to 1000 on reopening the connections. Although that number seems high it helped and get us from our previous best of 39 process to 80 process. When we have a lot of time and cannot think of anything better to do we will try for 500 process. Regards, Doug www.u2logic.com "Eclipse based tools for the U2 programmer" > > -----Original Message----- > From: u2-users-boun...@listserver.u2ug.org [mailto: > u2-users-boun...@listserver.u2ug.org] On Behalf Of HENDERSON MIKE, MR > Sent: Tuesday, 2 October 2012 1:18 PM > To: U2 Users List > Subject: Re: [U2] [u2] Parallel processing in Universe (Unclassified) > > I have often thought about this - mostly in an idle moment or as a > displacement activity for something less amusing that I ought to be doing. > ;-) > > > First of all, Universe is already extremely parallel: there's a separate > O/S thread for each TTY and for each phantom, and you can't get more > parallel than that for interactive processing. > > So you want more parallelism for your batch processes. > Different applications have different degrees of inherent parallelism. > For example in utility billing systems there is frequently the concept of > a group of premises - based on the old concept of a foot-borne meter reader > with a 'book' of readings to get. Each 'book' can be processed > independently of every other. In payroll, each employee's record can be > processed independently. Other areas of commerce have different > characteristics. > > I think that whatever unit of parallelism you settle for, you'd need three > processes: a 'dispatcher' that selects records for processing and queues > them into some structure for processing; a set of 'workers' that take > queued work items, process them, mark them as processed and put the results > in some common store; and a 'monitor' that looks for unprocessed records > and indications of stuck processes, and collates the results for final > output. > I've seen a couple of versions of this, one for electricity billings and > another for overnight batch-processing of report requests, both well over a > decade ago, and neither still in use although their underlying packages are > still being run. > > The major issue is that these days the whole entity in the general > commercial world is far more likely to be I/O limited than CPU limited, and > therefore introducing parallelism will be no help at all if the I/O system > is already choked. > Even if the system is currently CPU-limited, multi-threading may not > produce much improvement without very careful design of the record locking > philosophy - introducing parallelism will be no help if all the threads end > up contending serially for one record lock or a small set of locks. > > > If you want it to go faster, buy the CPU with the fastest clock you can > get (not the one with the most cores), and put your database on SSD like > Ross said. > The Power7+ chips being announced any day now are rumoured to go to > 5GHz+, maybe even more if you have half the cores on the chip disabled. > > > Regards > > > Mike > > -----Original Message----- > From: u2-users-boun...@listserver.u2ug.org > [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Ross Ferris > Sent: Tuesday, 2 October 2012 3:50 p.m. > To: U2 Users List > Subject: Re: [U2] [u2] Parallel processing in Universe > > If the file were big enough, and already had part files, then I believe > that you could have a phantom process each of the individual parts. > Failing that, get an SSD .... relatively cheap, and will give your > processing a reasonable kick along!! > > Ross Ferris > Stamina Software > Visage > Better by Design! > > > -----Original Message----- > From: u2-users-boun...@listserver.u2ug.org > [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of David Wolverton > Sent: Tuesday, 2 October 2012 8:47 AM > To: 'U2 Users List' > Subject: Re: [U2] [u2] Parallel processing in Universe > > OK - I was trying to create a 'smoother use' of the disk and 'read ahead' > -- this example the disk would be chattering from the heads moving all over > the place. I was trying to find a way to make this process more 'orderly' > -- is there one? > > -----Original Message----- > From: u2-users-boun...@listserver.u2ug.org > [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Robert Houben > Sent: Monday, October 01, 2012 4:48 PM > To: U2 Users List > Subject: Re: [U2] [u2] Parallel processing in Universe > > Create an index on a dict pointing at the first character of the key, and > have each phantom take two digits. (0-1, 2-3, 4-5, 6-7, 8-9) > > -----Original Message----- > From: u2-users-boun...@listserver.u2ug.org > [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of David Wolverton > Sent: October-01-12 2:43 PM > To: 'U2 Users List' > Subject: Re: [U2] [u2] Parallel processing in Universe > > So how would a user 'chop up' a file for parallel processing? Ideally, if > here was a Mod 10001 file (or whatever) it would seem like it would be > 'ideal' to assign 2000 groups to 5 phantoms -- but I don't know how 'start > a BASIC select at Group 2001 or 4001' ... > > -----Original Message----- > From: u2-users-boun...@listserver.u2ug.org > [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of George Gallen > Sent: Monday, October 01, 2012 3:29 PM > To: U2 Users List > Subject: Re: [U2] [u2] Parallel processing in Universe > > 0001: OPENSEQ "/tmp/pipetest" TO F.PIPE ELSE STOP "NO PIPE" > 0002: LOOP > 0003: READSEQ LINE FROM F.PIPE ELSE CONTINUE > 0004: PRINT LINE > 0005: REPEAT > 0006: STOP > 0007: END > > Although, not sure if you might need to sleep a litte between the > READSEQ's ELSE and CONTINUE > Might suck up cpu time when nothing is writing to the file. > > Then you could setup a printer in UV that did a "cat - >> /tmp/pipetest" > > Now your phantom just needs to print to that printer. > > George > > -----Original Message----- > From: u2-users-boun...@listserver.u2ug.org > [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of George Gallen > Sent: Monday, October 01, 2012 4:16 PM > To: U2 Users List > Subject: Re: [U2] [u2] Parallel processing in Universe > > The only thing about a pipe is that once it's closed, I believe it has to > be re-opened by both Ends again. So if point a opens one end, and point b > opens the other end, once either end closes, It closes for both sides, and > both sides would have to reopen again to use. > > To eliminate this, you could have one end open a file, and have the other > sides do a ">>" append To that file - just make sure you include some kind > of dataheader so the reading side knows which Process just wrote the data. > > -----Original Message----- > From: u2-users-boun...@listserver.u2ug.org > [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of u2ug > Sent: Monday, October 01, 2012 4:11 PM > To: U2 Users List > Subject: Re: [U2] [u2] Parallel processing in Universe > > pipes > > > -----Original Message----- > From: u2-users-boun...@listserver.u2ug.org > [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson > Sent: Monday, October 01, 2012 4:05 PM > To: u2-users@listserver.u2ug.org > Subject: [U2] [u2] Parallel processing in Universe > > > What's the largest dataset in the Universe user world? > In terms of number of records. > > I'm wondering if we have any potential for utilities that map-reduce. > I suppose you would spawn phantoms but how do they communicate back to the > master node? > _______________________________________________ > U2-Users mailing list > U2-Users@listserver.u2ug.org > http://listserver.u2ug.org/mailman/listinfo/u2-users > > > _______________________________________________ > U2-Users mailing list > U2-Users@listserver.u2ug.org > http://listserver.u2ug.org/mailman/listinfo/u2-users > _______________________________________________ > U2-Users mailing list > U2-Users@listserver.u2ug.org > http://listserver.u2ug.org/mailman/listinfo/u2-users > _______________________________________________ > U2-Users mailing list > U2-Users@listserver.u2ug.org > http://listserver.u2ug.org/mailman/listinfo/u2-users > > _______________________________________________ > U2-Users mailing list > U2-Users@listserver.u2ug.org > http://listserver.u2ug.org/mailman/listinfo/u2-users > _______________________________________________ > U2-Users mailing list > U2-Users@listserver.u2ug.org > http://listserver.u2ug.org/mailman/listinfo/u2-users > > _______________________________________________ > U2-Users mailing list > U2-Users@listserver.u2ug.org > http://listserver.u2ug.org/mailman/listinfo/u2-users > _______________________________________________ > U2-Users mailing list > U2-Users@listserver.u2ug.org > http://listserver.u2ug.org/mailman/listinfo/u2-users > The information contained in this Internet Email message is intended for > the addressee only and may contain privileged information, but not > necessarily the official views or opinions of the New Zealand Defence Force. > If you are not the intended recipient you must not use, disclose, copy or > distribute this message or the information in it. > > If you have received this message in error, please Email or telephone the > sender immediately. > _______________________________________________ > U2-Users mailing list > U2-Users@listserver.u2ug.org > http://listserver.u2ug.org/mailman/listinfo/u2-users > _______________________________________________ > U2-Users mailing list > U2-Users@listserver.u2ug.org > http://listserver.u2ug.org/mailman/listinfo/u2-users > _______________________________________________ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
