Re: [U2] Dynamic files, big transactions

Richard Nuckolls Mon, 12 Mar 2007 09:55:41 -0800

Andre,

I am not quite sure what you are doing a transaction on, but even ifyou successfully configured a Universe system to handle transactionswith that many elements simultaneously locked, I would guess that itis inefficient to grab that many locks at once.


Your transaction is too (bloody) large!

So I think that you may need to restructure your process somehow.

The first question that I would have is are you doing a BASICtransaction or a SQL one?

I am assuming that you are doing a BASIC one.

Beyond that, you may need to NOT use a formal TRANSACTION, but ratherkeep track of the state of your update somewhere, and do WRITESoutside of a TRANSACTION block. If you are just creating newrecords, and you ensure that the id is unique, you could save somethrashing by doing WRITES into a file without a previous READU.


One scheme might be:

READU the sequence counter record, increment by the number of"detail" records in the transaction.Create a transaction control record (TCR) in a file that you havecreated for this purpose. perhaps a usernumber together with time()could be used as an id. Include the range of detail ids in the TCR

WRITE (and unlock) the sequence counter.

Create the detail records in a work file (without using transactions,and possible without locking the records before writing)

Update the TCR, indicating that the previous step completed

Copy the records for the TCR's range from the work file to theproduction file, deleting them out of the work file.

stamp the TCR as complete.

-------
This is somewhat crude and incomplete, but you get the idea.
Out of curiosity, what kind of event has that many elements?

Good luck!

-Rick Nuckolls
Lynden, Inc.




On Mar 12, 2007, at 8:10 AM, Andre Meij wrote:

All,
I have done some more digging and I'm now stuck at writing about 6knewrecords within one transaction. I don't know what I might bemissing butdepending on different variations I get a max of 5k or 6k and thena READU
threshold reached error will break the transaction.
Any more insights will be greatly appreciated. I have pasted infoof my last
test below.

Regards,

Andre Meij
Innovate-IT
DELETE.FILE TEST06
DELETEd "TEST06", Type 30.
DELETEd file "D_TEST06", Type 3, Modulo 1.
DELETEd file definition record "TEST06" in the VOC file.
CREATE.FILE TEST06 30 MINIMUM.MODULUS 1024 SEQ.NUM
Creating file "TEST06" as Type 30.
Creating file "D_TEST06" as Type 3, Modulo 1, Separation 2.
Added "@ID", the default record for RetrieVe, to "D_TEST06".
ANALYZE.FILE TEST06
File name ..................   TEST06
Pathname ...................   TEST06
File type ..................   DYNAMIC
Hashing Algorithm ..........   SEQUENTIAL
No. of groups (modulus) ....   1024 current ( minimum 1024 )
Large record size ..........   1628 bytes
Group size .................   2048 bytes
Load factors ............... 80% (split), 50% (merge) and 0%(actual)
Total size .................   2101248 bytes
RUN BP test 15000 50000
Starting writing 50000 records
READU threshold reached, lock on 11521224 denied!
Completed writing 6223 records
Error: WARNING, 'TEST06' record '11521224' locked, try again later.
GROUP.STAT.DETAIL TEST06
 Type description= Hashed, keys end in numbers.
 Bytes  Record.id   File= TEST06  Modulo= 1024  Sep= 4  Type= 30
    32  11518037
    32  11520098
    32  11519718
    32  11516722
    32  11517008
------
   160 Bytes     5 Records in Group 1

    32  11518036
    32  11517009
    32  11520099
    32  11519719
    32  11516723
------
   160 Bytes     5 Records in Group 2

    32  11518038
    32  11517011
    32  11520096
    32  11519716
    32  11516720
LIST.READU (started before the end of the transaction after the READU
error.

Active Record Locks:
Device.... Inode....  Netnode Userno   Lmode        Pid Login Id
Item-ID.............
22282244 1258068 0 11 5 RU 21074 ameij1151501122282244 1258068 0 11 5 RU 21074 ameij1151503322282244 1258068 0 11 5 RU 21074 ameij1151506822282244 1258068 0 11 5 RU 21074 ameij1151510222282244 1258068 0 11 5 RU 21074 ameij1151513522282244 1258068 0 11 5 RU 21074 ameij1151514622282244 1258068 0 11 5 RU 21074 ameij1151516122282244 1258068 0 11 5 RU 21074 ameij1151517922282244 1258068 0 11 5 RU 21074 ameij1151521822282244 1258068 0 11 5 RU 21074 ameij1151523522282244 1258068 0 11 5 RU 21074 ameij1151528422282244 1258068 0 11 5 RU 21074 ameij1151529822282244 1258068 0 11 5 RU 21074 ameij1151531522282244 1258068 0 11 5 RU 21074 ameij1151533422282244 1258068 0 11 5 RU 21074 ameij1151535122282244 1258068 0 11 5 RU 21074 ameij1151536422282244 1258068 0 11 5 RU 21074 ameij1151542022282244 1258068 0 11 5 RU 21074 ameij1151545322282244 1258068 0 11 5 RU 21074 ameij1151546822282244 1258068 0 11 5 RU 21074 ameij1151547722282244 1258068 0 11 5 RU 21074 ameij1151551022282244 1258068 0 11 5 RU 21074 ameij1151554822282244 1258068 0 11 5 RU 21074 ameij1151556522282244 1258068 0 11 5 RU 21074 ameij1151557922282244 1258068 0 11 5 RU 21074 ameij1151560822282244 1258068 0 11 5 RU 21074 ameij1151562722282244 1258068 0 11 5 RU 21074 ameij1151564922282244 1258068 0 11 5 RU 21074 ameij1151568522282244 1258068 0 11 5 RU 21074 ameij1151571822282244 1258068 0 11 5 RU 21074 ameij1151575122282244 1258068 0 11 5 RU 21074 ameij1151576922282244 1258068 0 11 5 RU 21074 ameij1151579122282244 1258068 0 11 5 RU 21074 ameij1151579822282244 1258068 0 11 5 RU 21074 ameij1151582422282244 1258068 0 11 5 RU 21074 ameij1151586022282244 1258068 0 11 5 RU 21074 ameij1151589322282244 1258068 0 11 5 RU 21074 ameij1151592822282244 1258068 0 11 5 RU 21074 ameij1151594522282244 1258068 0 11 5 RU 21074 ameij1151596522282244 1258068 0 11 5 RU 21074 ameij1151599922282244 1258068 0 11 5 RU 21074 ameij1151601322282244 1258068 0 11 5 RU 21074 ameij1151604022282244 1258068 0 11 5 RU 21074 ameij1151604922282244 1258068 0 11 5 RU 21074 ameij1151608222282244 1258068 0 11 5 RU 21074 ameij1151612122282244 1258068 0 11 5 RU 21074 ameij1151613622282244 1258068 0 11 5 RU 21074 ameij1151617122282244 1258068 0 11 5 RU 21074 ameij1151617822282244 1258068 0 11 5 RU 21074 ameij1151621522282244 1258068 0 11 5 RU 21074 ameij1151624022282244 1258068 0 11 5 RU 21074 ameij1151627322282244 1258068 0 11 5 RU 21074 ameij1151629022282244 1258068 0 11 5 RU 21074 ameij1151632322282244 1258068 0 11 5 RU 21074 ameij1151635022282244 1258068 0 11 5 RU 21074 ameij1151635322282244 1258068 0 11 5 RU 21074 ameij1151641022282244 1258068 0 11 5 RU 21074 ameij1151643222282244 1258068 0 11 5 RU 21074 ameij1151645222282244 1258068 0 11 5 RU 21074 ameij1151646922282244 1258068 0 11 5 RU 21074 ameij1151648222282244 1258068 0 11 5 RU 21074 ameij1151649922282244 1258068 0 11 5 RU 21074 ameij1151651222282244 1258068 0 11 5 RU 21074 ameij1151656422282244 1258068 0 11 5 RU 21074 ameij1151658122282244 1258068 0 11 5 RU 21074 ameij1151662122282244 1258068 0 11 5 RU 21074 ameij1151665422282244 1258068 0 11 5 RU 21074 ameij1151666022282244 1258068 0 11 5 RU 21074 ameij1151667222282244 1258068 0 11 5 RU 21074 ameij1151671922282244 1258068 0 11 5 RU 21074 ameij1151672622282244 1258068 0 11 5 RU 21074 ameij1151676722282244 1258068 0 11 5 RU 21074 ameij1151679622282244 1258068 0 11 5 RU 21074 ameij1151682622282244 1258068 0 11 5 RU 21074 ameij1151685922282244 1258068 0 11 5 RU 21074 ameij1151688822282244 1258068 0 11 5 RU 21074 ameij1151691022282244 1258068 0 11 5 RU 21074 ameij1151692122282244 1258068 0 11 5 RU 21074 ameij1151695722282244 1258068 0 11 5 RU 21074 ameij1151699122282244 1258068 0 11 5 RU 21074 ameij1151700522282244 1258068 0 11 5 RU 21074 ameij1151701222282244 1258068 0 11 5 RU 21074 ameij1151705322282244 1258068 0 11 5 RU 21074 ameij1151708322282244 1258068 0 11 5 RU 21074 ameij1151710022282244 1258068 0 11 5 RU 21074 ameij1151711422282244 1258068 0 11 5 RU 21074 ameij1151714522282244 1258068 0 11 5 RU 21074 ameij1151716022282244 1258068 0 11 5 RU 21074 ameij1151719822282244 1258068 0 11 5 RU 21074 ameij1151722822282244 1258068 0 11 5 RU 21074 ameij1151723922282244 1258068 0 11 5 RU 21074 ameij1151725422282244 1258068 0 11 5 RU 21074 ameij1151728722282244 1258068 0 11 5 RU 21074 ameij1151731922282244 1258068 0 11 5 RU 21074 ameij1151733422282244 1258068 0 11 5 RU 21074 ameij1151734722282244 1258068 0 11 5 RU 21074 ameij1151737922282244 1258068 0 11 5 RU 21074 ameij1151740922282244 1258068 0 11 5 RU 21074 ameij1151743922282244 1258068 0 11 5 RU 21074 ameij1151746622282244 1258068 0 11 5 RU 21074 ameij1151748322282244 1258068 0 11 5 RU 21074 ameij1151752022282244 1258068 0 11 5 RU 21074 ameij1151753822282244 1258068 0 11 5 RU 21074 ameij1151755522282244 1258068 0 11 5 RU 21074 ameij1151757822282244 1258068 0 11 5 RU 21074 ameij1151759822282244 1258068 0 11 5 RU 21074 ameij1151761122282244 1258068 0 11 5 RU 21074 ameij1151764622282244 1258068 0 11 5 RU 21074 ameij1151767722282244 1258068 0 11 5 RU 21074 ameij1151769822282244 1258068 0 11 5 RU 21074 ameij1151775222282244 1258068 0 11 5 RU 21074 ameij1151776122282244 1258068 0 11 5 RU 21074 ameij1151778822282244 1258068 0 11 5 RU 21074 ameij1151782322282244 1258068 0 11 5 RU 21074 ameij1151784522282244 1258068 0 11 5 RU 21074 ameij1151785622282244 1258068 0 11 5 RU 21074 ameij1151787322282244 1258068 0 11 5 RU 21074 ameij1151790622282244 1258068 0 11 5 RU 21074 ameij1151792322282244 1258068 0 11 5 RU 21074 ameij1151797822282244 1258068 0 11 5 RU 21074 ameij1151800922282244 1258068 0 11 5 RU 21074 ameij1151802622282244 1258068 0 11 5 RU 21074 ameij1151803522282244 1258068 0 11 5 RU 21074 ameij1151807422282244 1258068 0 11 5 RU 21074 ameij1151811022282244 1258068 0 11 5 RU 21074 ameij1151812322282244 1258068 0 11 5 RU 21074 ameij1151814322282244 1258068 0 11 5 RU 21074 ameij1151817222282244 1258068 0 11 5 RU 21074 ameij1151818922282244 1258068 0 11 5 RU 21074 ameij1151821622282244 1258068 0 11 5 RU 21074 ameij1151825122282244 1258068 0 11 5 RU 21074 ameij1151825722282244 1258068 0 11 5 RU 21074 ameij1151827222282244 1258068 0 11 5 RU 21074 ameij1151830522282244 1258068 0 11 5 RU 21074 ameij1151833722282244 1258068 0 11 5 RU 21074 ameij1151835222282244 1258068 0 11 5 RU 21074 ameij1151837422282244 1258068 0 11 5 RU 21074 ameij1151840622282244 1258068 0 11 5 RU 21074 ameij1151843622282244 1258068 0 11 5 RU 21074 ameij1151845722282244 1258068 0 11 5 RU 21074 ameij1151849522282244 1258068 0 11 5 RU 21074 ameij1151851022282244 1258068 0 11 5 RU 21074 ameij1151854922282244 1258068 0 11 5 RU 21074 ameij1151856722282244 1258068 0 11 5 RU 21074 ameij1151858222282244 1258068 0 11 5 RU 21074 ameij1151860722282244 1258068 0 11 5 RU 21074 ameij1151861622282244 1258068 0 11 5 RU 21074 ameij1151863822282244 1258068 0 11 5 RU 21074 ameij1151866422282244 1258068 0 11 5 RU 21074 ameij1151869822282244 1258068 0 11 5 RU 21074 ameij1151872722282244 1258068 0 11 5 RU 21074 ameij1151878122282244 1258068 0 11 5 RU 21074 ameij1151878822282244 1258068 0 11 5 RU 21074 ameij1151881122282244 1258068 0 11 5 RU 21074 ameij1151884122282244 1258068 0 11 5 RU 21074 ameij1151886622282244 1258068 0 11 5 RU 21074 ameij1151888522282244 1258068 0 11 5 RU 21074 ameij1151890022282244 1258068 0 11 5 RU 21074 ameij1151893522282244 1258068 0 11 5 RU 21074 ameij1151895022282244 1258068 0 11 5 RU 21074 ameij1151898322282244 1258068 0 11 5 RU 21074 ameij1151900122282244 1258068 0 11 5 RU 21074 ameij1151903422282244 1258068 0 11 5 RU 21074 ameij1151907022282244 1258068 0 11 5 RU 21074 ameij1151909722282244 1258068 0 11 5 RU 21074 ameij1151912522282244 1258068 0 11 5 RU 21074 ameij1151914022282244 1258068 0 11 5 RU 21074 ameij1151917422282244 1258068 0 11 5 RU 21074 ameij1151919122282244 1258068 0 11 5 RU 21074 ameij1151920022282244 1258068 0 11 5 RU 21074 ameij1151924322282244 1258068 0 11 5 RU 21074 ameij1151927222282244 1258068 0 11 5 RU 21074 ameij1151930822282244 1258068 0 11 5 RU 21074 ameij1151933922282244 1258068 0 11 5 RU 21074 ameij1151934922282244 1258068 0 11 5 RU 21074 ameij1151937222282244 1258068 0 11 5 RU 21074 ameij1151940522282244 1258068 0 11 5 RU 21074 ameij1151942922282244 1258068 0 11 5 RU 21074 ameij1151944422282244 1258068 0 11 5 RU 21074 ameij1151946222282244 1258068 0 11 5 RU 21074 ameij1151951922282244 1258068 0 11 5 RU 21074 ameij1151953422282244 1258068 0 11 5 RU 21074 ameij1151956122282244 1258068 0 11 5 RU 21074 ameij1151958122282244 1258068 0 11 5 RU 21074 ameij1151959622282244 1258068 0 11 5 RU 21074 ameij1151964222282244 1258068 0 11 5 RU 21074 ameij1151965622282244 1258068 0 11 5 RU 21074 ameij1151970422282244 1258068 0 11 5 RU 21074 ameij1151971322282244 1258068 0 11 5 RU 21074 ameij1151972822282244 1258068 0 11 5 RU 21074 ameij1151974822282244 1258068 0 11 5 RU 21074 ameij1151978322282244 1258068 0 11 5 RU 21074 ameij1151980222282244 1258068 0 11 5 RU 21074 ameij1151981222282244 1258068 0 11 5 RU 21074 ameij1151984322282244 1258068 0 11 5 RU 21074 ameij1151986822282244 1258068 0 11 5 RU 21074 ameij1151988122282244 1258068 0 11 5 RU 21074 ameij1151989022282244 1258068 0 11 5 RU 21074 ameij1151992322282244 1258068 0 11 5 RU 21074 ameij1151993822282244 1258068 0 11 5 RU 21074 ameij1151999722282244 1258068 0 11 5 RU 21074 ameij1152002822282244 1258068 0 11 5 RU 21074 ameij1152004722282244 1258068 0 11 5 RU 21074 ameij1152009522282244 1258068 0 11 5 RU 21074 ameij1152010222282244 1258068 0 11 5 RU 21074 ameij1152011922282244 1258068 0 11 5 RU 21074 ameij1152014222282244 1258068 0 11 5 RU 21074 ameij1152015322282244 1258068 0 11 5 RU 21074 ameij1152016822282244 1258068 0 11 5 RU 21074 ameij1152020222282244 1258068 0 11 5 RU 21074 ameij1152023522282244 1258068 0 11 5 RU 21074 ameij1152027022282244 1258068 0 11 5 RU 21074 ameij1152027622282244 1258068 0 11 5 RU 21074 ameij1152030922282244 1258068 0 11 5 RU 21074 ameij1152032422282244 1258068 0 11 5 RU 21074 ameij1152037722282244 1258068 0 11 5 RU 21074 ameij1152038422282244 1258068 0 11 5 RU 21074 ameij1152042122282244 1258068 0 11 5 RU 21074 ameij1152044022282244 1258068 0 11 5 RU 21074 ameij1152045122282244 1258068 0 11 5 RU 21074 ameij1152047622282244 1258068 0 11 5 RU 21074 ameij1152051122282244 1258068 0 11 5 RU 21074 ameij1152052022282244 1258068 0 11 5 RU 21074 ameij1152055322282244 1258068 0 11 5 RU 21074 ameij1152057722282244 1258068 0 11 5 RU 21074 ameij1152059222282244 1258068 0 11 5 RU 21074 ameij1152061022282244 1258068 0 11 5 RU 21074 ameij1152064822282244 1258068 0 11 5 RU 21074 ameij1152068522282244 1258068 0 11 5 RU 21074 ameij1152068922282244 1258068 0 11 5 RU 21074 ameij1152071922282244 1258068 0 11 5 RU 21074 ameij1152075722282244 1258068 0 11 5 RU 21074 ameij1152079422282244 1258068 0 11 5 RU 21074 ameij1152083022282244 1258068 0 11 5 RU 21074 ameij1152086022282244 1258068 0 11 5 RU 21074 ameij1152087822282244 1258068 0 11 5 RU 21074 ameij1152089322282244 1258068 0 11 5 RU 21074 ameij1152089822282244 1258068 0 11 5 RU 21074 ameij1152091522282244 1258068 0 11 5 RU 21074 ameij1152093522282244 1258068 0 11 5 RU 21074 ameij1152096022282244 1258068 0 11 5 RU 21074 ameij1152099322282244 1258068 0 11 5 RU 21074 ameij1152103022282244 1258068 0 11 5 RU 21074 ameij1152104722282244 1258068 0 11 5 RU 21074 ameij1152105822282244 1258068 0 11 5 RU 21074 ameij1152109022282244 1258068 0 11 5 RU 21074 ameij1152112022282244 1258068 0 11 5 RU 21074 ameij1152115022282244 1258068 0 11 5 RU 21074 ameij1152115922282244 1258068 0 11 5 RU 21074 ameij1152119022282244 1258068 0 11 13 RU 21074 ameij1151501422282244 1258068 0 11 13 RU 21074 ameij1151506722282244 1258068 0 11 13 RU 21074 ameij1151507422282244 1258068 0 11 13 RU 21074 ameij1151509622282244 1258068 0 11 13 RU 21074 ameij1151512922282244 1258068 0 11 13 RU 21074 ameij1151515422282244 1258068 0 11 13 RU 21074 ameij1151516922282244 1258068 0 11 13 RU 21074 ameij11515184
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Andre Meij
Sent: Saturday, March 10, 2007 10:51 AM
To: u2-users@listserver.u2ug.org
Subject: RE: [U2] Dynamic files, big transactions

Rick, Charles, et others,
Thank you for the quick answers. I have quickly tried yoursolutions and
both the initial create with bigger hash and the change+resize on an
existing file seem to work, I can now easily lock 20k records in one
transaction.

I have yet to find out why we hit this issue on live servers in our
application (so why the hash on the affected files is apparently sosmall. I
will take a look at that Monday.


For now I am quite happy because I have a solution for the immediate
problem.
Also I have been trying to SELECT 40k records after the write of20k new
ones and that worked also, I am very confused now because I have been
"brought up" with the idea that there is no thing more disastrousthen aSELECT in a Universe transaction (locking table problems all aroundI am
told.)

Is that something you have heard of? Or is it just another fable?


Thanks again for the help, it is very much appreciated.


Regards,

Andre Meij
Innovate-IT



-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Stevenson,Charles
Sent: Saturday, March 10, 2007 7:04 AM
To: u2-users@listserver.u2ug.org
Subject: RE: [U2] Dynamic files, big transactions

Andre,

I'm with Rick.  He suggested "new partfile".  But maybe some kind of
queue or workfile, that routinely gets flushed, merging to modulo 1.
And maybe zero length record or very small, so that 250 ids allland in
the same group? Is group size 4KB?

What does that have to do with the lock table in memory, you (or some
lurker) may ask?

When a record is locked, UV uses the inode & group# to determine where
to plant the lock in the lock table.   So that means that all these
records will be assigned to the same lock group, since inode & group#
(i.e., 1) will be the same for all.

If you gave it a larger minimum.modulus, or converted that queue/work
file to static, then, when you lock many or all records at once, that
would spread the load across several lock groups, since theinode&group#
combo would vary from record to record.

cds

P.S.  I *think* splits and merges are suspended on groups that have
records currently in the lock table.  (Since group# determines where
something is in the lock table, you couldn't have that beingchanged out
from under you.)  So as long as a record remains locked, your dynamic
file will be not quite so dynamic.  You might be hitting that, too.
-------
u2-users mailing list
u2-users@listserver.u2ug.org
To unsubscribe please visit http://listserver.u2ug.org/
-------
u2-users mailing list
u2-users@listserver.u2ug.org
To unsubscribe please visit http://listserver.u2ug.org/
-------
u2-users mailing list
u2-users@listserver.u2ug.org
To unsubscribe please visit http://listserver.u2ug.org/

-------
u2-users mailing list
u2-users@listserver.u2ug.org
To unsubscribe please visit http://listserver.u2ug.org/

Re: [U2] Dynamic files, big transactions

Reply via email to