I'd agree with this approach...I've used it many times before... A good well thought out design well help you...design first: code second :-)
I've in the past done the following: - control program to configure, start/monitor/stop phantom processes - I find writing a phantom process which logs what it is doing/done is used for debugging purposes too (saves guessing what's going) - just remember to have a toggle to turn this on/off for production (to save disk space) - it is important that each phantom logs its progress, its process ID, start, last checkpoint reached and that it has successfully terminated. This helps prevent the case of you accidently firing off 100 phantoms and consuming all your UV licenses, etc. :-) The purpose of this is to: - make the process scalable and tuneable without re-coding via parameter record (include things like operating windows, etc) - have an application to manage the phantoms and monitor what they're doing (or if they're doing anything) - have phantom processes log what they're doing and allow options for a verbose logging mode to log everything they're doing in case of problems Some approaches I've used: a) each phantom could be started with a unique saved list of record keys to processed (generated by a control program or some other process beforehand) b) each phantom can perform its own query (but sleeping for a specified period so not to continuously performing disk I/O) c) Inbound and outbound transaction phantom process - each handles only response or send requests respectively. Such an approach only works if you can reconcile the response with the original request. Very application specific and not generally optimised for throughput. We first tested this to get ensure we weren't thrashing the disk I/O with too many phantom processes...as this was a pure database read/lock/write type transactional batch process. Use performance monitoring tools to help you do this. Plus your case - networking monitoring, etc. Yours needs to be optimised for sockets I/O as well as DB updates. It shouldn't be opening/closing socket connections unnecessarily (if that wasn't obvious already) - due to the high over head of doing so. Like disk I/O you need to be ensure that the path out to your internet connection is already optimised so it doesn't contribute to the delays. Likewise that it can support X number of connections and your third-party service provider allows you to do this. I'd say you need to optimise your socket program first and think about how to multi-stream (multi-thread implies low-level OS type functions) your processing. Care needs to be taken to ensure you design your process with the with sufficient error handling to ensure that you don't write an application which becomes part of the problem! This is so easy with stuff like this to go amiss. :-) Before you go live - in your test environment - try to break this process (overload/kill phantoms). Better it breaks there, than in production. It's really hard to fix UV problems, when you have no licenses left :-( Good luck. -----Original Message----- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Symeon Breen Sent: Monday, 7 March 2011 8:17 PM To: 'U2 Users List' Subject: Re: [U2] Multi-Threading Universe Socket traffic Ok the python solution mentioned is one way - however this multi threading you requiree is not a multi threading requirement - you are consuming a socket service, and not accepting connections, - we do this kind of thing all the time using phantoms. Lets say your batch of transactions is in a file, as you process each one set a flag saying it is done, or delete the record or something, then yourprogram can select the file, loop through the records and if the flag is set or the record does not exist it just skips onto the next one. You can then start 10 processes running all doing the same thing and they will work through the file. Or you could have process 1 doing all the ones beginning with a 1, 2 for 2 and so on. You may want a controlling program that runs up, counts the records on the file/in the batch and from that determined how many phantoms to run up. It then runs up the phantoms and then stops. Rgds Symeon. From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of nschroth Sent: 05 March 2011 16:55 To: u2-users@listserver.u2ug.org Subject: [U2] Multi-Threading Universe Socket traffic On Universe 10.1.14 over AIX 5.3, we currently communicate Credit Card transactions via sockets (ISO-8583) using the following logic (works fine): OPEN.ERR=openSocket(THIS.IP,THIS.PORT,TCP.MODE,TIMEOUT,THIS.HANDLE) INFO.ERR=getSocketInformation(THIS.HANDLE,PEER.FLAG,SOCKETINFO) WRITE.ERR=writeSocket(THIS.HANDLE,SEND.MSG,TIMEOUT,TCP.MODE,SEND.SIZE) READ.ERR=readSocket(THIS.HANDLE,RECV.MSG,RECV.LEN,TIMEOUT,TCP.MODE,RECV.SIZE ) I am looking into using some sort of multi-threading logic to allow increased volume in shorter timeframes. For example, we now send batches of up to 3,000 cards via a ftps mechanism that responds normally within 3-5 minutes. We want to benchmark doing this via the sockets instead. The Subroutine that does this Socket comm takes about 0.7 secs per trx, so a 3,000 card batch would take over 30 minutes single-threaded (unacceptable). Probably 80% of that 0.7 secs is transforming the data to send and then transforming the response back to process. I hearthat my BASIC applicaton program can utilize PHANTOM processes (unfamiliar territory) to launch multiple requests but I am not sure how to throw multiple trxs over the wall and make sure to put te correct responses back together with the appropriate request. Can anyone point me to some good documentation and/or example code for doing this? ************** IMPORTANT MESSAGE ***************************** This e-mail message is intended only for the addressee(s) and contains information which may be confidential. If you are not the intended recipient please advise the sender by return email, do not use or disclose the contents, and delete the message and any attachments from your system. Unless specifically indicated, this email does not constitute formal advice or commitment by the sender or the Commonwealth Bank of Australia (ABN 48 123 123 124) or its subsidiaries. We can be contacted through our web site: commbank.com.au. If you no longer wish to receive commercial electronic messages from us, please reply to this e-mail by typing Unsubscribe in the subject line. ************************************************************** _______________________________________________ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
