Re: [U2] Multi-Threading Universe Socket traffic
Does anyone have a good method to 'throttle' the number of child processes? We authorize a few $million per day in CC transactions doing each 'near' real-time auths as the orders arrive. This is single threaded master phantom opening / writing/ reading the socket for each transaction. Works fairly well. Very reliable but not real time enough to hold the customer on the phone till the 'approve' comes back. I've toyed with having the master phantom spawn child phantom processes but need a way to 'throttle' the thread count, and queue anything over a specified threshold. The method I tried, grabbing a 'token' from a control record when the child phantom spawns, and replacing it when the child process concludes does not work. I found that this becomes unworkable after 4 or 5 child phantoms get spawned. They conclude closely enough that the contention over the update lock for the 'tokens' record degrades any performance gain by going multi-threaded. Seems I can get an auth faster than the control record can be locked, updated, and written back. I fully agree with everything David said about logging the pid and have done this also. @nschroth - if you want some code snippets of the socket calls email me off list. Thank you. -Baker -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Hona, David Sent: Monday, March 07, 2011 7:04 PM To: 'U2 Users List' Subject: Re: [U2] Multi-Threading Universe Socket traffic I'd agree with this approach...I've used it many times before... A good well thought out design well help you...design first: code second :-) I've in the past done the following: - control program to configure, start/monitor/stop phantom processes - I find writing a phantom process which logs what it is doing/done is used for debugging purposes too (saves guessing what's going) - just remember to have a toggle to turn this on/off for production (to save disk space) - it is important that each phantom logs its progress, its process ID, start, last checkpoint reached and that it has successfully terminated. This helps prevent the case of you accidently firing off 100 phantoms and consuming all your UV licenses, etc. :-) The purpose of this is to: - make the process scalable and tuneable without re-coding via parameter record (include things like operating windows, etc) - have an application to manage the phantoms and monitor what they're doing (or if they're doing anything) - have phantom processes log what they're doing and allow options for a verbose logging mode to log everything they're doing in case of problems Some approaches I've used: a) each phantom could be started with a unique saved list of record keys to processed (generated by a control program or some other process beforehand) b) each phantom can perform its own query (but sleeping for a specified period so not to continuously performing disk I/O) c) Inbound and outbound transaction phantom process - each handles only response or send requests respectively. Such an approach only works if you can reconcile the response with the original request. Very application specific and not generally optimised for throughput. Good luck. From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of nschroth Sent: 05 March 2011 16:55 To: u2-users@listserver.u2ug.org Subject: [U2] Multi-Threading Universe Socket traffic On Universe 10.1.14 over AIX 5.3, we currently communicate Credit Card transactions via sockets (ISO-8583) using the following logic (works fine): OPEN.ERR=openSocket(THIS.IP,THIS.PORT,TCP.MODE,TIMEOUT,THIS.HANDLE) INFO.ERR=getSocketInformation(THIS.HANDLE,PEER.FLAG,SOCKETINFO) WRITE.ERR=writeSocket(THIS.HANDLE,SEND.MSG,TIMEOUT,TCP.MODE,SEND.SIZE) READ.ERR=readSocket(THIS.HANDLE,RECV.MSG,RECV.LEN,TIMEOUT,TCP.MODE,RECV.SIZE ) I am looking into using some sort of multi-threading logic to allow increased volume in shorter timeframes. For example, we now send batches of up to 3,000 cards via a ftps mechanism that responds normally within 3-5 minutes. We want to benchmark doing this via the sockets instead. The Subroutine that does this Socket comm takes about 0.7 secs per trx, so a 3,000 card batch would take over 30 minutes single-threaded (unacceptable). Probably 80% of that 0.7 secs is transforming the data to send and then transforming the response back to process. This communication, its contents and any file attachments transmitted with it are intended solely for the addressee(s) and may contain confidential proprietary information. Access by any other party without the express written permission of the sender is STRICTLY PROHIBITED. If you have received this communication in error you may not copy, distribute or use the contents, attachments or information in any way. Please destroy it and contact the sender
Re: [U2] Multi-Threading Universe Socket traffic
Baker, It sounds like you need some kind of scheduler setup. Instead of running things directly via one phantom you should consider putting the work in a queue and then scheduling the work processing cycle to be handled outside of the data request and return processes. You can avoid lock contention if you use a write-only and read-only file FIFO pair keyed by a worker phantom reference ID. Let each child monitor a shared request queue and pick things up in order of submission, keyed by each child's reference ID. Latency will be reduced as you add more master or worker phantoms depending on where the lag resides. Master phantoms: 1) get request (call, poll outside work queue, TCL execution, etc) 2) generate a sequential and unique ID based on round-robin selection of active phantom reference IDs 3) write request content to request-side of FIFO 4) wait for a response to be returned on the response-side of FIFO 5) return response to calling subject Child phantoms: 1) check processing queue and get first item ID matching the phantom's reference ID 2) read process request from FIFO per currently read ID 3) perform work and build response 4) write response to response-side of FIFO using the same ID Bear in mind that there is startup, shutdown and reinit logic that is not covered here. This is just an overview of how you can avoid lock contention with a FIFO pair and some strategic item ID usage. If you're interested I can give a more detailed example based on how I built the MVWWW request handler using file I/O. Regards, Glen Batchelor IT Director/CIO/CTO All-Spec Industries phone: (910) 332-0424 fax: (910) 763-5664 E-mail: webmas...@all-spec.com Web: http://www.all-spec.com Blog: http://blog.all-spec.com -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users- boun...@listserver.u2ug.org] On Behalf Of Baker Hughes Sent: Tuesday, March 08, 2011 4:07 PM To: 'U2 Users List' Subject: Re: [U2] Multi-Threading Universe Socket traffic Does anyone have a good method to 'throttle' the number of child processes? We authorize a few $million per day in CC transactions doing each 'near' real-time auths as the orders arrive. This is single threaded master phantom opening / writing/ reading the socket for each transaction. Works fairly well. Very reliable but not real time enough to hold the customer on the phone till the 'approve' comes back. I've toyed with having the master phantom spawn child phantom processes but need a way to 'throttle' the thread count, and queue anything over a specified threshold. The method I tried, grabbing a 'token' from a control record when the child phantom spawns, and replacing it when the child process concludes does not work. I found that this becomes unworkable after 4 or 5 child phantoms get spawned. They conclude closely enough that the contention over the update lock for the 'tokens' record degrades any performance gain by going multi-threaded. Seems I can get an auth faster than the control record can be locked, updated, and written back. I fully agree with everything David said about logging the pid and have done this also. @nschroth - if you want some code snippets of the socket calls email me off list. Thank you. -Baker -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users- boun...@listserver.u2ug.org] On Behalf Of Hona, David Sent: Monday, March 07, 2011 7:04 PM To: 'U2 Users List' Subject: Re: [U2] Multi-Threading Universe Socket traffic I'd agree with this approach...I've used it many times before... A good well thought out design well help you...design first: code second :-) I've in the past done the following: - control program to configure, start/monitor/stop phantom processes - I find writing a phantom process which logs what it is doing/done is used for debugging purposes too (saves guessing what's going) - just remember to have a toggle to turn this on/off for production (to save disk space) - it is important that each phantom logs its progress, its process ID, start, last checkpoint reached and that it has successfully terminated. This helps prevent the case of you accidently firing off 100 phantoms and consuming all your UV licenses, etc. :-) The purpose of this is to: - make the process scalable and tuneable without re-coding via parameter record (include things like operating windows, etc) - have an application to manage the phantoms and monitor what they're doing (or if they're doing anything) - have phantom processes log what they're doing and allow options for a verbose logging mode to log everything they're doing in case of problems Some approaches I've used: a) each phantom could be started with a unique saved list of record keys to processed (generated by a control program or some other
Re: [U2] Multi-Threading Universe Socket traffic
And remember, at the more recent UniData/UniVerse, this use of sockets will require a 'real' seat license -- it will make a 'phantom' (free) into an iPhantom (licensed) -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Symeon Breen Sent: Monday, March 07, 2011 3:17 AM To: 'U2 Users List' Subject: Re: [U2] Multi-Threading Universe Socket traffic Ok the python solution mentioned is one way - however this multi threading you requiree is not a multi threading requirement - you are consuming a socket service, and not accepting connections, - we do this kind of thing all the time using phantoms. Lets say your batch of transactions is in a file, as you process each one set a flag saying it is done, or delete the record or something, then yourprogram can select the file, loop through the records and if the flag is set or the record does not exist it just skips onto the next one. You can then start 10 processes running all doing the same thing and they will work through the file. Or you could have process 1 doing all the ones beginning with a 1, 2 for 2 and so on. You may want a controlling program that runs up, counts the records on the file/in the batch and from that determined how many phantoms to run up. It then runs up the phantoms and then stops. Rgds Symeon. From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of nschroth Sent: 05 March 2011 16:55 To: u2-users@listserver.u2ug.org Subject: [U2] Multi-Threading Universe Socket traffic On Universe 10.1.14 over AIX 5.3, we currently communicate Credit Card transactions via sockets (ISO-8583) using the following logic (works fine): OPEN.ERR=openSocket(THIS.IP,THIS.PORT,TCP.MODE,TIMEOUT,THIS.HANDLE) INFO.ERR=getSocketInformation(THIS.HANDLE,PEER.FLAG,SOCKETINFO) WRITE.ERR=writeSocket(THIS.HANDLE,SEND.MSG,TIMEOUT,TCP.MODE,SEND.SIZE) READ.ERR=readSocket(THIS.HANDLE,RECV.MSG,RECV.LEN,TIMEOUT,TCP.MODE,RECV.SIZE ) I am looking into using some sort of multi-threading logic to allow increased volume in shorter timeframes. For example, we now send batches of up to 3,000 cards via a ftps mechanism that responds normally within 3-5 minutes. We want to benchmark doing this via the sockets instead. The Subroutine that does this Socket comm takes about 0.7 secs per trx, so a 3,000 card batch would take over 30 minutes single-threaded (unacceptable). Probably 80% of that 0.7 secs is transforming the data to send and then transforming the response back to process. I hearthat my BASIC applicaton program can utilize PHANTOM processes (unfamiliar territory) to launch multiple requests but I am not sure how to throw multiple trxs over the wall and make sure to put te correct responses back together with the appropriate request. Can anyone point me to some good documentation and/or example code for doing this? -- View this message in context: http://old.nabble.com/Multi-Threading-Universe-Socket-traffic-tp31076011p310 76011.html Sent from the U2 - Users mailing list archive at Nabble.com. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users _ No virus found in this message. Checked by AVG - www.avg.com Version: 10.0.1204 / Virus Database: 1435/3482 - Release Date: 03/04/11 ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Multi-Threading Universe Socket traffic
Symeon, and others, How does a multi-threading strategy relate to the use of a multi-core, multi-cpu and/or clustered hardware server configuration using various operating systems and multi-value databases in terms of performance and thruput? (A complete answer gets you an honorary PhD in whatever subject you'd like). Dave Taylor Sysmark Information Systems, Inc. 49 Aspen Way Rolling Hills Estates, CA 90274 (O) 800-SYSMARK (800-797-6275) (F) 310-377-3550 (C) 310-561-5200 www.sysmarkinfo.com - Original Message - From: Symeon Breen syme...@gmail.com To: 'U2 Users List' u2-users@listserver.u2ug.org Sent: Monday, March 07, 2011 1:16 AM Subject: Re: [U2] Multi-Threading Universe Socket traffic Ok the python solution mentioned is one way - however this multi threading you requiree is not a multi threading requirement - you are consuming a socket service, and not accepting connections, - we do this kind of thing all the time using phantoms. Lets say your batch of transactions is in a file, as you process each one set a flag saying it is done, or delete the record or something, then yourprogram can select the file, loop through the records and if the flag is set or the record does not exist it just skips onto the next one. You can then start 10 processes running all doing the same thing and they will work through the file. Or you could have process 1 doing all the ones beginning with a 1, 2 for 2 and so on. You may want a controlling program that runs up, counts the records on the file/in the batch and from that determined how many phantoms to run up. It then runs up the phantoms and then stops. Rgds Symeon. From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of nschroth Sent: 05 March 2011 16:55 To: u2-users@listserver.u2ug.org Subject: [U2] Multi-Threading Universe Socket traffic On Universe 10.1.14 over AIX 5.3, we currently communicate Credit Card transactions via sockets (ISO-8583) using the following logic (works fine): OPEN.ERR=openSocket(THIS.IP,THIS.PORT,TCP.MODE,TIMEOUT,THIS.HANDLE) INFO.ERR=getSocketInformation(THIS.HANDLE,PEER.FLAG,SOCKETINFO) WRITE.ERR=writeSocket(THIS.HANDLE,SEND.MSG,TIMEOUT,TCP.MODE,SEND.SIZE) READ.ERR=readSocket(THIS.HANDLE,RECV.MSG,RECV.LEN,TIMEOUT,TCP.MODE,RECV.SIZE ) I am looking into using some sort of multi-threading logic to allow increased volume in shorter timeframes. For example, we now send batches of up to 3,000 cards via a ftps mechanism that responds normally within 3-5 minutes. We want to benchmark doing this via the sockets instead. The Subroutine that does this Socket comm takes about 0.7 secs per trx, so a 3,000 card batch would take over 30 minutes single-threaded (unacceptable). Probably 80% of that 0.7 secs is transforming the data to send and then transforming the response back to process. I hearthat my BASIC applicaton program can utilize PHANTOM processes (unfamiliar territory) to launch multiple requests but I am not sure how to throw multiple trxs over the wall and make sure to put te correct responses back together with the appropriate request. Can anyone point me to some good documentation and/or example code for doing this? -- View this message in context: http://old.nabble.com/Multi-Threading-Universe-Socket-traffic-tp31076011p310 76011.html Sent from the U2 - Users mailing list archive at Nabble.com. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users _ No virus found in this message. Checked by AVG - www.avg.com Version: 10.0.1204 / Virus Database: 1435/3482 - Release Date: 03/04/11 ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Multi-Threading Universe Socket traffic
Dave, That's a complex answer and you can't really answer it properly in a forum for such a broad range of setups. I'll try to answer as generically as possible focusing on O/S and CPU threading specifically. Please correct me if I'm wrong here. You don't really need multiple processors to handle multi-threading. Multi-threading has more to do with time slicing, priority marking and keeping up with all of the threads running at the same time. At the same time can vary from taking turns on one merry-go-round or taking turns on 16 merry-go-rounds depending on whether there are 50 kids or 50,000. The more load the more cores you need to sustain decent throughput of sharing. Think of the processor cores and level-one cache banks as turn-styles at the amusement park or subway entrance. Since a processor can only handle one instruction at a time on each core each thread will take a single-file turn submitting one instruction, or a few dozen depending on how the schedule is setup for that thread. On multi-core systems the multiple turn-styles allows more threads to run 'concurrently'. Regardless of core count, both thread priority and processor affinity/assignment are how scheduling affects the length of time a single kid can play on one specific merry-go-round and also how often that one kid is allowed to jumping in front of everyone else on other merry-go-rounds. Time priority and core assignment are handled a bit differently at the user /app level on different operating systems but they all do basically the same thing at the root. A multi-threaded application allows one parent process to spawn child threads (not fork()) to handle specific workloads. Each thread is basically a separate O/S process that has some shared memory space with the parent. The parent is responsible for keeping up with the children for all phases of the thread task cycle. Outside of signal/interrupt control, the child thread is on its own to make the task happen. So a multi-threaded strategy is irrelevant to hardware architecture in a general sense, but the O/S architecture plays a primary role in how threads are handled and how efficient the application will operate. Together, they both determine how overall efficient the threading will work. In a lot of cases the idea of throw more iron at it stems from a mismatch of efficiency between the hardware, the O/S and the application. Regards, Glen Batchelor IT Director/CIO/CTO All-Spec Industries phone: (910) 332-0424 fax: (910) 763-5664 E-mail: webmas...@all-spec.com Web: http://www.all-spec.com Blog: http://blog.all-spec.com -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users- boun...@listserver.u2ug.org] On Behalf Of Dave Taylor Sent: Monday, March 07, 2011 10:58 AM To: U2 Users List Subject: Re: [U2] Multi-Threading Universe Socket traffic Symeon, and others, How does a multi-threading strategy relate to the use of a multi-core, multi-cpu and/or clustered hardware server configuration using various operating systems and multi-value databases in terms of performance and thruput? (A complete answer gets you an honorary PhD in whatever subject you'd like). Dave Taylor Sysmark Information Systems, Inc. 49 Aspen Way Rolling Hills Estates, CA 90274 (O) 800-SYSMARK (800-797-6275) (F) 310-377-3550 (C) 310-561-5200 www.sysmarkinfo.com - Original Message - From: Symeon Breen syme...@gmail.com To: 'U2 Users List' u2-users@listserver.u2ug.org Sent: Monday, March 07, 2011 1:16 AM Subject: Re: [U2] Multi-Threading Universe Socket traffic Ok the python solution mentioned is one way - however this multi threading you requiree is not a multi threading requirement - you are consuming a socket service, and not accepting connections, - we do this kind of thing all the time using phantoms. Lets say your batch of transactions is in a file, as you process each one set a flag saying it is done, or delete the record or something, then yourprogram can select the file, loop through the records and if the flag is set or the record does not exist it just skips onto the next one. You can then start 10 processes running all doing the same thing and they will work through the file. Or you could have process 1 doing all the ones beginning with a 1, 2 for 2 and so on. You may want a controlling program that runs up, counts the records on the file/in the batch and from that determined how many phantoms to run up. It then runs up the phantoms and then stops. Rgds Symeon. From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of nschroth Sent: 05 March 2011 16:55 To: u2-users@listserver.u2ug.org Subject: [U2] Multi-Threading Universe Socket traffic On Universe 10.1.14 over AIX 5.3, we currently communicate
Re: [U2] Multi-Threading Universe Socket traffic
I'm going to go Contrarian: It does not relate. Dave Taylor wrote: Symeon, and others, How does a multi-threading strategy relate to the use of a multi-core, multi-cpu and/or clustered hardware server configuration using various operating systems and multi-value databases in terms of performance and thruput? (A complete answer gets you an honorary PhD in whatever subject you'd like). Dave Taylor Sysmark Information Systems, Inc. 49 Aspen Way Rolling Hills Estates, CA 90274 (O) 800-SYSMARK (800-797-6275) (F) 310-377-3550 (C) 310-561-5200 www.sysmarkinfo.com - Original Message - From: Symeon Breen syme...@gmail.com To: 'U2 Users List' u2-users@listserver.u2ug.org Sent: Monday, March 07, 2011 1:16 AM Subject: Re: [U2] Multi-Threading Universe Socket traffic Ok the python solution mentioned is one way - however this multi threading you requiree is not a multi threading requirement - you are consuming a socket service, and not accepting connections, - we do this kind of thing all the time using phantoms. Lets say your batch of transactions is in a file, as you process each one set a flag saying it is done, or delete the record or something, then yourprogram can select the file, loop through the records and if the flag is set or the record does not exist it just skips onto the next one. You can then start 10 processes running all doing the same thing and they will work through the file. Or you could have process 1 doing all the ones beginning with a 1, 2 for 2 and so on. You may want a controlling program that runs up, counts the records on the file/in the batch and from that determined how many phantoms to run up. It then runs up the phantoms and then stops. Rgds Symeon. From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of nschroth Sent: 05 March 2011 16:55 To: u2-users@listserver.u2ug.org Subject: [U2] Multi-Threading Universe Socket traffic On Universe 10.1.14 over AIX 5.3, we currently communicate Credit Card transactions via sockets (ISO-8583) using the following logic (works fine): OPEN.ERR=openSocket(THIS.IP,THIS.PORT,TCP.MODE,TIMEOUT,THIS.HANDLE) INFO.ERR=getSocketInformation(THIS.HANDLE,PEER.FLAG,SOCKETINFO) WRITE.ERR=writeSocket(THIS.HANDLE,SEND.MSG,TIMEOUT,TCP.MODE,SEND.SIZE) READ.ERR=readSocket(THIS.HANDLE,RECV.MSG,RECV.LEN,TIMEOUT,TCP.MODE,RECV.SIZE ) I am looking into using some sort of multi-threading logic to allow increased volume in shorter timeframes. For example, we now send batches of up to 3,000 cards via a ftps mechanism that responds normally within 3-5 minutes. We want to benchmark doing this via the sockets instead. The Subroutine that does this Socket comm takes about 0.7 secs per trx, so a 3,000 card batch would take over 30 minutes single-threaded (unacceptable). Probably 80% of that 0.7 secs is transforming the data to send and then transforming the response back to process. I hearthat my BASIC applicaton program can utilize PHANTOM processes (unfamiliar territory) to launch multiple requests but I am not sure how to throw multiple trxs over the wall and make sure to put te correct responses back together with the appropriate request. Can anyone point me to some good documentation and/or example code for doing this? -- View this message in context: http://old.nabble.com/Multi-Threading-Universe-Socket-traffic-tp31076011p310 76011.html Sent from the U2 - Users mailing list archive at Nabble.com. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users _ No virus found in this message. Checked by AVG - www.avg.com Version: 10.0.1204 / Virus Database: 1435/3482 - Release Date: 03/04/11 ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users -- Jeff Schasny - Denver, Co, USA jschasny at gmail dot com ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Multi-Threading Universe Socket traffic
From: Symeon Breen ...you are consuming a socket service, and not accepting connections, - we do this kind of thing all the time using phantoms. Lets say your batch of transactions is in a file... I agree with Symeon, phantoms are probably your answer. While I haven't benchmarked a solution I can easily picture a scenario where your primary process just writes transactions, and several phantoms are polling the transaction file for something to do. As each phantom grabs a batch of items it's working independently from others. It's the grocery store checkout lane model where the cashier says I can help you over here, and the more lanes that are open, the faster everyone gets out the door. Your existing socket code can make multiple outbound calls to the remote server, as long as that server can handle multiple calls on the same socket port and fork them off for processing. That's not your concern but of course it's a factor, and you have to know how many client processes you can expect to process simultaneously. With the grocery store analogy, it doesn't matter how many checkout lanes are open if there is only one person running around to get price checks. You're apparently OK with sockets but to avoid DBMS-specific socket implementations I generally offload such processing to a middle tier. So the MV app simply writes to a file, the external process extracts and handles the request, then writes a response back to another file. The BASIC app knows nothing about sockets, remote servers, or any such detail, it simply writes and reads data, which is what a database is good at - and this is more within the comfort zone of most MV developers. I can share more info about such a solution for anyone who is interested. HTH T Tony Gravagno Nebula Research and Development TG@ remove.pleaseNebula-RnD.com Nebula RD sells mv.NET and other Pick/MultiValue products worldwide, and provides related development services remove.pleaseNebula-RnD.com/blog Visit PickWiki.com! Contribute! http://Twitter.com/TonyGravagno ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Multi-Threading Universe Socket traffic
On 07/03/11 09:16, Symeon Breen wrote: Lets say your batch of transactions is in a file, as you process each one set a flag saying it is done, or delete the record or something, then yourprogram can select the file, loop through the records and if the flag is set or the record does not exist it just skips onto the next one. You can then start 10 processes running all doing the same thing and they will work through the file. Or you could have process 1 doing all the ones beginning with a 1, 2 for 2 and so on. You may want a controlling program that runs up, counts the records on the file/in the batch and from that determined how many phantoms to run up. It then runs up the phantoms and then stops. Just be careful. This is the sort of approach I would use, but be careful of BASIC SELECT. DON'T use this on a dynamic file unless you know what you're doing, or you don't need as fast as possible response and can run a pass say once a minute. The reason for saying that is that SELECT will halt any dynamicness in the file so you could end up with a badly mis-sized file. Probably not important, but something to be aware of. If you leave a fallow gap between SELECT passes, then the app writing to the file will cause the file to sort itself out. Cheers, Wol ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Multi-Threading Universe Socket traffic
I'd agree with this approach...I've used it many times before... A good well thought out design well help you...design first: code second :-) I've in the past done the following: - control program to configure, start/monitor/stop phantom processes - I find writing a phantom process which logs what it is doing/done is used for debugging purposes too (saves guessing what's going) - just remember to have a toggle to turn this on/off for production (to save disk space) - it is important that each phantom logs its progress, its process ID, start, last checkpoint reached and that it has successfully terminated. This helps prevent the case of you accidently firing off 100 phantoms and consuming all your UV licenses, etc. :-) The purpose of this is to: - make the process scalable and tuneable without re-coding via parameter record (include things like operating windows, etc) - have an application to manage the phantoms and monitor what they're doing (or if they're doing anything) - have phantom processes log what they're doing and allow options for a verbose logging mode to log everything they're doing in case of problems Some approaches I've used: a) each phantom could be started with a unique saved list of record keys to processed (generated by a control program or some other process beforehand) b) each phantom can perform its own query (but sleeping for a specified period so not to continuously performing disk I/O) c) Inbound and outbound transaction phantom process - each handles only response or send requests respectively. Such an approach only works if you can reconcile the response with the original request. Very application specific and not generally optimised for throughput. We first tested this to get ensure we weren't thrashing the disk I/O with too many phantom processes...as this was a pure database read/lock/write type transactional batch process. Use performance monitoring tools to help you do this. Plus your case - networking monitoring, etc. Yours needs to be optimised for sockets I/O as well as DB updates. It shouldn't be opening/closing socket connections unnecessarily (if that wasn't obvious already) - due to the high over head of doing so. Like disk I/O you need to be ensure that the path out to your internet connection is already optimised so it doesn't contribute to the delays. Likewise that it can support X number of connections and your third-party service provider allows you to do this. I'd say you need to optimise your socket program first and think about how to multi-stream (multi-thread implies low-level OS type functions) your processing. Care needs to be taken to ensure you design your process with the with sufficient error handling to ensure that you don't write an application which becomes part of the problem! This is so easy with stuff like this to go amiss. :-) Before you go live - in your test environment - try to break this process (overload/kill phantoms). Better it breaks there, than in production. It's really hard to fix UV problems, when you have no licenses left :-( Good luck. -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Symeon Breen Sent: Monday, 7 March 2011 8:17 PM To: 'U2 Users List' Subject: Re: [U2] Multi-Threading Universe Socket traffic Ok the python solution mentioned is one way - however this multi threading you requiree is not a multi threading requirement - you are consuming a socket service, and not accepting connections, - we do this kind of thing all the time using phantoms. Lets say your batch of transactions is in a file, as you process each one set a flag saying it is done, or delete the record or something, then yourprogram can select the file, loop through the records and if the flag is set or the record does not exist it just skips onto the next one. You can then start 10 processes running all doing the same thing and they will work through the file. Or you could have process 1 doing all the ones beginning with a 1, 2 for 2 and so on. You may want a controlling program that runs up, counts the records on the file/in the batch and from that determined how many phantoms to run up. It then runs up the phantoms and then stops. Rgds Symeon. From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of nschroth Sent: 05 March 2011 16:55 To: u2-users@listserver.u2ug.org Subject: [U2] Multi-Threading Universe Socket traffic On Universe 10.1.14 over AIX 5.3, we currently communicate Credit Card transactions via sockets (ISO-8583) using the following logic (works fine): OPEN.ERR=openSocket(THIS.IP,THIS.PORT,TCP.MODE,TIMEOUT,THIS.HANDLE) INFO.ERR=getSocketInformation(THIS.HANDLE,PEER.FLAG,SOCKETINFO) WRITE.ERR=writeSocket(THIS.HANDLE,SEND.MSG,TIMEOUT,TCP.MODE,SEND.SIZE) READ.ERR=readSocket(THIS.HANDLE,RECV.MSG,RECV.LEN,TIMEOUT,TCP.MODE,RECV.SIZE ) I am looking
[U2] Multi-Threading Universe Socket traffic
On Universe 10.1.14 over AIX 5.3, we currently communicate Credit Card transactions via sockets (ISO-8583) using the following logic (works fine): OPEN.ERR=openSocket(THIS.IP,THIS.PORT,TCP.MODE,TIMEOUT,THIS.HANDLE) INFO.ERR=getSocketInformation(THIS.HANDLE,PEER.FLAG,SOCKETINFO) WRITE.ERR=writeSocket(THIS.HANDLE,SEND.MSG,TIMEOUT,TCP.MODE,SEND.SIZE) READ.ERR=readSocket(THIS.HANDLE,RECV.MSG,RECV.LEN,TIMEOUT,TCP.MODE,RECV.SIZE) I am looking into using some sort of multi-threading logic to allow increased volume in shorter timeframes. For example, we now send batches of up to 3,000 cards via a ftps mechanism that responds normally within 3-5 minutes. We want to benchmark doing this via the sockets instead. The Subroutine that does this Socket comm takes about 0.7 secs per trx, so a 3,000 card batch would take over 30 minutes single-threaded (unacceptable). Probably 80% of that 0.7 secs is transforming the data to send and then transforming the response back to process. I hearthat my BASIC applicaton program can utilize PHANTOM processes (unfamiliar territory) to launch multiple requests but I am not sure how to throw multiple trxs over the wall and make sure to put te correct responses back together with the appropriate request. Can anyone point me to some good documentation and/or example code for doing this? -- View this message in context: http://old.nabble.com/Multi-Threading-Universe-Socket-traffic-tp31076011p31076011.html Sent from the U2 - Users mailing list archive at Nabble.com. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Multi-Threading Universe Socket traffic
I would pass it by socket to a waiting python socket server then you can use the multiprocessing module to break it up into multiple workers. Prob 20 lines of code. May I ask who your CC processor is? We need a new one for compliance. On 3/5/11, nschroth ngschr...@yahoo.com wrote: On Universe 10.1.14 over AIX 5.3, we currently communicate Credit Card transactions via sockets (ISO-8583) using the following logic (works fine): OPEN.ERR=openSocket(THIS.IP,THIS.PORT,TCP.MODE,TIMEOUT,THIS.HANDLE) INFO.ERR=getSocketInformation(THIS.HANDLE,PEER.FLAG,SOCKETINFO) WRITE.ERR=writeSocket(THIS.HANDLE,SEND.MSG,TIMEOUT,TCP.MODE,SEND.SIZE) READ.ERR=readSocket(THIS.HANDLE,RECV.MSG,RECV.LEN,TIMEOUT,TCP.MODE,RECV.SIZE) I am looking into using some sort of multi-threading logic to allow increased volume in shorter timeframes. For example, we now send batches of up to 3,000 cards via a ftps mechanism that responds normally within 3-5 minutes. We want to benchmark doing this via the sockets instead. The Subroutine that does this Socket comm takes about 0.7 secs per trx, so a 3,000 card batch would take over 30 minutes single-threaded (unacceptable). Probably 80% of that 0.7 secs is transforming the data to send and then transforming the response back to process. I hearthat my BASIC applicaton program can utilize PHANTOM processes (unfamiliar territory) to launch multiple requests but I am not sure how to throw multiple trxs over the wall and make sure to put te correct responses back together with the appropriate request. Can anyone point me to some good documentation and/or example code for doing this? -- View this message in context: http://old.nabble.com/Multi-Threading-Universe-Socket-traffic-tp31076011p31076011.html Sent from the U2 - Users mailing list archive at Nabble.com. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users -- Sent from my mobile device ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Multi-Threading Universe Socket traffic
PNC (First Data). But we want to adopt tokenization to avoid most of the PCI issues and they just started a socket-based tokenization...they are many months/years away from Skipjack or FuseBox. I appreciate your suggestion, but we are not a Python shop and mgmt does not want to get into anything that IT cannot fluently develope and support :-) slestak wrote: I would pass it by socket to a waiting python socket server then you can use the multiprocessing module to break it up into multiple workers. Prob 20 lines of code. May I ask who your CC processor is? We need a new one for compliance. On 3/5/11, nschroth ngschr...@yahoo.com wrote: On Universe 10.1.14 over AIX 5.3, we currently communicate Credit Card transactions via sockets (ISO-8583) using the following logic (works fine): OPEN.ERR=openSocket(THIS.IP,THIS.PORT,TCP.MODE,TIMEOUT,THIS.HANDLE) INFO.ERR=getSocketInformation(THIS.HANDLE,PEER.FLAG,SOCKETINFO) WRITE.ERR=writeSocket(THIS.HANDLE,SEND.MSG,TIMEOUT,TCP.MODE,SEND.SIZE) READ.ERR=readSocket(THIS.HANDLE,RECV.MSG,RECV.LEN,TIMEOUT,TCP.MODE,RECV.SIZE) I am looking into using some sort of multi-threading logic to allow increased volume in shorter timeframes. For example, we now send batches of up to 3,000 cards via a ftps mechanism that responds normally within 3-5 minutes. We want to benchmark doing this via the sockets instead. The Subroutine that does this Socket comm takes about 0.7 secs per trx, so a 3,000 card batch would take over 30 minutes single-threaded (unacceptable). Probably 80% of that 0.7 secs is transforming the data to send and then transforming the response back to process. I hearthat my BASIC applicaton program can utilize PHANTOM processes (unfamiliar territory) to launch multiple requests but I am not sure how to throw multiple trxs over the wall and make sure to put te correct responses back together with the appropriate request. Can anyone point me to some good documentation and/or example code for doing this? -- View this message in context: http://old.nabble.com/Multi-Threading-Universe-Socket-traffic-tp31076011p31076011.html Sent from the U2 - Users mailing list archive at Nabble.com. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users -- Sent from my mobile device ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users -- View this message in context: http://old.nabble.com/Multi-Threading-Universe-Socket-traffic-tp31076011p31076436.html Sent from the U2 - Users mailing list archive at Nabble.com. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Multi-Threading Universe Socket traffic
On 03/05/2011 01:26 PM, nschroth wrote: PNC (First Data). But we want to adopt tokenization to avoid most of the PCI issues and they just started a socket-based tokenization...they are many months/years away from Skipjack or FuseBox. I appreciate your suggestion, but we are not a Python shop and mgmt does not want to get into anything that IT cannot fluently develope and support :-) slestak wrote: I would pass it by socket to a waiting python socket server then you can use the multiprocessing module to break it up into multiple workers. Prob 20 lines of code. May I ask who your CC processor is? We need a new one for compliance. On 3/5/11, nschrothngschr...@yahoo.com wrote: On Universe 10.1.14 over AIX 5.3, we currently communicate Credit Card transactions via sockets (ISO-8583) using the following logic (works fine): OPEN.ERR=openSocket(THIS.IP,THIS.PORT,TCP.MODE,TIMEOUT,THIS.HANDLE) INFO.ERR=getSocketInformation(THIS.HANDLE,PEER.FLAG,SOCKETINFO) WRITE.ERR=writeSocket(THIS.HANDLE,SEND.MSG,TIMEOUT,TCP.MODE,SEND.SIZE) READ.ERR=readSocket(THIS.HANDLE,RECV.MSG,RECV.LEN,TIMEOUT,TCP.MODE,RECV.SIZE) I am looking into using some sort of multi-threading logic to allow increased volume in shorter timeframes. For example, we now send batches of up to 3,000 cards via a ftps mechanism that responds normally within 3-5 minutes. We want to benchmark doing this via the sockets instead. The Subroutine that does this Socket comm takes about 0.7 secs per trx, so a 3,000 card batch would take over 30 minutes single-threaded (unacceptable). Probably 80% of that 0.7 secs is transforming the data to send and then transforming the response back to process. I hearthat my BASIC applicaton program can utilize PHANTOM processes (unfamiliar territory) to launch multiple requests but I am not sure how to throw multiple trxs over the wall and make sure to put te correct responses back together with the appropriate request. Can anyone point me to some good documentation and/or example code for doing this? -- View this message in context: http://old.nabble.com/Multi-Threading-Universe-Socket-traffic-tp31076011p31076011.html Sent from the U2 - Users mailing list archive at Nabble.com. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users -- Sent from my mobile device ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users I would shy away from making U2 do sth it is not designed for. I understand your position though. The same thing can be done in java and perl as well relatively easily. And these technologies are designed for that use. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
