Re: [U2] Multi-threaded phantom processing
Hi Peter Clearly, you have no other option than to go the direction you have. Sounds like you're using the correct API and approach for the task at hand and with the constraints you have. Sounds like you have considered various implications and you're constrain by UV versions and UniRPC being blocked (a good thing if you're connectivity to third-parties and over unsecure/public networks :))... BTW: UV File triggers are light-weight triggers available in UV11.x - without the overhead and complexity of UV SQL Triggers. The MQSeries interface is good, but it's learning curve can be steep and is only feasible if both sites are licensed users. I've used it a lot, but the underlying complexity of the MQ setup and API can be up there with TCP/IP sockets :(... if you're new to it all! SQL BCI requires UniRPC access so that would be ruled out for your use as well. All of the higher-level APIs just save you the hassle of writing a complex low-level interface which could be hard to develop, debug and maintain in the future... In my experience, it is difficult to optimise any interface unless you have some minimum, average and maximum loads or throughput requirements to be met. It'll be difficult to optimise the interface other than setting minimum and maximum phantom processes - as you're and other have already mentioned. I guess time will tell how it works and what tweaking (aka optimisation) will be required...nothing like real-world usage to sort things out... Another potential consideration you need to consider or at least have test cases is loss of connectivity due to abnormal termination by the client, network (local/carrier/remote) or remote server. All these events may cause various issues that can impact UV locally, remotely and your TCP connectivity. Troubleshooting network and application connectivity is never easy - hence working through worse case scenarios in advance is a worthwhile - and somewhat very tedious exercise. Such issues have are applicable to all applications - of course. Such network issues impact the native UV APIs as well...but I am sure you'll get to know and loathe errors such as FIN_WAIT, etc., etc. as you get more usage and issues, etc. Cheers, David -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Peter Cheney Sent: Friday, 7 March 2014 6:00 PM To: 'U2 Users List' Subject: Re: [U2] Multi-threaded phantom processing Initially data volume will be small - probably about 20 customers per day and about 6 records per customer but it is likely to grow very quickly up to some unknown amount. We'll have to see I guess. The records per customer are likely to be the high volume stuff. There is transaction logging on the remote system that I can use and am relying on for data integrity but I did not know about the SQL stuff until this week. Not sure if it would work until I can find time to play with it and see how it might suit us. Some of the other constraints were: no access allowed to the remote unirpc port, the remote UV is below 10.3 and different from ours, neither site has a UV/Net licence in prod. So I think I'm pretty much stuck with file queuing and multiple phantoms for now at least. I also was able to get a successful remote file process going on our test system while waiting for confirmation of UV/Net accessibility. It is very nice and might have been a nice solution too. Then I did some experimenting with the performance requirements on the file triggers and found there to be too much of a lag for the triggers to fire directly to the other end. This would've caused unacceptable problems in testing not to mention going live so a file queuing process had to be devised. In all the file queuing works quite well and can be coded for rock solid reliability so it has a lot of pluses. I can also fine tune the phantom loop times and socket timeouts on the fly at both ends once the data starts flowing and we get some idea of what needs changing. Fingers, toes, arms, legs (and eyes!) all currently crossed. Cheers Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Hona, David Sent: Friday, 7 March 2014 13:41 To: U2 Users List Subject: Re: [U2] Multi-threaded phantom processing Hi Peter Sounds like an interested piece of work, just a couple of questions for you: - what sort of data volumes and update frequencies are involved? - do you need to know if the data has been successfully committed to the remote UV database? - did you consider SQL BCI or even UV/Net (read the different UV versions is an issue and so this may be a constraint) Gregor mentioned SQL BCI as an alternative interface - I've used that before to exchange data between systems
Re: [U2] Multi-threaded phantom processing
The only approach I would consider is via a web service , either hosted on iis or apache, with a uniobjects connection behind it - either uo.net or uo.java. web servers are designed to accept multiple hits, queue them up , offload to threads etc. why reinvent the wheel. One of my companies does adserving and we serve upwards of 20million ads per day using this approach with a .net webservice using uo.net to communicate to a udt host. Hi Everyone, A project that I m currently working on requires transmission of transactional and ancillary data from one UV site to site to another. I have the communications all set up and it is working very well via the UV socket functions with a client at one end and a server at the other. Both the client and the server both run as a UV phantom processes written in UV basic and the data being transferred (also encrypted) is more or less real time with various UV file triggers on the client end creating queue items for transmission based on the data content / transaction type. This set up is a single threaded affair and I am wondering how it will cope and what options I ll have as the volume of data increases. Multi-threading my server process to dish out some of the load might be one possibility and I ve seen examples in the past where the last digit of a numeric key (i.e. 0-9) is used to determine which one of 10 phantoms to call thus providing an effective distribution of the work. But even then, reading/writing processing data to a file for each child process seems to me to be the only way to achieve parent - child phantom communications at either end. Is there another way perhaps? It may be that I will need 2 sets of phantoms, 1 for the transactions and one for the ancillary data. Although we run UV on AIX the only other requirement is that I must be able to do it within UV Basic. Does anyone out there on the list know of a better way or any alternatives/advice so I can get a better appreciation of what else might be possible please? Many thanks in advance, Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au --- Note: This email (inc all attachments) is for the use of the intended recipient(s) only. Privileged or confidential information may be contained in this communication. If you have received this email in error, please notify the sender immediately and then delete all copies of this message from your computer network. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author's prior permission. If you are the intended recipient and you do not wish to receive similar electronic messages from us in future, then please respond to the sender to this effect. We have taken precautions to minimise the risk of transmitting software viruses, but advise you to carry out your own virus checks on this email and its attachments. We do not accept liability for any loss or damage caused by software viruses and do not represent that this transmission is free from viruses or other defects. Firstmac Limited (ABN 59 094 145 963) (AFSL 290600) --- ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ** IMPORTANT MESSAGE * This e-mail message is intended only for the addressee(s) and contains information which may be confidential. If you are not the intended recipient please advise the sender by return email, do not use or disclose the contents, and delete the message and any attachments from your system. Unless specifically indicated, this email does not constitute formal advice or commitment by the sender or the Commonwealth Bank of Australia (ABN 48 123 123 124) or its subsidiaries. We can be contacted through our web site: commbank.com.au. If you no longer wish to receive commercial electronic messages from us, please reply to this e-mail by typing Unsubscribe in the subject line. ** ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users --- Note: This email (inc all attachments) is for the use of the intended recipient(s) only. Privileged or confidential information may be contained in this communication. If you have received this email in error, please notify the sender immediately and then delete all copies of this message from your computer network. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the
Re: [U2] Multi-threaded phantom processing
On 07/03/14 06:59, Peter Cheney wrote: Some of the other constraints were: no access allowed to the remote unirpc port, the remote UV is below 10.3 and different from ours, neither site has a UV/Net licence in prod. So I think I'm pretty much stuck with file queuing and multiple phantoms for now at least. Can you mount a remote directory? Because my immediate idea was if the source computer can write to a directory on the target, the target computer can declare it as a type 1 file. And if the source computer also declares it as a type 1, it can dump its records into this directory for the target to pick up. If you're worried about the target trying to read them before they're fully saved, simply put a datestamp in the key, and the target just does a select for all records older than, say, one minute, then processes and deletes them. Cheers, Wol ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Multi-threaded phantom processing
Yes of course there are many solutions: rcp, ftp, nfa, http, mapped drives, etc. But the one that has the most security is the socket to socket communications with your own protocol. You get to control who has access and what commands they have access to. And of course you get to add logging as you need. Plus it runs native from U2 so that you can do command like compile, or move just a single attribute or sub-value. My customer needed a tool to keep the developers out of the live data server. By disabling the normal data transfer methods and installing a socket-to-socket approach we made the auditors very happy. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Rick Nuckolls Sent: Wednesday, March 05, 2014 5:57 PM To: U2 Users List Cc: U2 Users List Subject: Re: [U2] Multi-threaded phantom processing As elegant as some of these ideas are, it might be possible, depending on the number and nature of the transactions, to batch them up every n seconds and use rcp to copy them to an in box on the remote machine. -Rick On Mar 5, 2014, at 4:19 PM, Horacio Pellegrino pellegrin...@gmail.com wrote: Why don't you have one port listening for requests, and as soon as they come in, you create a new phantom to answer that request ( liberating the listener immediately for the next request ), then when the phantom finishes it could log off. Might be you want to keep a pool of phantoms alive for a while, on the assumption they might be reused. That way you are sill using one license per request, but you have the benefit of pooling... You can always set min-max limit for those phantom-workers. Horacio Pellegrino On Tue, Mar 4, 2014 at 11:30 PM, Peter Cheney peter.che...@firstmac.com.auwrote: Thanks David. As they say Bigger than Ben-Hur! That's a great idea. The server I have is listening on a predetermined port. I trolled through /etc/services and a netstat on the remote host to select something that looked unused. So then if it the main phantom spawns 1-n children (presumably more phantoms?), it would also need to select 1-n ports which I suppose must also be predetermined? Or is there a call to the OS that can return an unused port? We're on AIX but the remote host is HPUX. Cheers Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of David A. Green Sent: Wednesday, 5 March 2014 13:01 To: 'U2 Users List' Subject: Re: [U2] Multi-threaded phantom processing Peter, first of all let me congratulate you on this achievement. Having just finished a similar feat using UniData I understand the complexities that went into it. The system I'm current working with will be relatively low traffic. But I was pondering on the method of having a server socket that would just spawn a transfer server socket and send the socket address to the calling client. Then the client disconnects from the main server and just uses the transfer server to complete the task. Having 10 open phantoms with sockets will use up 10 licenses. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of Peter Cheney Sent: Tuesday, March 04, 2014 6:18 PM To: U2 Users List (u2-users@listserver.u2ug.org) Subject: [U2] Multi-threaded phantom processing Hi Everyone, A project that I'm currently working on requires transmission of transactional and ancillary data from one UV site to site to another. I have the communications all set up and it is working very well via the UV socket functions with a client at one end and a server at the other. Both the client and the server both run as a UV phantom processes written in UV basic and the data being transferred (also encrypted) is more or less real time with various UV file triggers on the client end creating queue items for transmission based on the data content / transaction type. This set up is a single threaded affair and I am wondering how it will cope and what options I'll have as the volume of data increases. Multi-threading my server process to dish out some of the load might be one possibility and I've seen examples in the past where the last digit of a numeric key (i.e. 0-9) is used to determine which one of 10 phantoms to call thus providing an effective distribution of the work. But even then, reading/writing processing data to a file for each child process seems to me to be the only way to achieve parent - child phantom communications at either end. Is there another way perhaps? It may be that I will need 2 sets of phantoms
Re: [U2] Multi-threaded phantom processing
Will you kinda lost me on this approach. Since the Server socket is on a different machine than the client they don't see each other's locks. Different clients might be on different servers too. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson Sent: Wednesday, March 05, 2014 5:26 PM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Multi-threaded phantom processing Peter is the only point of the parent child communications to address points like 1 I'm busy go away; OR 2 I'm available to work If so, you could simply use a lock-semaphore to address both of those issues.\ -Original Message- From: Horacio Pellegrino pellegrin...@gmail.com To: U2 Users List u2-users@listserver.u2ug.org Sent: Wed, Mar 5, 2014 4:19 pm Subject: Re: [U2] Multi-threaded phantom processing Why don't you have one port listening for requests, and as soon as they come in, you create a new phantom to answer that request ( liberating the listener immediately for the next request ), then when the phantom finishes it could log off. Might be you want to keep a pool of phantoms alive for a while, on the assumption they might be reused. That way you are sill using one license per request, but you have the benefit of pooling... You can always set min-max limit for those phantom-workers. Horacio Pellegrino On Tue, Mar 4, 2014 at 11:30 PM, Peter Cheney peter.che...@firstmac.com.auwrote: Thanks David. As they say Bigger than Ben-Hur! That's a great idea. The server I have is listening on a predetermined port. I trolled through /etc/services and a netstat on the remote host to select something that looked unused. So then if it the main phantom spawns 1-n children (presumably more phantoms?), it would also need to select 1-n ports which I suppose must also be predetermined? Or is there a call to the OS that can return an unused port? We're on AIX but the remote host is HPUX. Cheers Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of David A. Green Sent: Wednesday, 5 March 2014 13:01 To: 'U2 Users List' Subject: Re: [U2] Multi-threaded phantom processing Peter, first of all let me congratulate you on this achievement. Having just finished a similar feat using UniData I understand the complexities that went into it. The system I'm current working with will be relatively low traffic. But I was pondering on the method of having a server socket that would just spawn a transfer server socket and send the socket address to the calling client. Then the client disconnects from the main server and just uses the transfer server to complete the task. Having 10 open phantoms with sockets will use up 10 licenses. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of Peter Cheney Sent: Tuesday, March 04, 2014 6:18 PM To: U2 Users List (u2-users@listserver.u2ug.org) Subject: [U2] Multi-threaded phantom processing Hi Everyone, A project that I'm currently working on requires transmission of transactional and ancillary data from one UV site to site to another. I have the communications all set up and it is working very well via the UV socket functions with a client at one end and a server at the other. Both the client and the server both run as a UV phantom processes written in UV basic and the data being transferred (also encrypted) is more or less real time with various UV file triggers on the client end creating queue items for transmission based on the data content / transaction type. This set up is a single threaded affair and I am wondering how it will cope and what options I'll have as the volume of data increases. Multi-threading my server process to dish out some of the load might be one possibility and I've seen examples in the past where the last digit of a numeric key (i.e. 0-9) is used to determine which one of 10 phantoms to call thus providing an effective distribution of the work. But even then, reading/writing processing data to a file for each child process seems to me to be the only way to achieve parent - child phantom communications at either end. Is there another way perhaps? It may be that I will need 2 sets of phantoms, 1 for the transactions and one for the ancillary data. Although we run UV on AIX the only other requirement is that I must be able to do it within UV Basic. Does anyone out there on the list know of a better way or any alternatives/advice so I can get a better appreciation of what else might be possible please? Many thanks in advance, Peter
Re: [U2] Multi-threaded phantom processing
I was referring solely to the point of launching phantoms and how they could communicate back to their master who launched them. Not back to a different server. So the thread was mentioning a master-slave relationship on *a* server as I understood it. And the question asked was how do the slaves and master communicate. -Original Message- From: David A. Green dgr...@dagconsulting.com To: 'U2 Users List' u2-users@listserver.u2ug.org Sent: Thu, Mar 6, 2014 5:43 am Subject: Re: [U2] Multi-threaded phantom processing Will you kinda lost me on this approach. Since the Server socket is on a different machine than the client they don't see each other's locks. Different clients might be on different servers too. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson Sent: Wednesday, March 05, 2014 5:26 PM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Multi-threaded phantom processing Peter is the only point of the parent child communications to address points like 1 I'm busy go away; OR 2 I'm available to work If so, you could simply use a lock-semaphore to address both of those issues.\ -Original Message- From: Horacio Pellegrino pellegrin...@gmail.com To: U2 Users List u2-users@listserver.u2ug.org Sent: Wed, Mar 5, 2014 4:19 pm Subject: Re: [U2] Multi-threaded phantom processing Why don't you have one port listening for requests, and as soon as they come in, you create a new phantom to answer that request ( liberating the listener immediately for the next request ), then when the phantom finishes it could log off. Might be you want to keep a pool of phantoms alive for a while, on the assumption they might be reused. That way you are sill using one license per request, but you have the benefit of pooling... You can always set min-max limit for those phantom-workers. Horacio Pellegrino On Tue, Mar 4, 2014 at 11:30 PM, Peter Cheney peter.che...@firstmac.com.auwrote: Thanks David. As they say Bigger than Ben-Hur! That's a great idea. The server I have is listening on a predetermined port. I trolled through /etc/services and a netstat on the remote host to select something that looked unused. So then if it the main phantom spawns 1-n children (presumably more phantoms?), it would also need to select 1-n ports which I suppose must also be predetermined? Or is there a call to the OS that can return an unused port? We're on AIX but the remote host is HPUX. Cheers Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of David A. Green Sent: Wednesday, 5 March 2014 13:01 To: 'U2 Users List' Subject: Re: [U2] Multi-threaded phantom processing Peter, first of all let me congratulate you on this achievement. Having just finished a similar feat using UniData I understand the complexities that went into it. The system I'm current working with will be relatively low traffic. But I was pondering on the method of having a server socket that would just spawn a transfer server socket and send the socket address to the calling client. Then the client disconnects from the main server and just uses the transfer server to complete the task. Having 10 open phantoms with sockets will use up 10 licenses. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of Peter Cheney Sent: Tuesday, March 04, 2014 6:18 PM To: U2 Users List (u2-users@listserver.u2ug.org) Subject: [U2] Multi-threaded phantom processing Hi Everyone, A project that I'm currently working on requires transmission of transactional and ancillary data from one UV site to site to another. I have the communications all set up and it is working very well via the UV socket functions with a client at one end and a server at the other. Both the client and the server both run as a UV phantom processes written in UV basic and the data being transferred (also encrypted) is more or less real time with various UV file triggers on the client end creating queue items for transmission based on the data content / transaction type. This set up is a single threaded affair and I am wondering how it will cope and what options I'll have as the volume of data increases. Multi-threading my server process to dish out some of the load might be one possibility and I've seen examples in the past where the last digit of a numeric key (i.e. 0-9) is used to determine which one of 10 phantoms to call thus providing an effective distribution of the work. But even then, reading/writing processing data to a file for each child
Re: [U2] Multi-threaded phantom processing
My thoughts were that the parent will need to allocate the work and in doing so take the incoming data stream, launching a child and somehow pass the work to it. My only current knowledge of doing that within UV basic is writing to a queue and have the child phantom read from that and respond via another queue. Hence my question if there there's another possibly better way of doing it? It is nice to know if a particular phantom is busy or available though so that suggestion is certainly handy thanks. Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson Sent: Friday, 7 March 2014 02:52 To: u2-users@listserver.u2ug.org Subject: Re: [U2] Multi-threaded phantom processing I was referring solely to the point of launching phantoms and how they could communicate back to their master who launched them. Not back to a different server. So the thread was mentioning a master-slave relationship on *a* server as I understood it. And the question asked was how do the slaves and master communicate. -Original Message- From: David A. Green dgr...@dagconsulting.com To: 'U2 Users List' u2-users@listserver.u2ug.org Sent: Thu, Mar 6, 2014 5:43 am Subject: Re: [U2] Multi-threaded phantom processing Will you kinda lost me on this approach. Since the Server socket is on a different machine than the client they don't see each other's locks. Different clients might be on different servers too. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson Sent: Wednesday, March 05, 2014 5:26 PM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Multi-threaded phantom processing Peter is the only point of the parent child communications to address points like 1 I'm busy go away; OR 2 I'm available to work If so, you could simply use a lock-semaphore to address both of those issues.\ -Original Message- From: Horacio Pellegrino pellegrin...@gmail.com To: U2 Users List u2-users@listserver.u2ug.org Sent: Wed, Mar 5, 2014 4:19 pm Subject: Re: [U2] Multi-threaded phantom processing Why don't you have one port listening for requests, and as soon as they come in, you create a new phantom to answer that request ( liberating the listener immediately for the next request ), then when the phantom finishes it could log off. Might be you want to keep a pool of phantoms alive for a while, on the assumption they might be reused. That way you are sill using one license per request, but you have the benefit of pooling... You can always set min-max limit for those phantom-workers. Horacio Pellegrino On Tue, Mar 4, 2014 at 11:30 PM, Peter Cheney peter.che...@firstmac.com.auwrote: Thanks David. As they say Bigger than Ben-Hur! That's a great idea. The server I have is listening on a predetermined port. I trolled through /etc/services and a netstat on the remote host to select something that looked unused. So then if it the main phantom spawns 1-n children (presumably more phantoms?), it would also need to select 1-n ports which I suppose must also be predetermined? Or is there a call to the OS that can return an unused port? We're on AIX but the remote host is HPUX. Cheers Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of David A. Green Sent: Wednesday, 5 March 2014 13:01 To: 'U2 Users List' Subject: Re: [U2] Multi-threaded phantom processing Peter, first of all let me congratulate you on this achievement. Having just finished a similar feat using UniData I understand the complexities that went into it. The system I'm current working with will be relatively low traffic. But I was pondering on the method of having a server socket that would just spawn a transfer server socket and send the socket address to the calling client. Then the client disconnects from the main server and just uses the transfer server to complete the task. Having 10 open phantoms with sockets will use up 10 licenses. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of Peter Cheney Sent: Tuesday, March 04, 2014 6:18 PM To: U2 Users List (u2-users@listserver.u2ug.org) Subject: [U2] Multi-threaded phantom processing Hi Everyone, A project that I'm currently working on requires transmission of transactional and ancillary data from one UV site to site to another. I have the communications all set up and it is working very well via
Re: [U2] Multi-threaded phantom processing
The simplest approach is to use the queue, but that requires check...wait...check...wait Another way, is to have one master and several always awake slaves listening as devices. The master uses the SEND command to pipe a command directly into the input stream of the listeners. This makes them wake up instantly (more or less) instead of waiting a second or a minute to check the queue. -Original Message- From: Peter Cheney peter.che...@firstmac.com.au To: 'U2 Users List' u2-users@listserver.u2ug.org Sent: Thu, Mar 6, 2014 4:05 pm Subject: Re: [U2] Multi-threaded phantom processing My thoughts were that the parent will need to allocate the work and in doing so take the incoming data stream, launching a child and somehow pass the work to it. My only current knowledge of doing that within UV basic is writing to a queue and have the child phantom read from that and respond via another queue. Hence my question if there there's another possibly better way of doing it? It is nice to know if a particular phantom is busy or available though so that suggestion is certainly handy thanks. Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson Sent: Friday, 7 March 2014 02:52 To: u2-users@listserver.u2ug.org Subject: Re: [U2] Multi-threaded phantom processing I was referring solely to the point of launching phantoms and how they could communicate back to their master who launched them. Not back to a different server. So the thread was mentioning a master-slave relationship on *a* server as I understood it. And the question asked was how do the slaves and master communicate. -Original Message- From: David A. Green dgr...@dagconsulting.com To: 'U2 Users List' u2-users@listserver.u2ug.org Sent: Thu, Mar 6, 2014 5:43 am Subject: Re: [U2] Multi-threaded phantom processing Will you kinda lost me on this approach. Since the Server socket is on a different machine than the client they don't see each other's locks. Different clients might be on different servers too. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson Sent: Wednesday, March 05, 2014 5:26 PM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Multi-threaded phantom processing Peter is the only point of the parent child communications to address points like 1 I'm busy go away; OR 2 I'm available to work If so, you could simply use a lock-semaphore to address both of those issues.\ -Original Message- From: Horacio Pellegrino pellegrin...@gmail.com To: U2 Users List u2-users@listserver.u2ug.org Sent: Wed, Mar 5, 2014 4:19 pm Subject: Re: [U2] Multi-threaded phantom processing Why don't you have one port listening for requests, and as soon as they come in, you create a new phantom to answer that request ( liberating the listener immediately for the next request ), then when the phantom finishes it could log off. Might be you want to keep a pool of phantoms alive for a while, on the assumption they might be reused. That way you are sill using one license per request, but you have the benefit of pooling... You can always set min-max limit for those phantom-workers. Horacio Pellegrino On Tue, Mar 4, 2014 at 11:30 PM, Peter Cheney peter.che...@firstmac.com.auwrote: Thanks David. As they say Bigger than Ben-Hur! That's a great idea. The server I have is listening on a predetermined port. I trolled through /etc/services and a netstat on the remote host to select something that looked unused. So then if it the main phantom spawns 1-n children (presumably more phantoms?), it would also need to select 1-n ports which I suppose must also be predetermined? Or is there a call to the OS that can return an unused port? We're on AIX but the remote host is HPUX. Cheers Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of David A. Green Sent: Wednesday, 5 March 2014 13:01 To: 'U2 Users List' Subject: Re: [U2] Multi-threaded phantom processing Peter, first of all let me congratulate you on this achievement. Having just finished a similar feat using UniData I understand the complexities that went into it. The system I'm current working with will be relatively low traffic. But I was pondering on the method of having a server socket that would just spawn a transfer server socket and send the socket address to the calling client. Then the client disconnects from the main server and just uses the transfer server to complete the task. Having 10
Re: [U2] Multi-threaded phantom processing
Thanks for that suggestion too. I did not know the SEND command could be used like that. So I suppose the listeners would use the openpipe and wait then? Must try that. This also reminded me of a time during mid-late 90's programming on a Sequoia pick system. That version of basic had a PAUSE command that you could use to pause a phantom indefinitely. Then wake it up again via another command in another program (POKE I think) directed at its PID. It was a very handy tool back then but have not seen it or an equivalent since. Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson Sent: Friday, 7 March 2014 10:20 To: u2-users@listserver.u2ug.org Subject: Re: [U2] Multi-threaded phantom processing The simplest approach is to use the queue, but that requires check...wait...check...wait Another way, is to have one master and several always awake slaves listening as devices. The master uses the SEND command to pipe a command directly into the input stream of the listeners. This makes them wake up instantly (more or less) instead of waiting a second or a minute to check the queue. -Original Message- From: Peter Cheney peter.che...@firstmac.com.au To: 'U2 Users List' u2-users@listserver.u2ug.org Sent: Thu, Mar 6, 2014 4:05 pm Subject: Re: [U2] Multi-threaded phantom processing My thoughts were that the parent will need to allocate the work and in doing so take the incoming data stream, launching a child and somehow pass the work to it. My only current knowledge of doing that within UV basic is writing to a queue and have the child phantom read from that and respond via another queue. Hence my question if there there's another possibly better way of doing it? It is nice to know if a particular phantom is busy or available though so that suggestion is certainly handy thanks. Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson Sent: Friday, 7 March 2014 02:52 To: u2-users@listserver.u2ug.org Subject: Re: [U2] Multi-threaded phantom processing I was referring solely to the point of launching phantoms and how they could communicate back to their master who launched them. Not back to a different server. So the thread was mentioning a master-slave relationship on *a* server as I understood it. And the question asked was how do the slaves and master communicate. -Original Message- From: David A. Green dgr...@dagconsulting.com To: 'U2 Users List' u2-users@listserver.u2ug.org Sent: Thu, Mar 6, 2014 5:43 am Subject: Re: [U2] Multi-threaded phantom processing Will you kinda lost me on this approach. Since the Server socket is on a different machine than the client they don't see each other's locks. Different clients might be on different servers too. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson Sent: Wednesday, March 05, 2014 5:26 PM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Multi-threaded phantom processing Peter is the only point of the parent child communications to address points like 1 I'm busy go away; OR 2 I'm available to work If so, you could simply use a lock-semaphore to address both of those issues.\ -Original Message- From: Horacio Pellegrino pellegrin...@gmail.com To: U2 Users List u2-users@listserver.u2ug.org Sent: Wed, Mar 5, 2014 4:19 pm Subject: Re: [U2] Multi-threaded phantom processing Why don't you have one port listening for requests, and as soon as they come in, you create a new phantom to answer that request ( liberating the listener immediately for the next request ), then when the phantom finishes it could log off. Might be you want to keep a pool of phantoms alive for a while, on the assumption they might be reused. That way you are sill using one license per request, but you have the benefit of pooling... You can always set min-max limit for those phantom-workers. Horacio Pellegrino On Tue, Mar 4, 2014 at 11:30 PM, Peter Cheney peter.che...@firstmac.com.auwrote: Thanks David. As they say Bigger than Ben-Hur! That's a great idea. The server I have is listening on a predetermined port. I trolled through /etc/services and a netstat on the remote host to select something that looked unused. So then if it the main phantom spawns 1-n children (presumably more phantoms?), it would also need to select 1-n ports which I suppose must also be predetermined? Or is there a call to the OS that can return an unused port? We're on AIX but the remote host is HPUX. Cheers Peter
Re: [U2] Multi-threaded phantom processing
Hi Peter Sounds like an interested piece of work, just a couple of questions for you: - what sort of data volumes and update frequencies are involved? - do you need to know if the data has been successfully committed to the remote UV database? - did you consider SQL BCI or even UV/Net (read the different UV versions is an issue and so this may be a constraint) Gregor mentioned SQL BCI as an alternative interface - I've used that before to exchange data between systems and it works well. If you had the same UV version on both hosts - then UV/Net is simple effect way of transmitting data between UV systems with only licenses required on the remote system. With BCI you could utilise the Stored Procedure facility via an ExecDirect() call to push the data to the remote system in your UV file trigger. With UV/Net you could push the items to be copied the by your UV file trigger - thus the updates being written directly onto the remote host. BCI is free/included in UV and of course will utilise a license on the remote host. You need to write your own custom code. Probably a bit easier to maintain than any TCP/IP sockets based interface. Definitely a lot easier to troubleshoot ;-) ! YMMV. UV/Net is licensed on the number user connections on the remote host. Hence the above alternative as designed to eliminate the file-based queue - which in some instances by be desirable to maintain - depending on business, technical or operational needs/objections. Cheers, David -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Peter Cheney Sent: Wednesday, 5 March 2014 12:18 PM To: U2 Users List (u2-users@listserver.u2ug.org) Subject: [U2] Multi-threaded phantom processing Hi Everyone, A project that I’m currently working on requires transmission of transactional and ancillary data from one UV site to site to another. I have the communications all set up and it is working very well via the UV socket functions with a client at one end and a server at the other. Both the client and the server both run as a UV phantom processes written in UV basic and the data being transferred (also encrypted) is more or less real time with various UV file triggers on the client end creating queue items for transmission based on the data content / transaction type. This set up is a single threaded affair and I am wondering how it will cope and what options I’ll have as the volume of data increases. Multi-threading my server process to dish out some of the load might be one possibility and I’ve seen examples in the past where the last digit of a numeric key (i.e. 0-9) is used to determine which one of 10 phantoms to call thus providing an effective distribution of the work. But even then, reading/writing processing data to a file for each child process seems to me to be the only way to achieve parent - child phantom communications at either end. Is there another way perhaps? It may be that I will need 2 sets of phantoms, 1 for the transactions and one for the ancillary data. Although we run UV on AIX the only other requirement is that I must be able to do it within UV Basic. Does anyone out there on the list know of a better way or any alternatives/advice so I can get a better appreciation of what else might be possible please? Many thanks in advance, Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au --- Note: This email (inc all attachments) is for the use of the intended recipient(s) only. Privileged or confidential information may be contained in this communication. If you have received this email in error, please notify the sender immediately and then delete all copies of this message from your computer network. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author's prior permission. If you are the intended recipient and you do not wish to receive similar electronic messages from us in future, then please respond to the sender to this effect. We have taken precautions to minimise the risk of transmitting software viruses, but advise you to carry out your own virus checks on this email and its attachments. We do not accept liability for any loss or damage caused by software viruses and do not represent that this transmission is free from viruses or other defects. Firstmac Limited (ABN 59 094 145 963) (AFSL 290600) --- ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ** IMPORTANT MESSAGE * This e-mail message is intended only for the addressee(s) and contains
Re: [U2] Multi-threaded phantom processing
Initially data volume will be small - probably about 20 customers per day and about 6 records per customer but it is likely to grow very quickly up to some unknown amount. We'll have to see I guess. The records per customer are likely to be the high volume stuff. There is transaction logging on the remote system that I can use and am relying on for data integrity but I did not know about the SQL stuff until this week. Not sure if it would work until I can find time to play with it and see how it might suit us. Some of the other constraints were: no access allowed to the remote unirpc port, the remote UV is below 10.3 and different from ours, neither site has a UV/Net licence in prod. So I think I'm pretty much stuck with file queuing and multiple phantoms for now at least. I also was able to get a successful remote file process going on our test system while waiting for confirmation of UV/Net accessibility. It is very nice and might have been a nice solution too. Then I did some experimenting with the performance requirements on the file triggers and found there to be too much of a lag for the triggers to fire directly to the other end. This would've caused unacceptable problems in testing not to mention going live so a file queuing process had to be devised. In all the file queuing works quite well and can be coded for rock solid reliability so it has a lot of pluses. I can also fine tune the phantom loop times and socket timeouts on the fly at both ends once the data starts flowing and we get some idea of what needs changing. Fingers, toes, arms, legs (and eyes!) all currently crossed. Cheers Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Hona, David Sent: Friday, 7 March 2014 13:41 To: U2 Users List Subject: Re: [U2] Multi-threaded phantom processing Hi Peter Sounds like an interested piece of work, just a couple of questions for you: - what sort of data volumes and update frequencies are involved? - do you need to know if the data has been successfully committed to the remote UV database? - did you consider SQL BCI or even UV/Net (read the different UV versions is an issue and so this may be a constraint) Gregor mentioned SQL BCI as an alternative interface - I've used that before to exchange data between systems and it works well. If you had the same UV version on both hosts - then UV/Net is simple effect way of transmitting data between UV systems with only licenses required on the remote system. With BCI you could utilise the Stored Procedure facility via an ExecDirect() call to push the data to the remote system in your UV file trigger. With UV/Net you could push the items to be copied the by your UV file trigger - thus the updates being written directly onto the remote host. BCI is free/included in UV and of course will utilise a license on the remote host. You need to write your own custom code. Probably a bit easier to maintain than any TCP/IP sockets based interface. Definitely a lot easier to troubleshoot ;-) ! YMMV. UV/Net is licensed on the number user connections on the remote host. Hence the above alternative as designed to eliminate the file-based queue - which in some instances by be desirable to maintain - depending on business, technical or operational needs/objections. Cheers, David -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Peter Cheney Sent: Wednesday, 5 March 2014 12:18 PM To: U2 Users List (u2-users@listserver.u2ug.org) Subject: [U2] Multi-threaded phantom processing Hi Everyone, A project that I’m currently working on requires transmission of transactional and ancillary data from one UV site to site to another. I have the communications all set up and it is working very well via the UV socket functions with a client at one end and a server at the other. Both the client and the server both run as a UV phantom processes written in UV basic and the data being transferred (also encrypted) is more or less real time with various UV file triggers on the client end creating queue items for transmission based on the data content / transaction type. This set up is a single threaded affair and I am wondering how it will cope and what options I’ll have as the volume of data increases. Multi-threading my server process to dish out some of the load might be one possibility and I’ve seen examples in the past where the last digit of a numeric key (i.e. 0-9) is used to determine which one of 10 phantoms to call thus providing an effective distribution of the work. But even then, reading/writing processing data to a file for each child process seems to me to be the only way to achieve parent - child phantom communications at either end
Re: [U2] Multi-threaded phantom processing
I don't know how you would query the OS on the fly, but you can do a manual search for a bank of 10 or 20 unused ports and claim them. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Peter Cheney Sent: Tuesday, March 04, 2014 9:30 PM To: 'U2 Users List' Subject: Re: [U2] Multi-threaded phantom processing Thanks David. As they say Bigger than Ben-Hur! That's a great idea. The server I have is listening on a predetermined port. I trolled through /etc/services and a netstat on the remote host to select something that looked unused. So then if it the main phantom spawns 1-n children (presumably more phantoms?), it would also need to select 1-n ports which I suppose must also be predetermined? Or is there a call to the OS that can return an unused port? We're on AIX but the remote host is HPUX. Cheers Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Multi-threaded phantom processing
Instead of phantoms we us a different approach. Our middle-ware built on Apache Tomcat and UniObjects for Java replicates data from Universe to Universe or Unidata to Unidata. Using the built in routines in UOJ we are able to handle over 10,000 to 20,000 transactions per day. This technology uses a DataSet or a select list of data from a single file in a single operation. Then you write the DataSet to the other machine. Regards, Doug On Wed, Mar 5, 2014 at 5:34 AM, David A. Green dgr...@dagconsulting.comwrote: I don't know how you would query the OS on the fly, but you can do a manual search for a bank of 10 or 20 unused ports and claim them. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of Peter Cheney Sent: Tuesday, March 04, 2014 9:30 PM To: 'U2 Users List' Subject: Re: [U2] Multi-threaded phantom processing Thanks David. As they say Bigger than Ben-Hur! That's a great idea. The server I have is listening on a predetermined port. I trolled through /etc/services and a netstat on the remote host to select something that looked unused. So then if it the main phantom spawns 1-n children (presumably more phantoms?), it would also need to select 1-n ports which I suppose must also be predetermined? Or is there a call to the OS that can return an unused port? We're on AIX but the remote host is HPUX. Cheers Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Multi-threaded phantom processing
Why don't you have one port listening for requests, and as soon as they come in, you create a new phantom to answer that request ( liberating the listener immediately for the next request ), then when the phantom finishes it could log off. Might be you want to keep a pool of phantoms alive for a while, on the assumption they might be reused. That way you are sill using one license per request, but you have the benefit of pooling... You can always set min-max limit for those phantom-workers. Horacio Pellegrino On Tue, Mar 4, 2014 at 11:30 PM, Peter Cheney peter.che...@firstmac.com.auwrote: Thanks David. As they say Bigger than Ben-Hur! That's a great idea. The server I have is listening on a predetermined port. I trolled through /etc/services and a netstat on the remote host to select something that looked unused. So then if it the main phantom spawns 1-n children (presumably more phantoms?), it would also need to select 1-n ports which I suppose must also be predetermined? Or is there a call to the OS that can return an unused port? We're on AIX but the remote host is HPUX. Cheers Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of David A. Green Sent: Wednesday, 5 March 2014 13:01 To: 'U2 Users List' Subject: Re: [U2] Multi-threaded phantom processing Peter, first of all let me congratulate you on this achievement. Having just finished a similar feat using UniData I understand the complexities that went into it. The system I'm current working with will be relatively low traffic. But I was pondering on the method of having a server socket that would just spawn a transfer server socket and send the socket address to the calling client. Then the client disconnects from the main server and just uses the transfer server to complete the task. Having 10 open phantoms with sockets will use up 10 licenses. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of Peter Cheney Sent: Tuesday, March 04, 2014 6:18 PM To: U2 Users List (u2-users@listserver.u2ug.org) Subject: [U2] Multi-threaded phantom processing Hi Everyone, A project that I'm currently working on requires transmission of transactional and ancillary data from one UV site to site to another. I have the communications all set up and it is working very well via the UV socket functions with a client at one end and a server at the other. Both the client and the server both run as a UV phantom processes written in UV basic and the data being transferred (also encrypted) is more or less real time with various UV file triggers on the client end creating queue items for transmission based on the data content / transaction type. This set up is a single threaded affair and I am wondering how it will cope and what options I'll have as the volume of data increases. Multi-threading my server process to dish out some of the load might be one possibility and I've seen examples in the past where the last digit of a numeric key (i.e. 0-9) is used to determine which one of 10 phantoms to call thus providing an effective distribution of the work. But even then, reading/writing processing data to a file for each child process seems to me to be the only way to achieve parent - child phantom communications at either end. Is there another way perhaps? It may be that I will need 2 sets of phantoms, 1 for the transactions and one for the ancillary data. Although we run UV on AIX the only other requirement is that I must be able to do it within UV Basic. Does anyone out there on the list know of a better way or any alternatives/advice so I can get a better appreciation of what else might be possible please? Many thanks in advance, Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au --- Note: This email (inc all attachments) is for the use of the intended recipient(s) only. Privileged or confidential information may be contained in this communication. If you have received this email in error, please notify the sender immediately and then delete all copies of this message from your computer network. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author's prior permission. If you are the intended recipient and you do not wish to receive similar electronic messages from us in future, then please respond to the sender to this effect. We have taken precautions to minimise the risk of transmitting software viruses, but advise you to carry out your
Re: [U2] Multi-threaded phantom processing
Peter is the only point of the parent child communications to address points like 1 I'm busy go away; OR 2 I'm available to work If so, you could simply use a lock-semaphore to address both of those issues.\ -Original Message- From: Horacio Pellegrino pellegrin...@gmail.com To: U2 Users List u2-users@listserver.u2ug.org Sent: Wed, Mar 5, 2014 4:19 pm Subject: Re: [U2] Multi-threaded phantom processing Why don't you have one port listening for requests, and as soon as they come in, you create a new phantom to answer that request ( liberating the listener immediately for the next request ), then when the phantom finishes it could log off. Might be you want to keep a pool of phantoms alive for a while, on the assumption they might be reused. That way you are sill using one license per request, but you have the benefit of pooling... You can always set min-max limit for those phantom-workers. Horacio Pellegrino On Tue, Mar 4, 2014 at 11:30 PM, Peter Cheney peter.che...@firstmac.com.auwrote: Thanks David. As they say Bigger than Ben-Hur! That's a great idea. The server I have is listening on a predetermined port. I trolled through /etc/services and a netstat on the remote host to select something that looked unused. So then if it the main phantom spawns 1-n children (presumably more phantoms?), it would also need to select 1-n ports which I suppose must also be predetermined? Or is there a call to the OS that can return an unused port? We're on AIX but the remote host is HPUX. Cheers Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of David A. Green Sent: Wednesday, 5 March 2014 13:01 To: 'U2 Users List' Subject: Re: [U2] Multi-threaded phantom processing Peter, first of all let me congratulate you on this achievement. Having just finished a similar feat using UniData I understand the complexities that went into it. The system I'm current working with will be relatively low traffic. But I was pondering on the method of having a server socket that would just spawn a transfer server socket and send the socket address to the calling client. Then the client disconnects from the main server and just uses the transfer server to complete the task. Having 10 open phantoms with sockets will use up 10 licenses. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of Peter Cheney Sent: Tuesday, March 04, 2014 6:18 PM To: U2 Users List (u2-users@listserver.u2ug.org) Subject: [U2] Multi-threaded phantom processing Hi Everyone, A project that I'm currently working on requires transmission of transactional and ancillary data from one UV site to site to another. I have the communications all set up and it is working very well via the UV socket functions with a client at one end and a server at the other. Both the client and the server both run as a UV phantom processes written in UV basic and the data being transferred (also encrypted) is more or less real time with various UV file triggers on the client end creating queue items for transmission based on the data content / transaction type. This set up is a single threaded affair and I am wondering how it will cope and what options I'll have as the volume of data increases. Multi-threading my server process to dish out some of the load might be one possibility and I've seen examples in the past where the last digit of a numeric key (i.e. 0-9) is used to determine which one of 10 phantoms to call thus providing an effective distribution of the work. But even then, reading/writing processing data to a file for each child process seems to me to be the only way to achieve parent - child phantom communications at either end. Is there another way perhaps? It may be that I will need 2 sets of phantoms, 1 for the transactions and one for the ancillary data. Although we run UV on AIX the only other requirement is that I must be able to do it within UV Basic. Does anyone out there on the list know of a better way or any alternatives/advice so I can get a better appreciation of what else might be possible please? Many thanks in advance, Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au --- Note: This email (inc all attachments) is for the use of the intended recipient(s) only. Privileged or confidential information may be contained in this communication. If you have received this email in error, please notify the sender immediately and then delete all copies of this message from your computer network
Re: [U2] Multi-threaded phantom processing
As elegant as some of these ideas are, it might be possible, depending on the number and nature of the transactions, to batch them up every n seconds and use rcp to copy them to an in box on the remote machine. -Rick On Mar 5, 2014, at 4:19 PM, Horacio Pellegrino pellegrin...@gmail.com wrote: Why don't you have one port listening for requests, and as soon as they come in, you create a new phantom to answer that request ( liberating the listener immediately for the next request ), then when the phantom finishes it could log off. Might be you want to keep a pool of phantoms alive for a while, on the assumption they might be reused. That way you are sill using one license per request, but you have the benefit of pooling... You can always set min-max limit for those phantom-workers. Horacio Pellegrino On Tue, Mar 4, 2014 at 11:30 PM, Peter Cheney peter.che...@firstmac.com.auwrote: Thanks David. As they say Bigger than Ben-Hur! That's a great idea. The server I have is listening on a predetermined port. I trolled through /etc/services and a netstat on the remote host to select something that looked unused. So then if it the main phantom spawns 1-n children (presumably more phantoms?), it would also need to select 1-n ports which I suppose must also be predetermined? Or is there a call to the OS that can return an unused port? We're on AIX but the remote host is HPUX. Cheers Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of David A. Green Sent: Wednesday, 5 March 2014 13:01 To: 'U2 Users List' Subject: Re: [U2] Multi-threaded phantom processing Peter, first of all let me congratulate you on this achievement. Having just finished a similar feat using UniData I understand the complexities that went into it. The system I'm current working with will be relatively low traffic. But I was pondering on the method of having a server socket that would just spawn a transfer server socket and send the socket address to the calling client. Then the client disconnects from the main server and just uses the transfer server to complete the task. Having 10 open phantoms with sockets will use up 10 licenses. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of Peter Cheney Sent: Tuesday, March 04, 2014 6:18 PM To: U2 Users List (u2-users@listserver.u2ug.org) Subject: [U2] Multi-threaded phantom processing Hi Everyone, A project that I'm currently working on requires transmission of transactional and ancillary data from one UV site to site to another. I have the communications all set up and it is working very well via the UV socket functions with a client at one end and a server at the other. Both the client and the server both run as a UV phantom processes written in UV basic and the data being transferred (also encrypted) is more or less real time with various UV file triggers on the client end creating queue items for transmission based on the data content / transaction type. This set up is a single threaded affair and I am wondering how it will cope and what options I'll have as the volume of data increases. Multi-threading my server process to dish out some of the load might be one possibility and I've seen examples in the past where the last digit of a numeric key (i.e. 0-9) is used to determine which one of 10 phantoms to call thus providing an effective distribution of the work. But even then, reading/writing processing data to a file for each child process seems to me to be the only way to achieve parent - child phantom communications at either end. Is there another way perhaps? It may be that I will need 2 sets of phantoms, 1 for the transactions and one for the ancillary data. Although we run UV on AIX the only other requirement is that I must be able to do it within UV Basic. Does anyone out there on the list know of a better way or any alternatives/advice so I can get a better appreciation of what else might be possible please? Many thanks in advance, Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au --- Note: This email (inc all attachments) is for the use of the intended recipient(s) only. Privileged or confidential information may be contained in this communication. If you have received this email in error, please notify the sender immediately and then delete all copies of this message from your computer network. If you are not the intended recipient, you must not keep, use, disclose, copy
Re: [U2] Multi-threaded phantom processing
So I went looking for options to query the OS for a free port and found an answer on stackoverflow that I think is simply brilliant. (I hope it's OK to quote from others here? Apologies if not.) If your server was told what port to use, just bind() it. Seriously. Sure, you could parse /proc/net/tcp and see if the port's in use. But then what? You still need to call bind() now that you know your port is free, and it'll tell you if the port was free then anyway, and so there was no point in groveling through /proc/net/tcp and doing all that (slow!) string production and parsing and extra kernel trips through not-very-well-optimized (read: super slow compared to bind()) diagnostic paths, just to get information that could well be out of date before you even finished parsing it. So just call bind() and be happy. The full article is here for those interested: http://stackoverflow.com/questions/10294515/how-do-i-find-in-c-that-a-port-is-free-to-use So back in UV Basic we'd just call initServerSocket with the desired port number and action according to success or failure! Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of David A. Green Sent: Wednesday, 5 March 2014 22:35 To: 'U2 Users List' Subject: Re: [U2] Multi-threaded phantom processing I don't know how you would query the OS on the fly, but you can do a manual search for a bank of 10 or 20 unused ports and claim them. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Peter Cheney Sent: Tuesday, March 04, 2014 9:30 PM To: 'U2 Users List' Subject: Re: [U2] Multi-threaded phantom processing Thanks David. As they say Bigger than Ben-Hur! That's a great idea. The server I have is listening on a predetermined port. I trolled through /etc/services and a netstat on the remote host to select something that looked unused. So then if it the main phantom spawns 1-n children (presumably more phantoms?), it would also need to select 1-n ports which I suppose must also be predetermined? Or is there a call to the OS that can return an unused port? We're on AIX but the remote host is HPUX. Cheers Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users --- Note: This email (inc all attachments) is for the use of the intended recipient(s) only. Privileged or confidential information may be contained in this communication. If you have received this email in error, please notify the sender immediately and then delete all copies of this message from your computer network. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author's prior permission. If you are the intended recipient and you do not wish to receive similar electronic messages from us in future, then please respond to the sender to this effect. We have taken precautions to minimise the risk of transmitting software viruses, but advise you to carry out your own virus checks on this email and its attachments. We do not accept liability for any loss or damage caused by software viruses and do not represent that this transmission is free from viruses or other defects. Firstmac Limited (ABN 59 094 145 963) (AFSL 290600) --- ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Multi-threaded phantom processing
There are options you can explore using the BASIC SQL Client Interface. These will allow basic programs running as phantoms to establish their own connection to the remote site and run subroutines supplied with data as parameters. This would bypass the need for multiple server phantoms listening and reacting to socket connections. You could also look at hosting web services on the AIX server and use the submitrequest() functions. Gregor Scott System Group Manager 549 Blackburn Rd Mount Waverley VIC 3149 Australia P: +61 3 9535 F: +61 3 9535 2274 E: gregor.sc...@pentanasolutions.com W: pentanasolutions.com Pentana Solutions accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information provided, unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Gregor Scott System Group Manager 549 Blackburn Rd Mount Waverley VIC 3149 Australia P: +61 3 9535 F: +61 3 9535 2274 E: gregor.sc...@pentanasolutions.com W: pentanasolutions.com Pentana Solutions accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information provided, unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Multi-threaded phantom processing
Peter, first of all let me congratulate you on this achievement. Having just finished a similar feat using UniData I understand the complexities that went into it. The system I'm current working with will be relatively low traffic. But I was pondering on the method of having a server socket that would just spawn a transfer server socket and send the socket address to the calling client. Then the client disconnects from the main server and just uses the transfer server to complete the task. Having 10 open phantoms with sockets will use up 10 licenses. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Peter Cheney Sent: Tuesday, March 04, 2014 6:18 PM To: U2 Users List (u2-users@listserver.u2ug.org) Subject: [U2] Multi-threaded phantom processing Hi Everyone, A project that I’m currently working on requires transmission of transactional and ancillary data from one UV site to site to another. I have the communications all set up and it is working very well via the UV socket functions with a client at one end and a server at the other. Both the client and the server both run as a UV phantom processes written in UV basic and the data being transferred (also encrypted) is more or less real time with various UV file triggers on the client end creating queue items for transmission based on the data content / transaction type. This set up is a single threaded affair and I am wondering how it will cope and what options I’ll have as the volume of data increases. Multi-threading my server process to dish out some of the load might be one possibility and I’ve seen examples in the past where the last digit of a numeric key (i.e. 0-9) is used to determine which one of 10 phantoms to call thus providing an effective distribution of the work. But even then, reading/writing processing data to a file for each child process seems to me to be the only way to achieve parent - child phantom communications at either end. Is there another way perhaps? It may be that I will need 2 sets of phantoms, 1 for the transactions and one for the ancillary data. Although we run UV on AIX the only other requirement is that I must be able to do it within UV Basic. Does anyone out there on the list know of a better way or any alternatives/advice so I can get a better appreciation of what else might be possible please? Many thanks in advance, Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au --- Note: This email (inc all attachments) is for the use of the intended recipient(s) only. Privileged or confidential information may be contained in this communication. If you have received this email in error, please notify the sender immediately and then delete all copies of this message from your computer network. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author's prior permission. If you are the intended recipient and you do not wish to receive similar electronic messages from us in future, then please respond to the sender to this effect. We have taken precautions to minimise the risk of transmitting software viruses, but advise you to carry out your own virus checks on this email and its attachments. We do not accept liability for any loss or damage caused by software viruses and do not represent that this transmission is free from viruses or other defects. Firstmac Limited (ABN 59 094 145 963) (AFSL 290600) --- ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Multi-threaded phantom processing
Thanks Gregor I will look into the SQL interface and see what I can glean. I did look at the submit request stuff and had a working proof of concept between my workstation and our dev AIX host but the destination UV system is an earlier release and didn't support a lot of that stuff. So we had to find a middle of the road solution and sockets was it. For the moment at least until they upgrade. :) Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Gregor Scott Sent: Wednesday, 5 March 2014 11:29 To: U2 Users List Subject: Re: [U2] Multi-threaded phantom processing There are options you can explore using the BASIC SQL Client Interface. These will allow basic programs running as phantoms to establish their own connection to the remote site and run subroutines supplied with data as parameters. This would bypass the need for multiple server phantoms listening and reacting to socket connections. You could also look at hosting web services on the AIX server and use the submitrequest() functions. Gregor Scott System Group Manager 549 Blackburn Rd Mount Waverley VIC 3149 Australia P: +61 3 9535 F: +61 3 9535 2274 E: gregor.sc...@pentanasolutions.com W: pentanasolutions.com Pentana Solutions accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information provided, unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Gregor Scott System Group Manager 549 Blackburn Rd Mount Waverley VIC 3149 Australia P: +61 3 9535 F: +61 3 9535 2274 E: gregor.sc...@pentanasolutions.com W: pentanasolutions.com Pentana Solutions accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information provided, unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users --- Note: This email (inc all attachments) is for the use of the intended recipient(s) only. Privileged or confidential information may be contained in this communication. If you have received this email in error, please notify the sender immediately and then delete all copies of this message from your computer network. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author's prior permission. If you are the intended recipient and you do not wish to receive similar electronic messages from us in future, then please respond to the sender to this effect. We have taken precautions to minimise the risk of transmitting software viruses, but advise you to carry out your own virus checks on this email and its attachments. We do not accept liability for any loss or damage caused by software viruses and do not represent that this transmission is free from viruses or other defects. Firstmac Limited (ABN 59 094 145 963) (AFSL 290600) --- ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Multi-threaded phantom processing
Thanks David. As they say Bigger than Ben-Hur! That's a great idea. The server I have is listening on a predetermined port. I trolled through /etc/services and a netstat on the remote host to select something that looked unused. So then if it the main phantom spawns 1-n children (presumably more phantoms?), it would also need to select 1-n ports which I suppose must also be predetermined? Or is there a call to the OS that can return an unused port? We're on AIX but the remote host is HPUX. Cheers Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of David A. Green Sent: Wednesday, 5 March 2014 13:01 To: 'U2 Users List' Subject: Re: [U2] Multi-threaded phantom processing Peter, first of all let me congratulate you on this achievement. Having just finished a similar feat using UniData I understand the complexities that went into it. The system I'm current working with will be relatively low traffic. But I was pondering on the method of having a server socket that would just spawn a transfer server socket and send the socket address to the calling client. Then the client disconnects from the main server and just uses the transfer server to complete the task. Having 10 open phantoms with sockets will use up 10 licenses. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Peter Cheney Sent: Tuesday, March 04, 2014 6:18 PM To: U2 Users List (u2-users@listserver.u2ug.org) Subject: [U2] Multi-threaded phantom processing Hi Everyone, A project that I’m currently working on requires transmission of transactional and ancillary data from one UV site to site to another. I have the communications all set up and it is working very well via the UV socket functions with a client at one end and a server at the other. Both the client and the server both run as a UV phantom processes written in UV basic and the data being transferred (also encrypted) is more or less real time with various UV file triggers on the client end creating queue items for transmission based on the data content / transaction type. This set up is a single threaded affair and I am wondering how it will cope and what options I’ll have as the volume of data increases. Multi-threading my server process to dish out some of the load might be one possibility and I’ve seen examples in the past where the last digit of a numeric key (i.e. 0-9) is used to determine which one of 10 phantoms to call thus providing an effective distribution of the work. But even then, reading/writing processing data to a file for each child process seems to me to be the only way to achieve parent - child phantom communications at either end. Is there another way perhaps? It may be that I will need 2 sets of phantoms, 1 for the transactions and one for the ancillary data. Although we run UV on AIX the only other requirement is that I must be able to do it within UV Basic. Does anyone out there on the list know of a better way or any alternatives/advice so I can get a better appreciation of what else might be possible please? Many thanks in advance, Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au --- Note: This email (inc all attachments) is for the use of the intended recipient(s) only. Privileged or confidential information may be contained in this communication. If you have received this email in error, please notify the sender immediately and then delete all copies of this message from your computer network. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author's prior permission. If you are the intended recipient and you do not wish to receive similar electronic messages from us in future, then please respond to the sender to this effect. We have taken precautions to minimise the risk of transmitting software viruses, but advise you to carry out your own virus checks on this email and its attachments. We do not accept liability for any loss or damage caused by software viruses and do not represent that this transmission is free from viruses or other defects. Firstmac Limited (ABN 59 094 145 963) (AFSL 290600) --- ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
