Re: Batch process VS Started task
On 19 February 2012 17:04, Ed Gould wrote: > Not sure if this is what you are looking for but > There is a facility called "execution batch monitoring" (at least in JES2) > You "feed" it input and it creates output (to your specifications). It was > originally designed for batch compiles but it could be adapted to something > like you want (?). XBM was once a high performance and useful facility, because it avoided the overhead of traditional initiation. In particular, it was popular at universities where a large number of standard student compile & execute jobs could be processed efficiently. But a decade or so ago IBM removed this version of XBM, and replaced it with one that is best described by their own words in the JES2 Init & Tuning book: "Note that the majority of the previous benefits of using the execution batch scheduling facility no longer exist. You might want to convert jobs that currently use the facility to jobs with the appropriate execution JCL as soon as convenient." Today's XBM is just a slightly simplified way of submitting standard jobs, and offers no performance benefit. Tony H. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Batch process VS Started task
Hi Magen, Brief design points for a similar home-grown product that is 10 years in production: a) define a DB2 "workload" table where each batch process will post the DS-name of a dataset containing the "new work"; in our case this table is populated by the last step of the NDM transmission JCL [or ANY other job that wants a sequential file to be "processed"], so whenever a new batch file is created, its name is inserted in the DB2 table b) develop a long-running multi-tasking job, that would have a single task to periodically scan the above DB2 table loking for "new work" and once found, attach yet another subtask to "process" this particular "work"; in our case this subtask is merely copying the original sequential file to an MQ queue drained by multiple CICS draining tasks, while the whole job stays up for a full week before getting recycled; one more subtask is design to process operators commands that allow to pause, resume, stop transmission, etc. [actually, there are 2 such subtasks, the second is "listening" on an MQ queue designed to be populated from a CICS screen] c) you can also add much more granular control to the above by having some external "rules" [in our case DB2 tables] that would dictate exactly how to recognize and manage a particular data type [how to validate data headers, what MQ queue(s) to choose, how to recover after a failure, etc.] Please, e-mail me offline if you'd like a clarify some questions. HTH, -Victor- == In order to make systems more "online" we are looking for a way to run the process for each record all day long instead of a daily run, and doing so with minimum as possible application changes. Magen -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Batch process VS Started task
Both CICS and IMS have (rather different) facilities for mini-batching, triggering FIFO processing of all of them after an waiting-transactions queue reaches a specified length L > 0. This processing threshold L can be varied without making other changes, and doing so provides a mechanism for balancing initialization and transaction-processing path lengths in a way that minimizes their sum. This optimizing machinery is very important , and even if a RYO scheme is used it should be provided. Limiting the design alternatives considered to just 1) batch, all at the same time, or 2) one-at-a-time processing is unwise. John Gilmore, Ashland, MA 01721 - USA -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Batch process VS Started task
On Feb 19, 2012, at 5:27 PM, Joel C. Ewing wrote: The described record arrival rate averages at just under 58 records per second. Somehow I don't think XBM was designed with that high of transaction arrival rate in mind, processing each record as a separate transaction. I would think that a high-speed transaction processing environment (CICS, IMS) would do better than XBM, although those would probably still involve more overhead than a specialized transaction-processing STC optimized just for your own peculiar transactions (but I agree a separate specialized STC would be more of a maintenance headache). There is also an intermediate approach where you collect records by some means and then periodically fire off a process or transaction to process records collected since the last time the process was run. Another consideration: If the current batch processing is done at a time of lower system system load and involves significant work per record, moving this processing into peak processing times could have significant impact on your peak MSU requirement and a significant impact on your costs. I suspect your record arrival rate is not a constant throughout the day but also has its peaks. If those peak record arrival rates actually correspond to current periods of peak system load, then the impact on the peak system MSU requirements of moving this load would be even greater than one would expect from the average record rate alone. In other words, if there is a perceived business need to process the records in a more timely manner, be sure those footing the bill are aware it may not be a "free lunch". JC Ewing Joel: I think a LOT depends on how much processing is needed for each "transaction". If it is just syntax checking and minimal editing then maybe XBM is a possibility. Although your scenario is valid as well. It is hard to give out much information without a lot more detail. We did XBM jobs and the number varied all over the place. Typically each "job" took about 5 minutes of CPU time (168 slow I know) it was both cpu and IO intensive (essentially Assembler H compiles and with Major macros). I did not want to suggest that XBM is a final answer in any case. It was used by us in a highly unusual manner. I hope the original questioner comes back with a little bit more information. CICS or IMS is (like you suggested) viable alternative but it is complicated (bother suggestions) and hard to say this way or the otherway. Ed -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Batch process VS Started task
IMHO, doing a STC wont make your processing "Online". STC is just a STC - making your process running under a STC wont turn it "online". If you plan to process the "records" one by one, I would advice to use a transaction system, such as CICS or IMS. Using a STC like you described will just turn your current process into a "batch process that processes a record at time". There are several ways to trigger transactions based on events - just choose one that you have on your installation If you plan to process the "records" in batches - I would advice do that in batch. You can trigger batch jobs on most of Scheduling softwares based on events - even on record arrival or file creation. You can take a look at your Scheduling software to see what options it has available. Personally i would use the second approach - doing a batch triggered job. A batch job can process any "kind" of "file" (seq, vsam, databases) and you can program nice recovery without creating a maintenance nightmare. The only exception to this is if you have a business need to process each record as soon it arrives. ET Lima Mensagem Original From: "Jan MOEYERSONS" To: IBM-MAIN@bama.ua.edu Sent: Seg, Fevereiro 20, 2012 9:47 am Subject: Re: Batch process VS Started task On Sun, 19 Feb 2012 15:25:32 -0600, Magen Margalit wrote: >One idea that came up is to convert the process to a "self developed" STC >which will be triggered by a record on an MQ queue and will run as STC all the >batch process programs I think MQ is indeed a very good way to collect these records. You have plenty of configuration options in there to cater for recoverability (or not, if you don't need it), etc. And CICS or IMS can be used at the other end to drain the queue; using MQ triggering it becomes even simpler to start these processing transactions. Both CICS and IMS can easily handle the number of transactions and offer excelent recoverability. >To me it seems like a bad idea because having a "self developed" STC in production >create a maintenance gap (and where there is one STC a second one >will soon to follow...)... I agree on that. Most shops I know are working to eliminate as much as possible the home-grown stuff for exactly that reason: maintenance headache... >Are there any "self developed" STC's at your shop? We do have one. It caters for a very specific need of authentication in a client/server context. And each time we upgrade z/OS, we spend a lot of time testing... Cheers, Jantje. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Batch process VS Started task
On Sun, 19 Feb 2012 15:25:32 -0600, Magen Margalit wrote: >One idea that came up is to convert the process to a "self developed" STC >which will be triggered by a record on an MQ queue and will run as STC all the >batch process programs I think MQ is indeed a very good way to collect these records. You have plenty of configuration options in there to cater for recoverability (or not, if you don't need it), etc. And CICS or IMS can be used at the other end to drain the queue; using MQ triggering it becomes even simpler to start these processing transactions. Both CICS and IMS can easily handle the number of transactions and offer excelent recoverability. >To me it seems like a bad idea because having a "self developed" STC in >production >create a maintenance gap (and where there is one STC a second one >will soon to follow...)... I agree on that. Most shops I know are working to eliminate as much as possible the home-grown stuff for exactly that reason: maintenance headache... >Are there any "self developed" STC's at your shop? We do have one. It caters for a very specific need of authentication in a client/server context. And each time we upgrade z/OS, we spend a lot of time testing... Cheers, Jantje. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Batch process VS Started task
I would divide the problem into two: 1. how to make the file be transactional. to do this, you can use MQ as the mechanism to write into a "file". 2. where to fire the transaction. You can "write" the message into a queue managed by CICS and LINK to the application program (not START TRAN that will cause a lot of overhead on CICS). CICS will give you fine recovery and error handling and minimize the changes you need to make in the current program (is you run in under OTE). other alternatives are MQ to DB2 and define a trriger on the table to fire a Cobol (or other language) stored procedure. ITschak On Mon, Feb 20, 2012 at 9:03 AM, Magen Margalit wrote: > Hi. > > The records arrives from a few sources to a few seq files > The collection process collect the records from the files and then the > batch process begins. > In the new situation, the batch process or STC will handle each record and > the collection process will be eliminated. > > Regarding the maintenance gap, I'm not concern from a complicated code, > but from using functions that IBM will someday > will no longer support, i.e. LE functions. When you upgrade a zOS version > you contact all the software vendors and > check for compatibility to the new zOS version, on a "self developed" STC > you cann't do that... > > Magen > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN > -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Batch process VS Started task
Hi. The records arrives from a few sources to a few seq files The collection process collect the records from the files and then the batch process begins. In the new situation, the batch process or STC will handle each record and the collection process will be eliminated. Regarding the maintenance gap, I'm not concern from a complicated code, but from using functions that IBM will someday will no longer support, i.e. LE functions. When you upgrade a zOS version you contact all the software vendors and check for compatibility to the new zOS version, on a "self developed" STC you cann't do that... Magen -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Batch process VS Started task
On 20/02/2012 08:25 AM, Magen Margalit wrote: Hi list. We have a daily betch job that is processing as input records which has been collected all day. volume of records is about 5 millions for 24 hours. In order to make systems more "online" we are looking for a way to run the process for each record all day long instead of a daily run, and doing so with minimum as possible application changes. One idea that came up is to convert the process to a "self developed" STC which will be triggered by a record on an MQ queue and will run as STC all the batch process programs To me it seems like a bad idea because having a "self developed" STC in production create a maintenance gap (and where there is one STC a second one will soon to follow...)... Are there other advantages / dis-advantages regarding a "self developed" STC ? Are there any "self developed" STC's at your shop? Any other ideas on how to approach this issue? Thanks in advanced. Magen Magen, Some good ideas from the other responsders but IMHO they are all putting the cart before the horse. Look at the collection process, what is this process. Can it be modified to do the either batch process, more details would be nice, or batch the batch process. Ken -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Batch process VS Started task
If you can fall behind when the system is busy, would an IMS DC / Batch Message Program be appropiate? On Sun, Feb 19, 2012 at 4:40 PM, Lizette Koehler wrote: >> >> Hi list. >> >> We have a daily betch job that is processing as input records which has > been collected >> all day. >> volume of records is about 5 millions for 24 hours. >> >> In order to make systems more "online" we are looking for a way to run the > process >> for each record all day long instead of a daily run, and doing so with > minimum as >> possible application changes. >> <> >> Magen >> > > Perhaps you could review the BATCH PIPES process in z/OS or if you have MQ, > perhaps that could support what you need. > > Other options could be > IMS solution > MQ solution > CICS solution > DB2 Solution > > We currently have a need to send out orders when a DB2 table is updated. To > do this we use our CA-ESP Scheduling software to monitor the DB2 table, when > there is a row change, it triggers the process to action the order. > > If there were some more details on your problem, we might be able to come up > with more specific answers? > > Lizette > If you can fall behind when the system is busy (to avoid having to increase CPUs), would an IMS DC / Batch Message Program be appropiate? -- Mike A Schwab, Springfield IL USA Where do Forest Rangers go to get away from it all? -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Batch process VS Started task
I like the Batch PIPES idea. Of course, since Batch PIPES is a product which costs extra, we can't use it. So I'd use a named UNIX pipe instead. I'm fairly sure that you can have the creating transaction simply write to it like a sequential file. And the processing STC would read it like a sequential file. Should be rather simple to implement. On Sun, 2012-02-19 at 17:40 -0500, Lizette Koehler wrote: > > > > Hi list. > > > > We have a daily betch job that is processing as input records which has > been collected > > all day. > > volume of records is about 5 millions for 24 hours. > > > > In order to make systems more "online" we are looking for a way to run the > process > > for each record all day long instead of a daily run, and doing so with > minimum as > > possible application changes. > > > > One idea that came up is to convert the process to a "self developed" STC > which will be > > triggered by a record on an MQ queue and will run as STC all the batch > process > > programs > > > > To me it seems like a bad idea because having a "self developed" STC in > production > > create a maintenance gap (and where there is one STC a second one will > soon to > > follow...)... > > > > Are there other advantages / dis-advantages regarding a "self developed" > STC ? > > > > Are there any "self developed" STC's at your shop? > > > > Any other ideas on how to approach this issue? > > > > Thanks in advanced. > > > > Magen > > > > Perhaps you could review the BATCH PIPES process in z/OS or if you have MQ, > perhaps that could support what you need. > > Other options could be > IMS solution > MQ solution > CICS solution > DB2 Solution > > We currently have a need to send out orders when a DB2 table is updated. To > do this we use our CA-ESP Scheduling software to monitor the DB2 table, when > there is a row change, it triggers the process to action the order. > > If there were some more details on your problem, we might be able to come up > with more specific answers? > > Lizette > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN -- John McKown Maranatha! <>< -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Batch process VS Started task
On 2/19/2012 4:25 PM, Magen Margalit wrote: We have a daily betch job that is processing as input records which has been collected all day. volume of records is about 5 millions for 24 hours. With allowance for growth, you're looking at 100-200 records per second, and that's well within the capacity of modern equipment. Depending on how the records are created, you have overhead for serialization and backup (do you need an audit trail to recreate records if the file is lost?). You would need to do some testing to see whether the serialization and processing overhead for live processing is tolerable, since the batch presumably offers some savings in overhead. If the records are created by a single source, then instead of an STC program, you could consider a subtask, a PC routine, or (gasp) even a user SVC. For multiple tasks, either an SVC or a PC routine might do. The preferred method depends on how the data are created, how they are placed into the batch input file, and what processing they are used for. In order to make systems more "online" we are looking for a way to run the process for each record all day long instead of a daily run, and doing so with minimum as possible application changes. If you have a PUT, WRITE, or similar function to create the record, it's easy enough to replace that with a PC or SVC call. But you will need to develop contingency plans for data sets becoming full, or other occurrences that could prevent live updating. One idea that came up is to convert the process to a "self developed" STC which will be triggered by a record on an MQ queue and will run as STC all the batch process programs To me it seems like a bad idea because having a "self developed" STC in production create a maintenance gap (and where there is one STC a second one will soon to follow...)... When I worked at ADR in the sixties, we developed our own data collection and accounting routine long before IBM offered SMF. We had our own tape library software long before UCC-1 (or TLS). We had our own security software long before RACF (in addition we used IBM passwords, but only for critical and SYS1 data sets). In each case the staff contained more than one person familiar with the software and the ability to maintain it. While IBM is trying to make the system more accessible (i.e., dumbed down) your installation should have enough staff familiar with internals to write and maintain the code. Basically you need whatever the batch process is doing, and add provision for backup and serialization. Of course it's always possible to overcomplicate things to the point where nobody can maintain it. Gerhard Postpischil Bradford, VT -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Batch process VS Started task
The described record arrival rate averages at just under 58 records per second. Somehow I don't think XBM was designed with that high of transaction arrival rate in mind, processing each record as a separate transaction. I would think that a high-speed transaction processing environment (CICS, IMS) would do better than XBM, although those would probably still involve more overhead than a specialized transaction-processing STC optimized just for your own peculiar transactions (but I agree a separate specialized STC would be more of a maintenance headache). There is also an intermediate approach where you collect records by some means and then periodically fire off a process or transaction to process records collected since the last time the process was run. Another consideration: If the current batch processing is done at a time of lower system system load and involves significant work per record, moving this processing into peak processing times could have significant impact on your peak MSU requirement and a significant impact on your costs. I suspect your record arrival rate is not a constant throughout the day but also has its peaks. If those peak record arrival rates actually correspond to current periods of peak system load, then the impact on the peak system MSU requirements of moving this load would be even greater than one would expect from the average record rate alone. In other words, if there is a perceived business need to process the records in a more timely manner, be sure those footing the bill are aware it may not be a "free lunch". JC Ewing On 02/19/2012 04:04 PM, Ed Gould wrote: Magen: Not sure if this is what you are looking for but There is a facility called "execution batch monitoring" (at least in JES2) You "feed" it input and it creates output (to your specifications). It was originally designed for batch compiles but it could be adapted to something like you want (?). Ed On Feb 19, 2012, at 3:25 PM, Magen Margalit wrote: Hi list. We have a daily betch job that is processing as input records which has been collected all day. volume of records is about 5 millions for 24 hours. In order to make systems more "online" we are looking for a way to run the process for each record all day long instead of a daily run, and doing so with minimum as possible application changes. One idea that came up is to convert the process to a "self developed" STC which will be triggered by a record on an MQ queue and will run as STC all the batch process programs To me it seems like a bad idea because having a "self developed" STC in production create a maintenance gap (and where there is one STC a second one will soon to follow...)... Are there other advantages / dis-advantages regarding a "self developed" STC ? Are there any "self developed" STC's at your shop? Any other ideas on how to approach this issue? Thanks in advanced. Magen -- Joel C. Ewing,Bentonville, AR jcew...@acm.org -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Batch process VS Started task
> > Hi list. > > We have a daily betch job that is processing as input records which has been collected > all day. > volume of records is about 5 millions for 24 hours. > > In order to make systems more "online" we are looking for a way to run the process > for each record all day long instead of a daily run, and doing so with minimum as > possible application changes. > > One idea that came up is to convert the process to a "self developed" STC which will be > triggered by a record on an MQ queue and will run as STC all the batch process > programs > > To me it seems like a bad idea because having a "self developed" STC in production > create a maintenance gap (and where there is one STC a second one will soon to > follow...)... > > Are there other advantages / dis-advantages regarding a "self developed" STC ? > > Are there any "self developed" STC's at your shop? > > Any other ideas on how to approach this issue? > > Thanks in advanced. > > Magen > Perhaps you could review the BATCH PIPES process in z/OS or if you have MQ, perhaps that could support what you need. Other options could be IMS solution MQ solution CICS solution DB2 Solution We currently have a need to send out orders when a DB2 table is updated. To do this we use our CA-ESP Scheduling software to monitor the DB2 table, when there is a row change, it triggers the process to action the order. If there were some more details on your problem, we might be able to come up with more specific answers? Lizette -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Batch process VS Started task
On 2/19/2012 2:25 PM, Magen Margalit wrote: Hi list. We have a daily betch job that is processing as input records which has been collected all day. volume of records is about 5 millions for 24 hours. In order to make systems more "online" we are looking for a way to run the process for each record all day long instead of a daily run, and doing so with minimum as possible application changes. One idea that came up is to convert the process to a "self developed" STC which will be triggered by a record on an MQ queue and will run as STC all the batch process programs To me it seems like a bad idea because having a "self developed" STC in production create a maintenance gap (and where there is one STC a second one will soon to follow...)... Are there other advantages / dis-advantages regarding a "self developed" STC ? Are there any "self developed" STC's at your shop? Any other ideas on how to approach this issue? Thanks in advanced. Magen Well, you could set up your free HTTP server and handle it using HTML and CGIs. You could use CICS/TS if you have it and use the web interface. Or, use a z/OS UNIX daemon with multithreading. Not sure about the 5M trans a day rate, for any of these, though. Need to do some benchmarking. -- Kind regards, -Steve Comstock The Trainer's Friend, Inc. 303-355-2752 http://www.trainersfriend.com * To get a good Return on your Investment, first make an investment! + Training your people is an excellent investment * Try our tool for calculating your Return On Investment for training dollars at http://www.trainersfriend.com/ROI/roi.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Batch process VS Started task
Magen: Not sure if this is what you are looking for but There is a facility called "execution batch monitoring" (at least in JES2) You "feed" it input and it creates output (to your specifications). It was originally designed for batch compiles but it could be adapted to something like you want (?). Ed On Feb 19, 2012, at 3:25 PM, Magen Margalit wrote: Hi list. We have a daily betch job that is processing as input records which has been collected all day. volume of records is about 5 millions for 24 hours. In order to make systems more "online" we are looking for a way to run the process for each record all day long instead of a daily run, and doing so with minimum as possible application changes. One idea that came up is to convert the process to a "self developed" STC which will be triggered by a record on an MQ queue and will run as STC all the batch process programs To me it seems like a bad idea because having a "self developed" STC in production create a maintenance gap (and where there is one STC a second one will soon to follow...)... Are there other advantages / dis-advantages regarding a "self developed" STC ? Are there any "self developed" STC's at your shop? Any other ideas on how to approach this issue? Thanks in advanced. Magen -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Batch process VS Started task
Hi list. We have a daily betch job that is processing as input records which has been collected all day. volume of records is about 5 millions for 24 hours. In order to make systems more "online" we are looking for a way to run the process for each record all day long instead of a daily run, and doing so with minimum as possible application changes. One idea that came up is to convert the process to a "self developed" STC which will be triggered by a record on an MQ queue and will run as STC all the batch process programs To me it seems like a bad idea because having a "self developed" STC in production create a maintenance gap (and where there is one STC a second one will soon to follow...)... Are there other advantages / dis-advantages regarding a "self developed" STC ? Are there any "self developed" STC's at your shop? Any other ideas on how to approach this issue? Thanks in advanced. Magen -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
