Couldn't agree more that batch processing can be better handled as a service, its one of the "odd" things that vendors seemed to exclude because it wasn't in their product set (or worse claim an old ETL as SOA just to get some more license sales).
Steve 2009/1/29 Gervas Douglas <[email protected]>: > > > Batch Processing with SOA > > Document ID: ZAPFLASH-2009128 | Document Type: ZapFlash > By: Ronald Schmelzer > Posted: Jan. 28, 2009 > > Although typically thought of as an artifact of legacy computing, batch > processes remain vital to today's real-time enterprises. Behind the real > time systems that power the real time enterprise, such as customer order > fulfillment, account management, supply chain scheduling and optimization, > or financial trading systems, are regularly-updated back office business > systems. Today, batch processes remain essential for one key reason: it is > simply not efficient to regenerate a complete forecast or business plan > every time the business processes a single event such as an incoming > customer order. Real time enterprises do require systems that can support > dynamic processes; however, it is best to reserve that capacity for aspects > of data or processes for the most volatile high-velocity markets. > > Nonetheless, while the need for batch processes hasn't changed, the nature > of batch processing today has certainly evolved. For instance, while > scheduled batch processes remain relevant for time-related processes such as > end-of-period reporting, real time enterprises may require more flexibility > for adapting to temporary or permanent market fluctuations, or to planned or > unplanned changes in underlying IT infrastructure. Companies are faced with > new requirements for managing compliance with increasingly stringent > regulatory mandates. This may dictate the need for policy-driven workflows > with the capability for dynamically triggering batch processes when specific > scenarios arise. For instance, a Sarbanes Oxley violation might trigger a > rollback and new round of updating core financial systems outside of the > normal schedule. > > Service-Oriented Architecture (SOA) presents enterprises with the > opportunity to expose information and processes as self-contained Services > that can communicate and interoperate with each other in a standard, loosely > coupled fashion. Although the common impression is that Services expose > business processes, data processes, or application functionality, they are > also well-suited for exposing the very processes that drive batch-oriented > workloads. SOA enables the business to build flexible compositions of > Services that implement either business or IT processes in a loosely coupled > manner, which has important ramifications for IT service delivery, and the > batch processes that are part of it. > > The Evolution of Batch Processing > Over the years, batch technology has evolved from script-based automation to > rules or policy-driven workload automation. IT has long sought to automate > the running of batch jobs, initially with tools that automatically chained > them together in a schedule. As batch windows shrunk, IT organizations > embraced job management technologies enabling them to optimize their use of > limited resources over shorter timeframes. Workload automation is the > culmination of innovations in job automation, adapting it to the needs of > the real time enterprise with policy-driven, configurable workflows. > Compared to traditional job automation methods, configurable workflows are > far more flexible and maintainable. > > Yet, the objective has changed. Instead of optimizing time windows or > resource consumption in the data center, today the mission of batch > processing has broadened its support of the business. In the real time > enterprise, today's batch operation is morphing from a static, often > standalone process to a dynamic component of an application or composite > business process. The goals could include accelerating responsiveness, > improving corporate transparency, and enabling IT to adopt a more > business-centric and compliance-driven approach to managing its own > operations. > > The key to achieving these goals is consistency and agility. With > traditional, hard-coded or manual approaches, ensuring uniformity of batch > processes was often a hit or miss proposition. Batch, and its successor > workload automation, has evolved over time, with the latest innovations > adding policy or rules-driven workflows that transform batch jobs into > repeatable workflows. > > Ultimately, SOA provides the architectural underpinnings that bring this > vision to fruition: By invoking batch processes as a Service, they become > ingrained components of the composable business processes that power the > real time enterprise. For instance, via Service enablement, workload > automation can become extensions of applications ranging from supply chain > management to IT service desk. Long a stepchild of data center operations, > workload automation transforms into a first class citizen of the real time > enterprise. > > Embedding workload automation as a Service could also encourage cultural > changes that bring together IT operations, software development, and the > business. Typically, batch jobs are either scheduled or "thrown over the > wall" from the business to operations. When batch workloads or workflows > exposed as Services become extensions of the application, developers who > design or configure the application can now factor in the demand on IT > infrastructure by specifying the logic or rules under which enterprise > systems can request batch Services. Workload automation is the key to making > the batch window a first class citizen of the real time enterprise, which > dictates expanding the role of workflow automation from an IT to a > business-driven tool. > > Consequently, the goal of workload automation changes as well. Instead of IT > optimizing its own data center update workflows, applications or business > processes are now in the driver's seat, triggering workloads when specific > business events or scenarios occur. This requires a business process-focused > workflow engine for job automation, and a business rules engine for enabling > policy-based workflow management. In a business-driven workflow, requests > from applications, database stored procedures, or business processes trigger > workloads (which may consist of one or more batch jobs) as a result of > explicit business rules or policies. > > For instance, when order activity for a particular product SKU occurs, the > following rules could be used to automatically trigger new workloads: > > If a handful of unexpected orders arrive, alter shipment schedules, but do > not change supply chain plans. > If there is an unexpected run on the product, generate a new demand > forecast. > If the run causes actual or potential product shortages, regenerate > inventory, procurement, distribution, and demand plans. > > Under these scenarios, workloads no longer strictly focus on optimizing > database updates in the data center. Instead, they have become intrinsic > components of demand-driven supply chain business processes. Batch processes > would be exposed as Services that you might want to access in real time. The > idea of a batch is that the process is pre-defined (so, a batch is not a > flexible process), but it would be exposed as a Service that might be > consumed in a flexible process. In traditional workloads, a batch might be > scheduled as part of a daily activity, but a Service-oriented batch could be > executed on demand. From a Service-oriented perspective, the Service > contract and policy would have to specify how many times a batch can be > executed in a single day, how long the batch takes to operate, and any other > Service or data dependencies. > > The Contribution of SOA to Workload Automation > SOA is all about exposing information and processes as self-contained > Services that can communicate and interoperate with each other in a > standard, loosely coupled manner, enabling the business to build flexible > compositions of Services that implement business processes. Ultimately, SOA > enables IT to more effectively align with the business, because it changes > the way IT delivers the solutions. The self-contained nature of Services and > the standard connectivity empowers IT to rapidly compose solutions by > reusing existing Services, resulting in faster time-to benefit compared to > traditional ways of developing, modifying, and integrating conventional, > monolithic software applications. > > Although typically associated with functionality that is exposed from > software applications and/or composed business processes, SOA can also > expose IT infrastructure processes such as workload automation as Services. > Using SOA, business events trigger specific job types. In effect, the job > becomes part of a composite, Service-Oriented Business Application. The SOA > infrastructure coupled with the standards used in implementing SOA > facilitate a true loose coupling where business logic, business, rules, and > job specifications can each change without impacting the other components. > For instance, the workload automation system could invoke a batch update > Service as a result of Business Intelligence system triggering a scheduled > Extract-Transform-Load (ETL) operation to a data warehouse. If the user > changes their ETL system itself or some of the transformation routines, that > change is kept internal to the ETL Service and should not impact the > Workload Service. > > The ZapThink Take > During the height of the dot.com bubble, the conventional wisdom was that > the Internet was supposed to change everything in business. Although the > Internet helped enterprises forge new connections to business partners and > customers alike, it did not change the fundamentals of conducting business > or managing IT systems. Similarly, today's ascendance of the real time > enterprise has not eliminated the need for effective batch processing. By > leveraging Service-Oriented Architecture (SOA), enterprises can evolve batch > workloads from standalone data center operations to components that are > intrinsic to composite, Service-Oriented Business Applications. Where > workload automation and batch processing were once arcane tasks relegated > deep in the organization, they become critical Services delivered as part of > the Enterprise Architecture. > >
