ZapThink </>
Batch Processing with SOA
Document ID: ZAPFLASH-2009128 | Document Type: ZapFlash
/By: Ronald Schmelzer/
Posted: Jan. 28, 2009
Although typically thought of as an artifact of legacy computing, batch
processes remain vital to today?s real-time enterprises. Behind the real
time systems that power the real time enterprise, such as customer order
fulfillment, account management, supply chain scheduling and
optimization, or financial trading systems, are regularly-updated back
office business systems. Today, batch processes remain essential for one
key reason: it is simply not efficient to regenerate a complete forecast
or business plan every time the business processes a single event such
as an incoming customer order. Real time enterprises do require systems
that can support dynamic processes; however, it is best to reserve that
capacity for aspects of data or processes for the most volatile
high-velocity markets.
Nonetheless, while the need for batch processes hasn?t changed, the
nature of batch processing today has certainly evolved. For instance,
while scheduled batch processes remain relevant for time-related
processes such as end-of-period reporting, real time enterprises may
require more flexibility for adapting to temporary or permanent market
fluctuations, or to planned or unplanned changes in underlying IT
infrastructure. Companies are faced with new requirements for managing
compliance with increasingly stringent regulatory mandates. This may
dictate the need for policy-driven workflows with the capability for
dynamically triggering batch processes when specific scenarios arise.
For instance, a Sarbanes Oxley violation might trigger a rollback and
new round of updating core financial systems outside of the normal
schedule.
Service-Oriented Architecture (SOA) presents enterprises with the
opportunity to expose information and processes as self-contained
Services that can communicate and interoperate with each other in a
standard, loosely coupled fashion. Although the common impression is
that Services expose business processes, data processes, or application
functionality, they are also well-suited for exposing the very processes
that drive batch-oriented workloads. SOA enables the business to build
flexible compositions of Services that implement either business or IT
processes in a loosely coupled manner, which has important ramifications
for IT service delivery, and the batch processes that are part of it.
*The Evolution of Batch Processing*
Over the years, batch technology has evolved from script-based
automation to rules or policy-driven workload automation. IT has long
sought to automate the running of batch jobs, initially with tools that
automatically chained them together in a schedule. As batch windows
shrunk, IT organizations embraced job management technologies enabling
them to optimize their use of limited resources over shorter timeframes.
Workload automation is the culmination of innovations in job automation,
adapting it to the needs of the real time enterprise with policy-driven,
configurable workflows. Compared to traditional job automation methods,
configurable workflows are far more flexible and maintainable.
Yet, the objective has changed. Instead of optimizing time windows or
resource consumption in the data center, today the mission of batch
processing has broadened its support of the business. In the real time
enterprise, today?s batch operation is morphing from a static, often
standalone process to a dynamic component of an application or composite
business process. The goals could include accelerating responsiveness,
improving corporate transparency, and enabling IT to adopt a more
business-centric and compliance-driven approach to managing its own
operations.
The key to achieving these goals is consistency and agility. With
traditional, hard-coded or manual approaches, ensuring uniformity of
batch processes was often a hit or miss proposition. Batch, and its
successor workload automation, has evolved over time, with the latest
innovations adding policy or rules-driven workflows that transform batch
jobs into repeatable workflows.
Ultimately, SOA provides the architectural underpinnings that bring this
vision to fruition: By invoking batch processes as a Service, they
become ingrained components of the composable business processes that
power the real time enterprise. For instance, via Service enablement,
workload automation can become extensions of applications ranging from
supply chain management to IT service desk. Long a stepchild of data
center operations, workload automation transforms into a first class
citizen of the real time enterprise.
Embedding workload automation as a Service could also encourage cultural
changes that bring together IT operations, software development, and the
business. Typically, batch jobs are either scheduled or ?thrown over the
wall? from the business to operations. When batch workloads or workflows
exposed as Services become extensions of the application, developers who
design or configure the application can now factor in the demand on IT
infrastructure by specifying the logic or rules under which enterprise
systems can request batch Services. Workload automation is the key to
making the batch window a first class citizen of the real time
enterprise, which dictates expanding the role of workflow automation
from an IT to a business-driven tool.
Consequently, the goal of workload automation changes as well. Instead
of IT optimizing its own data center update workflows, applications or
business processes are now in the driver?s seat, triggering workloads
when specific business events or scenarios occur. This requires a
business process-focused workflow engine for job automation, and a
business rules engine for enabling policy-based workflow management. In
a business-driven workflow, requests from applications, database stored
procedures, or business processes trigger workloads (which may consist
of one or more batch jobs) as a result of explicit business rules or
policies.
For instance, when order activity for a particular product SKU occurs,
the following rules could be used to automatically trigger new workloads:
* If a handful of unexpected orders arrive, alter shipment
schedules, but do not change supply chain plans.
* If there is an unexpected run on the product, generate a new
demand forecast.
* If the run causes actual or potential product shortages,
regenerate inventory, procurement, distribution, and demand plans.
Under these scenarios, workloads no longer strictly focus on optimizing
database updates in the data center. Instead, they have become intrinsic
components of demand-driven supply chain business processes. Batch
processes would be exposed as Services that you might want to access in
real time. The idea of a batch is that the process is pre-defined (so, a
batch is not a flexible process), but it would be exposed as a Service
that might be consumed in a flexible process. In traditional workloads,
a batch might be scheduled as part of a daily activity, but a
Service-oriented batch could be executed on demand. From a
Service-oriented perspective, the Service contract and policy would have
to specify how many times a batch can be executed in a single day, how
long the batch takes to operate, and any other Service or data
dependencies.
*The Contribution of SOA to Workload Automation*
SOA is all about exposing information and processes as self-contained
Services that can communicate and interoperate with each other in a
standard, loosely coupled manner, enabling the business to build
flexible compositions of Services that implement business processes.
Ultimately, SOA enables IT to more effectively align with the business,
because it changes the way IT delivers the solutions. The self-contained
nature of Services and the standard connectivity empowers IT to rapidly
compose solutions by reusing existing Services, resulting in faster
time-to benefit compared to traditional ways of developing, modifying,
and integrating conventional, monolithic software applications.
Although typically associated with functionality that is exposed from
software applications and/or composed business processes, SOA can also
expose IT infrastructure processes such as workload automation as
Services. Using SOA, business events trigger specific job types. In
effect, the job becomes part of a composite, Service-Oriented Business
Application. The SOA infrastructure coupled with the standards used in
implementing SOA facilitate a true loose coupling where business logic,
business, rules, and job specifications can each change without
impacting the other components. For instance, the workload automation
system could invoke a batch update Service as a result of Business
Intelligence system triggering a scheduled Extract-Transform-Load (ETL)
operation to a data warehouse. If the user changes their ETL system
itself or some of the transformation routines, that change is kept
internal to the ETL Service and should not impact the Workload Service.
*The ZapThink Take*
During the height of the dot.com bubble, the conventional wisdom was
that the Internet was supposed to change everything in business.
Although the Internet helped enterprises forge new connections to
business partners and customers alike, it did not change the
fundamentals of conducting business or managing IT systems. Similarly,
today?s ascendance of the real time enterprise has not eliminated the
need for effective batch processing. By leveraging Service-Oriented
Architecture (SOA), enterprises can evolve batch workloads from
standalone data center operations to components that are intrinsic to
composite, Service-Oriented Business Applications. Where workload
automation and batch processing were once arcane tasks relegated deep in
the organization, they become critical Services delivered as part of the
Enterprise Architecture.
