<<Introduction
As with politics and religion, information technology has its armies
that fight for great truths in the name of half-truths. And one great
half truth is that Agile and Service-Oriented Architecture (SOA) are
incompatible.
On the face of it, that seems true. Service-oriented architecture must
be top-down in conception and execution for it to be effective. Agile is
a bottom-up systems development methodology that emerges from
self-organizing collectives. SOA and Agile have both demonstrated their
value and have firmly established themselves in the marketplace. And yet
the experience of more than a decade at many hundreds of firms has
exposed flaws in how SOA and Agile are practiced in the real world where
vast resources are at stake.
For purposes of this article, "top-down" means a rule-based management
structure of command and control. "Bottom-up" means a consensus-based
development structure. We are talking about tendencies rather than
absolutes. I acknowledge that Agile has planning and a waterfall SDLC is
somewhat team-driven.
The purpose of this article is to reconcile these two paradigms and to
suggest that the weaknesses that SOA and Agile have present
opportunities for integrating SOA and Agile into a complementary
partnership. SOAG-- Agile that informs SOA-- can have a decisive,
positive, and profitable organizational impact. My goal is not so much
to communicate specific technical solutions to SOA-implementation but to
put the question of SOA management into the broader context of
organizational transformation.
Systems development failure and success rarely has its roots in the
choice of language, platforms, methodology or the number and quality of
its staff. Rather, they are the result of how managers deploy those
resources. In the pages that follow, I will explore how managers can
successfully orchestrate SOA and Agile.
Outline
Here is what I hope to show:1.SOA implementation must be
top-down.2.Agile SDLC is bottom-up.3.To implement an effective SOA, we
can combine Agile in a top-down SOA waterfall SDLC.How to marry SOA to
Agile is the subject of the last part of this article. In this section,
I discuss the following:1.SOAG analysis and problem-solving.2.SOAG
SDLC3.SOAG and the human factorIn a topic as broad as this, more can be
said and much has been left out. Some of this may be the basis for
future articles as well as discussion by others in the SOA community.
Why SOA Is Top-Down
SOA is not a new name for an API or Web services or a marketing label.
And nor is it magic, easy, or cheap. It is, however, a potential
business facilitator and the wave of the future. SOA is based on open
standards, supports vendor diversity, fosters interoperability, promotes
discovery, and supports incremental implementation. The essence of SOA
is to decompose business processes into discrete functional units or
services and group existing or new business functions into business
services. Consumers can then discover and invoke services that the
provider publishes.
Top-down management is plan-based, hierarchical, and totalitarian.
Bottom-up management is humane, consensual, and egalitarian. Effective
SOA implementationin fact any kind of effective business
transformationmust be top-down. SOA management is an imposed vision
because we can design and implement SOA only by considering a
company's business processes in their totality from the highest
level along with the tools and governance structure that is needed as
bedrock for that architecture. And that can only be done with
championship from the highest levels of the organization. It also means
looking at the organization not as a collection of technologies but as a
collection of business processes. [REF-3]
"The bottom-up strategy is really not a strategy at all", Thomas
Erl writes. "Nor is it a valid approach in achieving contemporary
SOA. This is a realization that will hit many organizations as they
begin to take service-orientation, as an architectural model, more
seriously. Although the bottom-up design allows for the efficient
creation of Web services as required by applications, implementing a
proper SOA at a later point can result in a great deal of retro-fitting
or even the introduction of new standardized service layers positioned
over the top of the non-standardized services produced by this
approach." [REF-4]
This is not to say that SOA cannot be rolled out incrementally. Of
course it should. Nor should a top-down management structure be blind to
the experiences, thoughts, and feelings of all of those who may be
impacted by those changes. SOA will fail if it is not managed as a
top-down effort. SOA will also fail if it is only managed as a top-down
effort. But, as a question of architecture and strategy, a populist
approach is like building a new house around its appliances. A bottom-up
approach introduces needless complexities into already complex systems
that are at cross purposes with the company's umbrella strategy.
A big bang SOA deployment is not sensible. The most prudent approach is
to strive to understand the full business context and then undertake a
strategy of deployment from opportunistic projects to enterprise level
business solutions, taking into consideration that SOA enterprise is a
journey, filled with ambiguity, missteps, back steps, and discovery. It
is choreographydetermining how processes are linked together across
the enterprise. And it is orchestrationdetermining how to mesh
those processes underlying business applications so that they can be
exposed and linked to processes into other applications.
We can get insight into why SOA execution must be top-down by
considering ineffective and effective organizations. SOA attempts to
solve the corporation's strategic goals by replacing an ineffective
organization with effective organization.
Ineffective Organization: The Silo and the Cog
The silo and the cog are models of dysfunctional organizations(Wikimedia
Commons)
In the case of the silo, functions, such as marketing, finance, and
operations, evolve independently of each other into departments or
divisions, each sometimes containing duplicated IT resources and a
knitting ball of interrelationships. Silos can occur at the team or even
the individual level, where information is husbanded to build personal
power rather than to promote the corporate good. A more accurate model
might fill these silos with hives having barely sentient winterized
bees. Not only is there paralysis of communication cross-functionally
but also with the department, sometimes consisting of a myriad of echo
chambers.
The cog model views employees as static resources. The half truth that
employees need to smoothly mesh with each other is extended into a
liethat Mike and Jane are incapable of change and are thusly
expendable. Thus, an employee is branded as a "Java programmer",
for example, rather than as a problem solver who knows Java. So, when
the department's technology changes to .NET, management will replace
that cog with a Microsoft-branded cog. Of course, it is only a matter of
time when those same managers are replaced by yet other cogs.
Effective Organization: The Factory and The Guild
A more promising model is a hybrid of the factory's assembly line
and the guild from the Middle Ages.
Under the factory system, tasks are decomposed and executed
methodically, predictably, quickly, and cheaply. Under this model, time
to market is good, while innovative flexibility is poor. The guild
system gave us architectural gems of Western civilization, such as
Canterbury, Notre Dame and other cathedrals with their stonework and
stained glass windows. Under this system, beauty and craftsmanship was
paramount with scant regard to timeliness or cost-effectiveness.
Our challenge is to integrate the best of both models to give us the
time to market of the factory system combined with the innovation and
quality from the guild system.
Effective Organization: The Pyramid
The pyramid, a three-dimensional triangle, is the most-enduring man-made
structure and an appropriate model for a strong, stable business
enterprise.The pyramid is replicated as the most successful government,
religious, and corporate structures, and we only need to reflect on
organizations such as the governments of the United States or the
People's Republic of China, the Catholic Church or the Church of
Latter Day Saints, and IBM or Toyota to recognize this. However, such
structures also carry within them the seeds of self-sabotage. Consider
the Toyota Production System. Toyota's kaizen doctrine of continual
process put the authority to stop a production line in the hands of line
personnel. But such customer-driven drive for quality hit a speed bump
when cars with defective accelerators resulted in deaths and lawsuits.
This may have been a case where the process was so good that it seemed
inconceivable to Toyota's salary-men that a car that rolled off the
assembly line was defective.
As a bureaucratic organism, the corporate pyramid is constantly
mutatingblocks moving up, down, or out, and the pyramid expanding
or contracting. The strongest organizations have relatively few layers,
and such paucity of layers allows critical information to pass up and
down the chain of command efficiently.
One approach to implementing SOA is to conceptualize it as a triad of
governance, common services, and platforms and tooling.
SOA platforms and tooling, consisting of SOA-enabling middleware,
provides the backbone of a future distributed and interoperable
computing environment.
Commons services require an iterative governance SDLC.
Why Agile is Bottom-up
To understand the bottom-up nature of Agile, we must reflect on the
paradigm that it displaced-- the waterfall SDLC.
The Waterfall Model
The waterfall model grew out of the monolithic corporations that first
funded equally monolithic data centers in the 1950s through the 1980s.
Such data centers often had hundreds of programmers punching out cards
for projects that were managed with PERT or Gantt charts. Under this
model, developers could not go in their project from one step to another
until there was closure, usually indicated by signoffs. Upon getting
those signoffs, the team surfed to the next level in the waterfall.
Agile acolytes mock the waterfall model for its BDUFbig design up
frontthat makes for inflexibility to embrace downstream
requirements. But the waterfall model is effective. It built code that
put men on the moon, synthesized life-saving drugs, and launched
aircraft carriers. This model's singular advantage is that it
encouraged rigor and minimized re-work by enforcing a forward
development path. "A requirements defect that is left undetected until
construction or maintenance will cost (up to) 200 times as much to fix
the same bug later on in the process." [REF-6] The promise of the
waterfall method is that carefully analyzing all requirements using BDUF
would squeeze out vagueness and downstream errors while enforcing a
structure of accountability. This promise was not always realized in its
rigidity to meet dynamic customer requirements and to implement large
projects on time and within budget.
The Promise of Agile
The dominance of big iron receded with the advent of the web, object
orientation coding, distributed objects, the ubiquity of computers, the
plunge in the price of storage and chip prices, the Y2K hysteria, the
dot com boom and bust, and offshoring. These pressures forced developer
to re-think lifecycle development theory. One reaction to heavyweight
SDLC was Agile. Its basic principles were encapsulated in 2001 in the
Agile Manifesto [REF-7] by representatives of new methodologies such as
Extreme Programming, Scrum, and Pragmatic Programming.
The Practice of Agile
Some skepticism as to the merits of Agile is needed. Instead of BDUF, we
sometimes have BPOCbig person on campus relentless egomaniacs
that hijack and sometimes destroy projects and reputations. Agile can
become a cover for cowboy codinga flight from team interactions and
customer corroboration as Agile advertises. It can become an excuse not
to have any process, documentation, e-mails, or audit trail. Tribal
knowledge comes out of happenstance of ability and availability. What
some see as flexibility and humanity is often chaos, shoddiness, and
indolence with ten year-old systems with no documentation and
implementations without regard to impacts across the enterprise.
Agile can work. Much rests on the good will, energy, and collaborative
temperament of the team. But it often breaks down under the weight of
the cussedness of human nature. The Agile premise of dialectical
truththat the best truth comes out of the litigation of
clearly-articulated, strongly-held contrary points of viewis valid,
so long as there is honest, open debate where persuasion is not tied to
titles or personalities but to evidence.
It is not sufficient to say that when Agile fails, it is because its
disciples do not understand Agile. They often understand it all too
well, but cannot to make it succeed, due not to defects in its theory
but to defects in personalities that struggle to realize its theory. The
denial that Agile has within it the potential for bad practices falls
into the No True Scotsman fallacy, an argument that takes this form:
Argument:"No Scotsman puts sugar on his porridge."Reply:"But Angus likes
sugar with his porridge."Rebuttal:"Ah yes, but no true Scotsman puts
sugar on his porridge."Consider a parallel argument:Argument:"No Agile
implementer is a cowboy coder."Reply:"But Mark is a cowboy
coder."Rebuttal:"Ah yes, but no true Agile implementer is a cowboy
coder."The problem with this argument is that it derives an unproved
predicate ("puts sugar on porridge", "cowboy coder") from the
subject ("Scotsman", "Agile implementer"). Now, sometimes the
argument is valid as when the predicate derives from the subject, as in
"no true vegetarians eat beef." But there is nothing inherent in
Agile that will prevent someone from being a cowboy coder, and the
cowboy who codes under the banner of Agile will be the very flower of
Agile. Or so he or she will believe.
I believe that Agile and Agile-like methods such as Scrum are integral
to the success of a SOA implementation. But to ensure that Agile does
succeed, it is important to come to grips with how and why Agile can
fail. And the reason Agile can fail is because it is at the bottom a
personality-driven process that demands exceptional individual
discipline and a zeal for collaboration.
Conclusion
The second part to this article will be published in the May 2010 issue
of The SOA Magazine. >>
You can find this article at: http://www.soamag.com/I38/0410-1.php
<http://www.soamag.com/I38/0410-1.php>
Gervas
