SOAP and XML Schema support [Re: Project from hell?

[Aleksander Slominski]

Jeff,
did you try to use schema compiler (scomp) from XmlBeans?
i think it should be possible to create XmlBeans serializer and 
deserializer. it would add some memory overhead (XmlBeans uses its own 
XML infoset storage) but should work well - i have it working quite well 
in XSOAP4/XSUL.

thanks,
alek
Jeff wrote:
Hi Davanum,
Well, I have a small confession: the schema is so complex that the OGC have
yet to finalize and release it! Here are some pertinent points:
   - SCS (along with dependencies like SensorML, SWE, etc.) contains errors
that I had to fix (the XML Schema I used was fine after my fixes). It is not
unusual to find that scientists get a little carried away and are not
familiar with all the details of XML Schema. To be fair, not many people
are.
   - The OGC have yet to finalize the content (> 98% finished) and, in fact
the work I did was used to give them feedback on deficiencies, e.g. an SCS
client is expected to identify sensors associated with data collection
stations but in our case we were dealing with Water Quality Index values
that were _calculated_ from collected data (stored in a central database but
nonetheless identified with the water monitoring stations in the field) so
we have what I called a 'virtual sensor'. Following and OGC meeting last
week, it was resolved to make positive strides forward in the Fall...these
initiative take time!
   - The SCS I developed is, as far as we know, the only SCS in Canada and
possibly even then only _production_ SCS in the world. At the moment it's
used internally by the Environment Canada, a department of the Government of
Canada. Other software is being written by other parties to support SCS.
   - SCS, SensorML, et. al. are far-reaching in their scope. We used it for
terrestrial water monitoring stations but NASA will be using it for
satellite-borne sensors. Others will be using it for ship-borne sensors.
   - Interestingly, SCS clients can ask for different output formats: one
is XML (SWEObservation), another is GeoTIFF (multi-image files, one image
per station, lat/long, geographic reference system identified, etc.).
I have contacted the OGC and asked permission to supply you with the
fully-functioning XML Schema that I have. I very much expect that they will
grant permission for this (provided you don't publish it beyond use in
testing, I suppose) but I cannot give it to you if they decline.
Just to let you know what you are up against, I have attached an image that
illustrates the SCS schema. As I am sure you know, an XML Schema is
comprised of one or more XML Schema documents (54 in the case of SCS). When
an XML Schema document is imported into XchainJ the entire hierarchy of
related XML Schema documents is recursively traversed resulting in one
integrated XML Schema which is then depicted using a terse, colour-coded
(e.g. blue for elements, red for types) syntax. (A Java data model is
automatically generated and optimized; typically simplifying by an order of
magnitude.)
Warmest regards,
Jeff
Cogent Logic Corporation
Toronto, Canada

----- Original Message ----- 
From: "Davanum Srinivas" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Cc: <[EMAIL PROTECTED]>
Sent: Friday, May 14, 2004 2:42 PM
Subject: Re: Project from hell?

 

Jeff,
#1: I meant just a URL to the schema itself. Nothing more :)
#2: At apache, we have JaxMe and XMLBeans projects. This complicated
schema may be given to them as a test case. hence the question.
Best of luck with XchainJ.
thanks,
dims
On Fri, 14 May 2004 13:42:43 -0400, Jeff <[EMAIL PROTECTED]> wrote:
   

#1:
You must be joking! There are more than 2000 DIFFERENT elements and
     

complex
 

types. The problems tend to lie within the generated code and are not
obvious until you try to use that code...then you find that it doesn't
     

work.
 

I wasted way too much time trying make changes to the Axis-generated
     

code
 

before giving up. I concluded that Axis was an order of magnitude away
     

from
 

where I needed it to be in terms of complex payloads (though that
     

perception
 

might have been biased by the enormity of the SCS XML Schema). So I use
     

Axis
 

for connectivity (it's great at this, of course) and insert XML
     

documents
 

(that I generate) as document-literal content.
Well, okay, I'll check it out over the weekend and file a bug report :-)
#2:
No. I understand that nowadays some folks use Castor in cases like this.
Three years ago my customers needed something like a JAXB that supports
     

the
 

whole of XML Schema and connected to databases. There wasn't anything
     

then
 

so I wrote XchainJ. This product is now in version 2.3 and runs as a
     

fully
 

integrated Eclipse Plugin. This isn't a commercial plug though because,
oddly enough, I find life is easier if I don't sell it! XchainJ is great
     

for
 

really complex XML but, unfortunately, it has been my experience that
     

people
 

who have a requirement for this are not the sort of people who have the
technical expertise to use it! They tend to be scientists not Java
programmers. I could ramble on at length about user perceptions, etc.
     

but I
 

won't. When I work face-to-face with customers they really appreciate
     

the
 

product and either use it themselves or pay me to use it. Typically, I
     

can
 

turn around a project that would take a week using Castor, JAXB, etc. in
     

two
 

or three hours with XchainJ (it does XML/Java/DBMS interoperability).
     

The
 

whole process of dealing with potential customers in other countries and
over the Internet is more trouble than it's worth.
Interestingly, I presented XchainJ to the technical director of a
     

company
 

that sells Java software (on the basis that they could do the marketing,
etc.). The guy thought the product was great but found he was unable to
explain to his marketing folk what it did in terms that they could
understand! If they are not experts, people seem to get fogged beyond a
certain level of complexity.
It's a crazy situation: we have XML Schema that scientists are running
     

with
 

and producing very complex structures BUT they don't have the expertise
     

to
 

implement solutions. Then there's the computer industry that, while
populated with developers who can work on complex projects and after
     

great
 

effort can produce solutions, has mainstream tool vendors that are
completely out of touch with anything other than trivial XML! Some
commercial products have been written by programmers who were under the
impression that there will only ever be one XML Schema document that
     

targets
 

a given namespace. They generate error like "I've encountered this
     

namespace
 

before, what are you giving it to me again for!". Still other won't go
beyond a maximum of just one XML Schema document referenced from WSDL.
     

GML
 

comprises 27 XML Schema documents that target the GML namespace (plus
     

others
 

for xlink, etc.) (and GML is a basis schema that users are _intended_ to
incorporate as a component of other schemas).
An NGO approached me a year ago with a project that they had only a
     

month to
 

complete. It uses the CSDGM DTD (a.k.a. FGDC). They went to a big
     

software
 

development company before they approached me and were told (i)
     

something
 

that complex couldn't be done and (ii) if it could be done there's no
     

way it
 

could be done within a month. Using XchainJ 1.1, I completed the entire
project in one day. Had I had XchainJ 2.3 then, it would have taken half
     

a
 

day.
In a similar vein, my customers currently need XML Schema support in
rich-clients. The sort of support that doesn't exist today. They are
     

going
 

to get it in XchainJ 3.x. As I said earlier, there's a disconnect
     

between
 

the complexity supported by the computer software industry and the
complexity required by scientists. While the Eclipse folks are working
     

on
 

SWT-designer support, I'm working on 'XML Schema / SWT rich-client with
connections to controlled content web services plus authentication,
authorization, XML document management, etc.' support in a generic tool.
     

An
 

order of magnitude disparity.
Now, if only I was a marketeer instead of  a programmer... :-)
Warmest regards,
Jeff
Cogent Logic Corporation
Toronto, Canada
----- Original Message -----
From: "Davanum Srinivas" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Friday, May 14, 2004 12:18 PM
Subject: Re: Project from hell?
     

A few questions:
#1: Can you please open a bug report with a pointer to the schema that
       

fails?
     

#2: Did you try using any JAXB implementation against the schema?
thanks,
dims
On Fri, 14 May 2004 12:03:14 -0400, Jeff <[EMAIL PROTECTED]> wrote:
       

Hi!
On a similar note, there's a disconnect between the capabilities of
         

tools
     

created by the software industry and the requirements of the
         

scientific
 

community.
I have just completed a particular type of Open GIS Consortium (OGC)
         

web
 

service called a Sensor Collection Service. The XML Schema
         

referenced
 

from
     

the WSDL file comprises 54 XML Schema documents spanning 15
         

namespaces.
 

Not
     

only did the Axis bean code baulk at this but, when I had completed
         

the
 

project, clients found that the .NET tools couldn't handle anything
         

like
 

the
     

complexity of the SCS XML Schema. Consequently, I supplied 'client
software'.
The originators of SOAP are conning the software world and no one
         

seems
 

to
     

mind!
If it's legitimate to distribute platform-independent data (XML) it
         

must
 

be
     

legitimate to distribute the program logic that uses that data. If
         

only
 

we
     

had a platform-independent way to deliver program logic!
Forcing web service clients, as a matter of fiat, to write their own
         

program
     

logic is the antithesis of OOP: interfaces, inheritance,
         

polymorphism
 

all
     

exits to promote reuse. Reuse is the Holy Grail of software
         

development.
 

It could be argued that each client has their own needs and so it's
         

not
 

possible to write generic client-side code. Such an argument is
         

false.
 

The
     

fact that XML Schemas are used to formalise the data transmitted
         

within
 

SOAP
     

envelopes means that each web service is necessarily
         

application-specific
     

and, as such, is tractable to low-level client code. Such code
         

exposes
 

data
     

(in the form of XML, if appropriate) that can then be used in
         

whatever
 

way
     

the ultimate consumer-code requires.
I recently wrote a web service for the Government of Canada that
         

provided
     

document-literal content in the form of Web Ontology Language (OWL).
Everyone was pleased with the outcome and loved the OWL
         

implementation
 

BUT
     

the first thing they did was to nominate someone to write a generic
         

client
     

that dealt with the XML and provided the desired content through a
         

Java
 

component that everyone could use/reuse. Hey, that's an idea...I
         

wonder
 

if
     

we could supply Java client-side code with our web services. That
         

way,
 

the
     

.NET folks and all other non-Java folks could continue to do what
         

they
 

do
     

and the sane software developers can get back to the preferred
         

paradigm
 

of
     

using portable code.
XML and Java go together. Sun and all other interested parties seem
         

to
 

be
     

blind to the fact that making portable client-side code an
         

integrated
 

web
     

service deliverable would make those services far more viable. Not
         

everyone
     

wants to get into WSDL, etc. when they could simply use a bean! SOAP
         

and
 

web
     

services are infrastructure. Folks who use my web services want
         

turnkey
 

solutions. For them it's about access to scientific data. They want
         

to
 

operate at a higher level of abstraction than SOAP!
Warmest regards,
Jeff
Cogent Logic Corporation
Toronto, Canada
----- Original Message -----
From: "Galbreath, Mark A" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Friday, May 14, 2004 11:22 AM
Subject: RE: Project from hell?
         

EXACTOMUDO!  :-(
-----Original Message-----
From: Sherman, Dennis (END-CHI)
           

[mailto:[EMAIL PROTECTED]
 

Sent: Friday, May 14, 2004 9:12 AM
Your task sounds to me suspiciously like someone at an executive
           

level
 

having heard about web services, and thinking they've found the
           

silver
 

bullet to all their problems.
           

         

     

------------------------------------------------------------------------

--
The best way to predict the future is to invent it - Alan Kay