EDI-L,
Several people sited a white paper from Microsoft which covers (among other
things) Biztalk's limitations when dealing with "traditional EDI".
FYI - This is the address of the paper:
http://msdn.microsoft.com/library/default.asp?URL=/library/techart/EDIandBTS.htm
Additionally, several people have asked for more details on why I selected
Biztalk over the other EDI solutions and wanted some feedback on what I
thought of the issues in this white paper.
I ran across this white paper too while I was looking into Biztalk. I
called upon the help of a couple consulting firms who had some experience
with Biztalk to help me better understand these issues (as well as answer
the "RFI" questions I had initially when I was narrowing down the field of
which software packages to look at).
I dealt with one consulting firm for RFI questions/white paper issues and
another firm for some "mentoring" time.
I'll give credit to these firms at the bottom of this email.
Here are some of the issues which I copy/pasted from the white paper as
well as my views on them.
Outbound Batching
-----------------
Limitation - Although BizTalk Server can process aggregate collections of
EDI documents, it cannot produce them.
Solution - Include a predelivery batching routine on the outbound side of
BizTalk Server, and a debatching routine on the inbound side of BizTalk
Server. This is to accommodate aggregate responses to transmittals batched
outside BizTalk Server.
Effort Required - A low to medium level of effort is required to implement
a solution.
My view on this - According to one of the consulting firms I worked
with..."BizTalk takes each document from an EDI batch and processes it as a
separate document (except for receipting purposes, I believe). Therefore,
when documents appear on the outbound side they are no longer associated.
Even if you generated them internally, you couldn't create multiple
documents in a single interchange. I think the recommended solution is
probably only necessary for outbound documents in most cases, not inbound."
Currently, the only trading partner that we send outbound transactions to
is Caterpillar. We send ANSI X12 810 "invoice" transactions. These can
either be all wrapped up in a single ISA/GS/GE/IEA document or be sent
individually with each invoice generating their own ISA/GS/GE/IEA envelope.
I didn't see this as a serious limitation since (according to CAT) I can
send these wrapped however I need to. In the future, when we have more
than just the 810 documents to send, I believe CAT will still be OK with me
individually wrapping these documents up with their own ISA/GS/GE/IEA
envelopes.
Upfront, I already knew I was going to have to put some "pre/post"
processes in place for receiving and sending transactions to accommidate
CAT's ancient "mailbox system" (which is very similar to an old BBS dial-up
connection). So, accomidating this inability to batch outbound
transactions wasn't much more of an issue for me.
Segment Compound "Tags"
-----------------------
Limitation - Currently, the source tag identifier is the only mechanism in
BizTalk Server by which instance data is matched to schema-defined
structure. BizTalk Server cannot resolve parsing operations when the source
tag identifier in the document instance, by itself, is not sufficient for
determining a structure match in the schema. An example of this might occur
with the HL segments in an X12 856 advanced ship notice, where field data
other than the source tag identifier adds hierarchical context to the
meaning of the record's tag ("HL" in this case).
Solution - Create new EDI parsers that perform parsing look-ahead logic to
consider not only the tag but also the content qualifiers of various EDI
segments.
Effort Required - A high level of effort is required to implement a
solution.
My view on this - Per the consulting firm, "The HL tag should be correctly
processed in BizTalk except that the heirarchical stucture would not be
represented in the resulting XML document. You would have to scan the
resulting XML for the hierarchy level
number in order to understand it correctly. A custom parser is not
necessary, only a more intelligent map or Application Integration Component
(AIC). Level of effort: minimal."
>From what I saw, I could handle this via a decent map and document
specifications. I figured I would have to go through and element by
element, re-build or re-design each of the XML specifications to match each
of our EDI documents that need to be processed. That's usually the case
anyways, since EDI "standards" are so well established. I haven't had an
EDI document from CAT yet that was exactly in-line with the so-called
standard. So, even though Biztalk comes with pre-defined document specs
for the most common EDI documents, I'll have to pick through them and
configure them myself. However, I would have to do that no matter which
package I picked. My experience with Biztalk Editor and Biztalk Mapper was
very positive and I believe these products will allow me to handle this.
Envelope Creation
-----------------
Limitation - BizTalk Server cannot populate custom envelopes. Nor can
BizTalk Server deviate from the EDI-based envelopes that it provides, in
the case where it is necessary to use optional fields on the envelope.
Solution - Develop a predelivery process to add content to envelopes. This
might involve having BizTalk Server build an envelope, and then populating
the envelope in the add-on process. Alternatively, the add-on process could
create and populate the record. In either case, the data could be from the
BizTalk Messaging Management database, a private add-on database, or both.
Effort Required - A medium level of effort is required to implement a
solution.
My view on this - Per the consulting firm, "BizTalk does support custom
envelopes for XML documents. The comments above are correct for EDI
documents."
At first, I thought this would be a big problem for us, especially since
from dealing with CAT in a "prior life", we typically generated "custom"
envelopes by including non-EDI lines just above the ISA segment which told
the CAT mailbox system how to handle the uploading of the transactions.
This was all very specific to CAT's mailbox system and, when I looked back
over it, I realized that it really shouldn't have been done the way it was
at my old job. The person before me set it up that way to generate the
outbound EDI transactions with those extra "CAT Mailbox" segments, but
really this should have all been seperate from the EDI documents themselves
and handled via some "transport" process which takes the EDI documents and
wraps additional "CAT mailbox" information around them. Again, since I'll
be putting in place "pre/post" processes for handling transport issues,
I'll be dealing with the "CAT mailbox" structures outside of the EDI
document itself. So, since Biztalk can produce the "standard" EDI env
elopes, I don't have a problem with this. However, if it was absolutely
necessary, outbound documents could always be mapped and re-mapped and
wrapped and re-wrapped until you got the enveloping you needed.
Functional Acknowledgments
--------------------------
Limitation - There are four limitations in this area:
BizTalk Server parsers cannot take advantage of the range of EDI batching
or aggregation functionality that an EDI server can. For example, BizTalk
Server is unable to reject an entire group based on individual document
failure within the group.
Detail does not include the field level, so it is often impossible to know,
for example, which field failed validation.
The validation step stops at the first error.
BizTalk Server provides no notification or other action when receipts
become overdue.
Solution - There are no solutions for the first three limitations. To solve
the last limitation, you must set up stored procedures as Microsoft® SQL
Server? jobs. The purpose of this is to sweep the tracking database
periodically to look for overdue receipts and to perform notification as
needed.
Effort Required - A low to medium level of effort is required to implement
a solution.
My view on this - Per the consulting firm, re: the issue..."BizTalk Server
provides no notification or other action when receipts become overdue.",
their response was, "Not True. BizTalk stores interchanges in the "Retry"
queue while waiting for a receipt. After the configured retries are
exahsted, the message is moved to the "Suspend Queue". The Event Log and
the Suspend Queue are the notification mechanisms. Pages and E-Mail alerts
can be generated from these using standard NT/SQL Server tools."
Also, "The Suspend queue is the correct place to "sweep" for these errors.
This will also notify for parsing and validation errors."
I hired another consulting firm to do some "mentoring" with me for a couple
of days before I made the final decision and we worked with the Messaging
Queues, and the "retry" queues seemed to do the trick. I don't have a lot
of concern about this, at least not in our environment. Also, we don't
currently send functional acknowledgments (ANSI X12 997s) back to CAT. We
receive them re: our invoices and I'll need to provide document definitions
and mappings for these documents, but we don't currently send anything back
out to CAT.
EDI Data Types
--------------
Limitation - When creating a specification in BizTalk Editor, you cannot
specify EDI data types for X12 document field contents. This limitation is
most significant in custom data types that have to do with explicit or
implied decimal placement and the number of digits before and after the
decimal.
Solution - There is no solution for this limitation within the context of
BizTalk Server.
My view on this - Per the consulting firm, "The data types in schemas
defined by BizTalk are based on the BizTalk
extensions to the XDR standard. Data typing is supported, but it is
defined by those standards, not EDI. If a type can not be correctly
represented by a BizTalk-compatible datatype, it can be represented a
generic string or number and any necessary conversion can be handled in the
mapping."
I think everything can be handled via document definitions. All of the
data types I deal with in my transactions were available to me. If/when I
run across something that's not available, I can always spec it as a string
and handle it with some backend process or via the mapper or in the outside
application that data is going to. Not an issue for me.
Envelope Mapping
----------------
Limitation - BizTalk Server cannot map data from envelopes.
Solution - There is no solution for this limitation within the context of
BizTalk Server.
My view on this - Per the consulting firm, "Each interchange carries the
envelope information with it. Unfortunately,
it is not accessible from within the mapping. For the case mentioned
above, you would simply define multiple Organizations within Biztalk and
assign the appropriate identifiers from the envelope to the Organization.
This is the normal way for BizTalk to determine from whom an interchange
came."
This was an issue that I brought up with the other consulting firm during
our "mentoring" time and I haven't figured out how to deal with this one
yet. Again, from my "prior life", I'm used to having the ISA/GS segments
available to me because we could differentiate which CAT facility sent the
EDI vs. which facility bought the part vs. which facility was to receive
the shipment vs. which facility was to receive the ASN, using several
segments including the ISA segment.
I'm not sure if Biztalk Editor will let me add this information to the
document definition and make it available to the mapper or if I'll have to
handle this in the pre-processs solution. I'm not sure yet.
Binary Segment Content
----------------------
Limitation - BizTalk Server cannot specify a maximum size for a binary
field if that maximum is in excess of 32-bit MAXINT.
Solution - There is no solution for this limitation within the context of
BizTalk Server
My view on this - Per the consulting firm, "True. The solution is not to
specify a maximum size in the specification and check it later if
necessary."
Currently, not an issue for my environment.
Control Number Enforcement
--------------------------
Limitation - There is no way for BizTalk Server to know when duplicate
items are submitted for processing except through the use of the BizTalk
Framework.
Solution - Develop a preprocess that scans all data in the tracking
database (or in a replicated warehouse of all historic tracking data) to
verify the uniqueness of received data prior to submitting it to BizTalk
Messaging.
Effort Required - A medium level of effort is required to implement a
solution.
My view on this - Apparently, this is true. I'm not sure yet, how much of
an issue this will be for me.
Floating Segments
-----------------
Limitation - BizTalk Server does not support floating segments. Segments
defined in schemas are fixed to specific locations in the data according to
where the schema explicitly places them.
Solution - There is no solution for this limitation within the context of
BizTalk Server.
My view on this - Per the consulting firm, "BizTalk supports this
functionality through the use of multiple specifications. You are not
limited to one definition of an 850, for example. You could have a
different definition (including segment orders) for each interchange that
is defined in BizTalk. I believe that most EDI products support variations
in the "standard" from one trading partner to another. This is no
different."
I saw how you can define segments ("records") in the Biztalk Editor
document definitions and how these segments could occur multiple times, but
I didn't test to see if their pattern of occurance was hard-tied to the
spec--not allowing segments to "float".
I'll find this out when I start building my document definitions and maps.
However, I'm not sure that mutiple document definitions is the only
solution. I would think Biztalk Editor is slick enough to let you overcome
this somehow. It seems to be at the very root of XML's ability to define
EDI structures, so I'd be surprized if there was no way to accomidate this.
VAN Integration
---------------
Limitation - There are two limitations in this area:
BizTalk Server has no built-in VAN transport components for sending or
receiving data.
BizTalk Server has no mechanism for entering VAN sender and receiver status
reports into a tracking database.
Solution - Develop application integration components (AICs) to serve as
the transport mechanism to interact with a VAN. There might also be a need
to create tables related to the tracking database that would hold VAN
sender and receiver status reports. This is because there is also a foreign
key relationship between the new tables and existing tracking tables.
Effort Required - A medium to high level of effort is required to implement
a solution.
My view on this - Per the consulting firm, "True. This is a well-known
shortcoming. I would expect third party add-ons, possibly from the VAN's
themselves, to appear shortly."
I knew this going in....and as discussed earlier, I'll have to overcome
this with some custom components. But, mine are needed more for the lack
of dial-up...not VAN integration. FTP/HTTP/SHTTP are all there, so I guess
it depends on your VAN requirements as to whether this is an issue or not.
Envelope Data Viewing
---------------------
Limitation - It is not possible to see envelopes in BizTalk Document
Tracking, other than by viewing the parent interchange for documents being
searched.
Solution - Modify the BizTalk Document Tracking user interface so that the
envelopes are optionally displayed with the individual work items.
Effort Required - A high level of effort is required to implement a
solution.
My view on this - Per the consulting firm, "The solution offered here is
probably more expensive than is necessary in most cases. Depending on the
requirements of the client, this could be implemented in a more
straight-forward manner."
When I went through my "mentoring", I did see some basics on document
tracking, but what I saw lead me to believe that as long as you had this
information available, it could be tracked. Personally, it's not an issue
for me...others, however may have a problem with this.
***** IMPORTANT NOTE TO CONSIDER *****
One thing to take into consideration when I say we're going to use Biztalk
is that I belive it is the best "long term" choice for OUR particular
situation. My situation is this...
I just recently (February 1st) started working here as the EDI Specialist
and found that Harbinger TrustedLink was being used. Not only had I
personally had bad experience with the TLW product and it's support for the
past three years at my previous employer, but I also discovered that TLW
was very poorly implemented at G&D Transportation. The person who was here
before me was clearly not experienced in EDI and went to great extents to
compensate for Harbinger's weak mapping tool by writing several layers of
"Java" classes (apparently his development tool of choice) to format and
re-format documents as they were received and as they were sent.
Currently, we only have 3 "implementations" of TLW.
One deals with single facility at Cat called "Cat Defense". We receive
850s which are extracted to another system and we send 810s from that
system back to Cat. Likewise, we receive 997s, but they are currently not
being used for anything. (We have some issues with outstanding invoices so
I'm going to work on utilizing these 997s to link back to the 810 documents
they're acknowledging.)
Another implementation of TLW is where we receive two text (non-EDI)
documents from another CAT facility which, basically tells us what
shipments we need to make that day. This text data is imported into our
dispatching system via a "custom" import process. Again, these are text
documents, but their purpose is basically similar to an 830 (although they
bare little resemblance structurally).
Lastly, we use TLW to manually data enter ASNs for yet another CAT
facility--volume is very very low at no more than 4-5 ASNs per day.
So, as you can see, I'm very very lucky. I have a limited number of
implementations to replace and those that we have, aren't working well to
begin with so, basically, ANY package would be better.
In addition, both the MacIntosh system used for Cat Defense and the custom
built dispatch system are going to be replaced. In fact, during the
replacement of the MacIntosh system is where I'll implement Biztalk for the
first time. I'll have the luxury of being able to design my input/output
documents for this system from scratch and not be bound by a pre-existing
system. We'll be developing this new CAT Defense system ourselves and
Biztalk will be one component of that. Our development environment will be
ASP web pages running on IIS with a SQL Server 2000 backend, so XML
(Biztalk's middle name) will be a factor that we can learn about and take
advantage of during this process.
Knowing what we need to do in order to replace these systems, seeing the
development that needs to occur and the "development"/"customizable"
aspects that Biztalk has, make it a good choice for us.
I found that Biztalk Editor was very strong, Biztalk Mapper was very
strong, and I was impressed with how powerful OrchestrationDesigner was as
well.
Giving credit where credit is due:
==================================
As I mentioned, there were two consulting firms that I utilized when
looking at Biztalk:
plaNET Consulting, Inc. was kind enough to answer my initial questions
about Biztalk as well as answer the RFI questions.
Their address is http://planetci.com and the person I dealt with was John
Stahlnecker ([EMAIL PROTECTED]).
I also dealt with IMG University (http://www.imguniversity.com) through
Tony Saineghi ([EMAIL PROTECTED]) who had Tony Guidici ([EMAIL PROTECTED])
visit me for two days for a "mentoring" session where we went through some
Biztalk tutorials and tried to address some of the specific problems I'll
have to address when I implement Biztalk in our environment.
I owe both of these firms a "THANKS" for helping me out and I'm sure I'll
be working with both of them again as I get deeper into my implementation.
Thanks!
Dan Tharp
ÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝØ contact the list owner: mailto:[EMAIL PROTECTED]
Archives at http://www.mail-archive.com/edi-l%40listserv.ucop.edu/
