How about having a utility that simply outputs all 10 errors WITHOUT importing
anything, so the ten can be fixed, then the entire file is imported.
Si Chen wrote:
I think what the entity engine tool does it the right way, and I'm not
suggesting that we change it per se. The entity engine tool is very
good for importing seed and demo data as we have it.
There could be cases where a different process would be better. For
example, if you're importing a lot of initial data (say 10,000
records), and not all of it is "clean" (say 10 bad records). The
entity engine tool would basically fail on each bad record, so you'd go
back, fix one record, try all 10,000 again, and repeat that ten times.
It might be easier on the user if all 10,000 were read into a
"temporary" table, the good rows imported, then the user fix the
remaining bad rows in the database and insert those. The trouble with
this "process" is that it becomes kind of custom and is harder to
standardize. As I'm writing this I'm not sure what, if any of it, can
be made part of the project.
Si
On Jul 10, 2006, at 10:27 AM, David E. Jones wrote:
The point of transaction management is to have a set of operations
succeed or fail all together.
Would it really be easier with a large or small file to have all of
the data go in except for the failures and then try to pick out all
of the failures and fix and get those in individually in another file
or by slimming down the original?
Another problem with not failing the entire file is if there are
dependencies between the data or problems might be caused by having
partial data in the system, then you'll have cascading failures....
So how would a partial failure be handled, or what sort of a process
would it fit into?
-David
Si Chen wrote:
Still, one issue I have is "exceptions handling": it seems that
entity engine xml import would just rollback everything if one of
the records to be imported is bad. Isn't better (or standard) to
have a system where the data could be imported and then each
exception handled individually?
Si
On Jul 7, 2006, at 1:27 PM, David E. Jones wrote:
Yes, this looks like a good description of the general idea.
-David
Si Chen wrote:
David,
Yes, the idea was to create something that would be better for
some applications (namely, the one it's built for) than others.
But I'm very curious about this entity-engine-transform-xml. So
is this idea to paste a big block of XML into <entity-engine-
transform-xml> file and then write the .FTL to do the
transformation? And then run xml import on that file?
And what about the freemarker template? Does it fall under their
"Declarative XML Processing" documentation here: http://
freemarker.org/docs/xgui_declarative.html I looked at it briefly
and recognized some of syntax.
Si
On Jul 7, 2006, at 12:03 PM, David E. Jones wrote:
Stepping back a bit and looking at this conceptually:
What you are proposing would be an import format that would work
well for a certain piece of software that the data is coming from.
In fact, the normalized data model of OFBiz is far easier to
transform data to than a more normalized form would be.
Consider Software A and Software B that both have very
functionality specific, redundant, and generally non-normalized
data models. The cardinality of fields relative to a product may
be different, and incompatible, and they may treat similar
information in very different ways. Because the OFBiz model is
more flexible it may be possible to migrate A->OFBiz and B- >OFBiz
but it may not be possible to migrate A->B or B->A.
So, yes, something is needed to transform incoming data but
trying to create something that is less normalized will just make
it easier for some systems (those that are close to it) and far
more difficult for other systems (that have more conceptual
mismatches with it).
This is basically the unavoidable law of de-normalization.
As far as tools and such go, FreeMarker is actually a pretty good
XML transformation tool. The Entity Engine import stuff does
already support using FTL files along with an incoming XML file
to pre-transform the data. There is an example in the
ofbiz/applications/ecommerce/data/DemoContent.xml file. Notice
the root element is "entity-engine-transform-xml" instead of
"entity-engine-xml".
-David
Si Chen wrote:
Hi everybody. I've been thinking about importing data into
ofbiz. One of the issues is that we have a highly normalized
data model which would require extensive mapping from external
systems. Nevertheless, I think it is possible to create some
standard "template" entities for importing data and thus create
a standard practice and, eventually, standard tools for doing
it. Here's what I'm thinking, given in the context of importing
product data:
1. Create a standard non-normalized "holding" entity which
contains productId, quantity on hand, available to promise,
average cost, reorder point. Note that none of these are linked
to any entity in ofbiz.
2. Add a timestamp fields to identify when the data was loaded
into the "holding" entity and when it was imported into ofbiz.
3. Write a standard datafile xml import template which
corresponds to this "holding" entity.
4. Write some processing script or service which takes new data
from the holding entity and insert it into ofbiz. For example,
in this case it would be to create InventoryItem,
ProductAverageCost, and maybe some AcctgTrans (maybe . . .)
It may be most efficient to write these holding entities
corresponding to popular software programs where people might
want to import data from, so they can mirror those programs'
data layout more easily.
Si
