Si, What about setting a flag that fails the whole transaction at the end, but allows the import to continue.
This way a report of the bad data could be assembled. This looks like it would solve your problem and be an enhancement to the general import functionality. - Andrew On Mon, 2006-07-10 at 12:36 -0700, Si Chen wrote: > I think what the entity engine tool does it the right way, and I'm > not suggesting that we change it per se. The entity engine tool is > very good for importing seed and demo data as we have it. > > There could be cases where a different process would be better. For > example, if you're importing a lot of initial data (say 10,000 > records), and not all of it is "clean" (say 10 bad records). The > entity engine tool would basically fail on each bad record, so you'd > go back, fix one record, try all 10,000 again, and repeat that ten > times. It might be easier on the user if all 10,000 were read into a > "temporary" table, the good rows imported, then the user fix the > remaining bad rows in the database and insert those. The trouble > with this "process" is that it becomes kind of custom and is harder > to standardize. As I'm writing this I'm not sure what, if any of it, > can be made part of the project. > > Si > > On Jul 10, 2006, at 10:27 AM, David E. Jones wrote: > > > > > The point of transaction management is to have a set of operations > > succeed or fail all together. > > > > Would it really be easier with a large or small file to have all of > > the data go in except for the failures and then try to pick out all > > of the failures and fix and get those in individually in another > > file or by slimming down the original? > > > > Another problem with not failing the entire file is if there are > > dependencies between the data or problems might be caused by having > > partial data in the system, then you'll have cascading failures.... > > > > So how would a partial failure be handled, or what sort of a > > process would it fit into? > > > > -David > > > > > > Si Chen wrote: > >> Still, one issue I have is "exceptions handling": it seems that > >> entity engine xml import would just rollback everything if one of > >> the records to be imported is bad. Isn't better (or standard) to > >> have a system where the data could be imported and then each > >> exception handled individually? > >> Si > >> On Jul 7, 2006, at 1:27 PM, David E. Jones wrote: > >>> > >>> Yes, this looks like a good description of the general idea. > >>> > >>> -David > >>> > >>> > >>> Si Chen wrote: > >>>> David, > >>>> Yes, the idea was to create something that would be better for > >>>> some applications (namely, the one it's built for) than others. > >>>> But I'm very curious about this entity-engine-transform-xml. So > >>>> is this idea to paste a big block of XML into <entity-engine- > >>>> transform-xml> file and then write the .FTL to do the > >>>> transformation? And then run xml import on that file? > >>>> And what about the freemarker template? Does it fall under > >>>> their "Declarative XML Processing" documentation here: http:// > >>>> freemarker.org/docs/xgui_declarative.html I looked at it > >>>> briefly and recognized some of syntax. > >>>> Si > >>>> On Jul 7, 2006, at 12:03 PM, David E. Jones wrote: > >>>>> > >>>>> Stepping back a bit and looking at this conceptually: > >>>>> > >>>>> What you are proposing would be an import format that would > >>>>> work well for a certain piece of software that the data is > >>>>> coming from. > >>>>> > >>>>> In fact, the normalized data model of OFBiz is far easier to > >>>>> transform data to than a more normalized form would be. > >>>>> > >>>>> Consider Software A and Software B that both have very > >>>>> functionality specific, redundant, and generally non-normalized > >>>>> data models. The cardinality of fields relative to a product > >>>>> may be different, and incompatible, and they may treat similar > >>>>> information in very different ways. Because the OFBiz model is > >>>>> more flexible it may be possible to migrate A->OFBiz and B- > >>>>> >OFBiz but it may not be possible to migrate A->B or B->A. > >>>>> > >>>>> So, yes, something is needed to transform incoming data but > >>>>> trying to create something that is less normalized will just > >>>>> make it easier for some systems (those that are close to it) > >>>>> and far more difficult for other systems (that have more > >>>>> conceptual mismatches with it). > >>>>> > >>>>> This is basically the unavoidable law of de-normalization. > >>>>> > >>>>> As far as tools and such go, FreeMarker is actually a pretty > >>>>> good XML transformation tool. The Entity Engine import stuff > >>>>> does already support using FTL files along with an incoming XML > >>>>> file to pre-transform the data. There is an example in the > >>>>> ofbiz/applications/ecommerce/data/DemoContent.xml file. Notice > >>>>> the root element is "entity-engine-transform-xml" instead of > >>>>> "entity-engine-xml". > >>>>> > >>>>> -David > >>>>> > >>>>> > >>>>> Si Chen wrote: > >>>>>> Hi everybody. I've been thinking about importing data into > >>>>>> ofbiz. One of the issues is that we have a highly normalized > >>>>>> data model which would require extensive mapping from external > >>>>>> systems. Nevertheless, I think it is possible to create some > >>>>>> standard "template" entities for importing data and thus > >>>>>> create a standard practice and, eventually, standard tools for > >>>>>> doing it. Here's what I'm thinking, given in the context of > >>>>>> importing product data: > >>>>>> 1. Create a standard non-normalized "holding" entity which > >>>>>> contains productId, quantity on hand, available to promise, > >>>>>> average cost, reorder point. Note that none of these are > >>>>>> linked to any entity in ofbiz. > >>>>>> 2. Add a timestamp fields to identify when the data was > >>>>>> loaded into the "holding" entity and when it was imported into > >>>>>> ofbiz. > >>>>>> 3. Write a standard datafile xml import template which > >>>>>> corresponds to this "holding" entity. > >>>>>> 4. Write some processing script or service which takes new > >>>>>> data from the holding entity and insert it into ofbiz. For > >>>>>> example, in this case it would be to create InventoryItem, > >>>>>> ProductAverageCost, and maybe some AcctgTrans (maybe . . .) > >>>>>> It may be most efficient to write these holding entities > >>>>>> corresponding to popular software programs where people might > >>>>>> want to import data from, so they can mirror those programs' > >>>>>> data layout more easily. > >>>>>> Si > -- Kind Regards Andrew Sykes <[EMAIL PROTECTED]> Sykes Development Ltd http://www.sykesdevelopment.com
