[jira] [Commented] (OFBIZ-5410) External File Import/Export: A New Design
[ https://issues.apache.org/jira/browse/OFBIZ-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14110597#comment-14110597 ] Adrian Crum commented on OFBIZ-5410: Please see my comment on 28-Nov-2013. I tried to limit the scope of this issue to keep it achievable. > External File Import/Export: A New Design > - > > Key: OFBIZ-5410 > URL: https://issues.apache.org/jira/browse/OFBIZ-5410 > Project: OFBiz > Issue Type: Improvement > Components: framework >Reporter: Adrian Crum >Priority: Minor > Attachments: OFBIZ-5410-schema.patch, OFBIZ-5410.patch > > > Redesign the datafile component to make external file import/export easier > and more reliable. Details are in the comments. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OFBIZ-5410) External File Import/Export: A New Design
[ https://issues.apache.org/jira/browse/OFBIZ-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14110594#comment-14110594 ] Pierre Smits commented on OFBIZ-5410: - Ok. Will go with that. I guess if we have this, it should be reusable for importing and exporting to xml data files as well, or not? Or do we need to look at other solutions for that? > External File Import/Export: A New Design > - > > Key: OFBIZ-5410 > URL: https://issues.apache.org/jira/browse/OFBIZ-5410 > Project: OFBiz > Issue Type: Improvement > Components: framework >Reporter: Adrian Crum >Priority: Minor > Attachments: OFBIZ-5410-schema.patch, OFBIZ-5410.patch > > > Redesign the datafile component to make external file import/export easier > and more reliable. Details are in the comments. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OFBIZ-5410) External File Import/Export: A New Design
[ https://issues.apache.org/jira/browse/OFBIZ-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14110591#comment-14110591 ] Adrian Crum commented on OFBIZ-5410: I would prefer that all of the artifacts stay contained in the datafile component. Other components can reuse screens/forms from there. > External File Import/Export: A New Design > - > > Key: OFBIZ-5410 > URL: https://issues.apache.org/jira/browse/OFBIZ-5410 > Project: OFBiz > Issue Type: Improvement > Components: framework >Reporter: Adrian Crum >Priority: Minor > Attachments: OFBIZ-5410-schema.patch, OFBIZ-5410.patch > > > Redesign the datafile component to make external file import/export easier > and more reliable. Details are in the comments. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OFBIZ-5410) External File Import/Export: A New Design
[ https://issues.apache.org/jira/browse/OFBIZ-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14110588#comment-14110588 ] Pierre Smits commented on OFBIZ-5410: - As importing and exporting data can happen on any level and in every component, having it just available in webtools (as most functionality is currently concentrated), it seems to me that a patch having a (generic and extendable) hot-deployable front end app for this functionality would be the best approach. The deployable front end can also be uploaded here as patch files. This way we don't need to touch existing code until we're good and ready to integrate in whatever release, and others can also contribute. What do you think? > External File Import/Export: A New Design > - > > Key: OFBIZ-5410 > URL: https://issues.apache.org/jira/browse/OFBIZ-5410 > Project: OFBiz > Issue Type: Improvement > Components: framework >Reporter: Adrian Crum >Priority: Minor > Attachments: OFBIZ-5410-schema.patch, OFBIZ-5410.patch > > > Redesign the datafile component to make external file import/export easier > and more reliable. Details are in the comments. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OFBIZ-5410) External File Import/Export: A New Design
[ https://issues.apache.org/jira/browse/OFBIZ-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14110582#comment-14110582 ] Adrian Crum commented on OFBIZ-5410: The schema.patch file was just me whiteboarding some ideas for review and comment. The 11-Mar-2014 patch contains the code I had developed so far. > External File Import/Export: A New Design > - > > Key: OFBIZ-5410 > URL: https://issues.apache.org/jira/browse/OFBIZ-5410 > Project: OFBiz > Issue Type: Improvement > Components: framework >Reporter: Adrian Crum >Priority: Minor > Attachments: OFBIZ-5410-schema.patch, OFBIZ-5410.patch > > > Redesign the datafile component to make external file import/export easier > and more reliable. Details are in the comments. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OFBIZ-5410) External File Import/Export: A New Design
[ https://issues.apache.org/jira/browse/OFBIZ-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14110580#comment-14110580 ] Pierre Smits commented on OFBIZ-5410: - Are both patches required to implement, test and/or extend? Of does the latest supersede the first? > External File Import/Export: A New Design > - > > Key: OFBIZ-5410 > URL: https://issues.apache.org/jira/browse/OFBIZ-5410 > Project: OFBiz > Issue Type: Improvement > Components: framework >Reporter: Adrian Crum >Priority: Minor > Attachments: OFBIZ-5410-schema.patch, OFBIZ-5410.patch > > > Redesign the datafile component to make external file import/export easier > and more reliable. Details are in the comments. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OFBIZ-5410) External File Import/Export: A New Design
[ https://issues.apache.org/jira/browse/OFBIZ-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14110572#comment-14110572 ] Adrian Crum commented on OFBIZ-5410: I have some code fragments I can contribute. I already implemented this for a client, but that implementation was not generic enough for inclusion in the project. > External File Import/Export: A New Design > - > > Key: OFBIZ-5410 > URL: https://issues.apache.org/jira/browse/OFBIZ-5410 > Project: OFBiz > Issue Type: Improvement > Components: framework >Reporter: Adrian Crum >Priority: Minor > Attachments: OFBIZ-5410-schema.patch, OFBIZ-5410.patch > > > Redesign the datafile component to make external file import/export easier > and more reliable. Details are in the comments. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OFBIZ-5410) External File Import/Export: A New Design
[ https://issues.apache.org/jira/browse/OFBIZ-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14110568#comment-14110568 ] Pierre Smits commented on OFBIZ-5410: - I am willing to help on this. It will surely aid in the adoption of OFBiz when importing is handled in a consisted and well documented way. > External File Import/Export: A New Design > - > > Key: OFBIZ-5410 > URL: https://issues.apache.org/jira/browse/OFBIZ-5410 > Project: OFBiz > Issue Type: Improvement > Components: framework >Reporter: Adrian Crum >Priority: Minor > Attachments: OFBIZ-5410-schema.patch, OFBIZ-5410.patch > > > Redesign the datafile component to make external file import/export easier > and more reliable. Details are in the comments. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OFBIZ-5410) External File Import/Export: A New Design
[ https://issues.apache.org/jira/browse/OFBIZ-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836629#comment-13836629 ] Adrian Crum commented on OFBIZ-5410: Thank you for the review! I haven't decided on default values yet, I am only working on the general structure or hierarchy. > External File Import/Export: A New Design > - > > Key: OFBIZ-5410 > URL: https://issues.apache.org/jira/browse/OFBIZ-5410 > Project: OFBiz > Issue Type: Improvement >Reporter: Adrian Crum >Assignee: Adrian Crum >Priority: Minor > Attachments: OFBIZ-5410-schema.patch > > > Redesign the datafile component to make external file import/export easier > and more reliable. Details are in the comments. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (OFBIZ-5410) External File Import/Export: A New Design
[
https://issues.apache.org/jira/browse/OFBIZ-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836562#comment-13836562
]
Jacques Le Roux commented on OFBIZ-5410:
I just hastily tried, I found the schema use straightforward. I just wondered
why you did not use default values where you documented them as "typically".
For instance
{code}
The column delimiter or separator character.
Typically a comma or tab character. Default to comma.
{code}
> External File Import/Export: A New Design
> -
>
> Key: OFBIZ-5410
> URL: https://issues.apache.org/jira/browse/OFBIZ-5410
> Project: OFBiz
> Issue Type: Improvement
>Reporter: Adrian Crum
>Assignee: Adrian Crum
>Priority: Minor
> Attachments: OFBIZ-5410-schema.patch
>
>
> Redesign the datafile component to make external file import/export easier
> and more reliable. Details are in the comments.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (OFBIZ-5410) External File Import/Export: A New Design
[ https://issues.apache.org/jira/browse/OFBIZ-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13835318#comment-13835318 ] Adrian Crum commented on OFBIZ-5410: Jacopo - thank you very much for the review! I have considered the option of specifying a service so each record is mapped to service IN parameters and the service is invoked on each record. I am currently trying out that idea on my local copy. I plan to support CSV (delimited) and SDF (fixed length record) file formats initially. The design will accommodate plugging in additional formats. The "schema-less" imports is a great idea - I will include that. > External File Import/Export: A New Design > - > > Key: OFBIZ-5410 > URL: https://issues.apache.org/jira/browse/OFBIZ-5410 > Project: OFBiz > Issue Type: Improvement >Reporter: Adrian Crum >Assignee: Adrian Crum >Priority: Minor > > Redesign the datafile component to make external file import/export easier > and more reliable. Details are in the comments. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (OFBIZ-5410) External File Import/Export: A New Design
[ https://issues.apache.org/jira/browse/OFBIZ-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13835225#comment-13835225 ] Jacopo Cappellato commented on OFBIZ-5410: -- Just to give you an idea of what I meant with "configuration by convention" in this context, here is an example: from OFBiz you can export the data from an entity as CSV; by default the system will create a CSV file name like _.csv (for example Product_20131129-1350.csv) containing field names in the first row The import mechanism could be able to read a file in the same format and import it without the need of a schema definition. > External File Import/Export: A New Design > - > > Key: OFBIZ-5410 > URL: https://issues.apache.org/jira/browse/OFBIZ-5410 > Project: OFBiz > Issue Type: Improvement >Reporter: Adrian Crum >Assignee: Adrian Crum >Priority: Minor > > Redesign the datafile component to make external file import/export easier > and more reliable. Details are in the comments. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (OFBIZ-5410) External File Import/Export: A New Design
[
https://issues.apache.org/jira/browse/OFBIZ-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13835220#comment-13835220
]
Jacopo Cappellato commented on OFBIZ-5410:
--
Misc comments:
* I agree with the overall design
* I agree that the existing data import tool is old and has problems
* I see a potential for unifying some code that process xml-data imports with
the new code (not sure if it is feasible, I am just throwing out an idea)
* I agree that the data import tool should not deal with complex data
processing; a mechanism to support more complex logic could be that of
triggering services (one service call per row or one service call per file)
* I agree we should reuse Common CSV
* in my opinion it is not essential to support various import formats:
supporting one very well (e.g. CSV) it is better than providing several options
not well supported
* I also like the idea of schemaless imports ("configuration by convention") or
imports that accepts a schema and ignore fields that are not part of the schema
> External File Import/Export: A New Design
> -
>
> Key: OFBIZ-5410
> URL: https://issues.apache.org/jira/browse/OFBIZ-5410
> Project: OFBiz
> Issue Type: Improvement
>Reporter: Adrian Crum
>Assignee: Adrian Crum
>Priority: Minor
>
> Redesign the datafile component to make external file import/export easier
> and more reliable. Details are in the comments.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (OFBIZ-5410) External File Import/Export: A New Design
[ https://issues.apache.org/jira/browse/OFBIZ-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13834804#comment-13834804 ] Adrian Crum commented on OFBIZ-5410: I would like to limit the scope of this issue: 1. Support multiple file formats. 2. External data can be mapped to one or more OFBiz entities. 3. Support flat file formats only. (It appears the current code tries to accommodate hierarchical external files, but that will complicate things unnecessarily.) 4. Support data type conversion. 5. Support multiple character sets. 6. Support primary key generation during import. The basic goal will be to do a good job of importing/exporting, but not try to be all things to all people. Once the conversion is done, the receiving system can process the data further if necessary - it is not the responsibility of the datafile component to do complicated processing. > External File Import/Export: A New Design > - > > Key: OFBIZ-5410 > URL: https://issues.apache.org/jira/browse/OFBIZ-5410 > Project: OFBiz > Issue Type: Improvement >Reporter: Adrian Crum >Assignee: Adrian Crum >Priority: Minor > > Redesign the datafile component to make external file import/export easier > and more reliable. Details are in the comments. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (OFBIZ-5410) External File Import/Export: A New Design
[ https://issues.apache.org/jira/browse/OFBIZ-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13834796#comment-13834796 ] Adrian Crum commented on OFBIZ-5410: The CSV code needs to use the Commons CSV library (introduced in rev 1502093). The current CSV code is very crude and breaks easily. We need an easy way to integrate file export with the screen widgets. The current implementation is problematic. > External File Import/Export: A New Design > - > > Key: OFBIZ-5410 > URL: https://issues.apache.org/jira/browse/OFBIZ-5410 > Project: OFBiz > Issue Type: Improvement >Reporter: Adrian Crum >Priority: Minor > > Redesign the datafile component to make external file import/export easier > and more reliable. Details are in the comments. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (OFBIZ-5410) External File Import/Export: A New Design
[ https://issues.apache.org/jira/browse/OFBIZ-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13834784#comment-13834784 ] Adrian Crum commented on OFBIZ-5410: Another fundamental problem is the lack of generic import/export services. We need services that accept the location of the eternal file descriptor and perform the import/export task. > External File Import/Export: A New Design > - > > Key: OFBIZ-5410 > URL: https://issues.apache.org/jira/browse/OFBIZ-5410 > Project: OFBiz > Issue Type: Improvement >Reporter: Adrian Crum >Priority: Minor > > Redesign the datafile component to make external file import/export easier > and more reliable. Details are in the comments. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (OFBIZ-5410) External File Import/Export: A New Design
[ https://issues.apache.org/jira/browse/OFBIZ-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13834783#comment-13834783 ] Adrian Crum commented on OFBIZ-5410: One of the fundamental problems I see with the current implementation is the schema for the description of the external file(s) - datafiles.xsd. The element tries to describe every file format - which makes it confusing and difficult to use. All of it is undocumented - making things worse. I propose we begin with a new schema that does a better job of separating and describing file formats. > External File Import/Export: A New Design > - > > Key: OFBIZ-5410 > URL: https://issues.apache.org/jira/browse/OFBIZ-5410 > Project: OFBiz > Issue Type: Improvement >Reporter: Adrian Crum >Priority: Minor > > Redesign the datafile component to make external file import/export easier > and more reliable. Details are in the comments. -- This message was sent by Atlassian JIRA (v6.1#6144)
