[ https://issues.apache.org/jira/browse/OFBIZ-276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jacques Le Roux closed OFBIZ-276. --------------------------------- Resolution: Duplicate Assignee: Jacques Le Roux > Froogle Data Export > ------------------- > > Key: OFBIZ-276 > URL: https://issues.apache.org/jira/browse/OFBIZ-276 > Project: OFBiz > Issue Type: New Feature > Components: product > Affects Versions: SVN trunk > Reporter: Marco Risaliti > Assignee: Jacques Le Roux > Priority: Minor > Attachments: DataFile.patch, DataIO_Froogle.patch, > DataIO_Froogle2.patch, froogle.php, froogle_patch.patch, > ProductExportServices.java > > > This is a copy of the old-jira issue > http://jira.undersunconsulting.com/browse/OFBIZ-886 from Andrew Sykes. > ===================================================== > Attached is a _very_ rough outline of how I think a froogle export would go. > This needs a lot of work still, but I'm out of time, so hopefully someone > else can take it up. > > > All Comments Work Log Change History Sort Order: > Comment by Andrew Sykes [16/May/06 01:43 PM] [ Permlink ] > A rough outline > Comment by rohit [17/May/06 04:44 AM] [ Permlink ] > hi, > i am very new to java and ofbiz. i have been using the oscommerce shopping > cart for more than 3 years, but fell for the amazing functionality that ofbiz > provides. Though oscommerce performs a very small part of the what ofbiz can > it still has some very useful and essential functions for webmaster and web > marketeers. i am attaching a froogle data exporter from oscommerce site. The > file works independent of the oscommerce installation, i mean one need not > have oscommerce installed to get it working. It works excellently and i used > to set a cron object such that the file was run once a week. > i hope this file will give an idea of how to develop a froogle data exporter > for ofbiz. if i am not wrong we can also tweek this file to retreive the data > from ofbiz database and export to froogle. this might be a temporary solution > and may require one to have php installed, but i guess it will save valuable > time. > any comments or suggestions are welcome. > rohit > Comment by rohit [17/May/06 04:49 AM] [ Permlink ] > if one needs more information about the file attached, please visit > http://www.oscommerce.com/community/contributions,3876/category,all/search,froogle. > > oscommerce is a basic shoppinh cart, but with a very vibrant community, the > feature and modules are very useful to any webmaster. if ofbiz could include > some ideas from oscommerce, it could improve even better. > rohit > Comment by Andrew Sykes [17/May/06 06:39 AM] [ Permlink ] > Rohit, > This would be a VERY messy approach, and I'm not entirely convinced it would > even work with OFBiz > The code I've supplied is more than half of what is required to get a froogle > export working. > I have set it up as a service as that allows it to be scheduled using the > JobScheduler. > If you are looking for somewhere to start with OFBiz code, this is as good a > place as any :-) > Comment by David Yoon [17/May/06 01:19 PM] [ Permlink ] > Hi guys, I'm also working on this issue, although I'm making something a bit > more generic so you could add multiple comparison shopping sites. So far I've > configured it to work (update, export, upload) for froogle, and shopAmex. > Instead of recursing through all the product_categories, I was thinking of > using the inventory_item table as the base, since most of the cs sites want > the products that you upload to be available. currently I'm flattening the > product, inventory_item, category, price, tables into a single export_product > table which contains all the fields necessary for the various cs feed sites. > The update for this table is taking quite a long time though (like 30 mins, > since there are 22k products in the database). Any suggestions? > d. > Comment by Andrew Sykes [17/May/06 04:39 PM] [ Permlink ] > David, > I'm not sure it's a good idea to use the inventory_item table as it's not > actually necessary to use inventory management to run an OFBiz ecommerce > site. Some users just won't care about inventory (e.g. JIT ordering). While > others will have a thirdparty application for inventory management. > Andrew > Comment by Si Chen [18/May/06 11:07 AM] [ Permlink ] > Hey everybody - > I'd like to help. Sorry I got to the party late. We did this once before too, > but it got entombed in some custom code that's no longer worth extracting. > Here is what I'd suggest: > 1. Andrew' approach is mostly correctly, but I'd rather use productStoreId > than webSiteId. Andrew, you are actually using webSiteId to get > productStoreId, but productStoreId is more direct. Also very important: you > need to pull all your variant products as well, which I don't see, or people > would miss most of your real physical products. I definitely don't think > InventoryItem is what you should be using, by the way. > 2. You'll need to configure some links for the products and probabily also an > "atc=___" tag for tracking marketing campaigns (so you know if froogle or > whatever actually made you sales!) I'd suggest a .properties file. > 3. The best way to render the output is actually using the content manager to > create a FTL template of froogle's layout and then render against data. Look > at the DhlServices.java or the GLExportServices in our Financials module. > It's much easier this way than hardcoding formats into Java and more > maintainable as well should the formats change (as they inevitably do.) > Let me know if you need any help... > Comment by Andrew Sykes [18/May/06 12:15 PM] [ Permlink ] > The reason for using websiteId is that you need it further down to create the > links properly. Some comments in my code would have helped here :-) > I'm not sure that people would necessarily want to spam froogle with all > their variants, (for example, imagine a clothes store). Also, variants do not > necesssarily belong to any category, so that might lead to problems. > I totally agree about rendering, again I apologise for not adding this to my > comments. > Comment by Si Chen [18/May/06 12:37 PM] [ Permlink ] > Andrew, > I think you're right about the websiteId then. So you set the URL as well in > WebSite, like in the standardContentPrefix field? > Why don't we make variants/no variants a flag so people can choose? > No problem about the comments. > Anybody want to work on this and submit one we can put into OFBiz? > Si > Comment by Si Chen [18/May/06 12:41 PM] [ Permlink ] > Rohit, > So here's a question about your PHP file, totally off topic, but I'm just > curious about osCommerce: How does this patch work? You put it somewhere, and > somebody downloads the main osCommerce and then this patch? So it's tested > with a particular module...but not necessarily others? Do you just put it in > your www/ directory and it runs? > Now, what about licensing--there's no license terms in the header, so what do > we assume it's licensed under? GPL? Apache? > Please don't get me wrong--I'm not saying it's bad--I'm just curious. > Thanks, > Si > Comment by Ray Barlow [18/May/06 12:53 PM] [ Permlink ] > I have already done some mods to the version Andrew posted, and thrown a > screen in front of it. I had registered a Froogle account the weekend before > with a view to working on this so the whole thread has come at a good time. > The mods I've made already include output created using the DataFile class so > I won't rush to change that to FTL rendering. I'll try and polish a little > more in the next few days and then post it back here although I doubt that it > will be fit for commiting at that stage. > Comment by rohit [18/May/06 12:58 PM] [ Permlink ] > Hi Si, > Thanks for asking. Just to give you an idea Oscommerce is an open source > shopping cart based on PHP. The software and its modules like the file > attached are under GLP license. > Oscommerce has a basic module and the community members can create new > functionality and share with other. The froogle exporteer file is almost > independed on oscommerce files, though it depends on the database structure > of oscommerce. One can simply upload the file to a webserver and set a cron > object for it to run weekly. that is it. > i mentioned this file, to explore the possibility of something like this for > ofbiz, something thats very easy to install and run and that new users as > well as existing users of ofbiz can easy use for there websites. > i have using oscommerce for 3 years and believe me it does not do even 10% of > what ofbiz can do, but the ability of community to develop new contributions > such as froogle exporter, etc and the relative ease of integrating these > contributions into existing webstores, is where i guess oscommerce takes the > lead. i am still learning ofbiz and i hope i am proved wrong. And i did like > to add that you must visit oscommerce.com and take a look on how the 100K > community members have contributed to it. > Rohit > Comment by Andrew Sykes [18/May/06 01:49 PM] [ Permlink ] > I'm glad to hear that someone has been spurred into action by my > contribution, thanks for looking into this Ray. You may want to contact Ian > Gilbert (if you haven't already) as he was also planning to do something with > it, perhaps you guys could collaborate? > I think Si's suggestion of an optional include variants (may) be useful - can > you give an example scenario please Si? Also one thing to consider, there is > a filter category (whose name escapes me) that specifies which products to > show on a site, would this mess up variant posting? > I think it would also be good to add an optional categoryId parameter to the > service definition and a recurse[Y/N] flag. This would mean that without a > categoryId posting would start from the browse root, with a categoryId > posting would start from the specified category and if recurse==N posting > would be exclusively for that category. > In my additional comments on the user list, I suggested using the DataFile > stuff, as that was the first thing that sprung to mind, I really don't see > any objection to carrying on down that route, although someone else might > know different... > Has anyone considered whether setting the Froogle category in the > CategoryAttr is a good approach, I'd be interested to hear any comments on > this... > Also, what about the FTP part, it would be nice if OFBiz handled that > directly, anyone have any suggestioins? > Finally if FTP is integrated, we need somewhere to store the FTP address and > credentials - any suggestions? > Comment by Si Chen [18/May/06 02:22 PM] [ Permlink ] > Hey - > 1. Why variants: if someone searches for "white shirt" on froogle and all you > had was the virtual "shirt" but not variants, you might not get listed. > 2. No idea about the filter category. > 3. Recurse - yeah probably a good idea too. > 4. I still think the content template + render is better than DataFile, but I > could be convinced otherwise. > 5. I think there are some FtpServices.java in ofbiz now. Not sure what it > does though. > Comment by Andrew Sykes [18/May/06 02:43 PM] [ Permlink ] > Si, > Sure that could be an issue, but would you really want to send > 5 colours x 5 neck sizes x 5 chest sizes x 10 braiding colours > = 1250 products > (ok, a bit excessive, but you get the point I'm sure) > I'm guessing froogle would probably take quite a dim view of this especially > if they all had the same product image and description. > Also, having done a few clothing sites, they never have products this > generic, because... > A/ They'd never get a google rank > B/ The retailer would always find a more descriptive product name. > "White Shirt" is more likely to be a category name. > I've been trying to think of a scenario where variants would be useful, but I > just don't think product names would ever be generic enough to need this. > Comment by Si Chen [18/May/06 02:47 PM] [ Permlink ] > That's why it's a configurable parameter, so if you don't want it, you don't > have to use it. It's OK if you don't put it in--I can add it later if we > decide to use it. > Also: FTP address and credentials - use a .properties file in a config/ > directory. That's how the freight shipping and payment processors are > handled. > Comment by Si Chen [18/May/06 03:02 PM] [ Permlink ] > Rohit, > Thanks for your comments. I actually do know of oscommerce but wasn't quite > sure how it worked. Thanks for explaining it. > Well, we are definitely a community project as well and do indeed accept > contributions from a fairly large number of people. I think the difference > you find is not only a greater set of features but also a more formal > framework and process for building applications. Hopefully this will work for > you. All this stuff may seem intimidating at first, but you can definitely > learn it (if I can learn it, anybody can learn it), and once you know it, it > will make your work very easy. > Ok, end of my digressions. > Si > Comment by Ray Barlow [21/May/06 06:03 AM] [ Permlink ] > DataFile.patch adds support to write a DataFile with a header row. > Comment by Ray Barlow [21/May/06 06:10 AM] [ Permlink ] > I have added a menu on the catalog application called "DataIO" next to > "Configurations" and in there is a new screen for Froogle. > The screen contains comments about what is still required to be done which to > be honest still includes a fair amount to be a really cool implementation, > but this submission will at least generate the Froogle file. I've only run on > the demo data so far but will deploy it on one of my live sites asap and test > generating and submitting to Froogle for real. > Although not complete if this is acceptable then it would be nice to commit a > version sooner rather than wait for the holy grail of FTP automation etc. > Comment by rohit [21/May/06 10:28 PM] [ Permlink ] > Hi, > Can you please give some instructions on how to install it. I will surely > test it and give my feedback. > rohit > Comment by Ray Barlow [22/May/06 06:59 AM] [ Permlink ] > The two files I attached are created as SVN patch files. Depending on your > platform and tool depends on how you apply them and to be honest applying > patches is not something I've done a lot of. For Windows if your using > tortoise SVN (which is very good) try right click on the root folder and > "Apply Patch" from the sub menu. For the command line I think you'd use the > "patch" command, but for both I'd suggest a test folder and google would be a > good option. > On a more general note I forgot to mention that I saw the "DataIO" section > being an area for control of other features being talked about like Yahoo, > Amazon and maybe even some simple catalogue data import/export routines. > Comment by rohit [24/May/06 09:23 AM] [ Permlink ] > hi Ray, > i am sorry but i could not get it working. i was initially using > opentaps-0.9.0 but then tried it on SVN using the tortoise SVN client. > i will really appreciate if you could be more specific about how you went > about installing. For. e.g. in which folders are these files to be copied > into and if there is any code that needs to be edited manually. > i am new to ofbiz, hence a little more detailed instructions will help a lot. > thank, > rohit > Comment by rohit [24/May/06 11:43 PM] [ Permlink ] > hi Ray, > i was able to get the files install, but i am getting the following error, > when i try to use the froogle exporter: > 3057047[ServiceEventHandler.java:317:ERROR] > ---- exception report > ---------------------------------------------------------- > Service invocation error > Exception: org.ofbiz.service.GenericServiceException > Message: Service target threw an unexpected exception (null) > ---- stack trace > --------------------------------------------------------------- > org.ofbiz.service.GenericServiceException: Service target threw an unexpected > exception (null) > org.ofbiz.service.engine.StandardJavaEngine.serviceInvoker(StandardJavaEngine.java:113) > > org.ofbiz.service.engine.StandardJavaEngine.runSync(StandardJavaEngine.java:63) > > org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:340) > org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:214) > org.ofbiz.service.GenericDispatcher.runSync(GenericDispatcher.java:123) > org.ofbiz.webapp.event.ServiceEventHandler.invoke(ServiceEventHandler.java:301) > > org.ofbiz.webapp.control.RequestHandler.runEvent(RequestHandler.java:416) > org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:252) > org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:191) > org.ofbiz.webapp.control.ControlServlet.doPost(ControlServlet.java:87) > javax.servlet.http.HttpServlet.service(HttpServlet.java:615) > javax.servlet.http.HttpServlet.service(HttpServlet.java:688) > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) > > org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:258) > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) > > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) > > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:526) > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856) > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744) > > org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527) > > org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80) > > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) > > java.lang.Thread.run(Thread.java:534) > java.lang.NullPointerException > org.ofbiz.product.product.ProductExportServices.productsToCsv(ProductExportServices.java:73) > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > java.lang.reflect.Method.invoke(Method.java:324) > org.ofbiz.service.engine.StandardJavaEngine.serviceInvoker(StandardJavaEngine.java:101) > > org.ofbiz.service.engine.StandardJavaEngine.runSync(StandardJavaEngine.java:63) > > org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:340) > org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:214) > org.ofbiz.service.GenericDispatcher.runSync(GenericDispatcher.java:123) > org.ofbiz.webapp.event.ServiceEventHandler.invoke(ServiceEventHandler.java:301) > > org.ofbiz.webapp.control.RequestHandler.runEvent(RequestHandler.java:416) > org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:252) > org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:191) > org.ofbiz.webapp.control.ControlServlet.doPost(ControlServlet.java:87) > javax.servlet.http.HttpServlet.service(HttpServlet.java:615) > javax.servlet.http.HttpServlet.service(HttpServlet.java:688) > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) > > org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:258) > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) > > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) > > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:526) > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856) > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744) > > org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527) > > org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80) > > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) > > java.lang.Thread.run(Thread.java:534) > -------------------------------------------------------------------------------- > > 3057047[ RequestHandler.java:283:ERROR] Request froogleExport caused an error > with the following message: Error calling event: > org.ofbiz.webapp.event.EventHandlerException: Service invocation error (null) > 3057047[ RequestHandler.java:395:INFO ] [RequestHandler.doRequest]: Response > is a view. > 3057047[ RequestHandler.java:547:INFO ] servletName=control, view=Froogle > 3057109[ UtilXml.java:242:DEBUG] XML Read 0.062s: > file:/C:/ofbiz/applications/product/widget/catalog/DataIOScreens.xml > 3057109[ ScreenFactory.java:120:INFO ] Got 2 screens in 0.062s from: > file:/C:/ofbiz/applications/product/widget/catalog/DataIOScreens.xml > 3057281[ UtilXml.java:242:DEBUG] XML Read 0.156s: > file:/C:/ofbiz/applications/product/widget/catalog/CatalogCommonScreens.xml > 3057281[ ScreenFactory.java:120:INFO ] Got 13 screens in 0.156s from: > file:/C:/ofbiz/applications/product/widget/catalog/CatalogCommonScreens.xml > 3057328[ UtilXml.java:242:DEBUG] XML Read 0.031s: > file:/C:/ofbiz/framework/common/widget/CommonScreens.xml > 3057344[ ScreenFactory.java:120:INFO ] Got 8 screens in 0.047s from: > file:/C:/ofbiz/framework/common/widget/CommonScreens.xml > 3057578[ UtilXml.java:242:DEBUG] XML Read 0.047s: > file:/C:/ofbiz/applications/product/widget/catalog/CommonScreens.xml > 3057578[ ScreenFactory.java:120:INFO ] Got 9 screens in 0.047s from: > file:/C:/ofbiz/applications/product/widget/catalog/CommonScreens.xml > 3057750[ ServiceDispatcher.java:452:DEBUG] [[Sync service finished- > total:0.0,since last(Begin):0.0]] - 'catalog / > getProductCategoryAndLimitedMembers' > 3057812[ ControlServlet.java:258:DEBUG] [[[froogleExport] Done rendering > page, Servlet Finished- total:1.125,since last([froogleExport] S...):1.125]] > 3061609[ ControlServlet.java:98 :INFO ] The character encoding of the request > is: [null]. The character encoding we will use for the request and response > is: [UTF-8] > 3061625[ ControlServlet.java:134:DEBUG] [[[froogleExport] Servlet Starting, > doing setup- total:0.0,since last(Begin):0.0]] > 3061625[ ControlServlet.java:183:DEBUG] [[[froogleExport] Setup done, doing > Event(s) and View(s)- total:0.0,since last([froogleExport] S...):0.0]] > 3061625[ RequestHandler.java:212:INFO ] [Processing Request]: froogleExport > 3061672[ ServiceDispatcher.java:386:DEBUG] [[Sync service failed...- > total:0.0,since last(Begin):0.0]] - 'catalog / froogleExport' > 3061687[ ServiceDispatcher.java:389:ERROR] > Please advise what might be wrong. > rohit > Comment by Ray Barlow [05/Jun/06 12:17 PM] [ Permlink ] > Attached is an improved version that has a little more error checking and > also uses drop down lists to reduce user entry errors, for data such as > webSiteId. The NPE exception Rohit was experiencing would have been due to an > invalid webSiteId entry. > Also added support for tracking codes if defined in the marketing module and > a check to skip null images for products. > It would be nice to see this reviewed and committed if possible. DataFile > patch still stands as submitted before as I've not changed anything in it > that for this update. > Comment by rohit [21/Jul/06 01:58 AM] [ Permlink ] > hi, > i have attached a new patch file to work with nre requirement of google base > format. This patch has been created on the latest SVN 424158. > Comment by rohit [21/Jul/06 02:00 AM] [ Permlink ] > This patch has all the changes. you need only this file to implement froogle > on ofbiz. > Comment by Marco Risaliti [13/Sep/06 04:12 PM] [ Permlink ] > May I have to move it to the new jira server ? > Thanks > Marco -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.