[ http://issues.apache.org/jira/browse/OFBIZ-276?page=all ]
Jacopo Cappellato updated OFBIZ-276:
------------------------------------
Comment: was deleted
> Froogle Data Export
> -------------------
>
> Key: OFBIZ-276
> URL: http://issues.apache.org/jira/browse/OFBIZ-276
> Project: OFBiz (The Open for Business Project)
> Issue Type: New Feature
> Components: product
> Affects Versions: SVN trunk
> Reporter: Marco Risaliti
> Priority: Minor
> Attachments: DataFile.patch, DataIO_Froogle.patch,
> DataIO_Froogle2.patch, froogle.php, froogle_patch.patch,
> ProductExportServices.java
>
>
> This is a copy of the old-jira issue
> http://jira.undersunconsulting.com/browse/OFBIZ-886 from Andrew Sykes.
> =====================================================
> Attached is a _very_ rough outline of how I think a froogle export would go.
> This needs a lot of work still, but I'm out of time, so hopefully someone
> else can take it up.
>
>
> All Comments Work Log Change History Sort Order:
> Comment by Andrew Sykes [16/May/06 01:43 PM] [ Permlink ]
> A rough outline
> Comment by rohit [17/May/06 04:44 AM] [ Permlink ]
> hi,
> i am very new to java and ofbiz. i have been using the oscommerce shopping
> cart for more than 3 years, but fell for the amazing functionality that ofbiz
> provides. Though oscommerce performs a very small part of the what ofbiz can
> it still has some very useful and essential functions for webmaster and web
> marketeers. i am attaching a froogle data exporter from oscommerce site. The
> file works independent of the oscommerce installation, i mean one need not
> have oscommerce installed to get it working. It works excellently and i used
> to set a cron object such that the file was run once a week.
> i hope this file will give an idea of how to develop a froogle data exporter
> for ofbiz. if i am not wrong we can also tweek this file to retreive the data
> from ofbiz database and export to froogle. this might be a temporary solution
> and may require one to have php installed, but i guess it will save valuable
> time.
> any comments or suggestions are welcome.
> rohit
> Comment by rohit [17/May/06 04:49 AM] [ Permlink ]
> if one needs more information about the file attached, please visit
> http://www.oscommerce.com/community/contributions,3876/category,all/search,froogle.
>
> oscommerce is a basic shoppinh cart, but with a very vibrant community, the
> feature and modules are very useful to any webmaster. if ofbiz could include
> some ideas from oscommerce, it could improve even better.
> rohit
> Comment by Andrew Sykes [17/May/06 06:39 AM] [ Permlink ]
> Rohit,
> This would be a VERY messy approach, and I'm not entirely convinced it would
> even work with OFBiz
> The code I've supplied is more than half of what is required to get a froogle
> export working.
> I have set it up as a service as that allows it to be scheduled using the
> JobScheduler.
> If you are looking for somewhere to start with OFBiz code, this is as good a
> place as any :-)
> Comment by David Yoon [17/May/06 01:19 PM] [ Permlink ]
> Hi guys, I'm also working on this issue, although I'm making something a bit
> more generic so you could add multiple comparison shopping sites. So far I've
> configured it to work (update, export, upload) for froogle, and shopAmex.
> Instead of recursing through all the product_categories, I was thinking of
> using the inventory_item table as the base, since most of the cs sites want
> the products that you upload to be available. currently I'm flattening the
> product, inventory_item, category, price, tables into a single export_product
> table which contains all the fields necessary for the various cs feed sites.
> The update for this table is taking quite a long time though (like 30 mins,
> since there are 22k products in the database). Any suggestions?
> d.
> Comment by Andrew Sykes [17/May/06 04:39 PM] [ Permlink ]
> David,
> I'm not sure it's a good idea to use the inventory_item table as it's not
> actually necessary to use inventory management to run an OFBiz ecommerce
> site. Some users just won't care about inventory (e.g. JIT ordering). While
> others will have a thirdparty application for inventory management.
> Andrew
> Comment by Si Chen [18/May/06 11:07 AM] [ Permlink ]
> Hey everybody -
> I'd like to help. Sorry I got to the party late. We did this once before too,
> but it got entombed in some custom code that's no longer worth extracting.
> Here is what I'd suggest:
> 1. Andrew' approach is mostly correctly, but I'd rather use productStoreId
> than webSiteId. Andrew, you are actually using webSiteId to get
> productStoreId, but productStoreId is more direct. Also very important: you
> need to pull all your variant products as well, which I don't see, or people
> would miss most of your real physical products. I definitely don't think
> InventoryItem is what you should be using, by the way.
> 2. You'll need to configure some links for the products and probabily also an
> "atc=___" tag for tracking marketing campaigns (so you know if froogle or
> whatever actually made you sales!) I'd suggest a .properties file.
> 3. The best way to render the output is actually using the content manager to
> create a FTL template of froogle's layout and then render against data. Look
> at the DhlServices.java or the GLExportServices in our Financials module.
> It's much easier this way than hardcoding formats into Java and more
> maintainable as well should the formats change (as they inevitably do.)
> Let me know if you need any help...
> Comment by Andrew Sykes [18/May/06 12:15 PM] [ Permlink ]
> The reason for using websiteId is that you need it further down to create the
> links properly. Some comments in my code would have helped here :-)
> I'm not sure that people would necessarily want to spam froogle with all
> their variants, (for example, imagine a clothes store). Also, variants do not
> necesssarily belong to any category, so that might lead to problems.
> I totally agree about rendering, again I apologise for not adding this to my
> comments.
> Comment by Si Chen [18/May/06 12:37 PM] [ Permlink ]
> Andrew,
> I think you're right about the websiteId then. So you set the URL as well in
> WebSite, like in the standardContentPrefix field?
> Why don't we make variants/no variants a flag so people can choose?
> No problem about the comments.
> Anybody want to work on this and submit one we can put into OFBiz?
> Si
> Comment by Si Chen [18/May/06 12:41 PM] [ Permlink ]
> Rohit,
> So here's a question about your PHP file, totally off topic, but I'm just
> curious about osCommerce: How does this patch work? You put it somewhere, and
> somebody downloads the main osCommerce and then this patch? So it's tested
> with a particular module...but not necessarily others? Do you just put it in
> your www/ directory and it runs?
> Now, what about licensing--there's no license terms in the header, so what do
> we assume it's licensed under? GPL? Apache?
> Please don't get me wrong--I'm not saying it's bad--I'm just curious.
> Thanks,
> Si
> Comment by Ray Barlow [18/May/06 12:53 PM] [ Permlink ]
> I have already done some mods to the version Andrew posted, and thrown a
> screen in front of it. I had registered a Froogle account the weekend before
> with a view to working on this so the whole thread has come at a good time.
> The mods I've made already include output created using the DataFile class so
> I won't rush to change that to FTL rendering. I'll try and polish a little
> more in the next few days and then post it back here although I doubt that it
> will be fit for commiting at that stage.
> Comment by rohit [18/May/06 12:58 PM] [ Permlink ]
> Hi Si,
> Thanks for asking. Just to give you an idea Oscommerce is an open source
> shopping cart based on PHP. The software and its modules like the file
> attached are under GLP license.
> Oscommerce has a basic module and the community members can create new
> functionality and share with other. The froogle exporteer file is almost
> independed on oscommerce files, though it depends on the database structure
> of oscommerce. One can simply upload the file to a webserver and set a cron
> object for it to run weekly. that is it.
> i mentioned this file, to explore the possibility of something like this for
> ofbiz, something thats very easy to install and run and that new users as
> well as existing users of ofbiz can easy use for there websites.
> i have using oscommerce for 3 years and believe me it does not do even 10% of
> what ofbiz can do, but the ability of community to develop new contributions
> such as froogle exporter, etc and the relative ease of integrating these
> contributions into existing webstores, is where i guess oscommerce takes the
> lead. i am still learning ofbiz and i hope i am proved wrong. And i did like
> to add that you must visit oscommerce.com and take a look on how the 100K
> community members have contributed to it.
> Rohit
> Comment by Andrew Sykes [18/May/06 01:49 PM] [ Permlink ]
> I'm glad to hear that someone has been spurred into action by my
> contribution, thanks for looking into this Ray. You may want to contact Ian
> Gilbert (if you haven't already) as he was also planning to do something with
> it, perhaps you guys could collaborate?
> I think Si's suggestion of an optional include variants (may) be useful - can
> you give an example scenario please Si? Also one thing to consider, there is
> a filter category (whose name escapes me) that specifies which products to
> show on a site, would this mess up variant posting?
> I think it would also be good to add an optional categoryId parameter to the
> service definition and a recurse[Y/N] flag. This would mean that without a
> categoryId posting would start from the browse root, with a categoryId
> posting would start from the specified category and if recurse==N posting
> would be exclusively for that category.
> In my additional comments on the user list, I suggested using the DataFile
> stuff, as that was the first thing that sprung to mind, I really don't see
> any objection to carrying on down that route, although someone else might
> know different...
> Has anyone considered whether setting the Froogle category in the
> CategoryAttr is a good approach, I'd be interested to hear any comments on
> this...
> Also, what about the FTP part, it would be nice if OFBiz handled that
> directly, anyone have any suggestioins?
> Finally if FTP is integrated, we need somewhere to store the FTP address and
> credentials - any suggestions?
> Comment by Si Chen [18/May/06 02:22 PM] [ Permlink ]
> Hey -
> 1. Why variants: if someone searches for "white shirt" on froogle and all you
> had was the virtual "shirt" but not variants, you might not get listed.
> 2. No idea about the filter category.
> 3. Recurse - yeah probably a good idea too.
> 4. I still think the content template + render is better than DataFile, but I
> could be convinced otherwise.
> 5. I think there are some FtpServices.java in ofbiz now. Not sure what it
> does though.
> Comment by Andrew Sykes [18/May/06 02:43 PM] [ Permlink ]
> Si,
> Sure that could be an issue, but would you really want to send
> 5 colours x 5 neck sizes x 5 chest sizes x 10 braiding colours
> = 1250 products
> (ok, a bit excessive, but you get the point I'm sure)
> I'm guessing froogle would probably take quite a dim view of this especially
> if they all had the same product image and description.
> Also, having done a few clothing sites, they never have products this
> generic, because...
> A/ They'd never get a google rank
> B/ The retailer would always find a more descriptive product name.
> "White Shirt" is more likely to be a category name.
> I've been trying to think of a scenario where variants would be useful, but I
> just don't think product names would ever be generic enough to need this.
> Comment by Si Chen [18/May/06 02:47 PM] [ Permlink ]
> That's why it's a configurable parameter, so if you don't want it, you don't
> have to use it. It's OK if you don't put it in--I can add it later if we
> decide to use it.
> Also: FTP address and credentials - use a .properties file in a config/
> directory. That's how the freight shipping and payment processors are
> handled.
> Comment by Si Chen [18/May/06 03:02 PM] [ Permlink ]
> Rohit,
> Thanks for your comments. I actually do know of oscommerce but wasn't quite
> sure how it worked. Thanks for explaining it.
> Well, we are definitely a community project as well and do indeed accept
> contributions from a fairly large number of people. I think the difference
> you find is not only a greater set of features but also a more formal
> framework and process for building applications. Hopefully this will work for
> you. All this stuff may seem intimidating at first, but you can definitely
> learn it (if I can learn it, anybody can learn it), and once you know it, it
> will make your work very easy.
> Ok, end of my digressions.
> Si
> Comment by Ray Barlow [21/May/06 06:03 AM] [ Permlink ]
> DataFile.patch adds support to write a DataFile with a header row.
> Comment by Ray Barlow [21/May/06 06:10 AM] [ Permlink ]
> I have added a menu on the catalog application called "DataIO" next to
> "Configurations" and in there is a new screen for Froogle.
> The screen contains comments about what is still required to be done which to
> be honest still includes a fair amount to be a really cool implementation,
> but this submission will at least generate the Froogle file. I've only run on
> the demo data so far but will deploy it on one of my live sites asap and test
> generating and submitting to Froogle for real.
> Although not complete if this is acceptable then it would be nice to commit a
> version sooner rather than wait for the holy grail of FTP automation etc.
> Comment by rohit [21/May/06 10:28 PM] [ Permlink ]
> Hi,
> Can you please give some instructions on how to install it. I will surely
> test it and give my feedback.
> rohit
> Comment by Ray Barlow [22/May/06 06:59 AM] [ Permlink ]
> The two files I attached are created as SVN patch files. Depending on your
> platform and tool depends on how you apply them and to be honest applying
> patches is not something I've done a lot of. For Windows if your using
> tortoise SVN (which is very good) try right click on the root folder and
> "Apply Patch" from the sub menu. For the command line I think you'd use the
> "patch" command, but for both I'd suggest a test folder and google would be a
> good option.
> On a more general note I forgot to mention that I saw the "DataIO" section
> being an area for control of other features being talked about like Yahoo,
> Amazon and maybe even some simple catalogue data import/export routines.
> Comment by rohit [24/May/06 09:23 AM] [ Permlink ]
> hi Ray,
> i am sorry but i could not get it working. i was initially using
> opentaps-0.9.0 but then tried it on SVN using the tortoise SVN client.
> i will really appreciate if you could be more specific about how you went
> about installing. For. e.g. in which folders are these files to be copied
> into and if there is any code that needs to be edited manually.
> i am new to ofbiz, hence a little more detailed instructions will help a lot.
> thank,
> rohit
> Comment by rohit [24/May/06 11:43 PM] [ Permlink ]
> hi Ray,
> i was able to get the files install, but i am getting the following error,
> when i try to use the froogle exporter:
> 3057047[ServiceEventHandler.java:317:ERROR]
> ---- exception report
> ----------------------------------------------------------
> Service invocation error
> Exception: org.ofbiz.service.GenericServiceException
> Message: Service target threw an unexpected exception (null)
> ---- stack trace
> ---------------------------------------------------------------
> org.ofbiz.service.GenericServiceException: Service target threw an unexpected
> exception (null)
> org.ofbiz.service.engine.StandardJavaEngine.serviceInvoker(StandardJavaEngine.java:113)
>
> org.ofbiz.service.engine.StandardJavaEngine.runSync(StandardJavaEngine.java:63)
>
> org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:340)
> org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:214)
> org.ofbiz.service.GenericDispatcher.runSync(GenericDispatcher.java:123)
> org.ofbiz.webapp.event.ServiceEventHandler.invoke(ServiceEventHandler.java:301)
>
> org.ofbiz.webapp.control.RequestHandler.runEvent(RequestHandler.java:416)
> org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:252)
> org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:191)
> org.ofbiz.webapp.control.ControlServlet.doPost(ControlServlet.java:87)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:615)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:688)
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
>
> org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:258)
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
>
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:526)
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
>
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
>
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
>
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
>
> java.lang.Thread.run(Thread.java:534)
> java.lang.NullPointerException
> org.ofbiz.product.product.ProductExportServices.productsToCsv(ProductExportServices.java:73)
>
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> java.lang.reflect.Method.invoke(Method.java:324)
> org.ofbiz.service.engine.StandardJavaEngine.serviceInvoker(StandardJavaEngine.java:101)
>
> org.ofbiz.service.engine.StandardJavaEngine.runSync(StandardJavaEngine.java:63)
>
> org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:340)
> org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:214)
> org.ofbiz.service.GenericDispatcher.runSync(GenericDispatcher.java:123)
> org.ofbiz.webapp.event.ServiceEventHandler.invoke(ServiceEventHandler.java:301)
>
> org.ofbiz.webapp.control.RequestHandler.runEvent(RequestHandler.java:416)
> org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:252)
> org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:191)
> org.ofbiz.webapp.control.ControlServlet.doPost(ControlServlet.java:87)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:615)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:688)
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
>
> org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:258)
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
>
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:526)
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
>
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
>
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
>
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
>
> java.lang.Thread.run(Thread.java:534)
> --------------------------------------------------------------------------------
>
> 3057047[ RequestHandler.java:283:ERROR] Request froogleExport caused an error
> with the following message: Error calling event:
> org.ofbiz.webapp.event.EventHandlerException: Service invocation error (null)
> 3057047[ RequestHandler.java:395:INFO ] [RequestHandler.doRequest]: Response
> is a view.
> 3057047[ RequestHandler.java:547:INFO ] servletName=control, view=Froogle
> 3057109[ UtilXml.java:242:DEBUG] XML Read 0.062s:
> file:/C:/ofbiz/applications/product/widget/catalog/DataIOScreens.xml
> 3057109[ ScreenFactory.java:120:INFO ] Got 2 screens in 0.062s from:
> file:/C:/ofbiz/applications/product/widget/catalog/DataIOScreens.xml
> 3057281[ UtilXml.java:242:DEBUG] XML Read 0.156s:
> file:/C:/ofbiz/applications/product/widget/catalog/CatalogCommonScreens.xml
> 3057281[ ScreenFactory.java:120:INFO ] Got 13 screens in 0.156s from:
> file:/C:/ofbiz/applications/product/widget/catalog/CatalogCommonScreens.xml
> 3057328[ UtilXml.java:242:DEBUG] XML Read 0.031s:
> file:/C:/ofbiz/framework/common/widget/CommonScreens.xml
> 3057344[ ScreenFactory.java:120:INFO ] Got 8 screens in 0.047s from:
> file:/C:/ofbiz/framework/common/widget/CommonScreens.xml
> 3057578[ UtilXml.java:242:DEBUG] XML Read 0.047s:
> file:/C:/ofbiz/applications/product/widget/catalog/CommonScreens.xml
> 3057578[ ScreenFactory.java:120:INFO ] Got 9 screens in 0.047s from:
> file:/C:/ofbiz/applications/product/widget/catalog/CommonScreens.xml
> 3057750[ ServiceDispatcher.java:452:DEBUG] [[Sync service finished-
> total:0.0,since last(Begin):0.0]] - 'catalog /
> getProductCategoryAndLimitedMembers'
> 3057812[ ControlServlet.java:258:DEBUG] [[[froogleExport] Done rendering
> page, Servlet Finished- total:1.125,since last([froogleExport] S...):1.125]]
> 3061609[ ControlServlet.java:98 :INFO ] The character encoding of the request
> is: [null]. The character encoding we will use for the request and response
> is: [UTF-8]
> 3061625[ ControlServlet.java:134:DEBUG] [[[froogleExport] Servlet Starting,
> doing setup- total:0.0,since last(Begin):0.0]]
> 3061625[ ControlServlet.java:183:DEBUG] [[[froogleExport] Setup done, doing
> Event(s) and View(s)- total:0.0,since last([froogleExport] S...):0.0]]
> 3061625[ RequestHandler.java:212:INFO ] [Processing Request]: froogleExport
> 3061672[ ServiceDispatcher.java:386:DEBUG] [[Sync service failed...-
> total:0.0,since last(Begin):0.0]] - 'catalog / froogleExport'
> 3061687[ ServiceDispatcher.java:389:ERROR]
> Please advise what might be wrong.
> rohit
> Comment by Ray Barlow [05/Jun/06 12:17 PM] [ Permlink ]
> Attached is an improved version that has a little more error checking and
> also uses drop down lists to reduce user entry errors, for data such as
> webSiteId. The NPE exception Rohit was experiencing would have been due to an
> invalid webSiteId entry.
> Also added support for tracking codes if defined in the marketing module and
> a check to skip null images for products.
> It would be nice to see this reviewed and committed if possible. DataFile
> patch still stands as submitted before as I've not changed anything in it
> that for this update.
> Comment by rohit [21/Jul/06 01:58 AM] [ Permlink ]
> hi,
> i have attached a new patch file to work with nre requirement of google base
> format. This patch has been created on the latest SVN 424158.
> Comment by rohit [21/Jul/06 02:00 AM] [ Permlink ]
> This patch has all the changes. you need only this file to implement froogle
> on ofbiz.
> Comment by Marco Risaliti [13/Sep/06 04:12 PM] [ Permlink ]
> May I have to move it to the new jira server ?
> Thanks
> Marco
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
