Timing and load is critical. This set-up is used to keep 500,000 or so items up to date of which there are roughly 1000 item movements a day (update/delete/addition) after that the items are simply touched once in thirty days to update the expiration date.
Deletions are handled in bulk because all this is require is the google base itemID, updates are then handled then it continues on updating items to extent the expiration dates. So basically a rolling process, remembering though that static items can only be resubmitted once in a thirty day period. xml is saved to check size and keep record then deleted once submitted sucessfully, if theres a problem its stored for problem solving purposes if theres a high number of failures it takes required action to alert and halts processing until resolved. That said each action runs separately (deletes additions etc...) theres also a clean up process that runs in background and checks items against the Google Base database for problem. On Dec 15, 5:01 pm, icebackhaz <[email protected]> wrote: > Let me see if I'm following along. > > 1. You save the xml for every transmision? At what point do you hit > write-the-file? (And delete the file?) > > 2. You put only one item in a submission? The communication overhead > is of no concern to you? How thick is your pipe! :) > > On Dec 14, 5:44 pm, Tom Wilson <[email protected]> wrote: > > > Writing the xml message to a temp file check it size and then from > > there decide to send it was the default precaution i took as i do with > > other projects that fire messages between servers. Then you have a > > handy reference of failed messages. > > > I stick to an item a submital, and its never failed me yet. Even with > > over 500,000 items if its spread well and the system that drive it is > > efficient its always been more than enough. > > > On Dec 12, 4:33 pm, icebackhaz <[email protected]> wrote: > > > > Good question. Yes it is clear (and provable :) ) that one may only > > > put 1Mb in the xml payload, but it is not at all clear to me how one > > > either a) checks how full the payload is and b) detects after the fact > > > that the problem was an overly large payload. > > > > We have a lot to send and want to do it efficiently and correctly. > > > Not checking before hand means we would need a the issues (742, 921) > > > taken care of or we will not know the exact problem. If I could check > > > before hand I would never encounter overhead of the double failure > > > (921). > > > > Alternatives abound but don't really appeal. Continuously calculate > > > the xml based on known overhead and data lengths: fraught with > > > miscalculation errors and seriously exposed google api/xml changes. > > > Send ultra-conservative batch sizes (30 items might be the upper limit > > > of perfectly full (1000 char) values if I've done the arithmetic > > > correctly): seriously under uses the payload and increases the number > > > of submissions, traffic overhead. Send less conservatively (perhaps > > > aggressively) sized batches and on failure assume the payload was too > > > large and split it (recursively): many possible other reasons for > > > failure. > > > > So yes, I'm trying to solidify our end and both these, um, ah features > > > of the API get in the way. Btw, what we've decided to do on failure > > > is to generate the xml with on our own Writer and test the size of the > > > generated xml and react accordingly. Seems a reasonable compromise, > > > no? > > > > Keep in mind, I'm not sure 921 will be accepted as a bug. I do > > > believe the java.io.IOException is more the result of mis- > > > communication (using a closed connection) than anything else. Pretty > > > sure they thought they would be sending back a ServiceException. > > > InvalidEntryException really, and that's also wrong. HTTP_BAD_REQUEST > > > is the http error, and in this case it the result of payload > 1 Mb, > > > not that a particular entry is malformed. > > > > On Dec 11, 5:37 pm, Tom Wilson <[email protected]> wrote: > > > > > Can i ask why exactly your looking at this ? > > > > > The documentation states that it accepts batch requests up to 1MB so > > > > why are you checking the size before posting ? > > > > > There only so much the API will do for you but building a solid system/ > > > > app relies on checks on both ends. > > > > > Tom Wilson > > > > Freelance Google Base Developer and Consultantwww.tomthedeveloper.com > > > > > Google Base Tools -http://dev.tomthedeveloper.com/googlebase > > > > Featured Project > > > > :http://google-code-featured.blogspot.com/2008/02/google-base-competit... > > > > > On Dec 11, 11:28 pm, icebackhaz <[email protected]> wrote: > > > > > > Trying to see what happens when one stuffs too much into the xml > > > > > payload, we have discovered that there is no facility for detecting > > > > > the exact problem either before or after the send. > > > > > > The repsonse from the server is HttpURLConnection.HTTP_BAD_REQUEST. > > > > > It looks like the intension was to generate a InvalidEntryException > > > > > and that looks bogus too, but at least the getReason() might be useful > > > > > > Unfortunately the built-in second attempt to send the payload chokes > > > > > big time and we get a java.io.IOException from deep in Sun's code and > > > > > the 401 floats off into the either. > > > > > > I've sent the to the bugs forum, but not sure it will be recognized as > > > > > a bug:http://code.google.com/p/gdata-issues/issues/detail?id=921 > > > > > > This is further compounded by the fact that even if we hadn't been > > > > > blown out of the water there is no access to the httpRequest which has > > > > > the all important header: "Content-Length" (This has been recognized > > > > > as a > > > > > bug:http://code.google.com/p/gdata-issues/issues/detail?id=742&sort=-id&c...) --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google Base Data API" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/Google-Base-data-API?hl=en -~----------~----~----~----~------~----~------~--~---
