Do you grab the xml after the send? i.e. from the httprequest or do you write it out yourself.
Hate to belabour the point: Are you only using batch() for deletes? The rest are singletons (1000 movements + ~500000/30 resends per day)? On Dec 15, 3:19 pm, Tom Wilson <[email protected]> wrote: > Timing and load is critical. This set-up is used to keep 500,000 or so > items up to date of which there are roughly 1000 item movements a day > (update/delete/addition) after that the items are simply touched once > in thirty days to update the expiration date. > > Deletions are handled in bulk because all this is require is the > google base itemID, updates are then handled then it continues on > updating items to extent the expiration dates. So basically a rolling > process, remembering though that static items can only be resubmitted > once in a thirty day period. > > xml is saved to check size and keep record then deleted once submitted > sucessfully, if theres a problem its stored for problem solving > purposes if theres a high number of failures it takes required action > to alert and halts processing until resolved. That said each action > runs separately (deletes additions etc...) theres also a clean up > process that runs in background and checks items against the Google > Base database for problem. > > On Dec 15, 5:01 pm, icebackhaz <[email protected]> wrote: > > > Let me see if I'm following along. > > > 1. You save the xml for every transmision? At what point do you hit > > write-the-file? (And delete the file?) > > > 2. You put only one item in a submission? The communication overhead > > is of no concern to you? How thick is your pipe! :) > > > On Dec 14, 5:44 pm, Tom Wilson <[email protected]> wrote: > > > > Writing the xml message to a temp file check it size and then from > > > there decide to send it was the default precaution i took as i do with > > > other projects that fire messages between servers. Then you have a > > > handy reference of failed messages. > > > > I stick to an item a submital, and its never failed me yet. Even with > > > over 500,000 items if its spread well and the system that drive it is > > > efficient its always been more than enough. > > > > On Dec 12, 4:33 pm, icebackhaz <[email protected]> wrote: > > > > > Good question. Yes it is clear (and provable :) ) that one may only > > > > put 1Mb in the xml payload, but it is not at all clear to me how one > > > > either a) checks how full the payload is and b) detects after the fact > > > > that the problem was an overly large payload. > > > > > We have a lot to send and want to do it efficiently and correctly. > > > > Not checking before hand means we would need a the issues (742, 921) > > > > taken care of or we will not know the exact problem. If I could check > > > > before hand I would never encounter overhead of the double failure > > > > (921). > > > > > Alternatives abound but don't really appeal. Continuously calculate > > > > the xml based on known overhead and data lengths: fraught with > > > > miscalculation errors and seriously exposed google api/xml changes. > > > > Send ultra-conservative batch sizes (30 items might be the upper limit > > > > of perfectly full (1000 char) values if I've done the arithmetic > > > > correctly): seriously under uses the payload and increases the number > > > > of submissions, traffic overhead. Send less conservatively (perhaps > > > > aggressively) sized batches and on failure assume the payload was too > > > > large and split it (recursively): many possible other reasons for > > > > failure. > > > > > So yes, I'm trying to solidify our end and both these, um, ah features > > > > of the API get in the way. Btw, what we've decided to do on failure > > > > is to generate the xml with on our own Writer and test the size of the > > > > generated xml and react accordingly. Seems a reasonable compromise, > > > > no? > > > > > Keep in mind, I'm not sure 921 will be accepted as a bug. I do > > > > believe the java.io.IOException is more the result of mis- > > > > communication (using a closed connection) than anything else. Pretty > > > > sure they thought they would be sending back a ServiceException. > > > > InvalidEntryException really, and that's also wrong. HTTP_BAD_REQUEST > > > > is the http error, and in this case it the result of payload > 1 Mb, > > > > not that a particular entry is malformed. > > > > > On Dec 11, 5:37 pm, Tom Wilson <[email protected]> wrote: > > > > > > Can i ask why exactly your looking at this ? > > > > > > The documentation states that it accepts batch requests up to 1MB so > > > > > why are you checking the size before posting ? > > > > > > There only so much the API will do for you but building a solid > > > > > system/ > > > > > app relies on checks on both ends. > > > > > > Tom Wilson > > > > > Freelance Google Base Developer and Consultantwww.tomthedeveloper.com > > > > > > Google Base Tools -http://dev.tomthedeveloper.com/googlebase > > > > > Featured Project > > > > > :http://google-code-featured.blogspot.com/2008/02/google-base-competit... > > > > > > On Dec 11, 11:28 pm, icebackhaz <[email protected]> wrote: > > > > > > > Trying to see what happens when one stuffs too much into the xml > > > > > > payload, we have discovered that there is no facility for detecting > > > > > > the exact problem either before or after the send. > > > > > > > The repsonse from the server is HttpURLConnection.HTTP_BAD_REQUEST. > > > > > > It looks like the intension was to generate a InvalidEntryException > > > > > > and that looks bogus too, but at least the getReason() might be > > > > > > useful > > > > > > > Unfortunately the built-in second attempt to send the payload chokes > > > > > > big time and we get a java.io.IOException from deep in Sun's code > > > > > > and > > > > > > the 401 floats off into the either. > > > > > > > I've sent the to the bugs forum, but not sure it will be recognized > > > > > > as > > > > > > a bug:http://code.google.com/p/gdata-issues/issues/detail?id=921 > > > > > > > This is further compounded by the fact that even if we hadn't been > > > > > > blown out of the water there is no access to the httpRequest which > > > > > > has > > > > > > the all important header: "Content-Length" (This has been > > > > > > recognized > > > > > > as a > > > > > > bug:http://code.google.com/p/gdata-issues/issues/detail?id=742&sort=-id&c...) --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google Base Data API" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/Google-Base-data-API?hl=en -~----------~----~----~----~------~----~------~--~---
