On Tue, Mar 31, 2009 at 8:57 AM, Anirudh (Google) <[email protected]> wrote: > Before we analyze your issue, I would like to let you know that IMAP > migration and gdata library based migration ( EMAPI) are two > completely different interfaces to gmail.
Hi Anirudh, Thanks for the information. > From the data you have shared, I suspect this is a case of migrating > too fast. EMAPI is rate limited and the best strategy is to > exponentially back off when migration fails. HTTP Status Code 503 is > an indication that the migration failed. Ok, that's what I figured based on the other messages I have read. > To track the status of individual entries in a batch request, you can > check the <batch:status> element in the returned Feed for each entry: > http://code.google.com/apis/gdata/batch.html#Writing_Feed > > A '201' indicates a successful insert. Other possible status codes are > listed here: > http://code.google.com/apis/gdata/batch.html#Handling_Errors Ah, but there's the issue. ;) I found that api information before, but how am I supposed to get the status results with the Python modules? The pydoc information is, unfortunately, not helpful for this, there is no example code available for email migration, and the code.google.com site has no documentation for the Python modules. I spent the afternoon digging through the modules and trying to figure out what I could do. Let me throw out some code: import email import gdata.apps.migration.service gd_client = gdata.apps.migration.service.MigrationService( '[email protected]', 'password', 'example.com', 'None-Sample-1') gd_client.ProgrammaticLogin() mail = email.message_from_file(file('mbox')) gd_client.AddBatchEntry(mail.as_string(),[],[]) result = gd_client.SubmitBatch('username') While pydoc says result should be a HTTPResponse object, it's actually gdata.apps.migration.BatchMailEventFeed which is very misleading. There's no documented way to look at the batch entry results via BatchMailEventFeed, there are only methods to add new entries to the batch and a few URL-related methods. After a lot of digging I ended up figuring out where to find a list of BatchMailEntry objects as the response from the server: for e in result.entry: print str(e.batch_id) print e.batch_status.code print e.batch_status.reason So that's great, I can now see the return codes (201 versus 503), and each of the batch entry ids. Unfortunately, it turns out this information is rather useless. The problem is that MigrationService doesn't accept a batch_id when adding a message, it gets auto-generated through the AddBatchEntry call. BatchMailEntry objects here only have the status info as shown above (no original message, etc,) and so there is no way to relate the id/result back to the original message so it can be resubmitted. Sure, I can try to resend the whole set of messages by running them back through AddBatchEntry again, but that's very suboptimal and may not even be possible depending on how messages are being read in. To work around this I decided to be lazy and went with the "break the OO fourth wall" method: ... same code up until the SubmitBatch line ... batch = {} for entry in gd_client.mail_batch.entry: batch[str(entry.batch_id)] = entry while len(batch): result = gd_client.SubmitBatch('username') for entry in result.entry: status = entry.batch_status id = str(entry.batch_id) if status.code not in ('200', '201'): gd_client.mail_batch.AddBatchEntry(batch[id]) else: del batch[id] if len(batch): sleep(30) While this works, I think it's pretty horrible. My code has to reach into the object to manipulate mail_batch and then further the mail_batch.entry list object. Then it has to do it again after submitting the batch to create a new queue w/ the messages that didn't make it through. I'm probably going to end up writing my own version of MigrationService so that the code will still work if/when the gdata version is fixed/updated (please!). I can now figure out which messages didn't make it and resend them, and in testing this afternoon I was able to send in several thousand messages w/ no messages missing at the end. \o/ --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google Apps APIs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/google-apps-apis?hl=en -~----------~----~----~----~------~----~------~--~---
