Interesting. This is great information and gets us that much closer to being able to deterministically reproduce the error. I'm pretty sure I've seen this before myself, but it was a one off issue. I'll see if I can reproduce this.
On Thu, Nov 19, 2009 at 12:50 AM, Dave Cheong <d...@davecheong.com> wrote: > Hi Ikai, > > Thanks for the follow up. I can't say for sure, and the following is > my crystal ball guess of what's happening. > > The problem only occurred when I was developing my email webhook. It > has since stopped happening (I think). When developing my webhook, > often I had exceptions thrown as a result of not parsing the > Multiparts correctly. On the surface, it looks like when an exception > is thrown, the email is marked as undelivered and AppEngine retries to > send again later. > > This is further complicated when on some requests, I was processing a > plain text email as a MultiPart email, which resulted in > OutOfMemoryErrors. Due to the high CPU usage, AppEngine temporarily > shuts down my app. Not sure what happens next, but it would appear the > SMTP server retries whilst my app is down/suspended or otherwise > broken and ends up failing to deliver each time. Something gets out of > sync then, because it would build up a backlog of failed deliveries > and retries for each failed delivery! > > The other side effect was my app would never stay up - AppEngine would > restart it and then it would immediately get shut down again (due to > the high CPU usage from the OutOfMemoryErrors). > > Once I fixed my code, I noticed lots of repeated emails coming through > (and getting processed/saved to the datastore). I let it run for > several hours for it to clear the backlog of emails. It has since > stopped delivering these dups. > > Does this make sense? Plausible? > > Prior to using AppEngine's inbound mail funtionality, I was using > http://smtp2web.com which had a nice feature in that if an exception > is thrown when calling the webhook, it would send a bounce email to > the sender and stop trying. This was nice I thought because there was > user feedback that the mail didn't get through. > > Your thoughts? > > dave > > AppEngine may opt to suspend the app (eg due to high CPU usage). This > in turn could be signalling to the SMTP server that the mail delivery > has failed and signal it to try again later. However, what gets placed > in the queue may be a duplicate, so there might be a reference to two > copies of the same email that needs to be delivered. > > After a certain timeout, AppEngine restarts the app, which fails to > restart properly because it immediately fails again due to the email > resend. This is further complicated if a new version is deployed with > changes to the email webhook - so some requests would succeed and > others fail. > > > > On Nov 19, 8:35 am, "Ikai L (Google)" <ika...@google.com> wrote: > > Hi Dave, > > > > Have you had any luck reproducing this? If so, what did you do? > > > > > > > > > > > > On Tue, Nov 17, 2009 at 4:45 AM, Dave Cheong <d...@davecheong.com> > wrote: > > > Hi all, > > > > > I'm observing a situation where my email webhook (/_ah/mail/*) is > > > being invoked multiple times (seemingly randomly) for a given email. > > > Not sure the pattern, but I'm definitely seeing log messages with the > > > same email content multiple times, in no particular order or pattern. > > > > > I'm not sure if a red herring, but it does appear to be somewhat > > > related to the fact that I am deploying new versions and restarting > > > the app constantly, as I tweak my code which handles the inbound > > > email. > > > > > Has anyone observed this behaviour? Can someone in Google help > > > diagnose why my email webhook is being called repeatedly? > > > > > Any help is appreciated. > > > > > thanks, > > > dave > > > > > -- > > > > > You received this message because you are subscribed to the Google > Groups > > > "Google App Engine for Java" group. > > > To post to this group, send email to > > > google-appengine-j...@googlegroups.com. > > > To unsubscribe from this group, send email to > > > google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com><google-appengine-java%2B > unsubscr...@googlegroups.com> > > > . > > > For more options, visit this group at > > >http://groups.google.com/group/google-appengine-java?hl=. > > > > -- > > Ikai Lan > > Developer Programs Engineer, Google App Engine > > -- > > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to > google-appengine-j...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=. > > > -- Ikai Lan Developer Programs Engineer, Google App Engine -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=.
