Oh, yeah, that's right, there is the closing of the persistence manager.
All my JDO calls go through one function that has all the boilerplate, and
it ends with
} finally {
pm.close();
}
}
So we can safely assume that this is not the problem.
On Wed, Aug 5, 2015 at 1:36 PM, Jeff Schnitzer <j...@infohazard.org> wrote:
> JDO absolutely does have a session. You start a session by opening a
> persistencemanager and end by closing it. When you close the session, JDO
> does its dirty change detection and writes out any alterations. Also,
> repeated loads from an open session will return the same object. JPA works
> the same way.
>
> Objectify's session is very similar to this (minus the dirty change
> detection).
>
> If you aren't closing your PM sessions... this would explain the problem.
>
> Jeff
>
> On Wed, Aug 5, 2015 at 11:59 AM, Thomas Becker <walkswiththeb...@gmail.com
> > wrote:
>
>> I believe that session caching is specific to the Objectify interface.
>> Since I'm not using that, I don't think there even is a way for me to
>> open/close any sessions. I'm quite sure that there is no such a thing as a
>> "JDO session." (There is of course the HTTP session that's provided by the
>> Java servlet environment, but that's got nothing to do with the datastore.)
>>
>> Thomas
>>
>> On Wed, Aug 5, 2015 at 12:44 PM, Jeff Schnitzer <j...@infohazard.org>
>> wrote:
>>
>>> This sounds to me not like an issue with eventual consistency, but an
>>> issue with session caching. The thing about different *versions* is
>>> that it guarantees requests to go to different instances.
>>>
>>> I don't remember much about how JDO works, but are you sure you are
>>> opening/closing sessions correctly?
>>>
>>> Jeff
>>>
>>> On Tue, Aug 4, 2015 at 7:56 PM, Thomas Becker <
>>> walkswiththeb...@gmail.com> wrote:
>>>
>>>> Thanks again for eveybody's input. It looks like I have solved my
>>>> problem by falling back on the low-level datastore access when strong
>>>> consistency matters. Problems like this happen, and the bottom line is GAE
>>>> is a great product that's done a lot for me. So kudos to all members,
>>>> former and present, of the GAE team at Google.
>>>>
>>>> FWIW, here are two observations that could be helpful in understanding
>>>> the behavior of the gae/JDO function GetObjectById:
>>>>
>>>> 1) If you set the option "Unapplied job percentage" to 100% in the SDK,
>>>> that is, you force "eventually consistent" to mean "never consistent," then
>>>> GetObjectById still acts strongly consistent. This means that the SDK
>>>> thinks that GetObjectById should act strongly consistent, just like the low
>>>> level get function.
>>>>
>>>> 2) Demonstrating that GetObjectById can return stale data in the
>>>> production environment is difficult because the behavior is, by its very
>>>> nature, intermittent. But one can increase the probability of seeing stale
>>>> data vastly if one performs the write and subsequent read from two
>>>> different *versions* of the app. As others have observed, the
>>>> probability already increases when write and read are performed from
>>>> different instances of the app. But with different *versions *(one of
>>>> them being the default version), it's been happening pretty much every time
>>>> in my context. The low-level get, by contrast, has always acted strongly
>>>> consistent for me, even across versions.
>>>>
>>>> Thomas
>>>>
>>>> On Tue, Aug 4, 2015 at 12:07 AM, Jeff Schnitzer <j...@infohazard.org>
>>>> wrote:
>>>>
>>>>> Again, JDO is not my area of expertise, but if so, this seems like a
>>>>> shockingly obvious issue. Can you post a sample of the code you use to
>>>>> demonstrate the problem?
>>>>>
>>>>> Jeff
>>>>>
>>>>> On Mon, Aug 3, 2015 at 8:11 PM, Prashant <antsh...@gmail.com> wrote:
>>>>>
>>>>>> I am facings exact same issue. getObjectById returns stale data even
>>>>>> if call is made inside a transaction.
>>>>>>
>>>>>> I performed an experiment - I ran two crons in two different modules,
>>>>>> both trying to increment their execution count in an (common, shared)
>>>>>> entity. I was shocked to see that even when I transaction, both crons
>>>>>> are
>>>>>> running as if they are performing transaction in thier own copy of
>>>>>> DataStore. Evertime, each cron will get copy of entity last updated by
>>>>>> itself (ignoring updates made by other cron) and on top of that
>>>>>> transaction
>>>>>> will also commit without any exception.
>>>>>>
>>>>>> --
>>>>>> Prashant
>>>>>>
>>>>>> On 8:24AM, Tue, 4 Aug 2015 Thomas Becker <walkswiththeb...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks, Jeff! And one more thing, it's pretty clear from the doc,
>>>>>>> but let me make sure anyway. The same is true for ancestor queries, they
>>>>>>> guarantee strong consistency even when not performed inside a
>>>>>>> transaction,
>>>>>>> right?
>>>>>>>
>>>>>>> Thomas
>>>>>>>
>>>>>>> On Mon, Aug 3, 2015 at 8:48 PM, Jeff Schnitzer <j...@infohazard.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yes, that is correct. By default, if you perform a low-level
>>>>>>>> get-by-key operation, it is strongly consistent whether or not you put
>>>>>>>> it
>>>>>>>> in a transaction.
>>>>>>>>
>>>>>>>> Note that you can, if you want, explicitly request an eventually
>>>>>>>> consistent get-by-key operation in the low-level api. It does return
>>>>>>>> faster
>>>>>>>> since it does not require a quorum response. However, this is a pretty
>>>>>>>> obscure feature.
>>>>>>>>
>>>>>>>> Jeff
>>>>>>>>
>>>>>>>> On Mon, Aug 3, 2015 at 7:30 PM, Thomas Becker <
>>>>>>>> walkswiththeb...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Thanks for your detailed and thoughtful answer, Jeff. I was aware
>>>>>>>>> of the fact that within a transaction, all reads will return the
>>>>>>>>> original
>>>>>>>>> value even if one writes it within the transaction. That was not the
>>>>>>>>> problem, but thanks for pointing it out anyway, it's easy to forget!
>>>>>>>>>
>>>>>>>>> Other than that, after reading everything on the subject of GAE
>>>>>>>>> datastore consistency under JDO, I have decided to fall back to
>>>>>>>>> low-level
>>>>>>>>> access in those cases where strong consistency is of the essence. At
>>>>>>>>> the
>>>>>>>>> risk of sounding acerbic or snarky, it seems clear to me that nobody
>>>>>>>>> knows
>>>>>>>>> what's going on in JDO when it comes to consistency.
>>>>>>>>>
>>>>>>>>> So just to confirm, it is true that the low-level get-by-key
>>>>>>>>> guarantees strong consistency even *when performed outside of a
>>>>>>>>> transaction*, correct?
>>>>>>>>>
>>>>>>>>> Thanks again for your thoughtful answer. Very refreshing indeed.
>>>>>>>>>
>>>>>>>>> Thomas
>>>>>>>>>
>>>>>>>>> On Mon, Aug 3, 2015 at 8:13 PM, Jeff Schnitzer <
>>>>>>>>> j...@infohazard.org> wrote:
>>>>>>>>>
>>>>>>>>>> I'm no expert on JDO, but the low-level api does guarantee strong
>>>>>>>>>> consistency for get-by-key operations (that is, unless you
>>>>>>>>>> explicitly ask
>>>>>>>>>> for eventual consistency). One of those SO responses suggested it is
>>>>>>>>>> possible to have the GAE/JDO plugin log its low-level operations -
>>>>>>>>>> that
>>>>>>>>>> should at least point you in the right direction.
>>>>>>>>>>
>>>>>>>>>> One other thing that might be related: In the low-level API,
>>>>>>>>>> within a single transaction, all reads will return the original
>>>>>>>>>> value even
>>>>>>>>>> if you write it within the transaction. I don't know what JDO does
>>>>>>>>>> with
>>>>>>>>>> this; I would expect that subsequent reads would return a
>>>>>>>>>> session-cached
>>>>>>>>>> instance and hide this little quirk. But maybe it's related to what
>>>>>>>>>> you are
>>>>>>>>>> seeing?
>>>>>>>>>>
>>>>>>>>>> Jeff
>>>>>>>>>>
>>>>>>>>>> On Mon, Aug 3, 2015 at 1:52 PM, Thomas Becker <
>>>>>>>>>> walkswiththeb...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks for your quick reply, Ryan. I have found several postings
>>>>>>>>>>> on the Web that describe the same problem, namely, GetObjectById
>>>>>>>>>>> returning
>>>>>>>>>>> stale data. Here are two from StackOverflow:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> http://stackoverflow.com/questions/26377347/inconsistent-fetch-from-google-app-engine-datastore
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> http://stackoverflow.com/questions/22005349/appengine-jdo-with-multiple-server-instances-loses-transaction-updates-on-entiti
>>>>>>>>>>>
>>>>>>>>>>> Under the circumstances, I don't think it would be a good use of
>>>>>>>>>>> my time to make a demo app. It seems clear to me that the issue
>>>>>>>>>>> exists.
>>>>>>>>>>> Besides, the problem is so intermittent that a demo app is of very
>>>>>>>>>>> limited
>>>>>>>>>>> value. I would be perfectly happy with a statement of the kind,
>>>>>>>>>>> "GetObjectById() does not guarantee strong consistency, that's how
>>>>>>>>>>> it is,
>>>>>>>>>>> and here's what you do instead to get strongly consistent data." I
>>>>>>>>>>> just
>>>>>>>>>>> need to know what's going on.
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Aug 3, 2015 at 2:32 PM, Ryan (Cloud Platform Support) <
>>>>>>>>>>> rbruy...@google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> If you can create a sample app that shows this post on it on
>>>>>>>>>>>> the PIT
>>>>>>>>>>>> <https://code.google.com/p/googleappengine/issues/entry> so I
>>>>>>>>>>>> can investigate this further.
>>>>>>>>>>>>
>>>>>>>>>>>> On Saturday, August 1, 2015 at 6:24:55 PM UTC-4, Thomas Becker
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am using the app engine datastore with Java JDO. According
>>>>>>>>>>>>> to the documentation, ancestor queries are always strongly
>>>>>>>>>>>>> consistent.
>>>>>>>>>>>>> Recently, however, I've been seeing a lot of stale data from
>>>>>>>>>>>>> ancestor
>>>>>>>>>>>>> queries. It seems to happen randomly. A query may return the most
>>>>>>>>>>>>> recent
>>>>>>>>>>>>> data, and a moment later, the same query returns stale data.
>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> You received this message because you are subscribed to a topic
>>>>>>>>>>>> in the Google Groups "Google App Engine" group.
>>>>>>>>>>>> To unsubscribe from this topic, visit
>>>>>>>>>>>> https://groups.google.com/d/topic/google-appengine/1re1FtqllnQ/unsubscribe
>>>>>>>>>>>> .
>>>>>>>>>>>> To unsubscribe from this group and all its topics, send an
>>>>>>>>>>>> email to google-appengine+unsubscr...@googlegroups.com.
>>>>>>>>>>>> To post to this group, send email to
>>>>>>>>>>>> google-appengine@googlegroups.com.
>>>>>>>>>>>> Visit this group at
>>>>>>>>>>>> http://groups.google.com/group/google-appengine.
>>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>>> https://groups.google.com/d/msgid/google-appengine/9723734e-0b14-49c0-8c31-3407df8454ad%40googlegroups.com
>>>>>>>>>>>> <https://groups.google.com/d/msgid/google-appengine/9723734e-0b14-49c0-8c31-3407df8454ad%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>>>>>> .
>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>> Google Groups "Google App Engine" group.
>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from
>>>>>>>>>>> it, send an email to
>>>>>>>>>>> google-appengine+unsubscr...@googlegroups.com.
>>>>>>>>>>> To post to this group, send email to
>>>>>>>>>>> google-appengine@googlegroups.com.
>>>>>>>>>>> Visit this group at
>>>>>>>>>>> http://groups.google.com/group/google-appengine.
>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>> https://groups.google.com/d/msgid/google-appengine/CAPxj-GA3OKiGfjwgYgKmvZ%3DSi7ZXG3VLvts7%2BjWZ7c2CsC693Q%40mail.gmail.com
>>>>>>>>>>> <https://groups.google.com/d/msgid/google-appengine/CAPxj-GA3OKiGfjwgYgKmvZ%3DSi7ZXG3VLvts7%2BjWZ7c2CsC693Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>>>>>> .
>>>>>>>>>>>
>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> You received this message because you are subscribed to a topic
>>>>>>>>>> in the Google Groups "Google App Engine" group.
>>>>>>>>>> To unsubscribe from this topic, visit
>>>>>>>>>> https://groups.google.com/d/topic/google-appengine/1re1FtqllnQ/unsubscribe
>>>>>>>>>> .
>>>>>>>>>> To unsubscribe from this group and all its topics, send an email
>>>>>>>>>> to google-appengine+unsubscr...@googlegroups.com.
>>>>>>>>>> To post to this group, send email to
>>>>>>>>>> google-appengine@googlegroups.com.
>>>>>>>>>> Visit this group at
>>>>>>>>>> http://groups.google.com/group/google-appengine.
>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>> https://groups.google.com/d/msgid/google-appengine/CADK-0uh4Q8kswYMhD5RcxeQJnO3rFshgkfGEbB%2BXzHVXQLbGiw%40mail.gmail.com
>>>>>>>>>> <https://groups.google.com/d/msgid/google-appengine/CADK-0uh4Q8kswYMhD5RcxeQJnO3rFshgkfGEbB%2BXzHVXQLbGiw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>>>>> .
>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>> Groups "Google App Engine" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>> send an email to google-appengine+unsubscr...@googlegroups.com.
>>>>>>>>> To post to this group, send email to
>>>>>>>>> google-appengine@googlegroups.com.
>>>>>>>>> Visit this group at
>>>>>>>>> http://groups.google.com/group/google-appengine.
>>>>>>>>> To view this discussion on the web visit
>>>>>>>>> https://groups.google.com/d/msgid/google-appengine/CAPxj-GApGFORBdvJd88QvitX_bHLGvAtiQ943WBd0V3Ro9onZw%40mail.gmail.com
>>>>>>>>> <https://groups.google.com/d/msgid/google-appengine/CAPxj-GApGFORBdvJd88QvitX_bHLGvAtiQ943WBd0V3Ro9onZw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>>>> .
>>>>>>>>>
>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to a topic in
>>>>>>>> the Google Groups "Google App Engine" group.
>>>>>>>> To unsubscribe from this topic, visit
>>>>>>>> https://groups.google.com/d/topic/google-appengine/1re1FtqllnQ/unsubscribe
>>>>>>>> .
>>>>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>>>>> google-appengine+unsubscr...@googlegroups.com.
>>>>>>>> To post to this group, send email to
>>>>>>>> google-appengine@googlegroups.com.
>>>>>>>> Visit this group at http://groups.google.com/group/google-appengine
>>>>>>>> .
>>>>>>>>
>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/google-appengine/CADK-0ujMaP_xRg6Pjht%3DsCR%3Dc-3hGEJvhXGtOT0h1vcMKhQBqQ%40mail.gmail.com
>>>>>>>> <https://groups.google.com/d/msgid/google-appengine/CADK-0ujMaP_xRg6Pjht%3DsCR%3Dc-3hGEJvhXGtOT0h1vcMKhQBqQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>>> .
>>>>>>>>
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "Google App Engine" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to google-appengine+unsubscr...@googlegroups.com.
>>>>>>> To post to this group, send email to
>>>>>>> google-appengine@googlegroups.com.
>>>>>>> Visit this group at http://groups.google.com/group/google-appengine.
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/google-appengine/CAPxj-GBwjD%2BGO9LDk5ObgTQo7w5%2B7wnN32%3D664CrVg%2BPxD_6nw%40mail.gmail.com
>>>>>>> <https://groups.google.com/d/msgid/google-appengine/CAPxj-GBwjD%2BGO9LDk5ObgTQo7w5%2B7wnN32%3D664CrVg%2BPxD_6nw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>> .
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Google App Engine" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to google-appengine+unsubscr...@googlegroups.com.
>>>>>> To post to this group, send email to
>>>>>> google-appengine@googlegroups.com.
>>>>>> Visit this group at http://groups.google.com/group/google-appengine.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/google-appengine/CAPpOf-t%3Dzp2uRUnkN9446_QoSXN6XCKdzNAe1g4GwULN8gg93A%40mail.gmail.com
>>>>>> <https://groups.google.com/d/msgid/google-appengine/CAPpOf-t%3Dzp2uRUnkN9446_QoSXN6XCKdzNAe1g4GwULN8gg93A%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to a topic in the
>>>>> Google Groups "Google App Engine" group.
>>>>> To unsubscribe from this topic, visit
>>>>> https://groups.google.com/d/topic/google-appengine/1re1FtqllnQ/unsubscribe
>>>>> .
>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>> google-appengine+unsubscr...@googlegroups.com.
>>>>> To post to this group, send email to google-appengine@googlegroups.com
>>>>> .
>>>>> Visit this group at http://groups.google.com/group/google-appengine.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/google-appengine/CADK-0uhSRuiH6a3Wn%3DAYYm2DF5GjvQnT1FCVAnjBkga6MF6pew%40mail.gmail.com
>>>>> <https://groups.google.com/d/msgid/google-appengine/CADK-0uhSRuiH6a3Wn%3DAYYm2DF5GjvQnT1FCVAnjBkga6MF6pew%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Google App Engine" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to google-appengine+unsubscr...@googlegroups.com.
>>>> To post to this group, send email to google-appengine@googlegroups.com.
>>>> Visit this group at http://groups.google.com/group/google-appengine.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/google-appengine/CAPxj-GBWY%2BQXoN%2BS3sX4YbO%2BuhZjd8svpUMnjwzTahzOFd-%3Dyw%40mail.gmail.com
>>>> <https://groups.google.com/d/msgid/google-appengine/CAPxj-GBWY%2BQXoN%2BS3sX4YbO%2BuhZjd8svpUMnjwzTahzOFd-%3Dyw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>> --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "Google App Engine" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/google-appengine/1re1FtqllnQ/unsubscribe
>>> .
>>> To unsubscribe from this group and all its topics, send an email to
>>> google-appengine+unsubscr...@googlegroups.com.
>>> To post to this group, send email to google-appengine@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/google-appengine.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/google-appengine/CADK-0uimyzpW0N5Rhv_OtQ05hrvfKxnKotnv0C_N%3D024SunE7g%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/google-appengine/CADK-0uimyzpW0N5Rhv_OtQ05hrvfKxnKotnv0C_N%3D024SunE7g%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to google-appengine+unsubscr...@googlegroups.com.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> Visit this group at http://groups.google.com/group/google-appengine.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/google-appengine/CAPxj-GCTgAZ-rCchRCD-ZWaaj%3DTbZJaz9GhBHond3Acw0ZutMA%40mail.gmail.com
>> <https://groups.google.com/d/msgid/google-appengine/CAPxj-GCTgAZ-rCchRCD-ZWaaj%3DTbZJaz9GhBHond3Acw0ZutMA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Google App Engine" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/google-appengine/1re1FtqllnQ/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/CADK-0uhKzJLtTuAS6h812cY9tsg%3DTPnfgXDJZw%2BB6mG-rabb1Q%40mail.gmail.com
> <https://groups.google.com/d/msgid/google-appengine/CADK-0uhKzJLtTuAS6h812cY9tsg%3DTPnfgXDJZw%2BB6mG-rabb1Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit
https://groups.google.com/d/msgid/google-appengine/CAPxj-GAq7hauffHHK8xrCbS2npiidS88SQXLGedjix3ykprY3w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
