Re: [google-appengine] Question about datastore cost optimization by reusing old entities for new data
On Thu, Sep 15, 2011 at 9:31 PM, Rishi Arora wrote: > In my app I have a an Entity group that stores user activity data - like a > high level application log of user activity. This is the Entity Group that > I expect to grow the fastest over time, and I have decided to keep only the > last 3 months of "user activity" data to avoid indefinite datastore growth. > Regarding how to keep this "3 month moving window", the obvious answer was > - run a cron job daily that deletes entities older than 3 months, with some > upper bound on number of deletes per day. However, a better solution comes > to mind, and I'm wondering if someone can comment on whether it is indeed > better: > > When I need to create a new User Activity entity, I should first search for > an entity that is 3+ months old. If I find one, I'll "overwrite" it with > the new user activity. This costs me one read and one write operations (and > two more writes because there are two indexes associates with this entity > kind). That's a total of 1 read and 3 writes with the new solution. In my > current solution, I only have 3 writes and no reads, but 3 months later this > entity will have to be deleted, which will incur an additional read and 3 > writes (one for the entity itself and 2 for the two indexes). So, my theory > is that my current solution has a total of 1 read + 6 writes, and my new > solution saves me 3 writes. Anything wrong with this theory? Oh and also, > I save the CPU time too because I don't need to run a cron job every day. > It may be a minor downside, but if you do not run a cron job every day with the new approach, you may have some obsolete entities which have more than three months. Also, if you want to that all in an end-user thread (search for an old entity, override it and save), there is actually more to do. But of course you can delegate that job to a background task. Best, Daniel > Thanks in advance. > -Rishi > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] MAX_BLOB_FETCH_SIZE value
Hi there, I have been trying to use MAX_BLOB_FETCH_SIZE to determine maximal size of blobs that may be uploaded by the users. It is documented to be around 32 MB here [0] but the real value is actually less than one megabyte [1]. Could anyone explain the difference to me or it is an issue and we should create a new one for this? I can confirm that we do store larger files. Thank you, Daniel [0] https://developers.google.com/appengine/docs/python/blobstore/overview#Introducing_the_Blobstore [1] http://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/api/blobstore/blobstore.py#83 -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/HhmL7qj75S0J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: MAX_BLOB_FETCH_SIZE value
Hi Stuart, On Thu, Aug 30, 2012 at 1:07 AM, Stuart Langley wrote: > This is the maximum size that can be fetched in a single call using > BlobReader ( > https://developers.google.com/appengine/docs/python/blobstore/blobreaderclass) > - The blob size that can be uploaded by users is effectively unlimited. thanks for your reply. As I understand, if a file is larger than this value, more than one call is needed. Where does 32 MB value, which is mentioned in documentation [0], come from? [0] https://developers.google.com/appengine/docs/python/blobstore/overview Best, Daniel > > > On Thursday, 30 August 2012 05:05:56 UTC+10, Daniel Hans wrote: >> >> Hi there, >> >> I have been trying to use MAX_BLOB_FETCH_SIZE to determine maximal size >> of blobs that may be uploaded by the users. It is documented to be around >> 32 MB here [0] but the real value is actually less than one megabyte [1]. >> Could anyone explain the difference to me or it is an issue and we should >> create a new one for this? I can confirm that we do store larger files. >> >> Thank you, >> Daniel >> >> [0] https://developers.google.**com/appengine/docs/python/** >> blobstore/overview#**Introducing_the_Blobstore<https://developers.google.com/appengine/docs/python/blobstore/overview#Introducing_the_Blobstore> >> [1] http://code.google.com/p/**googleappengine/source/browse/** >> trunk/python/google/appengine/**api/blobstore/blobstore.py#83<http://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/api/blobstore/blobstore.py#83> >> > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/ZhIhVpD-lKQJ. > > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
