[google-appengine] What are the IP address ranges used by GAE for outgoing HTTP requests

2010-06-24 Thread Erwin Streur
We have developed a application as a Google App, which does call backs
into our corporate domain for sending updates to the backend systems
using HTTP. Of course the corporate network is firework protected and
we have set rules to allow trafic based on the IP address ranges which
were used then.

A couple of weeks ago traffic failed, which was caused by the HTTP
requests coming from a different IP address range. The range was added
also to the firewall rules. Last week it happened again.

So my questions are:
- What are now the IP address ranges used GAE for outgoing HTTP
transfic?
- Is there a webpage listing them and which is kept up to date?

Regards,

Erwin

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: Just verifying - App Engine is hosted on servers located in the USA right?

2010-09-30 Thread Erwin Streur
I agree. Rumors indicate that there are a couple in the Netherlands
and Belgium (Google appears to like water ;-), but there is hardly any
information about which ones are used for what.

I'm surprised by the statement of Ikai Lan (Google) about it and would
like more clarification on it. Does it mean that
a) (GAE) data might be stored in North America
b) (GAE) data is stored in North America as well (but actually it is
stored in all/most datacenter across the world)
c) (GAE) data is stored only in North America
?

On 27 sep, 13:25, Peter Ondruska  wrote:
> Well, this may not be truth always. Unfortunately Google does not
> disclose much about their data centres... One is being built in
> Finland.
>
> On Sep 27, 9:03 am, adams  wrote:
>
> > Just to clarify: does that mean, though, that it might be processed
> > outside of North America? And I assumme North America means USA and
> > Canada.
>
> > On Sep 24, 5:10 pm, "Ikai Lan (Google)" 
> > wrote:
>
> > > The data is stored in North American data centers.
>
> > > --
> > > Ikai Lan
> > > Developer Programs Engineer, Google App Engine
> > > Blogger:http://googleappengine.blogspot.com
> > > Reddit:http://www.reddit.com/r/appengine
> > > Twitter:http://twitter.com/app_engine
>
> > > On Fri, Sep 24, 2010 at 8:53 AM, John McLaughlin <
>
> > > johnmclaugh...@massanimation.com> wrote:
> > > > I don't think so.  I've been working with the opposite assumption.
> > > > When they say "running on Google's infrastructure" I'm assuming that
> > > > my application will be running worldwide with datastore information
> > > > replicated as needed to worldwide servers.  Perhaps Google AppEngine
> > > > for Business can make such guarantees when it launches.  I haven't
> > > > seen anything specific one way or another, but would love to know.
>
> > > > On Sep 23, 7:38 am, Spines  wrote:
> > > > > Just verifying - App Engine is hosted on servers located in the USA
> > > > > right?  And will stay that way?
>
> > > > > The reason I ask is because I've been reading up about "fair use" for
> > > > > using other people's images, and it seems that things are much clearer
> > > > > legally if your servers are located in the USA.
>
> > > > --
> > > > You received this message because you are subscribed to the Google 
> > > > Groups
> > > > "Google App Engine" group.
> > > > To post to this group, send email to google-appeng...@googlegroups.com.
> > > > To unsubscribe from this group, send email to
> > > > google-appengine+unsubscr...@googlegroups.com > > >  e...@googlegroups.com>
> > > > .
> > > > For more options, visit this group at
> > > >http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Startup performance icw Spring initialization

2010-11-04 Thread Erwin Streur
We have a Java application which uses Spring for dependency injection
and dispatching. The initialization of the WebApplicationContext and
the DispatcherServlet can take upto 20 seconds. When using a regular
webcontainer this is not a problem since initialization is done only
once and at container startup.

Unfortunately with GAE it becomes a problem. The application has a low
number of requests/min (=1 user request+ 10 system generated
requests / 2 minutes) and we are getting a lot of "This request caused
a new process to be started for your application, and thus caused your
application code to be loaded for the first time. This request may
thus take longer and use more CPU than a typical request for your
application."

The result is having slow performance/long response times when
processing user requests and very frequently HardDeadlineExceededError
and thus becoming a serious problem

I'm curious to know whether other developers have similar issues and
how they addressed them. I haven't made up my mind about the
resolution strategy for this (tweak Spring, remove Spring, add keep a
live pings....)

Regards,

Erwin Streur

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: how to work around this eror: "DeadlineExceededError"

2011-04-27 Thread Erwin Streur
To make sure that we are on the right track. You do get the deadline
ExceededException on the get statement and the work upto that
statement do not cost so much time (you have verified this by
profiling/ checking timestamps of log statement)

I primarily develop in Java, but what I understand from Python the
get() has an implied fetch limit of 1. You are performing a single
equality filter on an indexed property (GAE requirement) with no
ordering therefore no index iterating, no zigzag join and no large
data transfers are done. This should always be fast. Not as fast as
querying on primary key, but close anyway.

In case there are long running operations, like URL fetch on external
sites and large data transfer, performing the operation offline by
means of TaskQueue is good suggestion because then the deadline is not
30 sec, but 10 minutes. In case of large volume processing, you might
consider the MapReduce framework.

But again, your initial post doesn't indicate that to be the case and
the problem purely to be on the database get.
Does the problem happen always/often or was it just a fluke and was
there some temporary overall bad performance on the datastore? Is the
get part of a transaction? (it shouldn't matter because you are not
querying by ID, but just to be thorough;)

Regards,

Erwin

On 27 apr, 17:38, Noah McIlraith  wrote:
> Look into storing the user inside the key (key name), that way you will be
> able to very quickly grab the entity from the datastore (without needing to
> query).

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: how to work around this eror: "DeadlineExceededError"

2011-04-30 Thread Erwin Streur
Sasha, you are absolutely right.

Based on the orignal post I assumed that the number of Authority
records belonging to a given twitterUser is 0 or 1, because the get
method is used which returns the first result only (according to doc)
and not the fetch method which can return multiple results.
In case the code sniplet is wrong and large amount of data are being
fetched(and this is a real requirement) it become a whole different
ballgame.

Linh Truong, can clarify what exactly you are trying to do and how
much data/entities you have in the datastore and how many you expect
to retrieve using the query?

It is important to realize that the datastore is not a regular
(relational) database. Large amount of in the datastore is not/should
not be a problem, because GAE doesn't allow you to preform a full
table scan, but only index based retrieval.
Returning large amount of data in a single operation is a problem and
should be tackled by mechanisms as paging or sharding(MapReduce)

On 29 apr, 07:05, Sasha  wrote:
> DeadlineExceeded usually just means that you tried to do too much work
> in one request, and you need to choose a more efficient design.
> Usually this also means the user is waiting many seconds for a
> response, which will frustrate them, so it's good to avoid it. You
> need to push work out of the request into tasks, split up the work
> across multiple requests, or simply do less work in each request.
> Almost certainly, you don't want a huge SELECT statement.
>
> How many Authority records should normally have a given twitterUser?
> Do you really need to retrieve all of them at once? If it isn't really
> so many you are fetching, then it may just be that you are querying
> inefficiently, and this is why you should profile (for a start, see
> appstats:http://code.google.com/appengine/docs/python/tools/appstats.html).
> If the bottleneck is in how you are querying, you can make it fast in
> some simple ways, like reducing the number of round trips to the
> datastore, getting things by key name or key id instead of querying,
> storing frequently-read information in memcache, using __key__ queries
> where possible... not to mention having your application do some kinds
> of work incrementally and in advance rather than all at once. If you
> do many fetches instead of just one at a time, it will be slower.
>
> But... if the real cause of the problem is that the query returns too
> many records, and it won't work to use key names or __key__ queries to
> fetch records,
> then the solution strongly depends on WHY you are trying to get so
> many records at once. Probably, you won't really want to get all those
> records at once.
>
> If you want to show the results to the user, it's customary to show
> only a few at each time, in 'pages.' You should consider this
> carefully.
> A pretty simple and efficient way is to use query cursors, for this
> seehttp://code.google.com/appengine/docs/python/datastore/queries.html#Q...
> If the user can't stand to click 'Next,' then you could always write
> some Javascript to request each chunk after the last one completed, or
> retrieve it as the user scrolls down, so the user sees an 'infinite
> page'
> and the server can still be happy because each request does not take
> so long. If you don't like query cursors, you can use other tricks
> like inequality filters (rank >= 10 and rank < 20 limits you to only
> data from ranks 10-19) but this adds to the complexity.
>
> If you are NOT displaying data to users on a web page - if you need to
> do a mass export of some data to another computer, or to carry out an
> operation on every user, or generate a big report based on lots of
> data, then you were on the right track to look at the bulk loader
> (http://code.google.com/appengine/docs/python/tools/
> uploadingdata.html) and mapreduce. These tools really just split up
> big jobs into multiple requests themselves in a way that only makes it
> look simple. Regardless of the approach you choose, you will have to
> somehow ensure that each request takes only a short time to carry
> out.
>
> If you try to get all matching keys, that will take you a little
> farther but you will still hit a limit of how many you can grab. It's
> better to find a way to only ask the datastore for fewer records at
> each request. If something like mapreduce or pipeline or the bulk
> loader will work well for you, then by all means use those. It
> strongly depends on whether and why you really need to fetch a large
> number of records at once.
>
> If you really need a LOT of work to be done by the task queue, and
> cannot avoid it, you can always tell the user that it will take a
> while. Then when the job is done, you can refresh the page (in
> javascript) or send a notification. That is not unreasonable at all
> for tasks which are very computationally expensive.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send 

[google-appengine] System Status page misleading

2010-11-24 Thread Erwin Streur
Hi,

I find the System Status page (http://code.google.com/status/
appengine) misleading. It almost always shows 100% Availability and
"No significant issues". The non "No significant issues" was on 2nd
November: "Service disruption" even though I have see "Investigating"
in the mean time as well.

At the application level, our application is regularly returning HTTP
500 in production (logged as warnings) and we have deployment failures
in development. Given the regular posts in the fora other people are
experiencing the similar problems. So I think that Google should
report something different then "No significant issues", eg. "Minor
issues", "Major issues"

Any ideas/opinions on this? For example from Google on how they define
"No significant issues"?

Regards,

Erwin

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: App Engine serving cluster is under unexpectedly high or uneven load...please contact the App Engine team

2010-12-01 Thread Erwin Streur
As the message subtly indicated the problem is not in the execution of
your code handling the request, but the Google System preparing itself
for delegating the request to your code.

Probably it is trying to startup a new instance, but fails to do so
within 10 seconds ( check the Request Time/Latency value). There can
be a number of causes for this ranging from general stress on the
Google system to too many libraries and/or too many lengthy static
initialisation of classes.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: System Status page misleading

2010-12-02 Thread Erwin Streur
In another topic somebody mentioned api-status.com.
Indeed I find the following page to represent the GAE status much
better: http://api-status.com/6404/117406/Google-App-Engine-API

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: Newbie Q - Public IP address_DNS Server?

2010-12-17 Thread Erwin Streur
You might want to star the following issue

http://code.google.com/p/googleappengine/issues/detail?id=1269

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: How to do a batch operation without 30 sec time limit

2011-01-24 Thread Erwin Streur
Yes, you can. The Mapper API has been designed for that kind of batch
processing, especially for the type where the large amount of
execution time is caused by the large amount of individual entities to
process (instead of long execution time for each of the entities).

I recommend having a look at 
http://googleappengine.blogspot.com/2010/07/introducing-mapper-api.html
and other links mentioned on that page, including the slides of the
presentation about the concept, Ikai's post about it, the project home
page and the Getting started's.

The library is currently early experimental and clearly there is room
for improvement, but on the other hand improvements are being made; we
had an issue with very unbalanced shard distribution, which got fixed
by means of the __scatter__ property.

Are you using Python or Java? You might want to have a look at
http://code.google.com/p/appengine-mapreduce/issues/detail?id=41 In
case you are using Java you can fix it yourself and contribute to
project. The source code is available and not really rocket science.

Erwin

On 21 jan, 18:57, Mayumi Liyanage  wrote:
> Hi, I need to iterate over all the entities in the Datastore and send
> out emails once a day asynchronously to the actual app.
> Usual way to do this would be to invoke a Servlet using cron which
> would iterate over all the entities to send emails out. However, our
> data is growing at the rapid rate and sooner or later we will have a
> issue with 30 sec limit problem.
> What would be the best way to do this operation using app engine
> without worrying about 30 sec limit?
>
> Can we do above using the Mapper API? If so how can we invoke a mapper
> from the servlet?
>
> Thanks.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: How to configure a entity field to be unique?

2011-02-03 Thread Erwin Streur
Ed,

You could indeed follow the Google suggested approach and ignore it.

Or consider it to happen infrequently and expect to catch most
violating situations by performing a find on email in the Datastore
before writing the User entity, maybe with some additional checks and
reporting during later data reads if the violation hadn't been
detected during the update.

In case you want to have it really watertight, you probably have to
use a separate Kind for your uniqueness monitoring (Key=email,
value=Key of user entity) and updated it together the User updates.
Features like global transactions and optimistic concurrency control
will be required so the low level API and JDO won't do and you will
have to use richer frameworks like Slim3.

Erwin.

On 2 feb, 23:14, "Ikai Lan (Google)"  wrote:
> That sounds like an edge case to me that you'd be overengineering for unless
> you had a specific use case. How often have you changed your email address?
> I think the last time I changed mine was when Gmail launched in ... 2003?
> 2004? The only exception is if you are building something business related
> and you need work emails - those will change each time you change a job.
>
> --
> Ikai Lan
> Developer Programs Engineer, Google App Engine
> Blogger:http://googleappengine.blogspot.com
> Reddit:http://www.reddit.com/r/appengine
> Twitter:http://twitter.com/app_engine
>
> On Wed, Feb 2, 2011 at 1:21 PM, Jeff Schwartz wrote:
>
> > If the user changes their email address then all entities using it as their
> > part of their key would have to be deleted from the datastore (you can not
> > update an entities ID) and then saved again using their new email address.
> > That's doable but very expensive and IMHO something to be avoided at all
> > costs and I'd look to use something else.
>
> > On Wed, Feb 2, 2011 at 2:34 PM, Robert Kluin wrote:
>
> >> No but to implement a unique constraint, I find key_names are usually
> >> the easiest way.  But you're right, the email can change so he might
> >> need to add a second kind used  to track the uniqueness of that field.
>
> >> On Wed, Feb 2, 2011 at 12:58, Jeff Schwartz 
> >> wrote:
> >> > Since email addresses can change it isn't a good idea to use them in an
> >> > entity's id.
>
> >> > On Feb 2, 2011 12:41 PM, "dudu"  wrote:
> >> >> No solution folks? Any idea?
> >> >> How do you configure your unique fields?
>
> >> >> How can I use the email property to be part of the "Key".
> >> >> If is generated by* IDENTITY strategy*, do I need to interfere?
>
> >> >> Could someone just post some working code with a Key, and some unique
> >> >> fields?
>
> >> >> --
> >> >> You received this message because you are subscribed to the Google
> >> Groups
> >> >> "Google App Engine" group.
> >> >> To post to this group, send email to google-appengine@googlegroups.com
> >> .
> >> >> To unsubscribe from this group, send email to
> >> >> google-appengine+unsubscr...@googlegroups.com
> >> .
> >> >> For more options, visit this group at
> >> >>http://groups.google.com/group/google-appengine?hl=en.
>
> >> > --
> >> > You received this message because you are subscribed to the Google
> >> Groups
> >> > "Google App Engine" group.
> >> > To post to this group, send email to google-appengine@googlegroups.com.
> >> > To unsubscribe from this group, send email to
> >> > google-appengine+unsubscr...@googlegroups.com
> >> .
> >> > For more options, visit this group at
> >> >http://groups.google.com/group/google-appengine?hl=en.
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Google App Engine" group.
> >> To post to this group, send email to google-appengine@googlegroups.com.
> >> To unsubscribe from this group, send email to
> >> google-appengine+unsubscr...@googlegroups.com
> >> .
> >> For more options, visit this group at
> >>http://groups.google.com/group/google-appengine?hl=en.
>
> > --
> > *Jeff Schwartz*
>
> >  --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine+unsubscr...@googlegroups.com
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: How to configure a entity field to be unique?

2011-02-07 Thread Erwin Streur
You are not the only one with this requirement. You might want to star
http://code.google.com/p/googleappengine/issues/detail?id=178

On Feb 5, 12:47 am, dudu  wrote:
> Oh God...thanks folks, I used email just as example, I have other fields
> with this constraint problem.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: Application Over Quota - CPU limit

2011-03-17 Thread Erwin Streur
Same problem now with my app.

7 hours into budget day, 6.5 CPU hours used, but chart CPU Seconds
Used/Second shows values ranging from 0.01 to 0.14. No excess # of
requests nor long running requests reported in the logs.

Have also raised an issue for it.

On 17 mrt, 00:17, Arun Shanker Prasad 
wrote:
> Issue has been raised
>
> http://code.google.com/p/googleappengine/issues/detail?id=4745
>
> On Mar 16, 10:28 pm, Arun Shanker Prasad 
> wrote:
>
> > Hi All,
>
> > My application is inaccessible with message warning users that the app
> > is over quota. I had a look at the AppEngine dashboard and it is
> > mentioned that my application's CPU quota is over the limit. My
> > application has been online for years now, even at it's busiest times
> > it has never reached anywhere over 50% CPU quota, I had a look at the
> > logs and I cannot see any reason why my application could have been
> > over quota. It is night time in my country and it would be great if I
> > can get my application back up and running before start of business
> > tomorrow. Please have a look at my application quota and advice me as
> > to how to proceed.
>
> > Thanks,
> > Arun Shanker Prasad.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: Application Over Quota - CPU limit

2011-03-17 Thread Erwin Streur
:( The logs and dashboard are filtered by application version and
normally only that of the version flagged as default are displayed.
Just found out that a non-default version had been used intensely.

On 17 mrt, 15:01, Erwin Streur  wrote:
> Same problem now with my app.
>
> 7 hours into budget day, 6.5 CPU hours used, but chart CPU Seconds
> Used/Second shows values ranging from 0.01 to 0.14. No excess # of
> requests nor long running requests reported in the logs.
>
> Have also raised an issue for it.
>
> On 17 mrt, 00:17, Arun Shanker Prasad 
> wrote:
>
> > Issue has been raised
>
> >http://code.google.com/p/googleappengine/issues/detail?id=4745
>
> > On Mar 16, 10:28 pm, Arun Shanker Prasad 
> > wrote:
>
> > > Hi All,
>
> > > My application is inaccessible with message warning users that the app
> > > is over quota. I had a look at the AppEngine dashboard and it is
> > > mentioned that my application's CPU quota is over the limit. My
> > > application has been online for years now, even at it's busiest times
> > > it has never reached anywhere over 50% CPU quota, I had a look at the
> > > logs and I cannot see any reason why my application could have been
> > > over quota. It is night time in my country and it would be great if I
> > > can get my application back up and running before start of business
> > > tomorrow. Please have a look at my application quota and advice me as
> > > to how to proceed.
>
> > > Thanks,
> > > Arun Shanker Prasad.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: gae needs a counter service

2011-03-20 Thread Erwin Streur
I'm puzzled by all the answers, because to me the question is not
clear. Just like Robert says: different uses have different
requirements on the behavior of counters.

So what are you looking for and why do things like
DatastoreService.allocateIds or MemcacheService.increment not work for
you?

On another note: Brewer's CAP theorem is interesting reading material.
Might provide you some background on Nick's comments (and some other
features GAE is claimed to lack compared to SQL databases, including
difference between strong consistent vs eventually consistent)

On 19 mrt, 17:06, Jeff Schnitzer  wrote:
> On Fri, Mar 18, 2011 at 6:10 PM, Barry Hunter  wrote:
>
> > At least some can answer count queries using just index scans - ie
> > doesnt touch the actual data. Can get close to that with keys_only -
> > which persumably is just what the index contains - and counting the
> > rows, but it still requires transfer of all keys to the application (I
> > think anyway) just to be discarded.
>
> This is what PreparedQuery.countEntities() does.  It doesn't transfer
> keys to the application, but it's still an O(N) operation.
>
> Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: Neat way to remove obsolete properties from entities?

2011-04-06 Thread Erwin Streur
Robert,

Are you sure that this is going to work? In GAE there is a significant
difference between a property not being set/not existing at all and
being set to null/None.

Erwin

On 5 apr, 08:27, Robert Kluin  wrote:
> Hey Greg,
>   Assuming you are deleting the property definition from the Model,
> then I think you can also use:
>
>      your_entity._entity.pop('property_to_delete', None)
>
>   It is shorter, and probably a little clearer.
>
> Robert
>
>
>
> On Tue, Apr 5, 2011 at 00:17, Greg  wrote:
> > I've stumbled across what looks like a neat way to remove obsolete
> > properties from entities, without having to switch models to Expando
> > and back again. I convert the entity to a protocol buffer, which
> > reveals the obsolete properties. Then I remove them, convert back to
> > an entity and save it.
>
> > So far my testing has shown this doesn't have any ill effects, but I'd
> > like feedback in case what I'm doing is dangerous. In particular, I
> > don't understand what would happen to the indexes for the removed
> > properties - how does the datastore handle these?
>
> > Cheers
> > Greg.
>
> > # !!! EXPERIMENTAL CODE !!!
> > # !!! USE ENTIRELY AT YOUR OWN RISK !!!
> > # !!! NOT ENDORSED BY GOOGLE (YET) !!!
>
> > from google.appengine.ext import db
>
> > def remove_stale_properties(e):
> >        current_properties=e.properties().keys()
> >        epb=db.model_to_protobuf(e)
> >        for i in reversed(range(len(epb.property_))):
> >                if epb.property_[i].name() not in current_properties:
> >                        logging.info('Deleting property %s:%s'%
> > (epb.property_[i].name(),epb.property_[i].value()))
> >                        del epb.property_[i]
> >        for i in reversed(range(len(epb.raw_property_))):
> >                if epb.raw_property_[i].name() not in current_properties:
> >                        logging.info('Deleting raw property %s:%s'%
> > (epb.raw_property_[i].name(),epb.raw_property_[i].value()))
> >                        del epb.raw_property_[i]
> >        return db.model_from_protobuf(epb)
>
> > # Example usage
>
> > entity_list=Foo.all().fetch(1000)
> > for entity in entity_list:
> >        entity=remove_stale_properties(entity)
> > db.put(entity_list)
>
> > --
> > You received this message because you are subscribed to the Google Groups 
> > "Google App Engine" group.
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to 
> > google-appengine+unsubscr...@googlegroups.com.
> > For more options, visit this group 
> > athttp://groups.google.com/group/google-appengine?hl=en.- Tekst uit 
> > oorspronkelijk bericht niet weergeven -
>
> - Tekst uit oorspronkelijk bericht weergeven -

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.