[google-appengine] Re: solution to the 1mb limitation for google app engine?
split your data into parts for datastore entity 1M limitation. and combine parts to one file before respond to browser. @PROPERTY def Binary(): def fget(self): if self.size > MAX_BLOD_SIZE and self.binarylist: bin = StringIO() for id in self.binarylist: part = PhotoPart.get_by_id(id) bin.write(part.binary) return bin.getvalue() else: return self.binary def fset(self, bin): length = len(bin) if length > MAX_BLOD_SIZE: seq = 0 if not self.is_saved(): self.put() for i in range(0, length/MAX_BLOD_SIZE+1): part = PhotoPart() part.binary = bin[i*MAX_BLOD_SIZE:(i +1)*MAX_BLOD_SIZE] part.photo = self part.seq = seq part.put() self.binarylist.append(part.id) seq += 1 else: self.binary = bin return locals() your can see full source from my project GAEPhotos http://code.google.com/p/gaephotos/source/browse/trunk/models.py On Dec 13, 11:17 am, Stephen Johnson wrote: > Yes, the datastore has a 1MB limit on entities. You can use the blobstore > for larger objects. http://code.google.com/appengine/docs/java/blobstore/ > > On Sun, Dec 12, 2010 at 8:01 PM, dadada wrote: > > hi all, > > > is there a 1mb limitation for google app engine? > > > I am want to upload photos which are all more than 1mb. what can be > > the possible solution? the clients will be on browser and mobile. > > > can i compress the data on browser? > > > Thank you! > > bryan > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appeng...@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Username Already Exists For Application ID?
I went to create an Application with my Google username (usern...@gmail.com). It says the name is available, but then when I click to submit I get the message: Already Exists This application ID or version is already in use. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] solution to the 1mb limitation for google app engine?
Yes, the datastore has a 1MB limit on entities. You can use the blobstore for larger objects. http://code.google.com/appengine/docs/java/blobstore/ On Sun, Dec 12, 2010 at 8:01 PM, dadada wrote: > hi all, > > is there a 1mb limitation for google app engine? > > I am want to upload photos which are all more than 1mb. what can be > the possible solution? the clients will be on browser and mobile. > > can i compress the data on browser? > > Thank you! > bryan > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] solution to the 1mb limitation for google app engine?
hi all, is there a 1mb limitation for google app engine? I am want to upload photos which are all more than 1mb. what can be the possible solution? the clients will be on browser and mobile. can i compress the data on browser? Thank you! bryan -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Channel API CPU usage
> Better docs on Channel API quota use are coming, but the reason the > CPU charge is so high is that the CreateChannel charges up-front for 2 > hours of connect time by clients. It is true that if you have the same > user connecting repeatedly for a short time but spread out over > periods of time longer than two hours (obviously degenerate case would > be connecting for a couple of minutes every two hours and one second), > the charge will be high. One thing I'm looking at is the ability to > create a token that expires after an application-specified period, if > your application's usage patterns warrant it. > > Note that you can store the token for re-use in eg. a cookie so you > don't need to regenerate it more frequently than every two hours. > > Sending a message doesn't charge any CPU time at all (since the > connection's already established), just outbound bandwidth. It being the first version, I was hoping that it took so much CPU time because it had room for improvements. Does this mean that it's unlikely for the CPU costs of setting up a channel to come down? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Google App Engine Java URL Handling
I have a Java Google App Engine application with one module and one HTML page. I would like to serve different content to the user, using GWT, by using different URLs. For example: "http://myapp.appspot.com/"; Serves the main HTML page and calls the entry point method. This works fine. I would like to serve different UI content when I use a browse to a URL like "http://myapp.appspot.com/differentcontent/";. I know I can set up a servlet in WEB.XML and create a server side class for that URL pattern, but how do I get back to a GWT client side class from the servlet class when the servlet call did not originate from GWT? Any help would be much appreciated! -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] get key of a datastore entity
Hello,
I have the following code:
try {
Extent extent = pm.getExtent(Member.class,
false);
for (Member m : extent) {
m.addPurchase(p);
}
}
extent.closeAll();
}
So... with the line m.addPurchase(p) the p Object is automatically
saved to the datastore. How can i get the generated unique key for
this object from the datastore?
greets
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Unable to deploy due to "500 server error" since 5.45pm EST
Attempting to deploy, I get: Cloning 22 static files. Cloning 241 application files. Cloned 100 files. Cloned 200 files. Precompilation starting. Rolling back the update. Error 500: --- begin server output --- 500 Server Error Error: Server Error The server encountered an error and could not complete your request.If the problem persists, please http://code.google.com/appengine/community.html";>report your problem and mention this error message and the query that caused it. --- end server output --- Same happened whether with or without --no_precompilation. Started working again around 6.09pm EST. No indication anywhere in GAE stats that anything was wrong, but deploying from different networks to different appids all failed. Any status or information from google on what happened would be nice. Thanks. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: max_backoff_seconds - working?
Bump. GAE team can you comment please? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: sdk server too slow in 1.4.0
Incidentally, just experienced one of the sqlite related issues, and added some comments http://code.google.com/p/googleappengine/issues/detail?id=3124 -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] How to force urllib2 to fetch newly files.
someone like this: http://www.twse.com.tw/ch/trading/exchange/STOCK_DAY/STOCK_DAY_print.php?genpage=genpage/Report201012/201012_F3_1_8_8261.php&type=csv&r=72560 which is about Taiwan stock data.(stock no is 8261) If I fetch more times in one day, it can't to fetch the newly data, even if the stock is close. I also add random no. at the end of URL, but can not work. Thanks. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: memcache time
Two posts ago http://groups.google.com/group/google-appengine/browse_thread/thread/e5d6a1eae6080878/73a99164d7899859#73a99164d7899859 deal with the same issue. it seems like AppEngine is evicting sooner than you might expect... still no formal answer On Dec 12, 3:33 pm, Tomas Alaeus wrote: > 1. The time you specify when putting memcache entities is the maximum > lifetime. There is no guarantee that the memcache will hold its > contents for any period of time, they can be evicted immediately or > after a month, you have no way of knowing that. > > 2. Not that I know of. Part of the memcache can be flushed, or all of > it, at a time. So I don't think there is an easy way to implement it > either. Besides, since the memcache is stored in volatile memory the > "flush" can be triggered by a machine shutting down. In that case it > would be hard to know immediately which entities were removed. > > You should always make the code work even if the memcache always > return null. Easiest is to populate the memcache when a request > fetches a null entity, so it always have something to fall back on. > > On 12 Dec, 14:24, Sharma Anil wrote: > > > > > > > > > I have implemented the following on GAE. > > > 1. Fetch all entities in a Kind and identify the last updated > > entity. > > 2. Set the memcache with a key and a property from the entity from > > step one with specifying 6 hours longevity > > 3. Log the information and status > > 4. Define a cron job handle to do all steps 1, 2 and 3, in every 4 > > hours > > > Result: Works fine most of the time. > > Problem: Memcache entry does not exist for the duration set for > > it. > > > Question: 1. Is there any way to ensure Memcache key-value pair > > for a defined time ? > > 2. Is there anyway to get a message from GAE on > > forced Memcache flush out ? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: memcache time
1. The time you specify when putting memcache entities is the maximum lifetime. There is no guarantee that the memcache will hold its contents for any period of time, they can be evicted immediately or after a month, you have no way of knowing that. 2. Not that I know of. Part of the memcache can be flushed, or all of it, at a time. So I don't think there is an easy way to implement it either. Besides, since the memcache is stored in volatile memory the "flush" can be triggered by a machine shutting down. In that case it would be hard to know immediately which entities were removed. You should always make the code work even if the memcache always return null. Easiest is to populate the memcache when a request fetches a null entity, so it always have something to fall back on. On 12 Dec, 14:24, Sharma Anil wrote: > I have implemented the following on GAE. > > 1. Fetch all entities in a Kind and identify the last updated > entity. > 2. Set the memcache with a key and a property from the entity from > step one with specifying 6 hours longevity > 3. Log the information and status > 4. Define a cron job handle to do all steps 1, 2 and 3, in every 4 > hours > > Result: Works fine most of the time. > Problem: Memcache entry does not exist for the duration set for > it. > > Question: 1. Is there any way to ensure Memcache key-value pair > for a defined time ? > 2. Is there anyway to get a message from GAE on > forced Memcache flush out ? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] memcache time
I have implemented the following on GAE. 1. Fetch all entities in a Kind and identify the last updated entity. 2. Set the memcache with a key and a property from the entity from step one with specifying 6 hours longevity 3. Log the information and status 4. Define a cron job handle to do all steps 1, 2 and 3, in every 4 hours Result: Works fine most of the time. Problem:Memcache entry does not exist for the duration set for it. Question: 1. Is there any way to ensure Memcache key-value pair for a defined time ? 2. Is there anyway to get a message from GAE on forced Memcache flush out ? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] How GAE fits into Continuous Integration of cloud applications?
Hello, We are starting to design our continuous integration system. The path we choose to go is automated test cycle: Get fresh environment (17 web services, distributed on ~6 EC2 instances), plus GAE as the super scalable front end. Start running functionality tests on the system. Should the tests pass do auto deploy to staging. Now comes the tricky part: We can get EC2 instances up/down easily via API, not quite so for GAE. It would be cool to have the same API for GAE yet we don't... (yet?). The solutions we currently consider, are: 1. Have *N* GAE "applications". One of which will be the production one, use others as a pool for testing resources. This is not any way perfect because we need to clear the datastore after each test cycles finishes, plus you must always deploy the same "version number" to replace the existing version before you launch the test. 2. Reverse engineer the web interface of appengine.google.com. Then we would create a randomly named instance (UUID or whatever), run the tests on this instance and then delete it. 3. Deploy to a different version on the production application. 4. Have 2 applications "Production" & "Staging". For staging define *"N"*versions which will be used for testing. Implement DataStore and Memcache partitioning in code so that versions don't override each other data records. Then each test environment deploys to a specific "version" on the staging environment. The first solution is not cool because it has lots of possible failure options, plus it limits us to the number of environments we can concurrently run (probably not an issue, there shouldn't be more then 3-5 running at any given time). The second solution requires a process of RE the HTTPS calls, plus creating many GAE applications and then deleting them is something the seems to be not good for GAE. From what it seems applications are logically deleted -- Hench we would be creating lots of noise for short lived GAE applications... We would want to create a harmful process. The third option is still not bright, some of the versions might bring DataStore scheme changes. This is OK for rolling update as long as we don't need to roll back... but since the deployed version is part of a test cycle (haven't been deployed on staging yet) the currently in production version will use the older scheme so things will break. The forth option requires changes in code, and might become fragile to code changes. Plus it also requires cleaning the datastore & memcache at the end of a test cycle process. So, did any one here did something like this? I would appreciate input / suggestions. Thank you, Maxim. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: What's the red and blue bars in AppStats?
The usage of Appstats on the development environment is to determine if the request is not performing to many RPC calls. Doing transactions in a loop is a nice example that might be optimized. Don't compare the execution times of the development and production. They have very different implementations of the Datastore, and sometimes the production has to start a new app instance. For some RPC calls on the production server the API time (red bar) is larger then the wall-clock time (blue bar) because some queries/puts/get/fetch calls can be parallelized. The red bar is the sum of the processing cycles combined. The gaps in between the blue bars is the time spend in the code you have written that are not API calls. For example to construct a list of db.Key's you want to get. Those ms added together and converted to 1.2 GHz processor are the CPUms in the result. 2010/12/7 Fredrik Bonander : > Thanks for your (both of you) info. > In my mission to improve my application I'm on a constant hunt for > milliseconds. But while improving on every request I find it kinda strange > that the difference would be so huge from my development server. > In appstats the trouble some request that show's "3275ms 9140cpu_ms > 7296api_cpu_ms" is 1 memcache call, 4 db.RunQuery and 5 db.Get and in my > development it's about 400ms. When thinking about it I would think that the > red bar (the longest one) is time spent waiting on the request to return. > Especially since the red bar is not present in appstats on my development > server. > I think it would make sense or that for some odd reason my code is so bad > that would run that much slower on the production servers. > Another thing that bugs me a bit is that this is my third time asking about > how CPU/API time is calculated in different contexts but never any answers > from Santa's Elves (in sprit of keeping to the metaphor). > If I'm right (about the red bars) why the long waiting times for some > requests ? > It would be great to get some clarification in how this works so we as > developers can know how to proceed. I don't want to spend x amount of hours > optimizing my code if I can't to anything to help improve the CPU/API times. > ..fredrik > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: updating a relation isn't persisted (using Java/JPA)
I haven't found a solution to my problem.
I think I haven't correctly understood how persistance work with
Google App Engine.
I stop using directly JPA, and now use twig-persist as persistance
layer ( http://code.google.com/p/twig-persist/ )
On 9 déc, 01:11, david wrote:
> Hello,
>
> I got a simple relation between two object :
>
> @Entity
> class Spot {
> //
> �...@onetoone(targetEntity = Pics.class, cascade = CascadeType.ALL,
> fetch = FetchType.LAZY)
> private Pics pics;
> // ...
>
> }
>
> I have to update this relation by setting a new "Pics" objet for an
> already persisted Spot.
>
> Example :
>
> I persist a Spot object (with a Pics setted)
> I retrieve this last Spot from the datastore, then change pics field
> with a new Pics Object, and call persist again from the EntityManager.
> (in order to update the datastore)
> When I retrieve the Spot object, again from the datastore, the pics
> field is still setter with the first Pics object, and not the new one.
> So I'm a little bit confuse because the new relation isn't persisted !
>
> Thanks to App Engine Test helper, I created a small test which show
> this use case :https://gist.github.com/734144
>
> Where am I wrong ? My code is correct or I call the wrong method on
> the entityManager ? or maybe my mapping is wrong ? (or I simply miss
> something about JPA)
>
> Thank you for your help :)
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: sdk server too slow in 1.4.0
Hi Robert 1-As I indicated, I do use appstats in local development (that´s where I got the timing on the queries). The query rpc(keys_only) is very slow (coudl take 3 seconds). db.get rpcs are very fast. Turning appstat recording on or off makes no significant difference in the overall performance. The data recording for app stats is done via memcache 2-I assume you meant whether I´m doing bounding box queries. Well, I do make bounding box queries when map is dragged (bounding box queries are implemented by issuing an individual query per tile, which is surprisingly fast in production ). In any case, I´m not timing (via appstats) an aggregate of queries but a single query. For a city page, e.g http://www.askaro.com/es/CT/Barcelona just a few vainilla queries are needed to get latest questions, local heros, etc. Now, again, in production things are fast enough. I rely heavily on prefetching of reference properties, etc. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: memcache evictions
>From What I calculated, the miss ration is 20% (40902/(166694+40902)) and that's a lot considering that I cache everything and there are very little entities. Also my caching API calls do not even scrape the beginning of the quota Will AppEngine For Business help here? On Dec 12, 10:30 am, Tim Hoffman wrote: > HI > > I am afraid there is nothing to help with. Its a part of the behaviour > of the system that data > will be evicted from memcache. You absolutely cannot rely on things > being kept around in memcache. > > And your stats actually look pretty good, you have a 3% cache miss rate. > > Rgds > > Tim -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: memcache evictions
HI I am afraid there is nothing to help with. Its a part of the behaviour of the system that data will be evicted from memcache. You absolutely cannot rely on things being kept around in memcache. And your stats actually look pretty good, you have a 3% cache miss rate. Rgds Tim -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
