[appengine-java] Re: HashMap within an Embedded Class
This is a known bug and seems still not fixed yet. You will need to manually serialize in you code Also, if you just started coding on GAE, check out Objectify / Twig / Siena etc. These are all very good frameworks. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/B51GO7gHVksJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: Отг: Re: [appengine-java] Datastore Replication Options - can not be changed once app id is created
I also don't understand why we cannot switch. A lot of people are requesting this (including myself). On Jul 22, 7:14 am, Miroslav Genov mge...@gmail.com wrote: I have the opposite situation :). My app is using master/slave and I want to migrate it to use the multi-master replication. Is there are any idea the re-using of application id's to be enabled at some point of time or maybe someway of migration the datastore from master/slave to multi-master and vise-versa? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] How to run quatz(kind) of scheduler service on appengine?
You can have deploy your quatrz scheduler in EC2 and the only job is pinging your corresponding appengine URL according to the schedule. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/HzvkfqOeTWgJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Accessing same datastore from within 2 (local) JVM's
I'm starting my application local and I will perform some functional tests within a JUnit tests. From within this same JUnit test I also want to check if values are stored in datastore. Can I access the same datastore the running webapp is using, but then from within JUnit test running in a different JVM? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: New Pricing Model
Can anybody help me to know how much i will spend with GAE? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] How to run quatz(kind) of scheduler service on appengine?
http://code.google.com/appengine/docs/java/config/cron.html http://code.google.com/appengine/docs/java/config/cron.html http://code.google.com/appengine/docs/python/config/cron.html http://code.google.com/appengine/docs/python/config/cron.htmlIs that what you want? *Jayr Motta* Software Developer * * I'm on BlackBeltFactory.comhttp://www.blackbeltfactory.com/ui#!User/jmotta/ref=jmotta ! On Fri, Jul 22, 2011 at 5:22 AM, Max thebb...@gmail.com wrote: You can have deploy your quatrz scheduler in EC2 and the only job is pinging your corresponding appengine URL according to the schedule. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/HzvkfqOeTWgJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] How to run quatz(kind) of scheduler service on appengine?
Will Quartz run in a backend instance? http://code.google.com/appengine/docs/java/backends/overview.html I don't remember offhand if Quartz uses threads. I'm pretty sure it does which will mean it *won't* work, but I figured I'd throw it out there. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Fri, Jul 22, 2011 at 4:22 AM, Max thebb...@gmail.com wrote: You can have deploy your quatrz scheduler in EC2 and the only job is pinging your corresponding appengine URL according to the schedule. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/HzvkfqOeTWgJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] How to run quatz(kind) of scheduler service on appengine?
The OP mentioned that cron was too limiting for his particular needs. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Fri, Jul 22, 2011 at 10:19 AM, Jayr Motta jayrmo...@gmail.com wrote: http://code.google.com/appengine/docs/java/config/cron.html http://code.google.com/appengine/docs/java/config/cron.html http://code.google.com/appengine/docs/python/config/cron.html http://code.google.com/appengine/docs/python/config/cron.htmlIs that what you want? *Jayr Motta* Software Developer * * I'm on BlackBeltFactory.comhttp://www.blackbeltfactory.com/ui#!User/jmotta/ref=jmotta ! On Fri, Jul 22, 2011 at 5:22 AM, Max thebb...@gmail.com wrote: You can have deploy your quatrz scheduler in EC2 and the only job is pinging your corresponding appengine URL according to the schedule. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/HzvkfqOeTWgJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Scheduler features and load time limitations
Have you taken a look at the scheduler that has shipped? It should be under Application Settings. If there's anything there that you think might be needed, please let us know. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Mon, Jul 18, 2011 at 5:40 PM, Marcel Overdijk marceloverd...@gmail.comwrote: Everybody knows about load time issues when using frameworks that take a long time to startup. Take for example Grails. I wonder if we will still encounter these issues after the pricing changes. 1) Will the limit of 30s to start an application be lifted? 2) How smart will the scheduler be to startup new instances? Will it for example take into account the avg load time of previously started instances? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: HashMap within an Embedded Class
On Jul 22, 7:29 am, Max thebb...@gmail.com wrote: This is a known bug and seems still not fixed yet. You will need to manually serialize in you code It may be a known bug to you, but perhaps if someone actually register it in Google's issue log then they would know about it? http://code.google.com/p/datanucleus-appengine/issues/list If it exists there, then refer to it so the user knows which one to star -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] JSF session-scoped beans replication issue
There are some issues with JSF session-scoped beans. In some cases session bean changes are not replicated to another server instances. For details and workarounds look at this post http://java.zacheusz.eu/google-app-engine-http-session-vs-jsf-en/394/ -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] low-level api: many to may relatioship
Am wondering how in the low level api can i represent a man to many relationship , this is a good approximation? Entity entityA = new Entity(TypeA); entityA.setProperty(name, nameUserA); Entity entityB = new Entity(TypeA); entityA.setProperty(name, nameUserB); ds.put(entityA); ds.put(entityB); Entity entityChild = new Entity(entityChild,entityAKey); entityChild.setProperty(name,child); ds.put(entityChild); Entity entityChild = new Entity(entityChild,entityBKey); entityChild.setProperty(name,child); ds.put(entityChild); -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/PWTINDhJTh0J. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: [google-appengine] Re: CMS for GAE
Hello,
Thanks for your contributions for a GAE-Java CMS.
I had a look at the code of claymus. It seems that you use the
combination of gae and gwt, this is a very interesting point.
But on the client side, some architectural questions come to my
mind...
I don't understand your separation of code responsability?
Up to now, I don't see something like MVC or MVP or UI Builder ... It
seems that your write the html output directly from java (and that in
the same file that the DAO definition if I am not wrong)...
Could you explain a bit the advantages of your strategy?
A lot of people could think that a bad separation of responsibilities
will lead to a lot of difficulties in code maintenance and
evolution...
But it must be subtilities in your approach that I don't catch up to
now...?
Thanks in advance for your explanations,
Michel
PS:
I look for example in:
package com.claymus.site.module.page;
(...)
@PersistenceCapable(detachable = true)
public class Page extends PersistentCapable {
(...)
public void serve(ListListContent contents, ListListBlock
blocks, Theme theme) throws ServletException, IOException {
PrintWriter out = ClaymusMain.getResponse().getWriter();
// DOC TYPE
out.print(getDocType());
out.print(html);
(...)
On Jul 18, 5:50 pm, Prashant antsh...@gmail.com wrote:
Yes, you are right. Picking an opensource project and making it work for you
is very easy, but making something from scratch has its own fun :) . I
stared building Claymus because I felt need of Java based CMS which is made
for GAE, considering what GAE is, how it is different from other hosting
services, services/apis GAE is providing to developers, etc. etc. etc.
Claymus is designed to give you maximum flexibility and support to build
your app on top of Claymus. Along with all the GAE features you can take
advantage of Servlet Level caching to minimize delay and cpu usage, plugable
modules and themes, etc ... (complete list will be put on updated Claymus
Website :) ).
Regards,
Prashant
On Mon, Jul 18, 2011 at 7:28 PM, Niklas Rosencrantz nikla...@gmail.comwrote:
Having something like a CMS eg like wordpress, joomla or mediawiki
compatible then we won't have to start every project from scratch and
project would become more configuration than programming which naturally is
an advantage since programming takes more time than just configuring a new
CMS deployment. I used web2py for a project when a friend wanted a quick
website started and naturally to change pages we'd prefer a web-based editor
instead of changing templates and redeploying. I'm glad to see some CMS are
coming to app engine. Some frameworks offer features that are getting close
to that of a CMS, for instance GAE Framework (www.gaeframework.com) comes
with a blog engine and when you have a blog you are getting close the the
functions of a CMS. Do you agree?
On Sun, Jul 17, 2011 at 3:48 AM, Maneesh Arora mane...@gmail.com wrote:
What's the unique advantage of having a CMS in GAE?
thanks,
Maneesh
http://mightytext.net
On Sat, Jul 16, 2011 at 11:25 AM, Prashant antsh...@gmail.com wrote:
Hi All,
Claymus is close to its (first) preview release. Only few bug fixes and
minor modifications are remaining. Check out the live demo at
http://demo.claymus.com.
--
Prashant
claymus.googlecode.com
On Wed, May 25, 2011 at 10:29 AM, Jeff Schnitzer
j...@infohazard.orgwrote:
On Mon, May 23, 2011 at 11:17 AM, Brandon Wirtz drak...@digerat.com
wrote:
That's more deployed, not spoken. Every Blogger and IT guy on the
planet
can hack together a PHP Plugin for their CMS. Very Few can do that in
Python or Java.
...and even fewer can do it in Ruby or C#.
Truly, PHP is in a class by itself.
Jeff
--
You received this message because you are subscribed to the Google
Groups Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
[appengine-java] [OAuth + Provisioning ] Autentication - Python X Java
Hi, I need help to use Provisioning API with OAuth authentication, the following problem occurs, how to do domain validation in java api ? In Python it is possible to know the following code. service = gdata.apps.service.AppsService(domain=consumer_key) *-*service.SetOAuthInputParameters(sig_method, consumer_key, consumer_secret=consumer_secret, two_legged_oauth=True) res = service.RetrieveAllUsers() for entry in res.entry: print 'email;' + string.lower(entry.login.user_name) + '@' + consumer_key the emphasis on the coding can be seen that there is validation for key parametersusing OAuth authentication as CONSUMER_KEY and CONSUMER_SECRET, but inJava this is not possible .. As shown below: AppsForYourDomainClient client = new AppsForYourDomainClient (adminEmail, adminPassword, domain); //JavaCode Api - AppsForYourDomain how can I get something close to in java: AppsForYourDomainClient client = null; // client.SetOAuthInputParameters(sig_method, consumer_key, consumer_secret); //this method does not exist today in the api java Thanks... -- *Felipe Teixeira* *Desenvolvedor Java * IPNET Soluções Tel.: 55 21 3553 2717 / 21-7805-6867 Rua Visconde de Inhaúma, 134, Sala 615 Centro - RJ - CEP: 20091-007 *http://www.ipnetsolucoes.com.br** * -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] low-level api: many to may relatioship
A few comments:
1. That's more of a 1:1 relationship
2. Try to avoid using the term child. The reason is that when you start
working with entity groups, you will confuse yourself with terminology. If
you don't understand entity groups yet, take a look at these docs:
http://code.google.com/appengine/docs/python/datastore/entities.html#Entity_Groups_and_Ancestor_Paths
Basically, entity groups represent a hierarchical set of related models. You
should learn about them because they are the unit of atomicity in high
replication datastore (the default).
3. To represent a many-to-many relationship, you would probably want to have
a List property on each entity. A List can contain several keys. The thing
to be aware of here is that for bidirectional many-to-many relationships,
you have to manually make sure the key of the related object is in the
*other* object's list. Example:
Entity ikai = new Entity(User, ikai);
ikai.setProperty(friends, Arrays.asList(String[] { proppy, wesley,
nick});
Entity proppy = new Entity(User, proppy);
proppy.setProperty(friends, Arrays.asList(new String[] { ikai,
nick }));
If proppy (Johan) and I ever have a falling out, you have to remember to
remove johan from Ikai's list of friends and vice versa. Normalization
zealots are shaking their heads right now because is possible for many:many
relationships like this to get out of sync.
Some insights into working with non-relational persistence:
This isn't the only way to represent child objects. You want to optimize for
reads when possible because *most* applications are read heavy. A rule of
thumb is that the number of reads for a given entity are an order of
magnitude greater than the number of writes. With that in mind, let's talk
about an example of a real world data model I've had to recently write.
My problem: I needed to represent users and teams. A user has a role on a
team. Example:
Ikai - on App Engine team, roles: advocate, support, developer
proppy - on App Engine team, roles: advocate, support, on Apps team roles:
developer, tech writer
In an RDBMS, I'd likely represent this with join tables with an additional
field for the role. This would likely involve several join tables.
I examined my requirements, and I came to the conclusion that I'd need two
views:
- What roles does a person have? (What teams and roles does Ikai have?)
- Given a team, what other roles do people on that team have? (Give the App
Engine team, what other teams do people on this team have?)
How did I model this?
Two entity kinds: User entity and Team entity.
On the User entity, I stored a JSON serialized property that looks roughly
like this:
{ teams : {
App Engine : [ advocate, support, developer ],
}
When I write a User entity, I also write a Team entity. The Team has a JSON
serialized property that stores all the roles, team members, and a snippet
of information about Team members.
There are tradeoffs in this design, but it suits my purposes:
- It is possible things get out of sync. Since I don't put everything in the
same entity group, I can't do a transaction on a Team and User entity when I
make a change and need to write two entities. I accepted this, but there are
various ways I can mitigate the damage: one way is to use a task queue task
that retries until it executes correctly. Since the action to join or leave
a team can be idempotent, this is something I can run over and over.
- When I read a User entity, I can do this with a single key, so I don't
need to do a query and read related objects. This makes my reads very fast.
The same is true of Team entities. Now, granted, this might be a bit of an
overoptimization, since the number of teams, roles and users might be, worst
case scenario, in the tens of thousands and I can probably just read the
entire datastore and do operations in-memory.
It's possible I'll run into problems later with the way I've designed this,
but rather than try to get everything perfect, I opted with a good enough
design without obvious vlaws to get the project off the ground. Technical
debt sucks, but sometimes I see developers too much time overthinking
problems trying to get this perfect. I'm a cynic, but I've accepted that
I'll always look back on software projects and go, Man, I designed that
poorly anyway, so I might as well mess them up quickly, have a working
product, figure out where I've messed up, and fix the problems.
Ikai Lan
Developer Programs Engineer, Google App Engine
Blog: http://googleappengine.blogspot.com
Twitter: http://twitter.com/app_engine
Reddit: http://www.reddit.com/r/appengine
On Thu, Jul 21, 2011 at 3:51 PM, Alexander Herrera
edwinalexande...@gmail.com wrote:
Am wondering how in the low level api can i represent a man to many
relationship , this is a good approximation?
Entity entityA = new Entity(TypeA);
entityA.setProperty(name, nameUserA);
Entity entityB = new Entity(TypeA);
entityA.setProperty(name, nameUserB);
[appengine-java] Re: New Pricing Model
No body can answer that without knowing exactly how your app uses billed resources. Latest details can be found here: https://groups.google.com/d/topic/google-appengine/Hluog1_a3n4/discussion BTW, 10 queries is too much - your goal should be zero or one query per page by using denormalisation, memcache and in-memory caching. 10 consecutive queries would result in a very slow website and if your requests don't complete in roughly 600ms or less then your app will not be auto-scaled which effectively kills it. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/9hGSoMRIKkMJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] How to run quatz(kind) of scheduler service on appengine?
Yes.. I looked at Background instance. It allows to have long tasks but it doesn't allow to spawn threads. Quartz fires off events in threads. So I guess quartz is not an option on app-engine. We can use appengine cron jobs to some extent but if you need to fire an event at exact minute, it would be tough. It's tough to maintain 2 cloud providers for single product. Not sure how efficient it is to call appengine urls from EC2. May be move to EC2 completely may be better solution. We need object datastore for our product. Has anyone has experience with EC2 object datastore SimpleDB? Does it have same features and JPA/JDO interfaces like appengine. GAE datastore is pretty good but it doesn't give a VM ownership to the developer. The Backends in appengine still don't give any control of the server/instance to developer. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/nlrteFBD_2sJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: [google-appengine] Re: CMS for GAE
Hello Michel,
Thanks for checking out the code and the query :)
On Thu, Jul 21, 2011 at 9:49 PM, Michel Schroeder michels...@gmail.comwrote:
Hello,
Thanks for your contributions for a GAE-Java CMS.
I had a look at the code of claymus. It seems that you use the
combination of gae and gwt, this is a very interesting point.
But on the client side, some architectural questions come to my
mind...
I don't understand your separation of code responsability?
Code separation is done considering mainly, modularity and pluggability of
modules. All the classes related to a module are placed in a package (and
sub-packages) meant for that module. Modules are designed such that it won't
affect the functioning of App or other modules if a new module is added or
an old one (which is not a core module) is removed from the App.
Up to now, I don't see something like MVC or MVP or UI Builder ... It
seems that your write the html output directly from java (and that in
the same file that the DAO definition if I am not wrong)...
Yes, I didn't try to follow MVC or MVP strictly because I don't think that
is a strict requirement. I din't use UI Builder because none of the GWT
modules have very big and/or complex UI. If you are worried about supporting
mobile devices, that can supported just by creating a mobile theme (which
will include layout and look feel) and few tweeks in Content and Block
types.
I think you are thinking of following methods as Data-Access methods. They
are not Data-Access methods, they are just converting Data Object to Data
Transfer Object. PageData class, in the same package, is the DAO for page
module.
public PageDTO getDTO() {
PageDTO pageDTO = new PageDTO();
pageDTO.setUri(getUri());
pageDTO.setTitle(getTitle());
pageDTO.setLayout(getLayout().getClass().getSimpleName());
return pageDTO;
}
public void update(PageDTO pageDTO) {
setTitle(pageDTO.getTitle());
setLayout(pageDTO.getLayout());
}
And, I don't see any thing wrong in outputting html from a java servlet.
Could you explain a bit the advantages of your strategy?
I am not following popular design architectures, like MVP and MVC, that may
have some disadvantages that I am not aware about now.
I am trying to support maximum modularity and plaggability like any other
CMS. e.g. if you have to add a new Content or Block type, you just need to
write 2 small classes, one for backend and one for front-end (see
com/claymus/site/module/block/types/RichText.java and
com/claymus/site/module/block/types/gwt/RichText.java) and drop them in to
Content/Block Types package. Same for a theme.
A lot of people could think that a bad separation of responsibilities
will lead to a lot of difficulties in code maintenance and
evolution...
But it must be subtilities in your approach that I don't catch up to
now...?
Thanks in advance for your explanations,
Michel
PS:
I look for example in:
package com.claymus.site.module.page;
(...)
@PersistenceCapable(detachable = true)
public class Page extends PersistentCapable {
(...)
public void serve(ListListContent contents, ListListBlock
blocks, Theme theme) throws ServletException, IOException {
PrintWriter out = ClaymusMain.getResponse().getWriter();
// DOC TYPE
out.print(getDocType());
out.print(html);
(...)
On Jul 18, 5:50 pm, Prashant antsh...@gmail.com wrote:
Yes, you are right. Picking an opensource project and making it work for
you
is very easy, but making something from scratch has its own fun :) . I
stared building Claymus because I felt need of Java based CMS which is
made
for GAE, considering what GAE is, how it is different from other hosting
services, services/apis GAE is providing to developers, etc. etc. etc.
Claymus is designed to give you maximum flexibility and support to build
your app on top of Claymus. Along with all the GAE features you can take
advantage of Servlet Level caching to minimize delay and cpu usage,
plugable
modules and themes, etc ... (complete list will be put on updated Claymus
Website :) ).
Regards,
Prashant
On Mon, Jul 18, 2011 at 7:28 PM, Niklas Rosencrantz nikla...@gmail.com
wrote:
Having something like a CMS eg like wordpress, joomla or mediawiki
compatible then we won't have to start every project from scratch and
project would become more configuration than programming which
naturally is
an advantage since programming takes more time than just configuring a
new
CMS deployment. I used web2py for a project when a friend wanted a
quick
website started and naturally to change pages we'd prefer a web-based
editor
instead of changing templates and redeploying. I'm glad to see some CMS
are
coming to app engine. Some frameworks offer features that are getting
close
to that of a CMS, for instance GAE Framework (www.gaeframework.com)
comes
with a blog engine and when you have a blog you
Re: [appengine-java] How to run quatz(kind) of scheduler service on appengine?
One more option: you can set ETAs in task queue tasks. That is, when some action occurs, you can say, Run this at 6:30am, and the job will run at roughly that time: http://code.google.com/appengine/docs/java/javadoc/com/google/appengine/api/taskqueue/TaskOptions.Builder.html#withEtaMillis(long) Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Fri, Jul 22, 2011 at 2:09 PM, Kesava Neeli nke...@gmail.com wrote: Yes.. I looked at Background instance. It allows to have long tasks but it doesn't allow to spawn threads. Quartz fires off events in threads. So I guess quartz is not an option on app-engine. We can use appengine cron jobs to some extent but if you need to fire an event at exact minute, it would be tough. It's tough to maintain 2 cloud providers for single product. Not sure how efficient it is to call appengine urls from EC2. May be move to EC2 completely may be better solution. We need object datastore for our product. Has anyone has experience with EC2 object datastore SimpleDB? Does it have same features and JPA/JDO interfaces like appengine. GAE datastore is pretty good but it doesn't give a VM ownership to the developer. The Backends in appengine still don't give any control of the server/instance to developer. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/nlrteFBD_2sJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Scheduler features and load time limitations
Hi Ikai, I cannot find answer to: 1) Will the limit of 30s to start an application be lifted? The other question seems to be answered since the changes in 1.5.2 (http://code.google.com/appengine/docs/adminconsole/ performancesettings.html#Setting_the_Minimum_Pending_Latency) On Jul 22, 4:24 pm, Ikai Lan (Google) ika...@google.com wrote: Have you taken a look at the scheduler that has shipped? It should be under Application Settings. If there's anything there that you think might be needed, please let us know. Ikai Lan Developer Programs Engineer, Google App Engine Blog:http://googleappengine.blogspot.com Twitter:http://twitter.com/app_engine Reddit:http://www.reddit.com/r/appengine On Mon, Jul 18, 2011 at 5:40 PM, Marcel Overdijk marceloverd...@gmail.comwrote: Everybody knows about load time issues when using frameworks that take a long time to startup. Take for example Grails. I wonder if we will still encounter these issues after the pricing changes. 1) Will the limit of 30s to start an application be lifted? 2) How smart will the scheduler be to startup new instances? Will it for example take into account the avg load time of previously started instances? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Scheduler features and load time limitations
I haven't heard any plans to lift the 30s load time to start an application. 30s+ loading time seems pretty bad to me ... Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Fri, Jul 22, 2011 at 2:37 PM, Marcel Overdijk marceloverd...@gmail.comwrote: Hi Ikai, I cannot find answer to: 1) Will the limit of 30s to start an application be lifted? The other question seems to be answered since the changes in 1.5.2 (http://code.google.com/appengine/docs/adminconsole/ performancesettings.html#Setting_the_Minimum_Pending_Latency) On Jul 22, 4:24 pm, Ikai Lan (Google) ika...@google.com wrote: Have you taken a look at the scheduler that has shipped? It should be under Application Settings. If there's anything there that you think might be needed, please let us know. Ikai Lan Developer Programs Engineer, Google App Engine Blog:http://googleappengine.blogspot.com Twitter:http://twitter.com/app_engine Reddit:http://www.reddit.com/r/appengine On Mon, Jul 18, 2011 at 5:40 PM, Marcel Overdijk marceloverd...@gmail.comwrote: Everybody knows about load time issues when using frameworks that take a long time to startup. Take for example Grails. I wonder if we will still encounter these issues after the pricing changes. 1) Will the limit of 30s to start an application be lifted? 2) How smart will the scheduler be to startup new instances? Will it for example take into account the avg load time of previously started instances? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Scheduler features and load time limitations
Yes might be, but frameworks like Grails need a longer startup time. It does not matter for any application server nor cloud provider except GAE. That's why I'm asking about future working of scheduler. Further, if I keep up enough instances I would not bother the startup time at all... I'm not saying 30s limit is bad or good, I'm just looking if using Grails on GAE will be feasible or not. On Jul 22, 9:06 pm, Ikai Lan (Google) ika...@google.com wrote: I haven't heard any plans to lift the 30s load time to start an application. 30s+ loading time seems pretty bad to me ... Ikai Lan Developer Programs Engineer, Google App Engine Blog:http://googleappengine.blogspot.com Twitter:http://twitter.com/app_engine Reddit:http://www.reddit.com/r/appengine On Fri, Jul 22, 2011 at 2:37 PM, Marcel Overdijk marceloverd...@gmail.comwrote: Hi Ikai, I cannot find answer to: 1) Will the limit of 30s to start an application be lifted? The other question seems to be answered since the changes in 1.5.2 (http://code.google.com/appengine/docs/adminconsole/ performancesettings.html#Setting_the_Minimum_Pending_Latency) On Jul 22, 4:24 pm, Ikai Lan (Google) ika...@google.com wrote: Have you taken a look at the scheduler that has shipped? It should be under Application Settings. If there's anything there that you think might be needed, please let us know. Ikai Lan Developer Programs Engineer, Google App Engine Blog:http://googleappengine.blogspot.com Twitter:http://twitter.com/app_engine Reddit:http://www.reddit.com/r/appengine On Mon, Jul 18, 2011 at 5:40 PM, Marcel Overdijk marceloverd...@gmail.comwrote: Everybody knows about load time issues when using frameworks that take a long time to startup. Take for example Grails. I wonder if we will still encounter these issues after the pricing changes. 1) Will the limit of 30s to start an application be lifted? 2) How smart will the scheduler be to startup new instances? Will it for example take into account the avg load time of previously started instances? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Scheduler features and load time limitations
What's the startup time on Grails now? I can ask about the 30s startup time, but I'm sure there are some pretty serious consequences to an app with a 40s-50s startup time. I'm a bit surprised, honestly, because even a JRuby on Rails app shouldn't take more than 30 seconds to startup. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Fri, Jul 22, 2011 at 3:12 PM, Marcel Overdijk marceloverd...@gmail.comwrote: Yes might be, but frameworks like Grails need a longer startup time. It does not matter for any application server nor cloud provider except GAE. That's why I'm asking about future working of scheduler. Further, if I keep up enough instances I would not bother the startup time at all... I'm not saying 30s limit is bad or good, I'm just looking if using Grails on GAE will be feasible or not. On Jul 22, 9:06 pm, Ikai Lan (Google) ika...@google.com wrote: I haven't heard any plans to lift the 30s load time to start an application. 30s+ loading time seems pretty bad to me ... Ikai Lan Developer Programs Engineer, Google App Engine Blog:http://googleappengine.blogspot.com Twitter:http://twitter.com/app_engine Reddit:http://www.reddit.com/r/appengine On Fri, Jul 22, 2011 at 2:37 PM, Marcel Overdijk marceloverd...@gmail.comwrote: Hi Ikai, I cannot find answer to: 1) Will the limit of 30s to start an application be lifted? The other question seems to be answered since the changes in 1.5.2 (http://code.google.com/appengine/docs/adminconsole/ performancesettings.html#Setting_the_Minimum_Pending_Latency) On Jul 22, 4:24 pm, Ikai Lan (Google) ika...@google.com wrote: Have you taken a look at the scheduler that has shipped? It should be under Application Settings. If there's anything there that you think might be needed, please let us know. Ikai Lan Developer Programs Engineer, Google App Engine Blog:http://googleappengine.blogspot.com Twitter:http://twitter.com/app_engine Reddit:http://www.reddit.com/r/appengine On Mon, Jul 18, 2011 at 5:40 PM, Marcel Overdijk marceloverd...@gmail.comwrote: Everybody knows about load time issues when using frameworks that take a long time to startup. Take for example Grails. I wonder if we will still encounter these issues after the pricing changes. 1) Will the limit of 30s to start an application be lifted? 2) How smart will the scheduler be to startup new instances? Will it for example take into account the avg load time of previously started instances? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Scheduler features and load time limitations
Honestly I don't know what the current startup time for Grails on GAE would be now. Also depending on what plugins installed etc. Current version does not even work on GAE. I was thinking about trying it out again but remembered the 30s limit I hit some time ago. So I asked this question on the forum to see if it makes any sense investing the time, or just continue with my Spring MVC app which runs fine. On Jul 22, 9:20 pm, Ikai Lan (Google) ika...@google.com wrote: What's the startup time on Grails now? I can ask about the 30s startup time, but I'm sure there are some pretty serious consequences to an app with a 40s-50s startup time. I'm a bit surprised, honestly, because even a JRuby on Rails app shouldn't take more than 30 seconds to startup. Ikai Lan Developer Programs Engineer, Google App Engine Blog:http://googleappengine.blogspot.com Twitter:http://twitter.com/app_engine Reddit:http://www.reddit.com/r/appengine On Fri, Jul 22, 2011 at 3:12 PM, Marcel Overdijk marceloverd...@gmail.comwrote: Yes might be, but frameworks like Grails need a longer startup time. It does not matter for any application server nor cloud provider except GAE. That's why I'm asking about future working of scheduler. Further, if I keep up enough instances I would not bother the startup time at all... I'm not saying 30s limit is bad or good, I'm just looking if using Grails on GAE will be feasible or not. On Jul 22, 9:06 pm, Ikai Lan (Google) ika...@google.com wrote: I haven't heard any plans to lift the 30s load time to start an application. 30s+ loading time seems pretty bad to me ... Ikai Lan Developer Programs Engineer, Google App Engine Blog:http://googleappengine.blogspot.com Twitter:http://twitter.com/app_engine Reddit:http://www.reddit.com/r/appengine On Fri, Jul 22, 2011 at 2:37 PM, Marcel Overdijk marceloverd...@gmail.comwrote: Hi Ikai, I cannot find answer to: 1) Will the limit of 30s to start an application be lifted? The other question seems to be answered since the changes in 1.5.2 (http://code.google.com/appengine/docs/adminconsole/ performancesettings.html#Setting_the_Minimum_Pending_Latency) On Jul 22, 4:24 pm, Ikai Lan (Google) ika...@google.com wrote: Have you taken a look at the scheduler that has shipped? It should be under Application Settings. If there's anything there that you think might be needed, please let us know. Ikai Lan Developer Programs Engineer, Google App Engine Blog:http://googleappengine.blogspot.com Twitter:http://twitter.com/app_engine Reddit:http://www.reddit.com/r/appengine On Mon, Jul 18, 2011 at 5:40 PM, Marcel Overdijk marceloverd...@gmail.comwrote: Everybody knows about load time issues when using frameworks that take a long time to startup. Take for example Grails. I wonder if we will still encounter these issues after the pricing changes. 1) Will the limit of 30s to start an application be lifted? 2) How smart will the scheduler be to startup new instances? Will it for example take into account the avg load time of previously started instances? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Correlative number
Im trying to get a kind of auto increment number for an entity. I all ready create a counter class to take a track of the quantity of entities so i can use the counter to get a correlative number , but its not working the right way, so wonder if anybody know how to implement these and if its posible to use a kind of mutex to block code. Thank, you. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Task Queue rate not high enough
I noticed a new field since 1.5.2 : Enforced Rate. Could that parameter help us make sure enough instance are created to process our tasks ? Maxime -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/v4XreKVTGg4J. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [google-appengine] Bad Performance for Dedupe of 2 million records using mapreduce on Appengine
Hi, Is data going to be added while this job is running? Because that is important to know. I like something like Stephen's second idea. Change your data model to use those four fields (or a hash of them) as the key name. Map over your existing data rewriting it. Once your done, there will be no dupes. If you also change the code that creates data to do the same you can transactionally check for a record before creating it or simply write the new record -- either way will prevent duplicates. This will scale to millions of entities with no issue and if there are no other fields your trying to merge will be cheaper since you do not need to query, just write the new entity each time. Robert On Jul 21, 2011 12:26 PM, Stephen Johnson onepagewo...@gmail.com wrote: Hi, I'll take a stab at trying to help you out. I think I understand what you are attempting to do. Using mapreduce: The way I would do this is to create (if you don't already have one) is a composite index on your four fields of name, phone, address and fathername. With that being said, since these are all equality type comparisons a zig-zag merge join perhaps would also do what you want, but the composite index should offer higher throughput for such a large operation with increased up front cost for the index creation. So, then for every entity that is mapped, I would do a keys-only query. This should return at least one key, the current entity. If multiple entities are returned, then I would delete the additional entities. Now, I don't think I'd add it to the datastore mutation pool since there would be a delay before they are deleted and then they could also get mapped which could delete the entity that was just de-duped. Now, with that being said, there seems to exist the possibility that another mapper could be working on another duplicate entity at the same time and thus could delete the entity we just used for dedupe purposes. There seem to be a couple of ways out of this. 1.) If the keys-only query is performed and only one key is returned and it isn't the key value of the entity we are deduping then obviously it was a duplicate itself that was deleted so we ignore this one. 2.) Or whenever duplicates are found a secondary table is used to store the key values of the duplicated entries as a list and using the 4 attributes of name, phone, address and fathername as they key. Then when the entire dataset has been processed. A second mapreduce would go through the second table to remove the duplicate entries. For each entry, one of the key values would be saved and the other key values would be used to delete the duplicate values. Perhaps a better option is using a backend server and cursor: I think another way to go might be to use a backend server and a cursor. Create the composite index described above and then query over this index using a sort order matching the composite index. Now, since these entries are in sort order, then all the duplicates should be next to each other so for each entity, if the values match the entity before it then it is a duplicate and it can be deleted. This is a much simpler approach but doesn't have the parallelism as the map reduce, but it is simpler. I'm not sure if either of these suggestions helps you out, but hopefully they might. Stephen Founder CortexConnect www.cortexconnect.com On Wed, Jul 20, 2011 at 8:07 PM, charming30 charmin...@gmail.com wrote: I have about 2 million records which have about 4 string fields each which needs to be checked for duplicates. To be more specific I have name, phone, address and fathername as fields and I must check for dedupe using all these fields with rest of data. The resulting unique records need to be noted into db. I have been able to implement mapreduce, iterarate of all records. Task rate is set to 100/s and bucket-size to 100. Billing enabled. Currently, everything is working, but performance is very very slow. I have been able to complete only 1000 records dedupe processing among a test dataset of 10,000 records in 6 hours. The current design in java is: In every map iteration, I compare the current record with the previous record - Previous record is a single record in db which acts like a global variable which I overwrite with another previous record in each map iteration - Comparison is done using an algorithm and result is written as a new entity to db - At the end of one Mapreduce job, i programatically create another job - The previous record variable helps the job to compare with next candidate record with rest of the data - I am ready to increase any amount of GAE resources to achieve this in shortest time. My Questions are: - Will the accuracy of dedupe (checking for duplicates) affect due to parallel jobs/tasks? - How can this design be improved? - Will this scale to 20 million records - Whats the fastest way to read/write variables (not just counters) during map iteration which
Re: [google-appengine] Re: Need help understanding Appengine billing
Every site and app are different. Without any idea of what you're doing there is no way to compare. What kind of traffic levels do you expect? Will each request hit the data store, how many times? Will there be back end processing involved? Robert On Jul 21, 2011 8:01 AM, Kwame iweg...@gmail.com wrote: Thanks Johan, I see your billing scheme, and I can sort of understand what the different terms mean, but it is difficult to wrap my brain around what the costs actually end up being in a real website. It would be helpful to get some examples of what other people have actually paid, and what kind of usage they have. I just need to make some rough estimates, so any examples will do. On Jul 21, 7:50 am, Johan Euphrosine pro...@google.com wrote: Hi Kwame, The current billing documentation is here: http://code.google.com/appengine/docs/billing.html However later this year we will introduce a new billing scheme which is described here: http://www.google.com/enterprise/cloud/appengine/pricing.html Community members that are willing to share metrics about their QPS, current quota usage and billing are more than welcome to comment to this threads. Hope that helps. On Thu, Jul 21, 2011 at 6:55 AM, Kwame iweg...@gmail.com wrote: I've been using Appengine for a while, and I haven't yet exhausted my free quota. I'm running a business and I would need to set a budget for my server costs. I'm not able to understand how the billing units like CPU Time, Outgoing Bandwidth, Incoming Bandwidth Total Stored data apply in real world usage of a typical busy website. Can anyone give me some sample charges for a typical website and approximately how much of these resources can be expected to be consumed, so I can compare with my site to get an approximate estimate. Thanks. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group athttp:// groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Mailing alerts with appengine
Hey Michal, You can get good performance while controling the rate by using a combination of the two methods. Split the list into several large chunks, inserting a processor for each chunk. Then each of those processors can run over their respective sets of addresses either mailing or inserting mailer tasks. Just be sure to persist your current position in the lists and use small batch sizes in your mailer tasks. If a task reruns for some reason it will reduce the number of people who get duplicate mails. Robert On Jul 20, 2011 5:45 AM, Michal Makarewicz makarew...@google.com wrote: A have a scenario where I have to send customized alerts to large amount of users (~30k) in certain, specified beforehand, time. My main idea was to make a cron job, which would check every minute if there is an alert to send and if so mark this alert as queued and add task to task queue (this would be done in transactional manner). Now I have two available approaches: Approach I: Have two different queue commands: 1. Split email list into smaller chunks (I'm not sure how big, I was thinking about ~500) and for each of them put second command on queue. (this one will be queued in cron job) 2. Send mail for each email address in the list. My main concern with this approach is lack of control on how many emails will be send simultaneously and possibility that my email will be classified as spam. Approach II: Just one task: 1. Send emails for predetermined number of people (again ~500) starting from start_index add to queue the same task with start_index + ~500. This approach will be slower, so is less preferable. I have pretty much unlimited quota so that wouldn't be a problem. Which approach is better in this situation? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Need help
Just a note, if you want help you need to provide enough information that someone can help you. On Jul 18, 2011 7:50 AM, Madhusudhan G Revankar sudhansa...@gmail.com wrote: Hi, I am facing problem in run/debug setting in eclipse for python(webapp). Not able to run on google app engine. Please help me on this -- Thanks Regards Madhusudhan -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Need help
I am experiencing GAE in eclipse.its working fine with java programming.But its not happening same with Python. While i run webapp using python. I am getting *error *as follows : *Status: 404 Not Found Content-Type: text/html; charset=utf-8 Cache-Control: no-cache Expires: Fri, 01 Jan 1990 00:00:00 GMT Content-Length: 0* Please let me know,where i am missing in configuration. On Fri, Jul 22, 2011 at 12:20 PM, Robert Kluin robert.kl...@gmail.comwrote: Just a note, if you want help you need to provide enough information that someone can help you. On Jul 18, 2011 7:50 AM, Madhusudhan G Revankar sudhansa...@gmail.com wrote: Hi, I am facing problem in run/debug setting in eclipse for python(webapp). Not able to run on google app engine. Please help me on this -- Thanks Regards Madhusudhan -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Thanks Regards Madhusudhan -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: GAE starting unnecessary instances
@Johan, The issue is not about Always On instance being busy. Its actually the other way ... the Always On instance is never busy ... at least that is what we observed in last 3-4 days. Your explanation may be partly true since this behavior keeps on changing. For e.g. I have a snapshot of instances from July 19th and here's the details (for some reason I can't see a link to attach the snapshot images here): Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr This is under no load with only very light weight cron jobs running. This gets much much worse during the day under peak load with requests for dynamic instances reaching 1000+ in matter of minutes and resident instances have only 1 request served. As you see above Resident Instance 2 and 3 are hardly hit so I don't think they are busy at all. On the other hand, Dynamic Instance 2 and 3 get most of the hits. Dynamic Instance 1 is what is killing us. It keeps getting killed and reborn within that 5 minute window!! We use Spring framework and it is really very expensive for us when a new instance starts up. Just to give you a background, we had gone through a real roller coaster ride to make this to work on GAE by breaking the loading of framework into many different chunks. But still spinning was out of control. Then we found java threads to our rescue. We worked through the hack to load JDO to avoid UnsupportedOperationException. We finally got it to work where most of our requests were served by Always On instances with occasional spinning of Dynamic instances. It was quite impressive. Unfortunately, this was short lived when we hit this new behavior with GAE. The very last thing we want GAE to do is create a new instance every few minutes as it could easily reach 30 second deadline during the day and throw critical error. I am not sure when the new billing will come into effect but we really need this thing fixed as it literally brings down our app to a grinding halt. So I am open to any suggestions you guys think can help us. Another thought about new scheduler is to have a configurable schedule. For e.g. our users are mostly business users who work during normal business hours. We want to be able to spin more Always On instances during those hours and bring the number down during nights and weekends. Dynamic instances won't work for us due to reason explained above. Thanks, galoch On Jul 21, 5:56 pm, Johan Euphrosine pro...@google.com wrote: After speaking with Engs, I think I can explain what is going on: Here are the current scheduling rules: ( reads as has priority for handling the incoming request) 1/ Idle Always On instance Spawning a new Dynamic instance 2/ Spawning a new Dynamic instance Busy Always On instance 3/ Idle Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance I will give you an example to illustrate the behavior you all noticed, that is Dynamic instance handling request while Always On is idle. (Always On instance started) - Incoming request - Always On instance handle the request - another Incoming request (Always On instance busy) - A new Dynamic instance is spawned (Dynamic instance idle, Always on instance busy) - Dynamic instance handle the request - another Incoming request (Dynamic instance idle, Always on instance idle) - Dynamic instance handle the request - No request for more than idle-dynamic-instance-timeout - Dynamic instance shut down - another Incoming request (Always On instance idle) - Always On instance handle the request Hope it makes thing clearer. As part of the new billing model you will have a scheduler knob called 'max-idle-instances' that you can use if extra idling dynamic instances are undesired. The good news is that we are open to suggestion, if you think this behavior is the wrong default, feel free to comment on that thread and I will follow up your suggestion to the Engineering team. On Wed, Jul 20, 2011 at 12:18 AM, Galoch galoch...@gmail.com wrote: Same here. Seems like GAE is totally ignoring Always On instances. I also noticed that even with no user hitting our app and a single cron job that runs every 5 minutes it is still spinning instances every 3 minutes and then killing them in 2 minutes. This has been happening since after the upgrade on 14th July. During peak load this really gets nasty and brings down the performance. This is the feedback I got yesterday from one of our customers since it takes time to spin an instance (and yes we use Spring): 1) I found the GUI to be very laggy Can someone from Google please respond? -- You received this message because you are subscribed to the Google Groups Google App Engine
[google-appengine] SDK Upgrade 1.5.2: --datastore_path ignored?
Hi, After today's upgrade my datastore is empty and my testdata is not available. I start my SDK Server like this (usually with the MacOS Launcher): dev_appserver.py --datastore_path=/Users/cat/repositories/appengine/ my.datastore -p8080 . Strangely --datastore_path is ignored and the dev_appserver uses a new datastore in the tmp directory: INFO 2011-07-22 07:11:11,606 rdbms_sqlite.py:58] Connecting to SQLite database '' with file '/var/folders/u5/u5xmrm5gHPGXhfjlyv98u+++ +TI/-Tmp-/dev_appserver.rdbms' Can anyone tell me which flags to set to get my old test datastore back? I would have to create lots of testdata manually that accumlated over the last months. Regards, Cat -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: GAE starting unnecessary instances
Still very close from Galoch situation, mine is a bit different. My Dynamic instances dont seem to be recycled but still my Resident instances dont seem to be used at all (see attachment). Why ? Francois -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/_r0g9FOzxdwJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. attachment: Instances-VinoCities.jpg
Re: [google-appengine] Re: 1.5.2 is out!
@gops, You should join google-appengine-go. Go is unbelievable fast on GAE; the hello world takes 104ms to start a new instance and much less in the next requests. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/N1aoM55YawQJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Verification with SMS is failing
I got this resolved. Looks like some security/firewall issue in one of office machine. Now I installed Google App engine plugin in my home desktop eclipse and deployed the app. It works fine now. On Thu, Jul 21, 2011 at 10:44 PM, Veerendra Shukla mobilegeni...@gmail.comwrote: I created 3 apps. but when I mention one of the app while eclipse google deploy, none of them work... and I get the earlier mentioned error.. mobinexttryhttps://appengine.google.com/dashboard?app_id=s~mobinexttrymobinextHigh ReplicationNone Deployed mobisample https://appengine.google.com/dashboard?app_id=s~mobisamplemobisampleHigh ReplicationNone Deployed mobitry https://appengine.google.com/dashboard?app_id=s~mobitry First Time Mobile SMS High Replication None Deployed On Thu, Jul 21, 2011 at 10:39 PM, Robert Kluin robert.kl...@gmail.comwrote: You need to login at appspot.com and register your application. On Jul 22, 2011 1:13 AM, Veerendra Shukla mobilegeni...@gmail.com wrote: I used my colleague's cell phone to get this verified. Now I am getting following error when I am deploying the app in google app. Please visit http://code.google.com/appengine for the latest SDK. Created staging directory at: 'C:\Users\vshukla\AppData\Local\Temp\2\appcfg6442456847751386259.tmp' Scanning for jsp files. Scanning files on local disk. Initiating update. java.io.IOException: Error posting to URL: https://appengine.google.com/api/appversion/create?app_id=mobitryversion=1; 404 Not Found This application does not exist (app_id=u'mobitry'). Debugging information may be found in C:\Users\vshukla\AppData\Local\Temp\2\appengine-deploy7314225494612513236.log How should I get this resolved? On Thu, Jul 21, 2011 at 12:23 PM, Jose Montes de Oca jfmontesde...@google.com wrote: Hi, Please fill the following form: https://appengine.google.com/**waitlist/sms_issues https://appengine.google.com/waitlist/sms_issues As explained in the FAQ for SMS Verification for App Creation: http://code.google.com/**appengine/kb/sms.html http://code.google.com/appengine/kb/sms.html Hope this helps. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/my6HhKCPK_8J. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: New Channel API pricing scheme
Someone please correct me if I am wrong, but looks like the limit was changed drastically. Now I see 8,640 channels limit for free apps and 95,040 for accounts with billing enabled. Now I think we can use it. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/mSoRRuwudAoJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: SDK Upgrade 1.5.2: --datastore_path ignored?
Update: (I'm going nuts) I downgraded to 1.5.1 (restored GoogleAppEngineLauncher.app) and still the same now ... It seems that the upgrade to 1.5.2 did some permanent damage. I am absolutely puzzled why --datastore_path= is ignored suddenly. Here the full startup log: cat$ /Applications/GoogleAppEngineLauncher.app/Contents/Resources/ GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/ dev_appserver.py -d --datastore_path=/Users/cat/repositories/appengine/ my.datastore -p 8080 . INFO 2011-07-22 09:15:00,330 appengine_rpc.py:159] Server: appengine.google.com INFO 2011-07-22 09:15:21,578 rdbms_sqlite.py:58] Connecting to SQLite database '' with file '/var/folders/u5/u5xmrm5gHPGXhfjlyv98u+++ +TI/-Tmp-/dev_appserver.rdbms' INFO 2011-07-22 09:15:21,618 dev_appserver_multiprocess.py:637] Running application test on port 8080: http://localhost:8080 On 22 Jul., 09:22, Cat katz...@gmail.com wrote: Hi, After today's upgrade my datastore is empty and my testdata is not available. I start my SDK Server like this (usually with the MacOS Launcher): dev_appserver.py --datastore_path=/Users/cat/repositories/appengine/ my.datastore -p8080 . Strangely --datastore_path is ignored and the dev_appserver uses a new datastore in the tmp directory: INFO 2011-07-22 07:11:11,606 rdbms_sqlite.py:58] Connecting to SQLite database '' with file '/var/folders/u5/u5xmrm5gHPGXhfjlyv98u+++ +TI/-Tmp-/dev_appserver.rdbms' Can anyone tell me which flags to set to get my old test datastore back? I would have to create lots of testdata manually that accumlated over the last months. Regards, Cat -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Google app engine throwing NoSuchMethodError suddenly
Any update on this?? On Jul 19, 2:49 pm, Prateek Mathur prateek.mat...@metacube.com wrote: Hi, I started seeing the following exception on my production builds suddenly and my application is broken because of this. java.lang.NoSuchMethodError: com.google.appengine.repackaged.com.google.common.base.StringUtil.isEmpty(L java/ lang/String;)Z at com.metacube.ipathshala.manager.util.SearchRequestAssembler.assemble(Search RequestAssembler.java: 129) at com.metacube.ipathshala.manager.SearchManager.doSearch(SearchManager.java: 70) Was there any changes in the runtime sdk for theStringUtilclass..It seems to indicate that the method doesnt exist anymore, while few days back the application was running fine with the same api... Any news on what might have happened? Thanks, Prateek. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
回复：Re: [google-appengine] How to select version of Django when using 'import template'
I've tried adding
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
from google.appengine.dist import use_library
use_library('django', '1.1')
to my application ,but it reports 'No module named django'.
What's wrong?
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/jiB76qQGteQJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Transactions and ancestor in a different namespace
Is it anywhere explicitly documented that the ancestor of an entity must have it's key in the same namespace as the entity itself? I'm using namespaces a lot in our app and in production I get an error if I try to do a transactional update of an entity with parent x in a different namespace than the entity that I'm updating. Is this intentional? If it is, I guess the dev server should also check this, my unit tests pass :) Hugo -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] App Engine Weekly Community Update #1
*App Engine Weekly Community Update #1* Dear App Engine Community, As discussed on IRC, each week I will bring some news and metrics about the App Engine community activity. Thanks a lot to +Rodrigo Moraes for reviewing this with his sharp journalist eyes. *Highlights* 1.5.2 Release Notes http://googleappengine.blogspot.com/2011/07/app-engine-152-sdk-released.html Postmortem: Java App Engine outage, July 14, 2011 http://googleappengine.blogspot.com/2011/07/postmortem-java-app-engine-outage-july.html Go for App Engine is now generally available http://blog.golang.org/2011/07/go-for-app-engine-is-now-generally.html Tipfy is looking for a new maintainer http://groups.google.com/group/tipfy/browse_thread/thread/1359e76c69e89060 *Hot questions* Flask vs webapp2 for Google App Engine http://stackoverflow.com/questions/6774371/flask-vs-webapp2-for-google-app-engine Bad Performance for Dedupe of 2 million records using mapreduce on Appengine http://stackoverflow.com/questions/6770781/bad-performance-for-dedupe-of-2-million-records-using-mapreduce-on-appengine What is the best way of describing day of week? http://stackoverflow.com/questions/6763623/what-is-the-best-way-of-describing-day-of-week *IRC and Hangouts* IRC Office Hours on Wednesday, transcript there: https://groups.google.com/d/topic/google-appengine/ZzUDaN226bU/discussion +Guido van Rossum and +Ikai Lan held community hangouts and posted some followup on Google+: https://plus.google.com/u/1/115212051037621986145/posts/TmU4Sdx7yT3 https://plus.google.com/u/1/107011265359512082824/posts/42G7ZiXkAtR *1:1 with one member of the App Engine team* Who? Sumit Arora aka sumeets@, I'm with the App Engine SRE team What? As SRE, our mission is to balance availability, reliability and performance with agility of our services. Which essentially means: we work on developing and deploying systems that help us measure availability and reliability; monitor outages or degradation of service as well as capacity planning. In addition, we gatekeep all changes going to our production infrastructure, to ensure changes are rolled out without impacting end users Where? I work out of the Sydney Australia Office. When? I've been with this team for about 6 months now. Why? I'm passionate about the cloud computing domain in general... and App Engine represents a pioneer in the that domain, because it gives developers the means to spend their time where it matters the most, i.e. developing applications and not worrying about infrastructure to run it on Emacs or Vi? vi Python, Java or Go? Python and Java In that order. I will use python over java any day mostly because of my sysadmin background. And I’m just getting started with Go. Recommended App Engine framework, library or tool ? Django, without a doubt. Fun thought? I find it very ironic and amusing that every time a smoke detector goes off, the first thing people want to do is fix the smoke detector. There is nothing wrong with it, it's doing what its supposed to be doing... Fix the fire instead! Sadly this is true of monitoring and alerting as well, in large. *Issue tracking* 47 new issues were reported in the public issue tracker: - 31 Defects - 10 Production issues - 6 Feature request - 20 were Acknowledged and still need to be reproduced - 10 have been Accepted and escalated to the Engineer team when necessary - 17 of those issues were not actionable (Invalid, WontFix or Duplicate) 19 issues (including 8 from this week) have been fixed. All the Productions issues backlog (68 issues) has been triaged, and the New queue for Production issue is now empty. *Groups* [appengine-python] 115 messages in 25 threads 64 % of posts answered within 2 days Top Posters: - robert.kluin (9) - proppy (7) - ninjin32 (7) [appengine-java] 177 messages in 51 threads 43% of posts answered within 2 days Top Posters: - ikai (11) - venugopal.vassiredy (6) - durant.didier (4) [appengine] 330 messages in 84 threads 61% of posts answered within 2 days Top Posters: - robert.kluin (20) - proppy (16) - jeff (7) Feedbacks are welcome on this new format, let me know what you are looking forward to see in next week App Engine Community Update. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] [OAuth + Provisioning ] Autentication - Python X Java
Hi, I need help to use Provisioning API with OAuth authentication, the following problem occurs, how to do domain validation in java api ? In Python it is possible to know the following code. service = gdata.apps.service.AppsService(domain=consumer_key) *-*service.SetOAuthInputParameters(sig_method, consumer_key, consumer_secret=consumer_secret, two_legged_oauth=True) res = service.RetrieveAllUsers() for entry in res.entry: print 'email;' + string.lower(entry.login.user_name) + '@' + consumer_key the emphasis on the coding can be seen that there is validation for key parametersusing OAuth authentication as CONSUMER_KEY and CONSUMER_SECRET, but inJava this is not possible .. As shown below: AppsForYourDomainClient client = new AppsForYourDomainClient(adminEmail, adminPassword, domain); //JavaCode Api - AppsForYourDomain how can I get something close to in java: AppsForYourDomainClient client = null; // client.SetOAuthInputParameters(sig_method, consumer_key, consumer_secret); //this method does not exist today in the api Thanks... -- *Felipe Teixeira* *Desenvolvedor Java * IPNET Soluções Tel.: 55 21 3553 2717 / 21-7805-6867 Rua Visconde de Inhaúma, 134, Sala 615 Centro - RJ - CEP: 20091-007 *http://www.ipnetsolucoes.com.br** * * * -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] App Engine Weekly Community Update #1
61% of posts answered within 2 days 1. I am glad that you are tracking this. 2. That's NOT a good percentage. Since this is your main support channel, this should be 99.995% in 1 day. I'm not kidding. If you want to be in the software business, walk the walk. 3. I'm skeptical about your measurement technique. Answered doesn't mean answered unless the problem is actually resolved. The pattern I usually see is: - I post a question - You ask for some more information - I provide that information - Crickets. My bet is that you are counting that as answered since you responded to the initial question. But it ain't answered in the stack overflow sense of the word. And that sense is all that matters. Let's hope that when you come out of preview and start charging us all that money, you step it up a bit on the support side. I know that customer support has about negative priority at google, so I recognize that it will be a challenge. -Joshua -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] [OAuth + Provisioning ] Autentication - Python X Java
Hi, I need help to use Provisioning API with OAuth authentication, the following problem occurs, how to do domain validation in java api ? In Python it is possible to know the following code. service = gdata.apps.service.AppsService(domain=consumer_key) *-*service.SetOAuthInputParameters(sig_method, consumer_key, consumer_secret=consumer_secret, two_legged_oauth=True) res = service.RetrieveAllUsers() for entry in res.entry: print 'email;' + string.lower(entry.login.user_name) + '@' + consumer_key the emphasis on the coding can be seen that there is validation for key parametersusing OAuth authentication as CONSUMER_KEY and CONSUMER_SECRET, but inJava this is not possible .. As shown below: AppsForYourDomainClient client = new AppsForYourDomainClient (adminEmail, adminPassword, domain); //JavaCode Api - AppsForYourDomain how can I get something close to in java: AppsForYourDomainClient client = null; // client.SetOAuthInputParameters(sig_method, consumer_key, consumer_secret); //this method does not exist today in the api java Thanks... -- *Felipe Teixeira* *Desenvolvedor Java * IPNET Soluções Tel.: 55 21 3553 2717 / 21-7805-6867 Rua Visconde de Inhaúma, 134, Sala 615 Centro - RJ - CEP: 20091-007 *http://www.ipnetsolucoes.com.br** * * * -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Need help understanding Appengine billing
Thanks Robert. The reason I ask in such a general way is because it is difficult to make such estimates at this time. The site is still new so it doesn't have much traffic. I have accountant types breathing down my neck asking for ballpark numbers! So, basically I just need a scenario of any website on appspot, which has some metrics and also can provide the amount they spend monthly on Appengine. This way my guys can make projections about future costs, based on projected use. This will also have implications for design modifications. Thanks for you help Robert. On Jul 22, 2:08 am, Robert Kluin robert.kl...@gmail.com wrote: Every site and app are different. Without any idea of what you're doing there is no way to compare. What kind of traffic levels do you expect? Will each request hit the data store, how many times? Will there be back end processing involved? Robert On Jul 21, 2011 8:01 AM, Kwame iweg...@gmail.com wrote: Thanks Johan, I see your billing scheme, and I can sort of understand what the different terms mean, but it is difficult to wrap my brain around what the costs actually end up being in a real website. It would be helpful to get some examples of what other people have actually paid, and what kind of usage they have. I just need to make some rough estimates, so any examples will do. On Jul 21, 7:50 am, Johan Euphrosine pro...@google.com wrote: Hi Kwame, The current billing documentation is here: http://code.google.com/appengine/docs/billing.html However later this year we will introduce a new billing scheme which is described here: http://www.google.com/enterprise/cloud/appengine/pricing.html Community members that are willing to share metrics about their QPS, current quota usage and billing are more than welcome to comment to this threads. Hope that helps. On Thu, Jul 21, 2011 at 6:55 AM, Kwame iweg...@gmail.com wrote: I've been using Appengine for a while, and I haven't yet exhausted my free quota. I'm running a business and I would need to set a budget for my server costs. I'm not able to understand how the billing units like CPU Time, Outgoing Bandwidth, Incoming Bandwidth Total Stored data apply in real world usage of a typical busy website. Can anyone give me some sample charges for a typical website and approximately how much of these resources can be expected to be consumed, so I can compare with my site to get an approximate estimate. Thanks. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group athttp:// groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: SDK Upgrade 1.5.2: --datastore_path ignored?
I'm having a problem that may be related. I'm using django-nonrel which sets the datastore path to .gaedata in the project directory. When I start the server I get a log message that has the correct path: INFO:root:Connecting to SQLite database '' with file '/Users/me/workspace/myapp/.gaedata/rdbms' But when the app starts the datastore is empty. -Chris On Fri, Jul 22, 2011 at 4:19 AM, Cat katz...@gmail.com wrote: Update: (I'm going nuts) I downgraded to 1.5.1 (restored GoogleAppEngineLauncher.app) and still the same now ... It seems that the upgrade to 1.5.2 did some permanent damage. I am absolutely puzzled why --datastore_path= is ignored suddenly. Here the full startup log: cat$ /Applications/GoogleAppEngineLauncher.app/Contents/Resources/ GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/ dev_appserver.py -d --datastore_path=/Users/cat/repositories/appengine/ my.datastore -p 8080 . INFO 2011-07-22 09:15:00,330 appengine_rpc.py:159] Server: appengine.google.com INFO 2011-07-22 09:15:21,578 rdbms_sqlite.py:58] Connecting to SQLite database '' with file '/var/folders/u5/u5xmrm5gHPGXhfjlyv98u+++ +TI/-Tmp-/dev_appserver.rdbms' INFO 2011-07-22 09:15:21,618 dev_appserver_multiprocess.py:637] Running application test on port 8080: http://localhost:8080 On 22 Jul., 09:22, Cat katz...@gmail.com wrote: Hi, After today's upgrade my datastore is empty and my testdata is not available. I start my SDK Server like this (usually with the MacOS Launcher): dev_appserver.py --datastore_path=/Users/cat/repositories/appengine/ my.datastore -p8080 . Strangely --datastore_path is ignored and the dev_appserver uses a new datastore in the tmp directory: INFO 2011-07-22 07:11:11,606 rdbms_sqlite.py:58] Connecting to SQLite database '' with file '/var/folders/u5/u5xmrm5gHPGXhfjlyv98u+++ +TI/-Tmp-/dev_appserver.rdbms' Can anyone tell me which flags to set to get my old test datastore back? I would have to create lots of testdata manually that accumlated over the last months. Regards, Cat -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: SDK Upgrade 1.5.2: --datastore_path ignored?
Cat, do you have backups of you datastore files from before you ran 1.5.2? I was able to get running again by downgrading the SDK to 1.5.1 and restoring my data files from a backup. -Chris On Fri, Jul 22, 2011 at 4:19 AM, Cat katz...@gmail.com wrote: Update: (I'm going nuts) I downgraded to 1.5.1 (restored GoogleAppEngineLauncher.app) and still the same now ... It seems that the upgrade to 1.5.2 did some permanent damage. I am absolutely puzzled why --datastore_path= is ignored suddenly. Here the full startup log: cat$ /Applications/GoogleAppEngineLauncher.app/Contents/Resources/ GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/ dev_appserver.py -d --datastore_path=/Users/cat/repositories/appengine/ my.datastore -p 8080 . INFO 2011-07-22 09:15:00,330 appengine_rpc.py:159] Server: appengine.google.com INFO 2011-07-22 09:15:21,578 rdbms_sqlite.py:58] Connecting to SQLite database '' with file '/var/folders/u5/u5xmrm5gHPGXhfjlyv98u+++ +TI/-Tmp-/dev_appserver.rdbms' INFO 2011-07-22 09:15:21,618 dev_appserver_multiprocess.py:637] Running application test on port 8080: http://localhost:8080 On 22 Jul., 09:22, Cat katz...@gmail.com wrote: Hi, After today's upgrade my datastore is empty and my testdata is not available. I start my SDK Server like this (usually with the MacOS Launcher): dev_appserver.py --datastore_path=/Users/cat/repositories/appengine/ my.datastore -p8080 . Strangely --datastore_path is ignored and the dev_appserver uses a new datastore in the tmp directory: INFO 2011-07-22 07:11:11,606 rdbms_sqlite.py:58] Connecting to SQLite database '' with file '/var/folders/u5/u5xmrm5gHPGXhfjlyv98u+++ +TI/-Tmp-/dev_appserver.rdbms' Can anyone tell me which flags to set to get my old test datastore back? I would have to create lots of testdata manually that accumlated over the last months. Regards, Cat -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Google app engine throwing NoSuchMethodError suddenly
Are you using the version of StringUtils that is in com.repackaged? If so, I'd advise you import a version of that into WEB-INF/lib. I'll check if it has changed, but in general, you probably shouldn't depend on the stuff that is in com.repackaged (I'll see if this is our policy and get it added to the docs if so). Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Fri, Jul 22, 2011 at 5:28 AM, Prateek Mathur prateek.mat...@metacube.com wrote: Any update on this?? On Jul 19, 2:49 pm, Prateek Mathur prateek.mat...@metacube.com wrote: Hi, I started seeing the following exception on my production builds suddenly and my application is broken because of this. java.lang.NoSuchMethodError: com.google.appengine.repackaged.com.google.common.base.StringUtil.isEmpty(L java/ lang/String;)Z at com.metacube.ipathshala.manager.util.SearchRequestAssembler.assemble(Search RequestAssembler.java: 129) at com.metacube.ipathshala.manager.SearchManager.doSearch(SearchManager.java: 70) Was there any changes in the runtime sdk for theStringUtilclass..It seems to indicate that the method doesnt exist anymore, while few days back the application was running fine with the same api... Any news on what might have happened? Thanks, Prateek. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: SDK Upgrade 1.5.2: --datastore_path ignored?
+1 for me. I have tried deleting the application plist as detailed in this issue http://code.google.com/p/googleappengine/issues/detail?id=1187q=datastore_pathcolspec=ID%20Type%20Component%20Status%20Stars%20Summary%20Language%20Priority%20Owner%20Log but no joy. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] local Google Datastore Service
You can access the local datastore admin at: http://localhost:PORT/_ah/admin Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Thu, Jul 21, 2011 at 2:37 PM, Jamil Rzayev jamil.rza...@gmail.comwrote: Dear Folks, I am developing GWT-based application where I need to store/retrieve data at GAE. I decided to use Google Datastore low-level API. The service does work well, but I need to test my CRUD operations locally not always deploying to Google App Engine. Could you please recommend me what I can use for that? It is part of my thesis and is very important for me to do it well. thank you. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: For deployment only does the sdk version matter?
Only your version matters. However, it is a good idea to stay somewhat up to date, as some APIs (experimental ones) get deprecated. One more thing: I tried running a really old application a while ago that was developed using an SDK version that was 1.1.x, and it wouldn't work with the newest SDK. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Mon, Jul 18, 2011 at 3:49 PM, Ernesto Oltra ernestoka...@gmail.comwrote: As far as my knowledge goes, you upload the source code (allowing downloads of it later), and its compiled, precompiled or whatever (Python/Java/Go...) in their machines; so the only problem here is you might be using a feature that has change their behaviour since the version you locally tested it. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/3atHYio-ZoEJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] App Engine Weekly Community Update #1
On Fri, Jul 22, 2011 at 3:26 PM, Joshua Smith joshuaesm...@charter.net wrote: Hi Joshua, 61% of posts answered within 2 days 1. I am glad that you are tracking this. I'm glad that you find interest into those metrics. 2. That's NOT a good percentage. Since this is your main support channel, this should be 99.995% in 1 day. I'm not kidding. If you want to be in the software business, walk the walk. The purpose of the groups is for App Engine developer to help each others, as per the group description: Discussion group for Google App Engine. For Python-only dialog, see the Python group, same goes for Java. Also see Downtime Notify for maintenance info. API Gurus: experienced developers who watch this group, answering questions providing valuable feedback: Geoffrey (Wooble) Spear Robert Kluin And the purpose of this metrics was to measure level of engagement and reactivity of the community as a whole (not only Googler) in the different threads. 3. I'm skeptical about your measurement technique. Answered doesn't mean answered unless the problem is actually resolved. The pattern I usually see is: - I post a question - You ask for some more information - I provide that information - Crickets. My bet is that you are counting that as answered since you responded to the initial question. But it ain't answered in the stack overflow sense of the word. And that sense is all that matters. I'd love to track the actual resolution of questions but we miss metadata to check if a question has been answered. Maybe we (as the community) can agree on some convention were we use nickname++ people as a signal of the question of the current thread being answered, or we (community) can decide to shift those question to Stack Overflow and keep the group for more casual discussion. I'd love to hear what others think about that. Let's hope that when you come out of preview and start charging us all that money, you step it up a bit on the support side. I know that customer support has about negative priority at google, so I recognize that it will be a challenge. If you take a look a the new pricing model we announced: http://www.google.com/enterprise/cloud/appengine/pricing.html You will notice that we mention SLA and Operational support for Paid and Premium app, which is a strong indicator that we are serious about this. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: GAE starting unnecessary instances
HI Galoch, Thanks for the followup, I think you are experiencing a combinaison fo the two following rules I was pointing to in my previous email: ( reads as has priority for handling the incoming request) 2/ Spawning a new Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance Applied to your example it could means that: Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr - 1 Hours ago while all your Always On instance were busy and you had a burst of incoming requests and the scheduler spawned new Dynamic instances as per rule 2/ highlighted above. - After the burst and back to normal traffic the new Dynamic Instances were handing incoming requests in priority as per rule 4/ highlighted above. - 2 Minutes ago all your instances Always On + Dynamic were busy again and the scheduler spawned a new Dynamic instance that handle 7 incoming requests. Hope that make more sense for you and Francois, but as I said earlier we are open to suggestion and I will make sure someone working on the scheduler team monitor this thread for your input. On Fri, Jul 22, 2011 at 9:09 AM, Galoch galoch...@gmail.com wrote: @Johan, The issue is not about Always On instance being busy. Its actually the other way ... the Always On instance is never busy ... at least that is what we observed in last 3-4 days. Your explanation may be partly true since this behavior keeps on changing. For e.g. I have a snapshot of instances from July 19th and here's the details (for some reason I can't see a link to attach the snapshot images here): Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr This is under no load with only very light weight cron jobs running. This gets much much worse during the day under peak load with requests for dynamic instances reaching 1000+ in matter of minutes and resident instances have only 1 request served. As you see above Resident Instance 2 and 3 are hardly hit so I don't think they are busy at all. On the other hand, Dynamic Instance 2 and 3 get most of the hits. Dynamic Instance 1 is what is killing us. It keeps getting killed and reborn within that 5 minute window!! We use Spring framework and it is really very expensive for us when a new instance starts up. Just to give you a background, we had gone through a real roller coaster ride to make this to work on GAE by breaking the loading of framework into many different chunks. But still spinning was out of control. Then we found java threads to our rescue. We worked through the hack to load JDO to avoid UnsupportedOperationException. We finally got it to work where most of our requests were served by Always On instances with occasional spinning of Dynamic instances. It was quite impressive. Unfortunately, this was short lived when we hit this new behavior with GAE. The very last thing we want GAE to do is create a new instance every few minutes as it could easily reach 30 second deadline during the day and throw critical error. I am not sure when the new billing will come into effect but we really need this thing fixed as it literally brings down our app to a grinding halt. So I am open to any suggestions you guys think can help us. Another thought about new scheduler is to have a configurable schedule. For e.g. our users are mostly business users who work during normal business hours. We want to be able to spin more Always On instances during those hours and bring the number down during nights and weekends. Dynamic instances won't work for us due to reason explained above. Thanks, galoch On Jul 21, 5:56 pm, Johan Euphrosine pro...@google.com wrote: After speaking with Engs, I think I can explain what is going on: Here are the current scheduling rules: ( reads as has priority for handling the incoming request) 1/ Idle Always On instance Spawning a new Dynamic instance 2/ Spawning a new Dynamic instance Busy Always On instance 3/ Idle Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance I will give you an example to illustrate the behavior you all noticed, that is Dynamic instance handling request while Always On is idle. (Always On instance started) - Incoming request - Always On instance handle the request - another Incoming request (Always On instance busy) - A new Dynamic instance is spawned (Dynamic instance idle, Always on instance busy) - Dynamic instance handle the request - another Incoming request (Dynamic instance
[google-appengine] [OAuth + Provisioning ] Autentication - Python X Java
Hi, I need help to use Provisioning API with OAuth authentication, the following problem occurs, how to do domain validation in java api ? In Python it is possible to know the following code. service = gdata.apps.service.AppsService(domain=consumer_key) *-*service.SetOAuthInputParameters(sig_method, consumer_key, consumer_secret=consumer_secret, two_legged_oauth=True) res = service.RetrieveAllUsers() for entry in res.entry: print 'email;' + string.lower(entry.login.user_name) + '@' + consumer_key the emphasis on the coding can be seen that there is validation for key parametersusing OAuth authentication as CONSUMER_KEY and CONSUMER_SECRET, but inJava this is not possible .. As shown below: AppsForYourDomainClient client = new AppsForYourDomainClient (adminEmail, adminPassword, domain); //JavaCode Api - AppsForYourDomain how can I get something close to in java: AppsForYourDomainClient client = null; // client.SetOAuthInputParameters(sig_method, consumer_key, consumer_secret); //this method does not exist today in the api java Thanks... -- *Felipe Teixeira* *Desenvolvedor Java * IPNET Soluções Tel.: 55 21 3553 2717 / 21-7805-6867 Rua Visconde de Inhaúma, 134, Sala 615 Centro - RJ - CEP: 20091-007 *http://www.ipnetsolucoes.com.br** * -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: GAE starting unnecessary instances
When are threaded instances considered busy? If it is while they are serving only a one request it would explain why Always on instances for threaded Java are now severely under- utilized. /Tom On Jul 22, 10:57 am, Johan Euphrosine pro...@google.com wrote: HI Galoch, Thanks for the followup, I think you are experiencing a combinaison fo the two following rules I was pointing to in my previous email: ( reads as has priority for handling the incoming request) 2/ Spawning a new Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance Applied to your example it could means that: Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr - 1 Hours ago while all your Always On instance were busy and you had a burst of incoming requests and the scheduler spawned new Dynamic instances as per rule 2/ highlighted above. - After the burst and back to normal traffic the new Dynamic Instances were handing incoming requests in priority as per rule 4/ highlighted above. - 2 Minutes ago all your instances Always On + Dynamic were busy again and the scheduler spawned a new Dynamic instance that handle 7 incoming requests. Hope that make more sense for you and Francois, but as I said earlier we are open to suggestion and I will make sure someone working on the scheduler team monitor this thread for your input. On Fri, Jul 22, 2011 at 9:09 AM, Galoch galoch...@gmail.com wrote: @Johan, The issue is not about Always On instance being busy. Its actually the other way ... the Always On instance is never busy ... at least that is what we observed in last 3-4 days. Your explanation may be partly true since this behavior keeps on changing. For e.g. I have a snapshot of instances from July 19th and here's the details (for some reason I can't see a link to attach the snapshot images here): Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr This is under no load with only very light weight cron jobs running. This gets much much worse during the day under peak load with requests for dynamic instances reaching 1000+ in matter of minutes and resident instances have only 1 request served. As you see above Resident Instance 2 and 3 are hardly hit so I don't think they are busy at all. On the other hand, Dynamic Instance 2 and 3 get most of the hits. Dynamic Instance 1 is what is killing us. It keeps getting killed and reborn within that 5 minute window!! We use Spring framework and it is really very expensive for us when a new instance starts up. Just to give you a background, we had gone through a real roller coaster ride to make this to work on GAE by breaking the loading of framework into many different chunks. But still spinning was out of control. Then we found java threads to our rescue. We worked through the hack to load JDO to avoid UnsupportedOperationException. We finally got it to work where most of our requests were served by Always On instances with occasional spinning of Dynamic instances. It was quite impressive. Unfortunately, this was short lived when we hit this new behavior with GAE. The very last thing we want GAE to do is create a new instance every few minutes as it could easily reach 30 second deadline during the day and throw critical error. I am not sure when the new billing will come into effect but we really need this thing fixed as it literally brings down our app to a grinding halt. So I am open to any suggestions you guys think can help us. Another thought about new scheduler is to have a configurable schedule. For e.g. our users are mostly business users who work during normal business hours. We want to be able to spin more Always On instances during those hours and bring the number down during nights and weekends. Dynamic instances won't work for us due to reason explained above. Thanks, galoch On Jul 21, 5:56 pm, Johan Euphrosine pro...@google.com wrote: After speaking with Engs, I think I can explain what is going on: Here are the current scheduling rules: ( reads as has priority for handling the incoming request) 1/ Idle Always On instance Spawning a new Dynamic instance 2/ Spawning a new Dynamic instance Busy Always On instance 3/ Idle Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance I will give you an example to illustrate the behavior you all noticed, that is Dynamic instance handling request
Re: [google-appengine] Re: GAE starting unnecessary instances
1/ Idle Always On instance Spawning a new Dynamic instance 2/ Spawning a new Dynamic instance Busy Always On instance 3/ Idle Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance So App engine prefers to use bored Always On instances over spawning new dynamic once that's good. If the Always On instances are busy it spawns a new Dynamic instance good. If a Dynamic instance is bored but the Always On once are busy the Dynamic instance gets the load still good. But then you loose me if the Always one instance is idle the Dynamic instance still gets the load, why? In this last case I would expect the Always on instance to get the load otherwise the Dynamic instance will keep on being busy and will not get stopped because of it. I don't know what the cost are of spawning and later destroying a Dynamic instance but I cannot imagine that this is such a huge cost that you would have to prefer using the Dynamic instances over the Always On once. I believe that this last rule should read: 4/ Idle Always On instance Idle Dynamic instance In which case the Idle Always On instances would get the load and the bored Dynamic instances would get cleaned up much faster then they are now. I suspect this is a typo though as I cannot imagine that this is really the setup but if it is I would say that is a candidate for change. :-) The other thing I would suggest is altering the load balancing rules for Dynamic instances from the picture painted in this email it looks like the load balancing of multiple Dynamic instances is pretty much round robin (or equal load based). If this would be changed to always try and load use one Dynamic instance till the load reaches 80% or so before using the second one and so on this would allow the despawning of excess Dynamic instances much sooner then when one uses the current setup. This does mean a slightly bigger hit in case of a serious failure of the currently preferred Dynamic instance. Hence the 80% mark for the load which is arbitrarily chosen by randomly picking a number above 50 and might need some more scientific work to ensure that in an average scenario the remaining instances will usually be able to take the load caused by the sudden death of an Dynamic instance. From what I currently see it looks like the safest option has been chosen, meaning that in all cases the service will remain active no matter what happens but this means a significant cost on the customer side. I suspect that many customers are happy with that, but I think that an equal amount of them will want to see a situation where the costs are less likely to spike while providing a similar albeit slightly less high availability solution. It might be an interesting idea to offer several flavors of high availability, ranging from the current supper safe but relatively unpredictable cost to a pretty decent with very predictable costs one. I have no idea if this is technically possible but something tells me that it should not be that hard to do. And even if it is a little harder to do my guess is that Google would be up to that task. Well that's my two cents... Regards, Rob ( reads as has priority for handling the incoming request) 2/ Spawning a new Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance On Fri, Jul 22, 2011 at 4:57 PM, Johan Euphrosine pro...@google.com wrote: HI Galoch, Thanks for the followup, I think you are experiencing a combinaison fo the two following rules I was pointing to in my previous email: ( reads as has priority for handling the incoming request) 2/ Spawning a new Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance Applied to your example it could means that: Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr - 1 Hours ago while all your Always On instance were busy and you had a burst of incoming requests and the scheduler spawned new Dynamic instances as per rule 2/ highlighted above. - After the burst and back to normal traffic the new Dynamic Instances were handing incoming requests in priority as per rule 4/ highlighted above. - 2 Minutes ago all your instances Always On + Dynamic were busy again and the scheduler spawned a new Dynamic instance that handle 7 incoming requests. Hope that make more sense for you and Francois, but as I said earlier we are open to suggestion and I will make sure someone working on the scheduler team monitor this thread for your input. On Fri, Jul 22, 2011 at 9:09 AM, Galoch galoch...@gmail.com wrote: @Johan, The issue is not about Always On instance being busy. Its actually the other way ... the Always On instance is never busy ... at least that
Re: [google-appengine] App Engine Weekly Community Update #1
This is a really great summary, a nice service Johan. Thank you for taking the time. I missed the IRC discussion. I'm sure one of the things discussed there was something about the best way to publish this. Obviously this group is a must. I'm trying to figure out a way where I can get this topic Weekly Community Update in email or rss. Anyone have any ideas? -- Jay -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/N5ej2Ri4YiUJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Memcache assumptions for Counters
Hey there I'm building something like Google Analytics for Appengine but in real time (Including qps, hourly daily graphs, backend counters, monitors with alerts, etc...) The cool thing is that it only uses memcache to increase counters/stats so its really quick to use in prod code. Every minute I gather all counters and write them to datastore. It seems to work perfectly for my app (~250qps with about 1000 different counters, and about 1000 counter increases per second) I can also measure how correct my data is (if stuff flushed in memcache, but so far that never happened), but it's all based on one assumption: If I call: memcache.incr(a, intitail_value=0) ... memcache.incr(b, initial_value=0) memcache.incr(b, initial_value=0) *if a is still in the memcache b will also be in the memcache and wont have been flushed, correct?* * * or in other words: If the entity size for two items in the memcache is the same, does the memcache work like either a LRU or FIFO cache? Any response is greatly appreciated... -Andrin -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: GAE starting unnecessary instances
Johan, it seems to me the problem is rule 4/ below. Shouldn't the rule be: Idle Always On instance Idle Dynamic instance instead? In this way, if you have a dynamic instance on, it will not handle traffic unless the always-on ones are busy. So when the traffic decreases, and the dynamic instances are no longer needed, they will be correctly shut off. I also think an auxiliary problem is with rule 2/ If you spawn a new instance just because the others are all busy, you may tend to have too many instances. The fact of spawning new instances should be configurable, depending on how many requests are already queued for the instances you have already active -- 0, 1, more, etc. More in general, if you allowed users to specify how much it costs to them to delay serving a request, it would not be difficult to synthesize for each app an optimal decision policy to decide whether to switch another instance on or off. This can be done using tools from dynamic optimization / control theory. I would be glad to help if the people there need guidance on this (I used to be in the research group there till a month ago). Luca On Thursday, July 21, 2011 5:56:42 PM UTC-7, Johan Euphrosine (Google) wrote: After speaking with Engs, I think I can explain what is going on: Here are the current scheduling rules: ( reads as has priority for handling the incoming request) 1/ Idle Always On instance Spawning a new Dynamic instance 2/ Spawning a new Dynamic instance Busy Always On instance 3/ Idle Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/X8gH6jjIK0MJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Memcache assumptions for Counters
Memcache works like an LRU cache, but I don't see why a would force out b unless you ran out of space. Also, App Engine's Memcache has 2 LRU structures: an app specific LRU and a global LRU for that Memcache instance. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Fri, Jul 22, 2011 at 1:05 PM, Andrin von Rechenberg andri...@gmail.comwrote: Hey there I'm building something like Google Analytics for Appengine but in real time (Including qps, hourly daily graphs, backend counters, monitors with alerts, etc...) The cool thing is that it only uses memcache to increase counters/stats so its really quick to use in prod code. Every minute I gather all counters and write them to datastore. It seems to work perfectly for my app (~250qps with about 1000 different counters, and about 1000 counter increases per second) I can also measure how correct my data is (if stuff flushed in memcache, but so far that never happened), but it's all based on one assumption: If I call: memcache.incr(a, intitail_value=0) ... memcache.incr(b, initial_value=0) memcache.incr(b, initial_value=0) *if a is still in the memcache b will also be in the memcache and wont have been flushed, correct?* * * or in other words: If the entity size for two items in the memcache is the same, does the memcache work like either a LRU or FIFO cache? Any response is greatly appreciated... -Andrin -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Memcache assumptions for Counters
Oh, I see what you're getting at: you're asking me if B will still be in the cache if A is still in the cache. That depends on whether or not the keys hash to the same Memcache instance. FYI - in general, we don't make any guarantees of this behavior, so it potentially can be problematic down the line if this changes. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Fri, Jul 22, 2011 at 1:52 PM, Ikai Lan (Google) ika...@google.comwrote: Memcache works like an LRU cache, but I don't see why a would force out b unless you ran out of space. Also, App Engine's Memcache has 2 LRU structures: an app specific LRU and a global LRU for that Memcache instance. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Fri, Jul 22, 2011 at 1:05 PM, Andrin von Rechenberg andri...@gmail.com wrote: Hey there I'm building something like Google Analytics for Appengine but in real time (Including qps, hourly daily graphs, backend counters, monitors with alerts, etc...) The cool thing is that it only uses memcache to increase counters/stats so its really quick to use in prod code. Every minute I gather all counters and write them to datastore. It seems to work perfectly for my app (~250qps with about 1000 different counters, and about 1000 counter increases per second) I can also measure how correct my data is (if stuff flushed in memcache, but so far that never happened), but it's all based on one assumption: If I call: memcache.incr(a, intitail_value=0) ... memcache.incr(b, initial_value=0) memcache.incr(b, initial_value=0) *if a is still in the memcache b will also be in the memcache and wont have been flushed, correct?* * * or in other words: If the entity size for two items in the memcache is the same, does the memcache work like either a LRU or FIFO cache? Any response is greatly appreciated... -Andrin -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Memcache assumptions for Counters
One more thing to be aware of: there are times when Memcache needs to be flushed. If a flush happens sometime, B can have a value, but A would be unset. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Fri, Jul 22, 2011 at 1:53 PM, Ikai Lan (Google) ika...@google.comwrote: Oh, I see what you're getting at: you're asking me if B will still be in the cache if A is still in the cache. That depends on whether or not the keys hash to the same Memcache instance. FYI - in general, we don't make any guarantees of this behavior, so it potentially can be problematic down the line if this changes. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Fri, Jul 22, 2011 at 1:52 PM, Ikai Lan (Google) ika...@google.comwrote: Memcache works like an LRU cache, but I don't see why a would force out b unless you ran out of space. Also, App Engine's Memcache has 2 LRU structures: an app specific LRU and a global LRU for that Memcache instance. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Fri, Jul 22, 2011 at 1:05 PM, Andrin von Rechenberg andri...@gmail.com wrote: Hey there I'm building something like Google Analytics for Appengine but in real time (Including qps, hourly daily graphs, backend counters, monitors with alerts, etc...) The cool thing is that it only uses memcache to increase counters/stats so its really quick to use in prod code. Every minute I gather all counters and write them to datastore. It seems to work perfectly for my app (~250qps with about 1000 different counters, and about 1000 counter increases per second) I can also measure how correct my data is (if stuff flushed in memcache, but so far that never happened), but it's all based on one assumption: If I call: memcache.incr(a, intitail_value=0) ... memcache.incr(b, initial_value=0) memcache.incr(b, initial_value=0) *if a is still in the memcache b will also be in the memcache and wont have been flushed, correct?* * * or in other words: If the entity size for two items in the memcache is the same, does the memcache work like either a LRU or FIFO cache? Any response is greatly appreciated... -Andrin -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Memcache assumptions for Counters
Thanks for getting back to me so quickly! RE: Your last email, you wrote: B can have a value, but A would be unset. That wouldnt be a problem since i only want the implication A=B (If A exists then B exists). But I assume you meant: A can have a value, but B would be unset. Right? Can you (*really roughly*) say how many memcache instances you are running? Let's say I would create 1000 As, would I hit all memcaches (with a high probabilty)? Then the assumption if all A's exist, then B exists' would hold... Or can I influence the hash somehow to end up in the same instance (something else then finding hash collisions :)) ? Thanks again, for getting back to me so quickly. -Andrin On Fri, Jul 22, 2011 at 7:55 PM, Ikai Lan (Google) ika...@google.comwrote: One more thing to be aware of: there are times when Memcache needs to be flushed. If a flush happens sometime, B can have a value, but A would be unset. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Fri, Jul 22, 2011 at 1:53 PM, Ikai Lan (Google) ika...@google.comwrote: Oh, I see what you're getting at: you're asking me if B will still be in the cache if A is still in the cache. That depends on whether or not the keys hash to the same Memcache instance. FYI - in general, we don't make any guarantees of this behavior, so it potentially can be problematic down the line if this changes. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Fri, Jul 22, 2011 at 1:52 PM, Ikai Lan (Google) ika...@google.comwrote: Memcache works like an LRU cache, but I don't see why a would force out b unless you ran out of space. Also, App Engine's Memcache has 2 LRU structures: an app specific LRU and a global LRU for that Memcache instance. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Fri, Jul 22, 2011 at 1:05 PM, Andrin von Rechenberg andri...@gmail.com wrote: Hey there I'm building something like Google Analytics for Appengine but in real time (Including qps, hourly daily graphs, backend counters, monitors with alerts, etc...) The cool thing is that it only uses memcache to increase counters/stats so its really quick to use in prod code. Every minute I gather all counters and write them to datastore. It seems to work perfectly for my app (~250qps with about 1000 different counters, and about 1000 counter increases per second) I can also measure how correct my data is (if stuff flushed in memcache, but so far that never happened), but it's all based on one assumption: If I call: memcache.incr(a, intitail_value=0) ... memcache.incr(b, initial_value=0) memcache.incr(b, initial_value=0) *if a is still in the memcache b will also be in the memcache and wont have been flushed, correct?* * * or in other words: If the entity size for two items in the memcache is the same, does the memcache work like either a LRU or FIFO cache? Any response is greatly appreciated... -Andrin -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Memcache assumptions for Counters
Ha ha ha ... To be honest, I couldn't say off the top of my head, but it's not something you can depend on. Another thing you can think about doing is using the backend instances. Those are more or less guaranteed to stick around, though you might not be able to store as much data in them. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Fri, Jul 22, 2011 at 2:08 PM, MiuMeet Support r...@miumeet.com wrote: Thanks for getting back to me so quickly! RE: Your last email, you wrote: B can have a value, but A would be unset. That wouldnt be a problem since i only want the implication A=B (If A exists then B exists). But I assume you meant: A can have a value, but B would be unset. Right? Can you (*really roughly*) say how many memcache instances you are running? Let's say I would create 1000 As, would I hit all memcaches (with a high probabilty)? Then the assumption if all A's exist, then B exists' would hold... Or can I influence the hash somehow to end up in the same instance (something else then finding hash collisions :)) ? Thanks again, for getting back to me so quickly. -Andrin On Fri, Jul 22, 2011 at 7:55 PM, Ikai Lan (Google) ika...@google.comwrote: One more thing to be aware of: there are times when Memcache needs to be flushed. If a flush happens sometime, B can have a value, but A would be unset. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Fri, Jul 22, 2011 at 1:53 PM, Ikai Lan (Google) ika...@google.comwrote: Oh, I see what you're getting at: you're asking me if B will still be in the cache if A is still in the cache. That depends on whether or not the keys hash to the same Memcache instance. FYI - in general, we don't make any guarantees of this behavior, so it potentially can be problematic down the line if this changes. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Fri, Jul 22, 2011 at 1:52 PM, Ikai Lan (Google) ika...@google.comwrote: Memcache works like an LRU cache, but I don't see why a would force out b unless you ran out of space. Also, App Engine's Memcache has 2 LRU structures: an app specific LRU and a global LRU for that Memcache instance. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Fri, Jul 22, 2011 at 1:05 PM, Andrin von Rechenberg andri...@gmail.com wrote: Hey there I'm building something like Google Analytics for Appengine but in real time (Including qps, hourly daily graphs, backend counters, monitors with alerts, etc...) The cool thing is that it only uses memcache to increase counters/stats so its really quick to use in prod code. Every minute I gather all counters and write them to datastore. It seems to work perfectly for my app (~250qps with about 1000 different counters, and about 1000 counter increases per second) I can also measure how correct my data is (if stuff flushed in memcache, but so far that never happened), but it's all based on one assumption: If I call: memcache.incr(a, intitail_value=0) ... memcache.incr(b, initial_value=0) memcache.incr(b, initial_value=0) *if a is still in the memcache b will also be in the memcache and wont have been flushed, correct?* * * or in other words: If the entity size for two items in the memcache is the same, does the memcache work like either a LRU or FIFO cache? Any response is greatly appreciated... -Andrin -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at
[google-appengine] Re: SDK Upgrade 1.5.2: --datastore_path ignored?
I have tried to restore the datastore and the GoogleAppEngineLauncher.app (with the SDK bundle inside) back to 1.5.0 but I cannot get it to run. Its always the same error, as if the downgrade did not change the python files. Does anyone know how to get the SDK-version out of the running appserver (a --version flag is missing!)? On 22 Jul., 16:21, oakmad adam.oak...@gmail.com wrote: +1 for me. I have tried deleting the application plist as detailed in this issuehttp://code.google.com/p/googleappengine/issues/detail?id=1187q=data... but no joy. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: GAE starting unnecessary instances
Hi Johan, Thanks for the explanation. I have couple of questions on that. 1. 1 Hours ago while all your Always On instance were busy and you had a burst of incoming requests While this may be true when my Always On instances were busy running some stuff but what about when 2 Always On instances show only 1 request served which is the Warmup request itself. Does this mean Warmup requests are considered as traffic? If that is the case then Always On instances seem rather useless since they will never ever get called in this scenario. 2. As Tom mentioned, what qualifies busy. When threadsafe option was implemented in GAE these 3 Always On instances were able to do most of the heavy lifting with occasional spinning of dynamic instances. Nothing has changed on our side that should alter this behavior. With all these changes happening within GAE I am trying to figure out what changed and what we can do to contain this burst of traffic within 3 (or more ) Always On instances with less frequent spinning of Dynamic instances. 3. - 2 Minutes ago all your instances Always On + Dynamic were busy again and the scheduler spawned a new Dynamic instance that handle 7 incoming requests. Again what constitutes busy as I do not see any request being served by Always On instances 2 and 3 in last 1 hour. Note that number of requests served by Always On 2/3 are unchanged since they were created ... Here's my reading in this scenario: a. It kills Dynamic Instance 1 within 2 minutes of serving a request b. When traffic comes in it looks only for Dynamic Instances if they are busy and completely ignores Always On instances at this point c. It recreates Dynamic Instance 1 In other words, what rule is applied in this case? Also I fail to understand rule 4 as both Rob and Luca mentioned. That completely undermines having Always On instances under threadsafe mode. 4. I like Rob's suggestion of better load balancing techniques but again with a caveat that an instance needs to be able to serve multiple threads before reaching a set capacity (80% or so) 5. Luca's suggestion also makes sense but again with the same caveat ... it should be able to process multiple threads before queuing 6. I looked at the new sliders in the Admin console and with those the situation is even worse. I set the Max Idle Instances to 3 (that's the minimum I could choose) and Min Pending Latency to 15 secs ... Guess what our CPU usage has gone up to 15 in 12 hrs because of constant creation and killing of 3 dynamic instances. Bare minimum traffic and few light weight crons. But the good side is now I see requests coming in on the 3 Always On instances. Is that enough load they are serving ... I don't know yet but something to observe. Two things I suggest would be really helpful for us: A. The overall key here is to know the thread handling capacity of an instance. Better yet if it can be configured similar to Backends but dynamic in nature (and of course Backends pricing is outrageous ... but that's another topic) B. Able to add more Always On instances but again with a dependency explained in point A. Hope it makes sense. Thanks, galoch On Jul 22, 7:57 am, Johan Euphrosine pro...@google.com wrote: HI Galoch, Thanks for the followup, I think you are experiencing a combinaison fo the two following rules I was pointing to in my previous email: ( reads as has priority for handling the incoming request) 2/ Spawning a new Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance Applied to your example it could means that: Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr - 1 Hours ago while all your Always On instance were busy and you had a burst of incoming requests and the scheduler spawned new Dynamic instances as per rule 2/ highlighted above. - After the burst and back to normal traffic the new Dynamic Instances were handing incoming requests in priority as per rule 4/ highlighted above. - 2 Minutes ago all your instances Always On + Dynamic were busy again and the scheduler spawned a new Dynamic instance that handle 7 incoming requests. Hope that make more sense for you and Francois, but as I said earlier we are open to suggestion and I will make sure someone working on the scheduler team monitor this thread for your input. On Fri, Jul 22, 2011 at 9:09 AM, Galoch galoch...@gmail.com wrote: @Johan, The issue is not about Always On instance being busy. Its actually the other way ... the Always On instance is never busy ... at least that is what we observed in last 3-4 days. Your explanation may be partly true since this behavior keeps on changing. For e.g. I have a snapshot of instances from July 19th and here's the
[google-appengine] Re: SDK Upgrade 1.5.2: --datastore_path ignored?
hi all, i *think* that it is honoring your datastore location (though the log message is incorrect), but the change to rename your application to dev~your appname in development has just rendered all of our test data useless. after re-importing my test data it does look like it is stored where i ask it to be, but under the new application name. if you are lucky enough to be using sqlite you can connect to the db and rename some tables to get it to work: sqlite3 local_appname_dev_sqlite.datastore sqlite .tables Apps IdSeq Namespaces appname!!Entities appname!!EntitiesByProperty appname!namespace!Entities appname!namespace!EntitiesByProperty sqlite alter table `appname!!Entities` rename to `dev~appname!!Entities`; sqlite alter table `appname!!EntitiesByProperty` rename to `dev~appname!!EntitiesByProperty`; sqlite alter table `appname!namespace!Entities` rename to `dev~appname!namespace!Entities`; sqlite alter table `appname!namespace!EntitiesByProperty` rename to `dev~appname!namespace!EntitiesByProperty`; where you substitute 'appname' for your application's name, and 'namespace' for your datanamespace. cfh -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/HvLD9aX68dYJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] App Engine Weekly Community Update #1
Hi Jay, This was also posted on Google Plus: https://plus.google.com/111042085517496880918/posts/exjJoZwygq1 Until there're brand pages on g+, and then the weekly update could be posted on a 'App Engine' profile, you can follow Johan using the link above. -- rodrigo -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/lBsHKUaKDiUJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: SDK Upgrade 1.5.2: --datastore_path ignored?
That's a clever way to update the appid. I do not know if it works for all cases (e.g. it may not work for all reference properties (stored keys)) but is a neat trick. Another way to deal with it is to use the --default_partition= flag rather than using an older version of the sdk. --Matthew On Jul 22, 12:17 pm, c h how...@umich.edu wrote: hi all, i *think* that it is honoring your datastore location (though the log message is incorrect), but the change to rename your application to dev~your appname in development has just rendered all of our test data useless. after re-importing my test data it does look like it is stored where i ask it to be, but under the new application name. if you are lucky enough to be using sqlite you can connect to the db and rename some tables to get it to work: sqlite3 local_appname_dev_sqlite.datastore sqlite .tables Apps IdSeq Namespaces appname!!Entities appname!!EntitiesByProperty appname!namespace!Entities appname!namespace!EntitiesByProperty sqlite alter table `appname!!Entities` rename to `dev~appname!!Entities`; sqlite alter table `appname!!EntitiesByProperty` rename to `dev~appname!!EntitiesByProperty`; sqlite alter table `appname!namespace!Entities` rename to `dev~appname!namespace!Entities`; sqlite alter table `appname!namespace!EntitiesByProperty` rename to `dev~appname!namespace!EntitiesByProperty`; where you substitute 'appname' for your application's name, and 'namespace' for your datanamespace. cfh -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: GAE starting unnecessary instances
The current behavior makes me suspect we are being prepared for Always- on being replaced completely by the new scheduler knobs. Being able to turn up the number of idle instances does make always-on somewhat redundant, as long as the idle instances stick around for a while. Also, if always-on instances WERE being properly utilized right now, it would require artificial load to ascertain the effects of the new scheduler on dynamic instances (assuming the three AO instances were sufficient for your app previously). Many devs wouldn't learn about the new scheduler attributes until after their app (suddenly) becomes popular - not the best time for surprises. And Google wouldn't get as much feedback on the features/behavior of the new scheduler. Is Always-on going to be kept under the new model? /Tom On Jul 22, 10:57 am, Johan Euphrosine pro...@google.com wrote: HI Galoch, Thanks for the followup, I think you are experiencing a combinaison fo the two following rules I was pointing to in my previous email: ( reads as has priority for handling the incoming request) 2/ Spawning a new Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance Applied to your example it could means that: Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr - 1 Hours ago while all your Always On instance were busy and you had a burst of incoming requests and the scheduler spawned new Dynamic instances as per rule 2/ highlighted above. - After the burst and back to normal traffic the new Dynamic Instances were handing incoming requests in priority as per rule 4/ highlighted above. - 2 Minutes ago all your instances Always On + Dynamic were busy again and the scheduler spawned a new Dynamic instance that handle 7 incoming requests. Hope that make more sense for you and Francois, but as I said earlier we are open to suggestion and I will make sure someone working on the scheduler team monitor this thread for your input. On Fri, Jul 22, 2011 at 9:09 AM, Galoch galoch...@gmail.com wrote: @Johan, The issue is not about Always On instance being busy. Its actually the other way ... the Always On instance is never busy ... at least that is what we observed in last 3-4 days. Your explanation may be partly true since this behavior keeps on changing. For e.g. I have a snapshot of instances from July 19th and here's the details (for some reason I can't see a link to attach the snapshot images here): Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr This is under no load with only very light weight cron jobs running. This gets much much worse during the day under peak load with requests for dynamic instances reaching 1000+ in matter of minutes and resident instances have only 1 request served. As you see above Resident Instance 2 and 3 are hardly hit so I don't think they are busy at all. On the other hand, Dynamic Instance 2 and 3 get most of the hits. Dynamic Instance 1 is what is killing us. It keeps getting killed and reborn within that 5 minute window!! We use Spring framework and it is really very expensive for us when a new instance starts up. Just to give you a background, we had gone through a real roller coaster ride to make this to work on GAE by breaking the loading of framework into many different chunks. But still spinning was out of control. Then we found java threads to our rescue. We worked through the hack to load JDO to avoid UnsupportedOperationException. We finally got it to work where most of our requests were served by Always On instances with occasional spinning of Dynamic instances. It was quite impressive. Unfortunately, this was short lived when we hit this new behavior with GAE. The very last thing we want GAE to do is create a new instance every few minutes as it could easily reach 30 second deadline during the day and throw critical error. I am not sure when the new billing will come into effect but we really need this thing fixed as it literally brings down our app to a grinding halt. So I am open to any suggestions you guys think can help us. Another thought about new scheduler is to have a configurable schedule. For e.g. our users are mostly business users who work during normal business hours. We want to be able to spin more Always On instances during those hours and bring the number down during nights and weekends. Dynamic instances won't work for us due to reason explained
Re: [google-appengine] App Engine Weekly Community Update #1
Maybe a Google Alert for site:groups.google.com/group/google-appengine/ intitle:App Engine Weekly Community Update http://www.google.com/alerts/ On Fri, Jul 22, 2011 at 5:30 PM, Jay jbaker.w...@gmail.com wrote: This is a really great summary, a nice service Johan. Thank you for taking the time. I missed the IRC discussion. I'm sure one of the things discussed there was something about the best way to publish this. Obviously this group is a must. I'm trying to figure out a way where I can get this topic Weekly Community Update in email or rss. Anyone have any ideas? -- Jay -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/N5ej2Ri4YiUJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: App Engine Weekly Community Update #1
On Jul 22, 10:34 am, Johan Euphrosine pro...@google.com wrote: I'd love to track the actual resolution of questions but we miss metadata to check if a question has been answered. Maybe we (as the community) can agree on some convention were we use nickname++ people as a signal of the question of the current thread being answered, or we (community) can decide to shift those question to Stack Overflow and keep the group for more casual discussion. Since you mention StackOverflow and linked to some interesting questions in the update, I wonder if anyone with http://data.stackexchange.com skills could produce similar metrics from the google-app-engine tag on SO. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: SDK Upgrade 1.5.2: --datastore_path ignored?
Thanks, Matthew. I was able to update to 1.5.2 and use my existing datastore by adding that flag. It would have been useful if the release notes had mentioned that this would be necessary. -Chrsi On Fri, Jul 22, 2011 at 2:39 PM, Matthew Blain matthew.bl...@google.comwrote: That's a clever way to update the appid. I do not know if it works for all cases (e.g. it may not work for all reference properties (stored keys)) but is a neat trick. Another way to deal with it is to use the --default_partition= flag rather than using an older version of the sdk. --Matthew On Jul 22, 12:17 pm, c h how...@umich.edu wrote: hi all, i *think* that it is honoring your datastore location (though the log message is incorrect), but the change to rename your application to dev~your appname in development has just rendered all of our test data useless. after re-importing my test data it does look like it is stored where i ask it to be, but under the new application name. if you are lucky enough to be using sqlite you can connect to the db and rename some tables to get it to work: sqlite3 local_appname_dev_sqlite.datastore sqlite .tables Apps IdSeq Namespaces appname!!Entities appname!!EntitiesByProperty appname!namespace!Entities appname!namespace!EntitiesByProperty sqlite alter table `appname!!Entities` rename to `dev~appname!!Entities`; sqlite alter table `appname!!EntitiesByProperty` rename to `dev~appname!!EntitiesByProperty`; sqlite alter table `appname!namespace!Entities` rename to `dev~appname!namespace!Entities`; sqlite alter table `appname!namespace!EntitiesByProperty` rename to `dev~appname!namespace!EntitiesByProperty`; where you substitute 'appname' for your application's name, and 'namespace' for your datanamespace. cfh -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: path/wildcards on GAE Boto get_bucket()?
Cross posting answer .. thanks to Mike @ groups.google.com/group/gs-
discussion
The REST API doesn't directly support wildcard request. You can ask
for a
prefix, (you can see documentation for this here
http://code.google.com/apis/storage/docs/reference-
headers.html#prefix)
but not (for example) gs://bucket/*.txt
gsutil has a library that does the wildcarding you want: please see
http://code.google.com/p/gsutil/source/browse/trunk/src/gslib/wildcar...
(Or if you just need to do some wildcarding from the command line, you
can
use gsutil rather than the library inside gsutil)
==
On Jul 21, 1:34 pm, David Cheney david.che...@gmail.com wrote:
Apologies if this is a FAQ.. not found an answer via search..
From within GAE, using python, I want to get a list of objects in a
GSD bucket:
uri = boto.storage_uri('mybucket','gs')
names = ''
for obj in uri.get_bucket():
names = names + + obj.name
This works handily but returns all items.. when what I care about is a
subset.
My objective is to get a list of images on a certain path as I would
get with a /a/b/*.jpg
I can obviously parse the list myself, but was hoping I'd missed magic
somewhere in the libs.
Thanks,
DJC
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: SDK Upgrade 1.5.2: --datastore_path ignored?
yup the sqlite updates don't quite seem to work. i second that the release notes should be updated, the flag does seem to work for me cfh -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Wi0krMFaQ6MJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: GAE starting unnecessary instances
I've suspected the same thing. So far we've seen no pricing info for always on and questions about it have went unanswered. Not to mention that $2.10 / week for three always-on instances is a lot different than $40 / week. Robert On Fri, Jul 22, 2011 at 15:46, Tom Phillips tphill0...@gmail.com wrote: The current behavior makes me suspect we are being prepared for Always- on being replaced completely by the new scheduler knobs. Being able to turn up the number of idle instances does make always-on somewhat redundant, as long as the idle instances stick around for a while. Also, if always-on instances WERE being properly utilized right now, it would require artificial load to ascertain the effects of the new scheduler on dynamic instances (assuming the three AO instances were sufficient for your app previously). Many devs wouldn't learn about the new scheduler attributes until after their app (suddenly) becomes popular - not the best time for surprises. And Google wouldn't get as much feedback on the features/behavior of the new scheduler. Is Always-on going to be kept under the new model? /Tom On Jul 22, 10:57 am, Johan Euphrosine pro...@google.com wrote: HI Galoch, Thanks for the followup, I think you are experiencing a combinaison fo the two following rules I was pointing to in my previous email: ( reads as has priority for handling the incoming request) 2/ Spawning a new Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance Applied to your example it could means that: Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr - 1 Hours ago while all your Always On instance were busy and you had a burst of incoming requests and the scheduler spawned new Dynamic instances as per rule 2/ highlighted above. - After the burst and back to normal traffic the new Dynamic Instances were handing incoming requests in priority as per rule 4/ highlighted above. - 2 Minutes ago all your instances Always On + Dynamic were busy again and the scheduler spawned a new Dynamic instance that handle 7 incoming requests. Hope that make more sense for you and Francois, but as I said earlier we are open to suggestion and I will make sure someone working on the scheduler team monitor this thread for your input. On Fri, Jul 22, 2011 at 9:09 AM, Galoch galoch...@gmail.com wrote: @Johan, The issue is not about Always On instance being busy. Its actually the other way ... the Always On instance is never busy ... at least that is what we observed in last 3-4 days. Your explanation may be partly true since this behavior keeps on changing. For e.g. I have a snapshot of instances from July 19th and here's the details (for some reason I can't see a link to attach the snapshot images here): Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr This is under no load with only very light weight cron jobs running. This gets much much worse during the day under peak load with requests for dynamic instances reaching 1000+ in matter of minutes and resident instances have only 1 request served. As you see above Resident Instance 2 and 3 are hardly hit so I don't think they are busy at all. On the other hand, Dynamic Instance 2 and 3 get most of the hits. Dynamic Instance 1 is what is killing us. It keeps getting killed and reborn within that 5 minute window!! We use Spring framework and it is really very expensive for us when a new instance starts up. Just to give you a background, we had gone through a real roller coaster ride to make this to work on GAE by breaking the loading of framework into many different chunks. But still spinning was out of control. Then we found java threads to our rescue. We worked through the hack to load JDO to avoid UnsupportedOperationException. We finally got it to work where most of our requests were served by Always On instances with occasional spinning of Dynamic instances. It was quite impressive. Unfortunately, this was short lived when we hit this new behavior with GAE. The very last thing we want GAE to do is create a new instance every few minutes as it could easily reach 30 second deadline during the day and throw critical error. I am not sure when the new billing will come into effect but we really need this thing fixed as it literally brings down our app to a grinding halt. So I am open to any suggestions you guys think can help us. Another thought about new
