[google-appengine] Re: Flexible & Standard Appengine - Java

[David Hardwick]

Thanks for this, Adam, that's a great resource that I wasn't aware of until 
I read your thread response.  Your summary of the article was also helpful.

On Sunday, April 23, 2017 at 2:44:20 PM UTC-4, Adam (Cloud Platform 
Support) wrote:
>
> You can find a comprehensive list of changes in Migrating Services from 
> the Standard Environment to the Flexible Environment. 
>  You'll 
> need to migrate your app to use the public APIs instead of the standard 
> environment APIs. The most straightforward way is to use the 
> google-cloud-java  
> library.
>
> Some services are not available in the flexible environment, notably 
> Memcache 
>  
> and Push Task Queues 
> , 
> however there are replacements in early access alpha which you can sign up 
> for, or use alternative services like Redis Labs Memcache or Google Cloud 
> Pub/Sub / Pull Queues.
>
> On Sunday, April 23, 2017 at 8:55:42 AM UTC-4, Rajesh Gupta wrote:
>>
>> We are on java standard appengine for 5+ years.
>>
>> Just want to know, what is not available in Flexible env that is 
>> available in the Standard?
>>
>> What are the showstoppers to move to a flexible appengine env from a 
>> Standard appengine.
>>
>> -- 
>> Regards,
>> Rajesh
>> *www.VeersoftSolutions.com *
>> *www.GainERP.com *
>> *Accounting/Inventory/Orders/Sales/Purchase on Google Cloud Platform and 
>> Mobile*
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/a5cc5fae-3465-4c11-88de-555374ebf330%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Have you had success with Traffic Splitting? How do you use it to roll out new versions?

['David Hardwick' via Google App Engine]

nity to submit appropriate talks for the *next* OSCON I guess
> (Austin, TX, May 2016)...:-).
>
>
> > For example, I don't have a great solution for canary testing a version
> which changes the ndb model schema.  I would love to hear the concrete
> lessons learned from anyone who has done such a thing.
>
> Alas, schema changes are always a bother, no matter the underlying
> technologies and architectures. I can't offer more than applause to David's
> post below, for starting to highlight some of the relevant issues -- TL;DR,
> that schema changes must be done in an incremental way, always having the
> code that wants/prefers/supports schema version N to operate
> non-destructively on versions N-1 *and* N+1 as well. A bother indeed,
> but, I have no silver bullet to slay that particular werewolf, sorry.
>
> This isn't limited to canarying or incremental roll-out, though those
> patterns highlight the problem in particularly stark colors. But even back
> in the times of big-bang upgrades and hours of downtime to let them happen,
> I think I've witnessed more release/upgrade disasters tied to schema
> changes, than to any other single root cause... the risks are just more
> obvious and blocking today, rather than deeply hidden, and that extra
> visibility need not be a bad thing, in fact.
>
> Another example is how do you elegantly synchronize decoupled apps?  What
>> I mean is that e.g., we run our APIs in a different project than our web
>> front-end.  There are several ways to handle this, but again would love to
>> get war stories from anyone who has run something similar in production.
>>
>
> I may be missing something here -- I'd expect the web front-end to be a
> consumer of the APIs just like any other front-end would (an excellent
> architectural separation), so e.g a new API version would be handled by the
> web front-end just as it would by any other client (mobile native apps,
> etc, etc) -- use explicit versioning, version negotiation, and so forth --
> just general best-practice patterns of API architecture, no?
>
> I'm sure there are other use cases that show your point better, so, let's
> please discuss them!
>
>
> Alex
>
>
>
>>
>>
>>
>> On Friday, July 24, 2015 at 3:56:27 PM UTC-4, Jason Collins wrote:
>>>
>>> Traffic-splitting / canary releases on App Engine are definitely "a
>>> thing".
>>>
>>> Traffic-splits on non-default modules are now available via API:
>>>
>>>
>>> https://cloud.google.com/appengine/docs/admin-api/quickstart/#splitting_traffic
>>>
>>>
>>> On Friday, 24 July 2015 11:50:33 UTC-7, Michael Spainhower wrote:
>>>>
>>>> @David, I started following this thread because I have the exact same
>>>> question and agree the lack of response is worrisome.
>>>>
>>>> We are in the Cloud Startup Program and I plan to ask about canary
>>>> testing during my next engineering 1-on-1.  I will reply to this thread
>>>> with what I learn from their engineer.
>>>>
>>>>
>>>>
>>>> On Friday, July 24, 2015 at 9:14:02 AM UTC-4, David Hardwick wrote:
>>>>>
>>>>> Oh boy, the lack of response here is not encouraging
>>>>>
>>>>> On Tuesday, July 21, 2015 at 1:53:17 PM UTC-4, David Hardwick wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> We haven't used Traffic Splitting yet but it has been an available
>>>>>> feature for a while and rumor has it that traffic splitting for 
>>>>>> non-default
>>>>>> modules could be coming in as soon as a month.
>>>>>>
>>>>>> Any who, if you have experience use it, then I would like to hear how
>>>>>> you are using it to roll out new features or versions.  I've heard the 
>>>>>> term
>>>>>> 'canary' testing where you roll out a new version to 10% of folks...you
>>>>>> measure the results and then either rollback and fully roll it out.  So 
>>>>>> if
>>>>>> anyone is doing 'canary' testing and deployments as I've described it, 
>>>>>> then
>>>>>> I've like to hear from you.
>>>>>>
>>>>>> Thanks in advance,
>>>>>>   Hardwick
>>>>>>
>>>>>  --
>> You received this message because you are subscribed to the Google Groups
>> "Google

[google-appengine] Re: Have you had success with Traffic Splitting? How do you use it to roll out new versions?

['David Hardwick' via Google App Engine]

(For the model/schema change, I think that comes down for being prepared to
have the schema accessed both ways...e.g., you need to remove a column, you
roll out the new code that doesn't use the column...the old code still
works with the column there...after you get to 100% on the new version,
then you can nix the column...stuff like that.  So the QA team would
actually want to test the traffic split approach in lower (non-production)
environments prior to the production deployment to make sure it works out
as planned. )

This link [1] provides more details on Traffic Splitting options (cookies
vs IP) and some areas to be aware of with respect to caching issues.
[1]
https://cloud.google.com/appengine/docs/developers-console/#traffic-splitting

It would be good to see if folks favor IP over Cookie for the splitting,
and what lessons learned they had with regard to caching issues.

Rock on,
  Hardwick


On Fri, Jul 24, 2015 at 4:14 PM, Michael Spainhower 
wrote:

> Sure, I don't think David or I have any question of whether canary testing
> is a possible or intended use case of traffic splitting.  I am interested
> in how folks have implemented it in practice for production apps.
>
> For example, I don't have a great solution for canary testing a version
> which changes the ndb model schema.  I would love to hear the concrete
> lessons learned from anyone who has done such a thing.
>
> Another example is how do you elegantly synchronize decoupled apps?  What
> I mean is that e.g., we run our APIs in a different project than our web
> front-end.  There are several ways to handle this, but again would love to
> get war stories from anyone who has run something similar in production.
>
>
>
> On Friday, July 24, 2015 at 3:56:27 PM UTC-4, Jason Collins wrote:
>>
>> Traffic-splitting / canary releases on App Engine are definitely "a
>> thing".
>>
>> Traffic-splits on non-default modules are now available via API:
>>
>>
>> https://cloud.google.com/appengine/docs/admin-api/quickstart/#splitting_traffic
>>
>>
>> On Friday, 24 July 2015 11:50:33 UTC-7, Michael Spainhower wrote:
>>>
>>> @David, I started following this thread because I have the exact same
>>> question and agree the lack of response is worrisome.
>>>
>>> We are in the Cloud Startup Program and I plan to ask about canary
>>> testing during my next engineering 1-on-1.  I will reply to this thread
>>> with what I learn from their engineer.
>>>
>>>
>>>
>>> On Friday, July 24, 2015 at 9:14:02 AM UTC-4, David Hardwick wrote:
>>>>
>>>> Oh boy, the lack of response here is not encouraging
>>>>
>>>> On Tuesday, July 21, 2015 at 1:53:17 PM UTC-4, David Hardwick wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> We haven't used Traffic Splitting yet but it has been an available
>>>>> feature for a while and rumor has it that traffic splitting for 
>>>>> non-default
>>>>> modules could be coming in as soon as a month.
>>>>>
>>>>> Any who, if you have experience use it, then I would like to hear how
>>>>> you are using it to roll out new features or versions.  I've heard the 
>>>>> term
>>>>> 'canary' testing where you roll out a new version to 10% of folks...you
>>>>> measure the results and then either rollback and fully roll it out.  So if
>>>>> anyone is doing 'canary' testing and deployments as I've described it, 
>>>>> then
>>>>> I've like to hear from you.
>>>>>
>>>>> Thanks in advance,
>>>>>   Hardwick
>>>>>
>>>>


-- 
*David Hardwick* | CTO | w. 646-237-5388
3405 Piedmont Rd. NE, Suite 325, Atlanta, GA 30305
 <http://www.bettercloud.com>
*Featured Series:*
The Latest Trends in Cloud IT
<http://blog.bettercloud.com/category/trends-in-cloud-it/>

calendar availability
<https://www.google.com/calendar/embed?src=david.hardw...@bettercloud.com&ctz=America/New_York&mode=week>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAMYGC97pJQON8hpYjQ-1kvZPkdoc%2BjVx_hOp6CqDD8ZiHqU%2BAA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Have you had success with Traffic Splitting? How do you use it to roll out new versions?

[David Hardwick]

Oh boy, the lack of response here is not encouraging

On Tuesday, July 21, 2015 at 1:53:17 PM UTC-4, David Hardwick wrote:
>
> Hello,
>
> We haven't used Traffic Splitting yet but it has been an available feature 
> for a while and rumor has it that traffic splitting for non-default modules 
> could be coming in as soon as a month.  
>
> Any who, if you have experience use it, then I would like to hear how you 
> are using it to roll out new features or versions.  I've heard the term 
> 'canary' testing where you roll out a new version to 10% of folks...you 
> measure the results and then either rollback and fully roll it out.  So if 
> anyone is doing 'canary' testing and deployments as I've described it, then 
> I've like to hear from you.
>
> Thanks in advance,
>   Hardwick
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/1fa10753-6040-492b-af53-a16f38cb6a63%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Have you had success with Traffic Splitting? How do you use it to roll out new versions?

[David Hardwick]

Hello,

We haven't used Traffic Splitting yet but it has been an available feature 
for a while and rumor has it that traffic splitting for non-default modules 
could be coming in as soon as a month.  

Any who, if you have experience use it, then I would like to hear how you 
are using it to roll out new features or versions.  I've heard the term 
'canary' testing where you roll out a new version to 10% of folks...you 
measure the results and then either rollback and fully roll it out.  So if 
anyone is doing 'canary' testing and deployments as I've described it, then 
I've like to hear from you.

Thanks in advance,
  Hardwick

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/8f92c926-ceef-4709-95ce-d36336d1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: "One senses GAE is just not a major priority for Google"

[David Hardwick]

And how are you?

The GCP Live event was telling the story around the continuum the Google 
Cloud Platform now provides from PaaS to IaaS.

You have GAE and with Managed VMs (combined with Autoscaler) you have GAE 
2.0 in the PaaS category.  The GAE 1.0 docker images for Java/Php/Python 
can be deployed on Managed VMs if you wanted to, and when you want to 
customize that image ("I want Java 8"), okay, now you can.

You need to run certain software on cloud servers, go with GCE (servers, 
disks, load balancers, etc.) in the IaaS category.
  
You want something in the middle, go with GKE (Google Containers Engine). 
You can deploy the Docker items for Managed VMs to GKE, but if you want to 
use features like traffic splitting, etc. then you want to use Managed VM. 
Depends on your needs, your choice now. 

I try to start in the PaaS section until I need to get out of it due to 
certain technical requirements.  But excited to have all these choices 
along the PaaS to IaaS spectrum.

rock on,
  hardwick

On Wednesday, November 5, 2014 6:43:44 PM UTC-5, Doug Anderson wrote:
>
> I don't think there's any reason to migrate existing apps unless App 
> Engine no longer satisfies your requirements.  I don't see App Engine going 
> away... you just need to set your expectations of the platform accordingly 
> (don't expect bugs to get resolved unless you have a paid support plan, 
> don't expect earth shattering new services/features, etc).
>
> On Wednesday, November 5, 2014 3:31:40 PM UTC-5, Emanuele Ziglioli wrote:
>>
>> Thank you everyone for your insights, very interesting.
>>
>> What do you guys think it's the way forward?
>> Are you going to migrate your GAE apps to to Managed VMs, with Docker and 
>> the gcs command line tools?
>>
>> Also, is the Datastore still a valid option? 
>> I wish BigQuery just worked natively with it...
>>
>> Thanks 
>> Emanuele
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: GAE SDK Release announcements ???

[David Hardwick]

+1 on announcing here...I check this email digest every day looking for the 
SDK updates, had no idea we were at 1.9.9.

On Tuesday, August 19, 2014 1:34:18 PM UTC-4, PK wrote:
>
> It used to be that new GAE SDK releases would be announced in this group. 
> Lately, I have not seen any announcements. The last announcement I can find 
> is 1.9.3 and the downloads page and dashboard show 1.9.9.
>
> So my question is are the GAE SDK releases announced somewhere else? I 
> think the pre-release announcements was a good practice too although  I 
> understand that the last few releases are very minor on the SDK side.
>
> Thanks,
> PK
> http://www.gae123.com
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: NoSQL vs Relational DBs - pros and cons

[David Hardwick]

Thanks, Jeff.  Yes, I'm starting to get some calls in with folks and one
company told me they don't make the decision at an application level, they
make it per tableso they use both relational and NoSQL.   Not going to
have massive scale needs and need relational?  Then they use
relational...going to have mind blowing scale (X thousand records per
second) they use NoSQL.

Thanks for the tip on the Google search for how Facebook scaled MySQL, I'm
seeing some great articles and videos to get through here.

Rock on,
  Hardwick


On Mon, Jul 21, 2014 at 1:24 PM, Jeff Schnitzer  wrote:

> There is a tech talk video online (several years old at this point) by a
> Facebook engineer which describes the lengths they went to to get MySQL to
> scale. You can probably find it with some googling. The upshot is they
> basically use MySQL as a NoSQL system: Data is stored in schemaless blobs &
> there are no joins.
>
> Jeff
>
>
> On Mon, Jul 21, 2014 at 10:12 AM, David Hardwick <
> david.hardw...@bettercloud.com> wrote:
>
>> Thanks, Kaan.
>>
>> I realize my post looks too mired in opinion.  In retrospect, I should
>> have stated that we are using App Engine and the Datastore, and we ARE
>> operating at that scale mentioned (40,000 customers, 10million records each
>> across 50 or so entities), we don't have a DBA, etc..I just wanted to
>> talk to someone that has either:
>>
>>
>>- a)  figured out how to shard a relational database at multi-tenant
>>scale (e.g., 30,000 customers containing 10M records per customer, across
>>50 tables each) such that the relational database scales and is as 
>> reliable
>>as a NoSQL database.  For example, did they just shard on a
>>'namespace' column and shard on that using the database's sharding
>>capabilities vs creating a db/schema for every customer and having to
>>reshuffle these around.
>>
>> ...OR...
>>
>>- b) Overcome querying deficiencies of a NoSQL database by writing
>>the data to both a NoSQL datastore and something like Elastic Search 
>> (which
>>also scales easily).  This would make up for a number of the 'cons' of a
>>NoSQL database for this use case based on a small spike project we did 
>> last
>>summer.  So that's another option I would like to hear from someone about
>>that has implemented this architecture and are fans.  I've recently heard
>>from a person using a 200M record, multi-tentant app in mysql that used
>>Elastic Search in front of it too for speed reasons.
>>
>> Thanks again!
>>   Hardwick
>>
>>
>>
>>
>>  On Sun, Jul 20, 2014 at 3:33 PM, Kaan Soral  wrote:
>>
>>>  Relational databases also require indexes, however NoSQL indexes are
>>> much harder to generate and costly
>>>
>>> You generally have to build with precision when developing with NoSQL,
>>> always knowing what you want and how you want it, build the models and
>>> indexes / systems for that
>>>
>>> There are also many other gotchas you learn with NoSQL - or AppEngine
>>> specifically
>>>
>>> But I will say this, I've never felt hopeless while using AppEngine,
>>> there is always someone out there to help with issues, or an efficient
>>> documentation
>>>
>>> Long story short, I've been using AppEngine at large scale for a while
>>> now, I would suggest it
>>> (The taskqueue system is also pretty strong, you can easily build a
>>> mapping task (map of mapreduce) and iterate over all entities in a
>>> relatively short time, so if costs are not a big issue, you can always
>>> iterate over everything instead of running a flexible relational db query
>>> that you would do otherwise)
>>>
>>> (Issues also happen, but there is always the soothing fact that someone
>>> else solves them at scale, however, since you are intending for a corporate
>>> usage, you might take the responsibility yourself and build a custom
>>> system, would be much harder, much more costly)
>>>
>>> I'm guessing this isn't the detailed reply you were looking for, but
>>> just my .02
>>>
>>> --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "Google App Engine" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/google-appengine/txoW1CyVaPs/unsubscribe
>>> .
>>> To unsubscr

Re: [google-appengine] Re: NoSQL vs Relational DBs - pros and cons

[David Hardwick]

Thanks, Kaan.

I realize my post looks too mired in opinion.  In retrospect, I should have
stated that we are using App Engine and the Datastore, and we ARE operating
at that scale mentioned (40,000 customers, 10million records each across 50
or so entities), we don't have a DBA, etc..I just wanted to talk to
someone that has either:


   - a)  figured out how to shard a relational database at multi-tenant
   scale (e.g., 30,000 customers containing 10M records per customer, across
   50 tables each) such that the relational database scales and is as reliable
   as a NoSQL database.  For example, did they just shard on a 'namespace'
   column and shard on that using the database's sharding capabilities vs
   creating a db/schema for every customer and having to reshuffle these
   around.

...OR...

   - b) Overcome querying deficiencies of a NoSQL database by writing the
   data to both a NoSQL datastore and something like Elastic Search (which
   also scales easily).  This would make up for a number of the 'cons' of a
   NoSQL database for this use case based on a small spike project we did last
   summer.  So that's another option I would like to hear from someone about
   that has implemented this architecture and are fans.  I've recently heard
   from a person using a 200M record, multi-tentant app in mysql that used
   Elastic Search in front of it too for speed reasons.

Thanks again!
  Hardwick




On Sun, Jul 20, 2014 at 3:33 PM, Kaan Soral  wrote:

> Relational databases also require indexes, however NoSQL indexes are much
> harder to generate and costly
>
> You generally have to build with precision when developing with NoSQL,
> always knowing what you want and how you want it, build the models and
> indexes / systems for that
>
> There are also many other gotchas you learn with NoSQL - or AppEngine
> specifically
>
> But I will say this, I've never felt hopeless while using AppEngine, there
> is always someone out there to help with issues, or an efficient
> documentation
>
> Long story short, I've been using AppEngine at large scale for a while
> now, I would suggest it
> (The taskqueue system is also pretty strong, you can easily build a
> mapping task (map of mapreduce) and iterate over all entities in a
> relatively short time, so if costs are not a big issue, you can always
> iterate over everything instead of running a flexible relational db query
> that you would do otherwise)
>
> (Issues also happen, but there is always the soothing fact that someone
> else solves them at scale, however, since you are intending for a corporate
> usage, you might take the responsibility yourself and build a custom
> system, would be much harder, much more costly)
>
> I'm guessing this isn't the detailed reply you were looking for, but just
> my .02
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Google App Engine" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/google-appengine/txoW1CyVaPs/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
 David Hardwick, CTO

 p. 646.237.5388
m. 703.338.0741
a. 3405 Piedmont Road NE, Suite 325, Atlanta, GA 30305
e. david.hardw...@bettercloud.com
 New Guide: Best Practices for Managing Google Apps Using FlashPanel
<http://www.flashpanel.com/guide/?utm_source=email&utm_medium=click&utm_campaign=BCsig>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] NoSQL vs Relational DBs - pros and cons

[David Hardwick]

And how are you?

My purpose with this email to not to seek out a religious debate on NoSQL vs 
Relational databases, *I'm really looking for someone that has had a great 
experience with successfully sharding relational databases for 
multi-tentant applications operating at very large scale, and overcame the 
associated challenges. *(or someone that's combined a NoSQL database and 
Elastic search, for example, to have relational database like query 
features atop a NoSQL database).

*The reason I seek such a person out is to answer the question:   Can I 
have the querying features of a relational database while at the same time 
have the ease of maintaining a NoSQL database with respect to scale, 
replication, and availability when operating a massive, multi-tenant 
database?*

We know there are no magic architectures, and that such comparisons of NoSQL vs 
Relational database really depend on the need.  The need in this case is 
for: 

a highly scalable and reliable database for a multi-tenant application that 
will serve an estimated 40,000 companies with an average of 1000 users per 
company and about 10,000 entries per user (or 10 million entries per 
company spread across about 50 tables).  


Given this particular use case, which I admit is still a bit vague, here is 
a pro/con analysis on NoSQL vs Relational DB based on my experiences with 
both types of databases. 

*When NoSQL databases -* like Google Datastore, Amazon SimpleDB, Opensource 
Casandra and MongoDB - *are stacked up against this use case, they have the 
following pros (+) and cons ( - )*, IMHO:

   - + Scales easily - sharding on a key-blob entry in a NoSQL datastore is 
   fairly easy compared sharding a relational database
   - + Replicates easily - for the same reason as sharding
   - + Query speed is always fast as each query has to be based on a key 
   index. 
   - + Easy to update data structures, no schema management to deal with
   - + Easy to maintain...in cases of Datastore and SimpleDB, no DBA is 
   needed!
   - -  Ad hoc querying across entities/tables is difficult, typically done 
   in the code rather than by the database
   - -  More upfront data processing work is needed to get things like 
   counts on an entity
   - - Sorting & filtering on an entity requires pre-defined indexes

*When Relational Databases* - like SQL Server, Postgres, Oracle, MySQL - *are 
stacked up against this use case* (i.e., 30,000 customers with 10M entries 
each across about 50 tables) *have the following pros (+) and cons ( - )*, 
IMHO:

   - + Easier to query for ad-hoc reporting needs
   - + Easier to throw aggregate queries (group by/ counts/ max/ min) to 
   the database instead of doing it in code
   - + Pagination (go to page x of Y ) is doable, even on really large 
   datasets
   - + Sorting and filtering on an entity or even between entities is 
   easier; cross entity/table querying is easier
   - - Scaling is difficult - sharding strategy needs to be figured out 
   ahead of time.  How to shard for 100,000 customers, with 10M records for 
   each customers (but those 10M records are stored amoungst 50 different 
   tables)
   - - Replication and Scaling is complicated (??), need serious expertise 
   / experience
   - - Queries can drag because there are so many possibilities -- need to 
   have a represented dataset in a non production instance to do pre-analysis 
   and optimization on new code/queries getting introduced.  Need to monitor 
   and analyze the query stats on existing queries to see what needs tuning.
   - - Data structure changes are harder, need to main sequenced DDL and 
   DML scripts, and coordinate deployments to have the database changes go 
   before the code changes
   - - Need a DBA for replication, backups, performance, scalability, 
   monitoring


It almost comes down to saying...

hey, if you want your product team to be able to do any querying they want, 
then go with a relational database but the technology team will have to 
deal with the technical consequences and challenges.


...or ...

hey, you don't want to deal with the technical challenges for maintaining a 
massively scalable and reliable relational database, so go with NoSQL and 
have your product team deal with the consequences and challenges.


I lean toward NoSQL because I've never seen a database scale so easily and 
be so reliable compared to the very large relational database projects I've 
worked on (and I've seen some MASSIVE Oracle environments and datasets in 
my work history in both government and commercial installations, but they 
weren't multi-tenant). 

However, if I was able to hear the experiences of someone that's been able 
to either:

   - a)  figure out how to shard a relational database easily even for 
   30,000 customers containing 10M records per customer, across 50 tables such 
   that the relational database scales and is as reliable as a NoSQL database, 
   then I would definitely go with a Relational Databas

[google-appengine] Re: TaskQueue Rest Api

[David Hardwick]

Hello,

Yeah, this is tricky...we ran into the same...here's some info from past 
emails on the matter...

===
First, we had to get the ACLs setup...
-

The error you're receiving is due to the pull queue configuration rather 
than the explicit enabling in the developer's console:
 
https://developers.google.com/appengine/docs/java/config/queue#Defining_Pull_Queues

Are the identities you're using to interact with the queue via the API 
interface 
specified in the queue's ACL configuration?

-
and after we got past that and setup the ACLs, then needed certain scopes 
to work with OAuth


The problem was this doc:

https://developers.google.com/appengine/docs/java/taskqueue/rest/about_auth

Specifically:

Here's the OAuth 2.0 scope information for the Google App Engine Task Queue:
ScopeMeaninghttps://www.googleapis.com/auth/taskqueueRead/write access.
https://www.googleapis.com/auth/taskqueue.consumerRead-only access.


These work OK if you want to create per-user Access/Refresh tokens against
your App ID. The thing is, we didn't want to hard code these. We were
assuming that, like all the other Google Cloud * APIs we could use our
app's Service Account/PK (which is talked about on the OAuth details page
the auth page links to for "more info") to talk to this. We tried adding
the ACL entries, but using the Service Account without a user gave us the
"This looks disabled" error. If we specified a user to talk on behalf of we
got an "Invalid Scope" error.

Our breakthrough came when we saw this page:

https://developers.google.com/appengine/docs/java/taskqueue/rest/taskqueues/get

Where it has additional scopes:

This request requires authorization with at least one of the following
scopes (read more about authentication and
authorization<
https://developers.google.com/appengine/docs/java/taskqueue/rest/about_auth>
).
Scope
https://www.googleapis.com/auth/taskqueue
https://www.googleapis.com/auth/cloud-taskqueue
https://www.googleapis.com/auth/taskqueue.consumer
https://www.googleapis.com/auth/cloud-taskqueue.consumer


(With the link back to the other page). Turns out the cloud-* scopes work
just fine with the cloud service token. However, it was dumb luck we found
they existed on this page. They are also not part of the static final
Strings on the ComputeScopes class in the packaged client library. Once we
hard coded the cloud-* scope strings into our app, everything started
working as expected.



On Wednesday, June 18, 2014 12:25:28 AM UTC-4, poke wrote:
>
> Anyone here can use TaskQueue Rest Api  .
> I can't  even in API explorer
>
> https://developers.google.com/apis-explorer/#p/taskqueue/v1beta2/taskqueue.tasks.list
>
> 403 Forbidden
>
> - Show headers -
>  
> {
>  "error": {
>   "errors": [
>{
> "domain": "global",
> "reason": "forbidden",
> "message": "you are not allowed to make this api call"
>}
>   ],
>   "code": 403,
>   "message": "you are not allowed to make this api call"
>  }
> }
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Can't integrate GCS to existing AppEngine project

[David Hardwick]

We had a small stumble with Google Cloud Storage too, recently, but not 
calling from App Engine but from Compute.  We tried to make a call to it 
but it returned that we were over quota on our first call.  We could upload 
files from the web interface, so we were confused.  Turns out that the JSON 
version of the API we were using is v1/experimental, and they start you off 
at 0 quota!  

Something to keep in mind if you are using the Cloud Storage from outside 
of app engine and you turn on that "Google Cloud Storage" service in the 
console and then you turn on that "Google Cloud Storage JSON API" service 
right below it.

Rock on,
  Hardwick

We put in a request for quota and will wait a week (requesting more quota 
can take up to a week to be reviewed/approved...and make sure you give 
formulaic reasons behind the numbers you request).  If we don't get 
approved, we'll do the XML approach.

On Friday, August 30, 2013 2:27:07 AM UTC-4, Denis Sigov wrote:
>
> Vinny,
>
> Ultimately, as you adviced, I went to APIs console, selected my project 
> (created by integrating GAE app), and switched to ON Cloud Storage (it was 
> OFF).  Now Cloud Storage buckets are creating fine. 
>
> Answers for your questions: 
>
>> When you tried another project, was that a project created by the "New 
>> Project" button on the Google Cloud Console, or by integrating an App 
>> Engine application into the console?
>
> By "New Project" button from Cloud Console. Now, I've tried to enable 
> billing for that project and everything works fine. 
>
> Can you try this: Go to the cloud console and create a new project. Set up 
>> billing on that project. Then go to Cloud Storage and try to set up a 
>> bucket.
>
> Done. New project with billing enabled works fine. 
>
> Thanks a lot for your help!
>
> Den
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Search API questions for app engine team

[David Hardwick]

Can this limit be segmented by namespaces?  So you can have one search 
index for namespaceA and then another for namespaceB?  (each maxing out at 
250GB during the GA period)

On Sunday, June 30, 2013 8:19:52 PM UTC-4, Mahron wrote:
>
> This limit will be a problem down the road if it can not be lifted, so I 
> am not really sure if I should implement it or not.
>
> I also do not understand if those 250 GB are for the text only, or does it 
> include whatever storage if used for indexing (which would make things even 
> worse).
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Frustrated by logs

[David Hardwick]

We used Splunk to analyze our GAE logs and it was extremely helpful.  It 
can be costly, but the insight it gives you can also save you a lot of 
costs too.

On Friday, May 24, 2013 11:51:23 AM UTC-4, Kristopher Giesing wrote:
>
> I'm having a devil of a time using the logs to debug my application.
>
> There's a basic pattern of access that I'm trying to troubleshoot.  In 
> order to do this, I would like to analyze correlations between the 
> following properties of requests:
>
> - Request path
> - Request status
> - Instance ID used to serve the request
>
> If I could filter on the first two properties in the admin console, I 
> would probably be able to diagnose the issue.  But that never seems to work 
> properly; the admin console seems to give me some random subset of the 
> relevant logs - sometimes none, sometimes a few, but never all of them.
>
> So I tried downloading the logs using appcfg.sh, and then filtering the 
> logs myself.  That works - except that the amount of data per request that 
> you get from downloading logs doesn't include the instance ID used to serve 
> the request!
>
> I'm getting enormously frustrated by this.  Am I missing something about 
> the admin console, or is its search function really this broken?  Is there 
> some way to get more verbose logging from the logs-download feature of 
> appcfg.sh?
>
> Any insights or help would be greatly appreciated.
>
> Thanks,
>
> - Kris
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Task Queue - Transient Errors

[David Hardwick]

Hello,

Yes, we have been getting these tootoday was particularly bad because 
of the issues with the task queues - 
https://code.google.com/status/appengine/detail/taskqueue/2013/04/18#ae-trust-detail-taskqueue-add-latency

We believe the attempt to grab the task from queue is failing temporarily, 
and it wants you to try again in a another second or so.


On Thursday, April 18, 2013 10:21:41 AM UTC-4, Santiago wrote:
>
> Hi everyone, In the last days I've been seeing on my application a lot of 
> Transient Errors when I try to use task queues. is this happening to anyone 
> else? do you know what actually happen when this error is raised? all i get 
> from GAE documentation is "There was a transient error while accessing 
> the queue. Please try again later"  but I'm not very confortable about 
> having my logs flooded with this! thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Changing AppEngine's XMPP account profile picture

[David Hardwick]

No, I've had the same need and it cannot be done.  I'm pretty sure there's 
an issue tracker you vote up on it.

On Monday, November 12, 2012 6:24:37 PM UTC-5, Antonio Carioca wrote:
>
> Hi, 
>
> I understand that when using XMPP on AppEngine, the account used is based 
> on your app_id (i.e. *your_app_id*@appspot.com). Now, in the past, when I 
> used XMPP through other services, I created my a new account for the app, 
> which allowed me to customise it in terms of the name and profile picture 
> shown on the client. That made messages coming from the application look 
> much more professional - having the default blank picture showing up on 
> Google Talk makes it look like a half baked solution.
>
> So, my question is, is there anyway to customise the *your_app_id*@
> appspot.com account in the same way you do with a regular Google account, 
> particularly when it comes to changing its display name and profile picture?
>
> Thanks!
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/WVLsxA1j_T0J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Sending emails from gae using mailgun

[David Hardwick]

Do any of these email services offer template based emails, where the 
marketing department can control the copy with some ${thisIsAVariable} 
inserts into the copy, and the development team just needs to send over the 
variables to the email?

ExactTarget has this but I was just curious if others had any other Vendors 
that also provide this feature.  That way the marketing team can test 
different copy in the subject and body to increase conversion rates on 
opening and clicking through without requiring a deployment to update the 
email template buried in the code.

Rock on,
  -Hardwick



On Friday, November 9, 2012 4:30:46 PM UTC-5, Emmanuel Mayssat wrote:
>
> I am writing an app that requires email to be sent (notifications)
> I am contemplating using mailgun as a service.
>
> 1/ I s there a email web service you recommended?
>
> I can send emails through mailgun using the 'request' python module
>
> def send_simple_message():
> data = {"from": "Excited User 
> >",
>   "to": ["t...@gmail.com ", 
> "to...@outlook.com"], 
> #
>   "subject": "[https] Hello Mama (REQUEST)",
>   "text": "Testing some Mailgun awesomness!"}
> return requests.post(
> mailgun_url,
> auth = HTTPBasicAuth(username, password), 
>   # <-- or simply auth=(username, password)
> data = data )
>
> I can also send emails from the shell using urllib2 (More complex because 
> of https)
>
> Cutting and pasting my urllib2 code in my gae app doesn't report any error 
> or anything else
>
> 2/ Can email be send when using the local gae server or the app have to be 
> fully deployed?
>
> 3/ Is there a way to send emails through google app engine? (including for 
> users that do not have gmail accounts)
>
> 4/ I cannot make sense of the gae policy as far as sending emails
> What are administrator password?
> Where are the administrator accounts?
> My domain is not hosted on gae.
> I am just using gae for test right now.
>
> Regards
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/2VU9HsHY4ZwJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Generating Reports using GAE

[David Hardwick]

We use the Doc List API (but go with Drive these days) and it is reliable. 
 You can create a Google Doc, save as HTML, then input your REPLACE_THIS 
variables to create a template.  After doing so, you then upload that 
document with the convert option to create a Google Doc.  You can make 
another request to make that into a PDF (not sure about that last one, 
doing it from memory, we usually just go to Docs, but I'm pretty sure you 
get get that doc to a PDF using the API).


On Thursday, October 18, 2012 11:21:29 AM UTC-4, Richard Watson wrote:
>
> Thaseena, depending on your requirements there are various approaches.
>
> http://code.google.com/p/g2-report-engine/wiki/AppEngineSupport (found 
> via Google - no idea how good it is)
>
> I'm using Docmosis for simple PDF output.  http://docmosis.com/
>
> Alternatively you'll have to use a generator for e.g. PDF or Excel or 
> whatever you want to export as.  You can also connect to Google Documents 
> via the API, but I'm not sure how reliable that is.
>
> On Thursday, October 18, 2012 9:03:28 AM UTC+2, Thaseena Banu wrote:
>>
>> Thank you, could you help me using java language.
>>
>> On Wednesday, 17 October 2012 21:13:58 UTC+5:30, Moises Belchin wrote:
>>>
>>> If you need simple reports you can write your own reportes using PyFPDF 
>>> to generate PDF reports for your app.
>>>
>>> http://code.google.com/p/pyfpdf/
>>>
>>> PyFPDF is ported from php FPDF http://www.fpdf.org/
>>>
>>> If you need manipulate big amount of data I recomend you  Google 
>>> BigQuery.
>>>
>>> https://developers.google.com/bigquery/
>>>
>>> Happy coding !
>>>
>>> Regards.
>>> Moisés Belchín.
>>>
>>>
>>>
>>> 2012/10/17 Barry Hunter 
>>>
 By writing some code. 


 On Wed, Oct 17, 2012 at 4:12 PM, Thaseena Banu wrote:

>
> how do we Generate reports or print the form using GAE? 
>
> -- 
> You received this message because you are subscribed to the Google 
> Groups "Google App Engine" group.
> To view this discussion on the web visit 
> https://groups.google.com/d/msg/google-appengine/-/CXq1gltfBxsJ.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengi...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

  -- 
 You received this message because you are subscribed to the Google 
 Groups "Google App Engine" group.
 To post to this group, send email to google-a...@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengi...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.

>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/uegAGYOWB3oJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: java.util.ConcurrentModificationException on committing a transaction with only task queue entries

[David Hardwick]

We can say unequivocally that under high load, if you have a transaction 
around a task enqueue without any database calls, you will get the 
ConcurrentModificationException error.

This is how the Pipeline library is setup for FINALIZE_JOB.

If you need to implement this pattern, then I would create a non-sense 
database call to see if that will work...we didn't need to make a database 
call so we took it out of the transaction, but the code that did have 
database operations and task enqueues within the transaction did not have 
this error.

On Saturday, June 16, 2012 4:44:29 PM UTC-4, Michael Hermus wrote:
>
> I just saw the issue pop up again, so it appears that (as you correctly 
> predicted) Objectify had nothing to do with the issue. 
>
> Sigh... I am not sure how to approach a resolution for this, because my 
> app makes significant use of this pattern. Do I perhaps need to store at 
> least one entity with each transaction in order to avoid this?
>
> On Tuesday, June 12, 2012 3:20:43 PM UTC-4, Michael Hermus wrote:
>>
>> Jeff,
>>
>> I haven't seen the problem since I made the change, and it seemed 
>> consistent under certain circumstances before that point. However, I have 
>> relatively few data points, so I wouldn't call it conclusive. If I have 
>> time, I will switch the transaction back to Objectify and see if that 
>> causes the issue to crop up again.
>>
>> Mike
>>
>> On Monday, June 11, 2012 7:20:08 PM UTC-4, Jeff Schnitzer wrote:
>>>
>>> Ah, sorry that wasn't clear to me. 
>>>
>>> I'm not sure how the presence of Objectify in this chain could make 
>>> any difference.  There's nothing special about the way Objectify 
>>> starts or commits a transaction.  Was this behavior consistent? 
>>>
>>> Jeff 
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/jdreH4oxnLwJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] The cronjobs keep coming every hour.

[David Hardwick]

We have had similar support requests yesterday...every hour even though 
scheduled once per week.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/UJcdsdjTlMMJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: How to move data reliably from front to backends ?

[David Hardwick]

We do pull queues on backends, and use the pipeline library to insure
reliability.



On Jul 29, 7:42 pm, Jeff Schnitzer  wrote:
> On Sun, Jul 29, 2012 at 7:19 AM, Richard  wrote:
>
> > How can I do this reliably in a 10 second window ?   I have had using a PULL
> > queue suggested, but I don't want to do all the work of converting the app
> > over if it will be just as unreliable (I remember posts about tasksqueue's
> > getting "stuck")
>
> The task queues that have gotten "stuck" in the past were push queues;
> basically, the machines that do the pushing fell behind (or were
> temporarily suspended due to problems).  I haven't heard of any
> equivalent problem with pull queues.  The queue would have to start
> throwing errors on service calls, or "losing" tasks... probably not
> impossible, but on par with "datastore requests failing".  It would be
> a major failure.
>
> Does anyone have any comments about the reliability of pull queues?
>
> I'm only just now starting to work with pull queues.  While I do allow
> that my push queues may not fire in a timely manner, I'm engineering
> to expect pull queues to always be available.
>
> Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Differentiated service with Google App Engine

[David Hardwick]

Hi 

If you put in a feature request to have two+ apps access the same datastore 
natively then I will definitely star it!


On Thursday, July 26, 2012 3:11:48 PM UTC-4, rerngvit yanggratoke wrote:
>
> Hello,
>Anyone have an idea on how to achieve a differentiated service with 
> Google App Engine? I have two types of users. One is premium while another 
> one is free user. I would like to set different minimum latencies for 
> firing up new GAE instances between the two. In other words, I am willing 
> to pay extra for the traffic generated from the premium users. Nonetheless, 
> for the free users, I would prefer to pay at minimum or within the daily 
> free quota provided by Google. In my case, I expect that the free users are 
> able to tolerate higher latency than the premium ones. I believe that this 
> case should be quite general and apply to many people.
>Assuming that I can identify types of users by the URLs they use or 
> via other mechanisms. The only way I could think of is creating two 
> separated applications and letting the premium users access the one that 
> enables billing, while free users access the one that doesn't. However, my 
> free and premium users need to share the datastore because they are also 
> have to communicate within the application. Unfortunately, it seems that 
> sharing a datastore across applications is not possible at the moment (feel 
> free to correct me if I am wrong). 
>
> Any ideas or suggestions?
>
> Cheers,
> Rerngvit
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/-mTVi29GpDwJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Differentiated service with Google App Engine

[David Hardwick]

If you put in a feature request to have more than one app access the same 
datastore natively, then I will surely start that issue!

On Thursday, July 26, 2012 3:11:48 PM UTC-4, rerngvit yanggratoke wrote:
>
> Hello,
>Anyone have an idea on how to achieve a differentiated service with 
> Google App Engine? I have two types of users. One is premium while another 
> one is free user. I would like to set different minimum latencies for 
> firing up new GAE instances between the two. In other words, I am willing 
> to pay extra for the traffic generated from the premium users. Nonetheless, 
> for the free users, I would prefer to pay at minimum or within the daily 
> free quota provided by Google. In my case, I expect that the free users are 
> able to tolerate higher latency than the premium ones. I believe that this 
> case should be quite general and apply to many people.
>Assuming that I can identify types of users by the URLs they use or 
> via other mechanisms. The only way I could think of is creating two 
> separated applications and letting the premium users access the one that 
> enables billing, while free users access the one that doesn't. However, my 
> free and premium users need to share the datastore because they are also 
> have to communicate within the application. Unfortunately, it seems that 
> sharing a datastore across applications is not possible at the moment (feel 
> free to correct me if I am wrong). 
>
> Any ideas or suggestions?
>
> Cheers,
> Rerngvit
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/cs1aYijZARwJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Startup time exceeded...on F4?!

[David Hardwick]

Amazing, Andrei.  If you can get those startup times going with GWT and the 
complexities you mentioned in your app, then clearly stripping out DI and 
using low-level API (and likely jar'ing everything) is a mega contributor 
to reducing startup time.  I would hate to have to spend the team's time 
stripping that stuff out, i would just pay for higher idle instances until 
request #7865 gets implemented.

I agreed that only dispatching to warm instances makes the most sense and 
would negate the need to have high idle instances or stripping out 
frameworks like Guice and Objectify...but if at some point in the future as 
the code base grows even larger and we are consistently exceeding the hard 
limits then we have to either breakup the app as Brendan mentions or start 
stripping out frameworks as several of you have mentioned.

Clear GAE was having a bad day when i posted this in the first place 
because we haven't had these issues of exceeding the hard limits on warmup 
times and we've only been adding to the code base since.  

Maybe it was just that day, but the other question that I asked in my 
original post was why were our Staging cold-startup times faster than our 
Production warm-up requests?  I could understand a few more seconds for 
warmups, but the warmups in production on F4s were exceeding the hard 
limits and the cold starts on our staging environment on F1s were happening 
in like 30 seconds.  

Some other part of the infrastructure must be used during warmups and that 
"part" must have been having a really bad day.  Why else would warmup 
requeests on F4s be able to lose to cold starts on F2s?

Rock on,
  -Hardwick

On Wednesday, July 25, 2012 6:28:01 AM UTC-4, Andrei Volgin wrote:
>
> I have a large GWT app with over 50 complex data entities with very 
> complicated relationships between them. There are over 100 RPC methods, and 
> I use various GAE services (Users, Memcache,  Blobstore, Images, and Mail). 
> My new instance startup time ranges from 4 to 5 sec on F1. Sometimes it 
> goes a little higher. The lowest was 3.6 sec. I use the low-level API and 
> no DI. Productivity tools are nice, but they have their cost.
>
> In any case, if Jeff's suggestion is implemented (dispatch only to warm 
> instances), it would not matter if the start up time is 4 sec or 40. I 
> followed this discussion closely, and to me it is obvious that the existing 
> GAE mechanism is inferior to what Jeff is proposing (for any number of 
> instances). There may be a good technical reason why the GAE team chose 
> their current implementation. It is also possible that GAE was built by 
> engineers who did not think in terms of web user experience, and who 
> believed that a rare 30 seconds delay is the acceptable price to pay for 
> some other benefits of the current implementation. This proposal was 
> already starred by many people (issue 7865). It would be nice if the GAE 
> teams takes another look and tells us whether we should expect any changes. 
> I am sure a lot of app developers would like to know whether they should 
> optimize for fast cold starts. These optimizations are expensive (both in 
> direct development time and in lost productivity), so this is not an idle 
> question.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/ncTsPLWS7JUJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Startup time exceeded...on F4?!

[David Hardwick]

Some additional observations and questions...

After reading this [Link 1] stack overflow article that mentioned an
issue with having your Max Idle count below 6, we started looking at
our warmup request on our staging environment because that app-id has
Idle Instances set to Auto-Auto, while production had specific values.

But...Where did all the "/_ah/warmup" requests go?  When doing a label
search for these staging environment logs ["path:/_ah/warmup" (doing a
label search)] we couldn't find any warmup request!!(yes, we have
warmup requests turned on)...we would just see the first cold-start
request would take around 15 seconds to load (F1) and 10 seconds to
load on (F2).

I even shut down every instance and hit the staging server again to
see if I could find a warmup request in the logs...nope.  Honestly, I
would rather have a user wait 10 seconds for the first request to that
server as opposed risking the warmup requests failing again.

Where did all the "/_ah/warmup" requests go?   More importantly, why
would we have such different times for warmup requests compared to
cold starts?  Shouldn't they be nearly identical?!

Rock on,
  -Hardwick

[Link 1] - 
http://stackoverflow.com/questions/9422698/ah-warmup-producing-harddeadlineexceedederror


On Jul 12, 12:26 pm, David Hardwick 
wrote:
> Hello,
>
> I realize there's been a lot of discussion on startup times exceeded on
> this forum recently, but wanted needed to post this experience we had this
> morning to keep the attention on this important issue.
>
> We uploaded a point release of our app to a "not-live" version this morning
> and, of course, we were going to click around on that instance to make sure
> it's all kosher before making that version "live."   The warm-up requests
> for the "not-live" version were exceeding the deadline limit of 60s...
> __and__we__are__on__F4s__!_!.
>
> However, the LIVE version of the app crashed too, 500 server errors,
> instance counts went to zero, all sorts of whacky stuff was seen in the
> control panel.  All that happened to our LIVE version without when all we
> did was upload another "non-live" version and hit it with a single
> request...did I mention we were on F4s?  ;-)  Does the failure of any
> instance to exceed the 60s limit take down all instances to include live
> one?
>
> We did a few things as quickly as possible since our live application was
> down, so clearly we didn't have the time to take the scientific approach of
> only changing one thing at a time and wait to see if it that did it.
>
> We...
> 1. Switched from F4s to F2 (i figured if this would least get us on some
> new servers/instances)
> 2. Increased max idle instances from 1 to 2 (with F4s running, I'm fine
> with having just 1 idle instance and not at all happy about paying for 2
> idle instances, so maybe we'll just increase this prior to deployments and
> then back down again after the deployment succeeds until we know more)
> 3. Made the recently uploaded version live (hey, why not, the production
> app was down for 10 minutes, so how much more harm could we do?)
>
> We use GWT and Guice, we jar everything (as I have been paying attention to
> this startup time discussions for quite some time now.  We are also
> considering switching our Guice libraries to a non-AOP version as we saw
> suggested in another blog since we just need the injection.
>
> Any insight, and I'm all ears!  app_id=s~myflashpanel
>
> Regards,
>   -Hardwick
>
> --
>
>  *We make Google Apps even better.*
>
> *David Hardwick*
> *CTO*
> david.hardw...@bettercloud.com
>
> *Signature by Flashpanel <http://flashpanel.com/>*
>  *See us in Mashable: Growing Up Google: How Cloud Computing Is Changing a
> Generation <http://mashable.com/2012/04/30/generation-growing-up-google/>*

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Startup time exceeded...on F4?!

[David Hardwick]

Hello,

I realize there's been a lot of discussion on startup times exceeded on
this forum recently, but wanted needed to post this experience we had this
morning to keep the attention on this important issue.

We uploaded a point release of our app to a "not-live" version this morning
and, of course, we were going to click around on that instance to make sure
it's all kosher before making that version "live."   The warm-up requests
for the "not-live" version were exceeding the deadline limit of 60s...
__and__we__are__on__F4s__!_!.

However, the LIVE version of the app crashed too, 500 server errors,
instance counts went to zero, all sorts of whacky stuff was seen in the
control panel.  All that happened to our LIVE version without when all we
did was upload another "non-live" version and hit it with a single
request...did I mention we were on F4s?  ;-)  Does the failure of any
instance to exceed the 60s limit take down all instances to include live
one?

We did a few things as quickly as possible since our live application was
down, so clearly we didn't have the time to take the scientific approach of
only changing one thing at a time and wait to see if it that did it.

We...
1. Switched from F4s to F2 (i figured if this would least get us on some
new servers/instances)
2. Increased max idle instances from 1 to 2 (with F4s running, I'm fine
with having just 1 idle instance and not at all happy about paying for 2
idle instances, so maybe we'll just increase this prior to deployments and
then back down again after the deployment succeeds until we know more)
3. Made the recently uploaded version live (hey, why not, the production
app was down for 10 minutes, so how much more harm could we do?)

We use GWT and Guice, we jar everything (as I have been paying attention to
this startup time discussions for quite some time now.  We are also
considering switching our Guice libraries to a non-AOP version as we saw
suggested in another blog since we just need the injection.

Any insight, and I'm all ears!  app_id=s~myflashpanel

Regards,
  -Hardwick

-- 

 *We make Google Apps even better.*

*David Hardwick*
*CTO*
david.hardw...@bettercloud.com

*Signature by Flashpanel <http://flashpanel.com/>*
 *See us in Mashable: Growing Up Google: How Cloud Computing Is Changing a
Generation <http://mashable.com/2012/04/30/generation-growing-up-google/>*

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Will we be able to buy discounted instance-hours for B1 backends?

[David Hardwick]

Here's the feature request here...

http://code.google.com/p/googleappengine/issues/detail?id=7411

On Jul 1, 2:31 pm, Robert Fischer  wrote:
> Hi,
>
> It's been some time since this post and I've been wondering the same thing.
>  Are there any plans to apply purchased discounted hours toward B1 (or
> other B#) instance hours?
>
> Thanks,
> Robert
>
>
>
>
>
>
>
> On Friday, September 16, 2011 2:54:47 PM UTC-7, Greg D'Alesandre wrote:
>
> > We don't plan to provide it just yet but could do so in the future.
>
> > Greg
>
> > On Fri, Sep 16, 2011 at 1:14 PM, Jason Collins 
> > wrote:
>
> >> The B1 backend specs are the same as frontend specs.
>
> >> Will we be able to buy discounted instance-hours for backend B1
> >> instances?
>
> >> Any possibility of a similar discount being applied to B2/B3/B4 for
> >> pre-commitment?
>
> >> j
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Google App Engine" group.
> >> To post to this group, send email to google-appengine@googlegroups.com.
> >> To unsubscribe from this group, send email to
> >> google-appengine+unsubscr...@googlegroups.com.
> >> For more options, visit this group at
> >>http://groups.google.com/group/google-appengine?hl=en.
>
> On Friday, September 16, 2011 2:54:47 PM UTC-7, Greg D'Alesandre wrote:
>
> > We don't plan to provide it just yet but could do so in the future.
>
> > Greg
>
> > On Fri, Sep 16, 2011 at 1:14 PM, Jason Collins 
> > wrote:
>
> >> The B1 backend specs are the same as frontend specs.
>
> >> Will we be able to buy discounted instance-hours for backend B1
> >> instances?
>
> >> Any possibility of a similar discount being applied to B2/B3/B4 for
> >> pre-commitment?
>
> >> j
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Google App Engine" group.
> >> To post to this group, send email to google-appengine@googlegroups.com.
> >> To unsubscribe from this group, send email to
> >> google-appengine+unsubscr...@googlegroups.com.
> >> For more options, visit this group at
> >>http://groups.google.com/group/google-appengine?hl=en.
>
> On Friday, September 16, 2011 2:54:47 PM UTC-7, Greg D'Alesandre wrote:
>
> > We don't plan to provide it just yet but could do so in the future.
>
> > Greg
>
> > On Fri, Sep 16, 2011 at 1:14 PM, Jason Collins 
> > wrote:
>
> >> The B1 backend specs are the same as frontend specs.
>
> >> Will we be able to buy discounted instance-hours for backend B1
> >> instances?
>
> >> Any possibility of a similar discount being applied to B2/B3/B4 for
> >> pre-commitment?
>
> >> j
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Google App Engine" group.
> >> To post to this group, send email to google-appengine@googlegroups.com.
> >> To unsubscribe from this group, send email to
> >> google-appengine+unsubscr...@googlegroups.com.
> >> For more options, visit this group at
> >>http://groups.google.com/group/google-appengine?hl=en.
>
> On Friday, September 16, 2011 2:54:47 PM UTC-7, Greg D'Alesandre wrote:
>
> > We don't plan to provide it just yet but could do so in the future.
>
> > Greg
>
> > On Fri, Sep 16, 2011 at 1:14 PM, Jason Collins 
> > wrote:
>
> >> The B1 backend specs are the same as frontend specs.
>
> >> Will we be able to buy discounted instance-hours for backend B1
> >> instances?
>
> >> Any possibility of a similar discount being applied to B2/B3/B4 for
> >> pre-commitment?
>
> >> j
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Google App Engine" group.
> >> To post to this group, send email to google-appengine@googlegroups.com.
> >> To unsubscribe from this group, send email to
> >> google-appengine+unsubscr...@googlegroups.com.
> >> For more options, visit this group at
> >>http://groups.google.com/group/google-appengine?hl=en.
>
> On Friday, September 16, 2011 2:54:47 PM UTC-7, Greg D'Alesandre wrote:
>
> > We don't plan to provide it just yet but could do so in the future.
>
> > Greg
>
> > On Fri, Sep 16, 2011 at 1:14 PM, Jason Collins 
> > wrote:
>
> >> The B1 backend specs are the same as frontend specs.
>
> >> Will we be able to buy discounted instance-hours for backend B1
> >> instances?
>
> >> Any possibility of a similar discount being applied to B2/B3/B4 for
> >> pre-commitment?
>
> >> j
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Google App Engine" group.
> >> To post to this group, send email to google-appengine@googlegroups.com.
> >> To unsubscribe from this group, send email to
> >> google-appengine+unsubscr...@googlegroups.com.
> >> For more options, visit this group at
> >>http://groups.google.com/group/google-appengine?hl=en.
>
> On Friday, September 16, 2011 2:54:47 PM UTC-7, Greg D'Alesandre wrote:
>
> > We don't plan to provide it just yet but could do so in the future.
>
> > Greg
>
> > On Fri, Sep 16, 2011 at 1:14 PM, Jason Collins 
> > wrote:
>
> >>

[google-appengine] Re: Code breaks with upgrade to 1.6.6 - using google-api-java-client

[David Hardwick]

It is not his environment, it happened for us (our posted the solution/
work around on stack overflow), it is related to the latest release.

On May 30, 4:11 pm, Yaniv Inbar  wrote:
> Thanks for reporting your findings Koen.
>
> UrlFetchTransport is still supposed to be the recommended choice on Google
> App Engine.  There is a flaw in Google App Engine implementation of
> HttpURLConnection in terms of parsing HTTP headers, whereas UrlFetch
> doesn't have that flaw.  But if NetHttpTransport is working for you, then
> you may continue to use it.
>
> However, honestly I have not tried it with App Engine 1.6.6.  I have only
> tried it on 1.6.5.  I'd really like to see an investigation of what causes
> it to fail on 1.6.6.  Another possibility is that it is not new to 1.6.6,
> but rather something different about the way you set up your environment.
>
> -- Yaniv
>
>
>
> On Tuesday, May 29, 2012 4:47:53 PM UTC-4, Koen Maes wrote:
>
> > I got confirmation from another user in the google-api-java-client group.
> > The workaround is to drop UrlFetchTransport and use NetHttpTransport which
> > seems to be supported nowadays on AppEngine.
>
> > see here :
> >https://groups.google.com/forum/?fromgroups#!topic/google-api-java-cl...
>
> > My app  is working in production now.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Backends: always-on vs. dynamic when it comes to scaling up/down instances

[David Hardwick]

And how are you?

We are using backends in a dynamic capacity now, where the backends cycle
down to zero if we have no further requests to process in our task queues
and pipelines.

However, when we start up our 10 instance-dynamic backend, we have to wait
(and pay for) all 10 instances to start up and then these 10 instances will
typically go back down to 1 or 2 instances since the load is typically not
large enough to need all 10 instances to start, but can certainly build to
that load (which is why we go with 10 as the upper limit), and hour or two
later.

Sorry for asking these two questions that we can answer ourselves by doing
our own experiments, but the team is swamped so just wanted to ask around
first.

   1. Or is there a way to start dynamic backends with just 1 instance,
   then scale up to 10 as needed?
   2. If we do always-on the backend, then will it scale down to just one
   instance during non-peak times and then scale up to 10 if needed?  I'm fine
   with paying for just one instance to be always-on if it scales up and down
   dynamically from there.

Rock on,
  -Hardwick


-- 
David Hardwick
BetterCloud
 86 Chambers St. Suite 704
New York, NY 10007

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Reserved Instance Hours for Backends

[David Hardwick]

And how are you?

Please vote for this feature request for RESERVE INSTANCE HOURS FOR
BACKENDS if you use Backends and you want to save money ($0.08 ->
$0.05/hour is a 37.5% savings).

http://code.google.com/p/googleappengine/issues/detail?id=7411

Rock the vote!
  -Hardwick

On Apr 20, 8:51 pm, Takashi Matsuo  wrote:
> Hi David,
>
> On Sat, Apr 21, 2012 at 4:05 AM, David Hardwick
>
>  wrote:
> > And how are you?
>
> > Any talk of having Reserved Instance Hours for Backends anytime soon?
>
> > Backends CPU is our greatest cost right now, so if we could get $0.05/
> > hour instead of $0.08, then we have a 37.5% savings, which would be
> > huge.
>
> I think this is a very reasonable feature request.
>
> BTW, do you mind filing a issue on the public issue tracker, and call
> for starring it? This is one of the best way to show how many people
> needs a specific feature.
>
> Thanks,
>
> -- Takashi
>
>
>
> > Super stingy,
> >  -Hardwick
>
> > --
> > You received this message because you are subscribed to the Google Groups 
> > "Google App Engine" group.
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to 
> > google-appengine+unsubscr...@googlegroups.com.
> > For more options, visit this group 
> > athttp://groups.google.com/group/google-appengine?hl=en.
>
> --
> Takashi Matsuo | Developer Advocate | tmat...@google.com | 03-6384-9224

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Reserved Instance Hours for Backends

[David Hardwick]

And how are you?

Any talk of having Reserved Instance Hours for Backends anytime soon?

Backends CPU is our greatest cost right now, so if we could get $0.05/
hour instead of $0.08, then we have a 37.5% savings, which would be
huge.

Super stingy,
  -Hardwick

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: We have an index record for null value?

[David Hardwick]

Awesome, we are using Objectify!

Many thanks, Jeff!  You are a great asset to this list.

On Apr 12, 8:09 pm, Jeff Schnitzer  wrote:
> Nulls are perfectly reasonable things to index.  Sometimes nulls are
> what you want to query for.
>
> What you want is called a 'partial index'.  The facility to do partial
> indexes exists in GAE, but how to do it depends on your data access
> layer.  Are you python (db or ndb?) or java (jdo/jpa, objectify,
> low-level?).
>
> In Objectify-land it looks something like this:
>
> @Index(IfNotNull.class) String name;
>
> You can get effectively the same effect by flagging certain values as
> not saved.  But there are so many apis and so many ways to do this
> that I wouldn't want to try to list them all here.
>
> Jeff
>
> On Thu, Apr 12, 2012 at 7:56 PM, David Hardwick
>
>
>
>
>
>
>
>  wrote:
> > And how are you?
>
> > I see records in our custom index for what I know is a null value.  Meaning,
> > we index a string field on the model and it needs to be a custom index.
>
> > However, this field is not required to be filled out by the user, and 9
> > times out of 10 this field is not filled out by a user...but based on the
> > index statistics, I can see that we have an index record regardless if that
> > value is filled in or if it is null.  Why write to the index for a null
> > value?
>
> > Is this expected behavior and we have to take a custom index approach to not
> > get charged index writes for null values?  Or is this a known issues that
> > will be addressed at some point?
>
> > Rock on,
> >   -Hardwick
>
> > --
> > Check out these BetterCloud Products: SherpaTools - DomainWatch - Archive
> > Migrator - Google Gooru
>
> > David Hardwick
> > CTO,
> > BetterCloud
> > Mobile: 703-338-0741
> > 86 Chambers St. Suite 704
> > New York, NY 10007
> > tungle.me/davidhardwick (calender availability)
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine+unsubscr...@googlegroups.com.
> > For more options, visit this group at
> >http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] We have an index record for null value?

[David Hardwick]

And how are you?

I see records in our custom index for what I know is a null value.
 Meaning, we index a string field on the model and it needs to be a custom
index.

However, this field is not required to be filled out by the user, and 9
times out of 10 this field is not filled out by a user...but based on the
index statistics, I can see that we have an index record regardless if that
value is filled in or if it is null.  Why write to the index for a null
value?

Is this expected behavior and we have to take a custom index approach to
not get charged index writes for null values?  Or is this a known issues
that will be addressed at some point?

Rock on,
  -Hardwick

-- 
Check out these BetterCloud Products:
SherpaTools<http://www2.bettercloud.com/e/10212/er-amputm-campaign-sherpatools/5b2cf/21170992>
 - 
DomainWatch<http://www2.bettercloud.com/e/10212/er-amputm-campaign-sherpatools/5b2cr/21170992>
 - Archive 
Migrator<http://www2.bettercloud.com/e/10212/er-amputm-campaign-sherpatools/5b2d3/21170992>
 - Google 
Gooru<http://www2.bettercloud.com/e/10212/er-amputm-campaign-sherpatools/5b2df/21170992>

David Hardwick
CTO,
BetterCloud
 Mobile: 703-338-0741
86 Chambers St. Suite 704
New York, NY 10007
tungle.me/davidhardwick (calender availability)

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Admin Settings: Number of Instances of my Application

[David Hardwick]

Billing has to be enabled to use the slider.

On Apr 7, 6:42 am, Kwame  wrote:
> The slider is not working for me either. This is the only way provided to
> restrict the number of instances of the app. I'm wondering if there's a bug
> in te system now.
>
>
>
>
>
>
>
> On Friday, April 6, 2012 7:51:44 AM UTC-4, Ijaja wrote:
>
> > Hi,
>
> > is there a way to restrict the number of instances for my application
> > to really one?!!
>
> > Using the slider, to set maximum instances to 1 does not seem to work.
> > I understand, that this is obvious only intended to restrict the
> > amount i have to pay,
> > but google spends nevertheless more instances. I want really to
> > restrict it, because
> > my application uses caching, which may result in inconsistent data
> > views!
>
> > greets, ijaja

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Admin Settings: Number of Instances of my Application

[David Hardwick]

That's the number of idle instances...no way to set limit on instances
that will be created that I'm aware of, but you could set the Pending
Latency (wait time to keep a request in the queue before spinning up a
new instance) to a very high value...do so though only understanding
the consequences.

On Apr 6, 7:51 am, Ijaja  wrote:
> Hi,
>
> is there a way to restrict the number of instances for my application
> to really one?!!
>
> Using the slider, to set maximum instances to 1 does not seem to work.
> I understand, that this is obvious only intended to restrict the
> amount i have to pay,
> but google spends nevertheless more instances. I want really to
> restrict it, because
> my application uses caching, which may result in inconsistent data
> views!
>
> greets, ijaja

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: How to set Task Queue order?

[David Hardwick]

The Pipeline would be my recommendation.

http://code.google.com/p/appengine-pipeline/

On Apr 4, 4:10 am, Nicholas Verne  wrote:
> You could add task 2 from task 1, add task 3 from task 2 etc.
>
> To prevent task2 starting before task1 finishes, set
> max_concurrent_requests to 1 for the queue.
>
> Nick Verne
>
> On Wed, Apr 4, 2012 at 3:37 PM, Leandro Rezende
>
>
>
>
>
>
>
>  wrote:
> > im adding 10 taskes at the Queue, but i would like to a Task wait the
> > previous task  finish before it run..
>
> > Example:
>
> >  - Added 10 tasks.
> > - Task 1 Run
> > - Task 1 Finish
> > - Task 2 Begin
> > - Task 2 Finish
> > - Task 3 Begin
> > - ..
>
> > is it possible?
>
> > Leandro
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine+unsubscr...@googlegroups.com.
> > For more options, visit this group at
> >http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Disabled app but still getting billed for it.

[David Hardwick]

Okay, that makes sense that you could be using other services even
though the app is disabled. However, it would be nice to get a warning
when you disable your app to let you know you need to disable billing
too if you are not using other services.

The whole "don't be evil" principle applied here would at least garner
a modal window to let you know to disable billing if you are not using
any other services and are willing to nix your data, etc.

On Apr 3, 7:27 pm, Jeff Schnitzer  wrote:
> Hah!  So that's how they're trying to get around "just abandon the app
> instead of deleting all your data".
>
> I'm curious to know what happens when people simply change their CC#.
>
> Jeff
>
>
>
>
>
>
>
> On Tue, Apr 3, 2012 at 6:43 PM, Barry Hunter  wrote:
> > Yes. Google have said this is intentional.
>
> > An app - even disabled, could be using chargable services. Most
> > noteably storage.
>
> > So to stop being changed for an app - you need to explicitly disable
> > billing. A prerequisite of such, is that your app doesn't need more
> > than the free quota.
>
> > On Tue, Apr 3, 2012 at 11:48 AM, Gwyn Howell  
> > wrote:
> >> Same thing happened to us. It seems you have to disable billing before
> >> disabling the app or they'll continue to charge you.
>
> >> On Tuesday, 3 April 2012 03:26:29 UTC+1, David Hardwick wrote:
>
> >>> Hello,
>
> >>> I've notice that I have apps that have been disabled for a while but I'm
> >>> still getting the $2.10/week minimum charge.  I then went and disabled
> >>> billing as well for the app.
>
> >>> Do you always have to both disable the app and the billing?
>
> >>> Regards,
> >>>   - Hardwick
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Google App Engine" group.
> >> To view this discussion on the web visit
> >>https://groups.google.com/d/msg/google-appengine/-/zJy1lu3kq-AJ.
>
> >> To post to this group, send email to google-appengine@googlegroups.com.
> >> To unsubscribe from this group, send email to
> >> google-appengine+unsubscr...@googlegroups.com.
> >> For more options, visit this group at
> >>http://groups.google.com/group/google-appengine?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups 
> > "Google App Engine" group.
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to 
> > google-appengine+unsubscr...@googlegroups.com.
> > For more options, visit this group 
> > athttp://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Disabled app but still getting billed for it.

[David Hardwick]

Here's the issue if you want to star it.

http://code.google.com/p/googleappengine/issues/detail?id=7265

On Apr 3, 10:54 am, Jeff Schnitzer  wrote:
> Sounds like a bug - have you logged an issue here?
>
> http://code.google.com/p/googleappengine/issues/list
>
> Jeff
>
>
>
>
>
>
>
> On Tue, Apr 3, 2012 at 6:48 AM, Gwyn Howell  wrote:
> > Same thing happened to us. It seems you have to disable billing before
> > disabling the app or they'll continue to charge you.
>
> > On Tuesday, 3 April 2012 03:26:29 UTC+1, David Hardwick wrote:
>
> >> Hello,
>
> >> I've notice that I have apps that have been disabled for a while but I'm
> >> still getting the $2.10/week minimum charge.  I then went and disabled
> >> billing as well for the app.
>
> >> Do you always have to both disable the app and the billing?
>
> >> Regards,
> >>   - Hardwick
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To view this discussion on the web visit
> >https://groups.google.com/d/msg/google-appengine/-/zJy1lu3kq-AJ.
>
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine+unsubscr...@googlegroups.com.
> > For more options, visit this group at
> >http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Disabled app but still getting billed for it.

[David Hardwick]

Hello,

I've notice that I have apps that have been disabled for a while but I'm
still getting the $2.10/week minimum charge.  I then went and disabled
billing as well for the app.

Do you always have to both disable the app and the billing?

Regards,
  - Hardwick

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Datastore quota madness

[David Hardwick]

Is it really two indexes for both asc and desc for each simple
property that is indexed?  If so, then it would be great if we could
specify if we only wanted ASC or only DESC to sames some unwanted
writes.

With HRD and indexing, the writes add up real quick (especially using
the Pipeline library[1])

rock on,
   -hardwick

[1] - http://code.google.com/p/appengine-pipeline/


On Mar 23, 12:45 pm, Jeff Schnitzer  wrote:
> If you have not optimized your indexes, they will almost always
> account for the bulk of your data storage.
>
> Most of the datastore APIs (python db, ndb, java low-level, jdo/jpa,
> etc) default to creating two indexes (asc and desc) on every property.
>  Each of these index entries represents another row in a BigTable
> tablet; for small entities, each of these rows are only slightly
> smaller than the master blob.  So 5 indexed properties could easily
> cause a 5-10X expansion in your total stored data volume.
>
> You can't vacuum these indexes; unlike multi-property indexes defined
> in datastore-indexes.xml/yaml, single-property indexes are established
> when you save an entity and are only deleted by resaving the entity.
>
> If you are concerned about space (or datastore write operations), you
> should go through your data model and flag any properties you aren't
> querying on as unindexed.  Then you will need to resave your entities
> to reclaim index space.
>
> Jeff
>
>
>
>
>
>
>
> On Fri, Mar 23, 2012 at 11:01 AM, João Campos  wrote:
> > Thanks for your reply.
>
> > Even if logs do account for data storage, I've got a little less than 115MB
> > of those. With the 15MB of application data that would make up for a 130MB
> > total.
>
> > I've also ran a vacuum_indexes and I've got not custom indexes and a very
> > simple data model (only 3 entities).
>
> > With such a simple data model and such small amount of data, can the indexes
> > really be responsible for the missing 894MB (1024-130)?
>
> > On Friday, March 23, 2012 11:40:53 AM UTC, de Witte wrote:
>
> >> Indexes and logs also consume data storage.
>
> >> Op donderdag 22 maart 2012 11:46:11 UTC+1 schreef Константин Сабуров het
> >> volgende:
>
> >>> Hello!
>
> >>> i have an application on master/slave datastore and recently noticed
> >>> really strange things: datastore size on the dashboard is much bigger than
> >>> expected. in my app i proactively deleting all unused info to stay below 
> >>> 1Gb
> >>> quota (and there is not many users) but datastore size do not going down, 
> >>> it
> >>> going up instead and for now it is over 4Gb.
> >>> and this is not only mine point of view: datastore admin page and
> >>> datastore statistics both shows numbers which much easier to believe
> >>> i attached screenshots, may be i overlooked something... i just whant to
> >>> delete "invisible entities" which take all of my billable space
> >>> please help! :)
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To view this discussion on the web visit
> >https://groups.google.com/d/msg/google-appengine/-/s00-DD85r4cJ.
>
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine+unsubscr...@googlegroups.com.
> > For more options, visit this group at
> >http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Understanding channels and costs

[David Hardwick]

And how are you?

I see that we get charged for $0.01/100 opening on a channel.

*Opened Channels*
> $0.01/100 Opens


So if I have a 'Notification Channel' which I used to send updates on the
status of a completed backend process, then every time I need to open a
channel on the client-side it would cause that channel open ticker to
increment...is that correct?

In use-case terms, if a person logins into the application, and the first
thing my code does is connect to the Notification channel for that newly
logged in user (so that the user can now listen for any notifications),
then I would be incrementing my 'Open Channel' ticker each time someone
starts a session on my application...is that correct?

Rock on,
  -Hardwick

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Over quota, set new quota...but new settings not taking.

[David Hardwick]

Hello,

Our application is over quota, I set a new quota (twice) waited the 30 
minutes after each time, but the quota remains.  Google Wallet confirms the 
transactions went through so I am not sure why the new quotas are not being 
picked up.

If anyone can offer help so I can get this application (sherpatools) back 
up today, then I would greatly appreciate it.

Best regards,
  David Hardwick

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/gf-HA_BtSTMJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Using App Engine and Blob Store API to replace a file server...

[David Hardwick]

Thank you, Nick.  You talked me off the ledge after Tim's email ;-)

Rock on,
  -David Hardwick

On Jan 25, 1:53 am, Tim Hoffman  wrote:
> Hi Nick
>
> Sorry for that error, I was just thinking about images ;-(
>
> Rgds
>
> Tim

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Using App Engine and Blob Store API to replace a file server...

[David Hardwick]

Hello,

We are rebuilding an application that has 1000s of files on a file
server.  The application we are rebuilding has a sophisticated
architecture with an SSO appliance involved and Secure Data Connector,
so the files can remain on a file server and continue to be protected
by NTFS.  However, the question remains, 'Why not use the blob store
API for all these files?'

These files are static and are typically knowledge documents,
articles, user guides, etc. -- so lots of reads and very few writes/
edits.

Has anyone used App Engine and the Blob Store API for handling 1000s
of network files?  If so, any lessons learned or advice based on the
experience?

Thanks!
  David

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Mass email on GAE

[David Hardwick]

Hello,

We have an application that needs to send a lot of emails (10,000+)
but I am concerned about using GAE mail service because other
applications using GAE may get the mail servers blacklisted and
therefore start sending all the mail the application sends to spam
boxes instead of inboxes.  Is this concern valid?

This is the most recent conversation I found on mass email:

http://groups.google.com/group/google-appengine/browse_thread/thread/a9251f4a15c1db83/c00361c2ac43b227?lnk=gst&q=mass+email#c00361c2ac43b227

Constantin, does your app suffer from getting marked as SPAM?

Have others used GAE or external email providers instead to send mass
emails with custom content per email (e.g., include first and last
name in each email sent)?  Can you let me know which provider you used
if external or if you have issues with getting marked as SPAM if you
used GAE?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.