date:20110907

[google-appengine] Re: The 15 Minute Question?

2011-09-07 Thread zdravko

Greg,

> Mostly because it can take a long time to start an instance, so it is better
> to keep one running than start a new one for traffic.

Instead of starting up instances from scratch, could an image not be
saved and later used for almost instantaneous start-ups (similar to
desktop hibernation) ?

Thanks again.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Built-in indexes are costing me $200/day

2011-09-07 Thread Waleed Abdulla

Agree. My biggest cost, even more than instances, are datastore Ops, and the
biggest part of those are index writes (70M index writes vs 15M puts). In
light of the new changes, it's more reasonable to have indexing off by
default, and have it enabled only as explicitly requested.

Also, controlling indexes should be done in one place only, index.yaml. It's
often confusing what the effect of the indexed=False parameter is,
especially if you apply it later. Do you have old entities indexed and new
ones not? And what if you set it to True again? You'll probably have a gap
of unindexed entities. How do you fix that? Run a map-reduce job to
read/write the whole table. It's not as clean and manageable as having
everything in one index.yaml file.

Waleed

P.S. same posted on issue page as well.



On Tue, Sep 6, 2011 at 11:53 PM, renderpaz  wrote:

> Thanks for pointing that out, If you have any sort of frequent input
> to your application, this is a whale of an issue, just didn't notice
> until the billing change.
>
> Others, please vote for this issue, it can't be that much of a
> challenge to fix.
>
> On Sep 6, 11:34 pm, Robert Kluin  wrote:
> > I'd love this too. Star 4231:
> >  http://code.google.com/p/googleappengine/issues/detail?id=4231
> >
> > Robert
> >
> >
> >
> > On Wed, Sep 7, 2011 at 01:31, renderpaz  wrote:
> > > Hello,
> >
> > > Let me preface this by saying that I don't believe the planned pricing
> > > changes will have a dramatic impact on my costs - in-fact it has
> > > motivated me to trim the fat and I'm already seeing large savings.
> >
> > > That said, my biggest cost is datastore writes.  I completely
> > > understand that updating a composite index should "cost" a data store
> > > write, but the problem is 90% of my data store writes are coming from
> > > the "built-in" indexes on each property.  (I specifically have 6
> > > properties).
> >
> > > I would happily turn of the built in indexes on many of these
> > > properties, but using indexed=False also prevents me from including
> > > that property in a composite index as well!
> >
> > > In retrospect I could easily have hashed the 6 properties into 1
> > > indexable property - but that seems ridiculous as all I want to do is
> > > say, "Don't create built in indexes for these, but allow me to include
> > > them in composite indexes"
> >
> > > MAIN POINT:
> > > Is there a reason it is not possible to turn off built in indexes?  It
> > > seems unfair that I cannot disable these indexes but am charged full
> > > price for them (and its >>> my actual writes)
> >
> > > Thanks,
> > > Kevin
> >
> > > Datastore quota info provided below:
> > > Datastore API Calls  0%  127,540,291 of Unlimited
>  Okay
> > > Datastore Queries0%  517,774 of UnlimitedOkay
> > > Blobstore API Calls  0%  0 of Unlimited  Okay
> > > Total Stored Data10% 157.95 of 1,601.00 GBytes
>   Okay
> > > Blobstore Stored Data0%  0.00 of 1,601.00 GBytes
>   Okay
> > > Data Sent to Datastore API   0%  17.19 of Unlimited
> GBytes   Okay
> > > Data Received from Datastore API 0%  12.01 of
> Unlimited GBytes   Okay
> > > Datastore CPU Time   0%  1,187.75 of Unlimited CPU
> hours Okay
> > > Datastore Entity Fetch Ops   0%  75,875,047 of
> Unlimited Okay
> > > -
> > > Datastore Entity Put Ops 0%  35,815,086 of
> Unlimited Okay
> > > -
> > > Datastore Entity Delete Ops  0%  31,931 of Unlimited
>   Okay
> > > -
> > > Datastore Index Write Ops0%   288,168,806
> ** of Unlimited
> > > Okay
> > > ---
> > > Datastore Query Ops  0%  517,617 of UnlimitedOkay
> > > Datastore Key Fetch Ops  0%  14,858,227 of Unlimited
>   Okay
> > > Datastore Id Allocation Ops  0%  0 of Unlimited  Okay
> > > Number of Indexes2%  3 of 200Okay
> > > Prospective Search Subscriptions 0%  0 of 10,000
> >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups "Google App Engine" group.
> > > To post to this group, send email to google-appengine@googlegroups.com
> .
> > > To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> > > For more options, visit this group athttp://
> groups.google.com/group/google-appengine?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options,

[google-appengine] Re: Import XML data to a High Replication Datastore

2011-09-07 Thread Gerald Tan

Archive all the XMLs into one large ZIP file, then use java.util.zip for 
random access to the XML files within?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/hxl2MUc5k2YJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Best practice to save entity avoiding duplicate

2011-09-07 Thread Waleed Abdulla

Works, unless you want to allow users to edit their entries. If a user
changes the address after submitting then the new key is now different and
you'll end up saving a second entity. I wouldn't use any user editable data
in the key. Instead, I'd add a separate column to include your hash.



On Tue, Sep 6, 2011 at 11:32 PM, Robert Kluin wrote:

> Hi Thomas,
>  If you want to avoid duplicates, the hash solution is great.
> Generate the hash, then In a transaction try to fetch the entity by
> id.  If you can't get it, insert it.
>
>
> Robert
>
>
>
>
>
>
> On Tue, Sep 6, 2011 at 21:16, Thomas Wiradikusuma
>  wrote:
> > Hi guys,
> >
> > I'm working on something like AirBnB but much simpler. I want to hint
> > user whenever he/she tries to submit duplicate entry.
> >
> > Currently, my model looks like:
> >
> > House {
> > Id
> > Username
> > Address
> > State
> > Country
> > }
> >
> > The id is made of address+state+country MD5. Is this the "correct" way
> > to do it? Or, should I go with auto-generated Id and add extra field
> > named 'hash', or how?
> >
> >
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> > For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
> >
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] How do I set maximum total instances to 1 ? OR How do I run my app for free for 24 hrs a day ?

2011-09-07 Thread Prashant

Hi,

I have an app with very low traffic and I won't be able to go for payed
account unless and untill my app starts generating some revenue. With new
pricing scheme my app will face a serious problem. There is no way to set
Total Maximum Instance to 1 so there is no way to run the app for free for
24hrs with only "24 (free) Instance Hours". Does that mean that I will have
to shut down my app ? Does Google have any solution for low traffic app like
mine ?

Thanks,
Prashant

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Sample for Push Queues

2011-09-07 Thread Hariprasath Mohankumar

Hi Robert,
   Thanks for your reply. But i found a strange 
behaiviour, when i add tasks in the queue with the call below,
 taskqueue.add(url='/autobot', 
params={'key':game.userXid}, queue_name='autobot') 
the execution of the task is not happening ASYNCHRONOUSLY. the code waits 
for the task to be completed by executing the logic which i defined in 
'/autobot' url and is not moving ahead with the instructions that follow the 
taskqueue.add( )  function. Is there anything im doing wrong here.


Thanks,
Hariprasath Mohankumar

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/5qw1CmOMklIJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Sample for Push Queues

2011-09-07 Thread Hariprasath Mohankumar

Hi Robert,
   Thanks for your reply. But i found a strange 
behaiviour, when i add tasks in the queue with the call below,
 taskqueue.add(url='/autobot', 
params={'key':game.userXid}, queue_name='autobot') 
the execution of the task is not happening ASYNCHRONOUSLY. the code waits 
for the task to be completed by executing the logic which i defined in 
'/autobot' url and then it  moves ahead with the instructions that follow 
the taskqueue.add( )  function. Is there anything im doing wrong here?. 


Thanks,
Hariprasath Mohankumar

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/SnPkO6wub4oJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] If I enable billing & set budget to $0 will I be charged $9/month?

2011-09-07 Thread Jens Stoltenberg

I want to keep the $50 credit I received by enabling billing.

But I've heard you lose it if you disable billing.

What about if I leave billing enabled and set my spending level to $0?

Does that mean I will have to pay $9 a month?

In other words, the only way to avoid the $9 a month is to disable billing?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: the min instace time is 24 hours?

2011-09-07 Thread Tim


Jon,

Thanks for that explanation, but if it's authoritative (not meaning to sound 
rude, but I hope .. I take it you're on the AppEngine team rather than a 
user explaining your understanding of how pricing applies) then it perhaps 
illustrates what many of us may be worried about, and maybe a more detailed 
explanation of the scheduler and 15-minute rule may put some minds at rest 
and reduce some of the hostility to the new billing scheme.

It's the sort of thing that can be easily explained in person, but can be 
hard to get across on paper, so let me explain how I may have previously got 
the wrong impression and been more alarmed than you may consider seems 
appropriate.

Everything I've read about the "15 minute minimum instance" would imply in 
the above scenario described that the billing would be 

Time Period: [0:5] - 4 active instances
Time Period [5:20] - 4 active instances running for their 15 minute minimum 
idle time since last query
Time Period [20:inf] - 0 active instances (ie some of the above may still be 
idling awaiting being killed, but wouldn't be billed)

This would give a charge of 80 minutes, as opposed to 35 as explained by 
yourself - I might have set min-idle-instances to 1, but I understood that 
once an instance was started it would be charged for instance time until 15 
minutes after it's last activity.

Now I'm willing to take it that you're right, but the pricing as explained 
so far has led me to believe that the rule is

[explanation #1]
 - ANY instance, once started, will not be eligible to be killed until at 
least 15 minutes after last activity, and thus will lead to at least 15 
minutes billing each
 - Instances will be started up according to traffic demands
 - Instances will be killed off when the number of "instances that have each 
gone BEYOND their 15 minutes since their last activity" exceeds 
"max-idle-instances"

And the thing that worried me about the above understanding was that if, in 
the period [5:20], there were a few queries (say 1 query a minute) then the 
scheduler might hand these round-robin style to the 4 idling instances, 
thereby keeping 4 instances active for much longer, rather than giving them 
repeatedly to the same single instance and thereby killing instances 2,3 and 
4 at the 20 minute mark.

This is what was alarming me - the impression that, given the odd short 
burst of traffic (say a busy minute once every couple of hours) that started 
up a handful of extra instances, then the scheduler might keep all these 
instances hanging around for ages, each running mostly idle but sharing the 
"normal load", and thus leading to a massive bill following each burst no 
matter how short.

Now you seem to be saying that this reading is incorrect (to which I and 
many others may understandably say "phew!") and the rules are more like

[explanation #2]
 - Instances, once started, are kept in a list ordered by start time
 - Any activity causes the list to be checked in order (ie first started 
checked first) for a free active instance, and assigned to the first one 
that is idle
 - If none are idle, the query will be queued up to "max-pending" time to 
see if any instance becomes free before a new instance is started
 - Each time an instance in the list finishes serving a query, the scheduler 
checks how many instances are actually busy, and if the number exceeds 
max-idle-instances, then it chooses instances from the END of the list (most 
recently started) to kill (or logically killed as in stopped being billed 
and removed from the list)
 - 15 minutes after any instance last serviced a query, it is killed (or 
logically...) regardless of max-idle-instances unless that would breach 
min-idle-instances

Now I'd understand if the above has approximations (eg extra pauses in the 
above), or has got some details wrong, or you don;t want to commit to 
PRECISELY how it works so you can change it in future and in fact what 
happens is you don't do all these checks in realtime (other than choosing 
what instance to assign etc) but then apply the logic afterwards for billing 
purposes (ie you tend to err on keeping things active in case they're 
needed, but bill as if they'd been killed off according to the rule above).

But if scheduling (of queries to instances), and billing (of instance time) 
is closer to explanation #2 than #1, then I could probably live with the "15 
minute" rule, but I'd hope you could also see that if the scheduler works 
more like #1 than #2 then that would be why so many of us are worried about 
"15 minutes" and would like to see that interval dropped to more like "2 
minutes" !

An authoritative explanation of the intent (there's that word again Greg) of 
the scheduler / billing logic would be more than welcome and may reduce the 
gap between why you guys think the pricing is reasonable and many of us in 
here think it's not.

Cheers

--
Tim

-- 
You received this message because you are subscribed to the Google Groups 
"Google

[google-appengine] Re: How do I set maximum total instances to 1 ? OR How do I run my app for free for 24 hrs a day ?

2011-09-07 Thread Gerald Tan

I believe even if you have additional instances spawned for you, you are 
only charged for 1 instance (which should be your max idle instance setting) 
+ active instances, which should be a very small number. Now you will see 
either one of two situations:

1) You have enough time when there are 0 instances up to make up for the 
active instance numbers, in which case your app should have no problem 
serving requests all day
2) You have an instance up all the time, in which case your app will be 
automatically shutdown when you run out of quota near the end of the billing 
day, probably for around 30mins to an hour depending on your active instance 
time, until the daily quota resets.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/W4vItIXH-4kJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Billing history stopped 5 day ago

2011-09-07 Thread Alex Popescu

In the light of the new billing details, I've made some changes to the Max 
Idle Instances and Min Pending Latency (as per the optimization 
documentation) just to see how these would reflect in my bill. Anyways my 
billing history has stopped 5 days ago on Sept. 2nd.

I'm really confused on how to proceed right now. I know that I have to get 
ready for the moment the new pricing becomes effective, but if I don't have 
the tools to estimate my costs and understand what optimizations I can do, 
then I don't seem to be left with much to do.

Could anyone clarify why billing history has stopped? Also, how soon changes 
to the above configurations would be reflected in the billing history?

tia,

A://

PS: Application ID: twimblr. I do have other applications each with its own 
different billing status (most recent being from Sept.5th, which is OKish).

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/jm3te1hWJgQJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Why app engine new price model is totally wrong

2011-09-07 Thread Andrew Cassidy

I wasn't trying to say 'You can replicate GAE on AWS trivially", GAE is a
massive beast that does one thing in particular that I haven't experimented
with yet: automatically creating and destroying running instances. I was
trying to say you could host your own scaling application on any
infrastructure you like with a little know how.

You can host in multiple data centers using Anycast addresses, you can
round-robin your load balancers using clusterip or DNS, you can handle load
balancer failures using basic ip failover mechanisms.

As long as you use some form shared storage backend for session data, such
as memcache, you don't have to worry about sticky sessions on the load
balanacers.

I have a working prototype on a basic testing network using (nothing amazon
here)

- nginx as the load balancer and to serve static content
- uwsgi container for the applications
- mysql cluster for redundant data storage
- mysql proxy to load balance across the database servers without any
code modification
- memcached for session storage and output caching
- corosync/pacemaker for ip failover
- rabbitmq/celery for task queueing

As I say, it's only a working prototype, and in most cases you could host a
smallish application on a smallish vps anywhere. Most VPS providers now
provide fault-tolerance and failover as standard. I'm moving my application
to one such host.

Using AWS for fault-tolerance, I just read this blog post:
http://blog.rightscale.com/2008/03/26/setting-up-a-fault-tolerant-site-using-amazons-availability-zones/

If anyone wants to fund an experiment on setting up an application on AWS
I'd be more than happy to try it.

--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/YnQr6kcNwVMJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

1 2 >

1 - 100 of 139 matches

Mail list logo