[google-appengine] Re: Built-in indexes are costing me $200/day

2011-09-07 Thread renderpaz
Thanks for pointing that out, If you have any sort of frequent input
to your application, this is a whale of an issue, just didn't notice
until the billing change.

Others, please vote for this issue, it can't be that much of a
challenge to fix.

On Sep 6, 11:34 pm, Robert Kluin robert.kl...@gmail.com wrote:
 I'd love this too. Star 4231:
  http://code.google.com/p/googleappengine/issues/detail?id=4231

 Robert



 On Wed, Sep 7, 2011 at 01:31, renderpaz kev...@gmail.com wrote:
  Hello,

  Let me preface this by saying that I don't believe the planned pricing
  changes will have a dramatic impact on my costs - in-fact it has
  motivated me to trim the fat and I'm already seeing large savings.

  That said, my biggest cost is datastore writes.  I completely
  understand that updating a composite index should cost a data store
  write, but the problem is 90% of my data store writes are coming from
  the built-in indexes on each property.  (I specifically have 6
  properties).

  I would happily turn of the built in indexes on many of these
  properties, but using indexed=False also prevents me from including
  that property in a composite index as well!

  In retrospect I could easily have hashed the 6 properties into 1
  indexable property - but that seems ridiculous as all I want to do is
  say, Don't create built in indexes for these, but allow me to include
  them in composite indexes

  MAIN POINT:
  Is there a reason it is not possible to turn off built in indexes?  It
  seems unfair that I cannot disable these indexes but am charged full
  price for them (and its  my actual writes)

  Thanks,
  Kevin

  Datastore quota info provided below:
  Datastore API Calls              0%      127,540,291 of Unlimited        
  Okay
  Datastore Queries                0%      517,774 of Unlimited    Okay
  Blobstore API Calls              0%      0 of Unlimited  Okay
  Total Stored Data                10%     157.95 of 1,601.00 GBytes       
  Okay
  Blobstore Stored Data            0%      0.00 of 1,601.00 GBytes         
  Okay
  Data Sent to Datastore API               0%      17.19 of Unlimited GBytes  
       Okay
  Data Received from Datastore API                 0%      12.01 of Unlimited 
  GBytes       Okay
  Datastore CPU Time               0%      1,187.75 of Unlimited CPU hours    
       Okay
  Datastore Entity Fetch Ops               0%      75,875,047 of Unlimited    
       Okay
  -
  Datastore Entity Put Ops                 0%      35,815,086 of Unlimited    
       Okay
  -
  Datastore Entity Delete Ops              0%      31,931 of Unlimited     
  Okay
  -
  Datastore Index Write Ops                0%       288,168,806 ** of 
  Unlimited
  Okay
  ---
  Datastore Query Ops              0%      517,617 of Unlimited    Okay
  Datastore Key Fetch Ops          0%      14,858,227 of Unlimited         
  Okay
  Datastore Id Allocation Ops              0%      0 of Unlimited  Okay
  Number of Indexes                2%      3 of 200        Okay
  Prospective Search Subscriptions                 0%      0 of 10,000

  --
  You received this message because you are subscribed to the Google Groups 
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to 
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group 
  athttp://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



Re: [google-appengine] Re: Built-in indexes are costing me $200/day

2011-09-07 Thread Waleed Abdulla
Agree. My biggest cost, even more than instances, are datastore Ops, and the
biggest part of those are index writes (70M index writes vs 15M puts). In
light of the new changes, it's more reasonable to have indexing off by
default, and have it enabled only as explicitly requested.

Also, controlling indexes should be done in one place only, index.yaml. It's
often confusing what the effect of the indexed=False parameter is,
especially if you apply it later. Do you have old entities indexed and new
ones not? And what if you set it to True again? You'll probably have a gap
of unindexed entities. How do you fix that? Run a map-reduce job to
read/write the whole table. It's not as clean and manageable as having
everything in one index.yaml file.

Waleed

P.S. same posted on issue page as well.



On Tue, Sep 6, 2011 at 11:53 PM, renderpaz kev...@gmail.com wrote:

 Thanks for pointing that out, If you have any sort of frequent input
 to your application, this is a whale of an issue, just didn't notice
 until the billing change.

 Others, please vote for this issue, it can't be that much of a
 challenge to fix.

 On Sep 6, 11:34 pm, Robert Kluin robert.kl...@gmail.com wrote:
  I'd love this too. Star 4231:
   http://code.google.com/p/googleappengine/issues/detail?id=4231
 
  Robert
 
 
 
  On Wed, Sep 7, 2011 at 01:31, renderpaz kev...@gmail.com wrote:
   Hello,
 
   Let me preface this by saying that I don't believe the planned pricing
   changes will have a dramatic impact on my costs - in-fact it has
   motivated me to trim the fat and I'm already seeing large savings.
 
   That said, my biggest cost is datastore writes.  I completely
   understand that updating a composite index should cost a data store
   write, but the problem is 90% of my data store writes are coming from
   the built-in indexes on each property.  (I specifically have 6
   properties).
 
   I would happily turn of the built in indexes on many of these
   properties, but using indexed=False also prevents me from including
   that property in a composite index as well!
 
   In retrospect I could easily have hashed the 6 properties into 1
   indexable property - but that seems ridiculous as all I want to do is
   say, Don't create built in indexes for these, but allow me to include
   them in composite indexes
 
   MAIN POINT:
   Is there a reason it is not possible to turn off built in indexes?  It
   seems unfair that I cannot disable these indexes but am charged full
   price for them (and its  my actual writes)
 
   Thanks,
   Kevin
 
   Datastore quota info provided below:
   Datastore API Calls  0%  127,540,291 of Unlimited
  Okay
   Datastore Queries0%  517,774 of UnlimitedOkay
   Blobstore API Calls  0%  0 of Unlimited  Okay
   Total Stored Data10% 157.95 of 1,601.00 GBytes
   Okay
   Blobstore Stored Data0%  0.00 of 1,601.00 GBytes
   Okay
   Data Sent to Datastore API   0%  17.19 of Unlimited
 GBytes   Okay
   Data Received from Datastore API 0%  12.01 of
 Unlimited GBytes   Okay
   Datastore CPU Time   0%  1,187.75 of Unlimited CPU
 hours Okay
   Datastore Entity Fetch Ops   0%  75,875,047 of
 Unlimited Okay
   -
   Datastore Entity Put Ops 0%  35,815,086 of
 Unlimited Okay
   -
   Datastore Entity Delete Ops  0%  31,931 of Unlimited
   Okay
   -
   Datastore Index Write Ops0%   288,168,806
 ** of Unlimited
   Okay
   ---
   Datastore Query Ops  0%  517,617 of UnlimitedOkay
   Datastore Key Fetch Ops  0%  14,858,227 of Unlimited
   Okay
   Datastore Id Allocation Ops  0%  0 of Unlimited  Okay
   Number of Indexes2%  3 of 200Okay
   Prospective Search Subscriptions 0%  0 of 10,000
 
   --
   You received this message because you are subscribed to the Google
 Groups Google App Engine group.
   To post to this group, send email to google-appengine@googlegroups.com
 .
   To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group athttp://
 groups.google.com/group/google-appengine?hl=en.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App