Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Fred Sauer]

On Fri, Sep 2, 2011 at 12:18 AM, NM  wrote:

> Here is a usecase that may support M/S over HRD.
>
> I have a form where we capture appointment information such as date, time
> and some other details. Upon a successful save the resulting page is a
> calendar view that needs to read all appointments including this newly
> created appointment.
>
> Given this and eventual consistency mode of HRD the calendar at times does
> not show the newly saved appointment. When I refresh the screen after a few
> seconds it does; obviously by then the data would have replicated. This is
> obviously not an issue with M/S.
>
> Thoughts?
>

The calendar times you are querying are all for one user, you could put the
calendar entities in a single entity group and then use an ancestor query,
which will return consistent results.

If you need to query across entity groups, but still want the calendar entry
you just saved to be included in the results, then the recommended approach
is:
- Perform your cross entity group query in the normal way
- In your application code, insert the just inserted entity into the result
set if it's not there
- Show the user the merged result set

You can store away or track the "just inserted calender entity" in a cookie,
in a url / form parameter, in memcache, or in an HTTP session object (Java).

Fred



>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/lC6R_999_vQJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>



-- 
Fred Sauer
Developer Advocate
Google Inc.
1600 Amphitheatre Parkway
Mountain View, CA 94043
fre...@google.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[NM]

Here is a usecase that may support M/S over HRD.
 
I have a form where we capture appointment information such as date, time 
and some other details. Upon a successful save the resulting page is a 
calendar view that needs to read all appointments including this newly 
created appointment.
 
Given this and eventual consistency mode of HRD the calendar at times does 
not show the newly saved appointment. When I refresh the screen after a few 
seconds it does; obviously by then the data would have replicated. This is 
obviously not an issue with M/S.
 
Thoughts?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/lC6R_999_vQJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Ikai Lan (Google)]

Hooray! Good to hear =).

High replication makes me happy.

--
Ikai Lan
Developer Programs Engineer, Google App Engine
plus.ikailan.com | twitter.com/ikai



On Thu, Aug 25, 2011 at 2:36 PM, thstart  wrote:

> haha COBOL ;) my COBOL is 1 year old.
>
> *The code was fine. Case closed*, moved to HRD, the app on HRD is much
> faster that
> what used to be 1 month ago. No change in code, just moved to HRD. I
> suppose Google worked
> very hard to improve the speed.
>
> *Bottom line:*
> My users are reporting to me it works faster than M/S and faster than HRD 1
> month ago.
> And yes my code is up to date and following best practices. It turned out
> this was a temporary
> speed issue with Google servers after they announced HRD. I published some
> important code I use
> in my app and Google confirmed it is OK, I use keys, lists the right way.
> So it was not the code.
>
> Let's move on.
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/WVosQnUL4VYJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[thstart]

haha COBOL ;) my COBOL is 1 year old. 

*The code was fine. Case closed*, moved to HRD, the app on HRD is much 
faster that 
what used to be 1 month ago. No change in code, just moved to HRD. I suppose 
Google worked 
very hard to improve the speed. 

*Bottom line:*
My users are reporting to me it works faster than M/S and faster than HRD 1 
month ago. 
And yes my code is up to date and following best practices. It turned out 
this was a temporary 
speed issue with Google servers after they announced HRD. I published some 
important code I use
in my app and Google confirmed it is OK, I use keys, lists the right way. So 
it was not the code.

Let's move on.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/WVosQnUL4VYJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[love smebs]

Every API or service provider in the world will do their best to keep things 
up to date, but unless you're doing legacy COBOL maintenance, there's always 
going to bits where people will say, "Oh, by the way, don't use that way 
anymore.
--
Ikai Lan 
Developer Programs Engineer,
*G * | *Small Business* - All Things 
B2B

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/1m_8nL522ucJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[thstart]

+Fred Sauer 

>EVENTUAL_CONSISTENCY 

Thank you I will explore this possibility. Yesterday I moved back the app to 
HRD and my tests
show significant speed improvement. No change in my code. I suppose Google 
improved the 
speed somehow. If this is consistent then my users will be happy.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/d6rhj8mmiyIJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Fred Sauer]

On Sat, Aug 20, 2011 at 1:38 PM, thstart  wrote:

> +Jeff Schnitzer
>
> > pry the details out of you.
>
> I posted a code if you care to read, the advice is EVENTUAL_CONSISTENCY
> which is a compromise and I don't need to do it in M/S.
>


When fetching a list of keys use max_entity_groups_per_rpc=1 for speed.


Use EVENTUAL_CONSISTENCY whenever you can as it has to the potential to
speed up your app. I say "whenever you can" because sometimes you won't be
able to.




>
> >get better reception ...
>
> Yes I got a reception, Brett will update his presentations, Dan will update
> his book - I just purchased an access to his 2nd edition - a work in
> progress, and yes - I make every effort to do what is needed.
>
> The point is if you have a project which you really care you will do the
> same to make sure it works. If you have the time to thinker 24/7 with
> database experimenting, then good luck.
>
> Just checking - have you looked at the code I posted, made tests both in HR
> and M/S, performed benchmarks or you just want to talk not having  what else
> to do?
> Do this and then come back ton talk about real world situations.
>
> >get a terapist
> If you are at the point to have a waiting list of paying customers, and you
> invested a year in developing, testing and almost to the final, and at this
> point you have to completely redesign you code without knowing in advance,
> if you an in areal world situation probably you will get a terapist, I don't
> need one.
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/Q3CS2U8McpAJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>



-- 
Fred Sauer
Developer Advocate
Google Inc.
1600 Amphitheatre Parkway
Mountain View, CA 94043
fre...@google.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Fred Sauer]

On Sat, Aug 20, 2011 at 7:52 AM, Sekhar  wrote:

> We're about to launch a healthcare app, and we recently switched from M/S
> to HRD (both with reserved instances). I'm really, really happy with HRD so
> far, most importantly we don't see random hiccups in data access any more.
> Performance has been really consistent.


Great to hear. It's also exactly what I'm hearing from others I've talked to
who have switched from M/S to HRD.


We are occasionally getting a new error "too much contention on these
> datastore entities. please try again." however. Clearly it's not due to
> traffic because it's like 5 of us testing the site at this point. I'm a bit
> worried our patients/doctors will see this after launch next month. What's
> this and is there a way to avoid it?
>

In both cases, you should design your application for at most 1 datastore
write/second (=1qps) per entity group. You can write simultaneously to as
many entity groups as you like.

The problem you mention usually comes up in one of two scenarios:

   1. You have a some shared / global data structure which is updated by
   multiple users at the same time, ultimately at a rate > 1qps. See
   http://code.google.com/appengine/articles/sharding_counters.html for a
   common technique for dealing with this.
   2. You are testing with multiple users, but they are all logged in with
   the same account, or are otherwise artificially updating the same data. In
   this case you have a usage pattern which doesn't match what production
   traffic would look like.

If you're wanting to build high throughput pipelines, check out
http://www.google.com/events/io/2010/sessions/high-throughput-data-pipelines-appengine.html



>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/QygaPE8F_dwJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>



-- 
Fred Sauer
Developer Advocate
Google Inc.
1600 Amphitheatre Parkway
Mountain View, CA 94043
fre...@google.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[thstart]

+Ikai Lan

>As far as speed goes, let me give you an example of speed in action.

I very much appreciate your explanation. I understand we cannot know much of 
the 
internal working and don't need to. That is good enough for me to decide now 
to move back
to HRD. So did I - I just reactivated the HRD version of same app with same 
data and mentioned 
it is much faster now. I did not change anything, probably Google made some 
improvements.
So let we do some internal testing again to see how it will behave now.

Still I will go deeper and optimize from HRD in mind. Any written tutorials, 
guides, etc. will be 
of much help.

One more recent development makes the debate about M/S vs. HRD obsolete for 
Python developers
especially - the today's announcement for trusted beta for Python 2.7 under 
AppEngine which will work
only on HRD. It is clear any speed disadvantage HRD may have (small or not), 
we can overcome with
2.7 capabilities - concurrency and threading so I am fully in - just send an 
application for trusted testers.

Thank you!


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/aUl5xHkCqWYJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Ikai Lan (Google)]

Hi,

You're building the keys correctly for strong consistency.

As far as speed goes, let me give you an example of speed in action.

It takes me ~30 minutes to bike somewhere. I can also take the bus. The bus
takes me 20 minutes when there are no disruptions. Unfortunately, there are
disruptions several times a month. When there is a disruption, it takes me
an hour to get somewhere. Is the bus faster, or is biking faster? Would you
rather have a consistent 30 minute commute, or a 20 minute commute, that,
several times a month, fluctuates to a one hour commute? Now pretend it
randomly fluctuates to a 4 hour commute. You'd pick the bike commute every
single time. That's what it's like to be on master/slave. I'd love to say
it's not like that, but history will prove me wrong. While the work we've
done has gone a long way in improving reliability overall, it doesn't change
the fact that a single point of failure is subject to massive swings in
performance.

I remember seeing a presentation by Facebook engineering about how they
engineer for predictability instead of pure speed. It's the same logic. Slow
and steady is better than fast and unpredictable, especially in user facing
applications.

Another example is Azul System's design philosophy in pauseless GC. It's
actually quite interesting. Their view is that the way GC works in Hotspot
JVM (generational) optimizes the 99% use case, but in the 1% use case, GC
pauses which pause the entire JVM (sometimes for a second or two, which is a
deadly stall in real time applications), will exist, so their philosophy is
to build a JVM that focuses on more predictability instead of prioritizing
speed in the 99% use case. Disclosure: I still use Hotspot since JVM GC
pauses don't affect the applications I write, but if they did, I would
seriously look at their solution (
http://www.artima.com/lejava/articles/azul_pauseless_gc.html).

A single unpredictable component is easy to use. When you have multiple
unpredictable components, you see an exponential increase in complexity in
handling wild performance swings, and you want to minimize the number of
components whose reliability you cannot predict (URLFetch - you can't
predict how someone else's system will perform). The datastore is, very
often, at the absolute core of applications you will be building. It's very
important for the performance of the datastore to be as predictable as
possible.

In addition to predictability, you also get much stronger protection against
data loss to the point that it is a near statistical impossibility. There's
no reason why you wouldn't want to use high replication. The next time
someone says something is faster, ask why, because that's where you'll get
the real answers.

--
Ikai Lan
Developer Programs Engineer, Google App Engine
plus.ikailan.com | twitter.com/ikai



On Mon, Aug 22, 2011 at 8:53 AM, thstart  wrote:

> +Ikai Lan
>
> In my example:
>
> *
> root = MyRoot.get_profile()
> **
> skey = email
> *
> *
> key = db.Key.from_path('MyList',
>skey,
>parent=root.key())
> profile = db.get(key)
>
> For HR are there a need to change the way these keys are formed?
>
> Does your data migration tool can transfer my database from M/S to HR
> automatically?
> Do my users need to re-enter their data again?
>
>
> *
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/CBRAyB0paHwJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[thstart]

+Ikai Lan

In my example:

*
root = MyRoot.get_profile()
**
skey = email
*
*
key = db.Key.from_path('MyList',
   skey,
   parent=root.key())
profile = db.get(key)

For HR are there a need to change the way these keys are formed? 

Does your data migration tool can transfer my database from M/S to HR 
automatically?
Do my users need to re-enter their data again?


*

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/CBRAyB0paHwJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[thstart]

+Ikai Lan

>Dan's advice is correct: use get-by-key as much as possible, and any update 
to the book will likely include a section about high replication.

*I just bought an access to his book in progress. Asked him a question on G+ 
about speed in general, get the following answer:
"Compared to M/S, the HRD does indeed trade some speed for reliability. But 
that reliability is crucial.". Now if that is true what do you recommend to 
get same speed? Probably there is a way for better parallelization?*
*
*
*What is most important is how to achieve the maximum read speed, the writes 
I do in background in deferred tasks and the user can wait till the process 
finishes - this is not an issue.*

>The practice you're citing from Brett is a few years old and predates high 
replication; would you be willing to revisit it if I roped him into updating 
it >for high replication?

*How quick can we get that?*

*Because the information about migrating M/S to HR is dispersed in different 
places, *
*where is the best place to start? *
*
*
*Best,*
*--Constantine*
*
*

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/BXrr5vbtM-8J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[thstart]

Hi Mike,

Thank you, I am collecting the bits needed for eventual switch.

In fact I already switched my app to HR, it worked for 1 month but 
I never got the speed of M/S. So now I have to figure out why.

Best,
--Constantine

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/IUqfj2T8lU0J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Mike Lawrence]

In general, my app does 3 db gets for every db query.

No modifications were necessary with my app. Using slim3 db for
transactions.

1000 simutaneous users were logged in using my own authentication (not
federated) with sessions enabled.
I only use session to record the fact that a user has authenticated
I save app state in hidden form html.

hope that helps

Sincerely,  Mike Lawrence


On Mon, Aug 22, 2011 at 10:54 AM, thstart  wrote:

> +Mike Lawrence
>
> As you post I have couple of questions to you.
>
> >Average response time is still sub-second for the week.
>
> Do you use queries to fetch data or you use a lists?
>
> Do you modified your app when moving to HR, or just transferred it?
>
> >1000 simultaneous users.
>
> Are you users logged in or just retrieving data anonymously?
>
>
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/rpGj8t4Gy-0J.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[thstart]

+Mike Lawrence

As you post I have couple of questions to you.

>Average response time is still sub-second for the week.

Do you use queries to fetch data or you use a lists?

Do you modified your app when moving to HR, or just transferred it?

>1000 simultaneous users.

Are you users logged in or just retrieving data anonymously?




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/rpGj8t4Gy-0J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Mike Lawrence]

For the one week off of Master/Slave and on HR:


   - up-time has been 100%
   - max response time has improved... down from 22+ seconds to about 2-6 
   seconds.
   - Average response time is still sub-second for the week.








-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/FZ0hSiyK6aAJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Robert Kluin]

This could be for a wide range of reasons. You should start a new thread and
give us some more details. For example, are the entities with contention
warnings global to all users or user specific?  Is the contention caused by
user requests or tasks?  What are you doing, updating a counter?  Be sure to
give us enough information to help you.

Robert



On Aug 20, 2011 10:52 AM, "Sekhar"  wrote:
>
> We're about to launch a healthcare app, and we recently switched from M/S
to HRD (both with reserved instances). I'm really, really happy with HRD so
far, most importantly we don't see random hiccups in data access any more.
Performance has been really consistent.
>
> We are occasionally getting a new error "too much contention on these
datastore entities. please try again." however. Clearly it's not due to
traffic because it's like 5 of us testing the site at this point. I'm a bit
worried our patients/doctors will see this after launch next month. What's
this and is there a way to avoid it?
>
> --
> You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
> To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/QygaPE8F_dwJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[thstart]

> P.S. it is 3 hours and I still can not upload my app on the cloud (M/S).

>I get that you're upset about something being broken.

Yes there are many times when try to upload my code on the cloud but the 
uploads are rolled back. This lasted yesterday for 3 hours. 

Is it not clear what I want to say? What is your recommendation?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/6Fw0-Zu0uSsJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[thstart]

+Jeff Schnitzer

Just got an answer from Dan on G+: "Compared to M/S, the HRD does indeed 
trade some speed for reliability." 

That was my point. 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/9IjKpLqo90QJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[thstart]

+Jeff Schnitzer
> pry the details out of you.

I posted a code if you care to read, the advice is EVENTUAL_CONSISTENCY 
which is a compromise and I don't need to do it in M/S.

>get better reception ...

Yes I got a reception, Brett will update his presentations, Dan will update 
his book - I just purchased an access to his 2nd edition - a work in 
progress, and yes - I make every effort to do what is needed. 

The point is if you have a project which you really care you will do the 
same to make sure it works. If you have the time to thinker 24/7 with 
database experimenting, then good luck.

Just checking - have you looked at the code I posted, made tests both in HR 
and M/S, performed benchmarks or you just want to talk not having  what else 
to do?
Do this and then come back ton talk about real world situations.

>get a terapist
If you are at the point to have a waiting list of paying customers, and you 
invested a year in developing, testing and almost to the final, and at this 
point you have to completely redesign you code without knowing in advance, 
if you an in areal world situation probably you will get a terapist, I don't 
need one.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/Q3CS2U8McpAJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Jeff Schnitzer]

On Fri, Aug 19, 2011 at 11:30 PM, thstart  wrote:
> +Jeff Schnitzer
>>EVENTUAL_CONSISTENCY
> so with this we are 1 step back to M/S using HR? What is the point?

Huh?  This seems a complete non-sequitur.

> P.S. it is 3 hours and I still can not upload my app on the cloud (M/S).

I get that you're upset about something being broken.  As a
non-googler, let me offer you some objective advice:  You sound
awfully whiny.  You'd probably get better reception by taking a
different tone.

For example, you might post:  "My application, which is fast on M/S,
is slow on HRD.  Why is that?"  This is something we can all explore
and maybe learn something new - it runs contrary to everyone else's
experience.  Instead you post "HRD is slow!" and expect us to pry the
details out of you.  It doesn't sound like you're actually interested
in improving your product, you just want to rant.  Seriously, get a
therapist.

Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Sekhar]

We're about to launch a healthcare app, and we recently switched from M/S to 
HRD (both with reserved instances). I'm really, really happy with HRD so 
far, most importantly we don't see random hiccups in data access any more. 
Performance has been really consistent.

We are occasionally getting a new error "too much contention on these 
datastore entities. please try again." however. Clearly it's not due to 
traffic because it's like 5 of us testing the site at this point. I'm a bit 
worried our patients/doctors will see this after launch next month. What's 
this and is there a way to avoid it?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/QygaPE8F_dwJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[thstart]

+Jeff Schnitzer
>EVENTUAL_CONSISTENCY

so with this we are 1 step back to M/S using HR? What is the point?

P.S. it is 3 hours and I still can not upload my app on the cloud (M/S). 
Weekends is the time
for most productive work, and I consistently cannot do that. 

For the record now is 11:30 PM PST, 8/19/2011. Checked system status - it 
says it is OK.
It is not OK if I cannot upload for 3 hours.


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/QWksXz4zVqcJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Jeff Schnitzer]

On Fri, Aug 19, 2011 at 8:16 PM, thstart  wrote:
> My only concern if I will get the same speed. I don't want to spend all this
> effort
> costing me money and time and at the end to find it will be slower than M/S.

Did you try the solution Stephen mentioned?

   my_members = db.get(my_keys,
config=db.create_config(read_policy=db.EVENTUAL_CONSISTENCY))

I would be curious to know if this eliminated the speed issue.  From
all reports (and my own highly unscientific observation), HRD seems to
perform consistently better than M/S.  But I use EVENTUAL_CONSISTENCY
as a default - probably others do too.  I haven't even thought to try
the alternative.

Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[thstart]

+Fred Sauer

>To put it another way: I have yet to find a use case, when the whole app is 
considered, where M/S is preferable to HRD.

My only concern if I will get the same speed. I don't want to spend all this 
effort
costing me money and time and at the end to find it will be slower than M/S.

By the way while I was on HR (about 2 months) I mentioned there was a time I 
cannot upload
my app, had to try 10 times, sometimes to wait 1 hour. This happened very 
often.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/lO4CgTrqmZIJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[thstart]

+Fred Sauer

>Test where you like, but test the scenario which you plan to use.

the question is if these changes suggested will work on M/S because I will 
not change everything
in one time, but gradually on M/S and if it works to move it to HR.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/2oNR9zcNUh0J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Should Google deprecate Master/Slave data store option?

[Brandon Wirtz]

I have a set of apps that Run very well on MS and unless MS is all the way
down (which has happened) they ran just fine during maintenance, and
No-Write periods.  They are cheaper and faster on MS.  

 

HR is definitely the best for many situation, but there may be times when MS
is better especially if you are fighting costs.

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Fred Sauer
Sent: Friday, August 19, 2011 5:31 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Should Google deprecate Master/Slave data
store option?

 

You should mirate to HRD if you care about consistent, predictable speed,
performance and reliability. Your app will be happier. You will be happier.
Your customers will be happier.

The advice to migrate to HRD is unqualified: you should migrate.

Fred on my Android. 

On Aug 19, 2011 3:51 PM, "thstart"  wrote:
> A bit from business side reality - today my partners where on a meeting 
> with a BIG customer representatives. Functionality and speed especially 
> (M/S currently) impressed them so much that they wanted to add 500 users 
> immediately
> to our app and paid for it to be sure they get an access. 
> 
> Our app is with Billing On so I don't see this High Replication vs.
Master/ 
> Slave 
> thing as an issue for Google - we are paying customers. 
> 
> So now the question is open before me and my partners especially:
> 1) Things are moving very fast from business side and I will mark the app 
> from Alpha
> to Beta soon after looking how it works with more simultaneous users.
> 2) The users are already feeling the speed and any speed decrease will be 
> mentioned
> immediately. That is because the app is actually a mobile HTML5 app and
used 
> in 
> all kind of devices: Android, iPhone, Blackberry. For such kind of app the

> speed is of
> 1st priority and any decision about functionality is rejected if the speed

> is slowed down.
> 3) Now the question is what benefit we will get (if any) moving from M/S
to 
> HR WITHOUT
> disrupting the current functionality and speed? Does the amount of time, 
> effort and cost 
> which need to be invested in migration to HR is justified? As a start-up 
> this is critical for us.
> Every day we have a NEW list of priorities which are business related - 
> users need more and more -
> we get this from the feedback. Nobody cares about underlying technology if

> it works good enough.
> 
> Best,
> --Constantine
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
> To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/iDQzYaVOSioJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
<mailto:google-appengine%2bunsubscr...@googlegroups.com> .
> For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
> 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Fred Sauer]

You should mirate to HRD if you care about consistent, predictable speed,
performance and reliability. Your app will be happier. You will be happier.
Your customers will be happier.

The advice to migrate to HRD is unqualified: you should migrate.

Fred on my Android.
On Aug 19, 2011 3:51 PM, "thstart"  wrote:
> A bit from business side reality - today my partners where on a meeting
> with a BIG customer representatives. Functionality and speed especially
> (M/S currently) impressed them so much that they wanted to add 500 users
> immediately
> to our app and paid for it to be sure they get an access.
>
> Our app is with Billing On so I don't see this High Replication vs.
Master/
> Slave
> thing as an issue for Google - we are paying customers.
>
> So now the question is open before me and my partners especially:
> 1) Things are moving very fast from business side and I will mark the app
> from Alpha
> to Beta soon after looking how it works with more simultaneous users.
> 2) The users are already feeling the speed and any speed decrease will be
> mentioned
> immediately. That is because the app is actually a mobile HTML5 app and
used
> in
> all kind of devices: Android, iPhone, Blackberry. For such kind of app the

> speed is of
> 1st priority and any decision about functionality is rejected if the speed

> is slowed down.
> 3) Now the question is what benefit we will get (if any) moving from M/S
to
> HR WITHOUT
> disrupting the current functionality and speed? Does the amount of time,
> effort and cost
> which need to be invested in migration to HR is justified? As a start-up
> this is critical for us.
> Every day we have a NEW list of priorities which are business related -
> users need more and more -
> we get this from the feedback. Nobody cares about underlying technology if

> it works good enough.
>
> Best,
> --Constantine
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
> To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/iDQzYaVOSioJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Fred Sauer]

Feel free to set the default to 1 for your apps.

Fred on my Android.
On Aug 19, 2011 11:52 AM, "Stephen"  wrote:
> On Fri, Aug 19, 2011 at 6:49 PM, Fred Sauer  wrote:
>>>
>>> If you don't want to give you eventual consistency, try this:
>>>
>>> my_members = db.get(my_keys,
>>>  config=db.create_config(max_entity_groups_per_rpc=1))
>>
>> Also, for reference, here's what this does:
>>
>>   @ConfigOption
>>   def max_entity_groups_per_rpc(value):
>> """The maximum number of entity groups that can be represented in one
>> rpc.
>> For a non-transactional operation that involves more entity groups
than
>> the
>> maximum, the operation will be performed by executing multiple,
>> asynchronous
>> rpcs to the datastore, each of which has no more entity groups
>> represented
>> than the maximum.  So, if a put() operation has 8 entity groups and
the
>> maximum is 3, we will send 3 rpcs, 2 with 3 entity groups and 1 with
2
>> entity groups.  This is a performance optimization - in many cases
>> multiple, small, concurrent rpcs will finish faster than a single
large
>> rpc.  The optimal value for this property will be
application-specific,
>> so
>> experimentation is encouraged.
>> """
>
> It feels like you'd always want to set this to 1. What situations
> might make >1 perform better?
>
> --
> You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Fred Sauer]

Fred on my Android.
On Aug 19, 2011 11:18 AM, "thstart"  wrote:
>
> +Ikai Lan
> +Fred Sauer
> +Stephen
>
> Thank you, I need to rethink again with this information: it is useful. I
am fine
> with new technologies, in my case the product is very sophisticated and
the balance
> between new Google technology and what my business requires was tilted to
business
> side - e.g. less new technology and techniques, more pro business
solution.
>
> Definitely an extension to the book, an update to the excellent Brett
Slatkin (he explains very well)
> materials will be of much help and am awaiting them and am wondering how i
will know if they are ready?
>
> There is some practical questions: these changes obviously have to be
tested,

You can also deploy another app version for testing.

I am fine to set up a second
> app just for that, but still it would be not the real thing in real
situations. Right now I have a group of trusted people
> who are already using it and looking at the logs I can see real life
situations which never pop up in test accounts.
> So my question is - I obviously will want to implement the changes one by
one and see the effect, being right now
> on M/S could this be done in M/S environment and then move my users to HR?

The impact of these changes are material on HRD, and may have limited
performance impact on M/S.

Test where you like, but test the scenario which you plan to use.

Above all else, migrate to HRD as soon as practical.

>
> Also I suppose when Google goes full HR and obviously wants we to move out
of M/S, do Google has benchmarks
> for this kind of usage as I described in order to have a good motivation
to move to HR?

While you should care that your app has acceptable performance, and you
should use the suggested tips to get optimal performance out of HRD, your
primary concern or reasons for migrating off of M/S are
1 Avoiding scheduled read-only periods
2 Avoiding unscheduled outages and maintenance periods
3 Avoiding datastore latency spikes

Every developer who I've communicated with has been happy with their app on
HRD and none want to go back. In particular the 3rd point I mentioned often
weighs the heaviest for developers.

To put it another way: I have yet to find a use case, when the whole app is
considered, where M/S is preferable to HRD.

Hope that helps.
Fred

>
> Thanks to all!
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
> To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/cmDqAoiZDiwJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[thstart]

A bit from business side reality - today my partners where on a meeting 
with a BIG customer representatives. Functionality and speed especially 
(M/S currently) impressed them so much that they wanted to add 500 users 
immediately
to our app and paid for it to be sure they get an access. 

Our app is with Billing On so I don't see this High Replication vs. Master/ 
Slave 
thing as an issue for Google -  we are paying customers. 

So now the question is open before me and my partners especially:
1) Things are moving very fast from business side and I will mark the app 
from Alpha
to Beta soon after looking how it works with more simultaneous users.
2) The users are already feeling the speed and any speed decrease will be 
mentioned
immediately. That is because the app is actually a mobile HTML5 app and used 
in 
all kind of devices: Android, iPhone, Blackberry. For such kind of app the 
speed is of
1st priority and any decision about functionality is rejected if the speed 
is slowed down.
3) Now the question is what benefit we will get (if any) moving from M/S to 
HR WITHOUT
disrupting the current functionality and speed? Does the amount of time, 
effort and cost 
which need to be invested in migration to HR is justified? As a start-up 
this is critical for us.
Every day we have a NEW list of priorities which are business related - 
users need more and more -
we get this from the feedback. Nobody cares about underlying technology if 
it works good enough.

Best,
--Constantine



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/iDQzYaVOSioJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Stephen]

On Fri, Aug 19, 2011 at 6:49 PM, Fred Sauer  wrote:
>>
>> If you don't want to give you eventual consistency, try this:
>>
>> my_members = db.get(my_keys,
>>              config=db.create_config(max_entity_groups_per_rpc=1))
>
> Also, for reference, here's what this does:
>
>   @ConfigOption
>   def max_entity_groups_per_rpc(value):
>     """The maximum number of entity groups that can be represented in one
> rpc.
>     For a non-transactional operation that involves more entity groups than
> the
>     maximum, the operation will be performed by executing multiple,
> asynchronous
>     rpcs to the datastore, each of which has no more entity groups
> represented
>     than the maximum.  So, if a put() operation has 8 entity groups and the
>     maximum is 3, we will send 3 rpcs, 2 with 3 entity groups and 1 with 2
>     entity groups.  This is a performance optimization - in many cases
>     multiple, small, concurrent rpcs will finish faster than a single large
>     rpc.  The optimal value for this property will be application-specific,
> so
>     experimentation is encouraged.
>     """

It feels like you'd always want to set this to 1. What situations
might make >1 perform better?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[thstart]

*+Ikai Lan 
*
*+Fred Sauer*
*+Stephen*
*
*
Thank you, I need to rethink again with this information: it is useful. I am 
fine
with new technologies, in my case the product is very sophisticated and the 
balance
between new Google technology and what my business requires was tilted to 
business
side - e.g. less new technology and techniques, more pro business solution. 

Definitely an extension to the book, an update to the excellent Brett 
Slatkin (he explains very well) 
materials will be of much help and am awaiting them and am wondering how i 
will know if they are ready?

There is some practical questions: these changes obviously have to be 
tested, I am fine to set up a second
app just for that, but still it would be not the real thing in real 
situations. Right now I have a group of trusted people 
who are already using it and looking at the logs I can see real life 
situations which never pop up in test accounts.
So my question is - I obviously will want to implement the changes one by 
one and see the effect, being right now
on M/S could this be done in M/S environment and then move my users to HR?

Also I suppose when Google goes full HR and obviously wants we to move out 
of M/S, do Google has benchmarks 
for this kind of usage as I described in order to have a good motivation to 
move to HR?

Thanks to all!



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/cmDqAoiZDiwJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Fred Sauer]

On Fri, Aug 19, 2011 at 10:46 AM, Fred Sauer  wrote:

>
>
> On Fri, Aug 19, 2011 at 1:38 AM, Stephen  wrote:
>
>>
>>
>> On Thu, Aug 18, 2011 at 10:58 PM, thstart  wrote:
>>
>>>
>>> I think Master/Slave is a lot faster in my case up to 10 times faster. I
>>> tested it with only read, no write operations,
>>> no transactions, nothing special for example I have a heavy use of this
>>> expression:
>>>
>>> e = MyList.get_profile(email)
>>> my_keys = e.my_keys
>>> my_members = db.get(my_keys)
>>>
>>> for member in my_members:
>>> member., etc. <-- do something with the
>>> property
>>>
>>
>> Try this:
>>
>> e = MyList.get_profile(email)
>>  my_keys = e.my_keys
>> my_members = db.get(my_keys, config=db.create_config(
>> read_policy=db.EVENTUAL_CONSISTENCY))
>>
>
> If you don't want to give you eventual consistency, try this:
>
> my_members = db.get(my_keys,
>  config=db.create_config(max_entity_groups_per_rpc=1))
>

Also, for reference, here's what this does:
  @ConfigOption
  def max_entity_groups_per_rpc(value):
"""The maximum number of entity groups that can be represented in one
rpc.

For a non-transactional operation that involves more entity groups than
the
maximum, the operation will be performed by executing multiple,
asynchronous
rpcs to the datastore, each of which has no more entity groups
represented
than the maximum.  So, if a put() operation has 8 entity groups and the
maximum is 3, we will send 3 rpcs, 2 with 3 entity groups and 1 with 2
entity groups.  This is a performance optimization - in many cases
multiple, small, concurrent rpcs will finish faster than a single large
rpc.  The optimal value for this property will be application-specific,
so
experimentation is encouraged.
"""


>
>
> Similarly, in Java you can set max entity groups per RPC:
> - Set  the appengine.datastore.defaultMaxEntityGroupsPerRpc system property
> to 1
> - Use
> http://code.google.com/appengine/docs/java/javadoc/com/google/appengine/api/datastore/DatastoreServiceConfig.html#maxEntityGroupsPerRpc(int)
>
>
>
>
>>
>> for member in my_members:
>> member., etc. <-- do something with the
>> property
>>
>>
>> It's a relatively new option:
>>
>>
>> http://code.google.com/intl/en/appengine/docs/python/datastore/functions.html#create_config
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
>
>
> --
> Fred Sauer
> Developer Advocate
> Google Inc.
> 1600 Amphitheatre Parkway
> Mountain View, CA 94043
> fre...@google.com
>
>
>


-- 
Fred Sauer
Developer Advocate
Google Inc.
1600 Amphitheatre Parkway
Mountain View, CA 94043
fre...@google.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Fred Sauer]

On Fri, Aug 19, 2011 at 1:38 AM, Stephen  wrote:

>
>
> On Thu, Aug 18, 2011 at 10:58 PM, thstart  wrote:
>
>>
>> I think Master/Slave is a lot faster in my case up to 10 times faster. I
>> tested it with only read, no write operations,
>> no transactions, nothing special for example I have a heavy use of this
>> expression:
>>
>> e = MyList.get_profile(email)
>> my_keys = e.my_keys
>> my_members = db.get(my_keys)
>>
>> for member in my_members:
>> member., etc. <-- do something with the
>> property
>>
>
> Try this:
>
> e = MyList.get_profile(email)
>  my_keys = e.my_keys
> my_members = db.get(my_keys, config=db.create_config(
> read_policy=db.EVENTUAL_CONSISTENCY))
>

If you don't want to give you eventual consistency, try this:

my_members = db.get(my_keys,
 config=db.create_config(max_entity_groups_per_rpc=1))


Similarly, in Java you can set max entity groups per RPC:
- Set  the appengine.datastore.defaultMaxEntityGroupsPerRpc system property
to 1
- Use
http://code.google.com/appengine/docs/java/javadoc/com/google/appengine/api/datastore/DatastoreServiceConfig.html#maxEntityGroupsPerRpc(int)




>
> for member in my_members:
> member., etc. <-- do something with the
> property
>
>
> It's a relatively new option:
>
>
> http://code.google.com/intl/en/appengine/docs/python/datastore/functions.html#create_config
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>



-- 
Fred Sauer
Developer Advocate
Google Inc.
1600 Amphitheatre Parkway
Mountain View, CA 94043
fre...@google.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Nadun Kulatunge]

 wow cool fun stuff << http://j.gs/I9f >> i think you will like it

On Fri, Aug 19, 2011 at 2:38 PM, Stephen  wrote:

>
>
> On Thu, Aug 18, 2011 at 10:58 PM, thstart  wrote:
>
>>
>> I think Master/Slave is a lot faster in my case up to 10 times faster. I
>> tested it with only read, no write operations,
>> no transactions, nothing special for example I have a heavy use of this
>> expression:
>>
>> e = MyList.get_profile(email)
>> my_keys = e.my_keys
>> my_members = db.get(my_keys)
>>
>> for member in my_members:
>> member., etc. <-- do something with the
>> property
>>
>
> Try this:
>
> e = MyList.get_profile(email)
>  my_keys = e.my_keys
> my_members = db.get(my_keys, config=db.create_config(
> read_policy=db.EVENTUAL_CONSISTENCY))
>
> for member in my_members:
> member., etc. <-- do something with the
> property
>
>
> It's a relatively new option:
>
>
> http://code.google.com/intl/en/appengine/docs/python/datastore/functions.html#create_config
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>



-- 
Bye TC

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Stephen]

On Thu, Aug 18, 2011 at 10:58 PM, thstart  wrote:

>
> I think Master/Slave is a lot faster in my case up to 10 times faster. I
> tested it with only read, no write operations,
> no transactions, nothing special for example I have a heavy use of this
> expression:
>
> e = MyList.get_profile(email)
> my_keys = e.my_keys
> my_members = db.get(my_keys)
>
> for member in my_members:
> member., etc. <-- do something with the
> property
>

Try this:

e = MyList.get_profile(email)
 my_keys = e.my_keys
my_members = db.get(my_keys, config=db.create_config(
read_policy=db.EVENTUAL_CONSISTENCY))

for member in my_members:
member., etc. <-- do something with the
property


It's a relatively new option:


http://code.google.com/intl/en/appengine/docs/python/datastore/functions.html#create_config

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Ikai Lan (Google)]

Eventually consistent datastore gets make get by key operations faster.
Queries should perform the same, but better in failure scenarios in high
replication.

As far as things going out of date goes: that's just the technology world
for you. Every API or service provider in the world will do their best to
keep things up to date, but unless you're doing legacy COBOL maintenance,
there's always going to bits where people will say, "Oh, by the way, don't
use that way anymore. Use this new way instead." The high replication
datastore didn't exist when Dan wrote that book. Also, you should know how
the book writing cycle works: a writer works with a publisher, then writes
the book over the next year or two. During that time, the publisher markets
the book, and the writer furiously tries to keep it up to date, knowing that
chances are good there are bits that will be obsolete and his initial plan
has to change. I don't know off the top of my head if Dan is planning on
releasing a new version of his book or not. The first edition was printed in
November 2009. In general, Dan's advice is correct: use get-by-key as much
as possible, and any update to the book will likely include a section about
high replication.

--
Ikai Lan
Developer Programs Engineer, Google App Engine
plus.ikailan.com | twitter.com/ikai



On Thu, Aug 18, 2011 at 3:52 PM, thstart  wrote:

> >If you enable eventually consistent queries, a batch get by key in high
> replication should perform as fast as in master/slave.
>
> *is this related only to queries like:*
>
> q = db.GqlQuery("SELECT * FROM Person " +
>
> "WHERE last_name = :1 AND height < :2 " +
>
> "ORDER BY height DESC",
> "Smith", 72)
>
>
> http://code.google.com/appengine/docs/python/datastore/queries.html
>
> The Master/Slave Datastore and the High Replication 
> Datastore have
> different guarantees when it comes to query consistency. By default:
>
>- The Master/Slave datastore is strongly consistent for all queries.
>- The High Replication datastore is strongly consistent by default* for
>queries* within an entity 
> group.
>  With
>the High Replication Datastore, non-ancestor queries are always eventually
>consistent.
>
>
>
> *If that is the case I managed to get rid of queries - I don't use any any
> queries, just working with lists and keys -
> *
> *it is parallel, fast, don't need indexes, etc. Bret Slatkin had very
> useful presentations and found them to be very simple*
> *and to work in practice.*
> *
> *
> *By the way our customers don't know what technique I am using only thing
> they care is to be fast and they are telling me*
> *up to now they haven't used a web app with so much functionality to be so
> fast. *
> *
> *
> *I would say they to be fast is that I designed the database as separate
> entity for each user - e.g. it scales with numbers of users*
> *automatically. Very important is that I don;t use any queries, indexes,
> etc. just lists of keys and for ... in construct for search,*
> *also the keys and entities are composed in as many ways possible in
> advance so when the user is searching for data it gets*
> *it instantly because search is embedded in the keys. In Master/S**lave
> all this worked in parallel and was extremely fast.*
> *
> *
> *
> *
> >The practice you're citing from Brett is a few years old and predates high
> replication; would you be willing to revisit it if I roped him into updating
> it for high >replication?*
> *
>
> Yes this is true, but you have to admit this is the only useful information
> on designing efficient data-store. There is *only one* serious book
> explaining
> how *AppEngine Datastore* is working and in this book you can read the
> same recommendation like Brett Slatkin recommends. And the book
> is a year old:
>
> Programming Google App Engine
> by Dan Sanderson
> Copyright © *2010* Dan Sanderson.
>
> Dan Sanderson is working at Google and explains everything extremely well
> and I believe after so much time and efforts invested to understand how it
> works and to successfully implement a very useful app with waiting customers
> - *do I have to think a book and technique explained in 2010 is already
> out-dated?*
>
>
> >would you be willing to revisit it if I roped him into updating it for
> high replication?
>
> *I always want to know new techniques, so the quick answer is yes,
> definitely. *
> *
> *
> *You should also keep in mind that it is not a good business idea to
> constantly revisit *
> *and rewrite a perfectly working and tested code.*
> *
> *
> *Best,*
> *--Constantine*
>
>
>
>
>
>
> *
> *
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-ap

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[thstart]

>If you enable eventually consistent queries, a batch get by key in high 
replication should perform as fast as in master/slave.

*is this related only to queries like:*

q = db.GqlQuery("SELECT * FROM Person " +
"WHERE last_name = :1 AND height < :2 " +
"ORDER BY height DESC",
"Smith", 72)


http://code.google.com/appengine/docs/python/datastore/queries.html

The Master/Slave Datastore and the High Replication 
Datastore have 
different guarantees when it comes to query consistency. By default:

   - The Master/Slave datastore is strongly consistent for all queries.
   - The High Replication datastore is strongly consistent by default* for 
   queries* within an entity 
group.
 With 
   the High Replication Datastore, non-ancestor queries are always eventually 
   consistent.



*If that is the case I managed to get rid of queries - I don't use any any 
queries, just working with lists and keys - 
*
*it is parallel, fast, don't need indexes, etc. Bret Slatkin had very useful 
presentations and found them to be very simple*
*and to work in practice.*
*
*
*By the way our customers don't know what technique I am using only thing 
they care is to be fast and they are telling me*
*up to now they haven't used a web app with so much functionality to be so 
fast. *
*
*
*I would say they to be fast is that I designed the database as separate 
entity for each user - e.g. it scales with numbers of users*
*automatically. Very important is that I don;t use any queries, indexes, 
etc. just lists of keys and for ... in construct for search,*
*also the keys and entities are composed in as many ways possible in advance 
so when the user is searching for data it gets*
*it instantly because search is embedded in the keys. In Master/S**lave all 
this worked in parallel and was extremely fast.*
*
*
*
*
>The practice you're citing from Brett is a few years old and predates high 
replication; would you be willing to revisit it if I roped him into updating 
it for high >replication?*
*

Yes this is true, but you have to admit this is the only useful information 
on designing efficient data-store. There is *only one* serious book 
explaining
how *AppEngine Datastore* is working and in this book you can read the same 
recommendation like Brett Slatkin recommends. And the book 
is a year old:

Programming Google App Engine
by Dan Sanderson
Copyright © *2010* Dan Sanderson.

Dan Sanderson is working at Google and explains everything extremely well 
and I believe after so much time and efforts invested to understand how it 
works and to successfully implement a very useful app with waiting customers 
- *do I have to think a book and technique explained in 2010 is already 
out-dated?*


>would you be willing to revisit it if I roped him into updating it for high 
replication?

*I always want to know new techniques, so the quick answer is yes, 
definitely. *
*
*
*You should also keep in mind that it is not a good business idea to 
constantly revisit *
*and rewrite a perfectly working and tested code.*
*
*
*Best,*
*--Constantine*






*
*

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/BzQWzYVCk3UJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Ikai Lan (Google)]

If you enable eventually consistent queries, a batch get by key in high
replication should perform as fast as in master/slave.

The practice you're citing from Brett is a few years old and predates high
replication; would you be willing to revisit it if I roped him into updating
it for high replication?

--
Ikai Lan
Developer Programs Engineer, Google App Engine
plus.ikailan.com | twitter.com/ikai



On Thu, Aug 18, 2011 at 2:58 PM, thstart  wrote:

> +Joshua Smith
> >From my reading, M/S seems to have better throughput than HR, at the
> expense of more frequent downtime,
> >and spurious datastore timeouts.  I'd like to have that option in a
> multi-threaded app.
>
> This is more important for public app (site) with many anonymous users.
>
> My app is 100% user based - e.g. used after log-in, but heavy on data
> related to each particular user.
> The users are not accessing it very often (say they access it many times a
> day but this is not much)
> but when they access it they need to be as fast as possible.
>
> So  *frequent downtime *is a non-issue, the chance a user to access it at
> exactly this time are minimal,* *
> *spurious datastore timeouts *is a non-issue because this is rare - so I
> can live with Master / Slave
> imperfections - it is good enough and I don;t need to rewrite and optimize
> again.
>
> +Ikai Lan
> >Master/Slave doesn't have better throughput. In the 99th percentile,
> because of datastore latency spikes,
> >high replication far outperforms master/slave.
>
> I think Master/Slave is a lot faster in my case up to 10 times faster. I
> tested it with only read, no write operations,
> no transactions, nothing special for example I have a heavy use of this
> expression:
>
> *
> e = MyList.get_profile(email)
> *
> *
> my_keys = e.my_keys
> my_members = db.get(my_keys)
>
> for member in my_members:
> member., etc. <-- do something with the
> property
>
> *
> *and it is parallel in Master/Slave (according to Brett Slatkin
> presentation) . In High Replication I believe it is serial,*
> *probably I am wrong (I don't have a way to know) but this operation is
> very slow in High Replication.*
> *
> *
> *My code uses heavily this construct and it is extremely fast.*
> *
> *
> *Best,*
> *
> *
> *--Constantine*
> *
> *
> *
>
> *
>
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/DoYJ0jkPJ7cJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[thstart]

+Joshua Smith
>From my reading, M/S seems to have better throughput than HR, at the 
expense of more frequent downtime, 
>and spurious datastore timeouts.  I'd like to have that option in a 
multi-threaded app.

This is more important for public app (site) with many anonymous users. 

My app is 100% user based - e.g. used after log-in, but heavy on data 
related to each particular user. 
The users are not accessing it very often (say they access it many times a 
day but this is not much)
but when they access it they need to be as fast as possible.

So  *frequent downtime *is a non-issue, the chance a user to access it at 
exactly this time are minimal,* *
*spurious datastore timeouts *is a non-issue because this is rare - so I can 
live with Master / Slave 
imperfections - it is good enough and I don;t need to rewrite and optimize 
again.

+Ikai Lan
>Master/Slave doesn't have better throughput. In the 99th percentile, 
because of datastore latency spikes, 
>high replication far outperforms master/slave.

I think Master/Slave is a lot faster in my case up to 10 times faster. I 
tested it with only read, no write operations,
no transactions, nothing special for example I have a heavy use of this 
expression:

*
e = MyList.get_profile(email)
*
*
my_keys = e.my_keys  
my_members = db.get(my_keys)

for member in my_members:
member., etc. <-- do something with the 
property

*
*and it is parallel in Master/Slave (according to Brett Slatkin 
presentation) . In High Replication I believe it is serial,*
*probably I am wrong (I don't have a way to know) but this operation is very 
slow in High Replication.*
*
*
*My code uses heavily this construct and it is extremely fast.*
*
*
*Best,*
*
*
*--Constantine*
*
*
*

*



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/DoYJ0jkPJ7cJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[thstart]

*I am responding here to combined comments:*

*I followed Brett Slatikn recommendations how to perform most efficient 
operations on DataStore and it works extremely fast *
*in Master / Slave.*

+Robert Kluin
> For instance, fetching a large list of keys from multiple entity groups 
may be slower due to the transaction overhead.

*I have list of keys only from one entity.*

>So you might need to evaluate your use of entity groups.  It may also
>be that you've got a design that is making many datastore RPCs in
>serial, that will probably be more of an issue when on HR since there
>is a slightly higher average latency.

*I store keys in lists. When getting the list *
*
*
*for example*
*class MyList(BaseModel):*
*My_keys = db.ListProperty(db.Key) *
*
*
*Brett Slatkin recommends about no more than 3,000 max and so do I.*
*
*
*and this is parallel:*
*
e = MyList.get_profile(email)
*
*
my_keys = e.my_keys
my_key = db.get(my_keys)

#That's it! Super fast!
@classmethod
def get_profile(self, email):
if not email:
return None
root = MyRoot.get_profile()
#cache
self._cache = get_cache("memcached://")
skey = email
key_cache = 'MyList' + skey + str(root.key())
profile = self._cache.get(key_cache)
#cache
if not profile:
key = db.Key.from_path('MyList',
   skey,
   parent=root.key())
profile = db.get(key)
# not found - create
if not profile:
#def txn():
root = MyRoot.get_profile_del_cache()
profile = MyList(key_name=skey,
   parent=root.key())
profile.put()
return profile
#return db.run_in_transaction(txn)
#cache
self._cache.set(key_cache, profile, timeout=CACHE_TIMEOUT)
#cache
return profile

*

+Fred Sauer
>verifying whether you are performing batch gets across entity groups

*I have transactions in which I have an operations on different entity 
groups but this is by design. *

>I should also point out that I have yet to meet a developer whose not 
happier on HRD and migrating their app. 
>You may well be the first (although I'm not trying to imply that you are 
the only one)... :)

*The app is in development for more than a year and I followed best 
practices valid at this time before high replication*
*datastore was introduced. Redesign is not an option because I already have 
generated a lot of interest and waiting list*
*of customers ready to pay for the current functionality. The benefit I 
would get from going to High Replication does not *
*justifies the effort: I have already 20,000 lines of code - redesign is not 
practical.*
*
*
*Best,*
*
*
*--Constantine*

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/S6G9eOWRvVUJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Fred Sauer]

On Thu, Aug 18, 2011 at 11:48 AM, Joshua Smith wrote:

>
> On Aug 18, 2011, at 2:10 PM, Fred Sauer wrote:
>
> On Wednesday, August 17, 2011 5:24:48 AM UTC-7, Joshua Smith wrote:
>>
>> I certainly hope they don't deprecate M/S, unless/until the following
>> issues are addressed:
>>
>> 1) Ability to migrate without changing app ID.  This is fundamental, and
>> we've never seen any *reason* why this is impossible.  I suspect that this
>> is very hard, but not impossible.
>>
> We are able to create an alias for migrated apps, so that traffic sent to
> oldappid.appspot.com will be transparently routed to your new application.
> So while you won't be able to keep the same app id, you can keep using the
> same appspot.com domain, which is probably what you care about most.
>
>
> Perhaps that what some people care about most, but not me.  What I care
> about is what I spelled out in the next two bullet points :)
>
>
>>  Changing the app ID causes problems because:
>>
> - External references to entity keys (not IDs) are pervasive in many apps
>> (such as google search links to deep pages, user bookmarks, or RSS feed
>> URLs) because the documentation encourages the use of these keys in URLs
>>
> This remains an issues, but can be easily addressed in a couple of lines of
> application code in the appropriate request handlers. The string encoded
> keys that are referenced externally have encoded within them the old app id.
> If you take the decoded key and extract the parent, kind and id/name to
> create a new key (again, this is 1-2 lines of code), then your app can use
> that new key to locate the migrated entity within the new datastore.
>
> As far as reference (list) properties: those reference keys are
> automatically updated as part of the the datastore migration and should not
> cause you any issues.
>
> - Many users have keys wrapped up in properties that the migration tool
>> cannot recognize [this doesn't impact my app, but it's a common issue on
>> this list]
>>
> This does affect some apps. As I mentioned above, a couple of lines of code
> in the right place should take care of this.
>
>
> 1-2 lines of code, in dozens of handlers is the crux of the problem.
>  Dealing with this at the application layer is problematic because there are
> so many code paths that need to be examined.
>

>
> As an alternative, the infrastructure could be fixed to recognize the case
>> of using the old app ID instead of the new app ID in a key, and quietly make
>> the substitution.
>>
> That's a neat idea.
>
>
> Neat enough that you guys might implement it?
>
> I've suggested this before, but I'm glad that I've got your attention this
> time!
>

I'll certainly bring up with the team.



>
> This would be a pretty huge step toward making the migration practical for
> me.
>
>
>> 2) Migration of large data sets takes an extremely long time.  Given that
>> we regularly see M/S downtimes in the couple of hours range, a couple of
>> hours is probably a reasonable amount of time to migrate.  A couple of days
>> is much too much.  [Note that I have not personally gotten far enough into a
>> migration to see this duration; I'm basing this on many user complaints I've
>> read on this list.]
>>
> Applications with small amounts of data can easily self migrate in a short
> period of time using datastore_admin. Applications with more data will
> benefit from the new migration tool Ikai mentioned. We created the tool
> specifically to simplify and speed up the migration for larger apps, and to
> reduce the read-only period required to make the final switch.
>
>
> Yes, I saw that in his note, and if it works as advertised, I suspect it
> will resolve this issue.
>
>
>> 3) The problem of transactional consistency needs to be addressed.  It is
>> not acceptable for transactions to end in a "maybe this worked, maybe it
>> didn't" state.  They should either succeed or fail.  (Although the docs not
>> not say whether this problem is exclusive to HR, it seems to be, and
>> certainly it matters much more in HR since every put is implicitly
>> transactional.)
>>
> There are subtle differences between the consistency guarantees offered by
> M/S and HRD. In some cases, apps need to make a few tweaks in order to work
> correctly in the new environment. The benefit is much more
> predictable/consistency latency and increased reliability. A good article
> which reviews the consistency guarantees can be found here:
>   http://code.google.com/appengine/articles/transaction_isolation.html
>
>
> I'm not talking about consistency guarantees.  I get that things change
> under HR, and it's pretty easy to understand what the implications of that
> are.
>
> What I'm talking about is the problem where you have to write idempotent
> transactions because you can never tell whether a transaction that threw
> certain exceptions succeeded or failed.  That's really messed up, and you
> really need to fix that.
>
> But, to be fair, I'm not sure this problem will cause me

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Fred Sauer]

I recommend looking at Appstats, and verifying whether you are performing
batch gets across entity groups. I have not met an app yet that is unable to
get acceptable HRD performance.

I should note that apps built on M/S have (usually unintentionally) been
optimized for the M/S performance characteristics. It makes sense that some
apps may need to or want to make adjustments before migrating to HRD.

I should also point out that I have yet to meet a developer whose not
happier on HRD and migrating their app. You may well be the first (although
I'm not trying to imply that you are the only one)... :)

On Thu, Aug 18, 2011 at 12:11 PM, thstart  wrote:

> Let me share my experience.
>
> I was developing an AppEngine app for a year.
> It was very fast (extremely fast), with latency > 1 sec.
> When High Replication data-store was released I decided to give it
> a try and switched to it. Unfortunately the latency got >1 Sec
> sometimes > 2 Sec. The app worked a wile that way and because
> still in beta I could experiment. So I decided to go to Billed option
> to see if the reliability and more importantly the latency could improve.
> Unfortunately this do not affected latency. Then I decided to go back to
> Master/Slave - our customers mentioned immediately our app is 2 times
> faster - again < 1 Sec latency. Then I needed to add https which is
> very important for our app. This do not added significant latency so our
> customers are very happy with this solution. Soon we will go public.
>
> So the latency issue is a show stopper for high replication version
> and we would want to stay with master slave.
>
> Anybody with similar experience?
>
> Thank you,
>
>
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/P1c_gMSRtl0J.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>



-- 
Fred Sauer
Developer Advocate
Google Inc.
1600 Amphitheatre Parkway
Mountain View, CA 94043
fre...@google.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Fred Sauer]

That's accurate. Performing a batch get to get a N entities from different
entity groups is slower or HRD. As N increases the issue is more pronounced.
This is because the datastore needs to fetch a consistent copy for each
entity.

Placing these entities into a single entity group can help greatly. Fetching
fewer entities initially, so that the UI updates quickly, and then fetching
more as needed, is always a good idea.

A great tool to help understand your use of the datastore is Appstats, which
can be easily enabled and provides negligible overhead to most sites. (If
you have a high traffic site, or don't want to enable Appstats in production
for other reasons, you can easily deploy a non-default version of your app
with Appstats enabled). Appstats will provide you a graphical overview of
the datastore (and other RPC) calls you are making.

On Thu, Aug 18, 2011 at 12:19 PM, Robert Kluin wrote:

> When switching to the HR datastore, you might need to make some
> adjustments.  For instance, fetching a large list of keys from
> multiple entity groups may be slower due to the transaction overhead.
> So you might need to evaluate your use of entity groups.  It may also
> be that you've got a design that is making many datastore RPCs in
> serial, that will probably be more of an issue when on HR since there
> is a slightly higher average latency.
>
>
> Robert
>
>
>
>
>
> On Thu, Aug 18, 2011 at 15:11, thstart  wrote:
> > Let me share my experience.
> > I was developing an AppEngine app for a year.
> > It was very fast (extremely fast), with latency > 1 sec.
> > When High Replication data-store was released I decided to give it
> > a try and switched to it. Unfortunately the latency got >1 Sec
> > sometimes > 2 Sec. The app worked a wile that way and because
> > still in beta I could experiment. So I decided to go to Billed option
> > to see if the reliability and more importantly the latency could improve.
> > Unfortunately this do not affected latency. Then I decided to go back to
> > Master/Slave - our customers mentioned immediately our app is 2 times
> > faster - again < 1 Sec latency. Then I needed to add https which is
> > very important for our app. This do not added significant latency so our
> > customers are very happy with this solution. Soon we will go public.
> > So the latency issue is a show stopper for high replication version
> > and we would want to stay with master slave.
> > Anybody with similar experience?
> > Thank you,
> >
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msg/google-appengine/-/P1c_gMSRtl0J.
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine+unsubscr...@googlegroups.com.
> > For more options, visit this group at
> > http://groups.google.com/group/google-appengine?hl=en.
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>


-- 
Fred Sauer
Developer Advocate
Google Inc.
1600 Amphitheatre Parkway
Mountain View, CA 94043
fre...@google.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Robert Kluin]

When switching to the HR datastore, you might need to make some
adjustments.  For instance, fetching a large list of keys from
multiple entity groups may be slower due to the transaction overhead.
So you might need to evaluate your use of entity groups.  It may also
be that you've got a design that is making many datastore RPCs in
serial, that will probably be more of an issue when on HR since there
is a slightly higher average latency.


Robert





On Thu, Aug 18, 2011 at 15:11, thstart  wrote:
> Let me share my experience.
> I was developing an AppEngine app for a year.
> It was very fast (extremely fast), with latency > 1 sec.
> When High Replication data-store was released I decided to give it
> a try and switched to it. Unfortunately the latency got >1 Sec
> sometimes > 2 Sec. The app worked a wile that way and because
> still in beta I could experiment. So I decided to go to Billed option
> to see if the reliability and more importantly the latency could improve.
> Unfortunately this do not affected latency. Then I decided to go back to
> Master/Slave - our customers mentioned immediately our app is 2 times
> faster - again < 1 Sec latency. Then I needed to add https which is
> very important for our app. This do not added significant latency so our
> customers are very happy with this solution. Soon we will go public.
> So the latency issue is a show stopper for high replication version
> and we would want to stay with master slave.
> Anybody with similar experience?
> Thank you,
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/P1c_gMSRtl0J.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[thstart]

Let me share my experience. 

I was developing an AppEngine app for a year.
It was very fast (extremely fast), with latency > 1 sec.
When High Replication data-store was released I decided to give it
a try and switched to it. Unfortunately the latency got >1 Sec 
sometimes > 2 Sec. The app worked a wile that way and because 
still in beta I could experiment. So I decided to go to Billed option
to see if the reliability and more importantly the latency could improve.
Unfortunately this do not affected latency. Then I decided to go back to 
Master/Slave - our customers mentioned immediately our app is 2 times 
faster - again < 1 Sec latency. Then I needed to add https which is 
very important for our app. This do not added significant latency so our
customers are very happy with this solution. Soon we will go public.

So the latency issue is a show stopper for high replication version 
and we would want to stay with master slave. 

Anybody with similar experience?

Thank you,




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/P1c_gMSRtl0J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Joshua Smith]

On Aug 18, 2011, at 2:10 PM, Fred Sauer wrote:
> On Wednesday, August 17, 2011 5:24:48 AM UTC-7, Joshua Smith wrote:
> I certainly hope they don't deprecate M/S, unless/until the following issues 
> are addressed:
> 1) Ability to migrate without changing app ID.  This is fundamental, and 
> we've never seen any *reason* why this is impossible.  I suspect that this is 
> very hard, but not impossible.
> 
> We are able to create an alias for migrated apps, so that traffic sent to 
> oldappid.appspot.com will be transparently routed to your new application. So 
> while you won't be able to keep the same app id, you can keep using the same 
> appspot.com domain, which is probably what you care about most.

Perhaps that what some people care about most, but not me.  What I care about 
is what I spelled out in the next two bullet points :)
> 
>  Changing the app ID causes problems because: 
> 
> - External references to entity keys (not IDs) are pervasive in many apps 
> (such as google search links to deep pages, user bookmarks, or RSS feed URLs) 
> because the documentation encourages the use of these keys in URLs
> 
> This remains an issues, but can be easily addressed in a couple of lines of 
> application code in the appropriate request handlers. The string encoded keys 
> that are referenced externally have encoded within them the old app id. If 
> you take the decoded key and extract the parent, kind and id/name to create a 
> new key (again, this is 1-2 lines of code), then your app can use that new 
> key to locate the migrated entity within the new datastore.
> 
> As far as reference (list) properties: those reference keys are automatically 
> updated as part of the the datastore migration and should not cause you any 
> issues.
> 
> - Many users have keys wrapped up in properties that the migration tool 
> cannot recognize [this doesn't impact my app, but it's a common issue on this 
> list]
> 
> This does affect some apps. As I mentioned above, a couple of lines of code 
> in the right place should take care of this.

1-2 lines of code, in dozens of handlers is the crux of the problem.  Dealing 
with this at the application layer is problematic because there are so many 
code paths that need to be examined.

> As an alternative, the infrastructure could be fixed to recognize the case of 
> using the old app ID instead of the new app ID in a key, and quietly make the 
> substitution.
> 
> That's a neat idea.

Neat enough that you guys might implement it?

I've suggested this before, but I'm glad that I've got your attention this time!

This would be a pretty huge step toward making the migration practical for me.

> 
> 2) Migration of large data sets takes an extremely long time.  Given that we 
> regularly see M/S downtimes in the couple of hours range, a couple of hours 
> is probably a reasonable amount of time to migrate.  A couple of days is much 
> too much.  [Note that I have not personally gotten far enough into a 
> migration to see this duration; I'm basing this on many user complaints I've 
> read on this list.]
> 
> Applications with small amounts of data can easily self migrate in a short 
> period of time using datastore_admin. Applications with more data will 
> benefit from the new migration tool Ikai mentioned. We created the tool 
> specifically to simplify and speed up the migration for larger apps, and to 
> reduce the read-only period required to make the final switch.

Yes, I saw that in his note, and if it works as advertised, I suspect it will 
resolve this issue.

> 
> 3) The problem of transactional consistency needs to be addressed.  It is not 
> acceptable for transactions to end in a "maybe this worked, maybe it didn't" 
> state.  They should either succeed or fail.  (Although the docs not not say 
> whether this problem is exclusive to HR, it seems to be, and certainly it 
> matters much more in HR since every put is implicitly transactional.)
> 
> There are subtle differences between the consistency guarantees offered by 
> M/S and HRD. In some cases, apps need to make a few tweaks in order to work 
> correctly in the new environment. The benefit is much more 
> predictable/consistency latency and increased reliability. A good article 
> which reviews the consistency guarantees can be found here:
>   http://code.google.com/appengine/articles/transaction_isolation.html
> 

I'm not talking about consistency guarantees.  I get that things change under 
HR, and it's pretty easy to understand what the implications of that are.

What I'm talking about is the problem where you have to write idempotent 
transactions because you can never tell whether a transaction that threw 
certain exceptions succeeded or failed.  That's really messed up, and you 
really need to fix that.

But, to be fair, I'm not sure this problem will cause me actual problems if I 
switch to HR.  It just makes me really, really uncomfortable.

> 
> 4) As Ikai said, there are some unresolved issues in HR tha

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Fred Sauer]

On Wednesday, August 17, 2011 5:24:48 AM UTC-7, Joshua Smith wrote:
>
> I certainly hope they don't deprecate M/S, unless/until the following 
> issues are addressed:
>
> 1) Ability to migrate without changing app ID.  This is fundamental, and 
> we've never seen any *reason* why this is impossible.  I suspect that this 
> is very hard, but not impossible.
>
We are able to create an alias for migrated apps, so that traffic sent to 
oldappid.appspot.com will be transparently routed to your new application. 
So while you won't be able to keep the same app id, you can keep using the 
same appspot.com domain, which is probably what you care about most.

 

>  Changing the app ID causes problems because: 
>
- External references to entity keys (not IDs) are pervasive in many apps 
> (such as google search links to deep pages, user bookmarks, or RSS feed 
> URLs) because the documentation encourages the use of these keys in URLs
>
This remains an issues, but can be easily addressed in a couple of lines of 
application code in the appropriate request handlers. The string encoded 
keys that are referenced externally have encoded within them the old app id. 
If you take the decoded key and extract the parent, kind and id/name to 
create a new key (again, this is 1-2 lines of code), then your app can use 
that new key to locate the migrated entity within the new datastore.

As far as reference (list) properties: those reference keys are 
automatically updated as part of the the datastore migration and should not 
cause you any issues.


 

> - Many users have keys wrapped up in properties that the migration tool 
> cannot recognize [this doesn't impact my app, but it's a common issue on 
> this list]
>
This does affect some apps. As I mentioned above, a couple of lines of code 
in the right place should take care of this.

 

> As an alternative, the infrastructure could be fixed to recognize the case 
> of using the old app ID instead of the new app ID in a key, and quietly make 
> the substitution.
>
That's a neat idea.

 

> 2) Migration of large data sets takes an extremely long time.  Given that 
> we regularly see M/S downtimes in the couple of hours range, a couple of 
> hours is probably a reasonable amount of time to migrate.  A couple of days 
> is much too much.  [Note that I have not personally gotten far enough into a 
> migration to see this duration; I'm basing this on many user complaints I've 
> read on this list.]
>
Applications with small amounts of data can easily self migrate in a short 
period of time using datastore_admin. Applications with more data will 
benefit from the new migration tool Ikai mentioned. We created the tool 
specifically to simplify and speed up the migration for larger apps, and to 
reduce the read-only period required to make the final switch.

 

> 3) The problem of transactional consistency needs to be addressed.  It is 
> not acceptable for transactions to end in a "maybe this worked, maybe it 
> didn't" state.  They should either succeed or fail.  (Although the docs not 
> not say whether this problem is exclusive to HR, it seems to be, and 
> certainly it matters much more in HR since every put is implicitly 
> transactional.)
>
There are subtle differences between the consistency guarantees offered by 
M/S and HRD. In some cases, apps need to make a few tweaks in order to work 
correctly in the new environment. The benefit is much more 
predictable/consistency latency and increased reliability. A good article 
which reviews the consistency guarantees can be found here:
  http://code.google.com/appengine/articles/transaction_isolation.html


4) As Ikai said, there are some unresolved issues in HR that need to be 
> fixed, such as the current mystery in the 'Serious Problem: Rollback of data 
> on HRD' thread (which is eerily familiar: I think somebody else reported a 
> similar issue a while back).
>

I haven't looked at that thread yet, but I suspect that reviewing the above 
article 
(http://code.google.com/appengine/articles/transaction_isolation.html) would 
probably be beneficial in the context of that thread.


Even if all these are addressed, it would certainly be nice if M/S remained 
> an option.  It makes different trade-offs than HR, which may be better 
> trade-offs for some applications.  I will be very disappointed if they 
> decide to make HR a requirement for multi-threading Python, since the two 
> are completely orthogonal choices.  From my reading, M/S seems to have 
> better throughput than HR, at the expense of more frequent downtime, and 
> spurious datastore timeouts.  I'd like to have that option in a 
> multi-threaded app.
>

Thanks for the input. I hope the above response addresses some of your 
concerns.

 

> -Joshua
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/sLkBsAfFc84J.
To pos

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Renan Mobile]

Thanks Ikai,

I totally agree with you about prohibit new M/S applications.

Thanks and let's getting better always.  I really trust on Google.
--
Atenciosamente(thanks),
 Renan Franca
-
Presidente

- Renan Mobile ltda: http://renanmobile.com - Soluções em Dispositivos
Móveis (Smartphones) com integração via web.
- Soluções em:
 Android (Smartphones);
 Google Web Toolkit (Web);
Em 17/08/2011 13:36, "Ikai Lan (Google)"  escreveu:
> Master/Slave doesn't have better throughput. In the 99th percentile,
because
> of datastore latency spikes, high replication far outperforms
master/slave.
>
> I don't think there are plans to completely deprecate master/slave, but
> there's been serious talk of closing off the creation of new master/slave
> applications.
>
> --
> Ikai Lan
> Developer Programs Engineer, Google App Engine
> plus.ikailan.com | twitter.com/ikai
>
>
>
> On Wed, Aug 17, 2011 at 5:24 AM, Joshua Smith wrote:
>
>> I certainly hope they don't deprecate M/S, unless/until the following
>> issues are addressed:
>>
>> 1) Ability to migrate without changing app ID. This is fundamental, and
>> we've never seen any *reason* why this is impossible. I suspect that this
>> is very hard, but not impossible. Changing the app ID causes problems
>> because:
>>
>> - External references to entity keys (not IDs) are pervasive in many apps
>> (such as google search links to deep pages, user bookmarks, or RSS feed
>> URLs) because the documentation encourages the use of these keys in URLs
>>
>> - Many users have keys wrapped up in properties that the migration tool
>> cannot recognize [this doesn't impact my app, but it's a common issue on
>> this list]
>>
>> As an alternative, the infrastructure could be fixed to recognize the
case
>> of using the old app ID instead of the new app ID in a key, and quietly
make
>> the substitution.
>>
>> 2) Migration of large data sets takes an extremely long time. Given that
>> we regularly see M/S downtimes in the couple of hours range, a couple of
>> hours is probably a reasonable amount of time to migrate. A couple of
days
>> is much too much. [Note that I have not personally gotten far enough into
a
>> migration to see this duration; I'm basing this on many user complaints
I've
>> read on this list.]
>>
>> 3) The problem of transactional consistency needs to be addressed. It is
>> not acceptable for transactions to end in a "maybe this worked, maybe it
>> didn't" state. They should either succeed or fail. (Although the docs not
>> not say whether this problem is exclusive to HR, it seems to be, and
>> certainly it matters much more in HR since every put is implicitly
>> transactional.)
>>
>> 4) As Ikai said, there are some unresolved issues in HR that need to be
>> fixed, such as the current mystery in the 'Serious Problem: Rollback of
data
>> on HRD' thread (which is eerily familiar: I think somebody else reported
a
>> similar issue a while back).
>>
>> Even if all these are addressed, it would certainly be nice if M/S
remained
>> an option. It makes different trade-offs than HR, which may be better
>> trade-offs for some applications. I will be very disappointed if they
>> decide to make HR a requirement for multi-threading Python, since the two
>> are completely orthogonal choices. From my reading, M/S seems to have
>> better throughput than HR, at the expense of more frequent downtime, and
>> spurious datastore timeouts. I'd like to have that option in a
>> multi-threaded app.
>>
>> -Joshua
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Ikai Lan (Google)]

Master/Slave doesn't have better throughput. In the 99th percentile, because
of datastore latency spikes, high replication far outperforms master/slave.

I don't think there are plans to completely deprecate master/slave, but
there's been serious talk of closing off the creation of new master/slave
applications.

--
Ikai Lan
Developer Programs Engineer, Google App Engine
plus.ikailan.com | twitter.com/ikai



On Wed, Aug 17, 2011 at 5:24 AM, Joshua Smith wrote:

> I certainly hope they don't deprecate M/S, unless/until the following
> issues are addressed:
>
> 1) Ability to migrate without changing app ID.  This is fundamental, and
> we've never seen any *reason* why this is impossible.  I suspect that this
> is very hard, but not impossible.  Changing the app ID causes problems
> because:
>
> - External references to entity keys (not IDs) are pervasive in many apps
> (such as google search links to deep pages, user bookmarks, or RSS feed
> URLs) because the documentation encourages the use of these keys in URLs
>
> - Many users have keys wrapped up in properties that the migration tool
> cannot recognize [this doesn't impact my app, but it's a common issue on
> this list]
>
> As an alternative, the infrastructure could be fixed to recognize the case
> of using the old app ID instead of the new app ID in a key, and quietly make
> the substitution.
>
> 2) Migration of large data sets takes an extremely long time.  Given that
> we regularly see M/S downtimes in the couple of hours range, a couple of
> hours is probably a reasonable amount of time to migrate.  A couple of days
> is much too much.  [Note that I have not personally gotten far enough into a
> migration to see this duration; I'm basing this on many user complaints I've
> read on this list.]
>
> 3) The problem of transactional consistency needs to be addressed.  It is
> not acceptable for transactions to end in a "maybe this worked, maybe it
> didn't" state.  They should either succeed or fail.  (Although the docs not
> not say whether this problem is exclusive to HR, it seems to be, and
> certainly it matters much more in HR since every put is implicitly
> transactional.)
>
> 4) As Ikai said, there are some unresolved issues in HR that need to be
> fixed, such as the current mystery in the 'Serious Problem: Rollback of data
> on HRD' thread (which is eerily familiar: I think somebody else reported a
> similar issue a while back).
>
> Even if all these are addressed, it would certainly be nice if M/S remained
> an option.  It makes different trade-offs than HR, which may be better
> trade-offs for some applications.  I will be very disappointed if they
> decide to make HR a requirement for multi-threading Python, since the two
> are completely orthogonal choices.  From my reading, M/S seems to have
> better throughput than HR, at the expense of more frequent downtime, and
> spurious datastore timeouts.  I'd like to have that option in a
> multi-threaded app.
>
> -Joshua
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Renan Mobile]

Thanks Joshua,

I really like M/S, i don't have reasons to migrate my first application to
HR. If it's possible i would like stay on M/S.

Although I have a plan to build a new application that will works on HR,
because the specific business logic.

Oh! I'm a java developer.

Atenciosamente (thanks),

-
Presidente Renan Franca

Renan Mobile ltda: http://renanmobile.com - Soluções em Dispositivos Móveis
(Smartphones) com integração via web.
Soluções em:
Android (Smartphones);
Google Web Toolkit (Web);



2011/8/17 Joshua Smith 

> I certainly hope they don't deprecate M/S, unless/until the following
> issues are addressed:
>
> 1) Ability to migrate without changing app ID.  This is fundamental, and
> we've never seen any *reason* why this is impossible.  I suspect that this
> is very hard, but not impossible.  Changing the app ID causes problems
> because:
>
> - External references to entity keys (not IDs) are pervasive in many apps
> (such as google search links to deep pages, user bookmarks, or RSS feed
> URLs) because the documentation encourages the use of these keys in URLs
>
> - Many users have keys wrapped up in properties that the migration tool
> cannot recognize [this doesn't impact my app, but it's a common issue on
> this list]
>
> As an alternative, the infrastructure could be fixed to recognize the case
> of using the old app ID instead of the new app ID in a key, and quietly make
> the substitution.
>
> 2) Migration of large data sets takes an extremely long time.  Given that
> we regularly see M/S downtimes in the couple of hours range, a couple of
> hours is probably a reasonable amount of time to migrate.  A couple of days
> is much too much.  [Note that I have not personally gotten far enough into a
> migration to see this duration; I'm basing this on many user complaints I've
> read on this list.]
>
> 3) The problem of transactional consistency needs to be addressed.  It is
> not acceptable for transactions to end in a "maybe this worked, maybe it
> didn't" state.  They should either succeed or fail.  (Although the docs not
> not say whether this problem is exclusive to HR, it seems to be, and
> certainly it matters much more in HR since every put is implicitly
> transactional.)
>
> 4) As Ikai said, there are some unresolved issues in HR that need to be
> fixed, such as the current mystery in the 'Serious Problem: Rollback of data
> on HRD' thread (which is eerily familiar: I think somebody else reported a
> similar issue a while back).
>
> Even if all these are addressed, it would certainly be nice if M/S remained
> an option.  It makes different trade-offs than HR, which may be better
> trade-offs for some applications.  I will be very disappointed if they
> decide to make HR a requirement for multi-threading Python, since the two
> are completely orthogonal choices.  From my reading, M/S seems to have
> better throughput than HR, at the expense of more frequent downtime, and
> spurious datastore timeouts.  I'd like to have that option in a
> multi-threaded app.
>
> -Joshua
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Joshua Smith]

I certainly hope they don't deprecate M/S, unless/until the following issues 
are addressed:

1) Ability to migrate without changing app ID.  This is fundamental, and we've 
never seen any *reason* why this is impossible.  I suspect that this is very 
hard, but not impossible.  Changing the app ID causes problems because:

- External references to entity keys (not IDs) are pervasive in many apps (such 
as google search links to deep pages, user bookmarks, or RSS feed URLs) because 
the documentation encourages the use of these keys in URLs

- Many users have keys wrapped up in properties that the migration tool cannot 
recognize [this doesn't impact my app, but it's a common issue on this list]

As an alternative, the infrastructure could be fixed to recognize the case of 
using the old app ID instead of the new app ID in a key, and quietly make the 
substitution.

2) Migration of large data sets takes an extremely long time.  Given that we 
regularly see M/S downtimes in the couple of hours range, a couple of hours is 
probably a reasonable amount of time to migrate.  A couple of days is much too 
much.  [Note that I have not personally gotten far enough into a migration to 
see this duration; I'm basing this on many user complaints I've read on this 
list.]

3) The problem of transactional consistency needs to be addressed.  It is not 
acceptable for transactions to end in a "maybe this worked, maybe it didn't" 
state.  They should either succeed or fail.  (Although the docs not not say 
whether this problem is exclusive to HR, it seems to be, and certainly it 
matters much more in HR since every put is implicitly transactional.)

4) As Ikai said, there are some unresolved issues in HR that need to be fixed, 
such as the current mystery in the 'Serious Problem: Rollback of data on HRD' 
thread (which is eerily familiar: I think somebody else reported a similar 
issue a while back).

Even if all these are addressed, it would certainly be nice if M/S remained an 
option.  It makes different trade-offs than HR, which may be better trade-offs 
for some applications.  I will be very disappointed if they decide to make HR a 
requirement for multi-threading Python, since the two are completely orthogonal 
choices.  From my reading, M/S seems to have better throughput than HR, at the 
expense of more frequent downtime, and spurious datastore timeouts.  I'd like 
to have that option in a multi-threaded app.

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Ikai Lan (Google)]

Wow, thanks for the update, Mike!

HR isn't 100% problem free yet. We're still working through some of the
issues. Addressing infrastructure issues is one of our top priorities right
now, but we can only do so much for master/slave, which has been through a
bit of a rough patch lately.

I do want to urge people to look at migrating. We're getting really close to
releasing a self-migration tool, but developers are still going to have to
do some legwork. Make sure you understand the consistency guarantees of HR
datastore! I'm hearing a LOT of misconceptions from people who don't
understand get-by-key or ancestor questions. I recently did a talk about
this. Check out these slides here (they're in English; don't worry about the
title slide):

http://www.slideshare.net/ikailan/2011-auggddmexicocityhighreplicationdatastore

--
Ikai Lan
Developer Programs Engineer, Google App Engine
plus.ikailan.com | twitter.com/ikai



On Tue, Aug 16, 2011 at 6:30 PM, Mike Lawrence wrote:

> Just burned several days with Google Enterprise Support trying to
> troubleshoot intermittent long latencies;
> unnecessary dynamic instance starts; and failures during dynamic instance
> starts on a multi-threaded GAE Java app
> running on Master/Slave (MS) data store.
>
> My typical app latencies on MS were under 1 second per request.
> Periodically I'd see 5-20 second delays with very little traffic.
> My app starts in under 3 seconds (stripes/slim3), so when instances failed
> to start in 30 seconds, I suspected a GAE issue.
>
> After many attempts to resolve the issue, we finally decided to to try
> switching to the High Replication (HR) data store.
> All of the issues disappeared!!! The long latencies were all gone. My
> traffic was easily being served with a single instance.
> No more phantom dynamic instances. No more instance start crashes.
>
> I just volume tested my app. My median request went from 2.5 seconds on MS
> to under 1 second on HR with 1000 simultaneous users.
>
> Previously I was paying for 3 always-on instances. With HR enabled,
> there was no need for any always-on instances.
>
> So if High Replication is faster, and more stable, why should Google spend
> valuable resources supporting Master/Slave?
> Is the cost savings of MS worth all the issues it creates? Not for me.
>
> If you're experiencing similar problems on MS I highly recommend giving HR
> a shot.
>
> FYI, migration to HR was trivial:
> - created a new app id,
> - deployed my app to the new id,
> - used my app to initialize the data (this will be problematic for some),
> - had google point the old app id to the new app id.
> now serving from the original URL.
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/XCUAKZ5E1LYJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Should Google deprecate Master/Slave data store option?

[Mike Lawrence]

Just burned several days with Google Enterprise Support trying to 
troubleshoot intermittent long latencies; 
unnecessary dynamic instance starts; and failures during dynamic instance 
starts on a multi-threaded GAE Java app
running on Master/Slave (MS) data store.   

My typical app latencies on MS were under 1 second per request. Periodically 
I'd see 5-20 second delays with very little traffic.
My app starts in under 3 seconds (stripes/slim3), so when instances failed 
to start in 30 seconds, I suspected a GAE issue.

After many attempts to resolve the issue, we finally decided to to try 
switching to the High Replication (HR) data store.
All of the issues disappeared!!! The long latencies were all gone. My 
traffic was easily being served with a single instance.
No more phantom dynamic instances. No more instance start crashes.

I just volume tested my app. My median request went from 2.5 seconds on MS 
to under 1 second on HR with 1000 simultaneous users.

Previously I was paying for 3 always-on instances. With HR enabled, 
there was no need for any always-on instances.

So if High Replication is faster, and more stable, why should Google spend 
valuable resources supporting Master/Slave?
Is the cost savings of MS worth all the issues it creates? Not for me.

If you're experiencing similar problems on MS I highly recommend giving HR a 
shot.

FYI, migration to HR was trivial: 
- created a new app id, 
- deployed my app to the new id, 
- used my app to initialize the data (this will be problematic for some), 
- had google point the old app id to the new app id.
now serving from the original URL.


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/XCUAKZ5E1LYJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.