I've been using the same pattern as Jeff mentions for quite some time
-- even while I was on M/S. I use it because it reduces my problems
to "fetch by key" scenarios, and I can build multiple specialized
"indexes" in this way. Part of the reason I started doing this was
due to "exploding indexes" type issues; this lets me "control the
explosion," and possibly even defer the writes in some cases.
It also allows you to avoid contention issues when the Things are
"frequently" updated, but the indexed values may not be.
Robert
On Tue, Sep 20, 2011 at 15:03, Alfred Fuller
<arfuller+appeng...@google.com> wrote:
> An interesting notion. Although you could also just
> use ColorThings(key_name=color) as the parent entity for all the Things.
> This way the list of things would be queriable directly (using
> an ancestor query) and there would not be a limit on the number and size of
> Things. They also exist next to each other in the underlying big table so
> there is only one 'seek' to find them (which is the largest cost when
> looking things up if you don't count serialization).
>
> On Tue, Sep 20, 2011 at 12:37 PM, Jeff Schnitzer <j...@infohazard.org>
> wrote:
>>
>> I'm doing a lot of work lately with data that requires a large degree
>> of transactional consistency. One pattern I've found that makes some
>> of the pain of HRD eventuality go away is to add an extra entity that
>> uses your query field as a natural key. This really requires global
>> transactions to work (as announced, it's in trusted testing, wheee!)
>> but here's an example:
>>
>> Say you associate a facebook id with an account. In M/S, you'd
>> probably have something like this:
>>
>> class User {
>> @Id Long id;
>> long fbId;
>> ...
>> }
>>
>> ...and then when a request arrives with a facebook id, you would query
>> for the user record. No user record? Create one. With eventual
>> consistency, this creates a larger window (with M/S it was small)
>> where you can get duplicate Users for the same fbId.
>>
>> The solution to transactional integrity and strong consistency is to
>> add a FbId entity:
>>
>> class FbId {
>> @Id String fbId;
>> long userId;
>> }
>>
>> I've now got several of these mapping entities in place now. Using
>> global transactions to create the FbId and the User at the same time,
>> it seems to solve consistency issues entirely. I don't know how it
>> will perform yet under load, but obviously there's not heavy
>> contention in this situation so I would be surprised if the 2pc hurt
>> much.
>>
>> I'm starting to notice several of these FbId-type mapping objects
>> showing up in my code as a way to force queries (for unique items)
>> into strong consistency. I'm guessing you could do this for
>> multi-item queries using a list property instead:
>>
>> Instead of query(Thing.class).filter("color", someColor), you could
>> instead keep updating an entity like this:
>>
>> class ColorThings {
>> @Id String color;
>> List<Key<Thing>> things;
>> }
>>
>> ...which feels upside-down but really has a lot of advantages. If you
>> put ColorThings in memcache, it's like a query cache which actually
>> updates properly.
>>
>> Is anyone else noticing their code being pushed into this pattern by the
>> HRD?
>>
>> Jeff
>>
>> On Tue, Sep 20, 2011 at 10:10 AM, Ikai Lan (Google) <ika...@google.com>
>> wrote:
>> > Well, indexes are just Bigtable rows, so replication lag does apply to
>> > them
>> > as well.
>> > --
>> > Ikai Lan
>> > Developer Programs Engineer, Google App Engine
>> > plus.ikailan.com | twitter.com/ikai
>> >
>> >
>> > On Tue, Sep 20, 2011 at 7:42 AM, Mike Wesner <mbwes...@gmail.com> wrote:
>> >>
>> >> And then I went and used the word replication... i meant index lag.
>> >>
>> >> On Sep 20, 9:40 am, Mike Wesner <mbwes...@gmail.com> wrote:
>> >> > I don't think Ikai read your post...
>> >> >
>> >> > Robert and I wanted to write a little HRD status site to track this
>> >> > and get real data, but we haven't done so yet. I have never seen the
>> >> > replication take more than about 1s. I think 1s will cover about
>> >> > four
>> >> > 9's, but that is just an educated guess. Until we (the users)
>> >> > actually measure this over time I don't think we can know for sure.
>> >> >
>> >> > -Mike
>> >> >
>> >> > On Sep 19, 7:16 pm, Jeff Schnitzer <j...@infohazard.org> wrote:
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > > I know that an index update in the HRD will typically be visible
>> >> > > within a couple seconds. That's the average case. What is the
>> >> > > worst-case?
>> >> >
>> >> > > Assuming something in the datacenter goes wacky, how long might it
>> >> > > take for an index to update? Tens of seconds, minutes, hours,
>> >> > > days?
>> >> >
>> >> > > Thanks,
>> >> > > Jeff
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "Google App Engine" group.
>> >> To post to this group, send email to google-appengine@googlegroups.com.
>> >> To unsubscribe from this group, send email to
>> >> google-appengine+unsubscr...@googlegroups.com.
>> >> For more options, visit this group at
>> >> http://groups.google.com/group/google-appengine?hl=en.
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Google App Engine" group.
>> > To post to this group, send email to google-appengine@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > google-appengine+unsubscr...@googlegroups.com.
>> > For more options, visit this group at
>> > http://groups.google.com/group/google-appengine?hl=en.
>> >
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
