The problem with using a key-parent is that it limits to a single
index -- say I want to index Things by color and texture.
The downside of this multiple-thing index entity is that (like a
parent-key) it limits throughput. And since there's a 2pc involved,
it probably limits throughput quite a lot...
Jeff
On Tue, Sep 20, 2011 at 1:03 PM, Alfred Fuller
<arfuller+appeng...@google.com> wrote:
> An interesting notion. Although you could also just
> use ColorThings(key_name=color) as the parent entity for all the Things.
> This way the list of things would be queriable directly (using
> an ancestor query) and there would not be a limit on the number and size of
> Things. They also exist next to each other in the underlying big table so
> there is only one 'seek' to find them (which is the largest cost when
> looking things up if you don't count serialization).
>
> On Tue, Sep 20, 2011 at 12:37 PM, Jeff Schnitzer <j...@infohazard.org>
> wrote:
>>
>> I'm doing a lot of work lately with data that requires a large degree
>> of transactional consistency. One pattern I've found that makes some
>> of the pain of HRD eventuality go away is to add an extra entity that
>> uses your query field as a natural key. This really requires global
>> transactions to work (as announced, it's in trusted testing, wheee!)
>> but here's an example:
>>
>> Say you associate a facebook id with an account. In M/S, you'd
>> probably have something like this:
>>
>> class User {
>> @Id Long id;
>> long fbId;
>> ...
>> }
>>
>> ...and then when a request arrives with a facebook id, you would query
>> for the user record. No user record? Create one. With eventual
>> consistency, this creates a larger window (with M/S it was small)
>> where you can get duplicate Users for the same fbId.
>>
>> The solution to transactional integrity and strong consistency is to
>> add a FbId entity:
>>
>> class FbId {
>> @Id String fbId;
>> long userId;
>> }
>>
>> I've now got several of these mapping entities in place now. Using
>> global transactions to create the FbId and the User at the same time,
>> it seems to solve consistency issues entirely. I don't know how it
>> will perform yet under load, but obviously there's not heavy
>> contention in this situation so I would be surprised if the 2pc hurt
>> much.
>>
>> I'm starting to notice several of these FbId-type mapping objects
>> showing up in my code as a way to force queries (for unique items)
>> into strong consistency. I'm guessing you could do this for
>> multi-item queries using a list property instead:
>>
>> Instead of query(Thing.class).filter("color", someColor), you could
>> instead keep updating an entity like this:
>>
>> class ColorThings {
>> @Id String color;
>> List<Key<Thing>> things;
>> }
>>
>> ...which feels upside-down but really has a lot of advantages. If you
>> put ColorThings in memcache, it's like a query cache which actually
>> updates properly.
>>
>> Is anyone else noticing their code being pushed into this pattern by the
>> HRD?
>>
>> Jeff
>>
>> On Tue, Sep 20, 2011 at 10:10 AM, Ikai Lan (Google) <ika...@google.com>
>> wrote:
>> > Well, indexes are just Bigtable rows, so replication lag does apply to
>> > them
>> > as well.
>> > --
>> > Ikai Lan
>> > Developer Programs Engineer, Google App Engine
>> > plus.ikailan.com | twitter.com/ikai
>> >
>> >
>> > On Tue, Sep 20, 2011 at 7:42 AM, Mike Wesner <mbwes...@gmail.com> wrote:
>> >>
>> >> And then I went and used the word replication... i meant index lag.
>> >>
>> >> On Sep 20, 9:40 am, Mike Wesner <mbwes...@gmail.com> wrote:
>> >> > I don't think Ikai read your post...
>> >> >
>> >> > Robert and I wanted to write a little HRD status site to track this
>> >> > and get real data, but we haven't done so yet. I have never seen the
>> >> > replication take more than about 1s. I think 1s will cover about
>> >> > four
>> >> > 9's, but that is just an educated guess. Until we (the users)
>> >> > actually measure this over time I don't think we can know for sure.
>> >> >
>> >> > -Mike
>> >> >
>> >> > On Sep 19, 7:16 pm, Jeff Schnitzer <j...@infohazard.org> wrote:
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > > I know that an index update in the HRD will typically be visible
>> >> > > within a couple seconds. That's the average case. What is the
>> >> > > worst-case?
>> >> >
>> >> > > Assuming something in the datacenter goes wacky, how long might it
>> >> > > take for an index to update? Tens of seconds, minutes, hours,
>> >> > > days?
>> >> >
>> >> > > Thanks,
>> >> > > Jeff
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "Google App Engine" group.
>> >> To post to this group, send email to google-appengine@googlegroups.com.
>> >> To unsubscribe from this group, send email to
>> >> google-appengine+unsubscr...@googlegroups.com.
>> >> For more options, visit this group at
>> >> http://groups.google.com/group/google-appengine?hl=en.
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Google App Engine" group.
>> > To post to this group, send email to google-appengine@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > google-appengine+unsubscr...@googlegroups.com.
>> > For more options, visit this group at
>> > http://groups.google.com/group/google-appengine?hl=en.
>> >
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
