[google-appengine] Re: New bill did not draw from my Discounted Instance Hours pool
If you're on Python, the cheapest option is currently 0 discounted hours, as the normal frontend hours are already discounted, until December. Current rates are: Python Frontend: $0.04/hour (normal rate $0.08/hour) Discounted hours: $0.05/hour Unless I've missed something? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/8-xZS6YNQDQJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: The Dining Philosophers
I think it's basically possible to do locking, mutexes, semaphores, etc in memcache, by taking advantage of some of the semantics, plus the new-ish compare store stuff. There's the obvious caveat that it could be flushed without warning, or disabled for a while, so it's not a high reliability thing. def add(self, key, value, time=0, min_compress_len=0, namespace=None): Sets a key's value, iff item is not already in memcache. def replace(self, key, value, time=0, min_compress_len=0, namespace=None): Replaces a key's value, failing if item isn't already in memcache. def cas(self, key, value, time=0, min_compress_len=0, namespace=None): Compare-And-Set update. def incr(self, key, delta=1, namespace=None, initial_value=None): Atomically increments a key's value. def decr(self, key, delta=1, namespace=None, initial_value=None): Atomically decrements a key's value. It seems to me the you could carefully use the above in various creative ways to implement this, if you require frequent use and the datastore costs would be prohibitive for it. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/cAlYR6jxlogJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Datastore update without read
You can't update just a single property/attribute of an entity unless you already know the values for all of the others. If it's frequently updated, careful use of memcache may allow you to avoid the db.get, but otherwise you have to read it first. An alternative, if you just have say 1 property/attribute which changes often would be to split the entity into two separate entity models, one with the relatively static data, and the other with the frequently changing data. The downside there, of course, is that you're going to have 2 db.get operations when you need all of it, instead of 1. Internally, the datastore is entirely composed of (key, value) pairs, with a single (key,value) per entity (not counting indices, metadata, etc), and the SDK encodes/decodes those values into your entity models for you. They are actually encoded as protobufs, but there's nothing in the datastore's API to modify that data without reading it first. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/e6JrqffTES4J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Keeping sensitive information in memory
To be honest, if any part of the GAE servers are compromised, they have your entire application (everything you've uploaded) and data right there, if they have full control over the server host. Securing the system's memory swap is not going to offer terribly much at that point. This is just somewhere you're going to have to trust Google to get the security perimeter of the servers and network absolutely perfect. If trusting them with that is too high a risk for your data, then frankly you shouldn't be on GAE, and probably shouldn't be on any external cloud solution. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/JtKmw95VAkQJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Change Class of Polymodel Instance
Here's a method I've discovered for changing the class on PolyModel, which I think is relatively clean: ent1 = Model1.get(…) # Or get_by_id, etc dict = db.to_dict(ent1) del dict['class'] # Other changes to dict, as required, adding/removing properties ent2 = Model2(key=ent1.key(), **dict) ent2.put() I think that's fairly clean, although I've not used it heavily. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/-CAdxp8LpocJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Need Python example of Template System
You should probably use some other template system (e.g. django.template), as webapp.template is going to be deprecated soon. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/kvBZWObe1t0J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Change Class of Polymodel Instance
Nick, any comment on my approach of changing the PolyModel's class via db.to_dict and creating a new model with the same key? That seems to me like a relatively clean safe way to do it. The desire to be able to do this for my case, is to be able to move records from Person to User, then User to SuperUser, gaining extra fields needed as the person descends the hierarchy. I suspect that's a fairly common scenario, and seems almost exactly the type of thing that PolyModel was designed for. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/O3DQBp7xtBAJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Managing Dev and Live versions
You should also take a look at multitenancy / namespaces (same feature, 2 different names for it). http://code.google.com/appengine/docs/python/multitenancy/ I'd guess it can also be done in Java, my GAE experience is all Python. In Python, you setup the default namespace for a request in your appengine_config.py based on more or less whatever logic you want, so easy to do it based on the URL hostname prefix. The namespace effectively gives a soft partitioning of the datastore memcache (I say soft, as it's possible for a request handler to access all namespaces, if it wants to). -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Pk1TxNpIgz8J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Assigning datastore IDs based on UUIDs from an external DB
After a little testing, I discovered a flaw in my previously posted
technique, namely that it was generating uint64 IDs, when the datastore will
only accept non-zero uint63s. I'd also forgotten to include my UUIDProperty
class. Here's the updated version. The question remains whether there's
likely to be any horrible loss of efficiency or problems by assigning IDs
which will be essentially randomly distributed through the possible number
space (the external DB mostly uses UUIDv4, so the distribution should be
quite random). As previously stated, auto-assigned IDs will not be used for
these models, so collisions or disruption of auto-assignment are not a major
concerns. The code probably still needs the odd bit of polishing here and
there.
MASK_64 = 2**64-1
MASK_63 = 2**63-1
class UUID(uuid.UUID):
# Could use this, but Python doesn't guarantee future stability of
# hash values
# def get_id(self):
# return abs(hash(self.int))
def get_id(self):
Returns a positive, non-zero 63 bit integer from the
UUID's int128
x = ((self.int MASK_64) ^ (self.int 64)) MASK_63
if (x == 0):
x = 1
return x
id = property(get_id)
class UUIDProperty(Property):
A UUID property, stored as a 16 byte binary string.
data_type = UUID
def get_value_for_datastore(self, model_instance):
uuid = super(UUIDProperty,
self).get_value_for_datastore(model_instance)
return ByteString(uuid.bytes)
def make_value_from_datastore(self, value):
if value is None:
return None
return UUID(bytes=value)
def validate(self, value):
if value is not None and not isinstance(value, self.data_type):
try:
value = self.data_type(value)
except TypeError, err:
raise BadValueError('Property %s must be convertible '
'to a %s instance (%s)' %
(self.name, self.data_type.__name__,
err))
value = super(UUIDProperty, self).validate(value)
if value is not None and not isinstance(value, self.data_type):
raise BadValueError('Property %s must be a %s instance' %
(self.name, self.data_type.__name__))
return value
class UUIDModel(Model):
@classmethod
def get_by_uuid(cls, uuids, **kwds):
uuids, multiple = datastore.NormalizeAndTypeCheck(uuids, (UUID,
str))
def normalize(uuid):
if isinstance(uuid, str):
return UUID(uuid)
else:
return uuid
uuids = [normalize(uuid) for uuid in uuids]
ids = [uuid.id for uuid in uuids]
entities = cls.get_by_id(ids, **kwds)
for index, entity in enumerate(entities):
if entity is not None and entity.uuid != uuids[index]:
raise BadKeyError('UUID hash collision detected (class %s):
'
'%s / %s'
% (cls.kind(), entity.uuid, uuids[index]))
if multiple:
return entities
else:
return entities[0]
@classmethod
def get_or_insert_by_uuid(cls, uuid, **kwds):
if isinstance(uuid, str):
uuid = UUID(uuid)
id = uuid.id
def txn():
entity = cls.get_by_id(id, parent=kwds.get('parent'))
if entity is None:
entity = cls(key=Key.from_path(cls.kind(), id,
parent=kwds.get('parent')),
uuid=uuid,
**kwds)
entity.put()
elif entity.uuid != uuid:
raise BadKeyError('UUID hash collision detected (class %s):
'
'%s / %s'
% (cls.kind(), entity.uuid, uuid))
return entity
return db.run_in_transaction(txn)
uuid = UUIDProperty('UUID')
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/BEkUAv6h6icJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Assigning datastore IDs based on UUIDs from an external DB
Hi folks,
I've been pondering the best approach to modelling objects where the objects
in my GAE datastore correspond to a subset of objects in an external DB
where primary keys are UUIDs. As I expect most of my records in GAE to
really be quite small, I feel it's worth avoiding the storage size overhead
of just using the 36 character UUID as the key_name, and have come up with
the following to generate datastore uint63 IDs from UUIDs (Python, but the
questions are more general GAE efficiency questions):
MASK_64 = 2**64-1
class UUID(uuid.UUID):
def get_id(self):
return abs((self.int MASK_64) ^ (self.int 64))
id = property(get_id)
class UUIDModel(Model):
@classmethod
def get_by_uuid(cls, uuids, **kwds):
uuids, multiple = datastore.NormalizeAndTypeCheck(uuids, (UUID,
str))
def normalize(uuid):
if isinstance(uuid, str):
return UUID(uuid)
else:
return uuid
uuids = [normalize(uuid) for uuid in uuids]
ids = [uuid.id for uuid in uuids]
entities = cls.get_by_id(ids, **kwds)
for index, entity in enumerate(entities):
if entity is not None and entity.uuid != uuids[index]:
raise BadValueError('UUID hash collision detected!')
if multiple:
return entities
else:
return entities[0]
@classmethod
def get_or_insert_by_uuid(cls, uuid, **kwds):
if isinstance(uuid, str):
uuid = UUID(uuid)
id = uuid.id
def txn():
entity = cls.get_by_id(id, parent=kwds.get('parent'))
if entity is None:
entity = cls(key=Key.from_path(cls.kind(), id,
parent=kwds.get('parent')),
uuid=uuid,
**kwds)
entity.put()
elif entity.uuid != uuid:
raise BadValueError('UUID hash collision detected!')
return entity
return db.run_in_transaction(txn)
uuid = UUIDProperty('UUID')
I won't be using GAE's auto-assigned IDs for the model classes which have
IDs assigned from the external DB's UUID, so I'm not terribly worried about
the probability of ID collision, as 2**63 is still a very large number space
compared to the number of records that I expect to have. My reason for
using a custom hashing of the UUID into a uint63 is because Python's hash
function isn't guaranteed to remain consistent with future Python versions.
The reason for using uint63 is because the datastore classes throw an
exception on negative int64s used as IDs. Had the datastore supported
int128 or uint127 for IDs, I would have just used the UUIDs more directly
with it. I'm using the UUID as the GAE key to allow direct get_by_id()
calls when I already know the UUID, rather than having to do a filtered
query on it.
So, on to the questions. The above seems to work just fine for me in early
prototype stages of development, but I'm wondering if there's a downside to
this technique? Will I hit any performance, space, or general efficiency
penalties with the datastore by using IDs which are essentially randomly
assigned throughout the entire 63bit ID space? Is there anything about this
which strikes people as a terrible idea and would justify me having a major
rethink about my approach? What techniques are others using when they have
externally assigned UUIDs as primary keys for some of their model classes?
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/IqEKH0ZY5BkJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
