[google-appengine] Re: Reading existing entity from DB

[Louise Elmose Hedegaard]

Ok, thank you for your input!

-Louise

Den onsdag den 27. januar 2016 kl. 12.39.54 UTC+1 skrev Louise Elmose 
Hedegaard:
>
>
> Hi,
>
> I have an issue with too many entities being created in my GAE datastore. 
>
> I have an unique id on an entity (which is not the key in this case), and 
> there should thus only be one entity in the datastore per unique id.
> When an entity is updated or created by the user, I check whether the 
> entity already exists in the DB, and only create a new entity, if there is 
> not already an existing entity with a matching id. If there is an existing 
> entity, I update the parameters of the entity.
> This logic should ensure that there is only one entry/entity per unique 
> id, but somehow several entities with the same id are created in the DB - I 
> do not understand how/why!
>
> When I look at this page 
> https://cloud.google.com/appengine/docs/java/datastore/entities#Java_Updating_an_entity
>  
> is says "You can use a transaction 
>  to 
> test whether an entity with a given key exists before creating one.".
>
> Do I really need to use transactions to validate whether there is an 
> existing entity with a given id, or does that only apply when you use the 
> key as primary key??
>
> Thanks,
> -Louise
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/aa9bfa26-6bb2-4369-b58c-d19b2d17adfc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Reading existing entity from DB

[Nick]

I think you've been given some good pointers in the right direction, I'll just 
add that you need to read and understand the eventual consistency, 
transactional and cross entity group (XG) transaction docs for the datastore.

My guess is based on your question you haven't wrapped your head around it, and 
as a result your data models and subsequently data are going to get broken. 
Fixing them after go live will be hard, so you need to get a lot closer now.

Once you've got the basic concepts down, it should become clearer how to model 
this, and why this is happening.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/61e6a8a7-3aaa-4cc2-b077-3c58632a271f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Reading existing entity from DB

[Karl MacMillan]

> On Jan 28, 2016, at 7:22 AM, Louise Elmose Hedegaard  
> wrote:
> 
> Hi guys,
> 
> Thank you for your answers.
> I do not think I have explained my problem properly though when I read your 
> answers.
> 
> I have a table "order" where I save different information about an order. The 
> unique id of the order comes from an external system, and I want to save this 
> unique id for the order.
> When an order is updated in the external system, I want to update the GAE 
> datastore accordingly. 
> For this reason I first check whether the order that was updated in the 
> external system already exists in my datastore. If it already exists I update 
> it, if it does not already exist, I create the order in the datastore.
> My problem is that I use this:
> 
> datastore.prepare(query).asSingleEntity();
> 
> which fails, as there are more than one entity with the unique id.
> I do not understand how several entities can be created with the same unique 
> id, when my logic says:
> 

In order for this to work as you expect you need strong consistency, which 
typically means a) have your entities be a part of the same entity group and b) 
do the query inside of a transaction.

As others have pointed out it’s much, much easier to simply use the id as they 
key and fetch by key. Though you still need to do the logic below inside of a 
transaction.

Karl

> Entity entity = getEntity(uniqueId);
> if (entity == null){
>   createEntity();
> }else{
>   updateEntity();
> }
> 
> It is as if the getEntity check does not detect that there already is an 
> entity with the unique id.
> It does seem that this is caused by a timing/concurrency issue - the external 
> system sends two updates approximately at the same time, but shouldn't the 
> datastore be able to handle this?
> 
> My first attempt of getEntity was:
> 
> Query query = new Query(OrderDBFields.ORDER_TABLE_NAME).setFilter(new 
> Query.FilterPredicate(OrderDBFields.ID, Query.FilterOperator.EQUAL, id));
> return datastore.prepare(query).asSingleEntity();
> 
> My second (confused attempt) is:
> 
> 
> Transaction txn = datastore.beginTransaction();
> Key orderKey = KeyFactory.createKey(OrderDBFields.ORDER_TABLE_NAME, 
> id);
> Entity orderEntity = null;
> try {
> orderEntity = datastore.get(orderKey);
> } catch (EntityNotFoundException e) {
> orderEntity = null;
> }
> txn.commit();
> return orderEntity;
> 
> Any input is appreciated,
> Thanks
> -Louise
> 
> Den onsdag den 27. januar 2016 kl. 12.39.54 UTC+1 skrev Louise Elmose 
> Hedegaard:
> 
> Hi,
> 
> I have an issue with too many entities being created in my GAE datastore. 
> 
> I have an unique id on an entity (which is not the key in this case), and 
> there should thus only be one entity in the datastore per unique id.
> When an entity is updated or created by the user, I check whether the entity 
> already exists in the DB, and only create a new entity, if there is not 
> already an existing entity with a matching id. If there is an existing 
> entity, I update the parameters of the entity.
> This logic should ensure that there is only one entry/entity per unique id, 
> but somehow several entities with the same id are created in the DB - I do 
> not understand how/why!
> 
> When I look at this page 
> https://cloud.google.com/appengine/docs/java/datastore/entities#Java_Updating_an_entity
>  
> 
>  is says "You can use a transaction 
>  to test 
> whether an entity with a given key exists before creating one.".
> 
> Do I really need to use transactions to validate whether there is an existing 
> entity with a given id, or does that only apply when you use the key as 
> primary key??
> 
> Thanks,
> -Louise
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to google-appengine@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/google-appengine 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/c774c62a-98d8-467d-b97a-c5e817644cc5%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscri

[google-appengine] Re: Reading existing entity from DB

[Louise Elmose Hedegaard]

Hi guys,

Thank you for your answers.
I do not think I have explained my problem properly though when I read your 
answers.

I have a table "order" where I save different information about an order. 
The unique id of the order comes from an external system, and I want to 
save this unique id for the order.
When an order is updated in the external system, I want to update the GAE 
datastore accordingly. 
For this reason I first check whether the order that was updated in the 
external system already exists in my datastore. If it already exists I 
update it, if it does not already exist, I create the order in the 
datastore.
My problem is that I use this:

datastore.prepare(query).asSingleEntity();

which fails, as there are more than one entity with the unique id.
I do not understand how several entities can be created with the same 
unique id, when my logic says:

Entity entity = getEntity(uniqueId);
if (entity == null){
  createEntity();
}else{
  updateEntity();
}

It is as if the getEntity check does not detect that there already is an 
entity with the unique id.
It does seem that this is caused by a timing/concurrency issue - the 
external system sends two updates approximately at the same time, but 
shouldn't the datastore be able to handle this?

My first attempt of getEntity was:

Query query = new Query(OrderDBFields.ORDER_TABLE_NAME).setFilter(new 
Query.FilterPredicate(OrderDBFields.ID, Query.FilterOperator.EQUAL, id));
return datastore.prepare(query).asSingleEntity();

My second (confused attempt) is:


Transaction txn = datastore.beginTransaction();
Key orderKey = KeyFactory.createKey(OrderDBFields.ORDER_TABLE_NAME, 
id);
Entity orderEntity = null;
try {
orderEntity = datastore.get(orderKey);
} catch (EntityNotFoundException e) {
orderEntity = null;
}
txn.commit();
return orderEntity;

Any input is appreciated,
Thanks
-Louise

Den onsdag den 27. januar 2016 kl. 12.39.54 UTC+1 skrev Louise Elmose 
Hedegaard:
>
>
> Hi,
>
> I have an issue with too many entities being created in my GAE datastore. 
>
> I have an unique id on an entity (which is not the key in this case), and 
> there should thus only be one entity in the datastore per unique id.
> When an entity is updated or created by the user, I check whether the 
> entity already exists in the DB, and only create a new entity, if there is 
> not already an existing entity with a matching id. If there is an existing 
> entity, I update the parameters of the entity.
> This logic should ensure that there is only one entry/entity per unique 
> id, but somehow several entities with the same id are created in the DB - I 
> do not understand how/why!
>
> When I look at this page 
> https://cloud.google.com/appengine/docs/java/datastore/entities#Java_Updating_an_entity
>  
> is says "You can use a transaction 
>  to 
> test whether an entity with a given key exists before creating one.".
>
> Do I really need to use transactions to validate whether there is an 
> existing entity with a given id, or does that only apply when you use the 
> key as primary key??
>
> Thanks,
> -Louise
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/c774c62a-98d8-467d-b97a-c5e817644cc5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Reading existing entity from DB

[timh]

Having a unique id that is only a property and not the key of the entity 
makes it hard.
Are you generating the ID - is it sequential, or just unique (ie an email 
address) if the later, you could reliably use a combination of transactions 
and a child entity with a key of the unique id, and the ancestors key).

You don't even have store any data.

This can be reliable as you are able to use get operations in a transaction.

Anything entity you fetch by a query (unless it's an ancestor query) will 
be liable to eventual consistancy issues.

T

On Wednesday, January 27, 2016 at 7:39:54 PM UTC+8, Louise Elmose Hedegaard 
wrote:
>
>
> Hi,
>
> I have an issue with too many entities being created in my GAE datastore. 
>
> I have an unique id on an entity (which is not the key in this case), and 
> there should thus only be one entity in the datastore per unique id.
> When an entity is updated or created by the user, I check whether the 
> entity already exists in the DB, and only create a new entity, if there is 
> not already an existing entity with a matching id. If there is an existing 
> entity, I update the parameters of the entity.
> This logic should ensure that there is only one entry/entity per unique 
> id, but somehow several entities with the same id are created in the DB - I 
> do not understand how/why!
>
> When I look at this page 
> https://cloud.google.com/appengine/docs/java/datastore/entities#Java_Updating_an_entity
>  
> is says "You can use a transaction 
>  to 
> test whether an entity with a given key exists before creating one.".
>
> Do I really need to use transactions to validate whether there is an 
> existing entity with a given id, or does that only apply when you use the 
> key as primary key??
>
> Thanks,
> -Louise
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/fc9c1861-f446-42fe-8fbb-bbd884ed5be1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.