[google-appengine] Data restore through bulkupload - is there a risk of key duplicate + is this right way of doing it ?

[hugues2]

Hi,

I am putting together scripts to backup / restore data from the datastore. 
These backup / restore scripts offer the possibility to manipulate data at 
the namespace level which is not possible at this stage with the standard 
datastore admin features.

The process is currently the following :

- On a regular basis, scripts will backup each namespace through the 
following command ($1 = namespace, $2=Kind) :

appcfg.py download_data --secure --email=$BACKUP_USERID 
--filename=data_$1_$2.csv --kind=$2 --url=$WEBSITE --namespace=$1 --passin 
<<-EOF
$BACKUP_PASSWORD
EOF

- If a restore for a given namespace is required, the following would be 
done
  --> Delete all the entities in a namespace
  --> Restore through the following command :

appcfg.py upload_data --secure --email=$BACKUP_USERID 
--filename=data_$1_$2.csv --kind=$2 --url=$WEBSITE --namespace=$1 --passin 
<<-EOF
$BACKUP_PASSWORD
EOF

My questions would be the following : 
--> Is there a risk to have key duplicates after the data has been deleted 
then restored in the namespace ? 
--> I read in other posts about the options (--dump and --restore). What is 
the effect of these options and is it correct that these options do not 
exist anymore ?
--> Is this the right way of backup / restore for one namespace ?

Thanks,

Hugues







> Hi Jeff,
>
> Maybe one more question related to restoring entities created with
> objectify. What I am trying to do is to restore the data for one
> namespace. The process would be the following :
>
> 1) Use Datastore admin tool to backup the overall Datastore to prevent
> from global problems
>
> 2) On top of the datastore admin tool use the bulkuploader tool to
> backup by namespace using the command proposed by Dennis (see above)
>
> 3) If needed restore for one namespace, I was planning to
>   --> 1) delete all the entities for that namespace
>   --> 2) use the bulkuploader to restore in that namespace
>
> My question would be related to the uniqueness of the keys after the
> restore. Is there a risk that some keys would be duplicate when the
> users will create new entities in that namespace. That is, how would
> the datastore know that some entities have been deleted then
> restored ?
>
> Thanks a lot in advance,
>
> Hugues

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/QVM2nfB-2Z8J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Datastore Indexes stuck. vacuum_indexes didn't fix it

[Takashi Matsuo]

Okay, the indexes should be serving now. Please enjoy!

-- Takashi
On Jun 11, 2012 11:35 AM, "Takashi Matsuo"  wrote:

>
> Hi Gwyn,
>
> Sorry for the inconvenience. We're looking into it.
>
> On Sun, Jun 10, 2012 at 4:46 PM, Gwyn Howell wrote:
>
>> I have datastore indexes stuck in "Building" status. I have tried
>> deleting my index.yaml and running vacuum_indexes. That deleted all the
>> other indexes, but not the 5 stuck in "Building" status. How do I fix this?
>> (Googlers, my App ID is phoenixhealth-dms!)
>>
>> Thanks
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msg/google-appengine/-/aQW5zXh1HIsJ.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
>
>
> --
> Takashi Matsuo | Developer Advocate | tmat...@google.com
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: java.util.ConcurrentModificationException on committing a transaction with only task queue entries

[Jeff Schnitzer]

ConcurentModifcationException happens whenever you have transaction
collisions.  You will often have transaction collisions if you modify
the same piece of data from multiple tasks.

There are two possible situations here:

1) "It happens" - sometimes transactions collide.  The solution is
that you should always make transactions idempotent and you should
always retry them.  If you're using Objectify4, doing an
ofy.transact(new Work() { ... }) will do the retries for you.

2) You may need to rethink the way data flows in your app.  If you
have a bunch of tasks trying to modify the same piece of data, you're
going to get a lot of collisions and collisions slow things down.
Figure out how you can serialize the transactions or combine them.

Jeff

On Sun, Jun 10, 2012 at 5:10 PM, Michael Hermus
 wrote:
> As an expirement, I replaced my use of the Objectify transaction with the
> normal low level datastore API transaction; the problem appears to have gone
> away. There is really no need to use the Objecty wrapper, since there are no
> actual Entities involved. If the issue comes back up subsequent to this
> change, I will post an update.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/oNr1a1W0shwJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Datastore Statistics Not Updating

[Takashi Matsuo]

We're aware of this issue. Now the datastore stats are updated less often
than it should be.
We'll followup with more detailed information shortly.

Sorry for the inconvenience.

-- Takashi

On Mon, Jun 11, 2012 at 4:50 AM, Francois Masurel wrote:

> In my case, it's :  *Last updated: 5 days, 8:36:35 ago*
>
>
>
> On Sunday, June 10, 2012 5:12:19 AM UTC+2, Steve wrote:
>>
>> *Am I not understanding how this is supposed to work?*
>>
>>
>> 
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/TTClVtkdkGAJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>



-- 
Takashi Matsuo | Developer Advocate | tmat...@google.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Datastore Indexes stuck. vacuum_indexes didn't fix it

[Takashi Matsuo]

Hi Gwyn,

Sorry for the inconvenience. We're looking into it.

On Sun, Jun 10, 2012 at 4:46 PM, Gwyn Howell wrote:

> I have datastore indexes stuck in "Building" status. I have tried deleting
> my index.yaml and running vacuum_indexes. That deleted all the other
> indexes, but not the 5 stuck in "Building" status. How do I fix this?
> (Googlers, my App ID is phoenixhealth-dms!)
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/aQW5zXh1HIsJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>



-- 
Takashi Matsuo | Developer Advocate | tmat...@google.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: java.util.ConcurrentModificationException on committing a transaction with only task queue entries

[Michael Hermus]

As an expirement, I replaced my use of the Objectify transaction with the 
normal low level datastore API transaction; the problem appears to have 
gone away. There is really no need to use the Objecty wrapper, since there 
are no actual Entities involved. If the issue comes back up subsequent to 
this change, I will post an update.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/oNr1a1W0shwJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: java.util.ConcurrentModificationException on committing a transaction with only task queue entries

[Michael Hermus]

I have just experienced this same issue. I submit 4 tasks to 4 different 
queues as a part of a single transaction. If multiple requests come in 
rapid succession, the following exception is raised upon commit:
 

java.util.ConcurrentModificationException: too much contention on these 
datastore entities. please try again.
at 
com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java:39)
at 
com.google.appengine.api.datastore.DatastoreApiHelper$1.convertException(DatastoreApiHelper.java:76)
at 
com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:106)
at 
com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:90)
at 
com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:90)
at 
com.googlecode.objectify.cache.TriggerFuture.get(TriggerFuture.java:99)
at 
com.googlecode.objectify.util.FutureHelper.quietGet(FutureHelper.java:26)
at 
com.googlecode.objectify.cache.TransactionWrapper.commit(TransactionWrapper.java:44)

 
I would really like to know if this is a bug (I really hope so) or a 
limitiation of the Task Queue service.
 

On Tuesday, May 8, 2012 5:51:08 AM UTC-4, FastGeert wrote:

> Hi,  
>
> last night our application suffered 
> from java.util.ConcurrentModificationException on committing a transaction 
> with only task queue entries.
>
> The transaction only had to persist 5 task queue entries, no additional 
> changes needed to be committed in the transaction.
>
> After plunging through the documentation I conclude that this must be a 
> bug in App Engine.
>
> Any thoughts on this ?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/nMlDU-KyL5MJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Datastore Statistics Not Updating

[Francois Masurel]

In my case, it's :  *Last updated: 5 days, 8:36:35 ago* 



On Sunday, June 10, 2012 5:12:19 AM UTC+2, Steve wrote:
>
> *Am I not understanding how this is supposed to work?*
>
>
> 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/TTClVtkdkGAJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] 500: descriptor 'capitalize' requires a 'str' object but received a 'NoneType' Traceback

[paulmo]

Although I get this error in development with cgitb.enable() (form field 
obviously blank
when page loads), app runs fine locally. Not working on GAE though, Server 
Error.  

: descriptor 'capitalize' requires a 'str' object 
but received 
a 'NoneType' Traceback (most recent call last):

Thanks in advance for suggestions to run this. 


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/C69xutR_iyAJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] 500: 'capitalize' requires a 'str' object but received a 'NoneType' Traceback Help Plz.

[paulmo]

granted I got this error in develop with cgitab.enable(), but app ran fine even 
so. Have uploaded
app and getting Server Error. This is log:  
: descriptor 'capitalize' requires a 'str' object 
but received a 'NoneType' Traceback (most recent call last):


Thanks in advance for suggestions to get this working on gae. 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/BOEALJjPKlcJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Datastore Indexes stuck. vacuum_indexes didn't fix it

[Gwyn Howell]

I have datastore indexes stuck in "Building" status. I have tried deleting 
my index.yaml and running vacuum_indexes. That deleted all the other 
indexes, but not the 5 stuck in "Building" status. How do I fix this? 
(Googlers, my App ID is phoenixhealth-dms!)

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/aQW5zXh1HIsJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.