[google-appengine] Re: create_bulkloader_config doesn't reflect the latest schema changes

2013-06-26 Thread Hooman Korasani 
Any one from google here?

The statistics are failing since three days. Without up-to-date statistics 
there is no way to get the bulkloader working properly.  

http://stackoverflow.com/questions/17275202/how-to-force-create-bulkloader-config-to-pickup-latest-schema-data/17277052#comment25075099_17277052


On Tuesday, 25 June 2013 08:58:18 UTC+1, Hooman Korasani wrote:
>
> I have created a bulkloader.yaml like this:
>
> appcfg.py create_bulkloader_config --filename=tools/bulkloader.yaml --url=
> https://myid.appspot.com/_ah/remote_api
>
>
> However it doesn't reflect my changes from a day ago.  How long do I have 
> to wait until it picks it up?
>
> According to the data-store statistics, the last statistic update was *Last 
> updated: 1 day, 21:11:28 ago
> *However it also says *"**Statistics are updated at least once per day"
>
> *Is there any other way to get the bulkloader consider my latest changes?*
>
> *Many Thanks,
> Hooman*
>
> *
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: create_bulkloader_config doesn't reflect the latest schema changes

2013-06-27 Thread Jesse Rohwer 
Hi Hooman,

Bulkload operations on the HRD are eventually consistent, which means you 
cannot use bulkloader with HRD and be guaranteed to get up-to-date data.

You can read more in the bulkloader docs: 
https://developers.google.com/appengine/docs/python/tools/uploadingdata
In particular,
*Note: *This document applies to apps that use the master/slave 
datastore. 
If your app uses the High Replication datastore, it is possible to copy 
data from the app, but Google does not currently support this use case. If 
you attempt to copy from a High Replication datastore, you'll see a 
high_replication_warning error in the Admin Console, and the downloaded 
data might not include recently saved entities.

That said, it is true that in the past two days the HRD has had 
longer-than-usual eventual consistency. This is returning to a more normal 
range now, but again: for operations that are not strongly consistent, 
including bulkloader, there are no guarantees. I hope this helps clarify 
the situation.

Thanks,
Jesse Rohwer on behalf of the App Engine team

On Wednesday, June 26, 2013 2:41:40 AM UTC-7, Hooman Korasani wrote:
>
> Any one from google here?
>
> The statistics are failing since three days. Without up-to-date statistics 
> there is no way to get the bulkloader working properly.  
>
>
> http://stackoverflow.com/questions/17275202/how-to-force-create-bulkloader-config-to-pickup-latest-schema-data/17277052#comment25075099_17277052
>
>
> On Tuesday, 25 June 2013 08:58:18 UTC+1, Hooman Korasani wrote:
>>
>> I have created a bulkloader.yaml like this:
>>
>> appcfg.py create_bulkloader_config --filename=tools/bulkloader.yaml --url
>> =https://myid.appspot.com/_ah/remote_api
>>
>>
>> However it doesn't reflect my changes from a day ago.  How long do I have 
>> to wait until it picks it up?
>>
>> According to the data-store statistics, the last statistic update was *Last 
>> updated: 1 day, 21:11:28 ago
>> *However it also says *"**Statistics are updated at least once per day"
>>
>> *Is there any other way to get the bulkloader consider my latest changes?
>> *
>>
>> *Many Thanks,
>> Hooman*
>>
>> *
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: create_bulkloader_config doesn't reflect the latest schema changes

2013-06-27 Thread Hooman Korasani 

Hi Jesse,

First off thank you so much for replying to this issue. We are a young 
startup and are in process of migrating from AWS to GAE.
On day one (early this week) the bulkloader worked nicely even despite 
the HRD.  I understand I can't rely on it, but how else do we transfer 
our data to google app engine?


At least, if I were able to create one bulk_loader_config yaml, I would 
be able to transfer our data via cvs files.  Is there any other official 
way for HDS?


Many Thanks,
Hooman


On 27/06/13 18:58, Jesse Rohwer wrote:

Hi Hooman,

Bulkload operations on the HRD are eventually consistent, which means 
you cannot use bulkloader with HRD and be guaranteed to get up-to-date 
data.


You can read more in the bulkloader docs: 
https://developers.google.com/appengine/docs/python/tools/uploadingdata

In particular,
*Note: *This document applies to apps that use the master/slave 
datastore 
. If 
your app uses the High Replication datastore, it is possible to copy 
data from the app, but Google does not currently support this use 
case. If you attempt to copy from a High Replication datastore, you'll 
see a |high_replication_warning| error in the Admin Console, and the 
downloaded data might not include recently saved entities.


That said, it is true that in the past two days the HRD has had 
longer-than-usual eventual consistency. This is returning to a more 
normal range now, but again: for operations that are not strongly 
consistent, including bulkloader, there are no guarantees. I hope this 
helps clarify the situation.


Thanks,
Jesse Rohwer on behalf of the App Engine team

On Wednesday, June 26, 2013 2:41:40 AM UTC-7, Hooman Korasani wrote:

Any one from google here?

The statistics are failing since three days. Without up-to-date
statistics there is no way to get the bulkloader working properly.


http://stackoverflow.com/questions/17275202/how-to-force-create-bulkloader-config-to-pickup-latest-schema-data/17277052#comment25075099_17277052




On Tuesday, 25 June 2013 08:58:18 UTC+1, Hooman Korasani wrote:

I have created a bulkloader.yaml like this:

|
appcfg.py create_bulkloader_config
--filename=tools/bulkloader.yaml
--url=https://myid.appspot.com/_ah/remote_api

|


However it doesn't reflect my changes from a day ago.  How
long do I have to wait until it picks it up?

According to the data-store statistics, the last statistic
update was *Last updated: 1 day, 21:11:28 ago
*However it also says *"**Statistics are updated at least once
per day"

*Is there any other way to get the bulkloader consider my
latest changes?*

*Many Thanks,
Hooman*

*

--
You received this message because you are subscribed to a topic in the 
Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit 
https://groups.google.com/d/topic/google-appengine/1zv8LNucS7g/unsubscribe.
To unsubscribe from this group and all its topics, send an email to 
google-appengine+unsubscr...@googlegroups.com.

To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.




--
You received this message because you are subscribed to the Google Groups "Google 
App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: create_bulkloader_config doesn't reflect the latest schema changes

2013-06-28 Thread Jesse Rohwer 
Hi Hooman,

For uploading data, if your dataset is relatively small, the bulkloader 
will do fine. You might want to also consider uploading as CSV to Cloud 
Storage, then running a mapreduce to copy from Cloud Storage to Datastore. 
This is the method we would currently recommend for larger datasets, please 
see 
https://developers.google.com/appengine/docs/python/googlestorage/#Using_Google_Storage

Also, if your issue is just that the bulkloader auto config is not picking 
up recent datastore changes, you could manually edit the configuration as 
described here: 
https://developers.google.com/appengine/docs/python/tools/uploadingdata#Editing_the_Configuration_File
.

Hope that helps!

On Thursday, June 27, 2013 11:28:30 AM UTC-7, Hooman Korasani wrote:
>
>  Hi Jesse,
>
> First off thank you so much for replying to this issue. We are a young 
> startup and are in process of migrating from AWS to GAE.
> On day one (early this week) the bulkloader worked nicely even despite the 
> HRD.  I understand I can't rely on it, but how else do we transfer our data 
> to google app engine?
>
> At least, if I were able to create one bulk_loader_config yaml, I would be 
> able to transfer our data via cvs files.  Is there any other official way 
> for HDS?
>
> Many Thanks,
> Hooman
>
>
> On 27/06/13 18:58, Jesse Rohwer wrote:
>  
> Hi Hooman, 
>
>  Bulkload operations on the HRD are eventually consistent, which means 
> you cannot use bulkloader with HRD and be guaranteed to get up-to-date data.
>
>  You can read more in the bulkloader docs: 
> https://developers.google.com/appengine/docs/python/tools/uploadingdata
> In particular,
> *Note: *This document applies to apps that use the master/slave 
> datastore. 
> If your app uses the High Replication datastore, it is possible to copy 
> data from the app, but Google does not currently support this use case. If 
> you attempt to copy from a High Replication datastore, you'll see a 
> high_replication_warning error in the Admin Console, and the downloaded 
> data might not include recently saved entities.
>  
>  That said, it is true that in the past two days the HRD has had 
> longer-than-usual eventual consistency. This is returning to a more normal 
> range now, but again: for operations that are not strongly consistent, 
> including bulkloader, there are no guarantees. I hope this helps clarify 
> the situation.
>
>  Thanks,
> Jesse Rohwer on behalf of the App Engine team
>
> On Wednesday, June 26, 2013 2:41:40 AM UTC-7, Hooman Korasani wrote: 
>>
>> Any one from google here?
>>
>> The statistics are failing since three days. Without up-to-date 
>> statistics there is no way to get the bulkloader working properly.  
>>
>>
>> http://stackoverflow.com/questions/17275202/how-to-force-create-bulkloader-config-to-pickup-latest-schema-data/17277052#comment25075099_17277052
>>
>>
>> On Tuesday, 25 June 2013 08:58:18 UTC+1, Hooman Korasani wrote: 
>>>
>>> I have created a bulkloader.yaml like this:
>>>
>>>  appcfg.py create_bulkloader_config --filename=tools/bulkloader.yaml --
>>> url=https://myid.appspot.com/_ah/remote_api
>>>  
>>>
>>> However it doesn't reflect my changes from a day ago.  How long do I 
>>> have to wait until it picks it up?
>>>
>>> According to the data-store statistics, the last statistic update was *Last 
>>> updated: 1 day, 21:11:28 ago
>>> *However it also says *"**Statistics are updated at least once per day"
>>>
>>> *Is there any other way to get the bulkloader consider my latest 
>>> changes?*
>>>
>>> *Many Thanks,
>>> Hooman*
>>>
>>> *
>>>
>>  -- 
> You received this message because you are subscribed to a topic in the 
> Google Groups "Google App Engine" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/google-appengine/1zv8LNucS7g/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to 
> google-appengi...@googlegroups.com .
> To post to this group, send email to google-a...@googlegroups.com
> .
> Visit this group at http://groups.google.com/group/google-appengine.
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  
>
>
>  

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.