[google-appengine] Re: upload data via appcfg

['Adam (Cloud Platform Support)' via Google App Engine]

If you are getting errors related to the app's partition ('~s' vs '~dev') 
you can try adding the following to 'appengine_config.py' so that it 
matches the one used by prod:

from google.appengine.datastore.entity_pb import Reference
Reference.app = lambda *args: os.environ['APPLICATION_ID'].replace('dev~', 
's~')

To set the desired namespace, you can use the namespace manager:

from google.appengine.api import namespace_manager
namespace_manager.set_namespace('my_namespace')

Let me know if you run into other problems.

On Tuesday, May 10, 2016 at 2:49:57 AM UTC-4, Rotem Vilkovski wrote:
>
>
> Hey Adam, Thanks again.
>
> My ConfigureRemoteApiForOAuth code now on appengine_config looks like:
>
> if os.environ['SERVER_SOFTWARE'].startswith('Development'):
> from google.appengine.ext.remote_api import remote_api_stub
> from google.appengine.api import apiproxy_stub_map
> import dev_appserver
> dev_appserver.fix_sys_path()
> local_stub = apiproxy_stub_map.apiproxy.GetStub('urlfetch')
> remote_api_stub.ConfigureRemoteApiForOAuth('storemavendev2.appspot.com', 
> '/_ah/remote_api')
> apiproxy_stub_map.apiproxy.ReplaceStub('urlfetch', local_stub)
>
>
>
> And this split for three scenarios.
> First - when secure flag is True in ConfigureRemoteApiForOAuth (by 
> default) I get that:
>   File "D:\google_appengine\google\appengine\runtime\wsgi.py", line 240, 
> in Handle
> handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
>   File "D:\google_appengine\google\appengine\api\lib_config.py", line 354, 
> in __getattr__
> self._update_configs()
>   File "D:\google_appengine\google\appengine\api\lib_config.py", line 290, 
> in _update_configs
> self._registry.initialize()
>   File "D:\google_appengine\google\appengine\api\lib_config.py", line 165, 
> in initialize
> import_func(self._modname)
>   File "D:\storemaven\code\appengine_config.py", line 12, in 
> remote_api_stub.ConfigureRemoteApiForOAuth('mockappstore.appspot.com', 
> '/_ah/remote_api', True)
>   File 
> "D:\google_appengine\google\appengine\ext\remote_api\remote_api_stub.py", 
> line 769, in ConfigureRemoteApiForOAuth
> rpc_server_factory=rpc_server_factory)
>   File 
> "D:\google_appengine\google\appengine\ext\remote_api\remote_api_stub.py", 
> line 839, in ConfigureRemoteApi
> app_id = GetRemoteAppIdFromServer(server, path, rtok)
>   File 
> "D:\google_appengine\google\appengine\ext\remote_api\remote_api_stub.py", 
> line 569, in GetRemoteAppIdFromServer
> response = server.Send(path, payload=None, **urlargs)
>   File 
> "D:\google_appengine\google\appengine\tools\appengine_rpc_httplib2.py", 
> line 246, in Send
> url, method=method, body=payload, headers=headers)
>   File "D:\google_appengine\lib\oauth2client\oauth2client\client.py", line 
> 569, in new_request
> redirections, connection_type)
>   File "D:\google_appengine\lib\httplib2\httplib2\__init__.py", line 1464, 
> in request
> self.disable_ssl_certificate_validation)
>   File "D:\google_appengine\lib\httplib2\httplib2\__init__.py", line 1143, 
> in __init__
> strict, timeout, proxy_info, ca_certs, 
> disable_ssl_certificate_validation)
>   File "D:\google_appengine\lib\httplib2\httplib2\__init__.py", line 1092, 
> in __init__
> raise NotSupportedOnThisPlatform()
> NotSupportedOnThisPlatform
>
> Second - when I'm turning off secrure by passing secure=False I get 
> information from some other namespace, how can I pass from which namespace 
> I wanat to work with my data?
>
> Third - whem I'm turning off secure and try to reach to my other project 
> (id: storemavendev2) I get that(In other projects it works just not on the 
> wanted namespace):
> Traceback (most recent call last):
>   File "D:\google_appengine\google\appengine\runtime\wsgi.py", line 267, 
> in Handle
> result = handler(dict(self._environ), self._StartResponse)
>   File "D:\storemaven\code\com\lib\gaesessions\__init__.py", line 475, in 
> __call__
> return self.app(environ, my_start_response)
>   File "D:\storemaven\code/com/lib\flask\app.py", line 1836, in __call__
> return self.wsgi_app(environ, start_response)
>   File "D:\storemaven\code/com/lib\flask\app.py", line 1820, in wsgi_app
> response = self.make_response(self.handle_exception(e))
>   File "D:\storemaven\code/com/lib\flask\app.py", line 1403, in 
> handle_exception
> reraise(exc_type, exc_value, tb)
>   File "D:\storemaven\code/com/lib\flask\app.py", line 1817, in wsgi_app
> response = self.full_dispatch_request()
>   File "D:\storemaven\code/com/lib\flask\app.py", line 1477, in 
> full_dispatch_request
> rv = self.handle_user_exception(e)
>   File "D:\storemaven\code/com/lib\flask\app.py", line 1381, in 
> handle_user_exception
> reraise(exc_type, exc_value, tb)
>   File "D:\storemaven\code/com/lib\flask\app.py", line 1475, in 
> full_dispatch_request
> rv = self.dispatch_request()
>   File "D:\storemaven\code/com/lib\flask\app.py", line 

[google-appengine] Re: upload data via appcfg

[Rotem Vilkovski]

Hey Adam, Thanks again.

My ConfigureRemoteApiForOAuth code now on appengine_config looks like:

if os.environ['SERVER_SOFTWARE'].startswith('Development'):
from google.appengine.ext.remote_api import remote_api_stub
from google.appengine.api import apiproxy_stub_map
import dev_appserver
dev_appserver.fix_sys_path()
local_stub = apiproxy_stub_map.apiproxy.GetStub('urlfetch')
remote_api_stub.ConfigureRemoteApiForOAuth('storemavendev2.appspot.com', 
'/_ah/remote_api')
apiproxy_stub_map.apiproxy.ReplaceStub('urlfetch', local_stub)



And this split for three scenarios.
First - when secure flag is True in ConfigureRemoteApiForOAuth (by default) 
I get that:
  File "D:\google_appengine\google\appengine\runtime\wsgi.py", line 240, in 
Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "D:\google_appengine\google\appengine\api\lib_config.py", line 354, 
in __getattr__
self._update_configs()
  File "D:\google_appengine\google\appengine\api\lib_config.py", line 290, 
in _update_configs
self._registry.initialize()
  File "D:\google_appengine\google\appengine\api\lib_config.py", line 165, 
in initialize
import_func(self._modname)
  File "D:\storemaven\code\appengine_config.py", line 12, in 
remote_api_stub.ConfigureRemoteApiForOAuth('mockappstore.appspot.com', 
'/_ah/remote_api', True)
  File 
"D:\google_appengine\google\appengine\ext\remote_api\remote_api_stub.py", 
line 769, in ConfigureRemoteApiForOAuth
rpc_server_factory=rpc_server_factory)
  File 
"D:\google_appengine\google\appengine\ext\remote_api\remote_api_stub.py", 
line 839, in ConfigureRemoteApi
app_id = GetRemoteAppIdFromServer(server, path, rtok)
  File 
"D:\google_appengine\google\appengine\ext\remote_api\remote_api_stub.py", 
line 569, in GetRemoteAppIdFromServer
response = server.Send(path, payload=None, **urlargs)
  File 
"D:\google_appengine\google\appengine\tools\appengine_rpc_httplib2.py", 
line 246, in Send
url, method=method, body=payload, headers=headers)
  File "D:\google_appengine\lib\oauth2client\oauth2client\client.py", line 
569, in new_request
redirections, connection_type)
  File "D:\google_appengine\lib\httplib2\httplib2\__init__.py", line 1464, 
in request
self.disable_ssl_certificate_validation)
  File "D:\google_appengine\lib\httplib2\httplib2\__init__.py", line 1143, 
in __init__
strict, timeout, proxy_info, ca_certs, 
disable_ssl_certificate_validation)
  File "D:\google_appengine\lib\httplib2\httplib2\__init__.py", line 1092, 
in __init__
raise NotSupportedOnThisPlatform()
NotSupportedOnThisPlatform

Second - when I'm turning off secrure by passing secure=False I get 
information from some other namespace, how can I pass from which namespace 
I wanat to work with my data?

Third - whem I'm turning off secure and try to reach to my other project 
(id: storemavendev2) I get that(In other projects it works just not on the 
wanted namespace):
Traceback (most recent call last):
  File "D:\google_appengine\google\appengine\runtime\wsgi.py", line 267, in 
Handle
result = handler(dict(self._environ), self._StartResponse)
  File "D:\storemaven\code\com\lib\gaesessions\__init__.py", line 475, in 
__call__
return self.app(environ, my_start_response)
  File "D:\storemaven\code/com/lib\flask\app.py", line 1836, in __call__
return self.wsgi_app(environ, start_response)
  File "D:\storemaven\code/com/lib\flask\app.py", line 1820, in wsgi_app
response = self.make_response(self.handle_exception(e))
  File "D:\storemaven\code/com/lib\flask\app.py", line 1403, in 
handle_exception
reraise(exc_type, exc_value, tb)
  File "D:\storemaven\code/com/lib\flask\app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
  File "D:\storemaven\code/com/lib\flask\app.py", line 1477, in 
full_dispatch_request
rv = self.handle_user_exception(e)
  File "D:\storemaven\code/com/lib\flask\app.py", line 1381, in 
handle_user_exception
reraise(exc_type, exc_value, tb)
  File "D:\storemaven\code/com/lib\flask\app.py", line 1475, in 
full_dispatch_request
rv = self.dispatch_request()
  File "D:\storemaven\code/com/lib\flask\app.py", line 1461, in 
dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
  File "D:\storemaven\code\com\scene53\api\flask\headers.py", line 20, in 
decorated_function
resp = make_response(f(*args, **kwargs))
  File "D:\storemaven\code\com\scene53\api\flask\headers.py", line 62, in 
decorated_function
return f(*args, **kwargs)
  File "D:\storemaven\code\com\scene53\api\flask\console\__init__.py", line 
551, in listUnassignedUsers
return json.jsonify(status="OK", text="", 
payload=userFacade.listUnassignedUsers())
  File "D:\storemaven\code\com\scene53\data\userfacade.py", line 175, in 
listUnassignedUsers
users=self.userService.listUsers(None)
  File "D:\storemaven\code\com\scene53\data\user.py", line 180, in listUsers
for p in iter:
  File 

[google-appengine] Re: upload data via appcfg

['Adam (Cloud Platform Support)' via Google App Engine]

Using the Remote API on dev_appserver.py is essentially the same as using 
it in a local Python client, as per the example in 'Remote API for Python 
'. The most 
straightforward way to turn it on globally for your app in the devserver is 
to add the setup to your 'appengine_config.py 
' 
located in your project's root:

import os
if os.environ['SERVER_SOFTWARE'].startswith('Development'):
  from google.appengine.ext.remote_api import remote_api_stub
  from google.appengine.api import apiproxy_stub_map

  local_stub = apiproxy_stub_map.apiproxy.GetStub('urlfetch')
  remote_api_stub.ConfigureRemoteApiForOAuth('myapp.appspot.com',
'/_ah/remote_api')
  apiproxy_stub_map.apiproxy.ReplaceStub('urlfetch', local_stub)

(Note that replacing the 'urlfetch' stub is necessary to work around a bug, 
if interested you can read about it more here 
 and here 

).

Regarding the 404 with Datastore Admin, if you get this when following the 
link from the Cloud Console this usually means the 
'ah-builtin-python-bundle' version got deleted from your project. The 
simplest way to fix this is to deploy to a new project. If creating a new 
project is not an option, you can file a production issue on the issue 
tracker  and request 
a restore of this bundle.

On Saturday, May 7, 2016 at 7:40:58 AM UTC-4, Rotem Vilkovski wrote:
>
> BTW - in my production I get 404 when I try to reach datastore admin, my 
> project id is: 831908763256
> Thanks again.
>
> On Saturday, May 7, 2016 at 2:30:22 PM UTC+3, Rotem Vilkovski wrote:
>>
>> I've done a restore on my testing project. How can I configure my local 
>> project turn to the remote datastore? I don't want to upload a version each 
>> time I make a little change without check it out locally.
>> Thanks!
>>
>> On Saturday, May 7, 2016 at 12:50:55 PM UTC+3, Rotem Vilkovski wrote:
>>>
>>> Hey Adam, thank you for the comment.
>>> I tried to read some of your documentation about the remote api and it 
>>> hasn't helped me at all.
>>> Is there any example for something so basic? I mean this is the most 
>>> basic move you need to get you local environment to run and I can't find 
>>> anything that you've documented. every documentation just assume you have a 
>>> local datastore that fully functions. 
>>> Lets say I've done a DatastoreBackup, how do I restore it locally?
>>> Thanks!
>>>
>>> On Tuesday, May 3, 2016 at 6:20:33 PM UTC+3, Rotem Vilkovski wrote:

 Hey, 

 I'm trying to upload data to local Datastore using the file I've 
 downloaded from out production using appcfg download_data. 
 I get an error almost every time about ProtocolBufferDecodeError: 
 corrupted. Sometimes (on the same file) it is just working. This is really 
 disturbing and I'd like to know if there's another way to use my local 
 datastore with my local file and having a solution for this problem 
 please. 
 Everytime I update an entity in production and want to update my local 
 Datastore it takes me for 10-20 tries.

 Stack trace:
   File 
 "/home/rotem/Desktop/google_appengine/google/appengine/tools/adaptive_thread_pool.py",
  
 line 172, in WorkOnItems
 status, instruction = item.PerformWork(self.__thread_pool)
   File 
 "/home/rotem/Desktop/google_appengine/google/appengine/tools/bulkloader.py",
  
 line 750, in PerformWork
 transfer_time = self._TransferItem(thread_pool)
   File 
 "/home/rotem/Desktop/google_appengine/google/appengine/tools/bulkloader.py",
  
 line 921, in _TransferItem
 self.request_manager.PostEntities(self.content)
   File 
 "/home/rotem/Desktop/google_appengine/google/appengine/tools/bulkloader.py",
  
 line 1396, in PostEntities
 datastore.Put(entities)
   File 
 "/home/rotem/Desktop/google_appengine/google/appengine/api/datastore.py", 
 line 606, in Put
 return PutAsync(entities, **kwargs).get_result()
   File 
 "/home/rotem/Desktop/google_appengine/google/appengine/datastore/datastore_rpc.py",
  
 line 923, in get_result
 results = self.__rpcs[0].get_result()
   File 
 "/home/rotem/Desktop/google_appengine/google/appengine/api/apiproxy_stub_map.py",
  
 line 613, in get_result
 return self.__get_result_hook(self)
   File 
 "/home/rotem/Desktop/google_appengine/google/appengine/datastore/datastore_rpc.py",
  
 line 1881, in __put_hook
 self.check_rpc_success(rpc)
   File 
 "/home/rotem/Desktop/google_appengine/google/appengine/datastore/datastore_rpc.py",
  
 line 1371, in check_rpc_success
 

[google-appengine] Re: upload data via appcfg

[Rotem Vilkovski]

BTW - in my production I get 404 when I try to reach datastore admin, my 
project id is: 831908763256
Thanks again.

On Saturday, May 7, 2016 at 2:30:22 PM UTC+3, Rotem Vilkovski wrote:
>
> I've done a restore on my testing project. How can I configure my local 
> project turn to the remote datastore? I don't want to upload a version each 
> time I make a little change without check it out locally.
> Thanks!
>
> On Saturday, May 7, 2016 at 12:50:55 PM UTC+3, Rotem Vilkovski wrote:
>>
>> Hey Adam, thank you for the comment.
>> I tried to read some of your documentation about the remote api and it 
>> hasn't helped me at all.
>> Is there any example for something so basic? I mean this is the most 
>> basic move you need to get you local environment to run and I can't find 
>> anything that you've documented. every documentation just assume you have a 
>> local datastore that fully functions. 
>> Lets say I've done a DatastoreBackup, how do I restore it locally?
>> Thanks!
>>
>> On Tuesday, May 3, 2016 at 6:20:33 PM UTC+3, Rotem Vilkovski wrote:
>>>
>>> Hey, 
>>>
>>> I'm trying to upload data to local Datastore using the file I've 
>>> downloaded from out production using appcfg download_data. 
>>> I get an error almost every time about ProtocolBufferDecodeError: 
>>> corrupted. Sometimes (on the same file) it is just working. This is really 
>>> disturbing and I'd like to know if there's another way to use my local 
>>> datastore with my local file and having a solution for this problem please. 
>>> Everytime I update an entity in production and want to update my local 
>>> Datastore it takes me for 10-20 tries.
>>>
>>> Stack trace:
>>>   File 
>>> "/home/rotem/Desktop/google_appengine/google/appengine/tools/adaptive_thread_pool.py",
>>>  
>>> line 172, in WorkOnItems
>>> status, instruction = item.PerformWork(self.__thread_pool)
>>>   File 
>>> "/home/rotem/Desktop/google_appengine/google/appengine/tools/bulkloader.py",
>>>  
>>> line 750, in PerformWork
>>> transfer_time = self._TransferItem(thread_pool)
>>>   File 
>>> "/home/rotem/Desktop/google_appengine/google/appengine/tools/bulkloader.py",
>>>  
>>> line 921, in _TransferItem
>>> self.request_manager.PostEntities(self.content)
>>>   File 
>>> "/home/rotem/Desktop/google_appengine/google/appengine/tools/bulkloader.py",
>>>  
>>> line 1396, in PostEntities
>>> datastore.Put(entities)
>>>   File 
>>> "/home/rotem/Desktop/google_appengine/google/appengine/api/datastore.py", 
>>> line 606, in Put
>>> return PutAsync(entities, **kwargs).get_result()
>>>   File 
>>> "/home/rotem/Desktop/google_appengine/google/appengine/datastore/datastore_rpc.py",
>>>  
>>> line 923, in get_result
>>> results = self.__rpcs[0].get_result()
>>>   File 
>>> "/home/rotem/Desktop/google_appengine/google/appengine/api/apiproxy_stub_map.py",
>>>  
>>> line 613, in get_result
>>> return self.__get_result_hook(self)
>>>   File 
>>> "/home/rotem/Desktop/google_appengine/google/appengine/datastore/datastore_rpc.py",
>>>  
>>> line 1881, in __put_hook
>>> self.check_rpc_success(rpc)
>>>   File 
>>> "/home/rotem/Desktop/google_appengine/google/appengine/datastore/datastore_rpc.py",
>>>  
>>> line 1371, in check_rpc_success
>>> rpc.check_success()
>>>   File 
>>> "/home/rotem/Desktop/google_appengine/google/appengine/api/apiproxy_stub_map.py",
>>>  
>>> line 579, in check_success
>>> self.__rpc.CheckSuccess()
>>>   File 
>>> "/home/rotem/Desktop/google_appengine/google/appengine/api/apiproxy_rpc.py",
>>>  
>>> line 157, in _WaitImpl
>>> self.request, self.response)
>>>   File 
>>> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py",
>>>  
>>> line 286, in MakeSyncCall
>>> handler(request, response)
>>>   File 
>>> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py",
>>>  
>>> line 460, in _Dynamic_Put
>>> 'datastore_v3', 'Put', put_request, put_response)
>>>   File 
>>> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py",
>>>  
>>> line 201, in MakeSyncCall
>>> self._MakeRealSyncCall(service, call, request, response)
>>>   File 
>>> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py",
>>>  
>>> line 228, in _MakeRealSyncCall
>>> response_pb.ParseFromString(encoded_response)
>>>   File 
>>> "/home/rotem/Desktop/google_appengine/google/net/proto/ProtocolBuffer.py", 
>>> line 140, in ParseFromString
>>> self.MergeFromString(s)
>>>   File 
>>> "/home/rotem/Desktop/google_appengine/google/net/proto/ProtocolBuffer.py", 
>>> line 152, in MergeFromString
>>> self.MergePartialFromString(s)
>>>   File 
>>> "/home/rotem/Desktop/google_appengine/google/net/proto/ProtocolBuffer.py", 
>>> line 168, in MergePartialFromString
>>> self.TryMerge(d)
>>>   File 
>>> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_pb.py",
>>>  
>>> line 770, in TryMerge

[google-appengine] Re: upload data via appcfg

[Rotem Vilkovski]

I've done a restore on my testing project. How can I configure my local 
project turn to the remote datastore? I don't want to upload a version each 
time I make a little change without check it out locally.
Thanks!

On Saturday, May 7, 2016 at 12:50:55 PM UTC+3, Rotem Vilkovski wrote:
>
> Hey Adam, thank you for the comment.
> I tried to read some of your documentation about the remote api and it 
> hasn't helped me at all.
> Is there any example for something so basic? I mean this is the most basic 
> move you need to get you local environment to run and I can't find anything 
> that you've documented. every documentation just assume you have a local 
> datastore that fully functions. 
> Lets say I've done a DatastoreBackup, how do I restore it locally?
> Thanks!
>
> On Tuesday, May 3, 2016 at 6:20:33 PM UTC+3, Rotem Vilkovski wrote:
>>
>> Hey, 
>>
>> I'm trying to upload data to local Datastore using the file I've 
>> downloaded from out production using appcfg download_data. 
>> I get an error almost every time about ProtocolBufferDecodeError: 
>> corrupted. Sometimes (on the same file) it is just working. This is really 
>> disturbing and I'd like to know if there's another way to use my local 
>> datastore with my local file and having a solution for this problem please. 
>> Everytime I update an entity in production and want to update my local 
>> Datastore it takes me for 10-20 tries.
>>
>> Stack trace:
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/tools/adaptive_thread_pool.py",
>>  
>> line 172, in WorkOnItems
>> status, instruction = item.PerformWork(self.__thread_pool)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/tools/bulkloader.py", 
>> line 750, in PerformWork
>> transfer_time = self._TransferItem(thread_pool)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/tools/bulkloader.py", 
>> line 921, in _TransferItem
>> self.request_manager.PostEntities(self.content)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/tools/bulkloader.py", 
>> line 1396, in PostEntities
>> datastore.Put(entities)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/api/datastore.py", 
>> line 606, in Put
>> return PutAsync(entities, **kwargs).get_result()
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/datastore/datastore_rpc.py",
>>  
>> line 923, in get_result
>> results = self.__rpcs[0].get_result()
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/api/apiproxy_stub_map.py",
>>  
>> line 613, in get_result
>> return self.__get_result_hook(self)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/datastore/datastore_rpc.py",
>>  
>> line 1881, in __put_hook
>> self.check_rpc_success(rpc)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/datastore/datastore_rpc.py",
>>  
>> line 1371, in check_rpc_success
>> rpc.check_success()
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/api/apiproxy_stub_map.py",
>>  
>> line 579, in check_success
>> self.__rpc.CheckSuccess()
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/api/apiproxy_rpc.py", 
>> line 157, in _WaitImpl
>> self.request, self.response)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py",
>>  
>> line 286, in MakeSyncCall
>> handler(request, response)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py",
>>  
>> line 460, in _Dynamic_Put
>> 'datastore_v3', 'Put', put_request, put_response)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py",
>>  
>> line 201, in MakeSyncCall
>> self._MakeRealSyncCall(service, call, request, response)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py",
>>  
>> line 228, in _MakeRealSyncCall
>> response_pb.ParseFromString(encoded_response)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/net/proto/ProtocolBuffer.py", 
>> line 140, in ParseFromString
>> self.MergeFromString(s)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/net/proto/ProtocolBuffer.py", 
>> line 152, in MergeFromString
>> self.MergePartialFromString(s)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/net/proto/ProtocolBuffer.py", 
>> line 168, in MergePartialFromString
>> self.TryMerge(d)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_pb.py",
>>  
>> line 770, in TryMerge
>> d.skipData(tt)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/net/proto/ProtocolBuffer.py", 
>> line 673, in skipData
>> raise ProtocolBufferDecodeError, "corrupted"
>> ProtocolBufferDecodeError: corrupted
>>
>>
>> Thanks!
>>
>

-- 
You received this message because you are subscribed to the Google Groups 

[google-appengine] Re: upload data via appcfg

[Rotem Vilkovski]

Hey Adam, thank you for the comment.
I tried to read some of your documentation about the remote api and it 
hasn't helped me at all.
Is there any example for something so basic? I mean this is the most basic 
move you need to get you local environment to run and I can't find anything 
that you've documented. every documentation just assume you have a local 
datastore that fully functions. 
Lets say I've done a DatastoreBackup, how do I restore it locally?
Thanks!

On Tuesday, May 3, 2016 at 6:20:33 PM UTC+3, Rotem Vilkovski wrote:
>
> Hey, 
>
> I'm trying to upload data to local Datastore using the file I've 
> downloaded from out production using appcfg download_data. 
> I get an error almost every time about ProtocolBufferDecodeError: 
> corrupted. Sometimes (on the same file) it is just working. This is really 
> disturbing and I'd like to know if there's another way to use my local 
> datastore with my local file and having a solution for this problem please. 
> Everytime I update an entity in production and want to update my local 
> Datastore it takes me for 10-20 tries.
>
> Stack trace:
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/tools/adaptive_thread_pool.py",
>  
> line 172, in WorkOnItems
> status, instruction = item.PerformWork(self.__thread_pool)
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/tools/bulkloader.py", 
> line 750, in PerformWork
> transfer_time = self._TransferItem(thread_pool)
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/tools/bulkloader.py", 
> line 921, in _TransferItem
> self.request_manager.PostEntities(self.content)
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/tools/bulkloader.py", 
> line 1396, in PostEntities
> datastore.Put(entities)
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/api/datastore.py", 
> line 606, in Put
> return PutAsync(entities, **kwargs).get_result()
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/datastore/datastore_rpc.py",
>  
> line 923, in get_result
> results = self.__rpcs[0].get_result()
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/api/apiproxy_stub_map.py",
>  
> line 613, in get_result
> return self.__get_result_hook(self)
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/datastore/datastore_rpc.py",
>  
> line 1881, in __put_hook
> self.check_rpc_success(rpc)
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/datastore/datastore_rpc.py",
>  
> line 1371, in check_rpc_success
> rpc.check_success()
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/api/apiproxy_stub_map.py",
>  
> line 579, in check_success
> self.__rpc.CheckSuccess()
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/api/apiproxy_rpc.py", 
> line 157, in _WaitImpl
> self.request, self.response)
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py",
>  
> line 286, in MakeSyncCall
> handler(request, response)
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py",
>  
> line 460, in _Dynamic_Put
> 'datastore_v3', 'Put', put_request, put_response)
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py",
>  
> line 201, in MakeSyncCall
> self._MakeRealSyncCall(service, call, request, response)
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py",
>  
> line 228, in _MakeRealSyncCall
> response_pb.ParseFromString(encoded_response)
>   File 
> "/home/rotem/Desktop/google_appengine/google/net/proto/ProtocolBuffer.py", 
> line 140, in ParseFromString
> self.MergeFromString(s)
>   File 
> "/home/rotem/Desktop/google_appengine/google/net/proto/ProtocolBuffer.py", 
> line 152, in MergeFromString
> self.MergePartialFromString(s)
>   File 
> "/home/rotem/Desktop/google_appengine/google/net/proto/ProtocolBuffer.py", 
> line 168, in MergePartialFromString
> self.TryMerge(d)
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_pb.py",
>  
> line 770, in TryMerge
> d.skipData(tt)
>   File 
> "/home/rotem/Desktop/google_appengine/google/net/proto/ProtocolBuffer.py", 
> line 673, in skipData
> raise ProtocolBufferDecodeError, "corrupted"
> ProtocolBufferDecodeError: corrupted
>
>
> Thanks!
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 

[google-appengine] Re: upload data via appcfg

['Adam (Cloud Platform Support)' via Google App Engine]

Regarding the 'ProtocolBufferDecodeError: corrupted' error, this can happen 
when using a custom URL for Remote API which isn't mapped correctly. If you 
would like to post your app.yaml with the sensitive bits redacted I can 
take a look.

On Friday, May 6, 2016 at 2:44:01 PM UTC-4, Adam (Cloud Platform Support) 
wrote:
>
> Rather than manually trying to resync the production data locally every 
> time an update is required, you could use the Remote API 
>  to have 
> your development server app talk to the remote Datastore, and have both 
> your local and deployed app use the same data source.
>
> I don't recommend doing this for "real" production data - you should 
> create a new project used specifically for testing / development. You can 
> use the Datastore Backup 
> 
>  
> to backup any data you need from prod and restore it to the test project.
>
> On Tuesday, May 3, 2016 at 11:20:33 AM UTC-4, Rotem Vilkovski wrote:
>>
>> Hey, 
>>
>> I'm trying to upload data to local Datastore using the file I've 
>> downloaded from out production using appcfg download_data. 
>> I get an error almost every time about ProtocolBufferDecodeError: 
>> corrupted. Sometimes (on the same file) it is just working. This is really 
>> disturbing and I'd like to know if there's another way to use my local 
>> datastore with my local file and having a solution for this problem please. 
>> Everytime I update an entity in production and want to update my local 
>> Datastore it takes me for 10-20 tries.
>>
>> Stack trace:
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/tools/adaptive_thread_pool.py",
>>  
>> line 172, in WorkOnItems
>> status, instruction = item.PerformWork(self.__thread_pool)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/tools/bulkloader.py", 
>> line 750, in PerformWork
>> transfer_time = self._TransferItem(thread_pool)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/tools/bulkloader.py", 
>> line 921, in _TransferItem
>> self.request_manager.PostEntities(self.content)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/tools/bulkloader.py", 
>> line 1396, in PostEntities
>> datastore.Put(entities)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/api/datastore.py", 
>> line 606, in Put
>> return PutAsync(entities, **kwargs).get_result()
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/datastore/datastore_rpc.py",
>>  
>> line 923, in get_result
>> results = self.__rpcs[0].get_result()
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/api/apiproxy_stub_map.py",
>>  
>> line 613, in get_result
>> return self.__get_result_hook(self)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/datastore/datastore_rpc.py",
>>  
>> line 1881, in __put_hook
>> self.check_rpc_success(rpc)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/datastore/datastore_rpc.py",
>>  
>> line 1371, in check_rpc_success
>> rpc.check_success()
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/api/apiproxy_stub_map.py",
>>  
>> line 579, in check_success
>> self.__rpc.CheckSuccess()
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/api/apiproxy_rpc.py", 
>> line 157, in _WaitImpl
>> self.request, self.response)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py",
>>  
>> line 286, in MakeSyncCall
>> handler(request, response)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py",
>>  
>> line 460, in _Dynamic_Put
>> 'datastore_v3', 'Put', put_request, put_response)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py",
>>  
>> line 201, in MakeSyncCall
>> self._MakeRealSyncCall(service, call, request, response)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py",
>>  
>> line 228, in _MakeRealSyncCall
>> response_pb.ParseFromString(encoded_response)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/net/proto/ProtocolBuffer.py", 
>> line 140, in ParseFromString
>> self.MergeFromString(s)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/net/proto/ProtocolBuffer.py", 
>> line 152, in MergeFromString
>> self.MergePartialFromString(s)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/net/proto/ProtocolBuffer.py", 
>> line 168, in MergePartialFromString
>> self.TryMerge(d)
>>   File 
>> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_pb.py",
>>  
>> line 770, in TryMerge
>> d.skipData(tt)
>>   File 
>> 

[google-appengine] Re: upload data via appcfg

['Adam (Cloud Platform Support)' via Google App Engine]

Rather than manually trying to resync the production data locally every 
time an update is required, you could use the Remote API 
 to have 
your development server app talk to the remote Datastore, and have both 
your local and deployed app use the same data source.

I don't recommend doing this for "real" production data - you should create 
a new project used specifically for testing / development. You can use the 
Datastore 
Backup 

 
to backup any data you need from prod and restore it to the test project.

On Tuesday, May 3, 2016 at 11:20:33 AM UTC-4, Rotem Vilkovski wrote:
>
> Hey, 
>
> I'm trying to upload data to local Datastore using the file I've 
> downloaded from out production using appcfg download_data. 
> I get an error almost every time about ProtocolBufferDecodeError: 
> corrupted. Sometimes (on the same file) it is just working. This is really 
> disturbing and I'd like to know if there's another way to use my local 
> datastore with my local file and having a solution for this problem please. 
> Everytime I update an entity in production and want to update my local 
> Datastore it takes me for 10-20 tries.
>
> Stack trace:
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/tools/adaptive_thread_pool.py",
>  
> line 172, in WorkOnItems
> status, instruction = item.PerformWork(self.__thread_pool)
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/tools/bulkloader.py", 
> line 750, in PerformWork
> transfer_time = self._TransferItem(thread_pool)
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/tools/bulkloader.py", 
> line 921, in _TransferItem
> self.request_manager.PostEntities(self.content)
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/tools/bulkloader.py", 
> line 1396, in PostEntities
> datastore.Put(entities)
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/api/datastore.py", 
> line 606, in Put
> return PutAsync(entities, **kwargs).get_result()
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/datastore/datastore_rpc.py",
>  
> line 923, in get_result
> results = self.__rpcs[0].get_result()
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/api/apiproxy_stub_map.py",
>  
> line 613, in get_result
> return self.__get_result_hook(self)
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/datastore/datastore_rpc.py",
>  
> line 1881, in __put_hook
> self.check_rpc_success(rpc)
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/datastore/datastore_rpc.py",
>  
> line 1371, in check_rpc_success
> rpc.check_success()
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/api/apiproxy_stub_map.py",
>  
> line 579, in check_success
> self.__rpc.CheckSuccess()
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/api/apiproxy_rpc.py", 
> line 157, in _WaitImpl
> self.request, self.response)
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py",
>  
> line 286, in MakeSyncCall
> handler(request, response)
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py",
>  
> line 460, in _Dynamic_Put
> 'datastore_v3', 'Put', put_request, put_response)
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py",
>  
> line 201, in MakeSyncCall
> self._MakeRealSyncCall(service, call, request, response)
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py",
>  
> line 228, in _MakeRealSyncCall
> response_pb.ParseFromString(encoded_response)
>   File 
> "/home/rotem/Desktop/google_appengine/google/net/proto/ProtocolBuffer.py", 
> line 140, in ParseFromString
> self.MergeFromString(s)
>   File 
> "/home/rotem/Desktop/google_appengine/google/net/proto/ProtocolBuffer.py", 
> line 152, in MergeFromString
> self.MergePartialFromString(s)
>   File 
> "/home/rotem/Desktop/google_appengine/google/net/proto/ProtocolBuffer.py", 
> line 168, in MergePartialFromString
> self.TryMerge(d)
>   File 
> "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_pb.py",
>  
> line 770, in TryMerge
> d.skipData(tt)
>   File 
> "/home/rotem/Desktop/google_appengine/google/net/proto/ProtocolBuffer.py", 
> line 673, in skipData
> raise ProtocolBufferDecodeError, "corrupted"
> ProtocolBufferDecodeError: corrupted
>
>
> Thanks!
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to