> Looking over two tables here are the things I've identified as
> possible problems:
> 1 - This table has a 'double' in it. Not just int/string like the
> working one. For GAE uploading I defined the double field as a GAE
> "float" property. Is it possible web2py defines a 'double' as
> something else on GAE? If so is there a mapping that would tell me
> how to map web2py fields into GAE fields for uploading data?
On GAE, 'double' is a float looking at the top of gql.py.
>
> 2 - Searching the Data Viewer on GAE shows that due to a previously
> failed data upload I now have multiple copies of some of the
> entities. These copies the exact same data in the except for the ID/
> Name fields. I would like to remove the duplicates sometime but that
> wasn't my first concern. Would duplicate copies cause this problem?
> I figured since the ID is unique it would just show up as another
> entery in the table. I haven't set any of the entries to 'unique'.
It depends on your app, but you probably want to remove any duplicates
before reloading the data. I read that the bulk loader will over-
write records using the IDs given in the .csv, which is what you want.
>
> 3 - Both tables are set with migrate=False, specifically b/c I've
> seperated the data upload and process from the data reading process.
> I'm not even sure what web2py would do if I removed the migrate=false
> flag. What is the recommendation here?
AFAIK, migrate should have no effect on GAE.
>
> Inserting/deleting items from the GAE Datastore is very expensive in
> CPU time, I messed with this yesterday and in a matter of a few
> minutes (delete all, re-upload) was over my quota. Because of that
> I'd really like to avoid having to change data in the Datastore as
> much as I can. With 20K+ items you don't get a lot of 'do-overs'.
Have you enabled billing on your account? Enabiling billing greatly
increases your free quota.
Robin
On Sep 19, 2:45 pm, Chris S <sanders.ch...@gmail.com> wrote:
> Background:
> Working with an app on GAE currently I've run across a slight issue
> with uploading data into my app. I initially tried to download my
> tables to CSV from my development machine then use simply re-upload
> them into GAE. It's been a while since I tried that but it was timing
> out.
>
> Problem:
> Because of that I decided to upload with bulkloader.py. I did my
> first table (~2K entries) with out a hitch. My second Table is where
> I'm having an issue. This table is about 24K entities. After
> uploading all of the data trying to view it on the appadmin returns
> the following error (web2py error ticket):
> The table is "sqldb.item" if it matters.
> -------------
> Traceback (most recent call last):
> File "/base/data/home/apps/aion-info/2.336452968225099838/gluon/
> main.py", line 396, in wsgibase
> session._try_store_in_db(request, response)
> File "/base/data/home/apps/aion-info/2.336452968225099838/gluon/
> globals.py", line 342, in _try_store_in_db
> table._db(table.id == record_id).update(**dd)
> File "/base/data/home/apps/aion-info/2.336452968225099838/gluon/
> contrib/gql.py", line 698, in update
> (item, tablename, fields) = self._getitem_exception()
> File "/base/data/home/apps/aion-info/2.336452968225099838/gluon/
> contrib/gql.py", line 632, in _getitem_exception
> item = self._db[tablename]._tableobj.get_by_id(long(id))
> File "/base/python_lib/versions/1/google/appengine/ext/db/
> __init__.py", line 943, in get_by_id
> return get(*keys)
> File "/base/python_lib/versions/1/google/appengine/ext/db/
> __init__.py", line 1123, in get
> entities = datastore.Get(keys)
> File "/base/python_lib/versions/1/google/appengine/api/
> datastore.py", line 224, in Get
> apiproxy_stub_map.MakeSyncCall('datastore_v3', 'Get', req, resp)
> File "/base/python_lib/versions/1/google/appengine/api/
> apiproxy_stub_map.py", line 72, in MakeSyncCall
> apiproxy.MakeSyncCall(service, call, request, response)
> File "/base/python_lib/versions/1/google/appengine/api/
> apiproxy_stub_map.py", line 255, in MakeSyncCall
> rpc.CheckSuccess()
> File "/base/python_lib/versions/1/google/appengine/api/
> apiproxy_rpc.py", line 111, in CheckSuccess
> raise self.exception
> CancelledError: The API call datastore_v3.Get() was explicitly
> cancelled.
> -----------------------------
>
> Looking over two tables here are the things I've identified as
> possible problems:
> 1 - This table has a 'double' in it. Not just int/string like the
> working one. For GAE uploading I defined the double field as a GAE
> "float" property. Is it possible web2py defines a 'double' as
> something else on GAE? If so is there a mapping that would tell me
> how to map web2py fields into GAE fields for uploading data?
>
> 2 - Searching the Data Viewer on GAE shows that due to a previously
> failed data upload I now have multiple copies of some of the
> entities. These copies the exact same data in the except for the ID/
> Name fields. I would like to remove the duplicates sometime but that
> wasn't my first concern. Would duplicate copies cause this problem?
> I figured since the ID is unique it would just show up as another
> entery in the table. I haven't set any of the entries to 'unique'.
>
> 3 - Both tables are set with migrate=False, specifically b/c I've
> seperated the data upload and process from the data reading process.
> I'm not even sure what web2py would do if I removed the migrate=false
> flag. What is the recommendation here?
>
> Inserting/deleting items from the GAE Datastore is very expensive in
> CPU time, I messed with this yesterday and in a matter of a few
> minutes (delete all, re-upload) was over my quota. Because of that
> I'd really like to avoid having to change data in the Datastore as
> much as I can. With 20K+ items you don't get a lot of 'do-overs'.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~----------~----~----~----~------~----~------~--~---
