On Sep 19, 6:13 pm, Chris S <sanders.ch...@gmail.com> wrote:
> From what I've read enabling billing doesn't increase the CPU time
> just some of the other quotas. It's CPU time that gets eaten up so
> badly when you remove/add items to the datastore and I'm not keen on
> paying for extra CPU time until I at least have some sort of working
> app to expand on.
It is true that enabling billing does not increase the total free
monthly CPU, it does however increase the max rate you can consume CPU
by 4.8X, so you won't hit your over quota so quickly in the short
term.
http://code.google.com/appengine/docs/quotas.html#Resources
>
> As far as the duplicate entries, I removed the "ID" field before
> uploading since I knew the datastore would assign a different ID
> number than a SQL based web2py table. Is that the reason re-uploading
> data didn't overwrite old data? I would have preferred it to have
> over written it, but with out knowing the unique identifier that
> didn't happen. I'm sure I could use the remote_api_shell to recurse
> through the database and remove duplicates it's just that I don't see
> how that could be the problem.
Removing the IDs certainly created the duplicate entries, and if you
had references between entries, the data is probably invalid.
>
> I've tried uploading this same data to the development (local) copy of
> the GAE datastore and the table finds the entities as expected. So
> where do I go now since that's not the same thing I'm seeing on the
> production Datastore? Do I need to delete all enteries and re-upload
> all of the data again? I sure hope not.
I would try to remove the table entries. If you can't remove the
entries, you might instead create a brand new table with a different
name somehow.
>
> Thanks for pointing out where I can find info on what the web2py
> fields translate to in datastore terms. That will help me insure I
> don't have an issue with that in the future.
>
What were you trying to do? Backup/Restore? Schema migration?
Robin
> On Sep 19, 5:09 pm, Robin B <robi...@gmail.com> wrote:
>
> > > Looking over two tables here are the things I've identified as
> > > possible problems:
> > > 1 - This table has a 'double' in it. Not just int/string like the
> > > working one. For GAE uploading I defined the double field as a GAE
> > > "float" property. Is it possible web2py defines a 'double' as
> > > something else on GAE? If so is there a mapping that would tell me
> > > how to map web2py fields into GAE fields for uploading data?
>
> > On GAE, 'double' is a float looking at the top of gql.py.
>
> > > 2 - Searching the Data Viewer on GAE shows that due to a previously
> > > failed data upload I now have multiple copies of some of the
> > > entities. These copies the exact same data in the except for the ID/
> > > Name fields. I would like to remove the duplicates sometime but that
> > > wasn't my first concern. Would duplicate copies cause this problem?
> > > I figured since the ID is unique it would just show up as another
> > > entery in the table. I haven't set any of the entries to 'unique'.
>
> > It depends on your app, but you probably want to remove any duplicates
> > before reloading the data. I read that the bulk loader will over-
> > write records using the IDs given in the .csv, which is what you want.
>
> > > 3 - Both tables are set with migrate=False, specifically b/c I've
> > > seperated the data upload and process from the data reading process.
> > > I'm not even sure what web2py would do if I removed the migrate=false
> > > flag. What is the recommendation here?
>
> > AFAIK, migrate should have no effect on GAE.
>
> > > Inserting/deleting items from the GAE Datastore is very expensive in
> > > CPU time, I messed with this yesterday and in a matter of a few
> > > minutes (delete all, re-upload) was over my quota. Because of that
> > > I'd really like to avoid having to change data in the Datastore as
> > > much as I can. With 20K+ items you don't get a lot of 'do-overs'.
>
> > Have you enabled billing on your account? Enabiling billing greatly
> > increases your free quota.
>
> > Robin
>
> > On Sep 19, 2:45 pm, Chris S <sanders.ch...@gmail.com> wrote:
>
> > > Background:
> > > Working with an app on GAE currently I've run across a slight issue
> > > with uploading data into my app. I initially tried to download my
> > > tables to CSV from my development machine then use simply re-upload
> > > them into GAE. It's been a while since I tried that but it was timing
> > > out.
>
> > > Problem:
> > > Because of that I decided to upload with bulkloader.py. I did my
> > > first table (~2K entries) with out a hitch. My second Table is where
> > > I'm having an issue. This table is about 24K entities. After
> > > uploading all of the data trying to view it on the appadmin returns
> > > the following error (web2py error ticket):
> > > The table is "sqldb.item" if it matters.
> > > -------------
> > > Traceback (most recent call last):
> > > File "/base/data/home/apps/aion-info/2.336452968225099838/gluon/
> > > main.py", line 396, in wsgibase
> > > session._try_store_in_db(request, response)
> > > File "/base/data/home/apps/aion-info/2.336452968225099838/gluon/
> > > globals.py", line 342, in _try_store_in_db
> > > table._db(table.id == record_id).update(**dd)
> > > File "/base/data/home/apps/aion-info/2.336452968225099838/gluon/
> > > contrib/gql.py", line 698, in update
> > > (item, tablename, fields) = self._getitem_exception()
> > > File "/base/data/home/apps/aion-info/2.336452968225099838/gluon/
> > > contrib/gql.py", line 632, in _getitem_exception
> > > item = self._db[tablename]._tableobj.get_by_id(long(id))
> > > File "/base/python_lib/versions/1/google/appengine/ext/db/
> > > __init__.py", line 943, in get_by_id
> > > return get(*keys)
> > > File "/base/python_lib/versions/1/google/appengine/ext/db/
> > > __init__.py", line 1123, in get
> > > entities = datastore.Get(keys)
> > > File "/base/python_lib/versions/1/google/appengine/api/
> > > datastore.py", line 224, in Get
> > > apiproxy_stub_map.MakeSyncCall('datastore_v3', 'Get', req, resp)
> > > File "/base/python_lib/versions/1/google/appengine/api/
> > > apiproxy_stub_map.py", line 72, in MakeSyncCall
> > > apiproxy.MakeSyncCall(service, call, request, response)
> > > File "/base/python_lib/versions/1/google/appengine/api/
> > > apiproxy_stub_map.py", line 255, in MakeSyncCall
> > > rpc.CheckSuccess()
> > > File "/base/python_lib/versions/1/google/appengine/api/
> > > apiproxy_rpc.py", line 111, in CheckSuccess
> > > raise self.exception
> > > CancelledError: The API call datastore_v3.Get() was explicitly
> > > cancelled.
> > > -----------------------------
>
> > > Looking over two tables here are the things I've identified as
> > > possible problems:
> > > 1 - This table has a 'double' in it. Not just int/string like the
> > > working one. For GAE uploading I defined the double field as a GAE
> > > "float" property. Is it possible web2py defines a 'double' as
> > > something else on GAE? If so is there a mapping that would tell me
> > > how to map web2py fields into GAE fields for uploading data?
>
> > > 2 - Searching the Data Viewer on GAE shows that due to a previously
> > > failed data upload I now have multiple copies of some of the
> > > entities. These copies the exact same data in the except for the ID/
> > > Name fields. I would like to remove the duplicates sometime but that
> > > wasn't my first concern. Would duplicate copies cause this problem?
> > > I figured since the ID is unique it would just show up as another
> > > entery in the table. I haven't set any of the entries to 'unique'.
>
> > > 3 - Both tables are set with migrate=False, specifically b/c I've
> > > seperated the data upload and process from the data reading process.
> > > I'm not even sure what web2py would do if I removed the migrate=false
> > > flag. What is the recommendation here?
>
> > > Inserting/deleting items from the GAE Datastore is very expensive in
> > > CPU time, I messed with this yesterday and in a matter of a few
> > > minutes (delete all, re-upload) was over my quota. Because of that
> > > I'd really like to avoid having to change data in the Datastore as
> > > much as I can. With 20K+ items you don't get a lot of 'do-overs'.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~----------~----~----~----~------~----~------~--~---
