I was trying to move a MySQL based app to the GAE. The MySQL was built by inserting values into the SQL table with a python script, so it made since (to me) to do the same on the GAE. I'm guessing the easiest way to avoid this would have been to download a CSV of my data and use the appadmin to upload it again, however the larger of the two tables returns 'can't parse file' when I tried that. So I was trying to upload the data with a script.
However, it seems the error I'm seeing here might actually just be related to the appadmin displaying very large tables. I tried doing a serach for just one of my uploaded entries and it's actually there I just have to put in the URL to the appadmin manually b/c the default is to display all values. I'm uploading a version of my code now that I think will work since the data appears to be there and it's just an appadmin problem with the display. As for removing the duplicates it's not a big deal but anyone have a suggestion as to how I go about that? With the 1000 entity limit on selects I'm trying to think now how to best find only duplicates. Unfortunately I didn't put any 'time' variables in the table. I'm open to suggestions. On Sep 19, 8:02 pm, Robin B <robi...@gmail.com> wrote: > On Sep 19, 6:13 pm, Chris S <sanders.ch...@gmail.com> wrote: > > > From what I've read enabling billing doesn't increase the CPU time > > just some of the other quotas. It's CPU time that gets eaten up so > > badly when you remove/add items to the datastore and I'm not keen on > > paying for extra CPU time until I at least have some sort of working > > app to expand on. > > It is true that enabling billing does not increase the total free > monthly CPU, it does however increase the max rate you can consume CPU > by 4.8X, so you won't hit your over quota so quickly in the short > term. > > http://code.google.com/appengine/docs/quotas.html#Resources > > > > > As far as the duplicate entries, I removed the "ID" field before > > uploading since I knew the datastore would assign a different ID > > number than a SQL based web2py table. Is that the reason re-uploading > > data didn't overwrite old data? I would have preferred it to have > > over written it, but with out knowing the unique identifier that > > didn't happen. I'm sure I could use the remote_api_shell to recurse > > through the database and remove duplicates it's just that I don't see > > how that could be the problem. > > Removing the IDs certainly created the duplicate entries, and if you > had references between entries, the data is probably invalid. > > > > > I've tried uploading this same data to the development (local) copy of > > the GAE datastore and the table finds the entities as expected. So > > where do I go now since that's not the same thing I'm seeing on the > > production Datastore? Do I need to delete all enteries and re-upload > > all of the data again? I sure hope not. > > I would try to remove the table entries. If you can't remove the > entries, you might instead create a brand new table with a different > name somehow. > > > > > Thanks for pointing out where I can find info on what the web2py > > fields translate to in datastore terms. That will help me insure I > > don't have an issue with that in the future. > > What were you trying to do? Backup/Restore? Schema migration? > > Robin > > > On Sep 19, 5:09 pm, Robin B <robi...@gmail.com> wrote: > > > > > Looking over two tables here are the things I've identified as > > > > possible problems: > > > > 1 - This table has a 'double' in it. Not just int/string like the > > > > working one. For GAE uploading I defined the double field as a GAE > > > > "float" property. Is it possible web2py defines a 'double' as > > > > something else on GAE? If so is there a mapping that would tell me > > > > how to map web2py fields into GAE fields for uploading data? > > > > On GAE, 'double' is a float looking at the top of gql.py. > > > > > 2 - Searching the Data Viewer on GAE shows that due to a previously > > > > failed data upload I now have multiple copies of some of the > > > > entities. These copies the exact same data in the except for the ID/ > > > > Name fields. I would like to remove the duplicates sometime but that > > > > wasn't my first concern. Would duplicate copies cause this problem? > > > > I figured since the ID is unique it would just show up as another > > > > entery in the table. I haven't set any of the entries to 'unique'. > > > > It depends on your app, but you probably want to remove any duplicates > > > before reloading the data. I read that the bulk loader will over- > > > write records using the IDs given in the .csv, which is what you want. > > > > > 3 - Both tables are set with migrate=False, specifically b/c I've > > > > seperated the data upload and process from the data reading process. > > > > I'm not even sure what web2py would do if I removed the migrate=false > > > > flag. What is the recommendation here? > > > > AFAIK, migrate should have no effect on GAE. > > > > > Inserting/deleting items from the GAE Datastore is very expensive in > > > > CPU time, I messed with this yesterday and in a matter of a few > > > > minutes (delete all, re-upload) was over my quota. Because of that > > > > I'd really like to avoid having to change data in the Datastore as > > > > much as I can. With 20K+ items you don't get a lot of 'do-overs'. > > > > Have you enabled billing on your account? Enabiling billing greatly > > > increases your free quota. > > > > Robin > > > > On Sep 19, 2:45 pm, Chris S <sanders.ch...@gmail.com> wrote: > > > > > Background: > > > > Working with an app on GAE currently I've run across a slight issue > > > > with uploading data into my app. I initially tried to download my > > > > tables to CSV from my development machine then use simply re-upload > > > > them into GAE. It's been a while since I tried that but it was timing > > > > out. > > > > > Problem: > > > > Because of that I decided to upload with bulkloader.py. I did my > > > > first table (~2K entries) with out a hitch. My second Table is where > > > > I'm having an issue. This table is about 24K entities. After > > > > uploading all of the data trying to view it on the appadmin returns > > > > the following error (web2py error ticket): > > > > The table is "sqldb.item" if it matters. > > > > ------------- > > > > Traceback (most recent call last): > > > > File "/base/data/home/apps/aion-info/2.336452968225099838/gluon/ > > > > main.py", line 396, in wsgibase > > > > session._try_store_in_db(request, response) > > > > File "/base/data/home/apps/aion-info/2.336452968225099838/gluon/ > > > > globals.py", line 342, in _try_store_in_db > > > > table._db(table.id == record_id).update(**dd) > > > > File "/base/data/home/apps/aion-info/2.336452968225099838/gluon/ > > > > contrib/gql.py", line 698, in update > > > > (item, tablename, fields) = self._getitem_exception() > > > > File "/base/data/home/apps/aion-info/2.336452968225099838/gluon/ > > > > contrib/gql.py", line 632, in _getitem_exception > > > > item = self._db[tablename]._tableobj.get_by_id(long(id)) > > > > File "/base/python_lib/versions/1/google/appengine/ext/db/ > > > > __init__.py", line 943, in get_by_id > > > > return get(*keys) > > > > File "/base/python_lib/versions/1/google/appengine/ext/db/ > > > > __init__.py", line 1123, in get > > > > entities = datastore.Get(keys) > > > > File "/base/python_lib/versions/1/google/appengine/api/ > > > > datastore.py", line 224, in Get > > > > apiproxy_stub_map.MakeSyncCall('datastore_v3', 'Get', req, resp) > > > > File "/base/python_lib/versions/1/google/appengine/api/ > > > > apiproxy_stub_map.py", line 72, in MakeSyncCall > > > > apiproxy.MakeSyncCall(service, call, request, response) > > > > File "/base/python_lib/versions/1/google/appengine/api/ > > > > apiproxy_stub_map.py", line 255, in MakeSyncCall > > > > rpc.CheckSuccess() > > > > File "/base/python_lib/versions/1/google/appengine/api/ > > > > apiproxy_rpc.py", line 111, in CheckSuccess > > > > raise self.exception > > > > CancelledError: The API call datastore_v3.Get() was explicitly > > > > cancelled. > > > > ----------------------------- > > > > > Looking over two tables here are the things I've identified as > > > > possible problems: > > > > 1 - This table has a 'double' in it. Not just int/string like the > > > > working one. For GAE uploading I defined the double field as a GAE > > > > "float" property. Is it possible web2py defines a 'double' as > > > > something else on GAE? If so is there a mapping that would tell me > > > > how to map web2py fields into GAE fields for uploading data? > > > > > 2 - Searching the Data Viewer on GAE shows that due to a previously > > > > failed data upload I now have multiple copies of some of the > > > > entities. These copies the exact same data in the except for the ID/ > > > > Name fields. I would like to remove the duplicates sometime but that > > > > wasn't my first concern. Would duplicate copies cause this problem? > > > > I figured since the ID is unique it would just show up as another > > > > entery in the table. I haven't set any of the entries to 'unique'. > > > > > 3 - Both tables are set with migrate=False, specifically b/c I've > > > > seperated the data upload and process from the data reading process. > > > > I'm not even sure what web2py would do if I removed the migrate=false > > > > flag. What is the recommendation here? > > > > > Inserting/deleting items from the GAE Datastore is very expensive in > > > > CPU time, I messed with this yesterday and in a matter of a few > > > > minutes (delete all, re-upload) was over my quota. Because of that > > > > I'd really like to avoid having to change data in the Datastore as > > > > much as I can. With 20K+ items you don't get a lot of 'do-overs'. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~----------~----~----~----~------~----~------~--~---
