Here's a blog post my teammate Wesley (also on this team) wrote:
http://wesc.livejournal.com/1743.html
Sounds a bit like what is going on with your script.
On Fri, Dec 11, 2009 at 6:46 PM, Dominik Steiner <
dominik.j.stei...@googlemail.com> wrote:
> Thanks again Ikai for your help and fast reply.
>
> It might not help that I'm a Python newbie here, but still not able to get
> this going with the code you sent me, still the error it gives me is
>
> File "exporter.py", line 26, in decodeIfNotNone
> return utf8_string.decode("utf-8")
> File
> "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/encodings/utf_8.py",
> line 16, in decode
> return codecs.utf_8_decode(input, errors, True)
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in
> position 7: ordinal not in range(128)
>
> My product exporter looks like this
>
> def decodeIfNotNone(utf8_string):
> if utf8_string:
> return utf8_string.decode("utf-8")
> else:
> return ""
>
> class ProductExporter(bulkloader.Exporter):
> def __init__(self):
> bulkloader.Exporter.__init__(self, 'Product',
> [('name', decodeIfNotNone, None),
> ('userId', str, None),
> ('shop', decodeIfNotNone, None),
> ('brand', decodeIfNotNone, None),
> ('model', decodeIfNotNone, None),
> ('contact', decodeIfNotNone, None),
> ('imageId', str, None),
> ('userNickName', decodeIfNotNone,
> None),
> ('price', str, None),
> ('registeredDate', str, None),
> ('validUntilDate', str, None),
> ('ratingCount', str, None),
> ('ratingTotal', str, None),
> ('rating', str, None),
> ('commentCount', str, None),
> ('description', decodeIfNotNone,
> None)
> ])
> exporters = [ProductExporter]
>
> after reading this great article on unicode and python I also tried the
> following
>
> http://boodebr.org/main/python/all-about-python-and-unicode
>
> return unicode(utf8_string, 'utf-8')
>
> but that gives me
>
> File "exporter.py", line 26, in decodeIfNotNone
> return unicode(utf8_string, 'utf-8')
> TypeError: decoding Unicode is not supported
>
> Again I'm stucked here, there seems to have been a similar issue with the
> bulkuploader
>
> http://code.google.com/p/googleappengine/issues/detail?id=157
>
> but not sure if that applies to my exporter issue here too? (besides that
> it's already marked as fixed)
>
> Thanks for any further guidance you can give me
>
> Dominik
>
> That doesn't look like correct Python syntax to me. Lambda expressions are
> limited to a single expression (
> http://docs.python.org/tutorial/controlflow.html). I am pretty sure that
> just takes a callable, so you can do this:
>
> def decodeIfNotNone(utf8_string):
> if utf8_string:
> return utf8_string.decode("utf-8")
> else:
> return ""
>
> Then pass this callable as the second parameter to the method:
>
> ('brand', decodeIfNotNone, None),
>
> Let me know if this works for you.
>
> On Thu, Dec 10, 2009 at 5:40 PM, Dominik Steiner <
> dominik.j.stei...@googlemail.com> wrote:
>
>> Thanks Ikai again for the fast reply.
>> (
>> I tried the following
>>
>> ('brand', lambda x: x.decode('utf-8'), ""),
>>
>> so changing the default value from None to "" but that didn't help
>>
>> Then tried the following
>>
>> ('brand', if x is not None: lambda x: x.decode('utf-8'), ""),
>>
>> but this threw a syntax error.
>>
>> Is there anyway to check that the value is not null in order to decode it?
>>
>> Thanks
>>
>> Dominik
>>
>> It's basically the equivalent of a Null Pointer exception. This syntax:
>>
>> lambda x: x.someMethod()
>>
>> Is an inline anonymous function call. If x is None as opposed to blank
>> String, you will not be able to call methods on it.
>>
>> On Thu, Dec 10, 2009 at 1:45 PM, Dominik Steiner <
>> dominik.j.stei...@googlemail.com> wrote:
>>
>>> Thanks Ikai for the fast response,
>>>
>>> yes, some unicode characters might be the issue here as my data is in
>>> Spanish. I tried your suggestion and now have an exporter.py that looks like
>>> this
>>>
>>> from google.appengine.ext import db
>>>
>>> class Product(db.Model):
>>> userId = db.StringProperty()
>>> name = db.StringProperty()
>>> shop = db.StringProperty()
>>> brand = db.StringProperty()
>>> model = db.StringProperty()
>>> contact = db.StringProperty()
>>> imageId = db.StringProperty()
>>> userNickName = db.StringProperty()
>>> price = db.FloatProperty()
>>> registeredDate = db.DateProperty()
>>> validUntilDate = db.DateProperty()
>>> ratingCount = db.IntegerProperty()
>>> ratingTotal = db.IntegerProperty()
>>> rating = db.IntegerProperty()
>>> commentCount = db.IntegerProperty()
>>> description = db.TextProperty()
>>>
>>> from google.appengine.ext import db
>>> from google.appengine.tools import bulkloader
>>>
>>> class ProductExporter(bulkloader.Exporter):
>>> def __init__(self):
>>> bulkloader.Exporter.__init__(self, 'Product',
>>> [('name', lambda x:
>>> x.decode('utf-8'), None),
>>> ('userId', str, None),
>>> ('shop', lambda x:
>>> x.decode('utf-8'), None).
>>> ('brand', lambda x:
>>> x.decode('utf-8'), None),
>>> ('model', lambda x:
>>> x.decode('utf-8'), None),
>>> ('contact', lambda x:
>>> x.decode('utf-8'), None),
>>> ('imageId', str, None),
>>> ('userNickName', lambda x:
>>> x.decode('utf-8'), None),
>>> ('price', str, None),
>>> ('registeredDate', str, None),
>>> ('validUntilDate', str, None),
>>> ('ratingCount', str, None),
>>> ('ratingTotal', str, None),
>>> ('rating', str, None),
>>> ('commentCount', str, None),
>>> ('description', lambda x:
>>> x.decode('utf-8'), None)
>>> ])
>>> exporters = [ProductExporter]
>>>
>>>
>>> and I now get the following error (but at least it seems that it had been
>>> downloading for a while before throwing that error)
>>>
>>> Traceback (most recent call last):
>>> File "/usr/local/bin/appcfg.py", line 60, in <module>
>>> run_file(__file__, globals())
>>> File "/usr/local/bin/appcfg.py", line 57, in run_file
>>> execfile(script_path, globals_)
>>> File
>>> "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py",
>>> line 2548, in <module>
>>> main(sys.argv)
>>> File
>>> "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py",
>>> line 2539, in main
>>> result = AppCfgApp(argv).Run()
>>> File
>>> "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py",
>>> line 1640, in Run
>>> self.action(self)
>>> File
>>> "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py",
>>> line 2427, in __call__
>>> return method()
>>> File
>>> "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py",
>>> line 2293, in PerformDownload
>>> run_fn(args)
>>> File
>>> "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py",
>>> line 2215, in RunBulkloader
>>> sys.exit(bulkloader.Run(arg_dict))
>>> File
>>> "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py",
>>> line 3894, in Run
>>> return _PerformBulkload(arg_dict)
>>> File
>>> "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py",
>>> line 3800, in _PerformBulkload
>>> return_code = app.Run()
>>> File
>>> "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py",
>>> line 3158, in Run
>>> self.progress_thread.WorkFinished()
>>> File
>>> "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py",
>>> line 2207, in WorkFinished
>>> exporter.output_entities(self.result_db.AllEntities())
>>> File
>>> "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py",
>>> line 2788, in output_entities
>>> for entity in entity_generator)
>>> File
>>> "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py",
>>> line 2788, in <genexpr>
>>> for entity in entity_generator)
>>> File
>>> "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py",
>>> line 2769, in __SerializeEntity
>>> encoding = self.__EncodeEntity(entity)
>>> File
>>> "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py",
>>> line 2757, in __EncodeEntity
>>> writer.writerow(self.__ExtractProperties(entity))
>>> File
>>> "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py",
>>> line 2738, in __ExtractProperties
>>> encoding.append(fn(entity[name]))
>>> File "exporter.py", line 30, in <lambda>
>>> ('brand', lambda x: x.decode('utf-8'), None),
>>> AttributeError: 'NoneType' object has no attribute 'decode'
>>>
>>> Do you have any idea of what that could be?
>>>
>>> Thanks again for your help.
>>>
>>> Dominik
>>>
>>> Interesting. Do you have some unicode characters in any fields? There's
>>> a snippet on the bulk loader page about how to handle these characters in
>>> your Importer:
>>>
>>> http://code.google.com/appengine/docs/python/tools/uploadingdata.html
>>>
>>> import datetime
>>> from google.appengine.ext import db
>>> from google.appengine.tools import bulkloader
>>> import models
>>>
>>> class AlbumLoader(bulkloader.Loader):
>>> def __init__(self):
>>> bulkloader.Loader.__init__(self, 'Album',
>>> [('title', lambda x:
>>> x.decode('utf-8')),
>>> ('artist', lambda x:
>>> x.decode('utf-8')),
>>> ('publication_date',
>>> lambda x:
>>> datetime.datetime.strptime(x, '%m/%d/%Y').date()),
>>> ('length_in_minutes', int)
>>> ])
>>>
>>> loaders = [AlbumLoader]
>>>
>>> Your error may be caused by UTF-8/ASCII incompatibilities, and this is
>>> where I would start looking.
>>>
>>> On Wed, Dec 9, 2009 at 8:26 PM, Dominik Steiner <
>>> dominik.j.stei...@googlemail.com> wrote:
>>>
>>>> Hi Ikai,
>>>>
>>>> I followed the instructions in the cook book for the python uploader/
>>>> downloader in order to be able to download and backup the data of my
>>>> java application.
>>>>
>>>> As already posted in the cook book article, I got the following
>>>> situation:
>>>>
>>>> "I would love to be able to download data in order to backup the data
>>>> of my application and I tried your steps. One thing that didn't work
>>>> for me was the line
>>>>
>>>> appcfg.py --server=python.latest.appid.appspot.com download_data
>>>> exporter --filename=data.csv --kind=Thing --config_file=exporter/
>>>> thing_exporter.py
>>>>
>>>> but had to write
>>>>
>>>> appcfg.py --server=python.latest.xelavos.appspot.com download_data --
>>>> filename=data.csv --kind=Thing --config_file=exporter/
>>>> thing_exporter.py path/to/my/pythondirectory
>>>>
>>>> Then after being able to launch the exporter i got the following error
>>>> after a while of downloading
>>>>
>>>> .[INFO ] Product: No descending index on __key__, performing serial
>>>> download
>>>> .........................................
>>>> Traceback (most recent call last):
>>>> File "/usr/local/bin/appcfg.py", line 60, in
>>>> run_file(__file__, globals())
>>>> File "/usr/local/bin/appcfg.py", line 57, in run_file
>>>> execfile(script_path, globals_)
>>>> File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
>>>> GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
>>>> google/appengine/tools/appcfg.py", line 2548, in
>>>> main(sys.argv)
>>>> File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
>>>> GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
>>>> google/appengine/tools/appcfg.py", line 2539, in main
>>>> result = AppCfgApp(argv).Run()
>>>> File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
>>>> GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
>>>> google/appengine/tools/appcfg.py", line 1640, in Run
>>>> self.action(self)
>>>> File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
>>>> GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
>>>> google/appengine/tools/appcfg.py", line 2427, in __call__
>>>> return method()
>>>> File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
>>>> GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
>>>> google/appengine/tools/appcfg.py", line 2293, in PerformDownload
>>>> run_fn(args)
>>>> File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
>>>> GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
>>>> google/appengine/tools/appcfg.py", line 2215, in RunBulkloader
>>>> sys.exit(bulkloader.Run(arg_dict))
>>>> File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
>>>> GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
>>>> google/appengine/tools/bulkloader.py", line 3894, in Run
>>>> return _PerformBulkload(arg_dict)
>>>> File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
>>>> GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
>>>> google/appengine/tools/bulkloader.py", line 3800, in _PerformBulkload
>>>> return_code = app.Run()
>>>> File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
>>>> GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
>>>> google/appengine/tools/bulkloader.py", line 3158, in Run
>>>> self.progress_thread.WorkFinished()
>>>> File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
>>>> GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
>>>> google/appengine/tools/bulkloader.py", line 2207, in WorkFinished
>>>> exporter.output_entities(self.result_db.AllEntities())
>>>> File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
>>>> GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
>>>> google/appengine/tools/bulkloader.py", line 2788, in output_entities
>>>> for entity in entity_generator)
>>>> File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
>>>> GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
>>>> google/appengine/tools/bulkloader.py", line 2788, in
>>>> for entity in entity_generator)
>>>> File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
>>>> GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
>>>> google/appengine/tools/bulkloader.py", line 2769, in __SerializeEntity
>>>> encoding = self.__EncodeEntity(entity)
>>>> File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
>>>> GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
>>>> google/appengine/tools/bulkloader.py", line 2757, in __EncodeEntity
>>>> writer.writerow(self.__ExtractProperties(entity))
>>>> File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
>>>> GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
>>>> google/appengine/tools/bulkloader.py", line 2738, in
>>>> __ExtractProperties
>>>> encoding.append(fn(entity[name]))
>>>> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in
>>>> position 7: ordinal not in range(128)
>>>> "
>>>>
>>>> Do you know what this error means and how i can fix it?
>>>>
>>>> Thanks for any help
>>>>
>>>> Dominik
>>>>
>>>> --
>>>>
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Google App Engine for Java" group.
>>>> To post to this group, send email to
>>>> google-appengine-j...@googlegroups.com.
>>>> To unsubscribe from this group, send email to
>>>> google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com>
>>>> .
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/google-appengine-java?hl=en.
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Ikai Lan
>>> Developer Programs Engineer, Google App Engine
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Google App Engine for Java" group.
>>> To post to this group, send email to
>>> google-appengine-j...@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> google-appengine-java+unsubscr...@googlegroups.com.
>>> For more options, visit this group at
>>> http://groups.google.com/group/google-appengine-java?hl=en.
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Google App Engine for Java" group.
>>> To post to this group, send email to
>>> google-appengine-j...@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com>
>>> .
>>> For more options, visit this group at
>>> http://groups.google.com/group/google-appengine-java?hl=en.
>>>
>>
>>
>>
>> --
>> Ikai Lan
>> Developer Programs Engineer, Google App Engine
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine for Java" group.
>> To post to this group, send email to
>> google-appengine-j...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine-java+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine-java?hl=en.
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine for Java" group.
>> To post to this group, send email to
>> google-appengine-j...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com>
>> .
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine-java?hl=en.
>>
>
>
>
> --
> Ikai Lan
> Developer Programs Engineer, Google App Engine
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine for Java" group.
> To post to this group, send email to
> google-appengine-j...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine-java+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine-java?hl=en.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine for Java" group.
> To post to this group, send email to
> google-appengine-j...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine-java?hl=en.
>
--
Ikai Lan
Developer Programs Engineer, Google App Engine
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
