from:"alexandre . henry"

Re: [web2py] Re: UnicodeDecodeError "'utf-8' codec can't decode..." when inserting a blob in database (python 3.6.3)

2018-04-03 Thread alexandre . henry

Thank you Mathieu,

I found the place to create issue... Actually, I wanted to better qualify 
the issue before open it, restarting from a clean project, and I cannot 
reproduce the very first issue!
I encounter some other issues related to binary encoding but now I prefer 
triple check before speaking!

Le lundi 2 avril 2018 16:36:59 UTC+2, Mathieu Clabaut a écrit :
>
> Alexandre,
>
> just go to : https://github.com/web2py/web2py/issues/new
>
> And fill in the issue withe the information given in your mail.
>
> Le ven. 30 mars 2018 à 16:50,  a 
> écrit :
>
>> Sure! Let me see how to do that and I will. Thank you
>>
>>
>> Le mercredi 28 mars 2018 18:49:08 UTC+2, Massimo Di Pierro a écrit :
>>
>>> I think this is. Can you please open an issue about it?
>>>
>>> On Tuesday, 27 March 2018 13:32:15 UTC-5, alexand...@gmail.com wrote:

 Hello All,

 I'm an absolute newbie with web2py so I'm not sure if the issue lies in 
 web2py/3.6 or my own mistakes... My apologies in advance.

 I encounter this error in a db.my_table.update_or_insert(..)

 UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfe in position 
 96328: invalid start byte'

 What I'm trying to do:
 I have a text file that is the representation of a datamodel in a 
 specific COTS format.
 I want to parse it, list the tables to be created and create them 
 actually.
 I know that the file countains non-utf-8 characters, that's why I open 
 it in binary:

dpm_file = gzip.open(filepath,'rb')

 ... parse it using binary regex: (I imported the 'regex' module)
 regexp = regex.compile(b':columns 
 (\(((?>[^()]+)|(?1))*\))',regex.DOTALL)
 header_classes = regexp.findall(one_format_text)[0]

 ... and insert the results in db blob fields: (the error occurs here)
 db.dpm_header.update_or_insert(db.dpm_header.dpm == my_dpm_id, dpm 
 = my_dpm_id, header_classes  = header_classes, header_objects  = 
 header_objects)

 I defined blob fields because I thought that it was the way to write 
 directly in binary in database.
 I really wanted to use binary preciselly to avoid to manage the 
 encoding/decoding issues.

 In addition, this project is an attempt to transport in web2py an older 
 project that was working perfectly in pure python 3.6.3 (unix and windows) 
 (except that instead of creating tables in a DB, I created text files in a 
 filer...).
 Is it a issue on web2py/python 3.6? Or my misunderstanding on database 
 management...

 Thank you for your help!

 Here is the trace

 Traceback (most recent call last):
   File "D:\web\zz\web2py\gluon\restricted.py", line 219, in restricted
 exec(ccode, environment)
   File "D:/web/zz/web2py/applications/dpm_manager/controllers/default.py" 
 ,
  line 175, in 
   File "D:\web\zz\web2py\gluon\globals.py", line 419, in 
 self._caller = lambda f: f()
   File "D:\web\zz\web2py\gluon\tools.py", line 3982, in f
 return action(*a, **b)
   File "D:/web/zz/web2py/applications/dpm_manager/controllers/default.py" 
 ,
  line 131, in analyze
 header_classes  = header_classes)
   File "D:\web\zz\web2py\gluon\packages\dal\pydal\objects.py", line 811, 
 in update_or_insert
 record = self(_key)
   File "D:\web\zz\web2py\gluon\packages\dal\pydal\objects.py", line 565, 
 in __call__
 orderby_on_limitby=False).first()
   File "D:\web\zz\web2py\gluon\packages\dal\pydal\objects.py", line 2250, 
 in select
 return adapter.select(self.query, fields, attributes)
   File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\sqlite.py", 
 line 82, in select
 return super(SQLite, self).select(query, fields, attributes)
   File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\base.py", line 
 762, in select
 return self._select_aux(sql, fields, attributes, colnames)
   File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\base.py", line 
 741, in _select_aux
 return processor(rows, fields, colnames, cacheable=cacheable)
   File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\base.py", line 
 305, in parse
 for row in rows
   File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\base.py", line 
 302, in 
 self._parse(
   File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\base.py", line 
 231, in _parse
 value = self.parse_value(value, fit, ft, blob_decode)
   File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\base.py", line 
 198, in parse_value
 return self.parser.parse(value, field_itype, field_type)

[web2py] Re: UnicodeDecodeError "'utf-8' codec can't decode..." when inserting a blob in database (python 3.6.3)

2018-03-30 Thread alexandre . henry

Sure! Let me see how to do that and I will. Thank you

Le mercredi 28 mars 2018 18:49:08 UTC+2, Massimo Di Pierro a écrit :
>
> I think this is. Can you please open an issue about it?
>
> On Tuesday, 27 March 2018 13:32:15 UTC-5, alexand...@gmail.com 
>  wrote:
>>
>> Hello All,
>>
>> I'm an absolute newbie with web2py so I'm not sure if the issue lies in 
>> web2py/3.6 or my own mistakes... My apologies in advance.
>>
>> I encounter this error in a db.my_table.update_or_insert(..)
>>
>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfe in position 96328: 
>> invalid start byte'
>>
>>
>> What I'm trying to do:
>> I have a text file that is the representation of a datamodel in a 
>> specific COTS format.
>> I want to parse it, list the tables to be created and create them 
>> actually.
>> I know that the file countains non-utf-8 characters, that's why I open it 
>> in binary:
>>
>>dpm_file = gzip.open(filepath,'rb')
>>
>> ... parse it using binary regex: (I imported the 'regex' module)
>> regexp = regex.compile(b':columns 
>> (\(((?>[^()]+)|(?1))*\))',regex.DOTALL)
>> header_classes = regexp.findall(one_format_text)[0]
>>
>> ... and insert the results in db blob fields: (the error occurs here)
>> db.dpm_header.update_or_insert(db.dpm_header.dpm == my_dpm_id, dpm = 
>> my_dpm_id, header_classes  = header_classes, header_objects  = 
>> header_objects)
>>
>> I defined blob fields because I thought that it was the way to write 
>> directly in binary in database.
>> I really wanted to use binary preciselly to avoid to manage the 
>> encoding/decoding issues.
>>
>> In addition, this project is an attempt to transport in web2py an older 
>> project that was working perfectly in pure python 3.6.3 (unix and windows) 
>> (except that instead of creating tables in a DB, I created text files in a 
>> filer...).
>> Is it a issue on web2py/python 3.6? Or my misunderstanding on database 
>> management...
>>
>> Thank you for your help!
>>
>> Here is the trace
>>
>> Traceback (most recent call last):
>>   File "D:\web\zz\web2py\gluon\restricted.py", line 219, in restricted
>> exec(ccode, environment)
>>   File "D:/web/zz/web2py/applications/dpm_manager/controllers/default.py" 
>> ,
>>  line 175, in 
>>   File "D:\web\zz\web2py\gluon\globals.py", line 419, in 
>> self._caller = lambda f: f()
>>   File "D:\web\zz\web2py\gluon\tools.py", line 3982, in f
>> return action(*a, **b)
>>   File "D:/web/zz/web2py/applications/dpm_manager/controllers/default.py" 
>> ,
>>  line 131, in analyze
>> header_classes  = header_classes)
>>   File "D:\web\zz\web2py\gluon\packages\dal\pydal\objects.py", line 811, in 
>> update_or_insert
>> record = self(_key)
>>   File "D:\web\zz\web2py\gluon\packages\dal\pydal\objects.py", line 565, in 
>> __call__
>> orderby_on_limitby=False).first()
>>   File "D:\web\zz\web2py\gluon\packages\dal\pydal\objects.py", line 2250, in 
>> select
>> return adapter.select(self.query, fields, attributes)
>>   File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\sqlite.py", line 
>> 82, in select
>> return super(SQLite, self).select(query, fields, attributes)
>>   File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\base.py", line 
>> 762, in select
>> return self._select_aux(sql, fields, attributes, colnames)
>>   File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\base.py", line 
>> 741, in _select_aux
>> return processor(rows, fields, colnames, cacheable=cacheable)
>>   File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\base.py", line 
>> 305, in parse
>> for row in rows
>>   File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\base.py", line 
>> 302, in 
>> self._parse(
>>   File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\base.py", line 
>> 231, in _parse
>> value = self.parse_value(value, fit, ft, blob_decode)
>>   File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\base.py", line 
>> 198, in parse_value
>> return self.parser.parse(value, field_itype, field_type)
>>   File "D:\web\zz\web2py\gluon\packages\dal\pydal\parsers\__init__.py", line 
>> 101, in parse
>> return self.registered[field_itype](value, field_type)
>>   File "D:\web\zz\web2py\gluon\packages\dal\pydal\parsers\__init__.py", line 
>> 76, in __call__
>> return self.call(value, field_type)
>>   File "D:\web\zz\web2py\gluon\packages\dal\pydal\parsers\__init__.py", line 
>> 73, in _call
>> return self.f(self.parser, value)
>>   File "D:\web\zz\web2py\gluon\packages\dal\pydal\parsers\base.py", line 37, 
>> in _blob
>> return to_native(b64decode(to_bytes(value)))
>>   File "D:\web\zz\web2py\gluon\packages\dal\pydal\_compat.py", line 82, in 
>> to_native
>> return obj.decode(charset, errors)
>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfe in

[web2py] UnicodeDecodeError "'utf-8' codec can't decode..." when inserting a blob in database (python 3.6.3)

2018-03-27 Thread alexandre . henry

Hello All,

I'm an absolute newbie with web2py so I'm not sure if the issue lies in 
web2py/3.6 or my own mistakes... My apologies in advance.

I encounter this error in a db.my_table.update_or_insert(..)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfe in position 96328: 
invalid start byte'


What I'm trying to do:
I have a text file that is the representation of a datamodel in a specific 
COTS format.
I want to parse it, list the tables to be created and create them actually.
I know that the file countains non-utf-8 characters, that's why I open it 
in binary:

   dpm_file = gzip.open(filepath,'rb')

... parse it using binary regex: (I imported the 'regex' module)
regexp = regex.compile(b':columns 
(\(((?>[^()]+)|(?1))*\))',regex.DOTALL)
header_classes = regexp.findall(one_format_text)[0]

... and insert the results in db blob fields: (the error occurs here)
db.dpm_header.update_or_insert(db.dpm_header.dpm == my_dpm_id, dpm = 
my_dpm_id, header_classes  = header_classes, header_objects  = 
header_objects)

I defined blob fields because I thought that it was the way to write 
directly in binary in database.
I really wanted to use binary preciselly to avoid to manage the 
encoding/decoding issues.

In addition, this project is an attempt to transport in web2py an older 
project that was working perfectly in pure python 3.6.3 (unix and windows) 
(except that instead of creating tables in a DB, I created text files in a 
filer...).
Is it a issue on web2py/python 3.6? Or my misunderstanding on database 
management...

Thank you for your help!

Here is the trace

Traceback (most recent call last):
  File "D:\web\zz\web2py\gluon\restricted.py", line 219, in restricted
exec(ccode, environment)
  File "D:/web/zz/web2py/applications/dpm_manager/controllers/default.py" 
, 
line 175, in 
  File "D:\web\zz\web2py\gluon\globals.py", line 419, in 
self._caller = lambda f: f()
  File "D:\web\zz\web2py\gluon\tools.py", line 3982, in f
return action(*a, **b)
  File "D:/web/zz/web2py/applications/dpm_manager/controllers/default.py" 
, 
line 131, in analyze
header_classes  = header_classes)
  File "D:\web\zz\web2py\gluon\packages\dal\pydal\objects.py", line 811, in 
update_or_insert
record = self(_key)
  File "D:\web\zz\web2py\gluon\packages\dal\pydal\objects.py", line 565, in 
__call__
orderby_on_limitby=False).first()
  File "D:\web\zz\web2py\gluon\packages\dal\pydal\objects.py", line 2250, in 
select
return adapter.select(self.query, fields, attributes)
  File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\sqlite.py", line 82, 
in select
return super(SQLite, self).select(query, fields, attributes)
  File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\base.py", line 762, 
in select
return self._select_aux(sql, fields, attributes, colnames)
  File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\base.py", line 741, 
in _select_aux
return processor(rows, fields, colnames, cacheable=cacheable)
  File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\base.py", line 305, 
in parse
for row in rows
  File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\base.py", line 302, 
in 
self._parse(
  File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\base.py", line 231, 
in _parse
value = self.parse_value(value, fit, ft, blob_decode)
  File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\base.py", line 198, 
in parse_value
return self.parser.parse(value, field_itype, field_type)
  File "D:\web\zz\web2py\gluon\packages\dal\pydal\parsers\__init__.py", line 
101, in parse
return self.registered[field_itype](value, field_type)
  File "D:\web\zz\web2py\gluon\packages\dal\pydal\parsers\__init__.py", line 
76, in __call__
return self.call(value, field_type)
  File "D:\web\zz\web2py\gluon\packages\dal\pydal\parsers\__init__.py", line 
73, in _call
return self.f(self.parser, value)
  File "D:\web\zz\web2py\gluon\packages\dal\pydal\parsers\base.py", line 37, in 
_blob
return to_native(b64decode(to_bytes(value)))
  File "D:\web\zz\web2py\gluon\packages\dal\pydal\_compat.py", line 82, in 
to_native
return obj.decode(charset, errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfe in position 96328: 
invalid start byte

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [web2py] Re: UnicodeDecodeError "'utf-8' codec can't decode..." when inserting a blob in database (python 3.6.3)

[web2py] Re: UnicodeDecodeError "'utf-8' codec can't decode..." when inserting a blob in database (python 3.6.3)

[web2py] UnicodeDecodeError "'utf-8' codec can't decode..." when inserting a blob in database (python 3.6.3)

3 matches

Site Navigation

Mail list logo

Footer information