[galaxy-dev] Fatal Error while uploading a File with special Characters

[SAPET, Frederic]

Hello

Like Matthias 
(http://gmod.827538.n3.nabble.com/Fatal-Error-while-uploading-a-File-with-special-Characters-td4051157.html)
  I can't upload a file with special characters.

My test file is : Frédéric.txt

The database itself seems to be ok, I can see that the insertion is ok in 
"history_dataset_association" :
idhistory_id   dataset_id  create_time  
update_time copied_from_history_dataset_association_id hid  
   nameinfo
328 1 330 2017-03-01 08:13:56   2017-03-01 
08:14:05   NULL 293 Frédéric.txt uploaded txt 
file  1 line

But until I change Frédéric.txt to Frederic.txt within the name column, my 
history is broken.

Errors are the same as described by Matthias.

galaxy.web.framework.decorators ERROR 2017-03-01 09:14:18,321 Uncaught 
exception in exposed API method:
Traceback (most recent call last):
  File "/softs/bioinfo/galaxy-prod/lib/galaxy/web/framework/decorators.py", 
line 284, in decorator
rval = _format_return_as_json( rval, jsonp_callback, pretty=trans.debug )
  File "/softs/bioinfo/galaxy-prod/lib/galaxy/web/framework/decorators.py", 
line 317, in _format_return_as_json
json = safe_dumps( rval, **dumps_kwargs )
  File "/softs/bioinfo/galaxy-prod/lib/galaxy/util/json.py", line 67, in 
safe_dumps
dumped = json.dumps( obj, allow_nan=False, **kwargs )
  File "/usr/local/lib/python2.7/json/__init__.py", line 251, in dumps
sort_keys=sort_keys, **kw).encode(obj)
  File "/usr/local/lib/python2.7/json/encoder.py", line 209, in encode
chunks = list(chunks)
  File "/usr/local/lib/python2.7/json/encoder.py", line 431, in _iterencode
for chunk in _iterencode_list(o, _current_indent_level):
  File "/usr/local/lib/python2.7/json/encoder.py", line 332, in _iterencode_list
for chunk in chunks:
  File "/usr/local/lib/python2.7/json/encoder.py", line 390, in _iterencode_dict
yield _encoder(value)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 2: invalid 
continuation byte


I have read a lot of threads about this kind of error (mostly from 
stackoverflow)
It is related to Galaxy or it's related to my python installation ?

For a production server, it means that I will have to update manually the 
database each time a user try to upload such a file (find his id, find the 
broken history and the file name to correct !)

I'm running Galaxy 17.01 with a mysql (5.7.11-4, UTF-8) database.
I've modified lib/galaxy/datatypes/binary.py (for XLS datatypes) and 
lib/galaxy/jobs/runners/__init__.py (truncate the job name for PBS-PRO)

# On branch release_17.01
# Your branch is ahead of 'origin/release_17.01' by 3 commits.
#
# Changed but not updated:
#   (use "git add ..." to update what will be committed)
#   (use "git checkout -- ..." to discard changes in working directory)
#
#   modified:   lib/galaxy/datatypes/binary.py
#   modified:   lib/galaxy/jobs/runners/__init__.py

Thanks in advance!

Fred
---
Frederic Sapet
Bioinformatics Research Engineer
BIOGEMMA - Upstream Genomics Group
Centre de Recherche de Chappes
CS 90126
63720 CHAPPES
FRANCE
Tel : +33 (0)4 73 67 88 54
Fax : +33 (0)4 73 67 88 99
E-mail : frederic.sa...@biogemma.com

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Fatal Error while uploading a File with special Characters

[Matthias Enders]

Hi,

MySQL:
Server version: 5.5.44-0ubuntu0.14.04.1 (Ubuntu)

MySQL-GalaxyDB Encoding: UTF8

Galaxy: Where can I find the current release und changeset of my Galaxy 
Instance?

Mit freundlichen Grüßen
Matthias Enders
---
[cid:image001.png@01D03960.8254BEE0]

GERMAN SEED ALLIANCE GmbH
c/o Norddeutsche Pflanzenzucht
Hans-Georg Lembke KG
Hohenlieth, 24363 Holtsee
Tel.: +49 (0)4351/ 736-189
Fax: + 49 (0)4351/ 736-271
Mobil: +49 (0)151/ 14247360

Email: m.end...@german-seed-alliance.de<mailto:m.end...@german-seed-alliance.de>

Firmensitz Köln
Amtsgericht Köln, HRB 73844

Von: Dannon Baker [mailto:dannon.ba...@gmail.com]
Gesendet: Thursday, September 10, 2015 4:53 PM
An: Matthias Enders 
Cc: galaxy-dev@lists.galaxyproject.org
Betreff: Re: [galaxy-dev] Fatal Error while uploading a File with special 
Characters

Thanks.  Can you specify exactly which release (or even the exact changeset) of 
the galaxy revision you're running, and what encoding the database is currently 
set to?  I've reproduced the error with the latest galaxy release, mysql 5.6.26 
(utf8 encoding) and will get a fix out shortly, but I want to make sure I'm 
testing your situation as well.

On Thu, Sep 10, 2015 at 10:14 AM, Matthias Enders 
mailto:m.end...@german-seed-alliance.de>> 
wrote:
It was the name of the File:

Blühzeitpunkt.txt

(for example)

Von: Dannon Baker [mailto:dannon.ba...@gmail.com<mailto:dannon.ba...@gmail.com>]
Gesendet: Thursday, September 10, 2015 4:12 PM
An: Matthias Enders 
mailto:m.end...@german-seed-alliance.de>>
Cc: 
galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>
Betreff: Re: [galaxy-dev] Fatal Error while uploading a File with special 
Characters

You should definitely not have to purge a user to fix this (not that it should 
happen in the first place).  Was it the file with special characters, or the 
name of the file?



On Thu, Sep 10, 2015 at 2:28 AM, Matthias Enders 
mailto:m.end...@german-seed-alliance.de>> 
wrote:
Dear Galaxy Community,

a User of our Galaxy Platform has produced a fatal error by uploading a file 
containing a German special character  (ä / ü / ö / ß). This resulted in the 
following status (only for the given user):

- Upload was canceled: Status was "Failed Internal Server Error (500)"
- All Histories were corrupt stating: "An error was encountered while loading 
the contents."
- Tools can´t be used an stating (in Traceback):
" UnicodeDecodeError: 'utf8' codec can't decode byte 0xfc in position 5: 
invalid start byte"
"Module weberror.evalexception.middleware:364 in respond"


This totally destroyed to current user and we have had to purge him in order to 
get him back to work, even losing all his datasets & histories!

We are using an MySQL database as a backend. Is the Encoding of this database 
the problem? If yes: Which encoding is used/supported by galaxy?

Thanks in advance!

Matthias
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Fatal Error while uploading a File with special Characters

[Dannon Baker]

Thanks.  Can you specify exactly which release (or even the exact
changeset) of the galaxy revision you're running, and what encoding the
database is currently set to?  I've reproduced the error with the latest
galaxy release, mysql 5.6.26 (utf8 encoding) and will get a fix out
shortly, but I want to make sure I'm testing your situation as well.

On Thu, Sep 10, 2015 at 10:14 AM, Matthias Enders <
m.end...@german-seed-alliance.de> wrote:

> It was the name of the File:
>
>
>
> Blühzeitpunkt.txt
>
>
>
> (for example)
>
>
>
> *Von:* Dannon Baker [mailto:dannon.ba...@gmail.com]
> *Gesendet:* Thursday, September 10, 2015 4:12 PM
> *An:* Matthias Enders 
> *Cc:* galaxy-dev@lists.galaxyproject.org
> *Betreff:* Re: [galaxy-dev] Fatal Error while uploading a File with
> special Characters
>
>
>
> You should definitely not have to purge a user to fix this (not that it
> should happen in the first place).  Was it the file with special
> characters, or the name of the file?
>
>
>
>
>
>
>
> On Thu, Sep 10, 2015 at 2:28 AM, Matthias Enders <
> m.end...@german-seed-alliance.de> wrote:
>
> Dear Galaxy Community,
>
> a User of our Galaxy Platform has produced a fatal error by uploading a
> file containing a German special character  (ä / ü / ö / ß). This resulted
> in the following status (only for the given user):
>
> - Upload was canceled: Status was "Failed Internal Server Error (500)"
> - All Histories were corrupt stating: "An error was encountered while
> loading the contents."
> - Tools can´t be used an stating (in Traceback):
> " UnicodeDecodeError: 'utf8' codec can't decode byte 0xfc in position 5:
> invalid start byte"
> "Module weberror.evalexception.middleware:364 in respond"
>
>
> This totally destroyed to current user and we have had to purge him in
> order to get him back to work, even losing all his datasets & histories!
>
> We are using an MySQL database as a backend. Is the Encoding of this
> database the problem? If yes: Which encoding is used/supported by galaxy?
>
> Thanks in advance!
>
> Matthias
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Fatal Error while uploading a File with special Characters

[Matthias Enders]

It was the name of the File:

Blühzeitpunkt.txt

(for example)

Von: Dannon Baker [mailto:dannon.ba...@gmail.com]
Gesendet: Thursday, September 10, 2015 4:12 PM
An: Matthias Enders 
Cc: galaxy-dev@lists.galaxyproject.org
Betreff: Re: [galaxy-dev] Fatal Error while uploading a File with special 
Characters

You should definitely not have to purge a user to fix this (not that it should 
happen in the first place).  Was it the file with special characters, or the 
name of the file?



On Thu, Sep 10, 2015 at 2:28 AM, Matthias Enders 
mailto:m.end...@german-seed-alliance.de>> 
wrote:
Dear Galaxy Community,

a User of our Galaxy Platform has produced a fatal error by uploading a file 
containing a German special character  (ä / ü / ö / ß). This resulted in the 
following status (only for the given user):

- Upload was canceled: Status was "Failed Internal Server Error (500)"
- All Histories were corrupt stating: "An error was encountered while loading 
the contents."
- Tools can´t be used an stating (in Traceback):
" UnicodeDecodeError: 'utf8' codec can't decode byte 0xfc in position 5: 
invalid start byte"
"Module weberror.evalexception.middleware:364 in respond"


This totally destroyed to current user and we have had to purge him in order to 
get him back to work, even losing all his datasets & histories!

We are using an MySQL database as a backend. Is the Encoding of this database 
the problem? If yes: Which encoding is used/supported by galaxy?

Thanks in advance!

Matthias
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Fatal Error while uploading a File with special Characters

[Dannon Baker]

You should definitely not have to purge a user to fix this (not that it
should happen in the first place).  Was it the file with special
characters, or the name of the file?



On Thu, Sep 10, 2015 at 2:28 AM, Matthias Enders <
m.end...@german-seed-alliance.de> wrote:

> Dear Galaxy Community,
>
> a User of our Galaxy Platform has produced a fatal error by uploading a
> file containing a German special character  (ä / ü / ö / ß). This resulted
> in the following status (only for the given user):
>
> - Upload was canceled: Status was "Failed Internal Server Error (500)"
> - All Histories were corrupt stating: "An error was encountered while
> loading the contents."
> - Tools can´t be used an stating (in Traceback):
> " UnicodeDecodeError: 'utf8' codec can't decode byte 0xfc in position 5:
> invalid start byte"
> "Module weberror.evalexception.middleware:364 in respond"
>
>
> This totally destroyed to current user and we have had to purge him in
> order to get him back to work, even losing all his datasets & histories!
>
> We are using an MySQL database as a backend. Is the Encoding of this
> database the problem? If yes: Which encoding is used/supported by galaxy?
>
> Thanks in advance!
>
> Matthias
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Fatal Error while uploading a File with special Characters

[Matthias Enders]

Dear Galaxy Community,

a User of our Galaxy Platform has produced a fatal error by uploading a file 
containing a German special character  (ä / ü / ö / ß). This resulted in the 
following status (only for the given user):

- Upload was canceled: Status was "Failed Internal Server Error (500)"
- All Histories were corrupt stating: "An error was encountered while loading 
the contents."
- Tools can´t be used an stating (in Traceback):
" UnicodeDecodeError: 'utf8' codec can't decode byte 0xfc in position 5: 
invalid start byte"
"Module weberror.evalexception.middleware:364 in respond"


This totally destroyed to current user and we have had to purge him in order to 
get him back to work, even losing all his datasets & histories!

We are using an MySQL database as a backend. Is the Encoding of this database 
the problem? If yes: Which encoding is used/supported by galaxy?

Thanks in advance!

Matthias
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/