On Nov 4, 2011, at 8:03 PM, Chorny, Ilya wrote: > Looks like the mv is something I added and forgot about. I changed it to > shutil.copyfile and it works fine. This also has something to do with the > isilon. I pushed it to the actual user repository so you can check it out.
Okay, I'll have a look. What's the copy for? We try to avoid moving bits as much as possible during upload since it can slow down the entire upload tool run pretty significantly if there are multiple copies taking place. --nate > > Have a good weekend. > > Thanks, > > Ilya > > > -----Original Message----- > From: galaxy-dev-boun...@lists.bx.psu.edu > [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya > Sent: Friday, November 04, 2011 4:50 PM > To: Nate Coraor (n...@bx.psu.edu) > Cc: galaxy-dev@lists.bx.psu.edu > Subject: Re: [galaxy-dev] Looks like actual user breaks splitting > > I am getting a strange error when I am uploading a file > > mv: inter-device move failed: `/illumina/scratch/Swim/galaxy/tmpk2sbZl' to > `/home/galaxy/ichorny/galaxy-central/database/tmp/upload_file_data_isAnwO'; > unable to remove target: Permission denied > > I figured out that the mv is coming from > > lib/galaxy/datatypes/sniff.py > > if in_place: > fnamemode = stat.S_IMODE( os.stat(fname).st_mode ) > os.system('mv %s %s' %(temp_name,fname)) > # Return number of lines in file. > return ( i + 1, None ) > else: > return ( i + 1, temp_name ) > > Any thoughts on what might be going on? If I comment out the move the upload > works just fine. > > Thanks, > > Ilya > > -----Original Message----- > From: Chorny, Ilya > Sent: Friday, November 04, 2011 11:11 AM > To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu); Duddy, John > Cc: galaxy-dev@lists.bx.psu.edu > Subject: RE: [galaxy-dev] Looks like actual user breaks splitting > > Well I think I figured out why we get OSError: [Errno 39] Directory not > empty: './database/job_working_directory/451'. I changed it to an rm -rf and > got the following error: > > `./database/job_working_directory/455/.nfs0000000161e986fc00000068' ': Device > or resource busy > > It might have something to do with our mounts being Isilon. Any thought on > how to deal with this? > > > -----Original Message----- > From: galaxy-dev-boun...@lists.bx.psu.edu > [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya > Sent: Friday, November 04, 2011 10:50 AM > To: Nate Coraor (n...@bx.psu.edu) > Cc: galaxy-dev@lists.bx.psu.edu > Subject: Re: [galaxy-dev] Looks like actual user breaks splitting > > I came up with a fix for User killed running job, but error encountered > removing from DRM queue: 'DRMAAJobRunner' object has no attribute > 'job_user_uid' and pushed it. > > -----Original Message----- > From: Chorny, Ilya > Sent: Friday, November 04, 2011 10:24 AM > To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu) > Cc: galaxy-dev@lists.bx.psu.edu > Subject: RE: [galaxy-dev] Looks like actual user breaks splitting > > I am also getting the following error in the log file which makes no sense > because the directory is empty. I will try to track it down. Thoughts? > > Ilya > Traceback (most recent call last): > File "/home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/__init__.py", line > 732, in cleanup > shutil.rmtree( self.working_directory ) > File "/usr/lib64/python2.6/shutil.py", line 225, in rmtree > onerror(os.rmdir, path, sys.exc_info()) > File "/usr/lib64/python2.6/shutil.py", line 223, in rmtree > os.rmdir(path) > OSError: [Errno 39] Directory not empty: > './database/job_working_directory/451' > > > -----Original Message----- > From: galaxy-dev-boun...@lists.bx.psu.edu > [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya > Sent: Friday, November 04, 2011 9:07 AM > To: Duddy, John; Nate Coraor (n...@bx.psu.edu) > Cc: galaxy-dev@lists.bx.psu.edu > Subject: Re: [galaxy-dev] Looks like actual user breaks splitting > > Nate, > > I get the following error when I try to kill a job. > > User killed running job, but error encountered removing from DRM queue: > 'DRMAAJobRunner' object has no attribute 'job_user_uid'. > > Ilya > > -----Original Message----- > From: galaxy-dev-boun...@lists.bx.psu.edu > [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Duddy, John > Sent: Thursday, November 03, 2011 3:55 PM > To: Nate Coraor (n...@bx.psu.edu) > Cc: galaxy-dev@lists.bx.psu.edu > Subject: Re: [galaxy-dev] Looks like actual user breaks splitting > > I just came to that conclusion myself. If you need me to do anything, let me > know - but it sounds like you have a handle on it. > > Muchas gracias. > > John Duddy > Sr. Staff Software Engineer > Illumina, Inc. > 9885 Towne Centre Drive > San Diego, CA 92121 > Tel: 858-736-3584 > E-mail: jdu...@illumina.com > > > -----Original Message----- > From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] > Sent: Thursday, November 03, 2011 3:53 PM > To: Duddy, John > Cc: galaxy-dev@lists.bx.psu.edu > Subject: Re: [galaxy-dev] Looks like actual user breaks splitting > > Nate Coraor (n...@bx.psu.edu) wrote: >> Duddy, John wrote: >>> I'm not following you - it's been 6 months since I wrote that code >>> ;-} >> >> I know the feeling! >> >>> IT looks to me like a DatasetPath() object is always placed in that array, >>> and with one exception near then, it looks like the change I made generates >>> those objects the same way. >> >> It's creating a dict in self.output_dataset_paths, and that dict looks >> like this when outputs_to_working_directory = False: >> >> { output_param_name : [ HDA, DatasetPath ], ... } >> >> And this when True: >> >> { output_param_name : [ Dataset, DatasetPath ], ... } >> >>> Do you have a stack trace for the merge problem I can look at? >> >> If you put this in do_merge()'s except block: >> >> log.exception( stdout ) >> >> You'll get: >> >> Traceback (most recent call last): >> File >> "/space/nate/galaxy-central-ichorny/lib/galaxy/jobs/splitters/multi.py", >> line 128, in do_merge >> output_type = outputs[output][0].datatype >> AttributeError: 'Dataset' object has no attribute 'datatype' >> >> I could just change both methods to put an HDA in the list inside the >> dict there, but I haven't looked much into what output_dataset_paths >> is used for, so I wasn't sure what that might break. > > Sorta answered it myself, it looks like you created this precisely for > do_merge(), so changing it to contain the HDA fixes the problem (and > shouldn't break anything else). > > --nate > >> >> Thanks, >> --nate >> >>> John Duddy >>> Sr. Staff Software Engineer >>> Illumina, Inc. >>> 9885 Towne Centre Drive >>> San Diego, CA 92121 >>> Tel: 858-736-3584 >>> E-mail: jdu...@illumina.com >>> >>> >>> -----Original Message----- >>> From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] >>> Sent: Thursday, November 03, 2011 2:22 PM >>> To: Duddy, John >>> Cc: Chorny, Ilya; galaxy-dev@lists.bx.psu.edu >>> Subject: Re: Looks like actual user breaks splitting >>> >>> Hi John, >>> >>> It looks like the first issue is related to the change from >>> get_output_fnames() -> compute_outputs(). When >>> outputs_to_working_directory = False (default) this method >>> stores/returns a HistoryDatasetAssociation, but when True, >>> stores/returns a Dataset (the original method's behavior). Thus, >>> accessing the object's .datatype attribute in the splitter's >>> do_merge() fails. >>> >>> Thanks, >>> --nate >>> >>> Duddy, John wrote: >>>> I'll submit a pull request shortly... >>>> >>>> John Duddy >>>> Sr. Staff Software Engineer >>>> Illumina, Inc. >>>> 9885 Towne Centre Drive >>>> San Diego, CA 92121 >>>> Tel: 858-736-3584 >>>> E-mail: jdu...@illumina.com >>>> >>>> >>>> -----Original Message----- >>>> From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] >>>> Sent: Wednesday, November 02, 2011 12:24 PM >>>> To: Duddy, John >>>> Cc: Chorny, Ilya; galaxy-dev@lists.bx.psu.edu >>>> Subject: Re: Looks like actual user breaks splitting >>>> >>>> John, Ilya, >>>> >>>> I get further with sequence type inputs but it looks like >>>> JobWrapper.get_output_datasets_and_fnames() is not returning the >>>> right thing when outputs_to_working_directory = True >>>> >>>> BTW, the base Data.split() method is broken after the updates to >>>> Sequence.split() since it wasn't updated to expect >>>> HistoryDatasetAssociations rather than filenames. Could you take >>>> a look at that when you get a chance? >>>> >>>> --nate >>>> >>>> Duddy, John wrote: >>>>> The datatype you are using does not define a split method. Are you >>>>> working with our in-progress gz type or fastqillumina? >>>>> >>>>> John Duddy >>>>> Sr. Staff Software Engineer >>>>> Illumina, Inc. >>>>> 9885 Towne Centre Drive >>>>> San Diego, CA 92121 >>>>> Tel: 858-736-3584 >>>>> E-mail: jdu...@illumina.com<mailto:jdu...@illumina.com> >>>>> >>>>> From: Chorny, Ilya >>>>> Sent: Wednesday, November 02, 2011 11:50 AM >>>>> To: Duddy, John >>>>> Cc: Nate Coraor (n...@bx.psu.edu); galaxy-dev@lists.bx.psu.edu >>>>> Subject: Looks like actual user breaks splitting >>>>> >>>>> Hey John, >>>>> >>>>> Any thoughts? >>>>> >>>>> Ilya >>>>> >>>>> Traceback (most recent call last): >>>>> File >>>>> "/home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/runners/tasks.py", >>>>> line 73, in run_job >>>>> tasks = splitter.do_split(job_wrapper) >>>>> File >>>>> "/home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/splitters/multi.py", >>>>> line 73, in do_split >>>>> input_type.split(input_datasets, get_new_working_directory_name, >>>>> parallel_settings) >>>>> File "/home/galaxy/ichorny/galaxy-central/lib/galaxy/datatypes/data.py", >>>>> line 473, in split >>>>> raise Exception("Text file splitting does not support >>>>> multiple files") >>>>> Exception: Text file splitting does not support multiple files >>>>> >>>>> Ilya Chorny Ph.D. >>>>> Bioinformatics Scientist I >>>>> Illumina, Inc. >>>>> 9885 Towne Centre Drive >>>>> San Diego, CA 92121 >>>>> Work: 858.202.4582 >>>>> Email: icho...@illumina.com<mailto:icho...@illumina.com> >>>>> Website: www.illumina.com<http://www.illumina.com> >>>>> >>>>> >>>> >>> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this and other >> Galaxy lists, please use the interface at: >> >> http://lists.bx.psu.edu/ > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this and other Galaxy > lists, please use the interface at: > > http://lists.bx.psu.edu/ > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this and other Galaxy > lists, please use the interface at: > > http://lists.bx.psu.edu/ > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this and other Galaxy > lists, please use the interface at: > > http://lists.bx.psu.edu/ > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this and other Galaxy > lists, please use the interface at: > > http://lists.bx.psu.edu/ > ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
