Re: [galaxy-dev] Multi File upload api
This changeset is where I implemented collecting and assigning names for multiple file datasets: https://bitbucket.org/msiappdev/galaxy-extras/commits/bfb07a8c8b7b4c404339cd1efe50f9b2becb3b62 -John On Fri, May 31, 2013 at 1:21 AM, wrote: > Thanks John, > > We do have pretty old code, so I applied your patches. > > One minor problem, in multi_upload tool "Dataset Name" is ignored, just picks > the first file name. Do we need to update upoald.py or multi_upload.xml? > > -Alex > > -Original Message- > From: jmchil...@gmail.com [mailto:jmchil...@gmail.com] On Behalf Of John > Chilton > Sent: Friday, 31 May 2013 5:50 AM > To: Khassapov, Alex (CSIRO IM&T, Clayton) > Cc: Burdett, Neil (ICT Centre, Herston - RBWH); > Subject: Re: Multi File upload api > > Alex, if I am not mistaken you guys are using a fairly old version of the > multiple file dataset code. I think on the server side you would need to > apply the following three changesets to update the library backend to support > the creation of multifile datasets (I guess if you are doing path pasting > only the first changeset is really needed): > > https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=25e137130d32&at=stable > https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=a15ce4c00716&at=stable > https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=d2ae2265bc96&at=stable > > If your version of Galaxy is likewise pretty old you will need pull request > 123 as first > (https://bitbucket.org/galaxy/galaxy-central/pull-request/123/refactor-library_commonpy-toward-more/diff). > > Once those are in place updating example_watch_folder.py should be pretty > doable (Hagai Cohen had it working at some point): > > 1) You will need to replace: > > data['upload_option'] = 'upload_paths' > > with > > data['upload_option'] = 'upload_paths_multifile' > > 2) You will need to specify a name (e.g. add something like: > data['NAME'] = os.path.basename(fullpath)) > > 3) Multiple file datasets do not support data['ext'] = 'auto', so you will > need to specify an explicit format there (the script will hopefully be able > to determine this from the inputs). > > The most updated version of the multiple file dataset stuff > https://bitbucket.org/msiappdev/galaxy-extras/ paired with the most updated > version of blend4j (https://github.com/jmchilton/blend4j) are compatible and > if you do enough digging can provide insights on how to create multiple file > datasets via the API (library path pastes, library server directory uploads, > and history uploads). > > Hope this helps, > -John > > On Thu, May 30, 2013 at 1:40 AM, wrote: >> Hi John, >> >> >> >> Can you please have a look at Neil's question. >> >> >> >> Thank you, >> >> >> >> -Alex >> >> >> >> From: Burdett, Neil (ICT Centre, Herston - RBWH) >> Sent: Thursday, 30 May 2013 4:30 PM >> To: Khassapov, Alex (CSIRO IM&T, Clayton) >> Subject: Multi File upload api >> >> >> >> Hi Alex, >> >> The file "galaxy-dist/scripts/api/example_watch_folder.py" >> allows us to watch a folder then upload files that arrive in a >> specified input folder into the database and history. >> >> >> >> Can you ask your friend (who implemented the multi file upload tool), >> what changes are necessary to this file so we can upload multiple >> files as we do from the GUI. I assume he would know quite quickly what >> to do and hopefully quite simple >> >> >> >> Thanks >> >> Neil ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Multi File upload api
Hi guys, I am sure I will be looking into this shortly but now I want to get the one file case working first :). Do you guys have any idea why the example_watch_folder.pl will copy the file but not its contents? The file it copies into the library folder is empty. How can I go about resolving this issue? I appreciate any help. Regards, -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of alex.khassa...@csiro.au Sent: Friday, May 31, 2013 1:22 AM To: chil...@msi.umn.edu Cc: galaxy-...@bx.psu.edu; neil.burd...@csiro.au Subject: Re: [galaxy-dev] Multi File upload api Thanks John, We do have pretty old code, so I applied your patches. One minor problem, in multi_upload tool "Dataset Name" is ignored, just picks the first file name. Do we need to update upoald.py or multi_upload.xml? -Alex -Original Message- From: jmchil...@gmail.com [mailto:jmchil...@gmail.com] On Behalf Of John Chilton Sent: Friday, 31 May 2013 5:50 AM To: Khassapov, Alex (CSIRO IM&T, Clayton) Cc: Burdett, Neil (ICT Centre, Herston - RBWH); Subject: Re: Multi File upload api Alex, if I am not mistaken you guys are using a fairly old version of the multiple file dataset code. I think on the server side you would need to apply the following three changesets to update the library backend to support the creation of multifile datasets (I guess if you are doing path pasting only the first changeset is really needed): https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=25e137130d32&at=stable https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=a15ce4c00716&at=stable https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=d2ae2265bc96&at=stable If your version of Galaxy is likewise pretty old you will need pull request 123 as first (https://bitbucket.org/galaxy/galaxy-central/pull-request/123/refactor-library_commonpy-toward-more/diff). Once those are in place updating example_watch_folder.py should be pretty doable (Hagai Cohen had it working at some point): 1) You will need to replace: data['upload_option'] = 'upload_paths' with data['upload_option'] = 'upload_paths_multifile' 2) You will need to specify a name (e.g. add something like: data['NAME'] = os.path.basename(fullpath)) 3) Multiple file datasets do not support data['ext'] = 'auto', so you will need to specify an explicit format there (the script will hopefully be able to determine this from the inputs). The most updated version of the multiple file dataset stuff https://bitbucket.org/msiappdev/galaxy-extras/ paired with the most updated version of blend4j (https://github.com/jmchilton/blend4j) are compatible and if you do enough digging can provide insights on how to create multiple file datasets via the API (library path pastes, library server directory uploads, and history uploads). Hope this helps, -John On Thu, May 30, 2013 at 1:40 AM, wrote: > Hi John, > > > > Can you please have a look at Neil's question. > > > > Thank you, > > > > -Alex > > > > From: Burdett, Neil (ICT Centre, Herston - RBWH) > Sent: Thursday, 30 May 2013 4:30 PM > To: Khassapov, Alex (CSIRO IM&T, Clayton) > Subject: Multi File upload api > > > > Hi Alex, > > The file "galaxy-dist/scripts/api/example_watch_folder.py" > allows us to watch a folder then upload files that arrive in a > specified input folder into the database and history. > > > > Can you ask your friend (who implemented the multi file upload tool), > what changes are necessary to this file so we can upload multiple > files as we do from the GUI. I assume he would know quite quickly what > to do and hopefully quite simple > > > > Thanks > > Neil ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Multi File upload api
Thanks John, We do have pretty old code, so I applied your patches. One minor problem, in multi_upload tool "Dataset Name" is ignored, just picks the first file name. Do we need to update upoald.py or multi_upload.xml? -Alex -Original Message- From: jmchil...@gmail.com [mailto:jmchil...@gmail.com] On Behalf Of John Chilton Sent: Friday, 31 May 2013 5:50 AM To: Khassapov, Alex (CSIRO IM&T, Clayton) Cc: Burdett, Neil (ICT Centre, Herston - RBWH); Subject: Re: Multi File upload api Alex, if I am not mistaken you guys are using a fairly old version of the multiple file dataset code. I think on the server side you would need to apply the following three changesets to update the library backend to support the creation of multifile datasets (I guess if you are doing path pasting only the first changeset is really needed): https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=25e137130d32&at=stable https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=a15ce4c00716&at=stable https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=d2ae2265bc96&at=stable If your version of Galaxy is likewise pretty old you will need pull request 123 as first (https://bitbucket.org/galaxy/galaxy-central/pull-request/123/refactor-library_commonpy-toward-more/diff). Once those are in place updating example_watch_folder.py should be pretty doable (Hagai Cohen had it working at some point): 1) You will need to replace: data['upload_option'] = 'upload_paths' with data['upload_option'] = 'upload_paths_multifile' 2) You will need to specify a name (e.g. add something like: data['NAME'] = os.path.basename(fullpath)) 3) Multiple file datasets do not support data['ext'] = 'auto', so you will need to specify an explicit format there (the script will hopefully be able to determine this from the inputs). The most updated version of the multiple file dataset stuff https://bitbucket.org/msiappdev/galaxy-extras/ paired with the most updated version of blend4j (https://github.com/jmchilton/blend4j) are compatible and if you do enough digging can provide insights on how to create multiple file datasets via the API (library path pastes, library server directory uploads, and history uploads). Hope this helps, -John On Thu, May 30, 2013 at 1:40 AM, wrote: > Hi John, > > > > Can you please have a look at Neil's question. > > > > Thank you, > > > > -Alex > > > > From: Burdett, Neil (ICT Centre, Herston - RBWH) > Sent: Thursday, 30 May 2013 4:30 PM > To: Khassapov, Alex (CSIRO IM&T, Clayton) > Subject: Multi File upload api > > > > Hi Alex, > > The file "galaxy-dist/scripts/api/example_watch_folder.py" > allows us to watch a folder then upload files that arrive in a > specified input folder into the database and history. > > > > Can you ask your friend (who implemented the multi file upload tool), > what changes are necessary to this file so we can upload multiple > files as we do from the GUI. I assume he would know quite quickly what > to do and hopefully quite simple > > > > Thanks > > Neil ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Multi File upload api
Alex, if I am not mistaken you guys are using a fairly old version of the multiple file dataset code. I think on the server side you would need to apply the following three changesets to update the library backend to support the creation of multifile datasets (I guess if you are doing path pasting only the first changeset is really needed): https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=25e137130d32&at=stable https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=a15ce4c00716&at=stable https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=d2ae2265bc96&at=stable If your version of Galaxy is likewise pretty old you will need pull request 123 as first (https://bitbucket.org/galaxy/galaxy-central/pull-request/123/refactor-library_commonpy-toward-more/diff). Once those are in place updating example_watch_folder.py should be pretty doable (Hagai Cohen had it working at some point): 1) You will need to replace: data['upload_option'] = 'upload_paths' with data['upload_option'] = 'upload_paths_multifile' 2) You will need to specify a name (e.g. add something like: data['NAME'] = os.path.basename(fullpath)) 3) Multiple file datasets do not support data['ext'] = 'auto', so you will need to specify an explicit format there (the script will hopefully be able to determine this from the inputs). The most updated version of the multiple file dataset stuff https://bitbucket.org/msiappdev/galaxy-extras/ paired with the most updated version of blend4j (https://github.com/jmchilton/blend4j) are compatible and if you do enough digging can provide insights on how to create multiple file datasets via the API (library path pastes, library server directory uploads, and history uploads). Hope this helps, -John On Thu, May 30, 2013 at 1:40 AM, wrote: > Hi John, > > > > Can you please have a look at Neil’s question. > > > > Thank you, > > > > -Alex > > > > From: Burdett, Neil (ICT Centre, Herston - RBWH) > Sent: Thursday, 30 May 2013 4:30 PM > To: Khassapov, Alex (CSIRO IM&T, Clayton) > Subject: Multi File upload api > > > > Hi Alex, > > The file “galaxy-dist/scripts/api/example_watch_folder.py” allows > us to watch a folder then upload files that arrive in a specified input > folder into the database and history. > > > > Can you ask your friend (who implemented the multi file upload tool), what > changes are necessary to this file so we can upload multiple files as we do > from the GUI. I assume he would know quite quickly what to do and hopefully > quite simple > > > > Thanks > > Neil ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Multi File upload api
Hi John, Can you please have a look at Neil's question. Thank you, -Alex From: Burdett, Neil (ICT Centre, Herston - RBWH) Sent: Thursday, 30 May 2013 4:30 PM To: Khassapov, Alex (CSIRO IM&T, Clayton) Subject: Multi File upload api Hi Alex, The file "galaxy-dist/scripts/api/example_watch_folder.py" allows us to watch a folder then upload files that arrive in a specified input folder into the database and history. Can you ask your friend (who implemented the multi file upload tool), what changes are necessary to this file so we can upload multiple files as we do from the GUI. I assume he would know quite quickly what to do and hopefully quite simple Thanks Neil ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
