Re: [galaxy-dev] Introducing Conda as a new standard for Galaxy tool dependencies
Hi Bjorn, I read the documentation about conda shared early in this email and I have one more question: If I already a conda installation on computer and don't wish share this installation to control galaxy dependencies, galaxy works fine with your own conda auto installation or I have pay attention in more some detail? On Mon, Aug 29, 2016 at 5:37 AM Peter Cock wrote: > That's great Bjoern, > > I was thinking we should mention this on the wiki (with a link > to the documentation rather than duplicating it). There are > some existing pages about the tool_dependencies.xml > system - but no obvious top level introduction that I saw? > > https://wiki.galaxyproject.org/ToolDependencyRecipes > https://wiki.galaxyproject.org/ToolShed/PackageRecipes > https://wiki.galaxyproject.org/ToolDependenciesTagSets > https://wiki.galaxyproject.org/ToolsWithDependenciesInSeparateRepositories > > Where on the wiki do people think would be best? > > Peter > > On Sun, Aug 28, 2016 at 9:41 PM, Léo Biscassi > wrote: > > Nice job! I was really anxious for some documentation about this theme! > > Thanks all involved! > > > > On Fri, Aug 26, 2016 at 3:38 PM Björn Grüning > > > wrote: > >> > >> Hi all, > >> > >> Galaxy 16.07 was just released: > >> https://docs.galaxyproject.org/en/master/releases/16.07_announce.html > >> and the IUC has some news to share with you. > >> > >> For a more readable version please see: > >> https://docs.galaxyproject.org/en/master/admin/conda_faq.html > >> > >> > >> > >> Galaxy tools (also called wrappers) traditionally use Tool Shed package > >> recipes to install their dependencies. At the tool’s installation time > >> the recipe is downloaded and executed in order to provide the underlying > >> software executables. Introduction of these Galaxy-specific recipes was > >> a necessary step at the time, however nowadays there are other more > >> mature and stable options to install software in a similar manner. The > >> Galaxy community has taken steps to improve the tool dependency system > >> in order to enable new features and expand its reach. This document aims > >> to describe these and answer the FAQ. > >> > >> It is a pleasure to announce the adoption of a new standard for tool > >> dependencies in Galaxy which has been integrated over the last six > >> months: Conda packages! > >> Not only do Conda packages make tool dependencies more reliable and > >> stable, they are also easier to test and faster to develop than the > >> traditional Tool Shed package recipes . > >> > >> Conda is a package manager like apt-get, yum, pip, brew or guix. We > >> don’t want to argue about the relative merits of various package > >> managers here, in fact Galaxy supports multiple package managers and we > >> welcome community contributions (such as implementing a Guix package > >> manager or enhancing the existing brew support to bring it on par with > >> Conda). > >> > >> As a community, we have decided that Conda is the one that best fulfills > >> our needs. The following are some of the crucial Conda features that led > >> to this decision: > >> > >> * Installation of packages does not require root privileges > >> (Installation at any location the Galaxy user has write access to) > >> * Multiple versions of software can be installed in parallel > >> * HPC-ready > >> * Faster and more robust package installations through pre-compiled > >> packages (no build environment complications) > >> * Independent of programming language (R, Perl, Python, Julia, Java, > >> pre-compiled binaries, and more) > >> * Easy to write recipes (1 YAML description file + 1 Bash install > script) > >> * An active, large and growing community (with more and more software > >> authors managing their own recipes) > >> * Extensive documentation: Conda documentation > >> (http://conda.pydata.org/docs/building/build.html) and Conda > quick-start > >> (http://conda.pydata.org/docs/get-started.html) > >> > >> Below we answer some common questions (collected by Lance Parsons): > >> > >> 1. How do I enable Conda dependency resolution for existing Galaxy > >> installations? > >> --- > >> > >> Most Galaxy administrators have not set up a dependency resolvers > >> configuration file (dependency_resolvers_conf.xml) which means they are > >> using Galaxy’s default (dependency_resolvers_conf.xml.sample). Galaxy > >> has enabled Conda dependency resolution by default since release 16.04 > >> (if Conda was installed already), so many existing installations can use > >> Conda dependencies. Having Conda enabled in > >> dependency_resolvers_conf.xml means that Galaxy will look for > >> dependencies using the Conda system when it attempts to run tools. If > >> conda_auto_install is True, and a dependency is not found, Galaxy will > >> attempt to install it using the configured Conda channels. A graphical > >> user interface that allows administrators install Conda packages > >> direc
Re: [galaxy-dev] Introducing Conda as a new standard for Galaxy tool dependencies
That's great Bjoern, I was thinking we should mention this on the wiki (with a link to the documentation rather than duplicating it). There are some existing pages about the tool_dependencies.xml system - but no obvious top level introduction that I saw? https://wiki.galaxyproject.org/ToolDependencyRecipes https://wiki.galaxyproject.org/ToolShed/PackageRecipes https://wiki.galaxyproject.org/ToolDependenciesTagSets https://wiki.galaxyproject.org/ToolsWithDependenciesInSeparateRepositories Where on the wiki do people think would be best? Peter On Sun, Aug 28, 2016 at 9:41 PM, Léo Biscassi wrote: > Nice job! I was really anxious for some documentation about this theme! > Thanks all involved! > > On Fri, Aug 26, 2016 at 3:38 PM Björn Grüning > wrote: >> >> Hi all, >> >> Galaxy 16.07 was just released: >> https://docs.galaxyproject.org/en/master/releases/16.07_announce.html >> and the IUC has some news to share with you. >> >> For a more readable version please see: >> https://docs.galaxyproject.org/en/master/admin/conda_faq.html >> >> >> >> Galaxy tools (also called wrappers) traditionally use Tool Shed package >> recipes to install their dependencies. At the tool’s installation time >> the recipe is downloaded and executed in order to provide the underlying >> software executables. Introduction of these Galaxy-specific recipes was >> a necessary step at the time, however nowadays there are other more >> mature and stable options to install software in a similar manner. The >> Galaxy community has taken steps to improve the tool dependency system >> in order to enable new features and expand its reach. This document aims >> to describe these and answer the FAQ. >> >> It is a pleasure to announce the adoption of a new standard for tool >> dependencies in Galaxy which has been integrated over the last six >> months: Conda packages! >> Not only do Conda packages make tool dependencies more reliable and >> stable, they are also easier to test and faster to develop than the >> traditional Tool Shed package recipes . >> >> Conda is a package manager like apt-get, yum, pip, brew or guix. We >> don’t want to argue about the relative merits of various package >> managers here, in fact Galaxy supports multiple package managers and we >> welcome community contributions (such as implementing a Guix package >> manager or enhancing the existing brew support to bring it on par with >> Conda). >> >> As a community, we have decided that Conda is the one that best fulfills >> our needs. The following are some of the crucial Conda features that led >> to this decision: >> >> * Installation of packages does not require root privileges >> (Installation at any location the Galaxy user has write access to) >> * Multiple versions of software can be installed in parallel >> * HPC-ready >> * Faster and more robust package installations through pre-compiled >> packages (no build environment complications) >> * Independent of programming language (R, Perl, Python, Julia, Java, >> pre-compiled binaries, and more) >> * Easy to write recipes (1 YAML description file + 1 Bash install script) >> * An active, large and growing community (with more and more software >> authors managing their own recipes) >> * Extensive documentation: Conda documentation >> (http://conda.pydata.org/docs/building/build.html) and Conda quick-start >> (http://conda.pydata.org/docs/get-started.html) >> >> Below we answer some common questions (collected by Lance Parsons): >> >> 1. How do I enable Conda dependency resolution for existing Galaxy >> installations? >> --- >> >> Most Galaxy administrators have not set up a dependency resolvers >> configuration file (dependency_resolvers_conf.xml) which means they are >> using Galaxy’s default (dependency_resolvers_conf.xml.sample). Galaxy >> has enabled Conda dependency resolution by default since release 16.04 >> (if Conda was installed already), so many existing installations can use >> Conda dependencies. Having Conda enabled in >> dependency_resolvers_conf.xml means that Galaxy will look for >> dependencies using the Conda system when it attempts to run tools. If >> conda_auto_install is True, and a dependency is not found, Galaxy will >> attempt to install it using the configured Conda channels. A graphical >> user interface that allows administrators install Conda packages >> directly from Galaxy when tools are installed from the Tool Shed is >> available in the 16.07 release of Galaxy. >> >> >> 2. How do Conda dependencies work? Where do things get installed? >> - >> In contrast to the old dependency system, which was used exclusively by >> Galaxy, Conda is a pre-existing, independent project. With Conda it is >> possible for an admin to install packages without touching Galaxy at all >> – managing your dependencies independently from Galaxy. Galaxy can >> handle these depend
Re: [galaxy-dev] Introducing Conda as a new standard for Galaxy tool dependencies
Nice job! I was really anxious for some documentation about this theme! Thanks all involved! On Fri, Aug 26, 2016 at 3:38 PM Björn Grüning wrote: > Hi all, > > Galaxy 16.07 was just released: > https://docs.galaxyproject.org/en/master/releases/16.07_announce.html > and the IUC has some news to share with you. > > For a more readable version please see: > https://docs.galaxyproject.org/en/master/admin/conda_faq.html > > > > Galaxy tools (also called wrappers) traditionally use Tool Shed package > recipes to install their dependencies. At the tool’s installation time > the recipe is downloaded and executed in order to provide the underlying > software executables. Introduction of these Galaxy-specific recipes was > a necessary step at the time, however nowadays there are other more > mature and stable options to install software in a similar manner. The > Galaxy community has taken steps to improve the tool dependency system > in order to enable new features and expand its reach. This document aims > to describe these and answer the FAQ. > > It is a pleasure to announce the adoption of a new standard for tool > dependencies in Galaxy which has been integrated over the last six > months: Conda packages! > Not only do Conda packages make tool dependencies more reliable and > stable, they are also easier to test and faster to develop than the > traditional Tool Shed package recipes . > > Conda is a package manager like apt-get, yum, pip, brew or guix. We > don’t want to argue about the relative merits of various package > managers here, in fact Galaxy supports multiple package managers and we > welcome community contributions (such as implementing a Guix package > manager or enhancing the existing brew support to bring it on par with > Conda). > > As a community, we have decided that Conda is the one that best fulfills > our needs. The following are some of the crucial Conda features that led > to this decision: > > * Installation of packages does not require root privileges > (Installation at any location the Galaxy user has write access to) > * Multiple versions of software can be installed in parallel > * HPC-ready > * Faster and more robust package installations through pre-compiled > packages (no build environment complications) > * Independent of programming language (R, Perl, Python, Julia, Java, > pre-compiled binaries, and more) > * Easy to write recipes (1 YAML description file + 1 Bash install script) > * An active, large and growing community (with more and more software > authors managing their own recipes) > * Extensive documentation: Conda documentation > (http://conda.pydata.org/docs/building/build.html) and Conda quick-start > (http://conda.pydata.org/docs/get-started.html) > > Below we answer some common questions (collected by Lance Parsons): > > 1. How do I enable Conda dependency resolution for existing Galaxy > installations? > --- > > Most Galaxy administrators have not set up a dependency resolvers > configuration file (dependency_resolvers_conf.xml) which means they are > using Galaxy’s default (dependency_resolvers_conf.xml.sample). Galaxy > has enabled Conda dependency resolution by default since release 16.04 > (if Conda was installed already), so many existing installations can use > Conda dependencies. Having Conda enabled in > dependency_resolvers_conf.xml means that Galaxy will look for > dependencies using the Conda system when it attempts to run tools. If > conda_auto_install is True, and a dependency is not found, Galaxy will > attempt to install it using the configured Conda channels. A graphical > user interface that allows administrators install Conda packages > directly from Galaxy when tools are installed from the Tool Shed is > available in the 16.07 release of Galaxy. > > > 2. How do Conda dependencies work? Where do things get installed? > - > In contrast to the old dependency system, which was used exclusively by > Galaxy, Conda is a pre-existing, independent project. With Conda it is > possible for an admin to install packages without touching Galaxy at all > – managing your dependencies independently from Galaxy. Galaxy can > handle these dependencies for you, but admins are not required to use > Galaxy for dependency management. > > There are a few new config options in the galaxy.ini file, but by > default Galaxy will install Conda (the package manager) and the required > packages in the /_conda/ directory. In this > directory, Galaxy will create an envs folder with all of the > environments managed by Galaxy. Every environment contains a lib, bin, > share, and include subdirectory, depending on the tool, and is > sufficient to get a Galaxy tool up and running. Galaxy simply sources > this folder via Conda and makes everything available before the tool is > executed on your cluster. > > To summarize, there are four ways to manage
[galaxy-dev] Introducing Conda as a new standard for Galaxy tool dependencies
Hi all, Galaxy 16.07 was just released: https://docs.galaxyproject.org/en/master/releases/16.07_announce.html and the IUC has some news to share with you. For a more readable version please see: https://docs.galaxyproject.org/en/master/admin/conda_faq.html Galaxy tools (also called wrappers) traditionally use Tool Shed package recipes to install their dependencies. At the tool’s installation time the recipe is downloaded and executed in order to provide the underlying software executables. Introduction of these Galaxy-specific recipes was a necessary step at the time, however nowadays there are other more mature and stable options to install software in a similar manner. The Galaxy community has taken steps to improve the tool dependency system in order to enable new features and expand its reach. This document aims to describe these and answer the FAQ. It is a pleasure to announce the adoption of a new standard for tool dependencies in Galaxy which has been integrated over the last six months: Conda packages! Not only do Conda packages make tool dependencies more reliable and stable, they are also easier to test and faster to develop than the traditional Tool Shed package recipes . Conda is a package manager like apt-get, yum, pip, brew or guix. We don’t want to argue about the relative merits of various package managers here, in fact Galaxy supports multiple package managers and we welcome community contributions (such as implementing a Guix package manager or enhancing the existing brew support to bring it on par with Conda). As a community, we have decided that Conda is the one that best fulfills our needs. The following are some of the crucial Conda features that led to this decision: * Installation of packages does not require root privileges (Installation at any location the Galaxy user has write access to) * Multiple versions of software can be installed in parallel * HPC-ready * Faster and more robust package installations through pre-compiled packages (no build environment complications) * Independent of programming language (R, Perl, Python, Julia, Java, pre-compiled binaries, and more) * Easy to write recipes (1 YAML description file + 1 Bash install script) * An active, large and growing community (with more and more software authors managing their own recipes) * Extensive documentation: Conda documentation (http://conda.pydata.org/docs/building/build.html) and Conda quick-start (http://conda.pydata.org/docs/get-started.html) Below we answer some common questions (collected by Lance Parsons): 1. How do I enable Conda dependency resolution for existing Galaxy installations? --- Most Galaxy administrators have not set up a dependency resolvers configuration file (dependency_resolvers_conf.xml) which means they are using Galaxy’s default (dependency_resolvers_conf.xml.sample). Galaxy has enabled Conda dependency resolution by default since release 16.04 (if Conda was installed already), so many existing installations can use Conda dependencies. Having Conda enabled in dependency_resolvers_conf.xml means that Galaxy will look for dependencies using the Conda system when it attempts to run tools. If conda_auto_install is True, and a dependency is not found, Galaxy will attempt to install it using the configured Conda channels. A graphical user interface that allows administrators install Conda packages directly from Galaxy when tools are installed from the Tool Shed is available in the 16.07 release of Galaxy. 2. How do Conda dependencies work? Where do things get installed? - In contrast to the old dependency system, which was used exclusively by Galaxy, Conda is a pre-existing, independent project. With Conda it is possible for an admin to install packages without touching Galaxy at all – managing your dependencies independently from Galaxy. Galaxy can handle these dependencies for you, but admins are not required to use Galaxy for dependency management. There are a few new config options in the galaxy.ini file, but by default Galaxy will install Conda (the package manager) and the required packages in the /_conda/ directory. In this directory, Galaxy will create an envs folder with all of the environments managed by Galaxy. Every environment contains a lib, bin, share, and include subdirectory, depending on the tool, and is sufficient to get a Galaxy tool up and running. Galaxy simply sources this folder via Conda and makes everything available before the tool is executed on your cluster. To summarize, there are four ways to manage Conda dependencies for use with Galaxy. For all of these options, Conda dependency management must be configured in the dependency_resolvers_conf.xml and the galaxy.ini file. a) Manual Install - Conda dependencies may be installed by administrators from the command line. Conda (and thus the conda environments) should be
