[jira] [Commented] (AIRFLOW-4030) Add Singularity Container Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17042866#comment-17042866 ] ASF subversion and git services commented on AIRFLOW-4030: -- Commit 0bb687990b94da7445f4ba081592de8cea73119e in airflow's branch refs/heads/master from Vanessasaurus [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=0bb6879 ] [AIRFLOW-4030] second attempt to add singularity to airflow (#7191) * adding singularity operator and tests Signed-off-by: Vanessa Sochat * removing encoding pragmas and fixing up dockerfile to pass linting Signed-off-by: Vanessa Sochat * make workdir in /tmp because AIRFLOW_SOURCES not defined yet Signed-off-by: Vanessa Sochat * curl needs to follow redirects with -L Signed-off-by: Vanessa Sochat * moving files to where they are supposed to be, more changes to mock, no clue Signed-off-by: vsoch * removing trailing whitespace, moving example_dag for singularity, adding licenses to empty init files Signed-off-by: vsoch * ran isort on example dags file Signed-off-by: vsoch * adding missing init in example_dags folder for singularity Signed-off-by: vsoch * removing code from __init__.py files for singularity operator to fix documentation generation Signed-off-by: vsoch * forgot to update link to singularity in operators and hooks ref Signed-off-by: vsoch * command must have been provided on init of singularity operator instance Signed-off-by: vsoch * I guess I'm required to have a task_id? Signed-off-by: vsoch * try adding working_dir to singularity operator type definitions Signed-off-by: vsoch * disable too many arguments for pylint of singularity operator init Signed-off-by: vsoch * move pylint disable up to line 64 - doesnt catch at end of statement like other examples Signed-off-by: vsoch * two spaces before inline comment Signed-off-by: vsoch * I dont see task_id as a param for other providers, removing for singularity operator Signed-off-by: vsoch * adding debug print Signed-off-by: vsoch * allow for return of just image and/or lines Signed-off-by: vsoch * dont understand how mock works, but the image should exist after its pulled Signed-off-by: vsoch * try removing shutil, the client should handle pull folder instead Signed-off-by: vsoch * try changing pull-file to same uri that is expected to be pulled Signed-off-by: vsoch * import of AirflowException moved to exceptions Signed-off-by: vsoch * DAG module was moved to airflow.models Signed-off-by: vsoch * ensure pull is called with pull_folder Signed-off-by: vsoch > Add Singularity Container Operator > -- > > Key: AIRFLOW-4030 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4030 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Reporter: Vanessa >Assignee: Vanessa >Priority: Minor > Fix For: 2.0.0 > > > Singularity containers are comparable to Docker in the level of operation - > they provide an encapsulated environment with an OS, libraries, and custom > software for the user to run. The key difference is that Docker is not > optimized for scientific compute because it could never be installed on a > shared research cluster. Singularity, on the other hand, does not have these > issues and is installed across HCP centers internationally. > This issue is to add Singularity containers as an operator to Apache Airflow, > so that we can start to explore using airflow in an HPC environment. I work > with Encode DCC at Stanford, and am hopeful to explore Airflow as an > alternative to the workflow manager(s) we are using. I am one of the > [original Singularity developers see > |https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0177459] > that manages the Singularity Python client (spython), Singularity Hub and > Singularity Registry Server, and have started working on this issue here: > [https://github.com/apache/airflow/pull/4846.] Looking forward to working > with you! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4030) Add Singularity Container Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17042865#comment-17042865 ] ASF GitHub Bot commented on AIRFLOW-4030: - potiuk commented on pull request #7191: [AIRFLOW-4030] second attempt to add singularity to airflow URL: https://github.com/apache/airflow/pull/7191 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Singularity Container Operator > -- > > Key: AIRFLOW-4030 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4030 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Reporter: Vanessa >Assignee: Vanessa >Priority: Minor > > Singularity containers are comparable to Docker in the level of operation - > they provide an encapsulated environment with an OS, libraries, and custom > software for the user to run. The key difference is that Docker is not > optimized for scientific compute because it could never be installed on a > shared research cluster. Singularity, on the other hand, does not have these > issues and is installed across HCP centers internationally. > This issue is to add Singularity containers as an operator to Apache Airflow, > so that we can start to explore using airflow in an HPC environment. I work > with Encode DCC at Stanford, and am hopeful to explore Airflow as an > alternative to the workflow manager(s) we are using. I am one of the > [original Singularity developers see > |https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0177459] > that manages the Singularity Python client (spython), Singularity Hub and > Singularity Registry Server, and have started working on this issue here: > [https://github.com/apache/airflow/pull/4846.] Looking forward to working > with you! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4030) Add Singularity Container Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017309#comment-17017309 ] ASF GitHub Bot commented on AIRFLOW-4030: - vsoch commented on pull request #7191: [AIRFLOW-4030] second attempt to add singularity to airflow URL: https://github.com/apache/airflow/pull/7191 This is a second attempt to add Singularity Container support to Apache Airflow by way of Singularity Python. I am using the previously created JIRA ticket 4030 (created in March 2019) as it is still relevant. I am a new contributor and largely not familiar with the community here (and yes I've read the guidelines) so I would appreciate support and kindness from the individuals that act as maintainers here, and any additional support from other folks that are also interested in this integration. Thank you! Signed-off-by: Vanessa Sochat --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [ ] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Singularity Container Operator > -- > > Key: AIRFLOW-4030 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4030 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Reporter: Vanessa >Assignee: Vanessa >Priority: Minor > > Singularity containers are comparable to Docker in the level of operation - > they provide an encapsulated environment with an OS, libraries, and custom > software for the user to run. The key difference is that Docker is not > optimized for scientific compute because it could never be installed on a > shared research cluster. Singularity, on the other hand, does not have these > issues and is installed across HCP centers internationally. > This issue is to add Singularity containers as an operator to Apache Airflow, > so that we can start to explore using airflow in an HPC environment. I work > with Encode DCC at Stanford, and am hopeful to explore Airflow as an > alternative to the workflow manager(s) we are using. I am one of the > [original Singularity developers see > |https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0177459] > that manages the Singularity Python client (spython), Singularity Hub and > Singularity Registry Server, and have started working on this issue here: > [https://github.com/apache/airflow/pull/4846.] Looking forward to working > with you! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4030) Add Singularity Container Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017291#comment-17017291 ] ASF GitHub Bot commented on AIRFLOW-4030: - vsoch commented on pull request #4846: [AIRFLOW-4030] adding start to singularity for airflow URL: https://github.com/apache/airflow/pull/4846 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Singularity Container Operator > -- > > Key: AIRFLOW-4030 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4030 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Reporter: Vanessa >Assignee: Vanessa >Priority: Minor > > Singularity containers are comparable to Docker in the level of operation - > they provide an encapsulated environment with an OS, libraries, and custom > software for the user to run. The key difference is that Docker is not > optimized for scientific compute because it could never be installed on a > shared research cluster. Singularity, on the other hand, does not have these > issues and is installed across HCP centers internationally. > This issue is to add Singularity containers as an operator to Apache Airflow, > so that we can start to explore using airflow in an HPC environment. I work > with Encode DCC at Stanford, and am hopeful to explore Airflow as an > alternative to the workflow manager(s) we are using. I am one of the > [original Singularity developers see > |https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0177459] > that manages the Singularity Python client (spython), Singularity Hub and > Singularity Registry Server, and have started working on this issue here: > [https://github.com/apache/airflow/pull/4846.] Looking forward to working > with you! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4030) Add Singularity Container Operator
[
https://issues.apache.org/jira/browse/AIRFLOW-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948742#comment-16948742
]
ASF GitHub Bot commented on AIRFLOW-4030:
-
vsoch commented on pull request #4846: [AIRFLOW-4030] adding start to
singularity for airflow
URL: https://github.com/apache/airflow/pull/4846
Hey airflow maintainers! I'm working on an operator for [Singularity
containers](https://sylabs.io/guides/3.0/user-guide/), and I have a very basic
[example
working](https://github.com/researchapps/airflow-example/tree/master/singularity).
I'd like to do some more substantial tests beyond sleep (and I haven't written
proper tests yet) but wanted to open the pull request to start discussion about
tests and examples needed, etc. I'm new to airflow as of this afternoon, so
thank you (in the future) for teaching me all the things!
I haven't checked all the boxes below. I'm mostly afraid of Jira, so maybe
if someone wants to create an issue for this, would be greatly appreciated!
### Jira
- [x] My PR addresses the following [Airflow JIRA
issue](https://issues.apache.org/jira/browse/AIRFLOW-4030)
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Singularity Container Operator
> --
>
> Key: AIRFLOW-4030
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4030
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
>Reporter: Vanessa
>Assignee: Vanessa
>Priority: Minor
>
> Singularity containers are comparable to Docker in the level of operation -
> they provide an encapsulated environment with an OS, libraries, and custom
> software for the user to run. The key difference is that Docker is not
> optimized for scientific compute because it could never be installed on a
> shared research cluster. Singularity, on the other hand, does not have these
> issues and is installed across HCP centers internationally.
> This issue is to add Singularity containers as an operator to Apache Airflow,
> so that we can start to explore using airflow in an HPC environment. I work
> with Encode DCC at Stanford, and am hopeful to explore Airflow as an
> alternative to the workflow manager(s) we are using. I am one of the
> [original Singularity developers see
> |https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0177459]
> that manages the Singularity Python client (spython), Singularity Hub and
> Singularity Registry Server, and have started working on this issue here:
> [https://github.com/apache/airflow/pull/4846.] Looking forward to working
> with you!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4030) Add Singularity Container Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16921609#comment-16921609 ] ASF GitHub Bot commented on AIRFLOW-4030: - vsoch commented on pull request #4846: [AIRFLOW-4030] adding start to singularity for airflow URL: https://github.com/apache/airflow/pull/4846 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Singularity Container Operator > -- > > Key: AIRFLOW-4030 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4030 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Reporter: Vanessa >Assignee: Vanessa >Priority: Minor > > Singularity containers are comparable to Docker in the level of operation - > they provide an encapsulated environment with an OS, libraries, and custom > software for the user to run. The key difference is that Docker is not > optimized for scientific compute because it could never be installed on a > shared research cluster. Singularity, on the other hand, does not have these > issues and is installed across HCP centers internationally. > This issue is to add Singularity containers as an operator to Apache Airflow, > so that we can start to explore using airflow in an HPC environment. I work > with Encode DCC at Stanford, and am hopeful to explore Airflow as an > alternative to the workflow manager(s) we are using. I am one of the > [original Singularity developers see > |https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0177459] > that manages the Singularity Python client (spython), Singularity Hub and > Singularity Registry Server, and have started working on this issue here: > [https://github.com/apache/airflow/pull/4846.] Looking forward to working > with you! -- This message was sent by Atlassian Jira (v8.3.2#803003)
