[ https://issues.apache.org/jira/browse/AIRFLOW-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17042866#comment-17042866 ]
ASF subversion and git services commented on AIRFLOW-4030: ---------------------------------------------------------- Commit 0bb687990b94da7445f4ba081592de8cea73119e in airflow's branch refs/heads/master from Vanessasaurus [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=0bb6879 ] [AIRFLOW-4030] second attempt to add singularity to airflow (#7191) * adding singularity operator and tests Signed-off-by: Vanessa Sochat <vsoc...@stanford.edu> * removing encoding pragmas and fixing up dockerfile to pass linting Signed-off-by: Vanessa Sochat <vsoc...@stanford.edu> * make workdir in /tmp because AIRFLOW_SOURCES not defined yet Signed-off-by: Vanessa Sochat <vsoc...@stanford.edu> * curl needs to follow redirects with -L Signed-off-by: Vanessa Sochat <vsoc...@stanford.edu> * moving files to where they are supposed to be, more changes to mock, no clue Signed-off-by: vsoch <vsoc...@stanford.edu> * removing trailing whitespace, moving example_dag for singularity, adding licenses to empty init files Signed-off-by: vsoch <vsoc...@stanford.edu> * ran isort on example dags file Signed-off-by: vsoch <vsoc...@stanford.edu> * adding missing init in example_dags folder for singularity Signed-off-by: vsoch <vsoc...@stanford.edu> * removing code from __init__.py files for singularity operator to fix documentation generation Signed-off-by: vsoch <vsoc...@stanford.edu> * forgot to update link to singularity in operators and hooks ref Signed-off-by: vsoch <vsoc...@stanford.edu> * command must have been provided on init of singularity operator instance Signed-off-by: vsoch <vsoc...@stanford.edu> * I guess I'm required to have a task_id? Signed-off-by: vsoch <vsoc...@stanford.edu> * try adding working_dir to singularity operator type definitions Signed-off-by: vsoch <vsoc...@stanford.edu> * disable too many arguments for pylint of singularity operator init Signed-off-by: vsoch <vsoc...@stanford.edu> * move pylint disable up to line 64 - doesnt catch at end of statement like other examples Signed-off-by: vsoch <vsoc...@stanford.edu> * two spaces before inline comment Signed-off-by: vsoch <vsoc...@stanford.edu> * I dont see task_id as a param for other providers, removing for singularity operator Signed-off-by: vsoch <vsoc...@stanford.edu> * adding debug print Signed-off-by: vsoch <vsoc...@stanford.edu> * allow for return of just image and/or lines Signed-off-by: vsoch <vsoc...@stanford.edu> * dont understand how mock works, but the image should exist after its pulled.... Signed-off-by: vsoch <vsoc...@stanford.edu> * try removing shutil, the client should handle pull folder instead Signed-off-by: vsoch <vsoc...@stanford.edu> * try changing pull-file to same uri that is expected to be pulled Signed-off-by: vsoch <vsoc...@stanford.edu> * import of AirflowException moved to exceptions Signed-off-by: vsoch <vsoc...@stanford.edu> * DAG module was moved to airflow.models Signed-off-by: vsoch <vsoc...@stanford.edu> * ensure pull is called with pull_folder Signed-off-by: vsoch <vsoc...@stanford.edu> > Add Singularity Container Operator > ---------------------------------- > > Key: AIRFLOW-4030 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4030 > Project: Apache Airflow > Issue Type: Improvement > Components: operators > Reporter: Vanessa > Assignee: Vanessa > Priority: Minor > Fix For: 2.0.0 > > > Singularity containers are comparable to Docker in the level of operation - > they provide an encapsulated environment with an OS, libraries, and custom > software for the user to run. The key difference is that Docker is not > optimized for scientific compute because it could never be installed on a > shared research cluster. Singularity, on the other hand, does not have these > issues and is installed across HCP centers internationally. > This issue is to add Singularity containers as an operator to Apache Airflow, > so that we can start to explore using airflow in an HPC environment. I work > with Encode DCC at Stanford, and am hopeful to explore Airflow as an > alternative to the workflow manager(s) we are using. I am one of the > [original Singularity developers see > |https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0177459] > that manages the Singularity Python client (spython), Singularity Hub and > Singularity Registry Server, and have started working on this issue here: > [https://github.com/apache/airflow/pull/4846.] Looking forward to working > with you! -- This message was sent by Atlassian Jira (v8.3.4#803005)