I don't think it makes sense to include the create endpoint without also including dataset update and delete endpoints and updating the Datasets view in the UI to be able to manage externally created Datasets.
With that said, I don't think the fact that Datasets are tightly coupled with DAGs is a good reason not to include additional Dataset endpoints. It makes sense to me to be able to interact with Datasets from outside of Airflow. On Sat, Jan 20, 2024 at 6:13 AM Eduardo Nicastro <edu.nicas...@gmail.com> wrote: > Hello all, I have created a Pull Request ( > https://github.com/apache/airflow/pull/36929) to make it possible to > create a dataset through the API as a modest step forward. This PR is open > for your feedback. I'm preparing another PR to build upon the insights from > https://github.com/apache/airflow/pull/29433. Your thoughts and > contributions are highly encouraged. > > Best Regards, > Eduardo Nicastro > > On Thu, Jan 11, 2024 at 4:30 PM Eduardo Nicastro <edu.nicas...@gmail.com> > wrote: > >> Hello all, >> >> I'm reaching out to propose a topic for discussion that has recently >> emerged in our GitHub discussion threads (#36723 >> <https://github.com/apache/airflow/discussions/36723>). It revolves >> around enhancing the management of datasets in a multi-tenant Airflow >> architecture. >> >> Use case/motivation >> In our multi-instance setup, synchronizing dataset dependencies across >> instances poses significant challenges. With the advent of dataset >> listeners, a new door has opened for cross-instance dataset awareness. I >> propose we explore creating endpoints to export dataset updates to make it >> possible to trigger DAGs consuming from a Dataset across tenants. >> >> Context >> Below I will give some context about our current situation and solution >> we have in place and propose a new workflow that would be more efficient. >> To be able to implement this new workflow we would need a way to export >> Dataset updates as mentioned. >> >> Current Workflow >> In our organization, we're dealing with multiple Airflow tenants, let's >> say Tenant 1 and Tenant 2, as examples. To synchronize Dataset A across >> these tenants, we currently have a complex setup: >> >> 1. Containers run on a schedule to export metadata to CosmosDB (these >> will be replaced by the listener). >> 2. Additional scheduled containers pull data from CosmosDB and write >> it to a shared file system, enabling generated DAGS to read it and mirror >> a >> dataset across tenants. >> >> >> Proposed Workflow >> Here's a breakdown of our proposed workflow: >> >> 1. Cross-Tenant Dataset Interaction: We have Dags in Tenant 1 >> producing Dataset A. We need a mechanism to trigger all Dags consuming >> Dataset A in Tenant 2. This interaction is crucial for our data pipeline's >> efficiency and synchronicity. >> 2. Dataset Listener Implementation: Our approach involves >> implementing a Dataset listener that programmatically creates Dataset A in >> all tenants where it's not present (like Tenant 2) and export Dataset >> updates when they happen. This would trigger an update on all Dags >> consuming from that Dataset. >> 3. Standardized Dataset Names: We plan to use standardized dataset >> names, which makes sense since a URI is its identifier and uniqueness is a >> logical requirement. >> >> [image: image.png] >> >> Why This Matters: >> >> - It offers a streamlined, automated way to manage datasets across >> different Airflow instances. >> - It aligns with a need for efficient, interconnected workflows in a >> multi-tenant environment. >> >> >> I invite the community to discuss: >> >> - Are there alternative methods within Airflow's current framework >> that could achieve similar goals? >> - Any insights or experiences that could inform our approach? >> >> Your feedback and suggestions are invaluable, and I look forward to a >> collaborative discussion. >> >> Best Regards, >> Eduardo Nicastro >> >
