Hi IPMC, I would like to propose OpenDAL[1] as a new apache incubator project, you can find the proposal[2] of OpenDAL for more detail.
[1] https://github.com/datafuselabs/opendal [2] https://cwiki.apache.org/confluence/display/INCUBATOR/OpenDAL+Proposal Best, tison. Full text of the proposal below: ## Abstract OpenDAL means “Open Data Access Layer”. It’s a Rust library that helps developers access data freely, painlessly, and efficiently over multiple services, including AWS S3, HDFS, POSIX-compatible file systems, and so on. ## Proposal OpenDAL provides the following features to support developers accessing data freely, painlessly, and efficiently: - Freely - Access different storage services in the same way - Behavior tests for all services - Cross-language/project bindings (working in progress) - Painlessly - **100%** of documents covered - Powerful [Layers]( https://opendal.databend.rs/opendal/layers/index.html) - Automatic [retry]( https://opendal.databend.rs/opendal/layers/struct.RetryLayer.html) support - Full observability support: [logging]( https://opendal.databend.rs/opendal/layers/struct.LoggingLayer.html ), [tracing]( https://opendal.databend.rs/opendal/layers/struct.TracingLayer.html), and [metrics]( https://opendal.databend.rs/opendal/layers/struct.MetricsLayer.html). - Native chaos testing - Native service-side encryption support - Efficiently - Zero cost: mapping to underlying API calls directly - Best effort: auto-pick the best `read`/`seek`/`next` implementations based on services - [Auto metadata reuse]( https://opendal.databend.rs/rfcs/0561-list-metadata-reuse.html): avoid extra `metadata` calls OpenDAL was originally designed to be used by the Databend project but is now being used by Mozilla's sccache, DeepETH's mars, and several database startups. We believe that the OpenDAL project will provide diversity value to the community if OpenDAL is brought into the Apache incubator. ## Background OpenDAL is being developed by an open-source community from day one and is owned by DatafuseLabs. The project has been launched in February 2022. ## Rationale OpenDAL provides a unified storage abstraction layer that simplifies the interfacing of different storage services. In addition, OpenDAL provides further advanced storage encapsulation, enabling enhancements such as automatic retry, request optimization, and observability. OpenDAL makes it possible to develop once and run on any storage service. ## Initial Goals By transferring ownership of the project to the ASF, OpenDAL expects to ensure its neutrality and further encourage and facilitate the adoption of OpenDAL by the community. Some of the areas we would like to focus on during the Apache incubation phase include: - A healthier community: more maintainers and contributors will be able to participate in OpenDAL and own different modules. - Wider adoption: OpenDAL can be adopted by more open source/commercial projects, which in turn drives its own functionality. - Richer integration: OpenDAL enables greater integration of storage services and offers a wider range of language bindings. ## Current Status ### Meritocracy We intend to radically expand the initial developer and user community by running the project the 'Apache way'. Users and new contributors will be respected and welcomed. They will earn credit by participating in the community and providing quality patches/support to move the project forward. They will also be encouraged to provide non-code contributions (documentation, events, community management, etc.) and will be rewarded accordingly. Those with a proven track record of support and quality will be encouraged to become committers. ### Community Contributors: 37 Users: - Databend: A cloud data warehouse - GreptimeDB: A time-series database - Sccache: ccache with cloud storage - RisingWave: A Distributed SQL Database for Stream Processing ### Core Developers The core developers are all experienced open-source developers. They have been running the OpenDAL community for 1 year. ### Alignment ## Known Risks ### Project Name We have checked and believe that the name is appropriate and that the project has legal permission to continue using its current name. There are no other projects with this name found in a Google search. ### Orphan Products ### Inexperience with Open Source OpenDAL's core developers are all experienced open source contributors, and its main maintainer Xuanwo has 10 years of open source experience, having worked on a number of open source projects including Hexo, TiDB, TiKV, Databend, Sccache, and others. ### Length of Incubation Expect to enter incubation in two months and graduate in about two years. ### Homogenous Developers OpenDAL developers come from a variety of backgrounds and contribute to the OpenDAL project for different usage scenarios. ### Reliance on Salaried Developers Most developers are paid by their employers to contribute to this project. It's a big risk indeed, we expect to attract more maintainers and contributors from outside the DatafuseLabs to address this. ### Relationships with Other Apache Products - OpenDAL can be used to operate files such as parquet, avro, … - OpenDAL provides a shim that can be used with arrow object_store ### An Excessive Fascination with the Apache Brand ## Documentation The document of OpenDAL is hosted at [ https://opendal.databend.rs/](https://opendal.databend.rs/). And opendal’s document is self-contained, all its current and historical versions could be found at [ https://docs.rs/opendal/latest/opendal/](https://docs.rs/opendal/latest/opendal/) . ## Initial Source The project currently holds a GitHub repository and a Cargo crate: - [ https://github.com/datafuselabs/opendal](https://github.com/datafuselabs/opendal) - [https://crates.io/crates/opendal](https://crates.io/crates/opendal) - [https://opendal.databend.rs/](https://opendal.databend.rs/) The crate will retain its name, while the repository will be moved to apache org, and the website will be permanently moved to [opendal.apache.org ](http://opendal.apache.org) if the proposal gets accepted. ## Source and Intellectual Property Submission Plan ### External Dependencies > Generated by `cargo deny list` - Apache-2.0 (192): - ahash@0.7.6 - anes@0.1.6 - anyhow@1.0.68 - async-channel@1.8.0 - async-compat@0.2.1 - async-trait@0.1.60 - autocfg@1.1.0 - backon@0.2.0 - base64@0.13.1 - base64@0.21.0 - bitflags@1.3.2 - block-buffer@0.10.3 - bumpalo@3.11.1 - cast@0.3.0 - cc@1.0.78 - cfg-if@1.0.0 - ciborium@0.2.0 - ciborium-io@0.2.0 - ciborium-ll@0.2.0 - clap@3.2.23 - clap_lex@0.2.4 - concurrent-queue@2.0.0 - core-foundation@0.9.3 - core-foundation-sys@0.8.3 - cpufeatures@0.2.5 - criterion@0.4.0 - criterion-plot@0.5.0 - crossbeam-channel@0.5.6 - crossbeam-deque@0.8.2 - crossbeam-epoch@0.9.13 - crossbeam-utils@0.8.14 - crypto-common@0.1.6 - ctor@0.1.26 - deadpool@0.9.5 - deadpool-runtime@0.1.2 - diff@0.1.13 - digest@0.10.6 - dirs@4.0.0 - dirs-sys@0.3.7 - either@1.8.0 - encoding_rs@0.8.31 - env_logger@0.10.0 - event-listener@2.5.3 - fastrand@1.8.0 - flagset@0.4.3 - fnv@1.0.7 - form_urlencoded@1.1.0 - futures@0.3.25 - futures-channel@0.3.25 - futures-core@0.3.25 - futures-executor@0.3.25 - futures-io@0.3.25 - futures-lite@1.12.0 - futures-macro@0.3.25 - futures-sink@0.3.25 - futures-task@0.3.25 - futures-timer@3.0.2 - futures-util@0.3.25 - getrandom@0.1.16 - getrandom@0.2.8 - half@1.8.2 - hashbrown@0.12.3 - hermit-abi@0.1.19 - hermit-abi@0.2.6 - hex@0.4.3 - hmac@0.12.1 - http@0.2.8 - http-types@2.12.0 - httparse@1.8.0 - httpdate@1.0.2 - humantime@2.1.0 - hyper-rustls@0.23.2 - idna@0.3.0 - indexmap@1.9.2 - io-lifetimes@1.0.3 - ipnet@2.7.0 - itertools@0.10.5 - itoa@1.0.5 - js-sys@0.3.60 - lazy_static@1.4.0 - libc@0.2.139 - linux-raw-sys@0.1.4 - lock_api@0.4.9 - log@0.4.17 - md-5@0.10.5 - mime@0.3.16 - num-bigint@0.4.3 - num-integer@0.1.45 - num-traits@0.2.15 - num_cpus@1.15.0 - once_cell@1.17.0 - opendal@0.25.1 - openssl-probe@0.1.5 - opentelemetry@0.17.0 - opentelemetry-jaeger@0.16.0 - opentelemetry-semantic-conventions@0.9.0 - os_str_bytes@6.4.1 - parking@2.0.0 - parking_lot@0.12.1 - parking_lot_core@0.9.5 - paste@1.0.11 - percent-encoding@2.2.0 - pin-project@1.0.12 - pin-project-internal@1.0.12 - pin-project-lite@0.2.9 - pin-utils@0.1.0 - ppv-lite86@0.2.17 - pretty_assertions@1.3.0 - proc-macro2@1.0.49 - quote@1.0.23 - rand@0.7.3 - rand@0.8.5 - rand_chacha@0.2.2 - rand_chacha@0.3.1 - rand_core@0.5.1 - rand_core@0.6.4 - rand_hc@0.2.0 - rayon@1.6.1 - rayon-core@1.10.1 - regex@1.7.0 - regex-syntax@0.6.28 - reqsign@0.8.1 - reqwest@0.11.13 - rustix@0.36.6 - rustls@0.20.7 - rustls-native-certs@0.6.2 - rustls-pemfile@1.0.1 - ryu@1.0.12 - scopeguard@1.1.0 - sct@0.7.0 - security-framework@2.7.0 - security-framework-sys@2.6.1 - serde@1.0.152 - serde_derive@1.0.152 - serde_json@1.0.91 - serde_qs@0.8.5 - serde_urlencoded@0.7.1 - sha1@0.10.5 - sha2@0.10.6 - smallvec@1.10.0 - socket2@0.4.7 - syn@1.0.107 - thiserror@1.0.38 - thiserror-impl@1.0.38 - thread_local@1.1.4 - threadpool@1.8.1 - thrift@0.15.0 - time@0.3.17 - time-core@0.1.0 - time-macros@0.2.6 - tinytemplate@1.2.1 - tinyvec@1.6.0 - tinyvec_macros@0.1.0 - tokio-rustls@0.23.4 - typenum@1.16.0 - unicase@2.6.0 - unicode-bidi@0.3.8 - unicode-ident@1.0.6 - unicode-normalization@0.1.22 - ureq@2.6.1 - url@2.3.1 - uuid@1.2.2 - version_check@0.9.4 - waker-fn@1.1.0 - wasi@0.11.0+wasi-snapshot-preview1 - wasi@0.9.0+wasi-snapshot-preview1 - wasm-bindgen@0.2.83 - wasm-bindgen-backend@0.2.83 - wasm-bindgen-futures@0.4.33 - wasm-bindgen-macro@0.2.83 - wasm-bindgen-macro-support@0.2.83 - wasm-bindgen-shared@0.2.83 - web-sys@0.3.60 - winapi@0.3.9 - winapi-i686-pc-windows-gnu@0.4.0 - winapi-x86_64-pc-windows-gnu@0.4.0 - windows-sys@0.36.1 - windows-sys@0.42.0 - windows_aarch64_gnullvm@0.42.0 - windows_aarch64_msvc@0.36.1 - windows_aarch64_msvc@0.42.0 - windows_i686_gnu@0.36.1 - windows_i686_gnu@0.42.0 - windows_i686_msvc@0.36.1 - windows_i686_msvc@0.42.0 - windows_x86_64_gnu@0.36.1 - windows_x86_64_gnu@0.42.0 - windows_x86_64_gnullvm@0.42.0 - windows_x86_64_msvc@0.36.1 - windows_x86_64_msvc@0.42.0 - wiremock@0.5.16 - yansi@0.5.1 - Apache-2.0 WITH LLVM-exception (5) - io-lifetimes@1.0.3 - linux-raw-sys@0.1.4 - rustix@0.36.6 - wasi@0.11.0+wasi-snapshot-preview1 - wasi@0.9.0+wasi-snapshot-preview1 - BSD-3-Clause (3) - encoding_rs@0.8.31 - instant@0.1.12 - subtle@2.4.1 - BSL-1.0 (1): - ryu@1.0.12 - ISC (8): - hyper-rustls@0.23.2 - rustls@0.20.7 - rustls-native-certs@0.6.2 - rustls-pemfile@1.0.1 - sct@0.7.0 - simple_asn1@0.6.2 - untrusted@0.7.1 - webpki@0.22.0 - MIT (242) - ahash@0.7.6 - aho-corasick@0.7.20 - anes@0.1.6 - anyhow@1.0.68 - assert-json-diff@2.0.2 - async-channel@1.8.0 - async-compat@0.2.1 - async-trait@0.1.60 - atty@0.2.14 - autocfg@1.1.0 - base64@0.13.1 - base64@0.21.0 - bincode@2.0.0-rc.2 - bincode_derive@2.0.0-rc.2 - bitflags@1.3.2 - block-buffer@0.10.3 - bumpalo@3.11.1 - byteorder@1.4.3 - bytes@1.3.0 - cast@0.3.0 - cc@1.0.78 - cfg-if@1.0.0 - clap@3.2.23 - clap_lex@0.2.4 - concurrent-queue@2.0.0 - core-foundation@0.9.3 - core-foundation-sys@0.8.3 - cpufeatures@0.2.5 - criterion@0.4.0 - criterion-plot@0.5.0 - crossbeam-channel@0.5.6 - crossbeam-deque@0.8.2 - crossbeam-epoch@0.9.13 - crossbeam-utils@0.8.14 - crypto-common@0.1.6 - ctor@0.1.26 - deadpool@0.9.5 - deadpool-runtime@0.1.2 - diff@0.1.13 - digest@0.10.6 - dirs@4.0.0 - dirs-sys@0.3.7 - dlv-list@0.3.0 - dotenvy@0.15.6 - either@1.8.0 - encoding_rs@0.8.31 - env_logger@0.10.0 - event-listener@2.5.3 - fastrand@1.8.0 - fnv@1.0.7 - form_urlencoded@1.1.0 - futures@0.3.25 - futures-channel@0.3.25 - futures-core@0.3.25 - futures-executor@0.3.25 - futures-io@0.3.25 - futures-lite@1.12.0 - futures-macro@0.3.25 - futures-sink@0.3.25 - futures-task@0.3.25 - futures-timer@3.0.2 - futures-util@0.3.25 - generic-array@0.14.6 - getrandom@0.1.16 - getrandom@0.2.8 - h2@0.3.15 - half@1.8.2 - hashbrown@0.12.3 - hermit-abi@0.1.19 - hermit-abi@0.2.6 - hex@0.4.3 - hmac@0.12.1 - http@0.2.8 - http-body@0.4.5 - http-types@2.12.0 - httparse@1.8.0 - httpdate@1.0.2 - humantime@2.1.0 - hyper@0.14.23 - hyper-rustls@0.23.2 - idna@0.3.0 - indexmap@1.9.2 - infer@0.2.3 - integer-encoding@3.0.4 - io-lifetimes@1.0.3 - ipnet@2.7.0 - is-terminal@0.4.2 - itertools@0.10.5 - itoa@1.0.5 - js-sys@0.3.60 - jsonwebtoken@8.2.0 - lazy_static@1.4.0 - libc@0.2.139 - linux-raw-sys@0.1.4 - lock_api@0.4.9 - log@0.4.17 - matchers@0.1.0 - md-5@0.10.5 - memchr@2.5.0 - memoffset@0.7.1 - mime@0.3.16 - mime_guess@2.0.4 - mio@0.8.5 - nu-ansi-term@0.46.0 - num-bigint@0.4.3 - num-integer@0.1.45 - num-traits@0.2.15 - num_cpus@1.15.0 - once_cell@1.17.0 - oorandom@11.1.3 - openssl-probe@0.1.5 - ordered-float@1.1.1 - ordered-multimap@0.4.3 - os_str_bytes@6.4.1 - output_vt100@0.1.3 - overload@0.1.1 - parking@2.0.0 - parking_lot@0.12.1 - parking_lot_core@0.9.5 - paste@1.0.11 - pem@1.1.0 - percent-encoding@2.2.0 - pin-project@1.0.12 - pin-project-internal@1.0.12 - pin-project-lite@0.2.9 - pin-utils@0.1.0 - plotters@0.3.4 - plotters-backend@0.3.4 - plotters-svg@0.3.3 - ppv-lite86@0.2.17 - pretty_assertions@1.3.0 - proc-macro2@1.0.49 - quick-xml@0.27.1 - quote@1.0.23 - rand@0.7.3 - rand@0.8.5 - rand_chacha@0.2.2 - rand_chacha@0.3.1 - rand_core@0.5.1 - rand_core@0.6.4 - rand_hc@0.2.0 - rayon@1.6.1 - rayon-core@1.10.1 - redox_syscall@0.2.16 - redox_users@0.4.3 - regex@1.7.0 - regex-automata@0.1.10 - regex-syntax@0.6.28 - reqwest@0.11.13 - retain_mut@0.1.9 - rust-ini@0.18.0 - rustix@0.36.6 - rustls@0.20.7 - rustls-native-certs@0.6.2 - rustls-pemfile@1.0.1 - same-file@1.0.6 - schannel@0.1.20 - scopeguard@1.1.0 - sct@0.7.0 - security-framework@2.7.0 - security-framework-sys@2.6.1 - serde@1.0.152 - serde_derive@1.0.152 - serde_json@1.0.91 - serde_qs@0.8.5 - serde_urlencoded@0.7.1 - sha1@0.10.5 - sha2@0.10.6 - sharded-slab@0.1.4 - size@0.4.1 - slab@0.4.7 - smallvec@1.10.0 - socket2@0.4.7 - spin@0.5.2 - syn@1.0.107 - termcolor@1.1.3 - textwrap@0.16.0 - thiserror@1.0.38 - thiserror-impl@1.0.38 - thread_local@1.1.4 - threadpool@1.8.1 - time@0.3.17 - time-core@0.1.0 - time-macros@0.2.6 - tinytemplate@1.2.1 - tinyvec@1.6.0 - tinyvec_macros@0.1.0 - tokio@1.23.1 - tokio-macros@1.8.2 - tokio-rustls@0.23.4 - tokio-util@0.7.4 - tower-service@0.3.2 - tracing@0.1.37 - tracing-core@0.1.30 - tracing-log@0.1.3 - tracing-opentelemetry@0.17.4 - tracing-subscriber@0.3.16 - try-lock@0.2.3 - typenum@1.16.0 - unicase@2.6.0 - unicode-bidi@0.3.8 - unicode-ident@1.0.6 - unicode-normalization@0.1.22 - ureq@2.6.1 - url@2.3.1 - uuid@1.2.2 - valuable@0.1.0 - version_check@0.9.4 - virtue@0.0.8 - waker-fn@1.1.0 - walkdir@2.3.2 - want@0.3.0 - wasi@0.11.0+wasi-snapshot-preview1 - wasi@0.9.0+wasi-snapshot-preview1 - wasm-bindgen@0.2.83 - wasm-bindgen-backend@0.2.83 - wasm-bindgen-futures@0.4.33 - wasm-bindgen-macro@0.2.83 - wasm-bindgen-macro-support@0.2.83 - wasm-bindgen-shared@0.2.83 - web-sys@0.3.60 - winapi@0.3.9 - winapi-i686-pc-windows-gnu@0.4.0 - winapi-util@0.1.5 - winapi-x86_64-pc-windows-gnu@0.4.0 - windows-sys@0.36.1 - windows-sys@0.42.0 - windows_aarch64_gnullvm@0.42.0 - windows_aarch64_msvc@0.36.1 - windows_aarch64_msvc@0.42.0 - windows_i686_gnu@0.36.1 - windows_i686_gnu@0.42.0 - windows_i686_msvc@0.36.1 - windows_i686_msvc@0.42.0 - windows_x86_64_gnu@0.36.1 - windows_x86_64_gnu@0.42.0 - windows_x86_64_gnullvm@0.42.0 - windows_x86_64_msvc@0.36.1 - windows_x86_64_msvc@0.42.0 - winreg@0.10.1 - wiremock@0.5.16 - yansi@0.5.1 - MPL-2.0 (1): - webpki-roots@0.22.6 - Unicode-DFS-2016 (1): - unicode-ident@1.0.6 - Unlicense (8): - aho-corasick@0.7.20 - byteorder@1.4.3 - memchr@2.5.0 - regex-automata@0.1.10 - same-file@1.0.6 - termcolor@1.1.3 - walkdir@2.3.2 - winapi-util@0.1.5 - Zlib (2): - tinyvec@1.6.0 - tinyvec_macros@0.1.0 - Unlicensed (1): - ring@0.16.20 ring is licensed under a mix of [MIT, ISC, and OpenSSL’s licenses]( https://github.com/briansmith/ring/blob/main/LICENSE). OpenSSL itself is licensed under [Apache 2.0]( https://www.openssl.org/source/apache-license-2.0.txt). So ring is Apache 2.0 compatible. ### Cryptography N/A ## Required Resources ### Mailing Lists - priv...@opendal.apache.org - d...@opendal.apache.org - comm...@opendal.apache.org - iss...@opendal.apache.org - bu...@opendal.apache.org ### Subversion Directory N/A ### Git Repositories >From [ https://github.com/datafuselabs/opendal](https://github.com/datafuselabs/opendal) - [ https://gitbox.apache.org/asf/repos/incubator-opendal](https://gitbox.apache.org/asf/repos/incubator-opendal) - [ https://github.com/apache/incubator-opendal](https://github.com/apache/incubator-opendal) ### Issue Tracking The community would like to continue using GitHub Issues. ### Other Resources The community has already chosen GitHub actions as continuous integration tools. ## Initial Committers - Hao Ding [[git...@xuanwo.io](mailto:git...@xuanwo.io)] - Lue Cai [[cai...@bupt.edu.cn](mailto:cai...@bupt.edu.cn)] - Zhuoran Shang [[psi...@outlook.com](mailto:psi...@outlook.com)] - Benwang Li [[libenw1...@gmail.com](mailto:libenw1...@gmail.com)] ## Sponsors ### Champion - tison [ti...@apache.org] ### Nominated Mentors - hexiaoqiao [hexiaoq...@apache.org] - tedliu [ted...@apache.org] - tison [ti...@apache.org] - wusheng [wush...@apache.org] ### Sponsor Entity The Incubator
