+1 Like the slogan of OpenDAL. Storage software is increasingly adopting Rust, OpenDAL hides the complexity of cloud storage and enables users to leverage the benefits of cloud storage easily.
On Fri, Feb 17, 2023 at 7:21 AM tison <wander4...@gmail.com> wrote: > Hi IPMC, > > I would like to propose OpenDAL[1] as a new apache incubator project, you > can find the proposal[2] of OpenDAL for more detail. > > [1] https://github.com/datafuselabs/opendal > [2] https://cwiki.apache.org/confluence/display/INCUBATOR/OpenDAL+Proposal > > Best, > tison. > > Full text of the proposal below: > > ## Abstract > > OpenDAL means “Open Data Access Layer”. It’s a Rust library that helps > developers access data freely, painlessly, and efficiently over multiple > services, including AWS S3, HDFS, POSIX-compatible file systems, and so on. > > ## Proposal > > OpenDAL provides the following features to support developers accessing > data freely, painlessly, and efficiently: > > - Freely > - Access different storage services in the same way > - Behavior tests for all services > - Cross-language/project bindings (working in progress) > - Painlessly > - **100%** of documents covered > - Powerful [Layers]( > https://opendal.databend.rs/opendal/layers/index.html) > - Automatic [retry]( > https://opendal.databend.rs/opendal/layers/struct.RetryLayer.html) support > - Full observability support: [logging]( > https://opendal.databend.rs/opendal/layers/struct.LoggingLayer.html > ), [tracing]( > https://opendal.databend.rs/opendal/layers/struct.TracingLayer.html), and > [metrics]( > https://opendal.databend.rs/opendal/layers/struct.MetricsLayer.html). > - Native chaos testing > - Native service-side encryption support > - Efficiently > - Zero cost: mapping to underlying API calls directly > - Best effort: auto-pick the best `read`/`seek`/`next` implementations > based on services > - [Auto metadata reuse]( > https://opendal.databend.rs/rfcs/0561-list-metadata-reuse.html): avoid > extra `metadata` calls > > OpenDAL was originally designed to be used by the Databend project but is > now being used by Mozilla's sccache, DeepETH's mars, and several database > startups. > > We believe that the OpenDAL project will provide diversity value to the > community if OpenDAL is brought into the Apache incubator. > > ## Background > > OpenDAL is being developed by an open-source community from day one and is > owned by DatafuseLabs. The project has been launched in February 2022. > > ## Rationale > > OpenDAL provides a unified storage abstraction layer that simplifies the > interfacing of different storage services. In addition, OpenDAL provides > further advanced storage encapsulation, enabling enhancements such as > automatic retry, request optimization, and observability. OpenDAL makes it > possible to develop once and run on any storage service. > > ## Initial Goals > > By transferring ownership of the project to the ASF, OpenDAL expects to > ensure its neutrality and further encourage and facilitate the adoption of > OpenDAL by the community. > > Some of the areas we would like to focus on during the Apache incubation > phase include: > > - A healthier community: more maintainers and contributors will be able to > participate in OpenDAL and own different modules. > - Wider adoption: OpenDAL can be adopted by more open source/commercial > projects, which in turn drives its own functionality. > - Richer integration: OpenDAL enables greater integration of storage > services and offers a wider range of language bindings. > > ## Current Status > > ### Meritocracy > > We intend to radically expand the initial developer and user community by > running the project the 'Apache way'. Users and new contributors will be > respected and welcomed. They will earn credit by participating in the > community and providing quality patches/support to move the project > forward. They will also be encouraged to provide non-code contributions > (documentation, events, community management, etc.) and will be rewarded > accordingly. Those with a proven track record of support and quality will > be encouraged to become committers. > > ### Community > > Contributors: 37 > > Users: > > - Databend: A cloud data warehouse > - GreptimeDB: A time-series database > - Sccache: ccache with cloud storage > - RisingWave: A Distributed SQL Database for Stream Processing > > ### Core Developers > > The core developers are all experienced open-source developers. They have > been running the OpenDAL community for 1 year. > > ### Alignment > > ## Known Risks > > ### Project Name > > We have checked and believe that the name is appropriate and that the > project has legal permission to continue using its current name. There are > no other projects with this name found in a Google search. > > ### Orphan Products > > ### Inexperience with Open Source > > OpenDAL's core developers are all experienced open source contributors, and > its main maintainer Xuanwo has 10 years of open source experience, having > worked on a number of open source projects including Hexo, TiDB, TiKV, > Databend, Sccache, and others. > > ### Length of Incubation > > Expect to enter incubation in two months and graduate in about two years. > > ### Homogenous Developers > > OpenDAL developers come from a variety of backgrounds and contribute to the > OpenDAL project for different usage scenarios. > > ### Reliance on Salaried Developers > > Most developers are paid by their employers to contribute to this project. > It's a big risk indeed, we expect to attract more maintainers and > contributors from outside the DatafuseLabs to address this. > > ### Relationships with Other Apache Products > > - OpenDAL can be used to operate files such as parquet, avro, … > - OpenDAL provides a shim that can be used with arrow object_store > > ### An Excessive Fascination with the Apache Brand > > ## Documentation > > The document of OpenDAL is hosted at [ > https://opendal.databend.rs/](https://opendal.databend.rs/). And opendal’s > document is self-contained, all its current and historical versions could > be found at [ > > https://docs.rs/opendal/latest/opendal/](https://docs.rs/opendal/latest/opendal/) > . > > ## Initial Source > > The project currently holds a GitHub repository and a Cargo crate: > > - [ > > https://github.com/datafuselabs/opendal](https://github.com/datafuselabs/opendal) > - [https://crates.io/crates/opendal](https://crates.io/crates/opendal) > - [https://opendal.databend.rs/](https://opendal.databend.rs/) > > The crate will retain its name, while the repository will be moved to > apache org, and the website will be permanently moved to [ > opendal.apache.org > ](http://opendal.apache.org) if the proposal gets accepted. > > ## Source and Intellectual Property Submission Plan > > ### External Dependencies > > > Generated by `cargo deny list` > > - Apache-2.0 (192): > - ahash@0.7.6 > - anes@0.1.6 > - anyhow@1.0.68 > - async-channel@1.8.0 > - async-compat@0.2.1 > - async-trait@0.1.60 > - autocfg@1.1.0 > - backon@0.2.0 > - base64@0.13.1 > - base64@0.21.0 > - bitflags@1.3.2 > - block-buffer@0.10.3 > - bumpalo@3.11.1 > - cast@0.3.0 > - cc@1.0.78 > - cfg-if@1.0.0 > - ciborium@0.2.0 > - ciborium-io@0.2.0 > - ciborium-ll@0.2.0 > - clap@3.2.23 > - clap_lex@0.2.4 > - concurrent-queue@2.0.0 > - core-foundation@0.9.3 > - core-foundation-sys@0.8.3 > - cpufeatures@0.2.5 > - criterion@0.4.0 > - criterion-plot@0.5.0 > - crossbeam-channel@0.5.6 > - crossbeam-deque@0.8.2 > - crossbeam-epoch@0.9.13 > - crossbeam-utils@0.8.14 > - crypto-common@0.1.6 > - ctor@0.1.26 > - deadpool@0.9.5 > - deadpool-runtime@0.1.2 > - diff@0.1.13 > - digest@0.10.6 > - dirs@4.0.0 > - dirs-sys@0.3.7 > - either@1.8.0 > - encoding_rs@0.8.31 > - env_logger@0.10.0 > - event-listener@2.5.3 > - fastrand@1.8.0 > - flagset@0.4.3 > - fnv@1.0.7 > - form_urlencoded@1.1.0 > - futures@0.3.25 > - futures-channel@0.3.25 > - futures-core@0.3.25 > - futures-executor@0.3.25 > - futures-io@0.3.25 > - futures-lite@1.12.0 > - futures-macro@0.3.25 > - futures-sink@0.3.25 > - futures-task@0.3.25 > - futures-timer@3.0.2 > - futures-util@0.3.25 > - getrandom@0.1.16 > - getrandom@0.2.8 > - half@1.8.2 > - hashbrown@0.12.3 > - hermit-abi@0.1.19 > - hermit-abi@0.2.6 > - hex@0.4.3 > - hmac@0.12.1 > - http@0.2.8 > - http-types@2.12.0 > - httparse@1.8.0 > - httpdate@1.0.2 > - humantime@2.1.0 > - hyper-rustls@0.23.2 > - idna@0.3.0 > - indexmap@1.9.2 > - io-lifetimes@1.0.3 > - ipnet@2.7.0 > - itertools@0.10.5 > - itoa@1.0.5 > - js-sys@0.3.60 > - lazy_static@1.4.0 > - libc@0.2.139 > - linux-raw-sys@0.1.4 > - lock_api@0.4.9 > - log@0.4.17 > - md-5@0.10.5 > - mime@0.3.16 > - num-bigint@0.4.3 > - num-integer@0.1.45 > - num-traits@0.2.15 > - num_cpus@1.15.0 > - once_cell@1.17.0 > - opendal@0.25.1 > - openssl-probe@0.1.5 > - opentelemetry@0.17.0 > - opentelemetry-jaeger@0.16.0 > - opentelemetry-semantic-conventions@0.9.0 > - os_str_bytes@6.4.1 > - parking@2.0.0 > - parking_lot@0.12.1 > - parking_lot_core@0.9.5 > - paste@1.0.11 > - percent-encoding@2.2.0 > - pin-project@1.0.12 > - pin-project-internal@1.0.12 > - pin-project-lite@0.2.9 > - pin-utils@0.1.0 > - ppv-lite86@0.2.17 > - pretty_assertions@1.3.0 > - proc-macro2@1.0.49 > - quote@1.0.23 > - rand@0.7.3 > - rand@0.8.5 > - rand_chacha@0.2.2 > - rand_chacha@0.3.1 > - rand_core@0.5.1 > - rand_core@0.6.4 > - rand_hc@0.2.0 > - rayon@1.6.1 > - rayon-core@1.10.1 > - regex@1.7.0 > - regex-syntax@0.6.28 > - reqsign@0.8.1 > - reqwest@0.11.13 > - rustix@0.36.6 > - rustls@0.20.7 > - rustls-native-certs@0.6.2 > - rustls-pemfile@1.0.1 > - ryu@1.0.12 > - scopeguard@1.1.0 > - sct@0.7.0 > - security-framework@2.7.0 > - security-framework-sys@2.6.1 > - serde@1.0.152 > - serde_derive@1.0.152 > - serde_json@1.0.91 > - serde_qs@0.8.5 > - serde_urlencoded@0.7.1 > - sha1@0.10.5 > - sha2@0.10.6 > - smallvec@1.10.0 > - socket2@0.4.7 > - syn@1.0.107 > - thiserror@1.0.38 > - thiserror-impl@1.0.38 > - thread_local@1.1.4 > - threadpool@1.8.1 > - thrift@0.15.0 > - time@0.3.17 > - time-core@0.1.0 > - time-macros@0.2.6 > - tinytemplate@1.2.1 > - tinyvec@1.6.0 > - tinyvec_macros@0.1.0 > - tokio-rustls@0.23.4 > - typenum@1.16.0 > - unicase@2.6.0 > - unicode-bidi@0.3.8 > - unicode-ident@1.0.6 > - unicode-normalization@0.1.22 > - ureq@2.6.1 > - url@2.3.1 > - uuid@1.2.2 > - version_check@0.9.4 > - waker-fn@1.1.0 > - wasi@0.11.0+wasi-snapshot-preview1 > - wasi@0.9.0+wasi-snapshot-preview1 > - wasm-bindgen@0.2.83 > - wasm-bindgen-backend@0.2.83 > - wasm-bindgen-futures@0.4.33 > - wasm-bindgen-macro@0.2.83 > - wasm-bindgen-macro-support@0.2.83 > - wasm-bindgen-shared@0.2.83 > - web-sys@0.3.60 > - winapi@0.3.9 > - winapi-i686-pc-windows-gnu@0.4.0 > - winapi-x86_64-pc-windows-gnu@0.4.0 > - windows-sys@0.36.1 > - windows-sys@0.42.0 > - windows_aarch64_gnullvm@0.42.0 > - windows_aarch64_msvc@0.36.1 > - windows_aarch64_msvc@0.42.0 > - windows_i686_gnu@0.36.1 > - windows_i686_gnu@0.42.0 > - windows_i686_msvc@0.36.1 > - windows_i686_msvc@0.42.0 > - windows_x86_64_gnu@0.36.1 > - windows_x86_64_gnu@0.42.0 > - windows_x86_64_gnullvm@0.42.0 > - windows_x86_64_msvc@0.36.1 > - windows_x86_64_msvc@0.42.0 > - wiremock@0.5.16 > - yansi@0.5.1 > - Apache-2.0 WITH LLVM-exception (5) > - io-lifetimes@1.0.3 > - linux-raw-sys@0.1.4 > - rustix@0.36.6 > - wasi@0.11.0+wasi-snapshot-preview1 > - wasi@0.9.0+wasi-snapshot-preview1 > - BSD-3-Clause (3) > - encoding_rs@0.8.31 > - instant@0.1.12 > - subtle@2.4.1 > - BSL-1.0 (1): > - ryu@1.0.12 > - ISC (8): > - hyper-rustls@0.23.2 > - rustls@0.20.7 > - rustls-native-certs@0.6.2 > - rustls-pemfile@1.0.1 > - sct@0.7.0 > - simple_asn1@0.6.2 > - untrusted@0.7.1 > - webpki@0.22.0 > - MIT (242) > - ahash@0.7.6 > - aho-corasick@0.7.20 > - anes@0.1.6 > - anyhow@1.0.68 > - assert-json-diff@2.0.2 > - async-channel@1.8.0 > - async-compat@0.2.1 > - async-trait@0.1.60 > - atty@0.2.14 > - autocfg@1.1.0 > - base64@0.13.1 > - base64@0.21.0 > - bincode@2.0.0-rc.2 > - bincode_derive@2.0.0-rc.2 > - bitflags@1.3.2 > - block-buffer@0.10.3 > - bumpalo@3.11.1 > - byteorder@1.4.3 > - bytes@1.3.0 > - cast@0.3.0 > - cc@1.0.78 > - cfg-if@1.0.0 > - clap@3.2.23 > - clap_lex@0.2.4 > - concurrent-queue@2.0.0 > - core-foundation@0.9.3 > - core-foundation-sys@0.8.3 > - cpufeatures@0.2.5 > - criterion@0.4.0 > - criterion-plot@0.5.0 > - crossbeam-channel@0.5.6 > - crossbeam-deque@0.8.2 > - crossbeam-epoch@0.9.13 > - crossbeam-utils@0.8.14 > - crypto-common@0.1.6 > - ctor@0.1.26 > - deadpool@0.9.5 > - deadpool-runtime@0.1.2 > - diff@0.1.13 > - digest@0.10.6 > - dirs@4.0.0 > - dirs-sys@0.3.7 > - dlv-list@0.3.0 > - dotenvy@0.15.6 > - either@1.8.0 > - encoding_rs@0.8.31 > - env_logger@0.10.0 > - event-listener@2.5.3 > - fastrand@1.8.0 > - fnv@1.0.7 > - form_urlencoded@1.1.0 > - futures@0.3.25 > - futures-channel@0.3.25 > - futures-core@0.3.25 > - futures-executor@0.3.25 > - futures-io@0.3.25 > - futures-lite@1.12.0 > - futures-macro@0.3.25 > - futures-sink@0.3.25 > - futures-task@0.3.25 > - futures-timer@3.0.2 > - futures-util@0.3.25 > - generic-array@0.14.6 > - getrandom@0.1.16 > - getrandom@0.2.8 > - h2@0.3.15 > - half@1.8.2 > - hashbrown@0.12.3 > - hermit-abi@0.1.19 > - hermit-abi@0.2.6 > - hex@0.4.3 > - hmac@0.12.1 > - http@0.2.8 > - http-body@0.4.5 > - http-types@2.12.0 > - httparse@1.8.0 > - httpdate@1.0.2 > - humantime@2.1.0 > - hyper@0.14.23 > - hyper-rustls@0.23.2 > - idna@0.3.0 > - indexmap@1.9.2 > - infer@0.2.3 > - integer-encoding@3.0.4 > - io-lifetimes@1.0.3 > - ipnet@2.7.0 > - is-terminal@0.4.2 > - itertools@0.10.5 > - itoa@1.0.5 > - js-sys@0.3.60 > - jsonwebtoken@8.2.0 > - lazy_static@1.4.0 > - libc@0.2.139 > - linux-raw-sys@0.1.4 > - lock_api@0.4.9 > - log@0.4.17 > - matchers@0.1.0 > - md-5@0.10.5 > - memchr@2.5.0 > - memoffset@0.7.1 > - mime@0.3.16 > - mime_guess@2.0.4 > - mio@0.8.5 > - nu-ansi-term@0.46.0 > - num-bigint@0.4.3 > - num-integer@0.1.45 > - num-traits@0.2.15 > - num_cpus@1.15.0 > - once_cell@1.17.0 > - oorandom@11.1.3 > - openssl-probe@0.1.5 > - ordered-float@1.1.1 > - ordered-multimap@0.4.3 > - os_str_bytes@6.4.1 > - output_vt100@0.1.3 > - overload@0.1.1 > - parking@2.0.0 > - parking_lot@0.12.1 > - parking_lot_core@0.9.5 > - paste@1.0.11 > - pem@1.1.0 > - percent-encoding@2.2.0 > - pin-project@1.0.12 > - pin-project-internal@1.0.12 > - pin-project-lite@0.2.9 > - pin-utils@0.1.0 > - plotters@0.3.4 > - plotters-backend@0.3.4 > - plotters-svg@0.3.3 > - ppv-lite86@0.2.17 > - pretty_assertions@1.3.0 > - proc-macro2@1.0.49 > - quick-xml@0.27.1 > - quote@1.0.23 > - rand@0.7.3 > - rand@0.8.5 > - rand_chacha@0.2.2 > - rand_chacha@0.3.1 > - rand_core@0.5.1 > - rand_core@0.6.4 > - rand_hc@0.2.0 > - rayon@1.6.1 > - rayon-core@1.10.1 > - redox_syscall@0.2.16 > - redox_users@0.4.3 > - regex@1.7.0 > - regex-automata@0.1.10 > - regex-syntax@0.6.28 > - reqwest@0.11.13 > - retain_mut@0.1.9 > - rust-ini@0.18.0 > - rustix@0.36.6 > - rustls@0.20.7 > - rustls-native-certs@0.6.2 > - rustls-pemfile@1.0.1 > - same-file@1.0.6 > - schannel@0.1.20 > - scopeguard@1.1.0 > - sct@0.7.0 > - security-framework@2.7.0 > - security-framework-sys@2.6.1 > - serde@1.0.152 > - serde_derive@1.0.152 > - serde_json@1.0.91 > - serde_qs@0.8.5 > - serde_urlencoded@0.7.1 > - sha1@0.10.5 > - sha2@0.10.6 > - sharded-slab@0.1.4 > - size@0.4.1 > - slab@0.4.7 > - smallvec@1.10.0 > - socket2@0.4.7 > - spin@0.5.2 > - syn@1.0.107 > - termcolor@1.1.3 > - textwrap@0.16.0 > - thiserror@1.0.38 > - thiserror-impl@1.0.38 > - thread_local@1.1.4 > - threadpool@1.8.1 > - time@0.3.17 > - time-core@0.1.0 > - time-macros@0.2.6 > - tinytemplate@1.2.1 > - tinyvec@1.6.0 > - tinyvec_macros@0.1.0 > - tokio@1.23.1 > - tokio-macros@1.8.2 > - tokio-rustls@0.23.4 > - tokio-util@0.7.4 > - tower-service@0.3.2 > - tracing@0.1.37 > - tracing-core@0.1.30 > - tracing-log@0.1.3 > - tracing-opentelemetry@0.17.4 > - tracing-subscriber@0.3.16 > - try-lock@0.2.3 > - typenum@1.16.0 > - unicase@2.6.0 > - unicode-bidi@0.3.8 > - unicode-ident@1.0.6 > - unicode-normalization@0.1.22 > - ureq@2.6.1 > - url@2.3.1 > - uuid@1.2.2 > - valuable@0.1.0 > - version_check@0.9.4 > - virtue@0.0.8 > - waker-fn@1.1.0 > - walkdir@2.3.2 > - want@0.3.0 > - wasi@0.11.0+wasi-snapshot-preview1 > - wasi@0.9.0+wasi-snapshot-preview1 > - wasm-bindgen@0.2.83 > - wasm-bindgen-backend@0.2.83 > - wasm-bindgen-futures@0.4.33 > - wasm-bindgen-macro@0.2.83 > - wasm-bindgen-macro-support@0.2.83 > - wasm-bindgen-shared@0.2.83 > - web-sys@0.3.60 > - winapi@0.3.9 > - winapi-i686-pc-windows-gnu@0.4.0 > - winapi-util@0.1.5 > - winapi-x86_64-pc-windows-gnu@0.4.0 > - windows-sys@0.36.1 > - windows-sys@0.42.0 > - windows_aarch64_gnullvm@0.42.0 > - windows_aarch64_msvc@0.36.1 > - windows_aarch64_msvc@0.42.0 > - windows_i686_gnu@0.36.1 > - windows_i686_gnu@0.42.0 > - windows_i686_msvc@0.36.1 > - windows_i686_msvc@0.42.0 > - windows_x86_64_gnu@0.36.1 > - windows_x86_64_gnu@0.42.0 > - windows_x86_64_gnullvm@0.42.0 > - windows_x86_64_msvc@0.36.1 > - windows_x86_64_msvc@0.42.0 > - winreg@0.10.1 > - wiremock@0.5.16 > - yansi@0.5.1 > - MPL-2.0 (1): > - webpki-roots@0.22.6 > - Unicode-DFS-2016 (1): > - unicode-ident@1.0.6 > - Unlicense (8): > - aho-corasick@0.7.20 > - byteorder@1.4.3 > - memchr@2.5.0 > - regex-automata@0.1.10 > - same-file@1.0.6 > - termcolor@1.1.3 > - walkdir@2.3.2 > - winapi-util@0.1.5 > - Zlib (2): > - tinyvec@1.6.0 > - tinyvec_macros@0.1.0 > - Unlicensed (1): > - ring@0.16.20 > > ring is licensed under a mix of [MIT, ISC, and OpenSSL’s licenses]( > https://github.com/briansmith/ring/blob/main/LICENSE). OpenSSL itself is > licensed under [Apache 2.0]( > https://www.openssl.org/source/apache-license-2.0.txt). So ring is Apache > 2.0 compatible. > > > ### Cryptography > > N/A > > ## Required Resources > > ### Mailing Lists > > - priv...@opendal.apache.org > - d...@opendal.apache.org > - comm...@opendal.apache.org > - iss...@opendal.apache.org > - bu...@opendal.apache.org > > ### Subversion Directory > > N/A > > ### Git Repositories > > From [ > > https://github.com/datafuselabs/opendal](https://github.com/datafuselabs/opendal) > > - [ > > https://gitbox.apache.org/asf/repos/incubator-opendal](https://gitbox.apache.org/asf/repos/incubator-opendal) > - [ > > https://github.com/apache/incubator-opendal](https://github.com/apache/incubator-opendal) > > ### Issue Tracking > > The community would like to continue using GitHub Issues. > > ### Other Resources > > The community has already chosen GitHub actions as continuous integration > tools. > > ## Initial Committers > > - Hao Ding [[git...@xuanwo.io](mailto:git...@xuanwo.io)] > - Lue Cai [[cai...@bupt.edu.cn](mailto:cai...@bupt.edu.cn)] > - Zhuoran Shang [[psi...@outlook.com](mailto:psi...@outlook.com)] > - Benwang Li [[libenw1...@gmail.com](mailto:libenw1...@gmail.com)] > > ## Sponsors > > ### Champion > > - tison [ti...@apache.org] > > ### Nominated Mentors > > - hexiaoqiao [hexiaoq...@apache.org] > - tedliu [ted...@apache.org] > - tison [ti...@apache.org] > - wusheng [wush...@apache.org] > > ### Sponsor Entity > > The Incubator >
