This is an automated email from the ASF dual-hosted git repository. wusheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/skywalking.git
The following commit(s) were added to refs/heads/master by this push: new 3794b3dc91 Add a FAQ about other storage options. (#12059) 3794b3dc91 is described below commit 3794b3dc91022c8847672b88f758cf80c2782d11 Author: 吴晟 Wu Sheng <wu.sh...@foxmail.com> AuthorDate: Fri Mar 29 11:55:18 2024 +0800 Add a FAQ about other storage options. (#12059) --- docs/en/FAQ/README.md | 3 ++- docs/en/FAQ/why-clickhouse-not-supported.md | 38 +++++++++++++++++++++++++++++ docs/en/FAQ/why_mq_not_involved.md | 4 +-- docs/en/changes/changes.md | 1 + 4 files changed, 43 insertions(+), 3 deletions(-) diff --git a/docs/en/FAQ/README.md b/docs/en/FAQ/README.md index ed43bbf49a..09329589f2 100644 --- a/docs/en/FAQ/README.md +++ b/docs/en/FAQ/README.md @@ -2,7 +2,8 @@ These are known and frequently asked questions about SkyWalking. We welcome you to contribute here. ## Design -* [Why doesn't SkyWalking involve MQ in its architecture?](why_mq_not_involved.md) +* [Why does SkyWalking use RPC(gRPC and RESTful) rather than MQ as transport layer by default?](why_mq_not_involved.md) +* [Why is Clickhouse or Loki or xxx not supported as a storage option?](why-clickhouse-not-supported.md) ## Compiling * [Protoc plugin fails in maven build](Protoc-Plugin-Fails-When-Build.md) diff --git a/docs/en/FAQ/why-clickhouse-not-supported.md b/docs/en/FAQ/why-clickhouse-not-supported.md new file mode 100644 index 0000000000..dc85e296c3 --- /dev/null +++ b/docs/en/FAQ/why-clickhouse-not-supported.md @@ -0,0 +1,38 @@ +# Why is Clickhouse or Loki or xxx not supported as a storage option? + +## Background + +In the past several years, community users have asked why Clickhouse, Loki, or some other storage is not supported in the upstream. We have repeated the answer many times, but it is still happening, at here, I would like to write down the summary to help people understand more + +## Previous Discussions +All the following issues were about discussing new storage extension topics. +- Loki as storage + - https://github.com/apache/skywalking/discussions/9836 +- ClickHouse + - https://github.com/apache/skywalking/issues/11924 + - https://github.com/apache/skywalking/discussions/9011 +- Vertica + - https://github.com/apache/skywalking/discussions/8817 + +Generally, all those asking are about adding a new kind of storage. + +## Why they don't exist ? +First of all, `WHY` is not a suitable question. SkyWalking is a volunteer-driven community, the volunteers build this project including bug fixes, maintenance work, and new features from their personal and employer interests. What you saw about the current status is the combination of all those interests rather than responsibilities. +So, in SkyWalking, anything you saw existing is/was someone's interest and contributed to upstream. + +This logic is the same as this question, SkyWalking active maintainers are focusing on JDBC(MySQL and PostgreSQL ecosystem) Database and Elasticsearch for existing users, and moving forward on BanyanDB as the native one. We for now don't have people interested in ClickHouse or any other database. That is why they are not there. + +## How could add one? +To add a new feature, including a new storage plugin, you should go through [SWIP - SkyWalking Improvement Proposal](https://skywalking.apache.org/docs/main/next/en/swip/readme/) workflow, and have a full discussion with the maintenance team. +SkyWalking has a pluggable storage system, so, ideally new storage option is possible to implement a new provider for the storage module. Meanwhile, in practice, as storage implementation should be in high performance and well optimized, considering our experiences with JDBC and Elasticsearch implementations, some flags and annotations may need to be added in the kernel level and data model declarations. + +Furthermore, as current maintainers are not a fun of Clickhouse or others(otherwise, you should have seen those implementations), they are not going to be involved in the code implementations and they don't know much more from a general perspective about which kind of implementation in that specific database will have a better behavior and performance. So, if you want to propose this to upstream, you should be very experienced in that database, and have enough scale and environments to p [...] + +## What happens next if the new implementation gets accepted/merged/released? +Who proposed this new implementation(such as clickhouse storage), has to take the responsibilities of the maintenance. The maintenance means they need to +1. Join storage relative discussion to make sure SkyWalking can move forward on a kernel-level optimization without being blocked by these specific storage options. +2. Respond to this storage relative questions, bugs, CVEs, and performance issues. +3. Make the implementation performance match the expectation of the original proposal. Such as, about clickhouse, people are talking about how they are faster and have higher efficiency than Elasticsearch for large-scale deployments. Then we should always be able to see it has better benchmark and product side practice. + +Even if the storage gets accepted/merged/released, but **no one can't take the above responsibilities** or **the community doesn't receive the feedback and questions about those storages**, SkyWalking PMC(Project Management Committee) will start the process to remove the implementations. This happened before for Apache IoTDB and InfluxDB storage options. Here is the last vote about this, +- https://github.com/apache/skywalking/discussions/9059 \ No newline at end of file diff --git a/docs/en/FAQ/why_mq_not_involved.md b/docs/en/FAQ/why_mq_not_involved.md index 21b1bb39e6..3b02cda418 100644 --- a/docs/en/FAQ/why_mq_not_involved.md +++ b/docs/en/FAQ/why_mq_not_involved.md @@ -1,4 +1,4 @@ -# Why doesn't SkyWalking involve MQ in its architecture? +# Why does SkyWalking use RPC(gRPC and RESTful) rather than MQ as transport layer by default? This is often asked by those who are first introduced to SkyWalking. Many believe that MQ should have better performance and should be able to support higher throughput, like the following: <img src="MQ-involved-architecture.png"/> @@ -23,4 +23,4 @@ Even though MQ transport is not recommended from the production perspective, Sky `kafka-reporter` and `kafka-fetcher` for this feature since 8.1.0. ### How about MQ metrics data exporter? -The answer is that the MQ metrics data exporter is already readily available. The exporter module with gRPC default mechanism is there, and you can easily provide a new implementor of this module. +Log and trace exporters are using MQ as transport channel. And metrics exporter uses gRPC, as considering the scale. diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md index f45aaf4933..5b8e99c470 100644 --- a/docs/en/changes/changes.md +++ b/docs/en/changes/changes.md @@ -129,5 +129,6 @@ * Add `SWIP-5 Support ClickHouse Monitoring`. * Remove `OpenTelemetry Exporter` support from meter doc, as this has been flagged as unmaintained on OTEL upstream. * Add doc of one-line quick start script for different storage types. +* Add FAQ for `Why is Clickhouse or Loki or xxx not supported as a storage option?`. All issues and pull requests are [here](https://github.com/apache/skywalking/milestone/202?closed=1)