Hello team!!
Years of time and effort help move a product to the heights that CouchDB
has reached. And as a non-contributor, rather a very new CouchDB
user(1.5 years) who failed to find some relevant emails, I came up with
a version of the future for CouchDB that I thought would help us grow.
But Jan and Robert helped me realize that it takes a village to raise a
child(CouchDB). So this is a proposal to find a middle ground from where
we are headed and where the market is going next. The proposal I wrote
was solely driven by what I have read over the years about the growth of
the product and the community. I have attached the file or if you prefer
reading in a browser, then click here <https://gitlab.com/snippets/1873543>.
It will roughly take 4-5 minutes of your time. A proposed direction is
to start an entirely new project. That is not what I desire. I want to
join the community behind CouchDB not build a new one using it. My goal
from this proposal is to generate leverage by creating early mover
advantage and help grow the community.
Thanking you.
--
Chintan Mishra
Rebhu Computing
Founder and CEO
# **CouchDB: A database for your needs from web to IoT**
**C**luster **O**f **U**nreliable **C**ommodity **H**ardware **D**ata**B**ase
## ***Overview***
CouchDB is one of the leaders in database technologies. Advancements in CouchDB have been adopted in the industry by the existing market and the new products alike. Design decisions like JSON/HTTP/REST API for data handling and manipulation which were once considered too bold and far-fetched have now become industry standards. More so the unmatched ease of replication, Views, Mango, etc. further enhance developer productivity and reduce time to market for a product built using CouchDB. This proposal is written to assert the requirements of emerging technologies/industries while ensuring CouchDB's strengths in mind.
## ***Goals***
While writing this proposal the few things that were kept in the minds are:
- Build CouchDB's release procedure for server, web, mobile, and embedded systems
- Reduce fragmentation in the community
- Drive newer markets/users to CouchDB
- Scale out Replication strategy to billions of devices
## ***Rationale***
CouchDB supports a wide range of platforms. This ranges from web browsers to servers to mobile. The reason behind this wide range of support can be traced back to seamless replication and ease of data handling to name a few. In recent time this is often recognised as 'offline-first strategy'.
Whenever web developers had to sync data they had to take into account many factors like poor internet connectivity, dropped connection, etc. This created issues for web developers and they started seeking solutions for their issues. The seamless replication in CouchDB caught their eyes. The ease of syncing data between the the web browser client and server led to PouchDB and its likes. With advent of smarter mobile phones, the mobile developer community started experiencing problems similar to those already solved by the web developers. More and more system architects sought solutions that worked with the web and started trying them out in their mobile applications. This further allowed development of tools like IBM Cloudant's CDTDatastore or Sync-Android, CouchBase-Lite, and few other inactive projects.
With so many tools related to CouchDB ecosystem developers started experiencing decision overload which led them to seek a unified go-to solution. This led to community fragmentation and gave way for newer products to arise. Recent market changes allow us to invite a new set of engineers who are unprepared to handle similar problems. By providing them the tools to make their product a success, we can invite such developers to developer with CouchDB and succeed
Further, we now have tools and technologies which allow us to build for multiple platforms. They may extend from web browsers and servers to mobiles and embedded systems. This opens up a way to further enhance CouchDB's support for a wide range of platforms. All the while allowing us to reduce community fragmentation and ensuring growth of CouchDB community.
## ***Proposal***
CouchDB will be available in following flavors:
1. Server (RHEL, Debian, Windows, Mac, etc.)
2. Web Browser (Mozilla Firefox, Google Chrome, Apple Safari, Opera, etc.)
3. Mobile (iOS, Android)
4. Embedded Systems
In order to support all these we will rewrite some core functionalities in Rust and expose functions to access those functionalities natively. This will allow targeting multiple platforms. We can safely use NIFs between Elixir and Rust using project like Rustler. Thus, it can be safely assumed that same behaviour is expected for NIFs between Erlang and Rust.
### **Storage Engine**
A CouchDB data storage engine rewrite in Rust which can be used in [web](https://rustwasm.github.io/book/), [iOS](https://mozilla.github.io/firefox-browser-architecture/experiments/2017-09-06-rust-on-ios.html), [Android](https://mozilla.github.io/firefox-browser-architecture/experiments/2017-09-21-rust-on-android.html), and [embedded systems](https://www.rust-lang.org/what/embedded). This engine saves data as JSON in files. Exposed functions will allow developers to store data without HTTP API. These exposed functions will be accessible in all the flavors of CouchDB.
#### Common
This module will contain functions exposed to the user for read and write access natively. When the HTTP API will be invoked these functions will be called to store data.
#### Server
This module builds upon 'Common' and 'Mobile and Embedded (Standalone)'. It provides guarantees enjoyed by existing CouchDB users.
#### Web Browser
Stores data in IndexedDB and makes it available to 'Common'.
#### Mobile and Embedded (SQLite)
Mobile devices come with SQLite. Some mobile and embedded systems developers will feel comfortable using SQLite despite standalone solution. Uses 'Common' for exposed functions.
#### Standalone
A standalone lightweight storage solution for saving data. Uses 'Common' for exposed functions.
#### Eraser
This module will be responsible for deleting tombstones, and auxiliary data on devices with storage constraints like mobile and embedded systems.
Component required for different products while building a release:
| Sub-Module | Server | Web Browser | iOS | Android | Embedded System |
| ---------------------------- | -------- | ----------- | --------------- | --------------- | --------------- |
| Common | Required | Required | Required | Required | Required |
| Server | Required | No | No | No | No |
| Web | No | Required | No | No | No |
| Mobile and Embedded (SQLite) | No | No | SQLite only | SQLite only | SQLite only |
| Standalone | Required | No | Standalone only | Standalone only | Standalone only |
| Eraser | No | No | Required | Required | Required |
**Eraser** and **Server** are part of storage engine but they are build upon **Common** sub-module. In CouchDB-Mobile there will be a small codebase in Kotlin for Android and Swift for iOS. Its main purpose is to make functions available to mobile developers in a familiar way.
### **Index engine**
Design docs, Views, Mango, etc. will all be managed from here. This module will be responsible for handling B-trees and other tasks related to indexing. Use existing codebase for the server and rewrite Mango indexes in Rust for web, and mobile. Currently only Mango queries are implemented in Web, Mobile. In the future, mango query rewrite in Rust can be used in the server and replace Erlang implementation.
Add-on in the index engine: Filter functions emits data when a condition is met via changes feed.
Component required for different products while building a release:
| Sub-Module | Server | Web Browser | iOS | Android | Embedded System |
| ------------------------ | -------- | ----------- | -------- | -------- | --------------- |
| Existing indexing engine | Required | No | No | No | No |
| Rust based Mango | No | Optional | Optional | Optional | No |
### **Networking**
#### HTTP Server
Retain existing HTTP core infrastructure to support HTTP API for administrative functions along data manipulation and push/pull replication strategy.
#### HTTP Client (Erlang)
Retain existing codebase for establishing connection with other CouchDB servers in the cluster.
#### HTTP Client (Rust)
Fetching data from CouchDB servers or sending data to CouchDB servers from web browsers, mobile, and embedded systems.
#### MQTT Server
MQTT is a very simple to use PubSub protocol designed by IBM in 1990s and made open source in 2010s. It is used to send/receive data from embedded systems. Many IoTs rely on MQTT for lower battery consumption. HTTP establishes new TCP connection for every request-response pair. Whereas MQTT retains a persistent TCP connection. This reduces the battery consumption. Many IoT developers have already use an MQTT server like VerneMQ or Mosquitto. This may or may not be included in the CouchDB server.
#### MQTT Client (Erlang)
Required for implementing alternate replication strategy on the server
#### MQTT Client (Rust)
Required for implementing alternate replication strategy on the web, mobile, or embedded systems.
Components required while building a release:
| Sub-Module | Server | Web Browser | iOS | Android | Embedded System |
| -------------------- | -------- | ----------- | -------- | -------- | --------------- |
| HTTP Server | Required | No | No | No | No |
| HTTP Client (Erlang) | Required | No | No | No | No |
| HTTP Client (Rust) | No | Required | Required | Required | Optional |
| MQTT Server | Optional | No | No | No | No |
| MQTT Client (Erlang) | Required | No | No | No | No |
| MQTT Client (Rust) | No | Optional | Optional | Optional | Required |
### **Replication**
#### HTTP Based Push/Pull
Retain existing codebase.
#### MQTT Based Push/Pull
Build upon exiting flow. Clients can push data by publishing data and the server act based on received messages in its subscription. This will work independent of the time taken to process data on the server side. There is very small chance of timeouts, failed messages, etc.
Components required while building a release:
| Sub-Module | Server | Web Browser | iOS | Android | Embedded System |
| -------------------- | -------- | ----------- | -------- | -------- | --------------- |
| HTTP based Push/Pull | Required | Required | Required | Required | Optional |
| MQTT based Push/Pull | Required | Optional | Optional | Optional | Required |
## ***Design***
The following diagram displays how all the individual changes mentioned above fit in the exiting infrastructure.
## ***Closing Thoughts***
I may have misunderstood some ideas in the CouchDB. I am still reading the codebase. This is based on my perception of existing codebase with some inputs from what I have learned from conversations on Slack/IRC, GitHub issues, some mailing lists, FoundationDB forums, etc.
