Hi Chintan,
> On 10. Jul 2019, at 18:25, Chintan Mishra <chin...@rebhu.com> wrote:
>
> On 09/07/19 9:33 PM, Joan Touzet wrote:
>
>> Hi Chintan,
>>
>> Reading through your proposal, I have one main point to make.
>>
>> At the Apache Software Foundation, the people who lead the projects are
>> the people who do the work on them. We use the wrong word "meritocracy"
>> to explain this principle; a better word would be "do-ocracy."
>>
>> http://www.apache.org/foundation/how-it-works.html#decision-making
>> https://incubator.apache.org/guides/participation.html#as_a_developer
>> https://communitywiki.org/wiki/DoOcracy
>>
>> That means that your project can completely proceed on its own if it
>> wants to; the only thing over which you're not in control is whether
>> that project gets to call itself CouchDB or not. That decision is
>> reached by the people who have built CouchDB into what it is today.
> I appreciate that you shared these links. I now understand what I have to do
> next.
>> -----
>>
>> On that last point, there's a lot that would need to be done for you to
>> convince the PMC that your vision is the one, true future of CouchDB.
>>
>> What you propose is both a significant rewrite, as well as requiring an
>> entirely new set of skills from the developer base (Rust, MQTT, Kotlin,
>> Swift).
>
> From Slack conversations, it appears the community has some inclination
> towards building a Rust based CouchDB some day. As for other technologies
> those changes are not happening today. I do not propose to start with all the
> changes at once. Storage engine is a good place to start.
Since I brought it up in Slack, let me clarify: I do not suggest that
we should move CouchDB to Rust today or any time later.
What I am suggesting is that we should look at the things required to
support your idea of an IoT-capable CouchDB-like thing. My suggestion
is to not change CouchDB, but to make a new CouchDB compatible project.
Devices are only getting smaller, so a lower level language is needed
to ensure performance and good battery use. That leaves C, C++, Go and
Rust.
When I’m looking at what likely people I could excite to contribute to
such a thing, in my filter bubble that is folks getting into Rust or
and the rest of the Rust community.
If it ends up being Go, C or C++, because someone who runs with this
prefers those, I don’t really mind.
* * *
In particular, we should look at a more detailed IoT use-case and how
CouchDB can help.
Correct me if I’m wrong, but this is mostly about devices with sensors
generating measurements over time that should be aggregated into a
cloud service for analysis.
In that world, a hypothetical API for an IoT app using our new RustyCouch
could look like this:
db = RustyCouch.open('file.foo’)
db.save(measurement)
db.push('https://cloud.measurements.com’)
repeat
This is a very small subset of the CouchDB API, but it would cover the
majority of your billion IoT use-cases.
There are a few things to be considered about data persistence and
concurrency control, but in another email, you already mentioned
SQLite, which solves most of those for you already.
db.save() would generate a JSON document with a uuid as _id and
corresponding _rev and an entry in an index that allows us to query,
at a later point: in what order were these docs written, which we are
going to need for db.push()
db.push() then opens that index, checks with the cloud which docs
it already has (as per the standard CouchDB replication protocol)
and then sends all local docs that aren’t on the cloud yet in a
couple of _bulk_docs batches.
Voilá, a low-level, embeddable library that allows you to sync
stuff to CouchDB.
This is a scope that a single developer could make a prototype
of, even in a language that you are just starting out with.
With this in hand then, the next step is to talk to the folks
who build IoT platforms and applications to see if they want to
use something like that.
And once we have this, we can talk about changes to the replication
protocol.
* * *
If you want to take this further, and make a library that also
supports interactive querying, for say native applications on
phones and watches and whatnot, you already have a decent
foundation, but you’ll have a little more work to do.
* * *
But none of this requires changing CouchDB itself, or a 10 year
effort of porting something, while solving all the needs you
have.
* * *
Finally, I’d like to caution against being flippant about the
current project direction with FoundationDB. This is something
the team that has been doing this for over 10 years looked at
“in-depth” and decided it is the right thing to do.
The alternative would be to build a FoundationDB like thing
ourselves, which is a multi-million dollar investment that
I haven’t any one seen commit to at the moment.
In particular, I’m one of the champions for smaller CouchDB
installations in this project, and moving forward is always
a give-and-take. We are not in a position yet to gauge what
the problems are with an FDB-Couch for a single-node instance
but I’m sure going to work hard on making it easy for our
downstream users.
I’m the maintainer of the Mac binaries which are extremely
popular. Any database that can’t be set up with a download,
unzip and double click to start to get a dev environment is
going to have trouble attracting new developers. So I’ll make
sure we can retain this experience as much as possible.
* * *
Let’s be pragmatic and consider incremental change or small
scope side projects to move this forward. Grand visions,
in my experience almost never work out. The only reason I have
trust in an FDB transition is because someone with authority
and budget said “the team that ostensibly built CouchDB 2.x
is going to do this”. That’s the only way it could possible work.
And don’t mistake my RustyCouch suggestion about being
dismissive or sidelining. I’ve wanted something like this since
about 2008, and many people have tried with various attempts,
so my suggestion above is very serious *and* fed by the
experience or all these failed attempts.
CouchDB’s strength is its replication protocol. We didn’t
rewrite CouchDB in JavaScript because we suddenly realised
there are a billion browsers, but PouchDB came along with
a compatible data model and replication engine so that the
two projects complement each other perfectly and anyone on
the CouchDB will tell you that PouchDB is one of the biggest
drivers of CouchDB adoption.
How about we just re-run this strategy for IoT: build a
small thing that is useful for one use-case and make it work,
then make it more complicated to be useful for more use-cases.
At each point, make sure replication with CouchDB works.
That’s a winning strategy. We already know it.
Best
Jan
—
>
>> It is in direct competition with the proposal being worked on
>> this list for the FoundationDB backend swap. With the addition of MQTT,
>> it sounds like the entire replication protocol and methodology would
>> need to be revisited, as the semantic changes you're proposing would
>> break existing client replication.
> The HTTP replication protocol more or less remains the same in the
> foreseeable future. A new MQTT replication strategy will be built upon the
> existing method. The two will not work in parallel. Either one of these will
> work per database.
>> Finally, the proposal to push into
>> the mobile space would directly compete with our sister project PouchDB,
>> who have put in tens of thousands of development hours as well.
> The community will evolve at some point. And bringing people from sister
> project onto CouchDB will allow faster development. The diagram in the
> proposal missed a part for Web Browser based CouchDB. This missed part is an
> interface for JavaScript and CouchDB-Web Browser. So, we will need some
> JavaScript developers too. And they can help improve Fauxton.
>> This all
>> adds up to a much bigger scoped project than CouchDB is today, and I
>> daresay may be bigger than I think even you realize.
>
> I do realize that I want CouchDB to be in a billion mobile and embedded
> device by 2025. I understand this is a challenging scale. I brought this here
> because I see how much we need a DB for a "Cluster Of Unreliable Commodity
> Hardware". I assume proposed path will take somewhere between 18-21 months to
> come to fruition for a team of 15 people working 40 hours/week.
>
>> With my PMC hat on, I have to ask:
>>
>> * Do you already have developers versed in these skills you can bring to
>> the project (beyond yourself)? Are they ready to commit the 40+ hours
>> a week each to making it a reality?
> No, I do not have a team in place for this.
>> * Do you have experience in building a distributed system of this scale,
>> using the specific technologies you propose?
> I have been reading about distributed systems. I want to take up an Open
> Source project which solves replication problem for devices coming up with
> emerging technologies.CouchDB is the best fit as it already solves theproblem
> of replication across remote devices.
>> * How do you plan to convince other developers of your approach
>> specifically?
>
> What got us(you) here, won't get us(you) there! -- Marshall Goldsmith
>
> CouchDB led the way by being years ahead. This is just the same thing
> happening again in a newer market. CouchDB is already great at replication.
> What I am proposing is taking this simple-but-powerful methodology a step
> further and building it for planetary scale use-cases(idea derived from
> Lasp-Lang).Here are some ways with which we drive more developers, users, and
> eyes.
>
> * Helping users realize that CouchDB lets them relax while building
> applications for devices with any form factor.
> * Reaching out to the developers who have built their own solution for
> replicating stuff from their device of any form factor to CouchDB
> * On-boarding developers who will become early adopter and test it out
> on their IoT devices. Thus, proving an unmet market need.
> * Promoting offline-first strategy among mobile and embedded
> developers will drive contributors from these communities.
> * Documenting comparisons between existing mobile and embedded
> solutions which provide replication solutions like Realm, and CouchBase.
>
>> * How do you intend to train up our existing developers on the new
>> languages and technologies involved?
> If people are excited about the future they are building then this is a
> smaller problem to tackle. People in this community when and if they come to
> a consensus about the proposal then this can be tackled by 'Each one, teach
> one' followed by Yamaha Motors. This is a buddy system where people get new
> partners to tackle a problem/PR. They share issues, their understanding of
> the codebase and language, etc. with each other. As buddies rotate everyone
> gets on the same page after a few cycles. I have found 3-pair buddy system
> works best in software.But this may differ based on culture, language,
> timezone, and availability.
>> * How do you perceive the advantages and disadvantages of your approach
>> *specifically* vs. the FDB approach already outlined?
>
> Value addition (Horizontal) > >
> ----
> Proposal (Vertical) \/ \/
>
> Pros
>
>
> Cons
>
> FoundationDB
>
>
> * Improving what works for majority of existing users
> * Iterates CouchDB to a better form
> * Prospect of immediate consistency for ACID transactions
>
>
>
> * Losing some small and mid-sized developers
> * Fragments community
>
>
> Polyglot-unification
>
>
> * Growth by tapping newer prospects
> * Reduces fragmentation of user community and codebase
> * Reimagines CouchDB as if it was built in 2019
>
>
>
> * Tons of work
> * Uses RocksDB, overlooks FoundationDB migration
>
>
> Email with subject "CouchDb Rewrite/Fork" by 'Reddy B. <redd...@live.fr>' has
> mentioned some other concerns. This proposal introduces a new story for
> CouchDB. This proposal would require using RocksDB instead of FoundationDB.
>
>> -Joan
>>
>> On 2019-07-09 10:28, Chintan Mishra wrote:
>>> Hello team!!
>>>
>>> Years of time and effort help move a product to the heights that CouchDB
>>> has reached. And as a non-contributor, rather a very new CouchDB
>>> user(1.5 years) who failed to find some relevant emails, I came up with
>>> a version of the future for CouchDB that I thought would help us grow.
>>> But Jan and Robert helped me realize that it takes a village to raise a
>>> child(CouchDB). So this is a proposal to find a middle ground from where
>>> we are headed and where the market is going next. The proposal I wrote
>>> was solely driven by what I have read over the years about the growth of
>>> the product and the community. I have attached the file or if you prefer
>>> reading in a browser, then click here<https://gitlab.com/snippets/1873543>.
>>>
>>> It will roughly take 4-5 minutes of your time. A proposed direction is
>>> to start an entirely new project. That is not what I desire. I want to
>>> join the community behind CouchDB not build a new one using it. My goal
>>> from this proposal is to generate leverage by creating early mover
>>> advantage and help grow the community.
>>>
>>> Thanking you.
>>>
>>> --
>>> Chintan Mishra
>>> Rebhu Computing
>>> Founder and CEO
--
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
