Re: CouchDB data processing endpoints

[Johs. E]

Hi Robert,

As long as design documents have meaningful processing endpoints, they will 
remain the extremely useful babel fish of CouchDB.

Does your "for some time to come” mean that you support a LTS for 3.0?
(ref Jan 19 Aug 2019)

Johs


> On 8 Sep 2019, at 16:24, ermouth  wrote:
> 
> Hi.
> 
>> the majority view seems to be that these can all be done better externally
> 
> As Johs mentioned, it looks like couchapp community here is a sort of
> minority. I’m sorry to say, but telling minorities what is, well, the right
> way to use endpoints, is an attitude widely considered at least outdated.
> Even if the majority think they know how to do better, even if some of them
> experienced nausea one or two times long ago.
> 
> Minorities better be embraced, not repelled or expelled.
> 
>> there doesn't even need to be a performance impact
> 
> First, this is not always true, and I even know in which particular cases –
> because I’ve tested. But did you?
> 
> Secondly, as I pointed out several times, slight performance differences
> are not always important, for large meshes deployment is much more vital
> thing. As from deployment pov there is no reasonably lightweight
> substitute.
> 
> Best regards,
> ermouth
> 
> 
> вс, 8 сент. 2019 г. в 11:33, Robert Samuel Newson :
> 
>> Hi,
>> 
>> My rule of thumb here is whether any particular 'data processing endpoint'
>> can be done better (or at all) within the database server than otherwise,
>> as opposed to the original CouchDB position of adding such things to the
>> database to enable application hosting (of a limited form, as we've all
>> noted ad nauseam). For _show, _list, _rewrite and _update, the majority
>> view seems to be that these can all be done better externally. With Joan's
>> note on co-locating couchdb, node.js and a proxy like nginx or haproxy,
>> there doesn't even need to be a performance impact.
>> 
>> I think Joan might have been referring to "validate_doc_update" not
>> "_update" with the "going nowhere" comment, as I think we all agree that
>> validate_doc_update is an important part of the core database (though it
>> might be enhanced to not require a javascript evaluation in most
>> circumstances) or at least something like it that allows users to enforce
>> arbitrary constraints on the form of any given update.
>> 
>> In summary, I still think we're deprecating several of the 'data
>> processing endpoints' in 3.0 and removing them (or not re-implementing them
>> as the case may be) in 4.0. CouchDB 3.0 will be around for some time to
>> come.
>> 
>> B.
>> 
>>> On 7 Sep 2019, at 04:13, Johs Ensby  wrote:
>>> 
>>> Hi Joan,
>>> 
 On 7 Sep 2019, at 00:59, Joan Touzet  wrote:
 More accurately, the current plan is they won't be re-implemented for
>> 4.0, since the existing implementations won't work in 4.0 against
>> FoundationDB.
>>> 
>>> About the discussions on dropping the functions that make design
>> documents so useful to many of us:
>>> Thanks again for clarifying.
>>> 
>>> This provides predictability for a group of users that might otherwise
>> feel like a week minory.
>>> Together with Jan's LTS commitment in the August report below, this
>> predictability is highly appreciated.
>>> 
 On 19 Aug 2019, at 11:51, Jan Lehnardt  wrote:
 
 3.0
 will include the best version of the current, mostly Erlang-based
>> project,
 with many new features contributed by various project partners (but
>> notably
 IBM). This will be the LTS version for people who won’t be able to
>> migrate to
 the newer technology foundation. There are a number of technical
>> limitations
 that we are happy to adopt as a project going forward, but that might
>> be deal-
 breakers for some users. As such, we’ll serve those users best with an
>> excellent
 edition of the original technology stack. LTS-timelines are TBD.
 
>>> 
>>> 
>>> Johs
>>> 
>>> PS
 On 7 Sep 2019, at 00:59, Joan Touzet  wrote:
 We've already dropped the 'couchapp' term in the documentation, over a
>> year ago. There is a single reference to them in the docs that states these
>> functions should not be used for new designs:
>>> As for the discussion about CouchDB as a catch-all platform (node.js,
>> haproxy, and nginx etc) – it is lost and buried.
>>> Thanks to @ermouth for narrowing down this to "prossesing endpoints" and
>> "data pre/post prossessing", useful terms in redirecting the discussion
>> towards the usefulness of design documents that sync can with the data.
>> 
>> 

Re: [NEWS] The CouchDB weekly news for May 19 is out!

[Johs. E]

> On 20 May 2016, at 19:33, Matthew Rijk  wrote:
> 
> How do i unsubscribe to this

Click
dev-unsubscr...@couchdb.apache.org 

Re: [PROPOSAL] CouchApp Work Group

[Johs. E]

Good luck, Alexander
I have some articles under development that I hope will help the discussion on 
how to use Couchapps to  expand the CouchDB ecosystem.
johs


> On 31 Oct 2015, at 15:35, Michelle Phung  wrote:
> 
> :) i think its a really good idea! 
> 
> Michelle
> 
>> On Oct 31, 2015, at 9:33 AM, Alexander Shorin  wrote:
>> 
>> Good news that there are no bad news (:
>> 
>> Since noone objects, will proceed with following plan.
>> --
>> ,,,^..^,,,
>> 
>> 
>>> On Mon, Oct 26, 2015 at 4:38 PM, Alexander Shorin  wrote:
>>> Hi everyone.
>>> 
>>> Recently we had quite hot discussions around CouchApps feature: what
>>> the future it has, what the problems it has, what the plans people
>>> have on it and so on, but basically we have a lot of talks, but only.
>>> However, these discussions showed that we have quite enough people in
>>> our community who loves that feature, who wishes it to become better
>>> and who wanted to make it better.
>>> 
>>> While CouchDB Team has no enough resources to completely support this
>>> feature and improve it, I propose to create CouchApp Work Group (CAWG)
>>> which will be organized by active CouchApp activists and people who
>>> would like to work on it and provide support of any kind that can.
>>> 
>>> 
>>> === What CAWG will do?
>>> 
>>> - Restore good name of CouchApp and couchapp.org website;
>>> - Search for solutions of technical and ideological issues;
>>> - Provide informational and technical support;
>>> - Involve more people into CouchDB ecosystem;
>>> - Provide feedback to CouchDB dev team for the existed features that
>>> CouchApps counts vital, proposals and, I wish, pull requests for new
>>> features and improvements;
>>> 
>>> As was previously wisely noticed, CouchApp is a metafeature that based
>>> on top of CouchDB, but it tightly coupled with CouchDB and we cannot
>>> ignore it existence nor try to bury history of our project.
>>> 
>>> 
>>> === How CAWG will interact with core devs?
>>> 
>>> There were asked some concerns that CouchDB devs cannot maintains
>>> CouchApp related requests now, work on new features and so on, and so
>>> on, so all the new feature requests discussion almost always ended
>>> with the same resolution.
>>> 
>>> Accounting past experience, I propose the following format: CAWG works
>>> on it own without disturbing core developers by ideas, requests and
>>> what else.
>>> 
>>> All feature requests are forms as good proposals on CWiki or ML with
>>> rationale part. If CWAG could generate pull requests with features
>>> implementations and bug fixes - that is the best case of our
>>> cooperation.
>>> 
>>> 
>>> === Who the tools CAWG will use?
>>> 
>>> - New ML couch...@couchdb.apache.org for discussion, support, etc.
>>> - CWiki for designing proposals, ideas, documenting workflow etc.
>>> - JIRA with CouchApp component to track issues and work on them
>>> - couchapp.org website as main informational base. I guess, we'll be
>>> need in git repository to manage it content on the first steps, unless
>>> there are other alternatives.
>>> 
>>> 
>>> === Who are all these peoples?
>>> 
>>> Everyone in CouchDB community who want to work on CouchApp feature.
>>> Keyword - work, because there are a lot of people here with a lot of
>>> ideas, but without implementing them nothing will get changed.
>>> 
>>> So far I'd propose to promote these folks:
>>> - Ermouth
>>> - Giovanni Lenzi
>>> - Harald Kisch
>>> - Josh E.
>>> - Benjamin Young
>>> 
>>> Anyone else?
>>> 
>>> (Note, I know that some devs had not the best discussions with some of
>>> these people, but I would like if we can abstract from emotions of the
>>> past here, shake our hands and try to work together in the new format)
>>> 
>>> My personal role here is to be a bridge: connect CAWG with CouchDB and
>>> provide help with Erlang bits and other project routines.
>>> 
>>> 
>>> === How this all means to CouchDB Future?
>>> 
>>> Our last mottos and slogans were focused on that we're, basically,
>>> database for web that replicates. For now CouchApp feature is not in
>>> the best state to put it on the front of CouchDB marketing.
>>> 
>>> If we can get CouchApps back to live and this will really work and
>>> involve more people into CouchDB project and ecosystem - we're all in
>>> win-win situation.
>>> 
>>> 
>>> 
>>> Thoughts?
>>> 
>>> 
>>> --
>>> ,,,^..^,,,

Re: [PROPOSAL] Allow rewrites to be JS function

[Johs. E]

Hi Dale
I dont understand the “integrate with their use case as best as we possibly 
can” when it comes to rewrite and reverse proxy.
As far as I know there is nothing to integrate here.
The joy for PouchDB to be a pure database is very understandable, since it sits 
in this rich environment that allow you to do anything.
At the server side CouchDB allows for a very limited portion of javaScript 
processing, it would be good if it was less restrictive.
I am not advocating arbitrary application platform functionality.
Just take out some limitations for the javascript crowd out there to love 
CouchDB

Thanks and best regards,
Johs


> On 20 Oct 2015, at 11:10, Dale Harvey  wrote:
> 
> This discussion has gone round and round a couple of times in different
> forms
> so I will avoid repeating my previous points but from working on PouchDB,
> the focus on having 'PouchDB' be a database only is fairly liberating, by
> not trying
> to add fairly arbitrary "application platform" features into the core
> codebase we can focus
> on integrating with what does provide those application platform features
> much better.
> 
> Users do not need to wait for reverse proxying or url rewriting to be an
> agreed, implemented
> and maintained feature, they can use the many that already exist and we
> will ensure
> that we integrate with their use case as best as we possibly can.
> 
> On 19 October 2015 at 23:57, Harald Kisch  wrote:
> 
>> Hi Garren,
>> 
>> look at MSSQL (CLR) or ORACLE (JAVA Forms) any database was trying to
>> support their users with markup languages like XML, HTML, etc. for instance
>> directly out of the database core (performance, simplicity,
>> scalability,..).
>> Lotus Notes did also integrate JavaScript inside of their core (Do you know
>> which guy did take part of it?). This have different reasons, but one of
>> this reasons is to support users with dynamic mutable data directly into
>> their GUI in JSON format which in my opinion is the fundamental part of
>> CouchDB to be a database for the web.
>> Improvements get lost if we look at others and try not to be different. In
>> my opinion we should more think about replacing spidermonkey with the
>> google V8 engine and itegrate node completely into the CouchDB core to
>> consume npm-packages directly instead of using them in the local filesystem
>> outside of CouchDB, where unfortunatelly complexity rise up at scaling.
>> 
>> --Harald
>> 
>> On Mon, Oct 19, 2015 at 9:24 PM, Johs Ensby  wrote:
>> 
>>> Hi Garren,
>>> thanks for the "not standing in the way", I hope for more volunteers to
>>> iron out some of CouchDB's old akward wrinkles.
>>> I am all with you for simplification:) ermouth's rewrite function is a
>>> huge simplifier.
>>> 
>>> Where I disagree with you is where you say "probably a sign that this
>> idea
>>> is not something worth pursuing".
>>> Whenever you discover that you have a differentiator, it's always a good
>>> idea to look closely before discaring it and blend in with the rest.
>>> It's all about attracting the next million web developers.
>>> 
>>> johs
>>> 
>>> 
>>> 
>>> 
>>>> On 19. okt. 2015, at 20.08, Garren Smith  wrote:
>>>> 
>>>> I’m really struggling with these proposals. I love the enthusiasm of
>>> everyone but I keep thinking we should rather simplify CouchDB.
>>>> CouchDB is ultimately a database. One with excellent sync capabilities.
>>> And combining that with libraries like PouchDB and Hoodie make it an
>>> amazing database to build applications with.
>>>> Adding routers and reverse proxies just makes it feel like we trying to
>>> push CouchDB into being more than it needs to be.
>>>> 
>>>> For example building Couchapp like functionality in Node.js is so
>> simple
>>> and way better. Languages like Go also do that really well. Far superior
>>> than what we can do with a database.
>>>> I would rather let the Node.js and Go web libraries serve content and
>>> let us focus on building a clustered replicating database. We will draw
>>> more people to this community if we can do that properly over creaky,
>> slow
>>> and limited web serving mashed on top of a database.
>>>> If I look at other popular databases, I don’t see any of them serving
>>> web content which is probably a sign that this idea is not something
>> worth
>>> pursuing.
>>>> 
>>>> However if there is a burn

Re: [PROPOSAL] Allow rewrites to be JS function

[Johs. E]

Thanks Andy,
I will try and  get some use cases up on confluence.
As for whoever would pick up the work after ermouth,  I have of course one big 
thing on the wish list that goes well with a new router solution..
reverse proxy
I remember asking about it when I first started to work w CouchDB and there 
were some concerns regarding security.
Since then I think node.js has paved the way with content scraping and all 
sorts of outgoing traffic.

Has anyone work on a reverse proxy solution for Couch?

johs


> On 18 Oct 2015, at 21:36, Andy Wenk  wrote:
> 
> Hey Johs,
> 
> thanks a lot for this. I need some time to dig into it. We need a place to
> write the user stories / use case down. So I suggest we find good place at
> the cwiki. So I suggest to use
> https://cwiki.apache.org/confluence/display/COUCHDB/Enhancement+Proposals .
> Do you have write access there? If not, please ping me.
> 
> Great work!
> 
> All the best
> 
> Andy
> 
> P.S.: Jan already mentioned the feature freeze. Please take it not as a
> demotivation but as the possibility to have a bit more time to work on it.
> 
> On 17 October 2015 at 17:32, Johs Ensby  wrote:
> 
>> Andy,
>> I will make my first use case for function in _rewrite a high level one:
>> to create a standalone server that is an all-in-one DB server, application
>> server, api server and web server.
>> 
>> I have played with the build of CouchDB 2 with rewrite function
>> implemented that  ermouth put up on the irish AWS community AMI list and
>> the new use cases are endless.
>> First, I find that there are a few things that people fail to notice about
>> ddocs.
>> you need a tool to build a ddoc, editing JSON is not a viable option. The
>> Ddoc Lab of ermouth is in a class of its own. If you havent tried it yet,
>> do so from http://ddoc.me/ . Installing on your own
>> couch it is as easy as storing the application, all included as one
>> document in any database. Ddoc Lab is a component oriented IDE with syntax
>> checking, less preprosessor and other build tools that let you keep a well
>> organized ddoc as a source project (in one couchdb document) and you
>> publich a ddoc to any target db.
>> with this tool you can organize your js modules and templates etc and
>> basically...
>> set up the API of your application in a ddoc. You can switch between
>> databases and their ddoc functionality based on username, role or
>> geolocation and limit access to parts of the Couch API as needed
>> 
>> This is the method I would recommend to explore powerful simplicity with
>> function in rewrites
>> redirect port 80 directly to couch
>> set up 2 vhosts, one for public access pointing to youdb/_design/api and
>> one for sysadm pointing to /
>> for admin use Fauxton and Ddoc Lab on the sysadm vhost
>> you are set to develop great systems, no big tool stack to learn, just
>> bring in whatever js modules you like, the template engine you like, the
>> router you like, the HTML5 stuff you like..
>> .. or just write some very compact js code in one place where you ealier
>> had to mess around with a whole stack of tools and systems
>> 
>> below is the req object that the function takes
>> 
>> Johs
>> 
>> 
>> 
>> The rewrite function has this syntax
>> function(req) {
>>.. your code that will
>>return
>>path
>>// optional
>>headers
>>method // you can change this on the fly
>>code
>>body
>> }
>> 
>> the function receives this req object
>> method
>> path
>> raw_path
>> query
>> headers
>>Accept
>>Accept-Encoding
>>Connection
>>Host
>>Upgrade-Insecure-Requests
>>User-Agent
>>x-couchdb-vhost-path
>> body
>> peer
>> cookie
>> userCtx
>> db
>> name
>> roles
>> secObj
>> 
>>> On 1. okt. 2015, at 13.40, Andy Wenk  wrote:
>>> 
>>> Johs,
>>> 
>>> Yes for sure! That's always great. Maybe you can also write some user
>> stories (given when then) or scribble some graphics. Everything is useful
>> and will fasten the process ;-)
>>> 
>>> Cheers
>>> 
>>> Andy
>>> 
>>> On 1 Oct 2015 12:38, "Johs Ensby" mailto:j...@b2w.com>>
>> wrote:
>>> Thanks for this Andy,
>>> 
>>> I can contribute to the discussion of the feature seen from a user
>> perspective.
>>> Would it be appropriate to present some use cases?
>>> 
>>> best
>>> Johs
>>> 
 On 1. okt. 2015, at 12.33, Andy Wenk > andyw...@apache.org>> wrote:
 
 Johs,
 
 Let me please show the steps needed.
 
 * discuss the feature very clearly on the dev@. Please make sure that
>> core
 developers as committers with commit bits are involved
 
 * code the feature. Make sure to implement tests
 
 * send a pull request and show it to dev@
 
 * finally the community will accept or decline the feature (this will
 involve refactoring and changes)
 
 As Alex said. The PMC or Jan do not decide about the feature.
 
>>>

Re: CouchDB _rewrite

[Johs. E]

Thanks Joan,
we will prepare a strong case for it — probably needs to be part of a the 
“couchapp story” discussion that we had a while ago
johs

> On 04 Sep 2015, at 19:01, Joan Touzet  wrote:
> 
> Proedurally, CouchDB 2.0 is effectively in feature freeze - we need to
> get it out the door, it is far overdue.
> 
> Looking at changing basic functionality like _rewrite can come after
> 2.0 is done, in the 3.0 timeframe.
> 
> -Joan
> 
> - Original Message -
>> From: "Johs. E" 
>> To: "dev@couchdb.apache.org Developers" 
>> Cc: market...@couchdb.apache.org
>> Sent: Friday, September 4, 2015 8:21:12 AM
>> Subject: CouchDB _rewrite
>> 
>> Fellow CouchDB enthusiasts,
>> 
>> Let me quote a dialogue I had the other day with a colleague on
>> Couchapps and _rewrite:
>> 
>>>> I would like to know what is so horrible with the vhost/rewrite
>>>> of CouchDB
>>> You must concentrate all rules in one place, that is totally out of
>>> idea ‘one app – one ddoc’
>>> Capturing mechanics is outrageously ugly and limiting. You can‘t
>>> capture on query, only on path, and in very limiting manner.
>>> Obsolete for at least 15 years.
>>> Rule lists are flat – they must be trees, since it‘s json, not SQL
>>> table of directory with files.
>>> It‘s all very brittle, error prone and imposes all possible hurdles
>>> during debug – no err messages, no log, no validator.
>>> And most important: it creates illusion, that it can fit everything
>>> – but it only fits small static-like sites.
>>>> Is it something that could be fed to the developers?
>>> 
>>> Don‘t think anybody of them is interested. This functions assumed
>>> obsolete or impractical by the vast majority of community, as I
>>> see. And I agree with them.
>> 
>> Still with its limitations, I love _rewrite
>> You direct the vhost to db/_design/api/_rewrite
>> using so-called “unsafe” rewrites, you create an API for your many
>> databases and their couchapps there.
>> It works beautifully.
>> That is at Cloudant. I think I learned from an earlier discussion
>> that the lack of a “default vhost” is a problem outside Cloudant.
>> Now Cloudant does not offer SSL unless you enter into a relationship
>> with your local IBM organization and buy a dedicated cluster under a
>> std IBM contract, so
>> 
>> Of course I would like to see a better rewrite function, my priority
>> would be
>> A tree structure of rules
>> Capture query in the “to”
>> That would be a great enhancement to go with version 2.0
>> 
>> br
>> Johs
>> 
>> 

CouchDB _rewrite

[Johs. E]

Fellow CouchDB enthusiasts,

Let me quote a dialogue I had the other day with a colleague on Couchapps and 
_rewrite:

> > I would like to know what is so horrible with the vhost/rewrite of CouchDB
> You must concentrate all rules in one place, that is totally out of idea ‘one 
> app – one ddoc’
> Capturing mechanics is outrageously ugly and limiting. You can‘t capture on 
> query, only on path, and in very limiting manner. Obsolete for at least 15 
> years.
> Rule lists are flat – they must be trees, since it‘s json, not SQL table of 
> directory with files.
> It‘s all very brittle, error prone and imposes all possible hurdles during 
> debug – no err messages, no log, no validator.
> And most important: it creates illusion, that it can fit everything – but it 
> only fits small static-like sites.
> > Is it something that could be fed to the developers?
> 
> Don‘t think anybody of them is interested. This functions assumed obsolete or 
> impractical by the vast majority of community, as I see. And I agree with 
> them.

Still with its limitations, I love _rewrite
You direct the vhost to db/_design/api/_rewrite
using so-called “unsafe” rewrites, you create an API for your many databases 
and their couchapps there.
It works beautifully.
That is at Cloudant. I think I learned from an earlier discussion that the lack 
of a “default vhost” is a problem outside Cloudant.
Now Cloudant does not offer SSL unless you enter into a relationship with your 
local IBM organization and buy a dedicated cluster under a std IBM contract, so 

Of course I would like to see a better rewrite function, my priority would be
A tree structure of rules
Capture query in the “to”
That would be a great enhancement to go with version 2.0

br
Johs