Re: Call for "Must-fix" issues
I’m sorry to say, but: — Fauxton is unable to show doc attachments list in any modern browser on http connection — Fauxton is still unable to make “Back” button working in _config section — Fauxton doesn’t work on ~20% of browsers, polyfills for missing browser features is still an unknown term for Fauxton — Fauxton is still unable to validate at least JSON syntax on typing, I’m not saying about Mango syntax — Fauxton is still all about making as much UI noise as possible, sacrificing data density I discovered all this in first 2 minutes after install – just to check have anything changed. You said tables? I’m sorry to say, but tables is the least problem. Anyway, Fauxton is absolutely on par with CouchDB 2 in terms of product quality. An you enforce people to migrate on _this_? Are you kidding? And, to prevent known spells: sorry, no bug reports and no pull requests, all above problems were already reported. ermouth
Re: On 1.x deprecation
> Approximately how many 1.x nodes to you have in production now? > What is your estimated total no of users on these? Sorry, can’t disclose this kind of info publicly. All I can say I have zero nodes with 2.x in prod, despite we analyzed can we migrate several times. We can’t, 2.x just does not fit for production, which is well proved by Joan words about paid support requests, github issues and my own tests. ermouth
On 1.x deprecation
TLDR; Please vote -1 at @dev thread ‘[PROPOSAL] Officially deprecate CouchDB 1.x.’ All arguments of the proposal are of no basis. --- Hope, we all read deprecation statement from Joan. She provided one reason directly, also several were coined in indirect way. 1. “No one is maintaining the 1.x.x branches of CouchDB anymore” Probably that’s because they just do not need daily maintenance. 2. “Issues stack up on the tracker with no response.” Ok, lets count. Right now couchdb github repo has 133 issues open, 11 of them have 1.x tag. 21 of those 133 are marked with red bug tag, but none of them with 1.x. So seems 1.x have no serious bugs, but 2.x have a pile. How about “no response”? All 1.x issues have at least one. 33% of non-1.x issues have zero answers. So I’d say if there exist a stack up, it’s not related to 1.x, it’s more about 2.x. 3. “Having to patch 1.x as well as 2.x slows down the security team's ability to push releases quickly out the door” First, this game is two-sided. Probably, necessaty to fix not so adopted version slowed down the team from releasing very minor upgrade for very stable version widely used in production. Probably, back-porting of not widely adopted 2.x features into 1.7 also slowed process down. Or, probably, there were no slowdown at all? As I know, that ‘slow down’ was never complained. 4. “By focusing on what we do best - supporting 2.x” As I showed earlier it’s looks incomparably easier to support 1.x than 2.x. As I can see from numbers, supporting 2.x issues have already taken all focus. It happened about 1.5 years ago I’d say. Ok, have anyone ever complained? Today I consulted a guy from BY by phone, yesterday from IN – also privately. Classic reasonable CouchDB guys seem to be pretty well aware they won’t have a lot of help from official support anyway. They do not need bug fixes, they want to discuss use strategies, solution details, they want hints and opinions how to improve things that already works fine. So 1.x doesn’t blur developers’ focus, unless developers start demnding granny’s blood. Granny is still pretty healthy. 5. “I’m definitely seeing more people on 2.x these days than 1.x, *significantly* more - both from our informal support channels as well as through GitHub Issues and requests for paid support” This argument is very strange. It looks like someone already have a lot of 2.x tickets and want even more. Sorry, but I don’t see how it reflects real users distribution, I only see that 2.x is more buggy and though sells paid support better. --- So, as I see all above arguments for deprecation have no relation to real accountable state of things, unless I suppose the goal is paid support. Honestly, I can’t believe in it. Then why deprecate? The deprecation notice by itself breaks reputation of 1.6/1.7, which is for now waaay more reliable than 2.x. At least 1.6 doesn’t bite unexpectedly. I hope keeping things as is for about a year is well enough to fix most visible 2.x issues, and then – long live 1.x. Unfortunately I’m not subscribed to @dev and can’t reply into past proposal thread even if get subscribed now. So if you feel my arguments are reasonable, I ask you to cast ‘-1’ under that proposal. Thank you. ermouth
Re: Call for "Must-fix" issues
> This is rather inaccurate > A regrettable regression, > but only a config issue Well enough to block upgrade. Anyway, as we see now those non-technical issues of different nature improve the number of requests for paid support. I wouldn’t account this fact as popularity, as Joan did, however hope it at least sweetens regrets. ermouth
Re: Call for "Must-fix" issues
> convenience of one file on disk per database is gone, but is there anything else There were serious issues with single node shards (or whatever those pieces were) during upgrade from 2.1.0 to 2.1.1. I read only brief report, but remember it completely broken tests in some bizzare way, like inability to save docs with particular _id ranges. > could you unveil this secret, please? There’s nothing secret here, theoretically you have to insert two oneliners into chttpd_auth and then rebuild. Obvious task for end user ) -export([proxy_authentication_handler/1]). and proxy_authentication_handler(Req) -> couch_httpd_auth:proxy_authentication_handler(Req, chttpd_auth_cache). But as I can recollect it also finally broke things in some bizzare way. ermouth 2018-07-05 11:59 GMT+03:00 Johs Ensby : > >> > >> Proxy authorization which is also in docs, but absent in 2.x. > > > > it is not absent, just not exposed by default. You can configure to > enable it (it’s just not documented ;) > > Hi Jan, > could you unveil this secret, please? > johs > > > > > On 5 Jul 2018, at 10:53, Jan Lehnardt wrote: > > > > > > > >> On 5. Jul 2018, at 10:20, ermouth wrote: > >> > >> Proxy authorization which is also in docs, but absent in 2.x. > > > > it is not absent, just not exposed by default. You can configure to > enable it (it’s just not documented ;) > > > >> Single-node install without shards. > > > > Both the /_setup_cluster endpoint and the Mac binaries already set q=1 & > n = 1 on setup. Everyone else can configure it. > > > > Best > > Jan > > — > > > >> > >> ermouth > >> > >> 2018-07-05 10:10 GMT+03:00 Johs Ensby : > >> > >>> This thread reach out to CouchDB 1.x users to generate a list of > >>> "must-fix" issues that is preventing users to upgrade to the latest > >>> version of CouchDB. > >>> > >>> It is in response to Joan's comment below regarding the > >>> non-technical proposal to make a project decision to terminate > >>> official Apache support for CouchDB 1.x. > >>> > >>>> On 5 Jul 2018, at 06:31, Joan Touzet wrote: > >>>> > >>>> As for things in 2.x that are "must fix" before people can upgrade, I > >>>> too would like to see pull requests for those. Once again, your help > is > >>>> most welcome in this - both in identifying a definitive list of those > >>>> must-fix issues, as well as code towards fixing them. If you'd like > >>>> to help with this important work, please start a new thread. > >>> > >>> > >>> > >>> Mine is CouchDB as a proxy. > >>> The feature described here is not working in 2.1.1 > >>> http://docs.couchdb.org/en/2.1.1/config/proxying.html?highlight=_proxy > >>> > >>> Johs > >>> > > > > -- > > Professional Support for Apache CouchDB: > > https://neighbourhood.ie/couchdb-support/ > > > > ……… > Johannes Ensby > > > Business to Web AS > Tollbugata 8, N- 0152 Oslo, Norway > +47 611 00 006 (mobile) > +47 611 00 700 (switchboard) > j...@b2w.com > www.linkedin.com/in/ensby > www.b2w.com > >
Re: Call for "Must-fix" issues
Single-node install without shards. ermouth 2018-07-05 11:20 GMT+03:00 ermouth : > Proxy authorization which is also in docs, but absent in 2.x. > > ermouth > > 2018-07-05 10:10 GMT+03:00 Johs Ensby : > >> This thread reach out to CouchDB 1.x users to generate a list of >> "must-fix" issues that is preventing users to upgrade to the latest >> version of CouchDB. >> >> It is in response to Joan's comment below regarding the >> non-technical proposal to make a project decision to terminate >> official Apache support for CouchDB 1.x. >> >> > On 5 Jul 2018, at 06:31, Joan Touzet wrote: >> > >> > As for things in 2.x that are "must fix" before people can upgrade, I >> > too would like to see pull requests for those. Once again, your help is >> > most welcome in this - both in identifying a definitive list of those >> > must-fix issues, as well as code towards fixing them. If you'd like >> > to help with this important work, please start a new thread. >> >> >> >> Mine is CouchDB as a proxy. >> The feature described here is not working in 2.1.1 >> http://docs.couchdb.org/en/2.1.1/config/proxying.html?highlight=_proxy >> >> Johs >> > >
Re: Call for "Must-fix" issues
Proxy authorization which is also in docs, but absent in 2.x. ermouth 2018-07-05 10:10 GMT+03:00 Johs Ensby : > This thread reach out to CouchDB 1.x users to generate a list of > "must-fix" issues that is preventing users to upgrade to the latest > version of CouchDB. > > It is in response to Joan's comment below regarding the > non-technical proposal to make a project decision to terminate > official Apache support for CouchDB 1.x. > > > On 5 Jul 2018, at 06:31, Joan Touzet wrote: > > > > As for things in 2.x that are "must fix" before people can upgrade, I > > too would like to see pull requests for those. Once again, your help is > > most welcome in this - both in identifying a definitive list of those > > must-fix issues, as well as code towards fixing them. If you'd like > > to help with this important work, please start a new thread. > > > > Mine is CouchDB as a proxy. > The feature described here is not working in 2.1.1 > http://docs.couchdb.org/en/2.1.1/config/proxying.html?highlight=_proxy > > Johs >
CloudWall 2.2
Hello all, happy to announce CloudWall 2.2, http://cloudwall.me. It‘s a toy in-browser noBackend OS, which uses PouchDB/CouchDB instead of file system. New features: * CW, both as a compiled design doc and as sources, now installs using replication. * All apps and documentation were upgraded. * Added latest Photon, Ddoc Lab and Inliner apps, with sources. CW 2.2 can run a) from static hosting, b) as a couchapp bound to a dedicated domain, c) as a couch-hosted app, directly from index.html attach. CloudWall is built by itself, so sources are good example how to build couch-hosted app of such complexity using nothing but browser. ermouth
1.6.1: interim patch
Upgrading to 1.7.1 is important since CVE-2017-12635 is a serious hole.
However, there exist an interim way. Below design doc just rejects .roles
dupes. You can check it putting into any other db first and trying to put
new doc like {"roles":[],"roles":[1]}. If validator is ok, rejection reason
is ‘You can't hack roles’.
{
"_id": "_design/X12635",
"language": "erlang",
"validate_doc_update": "fun ({NewDoc}, OldDoc, UserCtx, SecObj)->\n\t%%
Covers CVE-2017-12635\n\tRoles = proplists:lookup_all(<<\"roles\">>,
NewDoc),\n\tcase length(Roles) < 2 of\n\t\ttrue -> ok;\n\t\tfalse ->
throw({[{<<\"forbidden\">>, <<\"You can’t hack roles,
sorry\">>}]})\n\tend,\n\t1\nend."
}
Since ability to save json with double entry of .roles array is a key of
the 12635 vulnerability, the ddoc seems to fix it, if put into /_users
bucket.
Nothing comes without price: you need to set native_query_servers / erlang
= {couch_native_process, start_link, []} in DB config. Since enabling
erlang might affect security, each case should be carefully assessed.
Although this trick is acceptable if you postponing upgrade to 1.7.1 for
reasons not under your control, I highly recommend upgrade as soon as
possible.
ermouth
Re: IBM Cloudant - Support for Virtual Hosts Ending December 4, 2017
Martin! You may use smileupps.com, max version is 1.6.0. Also you may use ami-29ecda5e AWS EC2 image (Ireland), which is 1.6.1 with 2.x-style JS rewrites. However, taking in account last CouchDB critical vulnerability, you should be very careful with your _users DB and protect it from changing by any user except admins. ermouth
Re: Photon, 90% progress
Thank you, Giovanni! BTW, tested Photon at Smileupps, seems to work. However, I missed one important detail: if the DB list tab is constantly open and visible, Photon makes a request every several seconds, polling tasks. Not so good for pay-for-request services. Gonna fix it.
Re: Photon, 90% progress
Thank you, Aurélien! > and a convincing demo I appreciate your sense of humor, but I afraid my awful English is not too convincing ) ermouth
Re: Photon, 90% progress
Thank you, Harald! Actually, it‘s not a couchapp from thin air, Photon was created inside browser using couchapps. Photon never existed as source files, only source docs. So entire couchapp (db hosted app) technology is powerful, its hidden potential is still nearly untouched, as I see it. ermouth 2017-09-15 11:41 GMT+03:00 Harald Kisch : > Thank You, ermouth! > For me this is the best editor for CouchDB and the best way to edit large > JSON structures. > It is also the simplest way to install a CouchApp, simply by pasting your > JSON code into a design document and click save. > Photon also shows how powerful this kind of application is. > > Great work! > > harald > > On Thu, Sep 14, 2017 at 10:14 PM, ermouth wrote: > > > Now 100% Futon features coverage. Recorded a screencast about Photon > > features https://www.youtube.com/watch?v=MHc6tozNhWU > > > > ermouth > > > > 2017-09-08 7:02 GMT+03:00 ermouth : > > > > > Think Photon is ready for more wide testing, https://github.com/ > > > ermouth/couch-photon > > > > > > As for now, 0.2.830, all original Futon features are covered, except > two: > > > creating sync tasks and view compaction on per ddoc basis. > > > > > > Most original Futon features were bit improved in Photon. Also Photon > > > includes several features not present in Futon, see boasting in PS. > > > > > > The sweetest new feature is instant update🌟. Right navbar button, > which > > > opens About tab, now allows to check for updates from AWS S3 CDN (no > > > logging there). It means Photon can be kept fresh in literally two > > clicks. > > > > > > Bug reports and suggestions are highly welcomed. > > > > > > ermouth > > > > > > PS. Photon boasting: > > > > > > * In-app tabs, restored on reload; > > > * Valid state-to-URL# reflection, so most states have unique URLs; > > > * Group actions for DBs and docs; > > > * JSON for config, so group updates for config keys or even sections; > > > * View navigation starting from a specific key; > > > * You can choose to purge or retain doc content on deletion; > > > * Doc revision navigation includes conflicts if any. > > > > > > > > > -- > -- > > Dipl.-Inf. Harald R. Kisch > > Falkenstraße 19C > 81541 München > Germany > > Mobil DE: +49 (0) 176 56 58 58 38 > > Skype: harald.kisch > Mail: haraldki...@gmail.com >
Re: Photon, 90% progress
Now 100% Futon features coverage. Recorded a screencast about Photon features https://www.youtube.com/watch?v=MHc6tozNhWU ermouth 2017-09-08 7:02 GMT+03:00 ermouth : > Think Photon is ready for more wide testing, https://github.com/ > ermouth/couch-photon > > As for now, 0.2.830, all original Futon features are covered, except two: > creating sync tasks and view compaction on per ddoc basis. > > Most original Futon features were bit improved in Photon. Also Photon > includes several features not present in Futon, see boasting in PS. > > The sweetest new feature is instant update🌟. Right navbar button, which > opens About tab, now allows to check for updates from AWS S3 CDN (no > logging there). It means Photon can be kept fresh in literally two clicks. > > Bug reports and suggestions are highly welcomed. > > ermouth > > PS. Photon boasting: > > * In-app tabs, restored on reload; > * Valid state-to-URL# reflection, so most states have unique URLs; > * Group actions for DBs and docs; > * JSON for config, so group updates for config keys or even sections; > * View navigation starting from a specific key; > * You can choose to purge or retain doc content on deletion; > * Doc revision navigation includes conflicts if any. >
Photon, 90% progress
Think Photon is ready for more wide testing, https://github.com/ermouth/couch-photon As for now, 0.2.830, all original Futon features are covered, except two: creating sync tasks and view compaction on per ddoc basis. Most original Futon features were bit improved in Photon. Also Photon includes several features not present in Futon, see boasting in PS. The sweetest new feature is instant update🌟. Right navbar button, which opens About tab, now allows to check for updates from AWS S3 CDN (no logging there). It means Photon can be kept fresh in literally two clicks. Bug reports and suggestions are highly welcomed. ermouth PS. Photon boasting: * In-app tabs, restored on reload; * Valid state-to-URL# reflection, so most states have unique URLs; * Group actions for DBs and docs; * JSON for config, so group updates for config keys or even sections; * View navigation starting from a specific key; * You can choose to purge or retain doc content on deletion; * Doc revision navigation includes conflicts if any.
Re: Fauxton, Futon and licensing obstacles
> The ones I miss now are: mango queries and attachments access. Attachments management is underway. As for mango. I will be happy if you could record a short screencast with several examples how you use mango in Fauxton. Query examples, expected output etc. We do not use mango at all, so I just don‘t have enough details on mango-related UX. So I‘m asking for hints ) ermouth
Re: Fauxton, Futon and licensing obstacles
> tabs sometimes dont respond to click Yet unknown issue. Any additional details? > numeric startkey not allowed They are allowed actually: if you input a number, it‘s submitted as a number unless you add quot marks around it. What was the experience that made you think they are not allowed? > codemirror box crops content How this ‘crop’ looks like? Long lines are not wrapped and you can not scroll? Can you please provide more details. > Missed direct access to attachments It‘s just not yet coded. For now I think there will be narrow (~200px) right aside in the doc view. The aside will contain attachments navigation. > I have opinions on UX, of course Feel free to post suggestions to github ) ermouth
Re: Fauxton, Futon and licensing obstacles
> Opening design documents seems not working Fixed. ermouth
Re: Fauxton, Futon and licensing obstacles
Opening ddocs indeed does not work for some configs (smileupps ie), this issue will be fixed soon. Encoding _design%2F <=> _design/, you know. ermouth 2017-09-06 13:24 GMT+03:00 Giovanni Lenzi : > Great tool ermouth, > > seems very fast and tabs greatly simplify administration tasks. > > Opening design documents seems not working... Are you planning to include > different editor for them? > > Thanks for work and great idea! > --Giovanni > > 2017-09-06 8:40 GMT+02:00 Johs. E : > > > Congratulations with fantastic progress, ermouth! > > Will test tonight. > > johs > > > > > On 6 Sep 2017, at 08:37, ermouth wrote: > > > > > > Progress so far: very early public beta > > > https://github.com/ermouth/couch-photon > > > > > > About 80% ready I hope, not tested in 2.x at all. > > > > > > ermouth > > > > > > 2017-08-30 21:45 GMT+03:00 ermouth : > > > > > >> Work in progress, watch video https://youtu.be/lBVdIPZhDc4 to see > > current > > >> state. > > >> > > >> Right now codename Photon is a json with footprint well below 1.5Mb. > > >> > > >> ermouth > > >> > > >> > > > > >
Re: Fauxton, Futon and licensing obstacles
Progress so far: very early public beta https://github.com/ermouth/couch-photon About 80% ready I hope, not tested in 2.x at all. ermouth 2017-08-30 21:45 GMT+03:00 ermouth : > Work in progress, watch video https://youtu.be/lBVdIPZhDc4 to see current > state. > > Right now codename Photon is a json with footprint well below 1.5Mb. > > ermouth > >
Re: Fauxton, Futon and licensing obstacles
Work in progress, watch video https://youtu.be/lBVdIPZhDc4 to see current state. Right now codename Photon is a json with footprint well below 1.5Mb. ermouth
Re: Fauxton, Futon and licensing obstacles
No CloudWall runtime, it would be too limiting in this particular case. ermouth 2017-08-24 8:08 GMT+03:00 Johs Ensby : > ermouth, > you are doing this as a cloudwell app with the cloudwall runtime > supporting the single ddoc app, right? > Johs > > On 23 Aug 2017, at 21:56, ermouth wrote: > > > > Good idea, however I gonna start with CouchDB ) > > > > I‘ve recordered 5min screencast about JSON editor for CN Photon > > https://youtu.be/OQ_YJ0EZQjE > > > > ermouth > > > > 2017-08-23 9:10 GMT+03:00 Martin Broerse : > > > >> Perhaps also include pouchdb databases in Photon. You would need to type > >> the database name you like to connect to because there is no index > >> implemented. See https://github.com/w3c/IndexedDB/issues/31 . Caching > the > >> pouchdb database names would be be a nice feature for Photon. If Photo > is > >> build in Ember we can use https://github.com/pouchdb- > community/ember-pouch > >> to connect to the pouchdb databases. > >> > >> On Tue, Aug 22, 2017 at 5:09 PM, ermouth wrote: > >> > >>> Several sketches http://jquerymy.com/kod/photon1.pdf > >>> > >>> ermouth > >>> > >>> 2017-08-20 15:07 GMT+03:00 ermouth : > >>> > >>>> Seems that Fauxton experiencing some troubles with React unfriendly > >>>> licensing. It‘s a pity taking in account CouchDB 2 moved to fairly > >> stable > >>>> state, and Fauxton itself received a lot of great improvements since > >> last > >>>> year. > >>>> > >>>> However, in new circumstances, what do you think about re-creating > >>>> something like Futon+ (for Couch 2) as a single ddoc? So you copy > >>>> open-sourced ddoc json to any DB, run index.html attachment from the > >>> ddoc – > >>>> and you‘re in. > >>>> > >>>> Basically, the idea is to re-create Futon-like app as a couchapp. > >>>> > >>>> If an idea is reasonable, is it reasonable to make the process openly > >>>> funded in some way, to make it more rapid and controllable? > >>>> > >>>> ermouth > >>>> > >>> > >> > >
Re: Fauxton, Futon and licensing obstacles
Good idea, however I gonna start with CouchDB ) I‘ve recordered 5min screencast about JSON editor for CN Photon https://youtu.be/OQ_YJ0EZQjE ermouth 2017-08-23 9:10 GMT+03:00 Martin Broerse : > Perhaps also include pouchdb databases in Photon. You would need to type > the database name you like to connect to because there is no index > implemented. See https://github.com/w3c/IndexedDB/issues/31 . Caching the > pouchdb database names would be be a nice feature for Photon. If Photo is > build in Ember we can use https://github.com/pouchdb-community/ember-pouch > to connect to the pouchdb databases. > > On Tue, Aug 22, 2017 at 5:09 PM, ermouth wrote: > > > Several sketches http://jquerymy.com/kod/photon1.pdf > > > > ermouth > > > > 2017-08-20 15:07 GMT+03:00 ermouth : > > > > > Seems that Fauxton experiencing some troubles with React unfriendly > > > licensing. It‘s a pity taking in account CouchDB 2 moved to fairly > stable > > > state, and Fauxton itself received a lot of great improvements since > last > > > year. > > > > > > However, in new circumstances, what do you think about re-creating > > > something like Futon+ (for Couch 2) as a single ddoc? So you copy > > > open-sourced ddoc json to any DB, run index.html attachment from the > > ddoc – > > > and you‘re in. > > > > > > Basically, the idea is to re-create Futon-like app as a couchapp. > > > > > > If an idea is reasonable, is it reasonable to make the process openly > > > funded in some way, to make it more rapid and controllable? > > > > > > ermouth > > > > > >
Re: Fauxton, Futon and licensing obstacles
Several sketches http://jquerymy.com/kod/photon1.pdf ermouth 2017-08-20 15:07 GMT+03:00 ermouth : > Seems that Fauxton experiencing some troubles with React unfriendly > licensing. It‘s a pity taking in account CouchDB 2 moved to fairly stable > state, and Fauxton itself received a lot of great improvements since last > year. > > However, in new circumstances, what do you think about re-creating > something like Futon+ (for Couch 2) as a single ddoc? So you copy > open-sourced ddoc json to any DB, run index.html attachment from the ddoc – > and you‘re in. > > Basically, the idea is to re-create Futon-like app as a couchapp. > > If an idea is reasonable, is it reasonable to make the process openly > funded in some way, to make it more rapid and controllable? > > ermouth >
Fauxton, Futon and licensing obstacles
Seems that Fauxton experiencing some troubles with React unfriendly licensing. It‘s a pity taking in account CouchDB 2 moved to fairly stable state, and Fauxton itself received a lot of great improvements since last year. However, in new circumstances, what do you think about re-creating something like Futon+ (for Couch 2) as a single ddoc? So you copy open-sourced ddoc json to any DB, run index.html attachment from the ddoc – and you‘re in. Basically, the idea is to re-create Futon-like app as a couchapp. If an idea is reasonable, is it reasonable to make the process openly funded in some way, to make it more rapid and controllable? ermouth
Introducing Couchbox
Happy to introduce Couchbox, CouchDB query server augment/substitute. See it at https://gitlab.com/Couchbox/couchbox Questions are welcomed. ermouth
New query server
I‘m happy to pre-announce new query server – Couchbox. Couchbox is an augment for CouchDB, that extends built-in query server with so long desired hooks/triggers and async api. Both api methods and hooks are defined as branches of design documents, very similar to lists, shows and updates. Unlike native QS functions, Couchbox hook and api functions are async and see extensions as this._email or this._cache or whatever, bound to current design doc. Not to mention you can query CouchDB during function execution. Couchbox is a combo of CouchDB, node.js 7+, Redis and, optionally, nginx. There is a script to install all dependencies in nearly one click. Couchbox employs one of best CouchDB principles: ‘deploy it, start it once and forget’. That‘s why Couchbox is configured using native CouchDB config, so it can be (and intended to be) managed from Futon. Since Couchbox runs in node.js as a swarm of VMs inside several workers, api and hooks functions fully support ES6 syntax. It also means adding new plugins (new functions visible as this._something) to Couchbox is an easy task. Just take known node.js lib and wrap it with several lines of code – and you have new plugin, available from inside your hooks or api. Couchbox mimics CouchDB practices whenever possible. Request object, inhaled by api functions is well known Req object like in Couch, require() and isArray() work exactly same, so making old lists/updates work in new Couchbox literally takes minutes (and we sometimes observed more then order of magnitude performance boost for old lists running in new QS). Couchbox is specifically designed to work in large distributed nets of CouchDB instances, actually it was created to run pretty complex commercial project (which it now runs). We plan to make Couchbox fully OSS at the end of June, now it lives in restricted dev repo. If somebody wants to preview and play with it right now, please, register at Gitlab and let me know your nickname, I‘ll grant a guest access. Questions are welcomed. ermouth
Re: CouchDB 2.0 and "Intuitive HTTP/JSON API and designed for Reliability" for couchapps
Couch 2.0 fails to process rewritten paths with ampersands (more than 1 param in query). Afaik (however I have not tested) this issue is fixed in 2.1. Although I‘m not sure it is the reason. To note: I plan to uncover a preview of new query server on Monday, as a full replacement for lists, shows, rewrites, updates and so on. With SMS, emails and all missing stuff. So stay tuned. ermouth 2017-05-23 13:47 GMT+03:00 Johs Ensby : > Hi, > we have been using Couch 1.6 with a patch for the rewrite function, but I > wanted to see how 2.0 in single node mode was doing these past two days. > The test case was a rather big web site with a few databases holding a few > thousand documents, images, PDF attachments. It is performing very well on > Couch 1.6 > > Moving it from the cloud instance offline to a local Mac was as > straigthforward as replication can be and with a few tweeks to the > configration the site popped up on my iMac with all bells and whistles. > That is, a few records did not replicate from Couch 1.6 on Ubuntu/AWS to my > local Mac. > > But what quickly turned out to be a bigger problem was that attachments > did not load into the browser reliably. > CSS files and images would sometimes load, somtimes not. Normally the CSS > files would load once and cache in the browser, but that did not prevent > pages from loaded occationally as if the css file was missing (not with a > 404, just show a red GET in the inspector without an error code). > I seem to remember that there was a discussion about Etags in 2.0, but > dont know if this is the issue here. > Strange things like the images coming up instead of the requested images > also happened. > Safari was a lot worse than Chrome. > > Could anyone tell me if these are known bugs that are fixed in 2.1? > > Best regards, > johs > > >
Re: _user database authentication
Please read this http://docs.couchdb.org/en/1.6.1/intro/security.html#users-public-information, hope it help. ermouth 2017-03-01 16:54 GMT+03:00 Nick : > Hello, > > I'm having a problem with a simple couchapp which has users. > > I want to prevent anonymous meddling with the _users database, so I have > an couchdb-wide admin user as described in the docs. > > However, I also want to allow anonymous users to see (via the app) what > users exist. However, I can't quite see how to do that, as anonymous users > are completely denied access to the _user database, even via views. > > Is this even possible? i..e. Allow access to a specific view generated > from _users, but not anything else? > > (I'm using couchdb v1.6.) > > > Thanks! > > Nick > >
Changing replication filter on the fly
There is known feature of continuous filtered replication: until
replication restarts, filter function is frozen.
It means you can start continuous filtered replication using _replicator
DB, then change (or even remove) filter function – and have your sync still
running with an old filter.
To ensure your changed filter can take control, you must explicitly remove
and re-create _replicator doc. It means stop/restart of sync flow and bit
too complex.
There exist another way.
Imagine we have filter fn, distributing docs between nodes according to
some topology. Let‘s say our topology is just a branch of a filter design
doc, and accessible from inside filter fn as this.topology property.
So our ddoc looks like:
{_id:"_design/topo",
topology:{node1:["org1","org2"],node2:["org3"]},
filter:"function(doc, req){ /* whatever */}"}
What happens when we change topology property and save ddoc? Our already
running filter function is not hot-swapped, however it immediately receives
its updated parent ddoc as an input.
So filter fn takes new topology and persists it into this.topology
property. One step:
if (doc._id=="_design/topo") this.topology = doc.topology;
All subsequent changes will be processed with updated topology settings,
and we achieved this without touching _replication DB.
Works like a charm.
ermouth
Re: Couchapp experience in 2016
Thank you, Giovanni! > For me, choosing a 1-tier architecture Actually, we still need kinda reverse proxy in front, mostly to have gzipped answers (and, optionally, prepend Cache-control headers). Without js rewrites I had to use Apache or node to avoid web server recompilation on rules change. Now I can use nginx since rule list is stable and I need to compile nginx only once don‘t bothering about internal routing rules. > by choosing extreme ease of > development(in-browser development, same javascript language for frontend > and backend) I still don‘t think couchapps are easy in common sense, mostly because technologies are limiting and concepts are off mainstream. However, I admit now that couchapp concept is less limiting than I thought a year ago ) ermouth 2016-09-05 11:28 GMT+03:00 Giovanni Lenzi : > Great job ermouth, you are always on the front line on pushing couchapp > borders! > > JS rewrites seems to really enable them to any kind of use. Performance may > be a limit if you are really interested to squeeze each available cpu > cycle, ok but, from a business perspective, how does it cost to reach > and mantain this speed? May be an interesting question for your article > too. > > For me, choosing a 1-tier architecture(couchdb with js rewrites and default > vhost) over the usual 3-tier(web-app-db) is mostly a matter of costs. > Choosing couchdb means keeping costs low by choosing extreme ease of > development(in-browser development, same javascript language for frontend > and backend) ease of maintenance and reduced set of skills(one server only > to learn, configure and mantain with no licenses). > > Would I mind spending some more bucks on hardware (which keeps constantly > improving and getting cheaper) to reach the same performance level of a > more complicated and thus expensive architecture? Of course not. > > Thanks for your work, > --Giovanni > > 2016-09-02 0:00 GMT+02:00 ermouth : > > > Tried out more or less pure couchapp approach in 2016 realities, I mean > JS > > rewrites and PouchDB. > > > > Written down a story about the project, from the couchapp side: > > http://lesorub.pro/how-it-works > > > > It was interesting experience, couchapps still might be useful, in very > > rare cases ) > > > > ermouth > > >
Re: Couchapp experience in 2016
> using more CouchDB design documents simultanously is not fast enogh? It‘s not. There are 2 basic considerations: a) node is faster as a compiler and generates code, that runs faster, b) we can avoid excessive serialize/parse calls between OTP and SM, which are both CPU and IO hungry for large branchy docs. Although we add several more layers here, and they add latency, which diminish our performance boost. So I‘m not sure if 5x rps and bandwidth per core achievable, gonna play with it a little. ermouth
Re: Couchapp experience in 2016
> from someone reaching the end of couchapp > possibilities by developing a software stack which does'nt need local > installation Thanks. Reaching the end means going back with new experience. Ddoc Lab is able to generate attaches with npm package tarballs, and npm allows direct install from CouchDB. I mean deploying from Couch to FS, not vice versa as we usually do. That is my "going back" from Couch to FS-based solutions: FS apps still follow data. Now I‘m thinking about composing kinda node.js module, able to augment query server in part of show, list, update and, possibly, reduce. There exist nice set of modules like pouchdb/pouchdb-list. I gonna try to combine couple of them, and, intercepting appropriate requests by node.js, exec lists and similar queries inside node.js. Just for an experiment, to see what it gives. Actually (couchapp + JSrewrite + PouchDB) architecture is pretty nice and even attractive in terms of stability and, well, purity – but lacks performance. So if we could have a performance booster, say 5x, I‘d use couchapps not only for small things or interims. ermouth
Re: Couchapp experience in 2016
Hi, Robin! > I'm very interested in couchapps > although the definition isn't too > clear for me Nobody knows exact definition. The variety of solutions entitling itself ‘couchapp’ it quite wide. They use different client-side frameworks and architectures, since CouchDB does not dictate. However they all are served from CouchDB and often use CouchDB queries to fetch/store data. So if we take the term in fuzzy meaning, couchapp is a web app hosted inside CouchDB and using CouchDB features to run. ermouth
Couchapp experience in 2016
Tried out more or less pure couchapp approach in 2016 realities, I mean JS rewrites and PouchDB. Written down a story about the project, from the couchapp side: http://lesorub.pro/how-it-works It was interesting experience, couchapps still might be useful, in very rare cases ) ermouth
Re: Caching results of _list functions
Tested approach, works like a charm )
Surprisingly, it appears that approach is more suitable not for rendering
pages, but for requests view->list, that respond with hundreds or thousands
of small rows. In those cases I observe sometimes 2 orders of magnitude
gain or even more, ie 300-500ms query server exec time turns into 3-5ms.
Even adding net latencies, that are around 100ms, gain is significant.
For page render, net latencies dim the gain, cause there is no visible
difference between 100 and 150 ms lag at client side.
ermouth
2016-05-19 11:41 GMT+03:00 ermouth :
> > - a function in a _list of a certain ddoc called directly via the _list
> endpint
>
> It‘s called via rewrite. Rewrite decides to call list. If rewrite already
> have a result in cache and the result is not too old, rewrite sends results
> directly.
>
> If there is no result cached or it‘s too old, rewrite redirects to list.
> And list, in turn, not only sends result, but caches it as well.
>
> > - could you give us som sinppets
>
> You make branch in ddoc, say .Cache, with code like:
>
> (function(){
> var cache = {};
> exports.Cache = {
> get: function (key) {return cache[key]},
> put: function (key, val) {cache[key] = val;}
> }
> })();
>
> In both rewrite and list to access cache you just use
>
> var Cache = require ('Cache').Cache;
>
> Now you can use it like Cache.get(key) and Cache.put(key, val) in rewrite
> and list. Your cache will persist between requests until you change ddoc or
> SM crashes.
>
>
> ermouth
>
> 2016-05-19 7:42 GMT+03:00 Johs Ensby :
>
>> Hi Ermouth,
>> this sounds like a bonus feature of significant value:)
>> Together with the rewrite function this discovery would keep the couchapp
>> enthusiasts of the community happy for a looong time, I believe.
>>
>> If I understand you right,
>> - a function in a _list of a certain ddoc called directly via the _list
>> endpint (NOT via _rewrite)
>> - can cache a rendered result (e.g. a html page)
>> - for a subsequent request via _rewrite (of the same ddoc) to pick up the
>> cached result from RAM
>> - using a timestamp inside that _rewrite function to compare against the
>> timestamped name of the cached result to validate that the age is within a
>> chosen tolerance
>>
>> Following this 2-step approach (_list, then _rewrite)..
>> - the _list function would have the full request context
>> - while the _rewrite would have the limited request object available
>> - the _rewrite and _list endpoints would need to be available
>>
>> Control questions
>> - does this mean that you cannot point a vhost to _rewrite and run all
>> calls to _list via rewrite functions?
>> - if you have one vhost pointing to _rewrite of a ddoc and another to
>> e.g. _list would they still share the cache?
>>
>> Testing this
>> - could you give us som sinppets on how to test this, I would love to see
>> this in action and test it towards my JS rewrite functions.
>>
>> br
>> johs
>>
>>
>> > On 19. mai 2016, at 00.01, ermouth wrote:
>> >
>> > Since JS rewrites landed in 2.0, and I use JS rewrites heavily, I
>> discover
>> > some tricks with the new feature time to time. Despite I use patched
>> 1.6.1
>> > with js rewrites, tricks are suitable for at least single node 2.0 as
>> well.
>> >
>> > JS rewrites function runs in the context of a query server instance. QS
>> > instance is a Spidermonkey process with 64Mb of RAM by default. It‘s
>> single
>> > threaded.
>> >
>> > Actually, seems that all query server functions of one particular design
>> > doc are executed inside one dedicated Spidermonkey instance. It means
>> they
>> > all share global context (also it means they share one thread).
>> >
>> > Ddoc global context persists until Spidermonkey process is restarted. It
>> > happens when SM instance crashes (runs out of RAM quota, unrecoverable
>> > error happens etc), or when underlying design doc is changed.
>> >
>> > It means _list function can cache results of rendering into RAM, and
>> > _rewrite function can later quickly respond with cached version,
>> skipping
>> > redirect to heavy map/list. Cache container is just JS closure,
>> > instantiated/accessed using built-in require().
>> >
>> > Unlike _list, JS rewrite function does not receive req.info, it would
>> be
>> > too slow to tap DB on each request. It means we can not invalidate
>> cache on
>> > local sequence change (DB update) – rewrite function just does not know
>> DB
>> > was updated.
>> >
>> > However we can use timestamp-based caching, even several seconds TTL of
>> > cache entries is enough to reduce average response time significantly
>> (say,
>> > order of magnitude).
>> >
>> > I‘ve tested approach a little and found it very promising.
>> >
>> > Since this dirty hack is possible with _list fns only, without js
>> rewrites,
>> > may be someone already employs it? Or at least tried?
>> >
>> > ermouth
>>
>>
>
Re: Caching results of _list functions
> - a function in a _list of a certain ddoc called directly via the _list
endpint
It‘s called via rewrite. Rewrite decides to call list. If rewrite already
have a result in cache and the result is not too old, rewrite sends results
directly.
If there is no result cached or it‘s too old, rewrite redirects to list.
And list, in turn, not only sends result, but caches it as well.
> - could you give us som sinppets
You make branch in ddoc, say .Cache, with code like:
(function(){
var cache = {};
exports.Cache = {
get: function (key) {return cache[key]},
put: function (key, val) {cache[key] = val;}
}
})();
In both rewrite and list to access cache you just use
var Cache = require ('Cache').Cache;
Now you can use it like Cache.get(key) and Cache.put(key, val) in rewrite
and list. Your cache will persist between requests until you change ddoc or
SM crashes.
ermouth
2016-05-19 7:42 GMT+03:00 Johs Ensby :
> Hi Ermouth,
> this sounds like a bonus feature of significant value:)
> Together with the rewrite function this discovery would keep the couchapp
> enthusiasts of the community happy for a looong time, I believe.
>
> If I understand you right,
> - a function in a _list of a certain ddoc called directly via the _list
> endpint (NOT via _rewrite)
> - can cache a rendered result (e.g. a html page)
> - for a subsequent request via _rewrite (of the same ddoc) to pick up the
> cached result from RAM
> - using a timestamp inside that _rewrite function to compare against the
> timestamped name of the cached result to validate that the age is within a
> chosen tolerance
>
> Following this 2-step approach (_list, then _rewrite)..
> - the _list function would have the full request context
> - while the _rewrite would have the limited request object available
> - the _rewrite and _list endpoints would need to be available
>
> Control questions
> - does this mean that you cannot point a vhost to _rewrite and run all
> calls to _list via rewrite functions?
> - if you have one vhost pointing to _rewrite of a ddoc and another to e.g.
> _list would they still share the cache?
>
> Testing this
> - could you give us som sinppets on how to test this, I would love to see
> this in action and test it towards my JS rewrite functions.
>
> br
> johs
>
>
> > On 19. mai 2016, at 00.01, ermouth wrote:
> >
> > Since JS rewrites landed in 2.0, and I use JS rewrites heavily, I
> discover
> > some tricks with the new feature time to time. Despite I use patched
> 1.6.1
> > with js rewrites, tricks are suitable for at least single node 2.0 as
> well.
> >
> > JS rewrites function runs in the context of a query server instance. QS
> > instance is a Spidermonkey process with 64Mb of RAM by default. It‘s
> single
> > threaded.
> >
> > Actually, seems that all query server functions of one particular design
> > doc are executed inside one dedicated Spidermonkey instance. It means
> they
> > all share global context (also it means they share one thread).
> >
> > Ddoc global context persists until Spidermonkey process is restarted. It
> > happens when SM instance crashes (runs out of RAM quota, unrecoverable
> > error happens etc), or when underlying design doc is changed.
> >
> > It means _list function can cache results of rendering into RAM, and
> > _rewrite function can later quickly respond with cached version, skipping
> > redirect to heavy map/list. Cache container is just JS closure,
> > instantiated/accessed using built-in require().
> >
> > Unlike _list, JS rewrite function does not receive req.info, it would be
> > too slow to tap DB on each request. It means we can not invalidate cache
> on
> > local sequence change (DB update) – rewrite function just does not know
> DB
> > was updated.
> >
> > However we can use timestamp-based caching, even several seconds TTL of
> > cache entries is enough to reduce average response time significantly
> (say,
> > order of magnitude).
> >
> > I‘ve tested approach a little and found it very promising.
> >
> > Since this dirty hack is possible with _list fns only, without js
> rewrites,
> > may be someone already employs it? Or at least tried?
> >
> > ermouth
>
>
Caching results of _list functions
Since JS rewrites landed in 2.0, and I use JS rewrites heavily, I discover some tricks with the new feature time to time. Despite I use patched 1.6.1 with js rewrites, tricks are suitable for at least single node 2.0 as well. JS rewrites function runs in the context of a query server instance. QS instance is a Spidermonkey process with 64Mb of RAM by default. It‘s single threaded. Actually, seems that all query server functions of one particular design doc are executed inside one dedicated Spidermonkey instance. It means they all share global context (also it means they share one thread). Ddoc global context persists until Spidermonkey process is restarted. It happens when SM instance crashes (runs out of RAM quota, unrecoverable error happens etc), or when underlying design doc is changed. It means _list function can cache results of rendering into RAM, and _rewrite function can later quickly respond with cached version, skipping redirect to heavy map/list. Cache container is just JS closure, instantiated/accessed using built-in require(). Unlike _list, JS rewrite function does not receive req.info, it would be too slow to tap DB on each request. It means we can not invalidate cache on local sequence change (DB update) – rewrite function just does not know DB was updated. However we can use timestamp-based caching, even several seconds TTL of cache entries is enough to reduce average response time significantly (say, order of magnitude). I‘ve tested approach a little and found it very promising. Since this dirty hack is possible with _list fns only, without js rewrites, may be someone already employs it? Or at least tried? ermouth
UTF-8 encoding for JSONs with non-latin chars
CouchDB normally delivers non-latin chars in JSON encoded as \u entities, that is in most cases meaningless overhead. Known roundtrip, lists that re-encode responses, is very slow for views and not applicable to replication traffic at all. Good way is to rebuild CouchDB from sources, making these changes: 1. Replace mochijson2.erl with https://github.com/mochi/mochiweb/blob/master/src/mochijson2.erl, that has critical fix 2. Change utf8=false to utf8=true at LOC #95 3. Recompile CouchDB. Voila, now all json responces are utf-8. ermouth
Re: Pattern matching lib for couchapps
See this screenshot for example: http://jquerymy.com/lj/f554d63c0fa4_7167/201511267.48.14.png It‘s just a fragment, but it‘s quiet transparent. ermouth 2016-01-05 9:38 GMT+03:00 Johs. E : > Ermouth, > could you share a snippet to show how this is used in the new rewrite > function? > br > Johs > > ps > happy new year to all @couchapp > > > On 16 Nov 2015, at 20:37, ermouth wrote: > > > > There exist nice lib for Erlang-style pattern-matching in JS – > matches.js ( > > https://github.com/natefaubion/matches.js). > > > > I‘ve prepared gist http://bit.ly/1QqQy4w, that contains bit extended and > > modified matches.js, see comment under the gist. > > > > In short: now ready for ddocs (knows about CouchDB require) and supports > > recursive syntax. See comment under the gist and original matches.js > docs. > > > > Note, that compilation of matcher is not lazy, but it‘s nearly > > indistinguishable comparing to other penalties of typical JS-backed > request > > in CouchDB. > > > > Good for new js rewrites (routing) and for (validate_doc_)update update > > functions (type check). > > > > ermouth > >
Re: [PROPOSAL] Deprecate global functions in query server
> this.emit(doc.key)
Normally this.propName points to the ddoc property, if any. So better
this._emit since properties with _ are prohibited for docs.
ermouth
2015-12-12 16:03 GMT+03:00 Jan Lehnardt :
> I’m in favour of doing something like this.
>
> How about adding them to the function in question?
>
> So
>
> function(doc) {
> emit(doc.key)
> }
>
> would become
>
> function(doc) {
> this.emit(doc.key)
> }
>
> +1. no need to mess with the function signature.
> +2. no clunky here’s-a-bunch-functions object that isn’t already there.
> +3. most upgrades are string replace away.
>
> -1. overloading this could be seen as clunky in itself.
> -2. still doesn’t solve that `function() {}` is not valid JavaScript and
> doesn’t work in newer SpiderMonkeys or other engines. But that might be out
> of scope.
>
> Best
> Jan
> --
>
>
>
> > On 02 Dec 2015, at 14:06, Alexander Shorin wrote:
> >
> > While I like require(...) way to handle this issue, I will allow
> > myself to disagree with the emit argument in function signature.
> > Because if we upgrade SM to get generators feature then we can throw
> > away emit at all. Map functions are perfect generators and Python
> > query server experience showed that this is great idea.
> > SM / JS support upgrade is unavoidable, so let's look a little bit far
> > in the future to now fix same things twice (;
> >
> > --
> > ,,,^..^,,,
> >
> >
> > On Wed, Dec 2, 2015 at 4:01 PM, Garren Smith wrote:
> >> Ok you make good points and I wasn’t aware of all these functions
> http://docs.couchdb.org/en/1.6.1/query-server/javascript.html <
> http://docs.couchdb.org/en/1.6.1/query-server/javascript.html>
> >> I’m going to do one more bike shed and then I will leave it up to you
> and other people who actually want to implement this.
> >> For the case of map reduce, I would go with a function call like I said
> before
> >>
> >> function (doc, emit) {
> >>
> >> }
> >>
> >> Then for all our other helpers I would make them available through a
> require. This in my mind makes it similar to node.js or to a browser if you
> use browserify, web pack, requirejs etc
> >> So a full example would be
> >>
> >> function (doc, emit) {
> >> var isArray = require(‘helpers’).isArray;
> >> var allHelpers = require(‘helpers’);
> >>
> >>
> >> }
> >>
> >> That is my 2 cents worth.
> >>
> >> Cheers
> >> Garren
> >>
> >>> On 02 Dec 2015, at 2:42 PM, Alexander Shorin wrote:
> >>>
> >>> Hi Garren,
> >>>
> >>> On Wed, Dec 2, 2015 at 3:35 PM, Garren Smith
> wrote:
> >>>>
> >>>> This is an interesting idea. I think passing the emit function in is
> a good idea and might make somethings easier in PouchDB. I would rather a
> map function looked something like this
> >>>>
> >>>> function (doc, emit) {
> >>>>
> >>>> }
> >>>
> >>> That's nice, but won't work: map function has access to a lot of other
> >>> global functions we provide, so you'll end with signature of 7
> >>> arguments. I don't think there is any reason to fix this problem by a
> >>> half. Just fix all globals or no one.
> >>>
> >>>> I don’t like the idea of passing a userCtx object. That feels overly
> bulky. Things like JSON/require are global variables in Node.js or the
> browser so my feeling is to follow their lead on those.
> >>>
> >>> ctx referenced not to userCtx, but a generic context object that hold
> >>> all the global function and objects we provide now. These are listed
> >>> in reference on documentation. May be we can find a better name to not
> >>> cause confusion. funcs? env? Suggestions welcome (:
> >>>
> >>> --
> >>> ,,,^..^,,,
> >>
>
>
Re: [PROPOSAL] Chainable requests
> Are all the scheduled hops completely
> executed, or the chain will be interrupted?
I also have this question in mind (and several others of comparable
complexity) – thats why I‘m still deciding is overall ‘loop’ approach
reasonable.
As for me, it should not be interrupted until one of chain members
explicitly tries to send data to closed connection. But even in this case
I‘m not sure chain must stop.
Your considerations?
ermouth
2015-12-03 12:25 GMT+03:00 Giovanni Lenzi :
> 2015-12-01 18:04 GMT+01:00 ermouth :
>
> > Off: I like your new logo (your avatar?) very much! Nice and professional
> > work, gratz to logo‘s author.
> >
>
> :) Thanks, really appreciated!!
>
>
> >
> > > Would it be possible to make external http calls too?
> >
> > Not sure if it‘s possible. But will investigate, mb loop through proxy or
> > smth.
> >
> > > requests generating infinite loops
> >
> > Well, rewrite counts number of hops. This feature also can count and have
> > upper limit of iterations.
> >
>
> Great, a hop counter is simple and nice to have. It may eventually allow
> developers to catch unwanted long-running operations or simply know how
> many hops are left.
>
> One question: from what I understand client connections are kept open by
> the server until all hops are finished. But what if a client closes the
> connection before hops chain end? Are all the scheduled hops completely
> executed, or the chain will be interrupted?
>
>
> -- Giovanni
>
>
>
> >
> > 2015-12-01 16:17 GMT+03:00 Giovanni Lenzi :
> >
> > > Wow, very useful!!!
> > >
> > > It would be finally possible to aggregate multiple document updates on
> > the
> > > server-side, thus implementing server-side actions in a secure and
> > complete
> > > way, calling a single application api method. I also like the two new
> > ways
> > > of priting output: chunked and reduced.
> > >
> > > Would it be possible to make external http calls too? That would allow
> > > developers to build full js integration libraries for third party
> > services,
> > > withouth the need for any node.js external process!!
> > >
> > > I'm afraid about requests generating infinite loops, but I'm more
> afraid
> > > that trying to simply kill them, may negatively affect some use cases.
> > >
> > >
> > > --Giovanni
> > >
> > > 2015-11-26 6:07 GMT+01:00 ermouth :
> > >
> > > > > is not like multiple lines of code in a server side .asp php or
> node
> > > > program
> > > > > it is more like "ask for this, and depending on the answer,
> > > > > ask for this" like a "request tree"
> > > >
> > > > Actually, that _is_ like asp, php or node program :) As for php guys
> > > > approach is nearly native, sync program code runs inbetween DB
> > requests.
> > > >
> > > > As for node.js guys it also could be understood in native paradigm:
> > think
> > > > couchapp functions are kinda middleware in request processing chain,
> > and
> > > > each middleware function can determine next step.
> > > >
> > > > Somehow like express.js maybe, but in express you call next(args)
> when
> > in
> > > > CouchDB you just return {path:"...", body:"args"}. Also in node your
> > > > middleware can be async, but in CouchDB it should be sync.
> > > >
> > > >
> > > >
> > > > ermouth
> > > >
> > > > 2015-11-26 5:11 GMT+03:00 Johs Ensby :
> > > >
> > > > > Ermouth,
> > > > >
> > > > > > On 25. nov. 2015, at 18.18, ermouth wrote:
> > > > > > chunked response and reduce approach
> > > > >
> > > > >
> > > > > I think both modes are valuable, conceptually we end up with 3
> modes
> > of
> > > > > respons
> > > > > Technically it makes sense to describe as server response.
> > > > >
> > > > > I am trying to think of how we want to spin this to the new
> > developers
> > > > >
> > > > > I would recommend that we name the feature as seen from the
> front-end
> > > > > developer
> > > > >
> > > > > - single request
> > > > > - chained request
> > > > > - progressive load
> > > > >
> > > > > The 3rd being a variant of chained request not accumulating but
> > > spitting
> > > > > output into the client for as long as it takes
> > > > >
> > > > > "Single reqest" is the normal thing, but what we see as one of the
> > > > painful
> > > > > limitations of Couch
> > > > > "Chained request" is the new thing that is not like multiple lines
> of
> > > > code
> > > > > in a server side .asp php or node program, it is more like "ask for
> > > this,
> > > > > and depending on the answer, ask for this" like a "rewuest tree"
> > > > > "Progressive load" is a super way to improve performance, it is
> > writing
> > > > > for UX, "what the user needs right away is... then and
> > eventually
> > > > > she/he will be looking for .. if a link in the first chucks has
> been
> > > > > clicked yet"
> > > > >
> > > > > johs
> > > >
> > >
> >
>
Re: [PROPOSAL] Deprecate global functions in query server
> It is not possible to check how many arguments functions accepts?
It is, `var fn=function(x,y){};fn.length ==> 2`. Although, you not
necesserily write js function with all args. For example, I can define
update (doc), without req obj if I don‘t need it. Gist below uses only
first arg of 4 in validate_doc_update.
> How this._* will be better than global context?
First, we have global context clean. Second, we do not have to change args
length for existing functions, so we can in many cases just find/replace to
refactore.
I thought a little, and if main reason is security, we better write
validate_doc_update, that parses ddocs and checks if functions use
inappropriate globals or, for example, eval. Validation of this kind is
simple, see here working example
https://gist.github.com/ermouth/953691b6b62c9cab4ff6.
ermouth
2015-12-02 19:17 GMT+03:00 Alexander Shorin :
> On Wed, Dec 2, 2015 at 6:44 PM, ermouth wrote:
> > Ain‘t it all too stressful?
> >
> > Approach will make _all_ existing code broken, that will, in turn,
> > effectively deny upgrade for most of users.
>
> Deprecate != remove. Both approaches may co-exist for some time till
> we find a way how to gracefully upgrade the code during yet another BC
> release. The drawback of such upgrade will be all views rebuild,
> suddenly.
>
> > Think old and new approaches better co-exist during several subversions.
> > Note, that co-existanse makes passing ctx via 1st arg impossible.
>
> It is not possible to check how many arguments functions accepts? I
> thought about this way to deal with this problem.
>
> > As for me, I‘d prefer to have emit and co as `this` childs. Like
> > this._emit, this._send, this._getRow and so on. Since properties with
> > leading _ can not exist in ddoc, referenced by `this`, we can extend
> `this`
> > free.
> >
> > Actually, we must have several shallow copies of ddoc, each extended with
> > own set of underscored methods. Each calll to QS function performed using
> > .apply with and appropriate shallow copy.
>
> How this._* will be better than global context? Just curious.
>
> --
> ,,,^..^,,,
>
Re: [PROPOSAL] Deprecate global functions in query server
Ain‘t it all too stressful? Approach will make _all_ existing code broken, that will, in turn, effectively deny upgrade for most of users. Think old and new approaches better co-exist during several subversions. Note, that co-existanse makes passing ctx via 1st arg impossible. As for me, I‘d prefer to have emit and co as `this` childs. Like this._emit, this._send, this._getRow and so on. Since properties with leading _ can not exist in ddoc, referenced by `this`, we can extend `this` free. Actually, we must have several shallow copies of ddoc, each extended with own set of underscored methods. Each calll to QS function performed using .apply with and appropriate shallow copy. ermouth
Re: [PROPOSAL] Chainable requests
Off: I like your new logo (your avatar?) very much! Nice and professional
work, gratz to logo‘s author.
> Would it be possible to make external http calls too?
Not sure if it‘s possible. But will investigate, mb loop through proxy or
smth.
> requests generating infinite loops
Well, rewrite counts number of hops. This feature also can count and have
upper limit of iterations.
> trying to simply kill them, may negatively affect some use cases
It‘s in any way inevitable if you have transactional-style architecture for
a DB that can not perform transactions. So make it in non-transactional
style )
ermouth
2015-12-01 16:17 GMT+03:00 Giovanni Lenzi :
> Wow, very useful!!!
>
> It would be finally possible to aggregate multiple document updates on the
> server-side, thus implementing server-side actions in a secure and complete
> way, calling a single application api method. I also like the two new ways
> of priting output: chunked and reduced.
>
> Would it be possible to make external http calls too? That would allow
> developers to build full js integration libraries for third party services,
> withouth the need for any node.js external process!!
>
> I'm afraid about requests generating infinite loops, but I'm more afraid
> that trying to simply kill them, may negatively affect some use cases.
>
>
> --Giovanni
>
> 2015-11-26 6:07 GMT+01:00 ermouth :
>
> > > is not like multiple lines of code in a server side .asp php or node
> > program
> > > it is more like "ask for this, and depending on the answer,
> > > ask for this" like a "request tree"
> >
> > Actually, that _is_ like asp, php or node program :) As for php guys
> > approach is nearly native, sync program code runs inbetween DB requests.
> >
> > As for node.js guys it also could be understood in native paradigm: think
> > couchapp functions are kinda middleware in request processing chain, and
> > each middleware function can determine next step.
> >
> > Somehow like express.js maybe, but in express you call next(args) when in
> > CouchDB you just return {path:"...", body:"args"}. Also in node your
> > middleware can be async, but in CouchDB it should be sync.
> >
> >
> >
> > ermouth
> >
> > 2015-11-26 5:11 GMT+03:00 Johs Ensby :
> >
> > > Ermouth,
> > >
> > > > On 25. nov. 2015, at 18.18, ermouth wrote:
> > > > chunked response and reduce approach
> > >
> > >
> > > I think both modes are valuable, conceptually we end up with 3 modes of
> > > respons
> > > Technically it makes sense to describe as server response.
> > >
> > > I am trying to think of how we want to spin this to the new developers
> > >
> > > I would recommend that we name the feature as seen from the front-end
> > > developer
> > >
> > > - single request
> > > - chained request
> > > - progressive load
> > >
> > > The 3rd being a variant of chained request not accumulating but
> spitting
> > > output into the client for as long as it takes
> > >
> > > "Single reqest" is the normal thing, but what we see as one of the
> > painful
> > > limitations of Couch
> > > "Chained request" is the new thing that is not like multiple lines of
> > code
> > > in a server side .asp php or node program, it is more like "ask for
> this,
> > > and depending on the answer, ask for this" like a "rewuest tree"
> > > "Progressive load" is a super way to improve performance, it is writing
> > > for UX, "what the user needs right away is... then and eventually
> > > she/he will be looking for .. if a link in the first chucks has been
> > > clicked yet"
> > >
> > > johs
> >
>
Re: [PROPOSAL] Chainable requests
> is not like multiple lines of code in a server side .asp php or node
program
> it is more like "ask for this, and depending on the answer,
> ask for this" like a "request tree"
Actually, that _is_ like asp, php or node program :) As for php guys
approach is nearly native, sync program code runs inbetween DB requests.
As for node.js guys it also could be understood in native paradigm: think
couchapp functions are kinda middleware in request processing chain, and
each middleware function can determine next step.
Somehow like express.js maybe, but in express you call next(args) when in
CouchDB you just return {path:"...", body:"args"}. Also in node your
middleware can be async, but in CouchDB it should be sync.
ermouth
2015-11-26 5:11 GMT+03:00 Johs Ensby :
> Ermouth,
>
> > On 25. nov. 2015, at 18.18, ermouth wrote:
> > chunked response and reduce approach
>
>
> I think both modes are valuable, conceptually we end up with 3 modes of
> respons
> Technically it makes sense to describe as server response.
>
> I am trying to think of how we want to spin this to the new developers
>
> I would recommend that we name the feature as seen from the front-end
> developer
>
> - single request
> - chained request
> - progressive load
>
> The 3rd being a variant of chained request not accumulating but spitting
> output into the client for as long as it takes
>
> "Single reqest" is the normal thing, but what we see as one of the painful
> limitations of Couch
> "Chained request" is the new thing that is not like multiple lines of code
> in a server side .asp php or node program, it is more like "ask for this,
> and depending on the answer, ask for this" like a "rewuest tree"
> "Progressive load" is a super way to improve performance, it is writing
> for UX, "what the user needs right away is... then and eventually
> she/he will be looking for .. if a link in the first chucks has been
> clicked yet"
>
> johs
[PROPOSAL] Chainable requests
TL;DR: Allow list, show and update (lsu) functions return special object
for chaining requests. In short, I propose lsu fns could return object of
same structure as JS rewrite does:
{path:"...", method:"...", body:"..."}.
If query server receives this kind of object as lsu fn result, user
connection kept open and next internal request performed. We can use .body
as accumulator inbetween requests.
This feature allows multistep map/reduce, deep data manipulation and
analysis, sites with server-rendered composite pages, service DB scripts
and so on.
Background
=
There can exist several approaches for multi-requests. Most of them tend to
use some kind of new DSL to describe requests server should perform, and in
which order results should be returned. In some cases I saw attempts to add
kinda postprocessor at the end: adapted _list or so.
Also despite CouchDB users understand inevitable penalties of multi-request
approach, feature is highly needed.
Multistep map/reduce
If we allow lsu functions to define, where to go next after they finished,
we can chain several query server requests during one user request. To make
our chain a reducer, we must have accumulator.
Actually, correctly formatted .body property could be used without adding
new functionality. For example, we can reserve key acc in form-encoded
.body and pass intermediate results using this key.
To avoid parse/serialize acc data to form-encoded string we can add .acc
property to the returned object, or smth like this.
Chunked response
===
For chains, intended to generate HTML response, we can go without
accumulator at all. Each part of chain sends its part of response, and last
chain member ends up user connection.
Example: view1/list1 sends headers and page menu, then rewrites to
view2/list2 which sends main article and rewrites to view3/list, which
sends readmore section and footer. All in one user request.
Pseudographics, illustrating both chunked response and reduce approach:
http://bit.ly/1NrTddP
Service scripts
===
Since we can perform update fns along the chain, we can write scripts like
‘Copy all docs of type N from DB1 to DB2, removing props X, Y and Z from
docs.’
Performance
==
Well, at most average ) If we could cache req.info along chain and optimize
.body/.form processing, it might improve situation a little. But main lags
appear during Erlang<->JS communication, not JS executuion per se.
Implementation and niceties
=
Implementation doesn‘t seem to be very difficult – only existing well known
paradigms are used, nearly nothing new added.
Think I could try to implement it, my modest Erlang kung-fu should be
enough for features described.
To note: these features will make CouchDB Turing complete, really.
Opinions welcome. Preliminary discussions showed both negative and positive
feedback, so putting it to public.
ermouth
Re: serving html pages built from multiple lists or shows
> If you could find the time to do a screen recording of what you just described https://www.youtube.com/watch?v=_FoI2An3JRc Bit messy since it‘s impromptu ermouth
Re: serving html pages built from multiple lists or shows
> I think it should be possible to use ermouth's ddoc.me to obtain the same > result much more easily. Maybe he can speak for us.. Since Ddoc Lab can post-process each item with individual code for an item, you can precompile your data in any suitable way before publishing. You can even keep you templates in external docs – Ddoc Lab is able to fetch externals before processing. To have handlebars inside Ddoc Lab document, you should: - create any code snippet, it can be dummy - mark it as having postprocessor - put HB code into postprocessor and make pre-checker wrapper, which will prevent HB init, if it was already initialized Now each postprocessor have access to window.Handlebar. This approach is usefull for other libs also. So to have your template precompiled, just write a template and postprocessor for it (think `return Handlebars.precompile(item.data)` – will be enough). ermouth 2015-11-19 4:52 GMT+03:00 Giovanni Lenzi : > Hi Robin and Nick, > You definitely CAN output full html, non js, SEO optimized web pages. We > use that for our store index and apps pages (on > https://www.smileupps.com/store). > > We needed to: > 1. create html templates of your pages with a templating library(we used > handlebars.js), > 2. precompile these templates before 'couchapp push' as server side > javascript file in ddoc, such as lib/templates.js > 3. create a view with name(path) of your page as key, emitting documents > containing information of a same page under the same key. These elements > may be html for header, links for navigation bar, markdown or raw html for > your page content, a list of apps, or anything else > 4. create a list which uses: > - getrow() to fetch above elements in memory, > - var tpls=require(lib/templates), > - obtain html by merging templates and documents: > htmlforheader=tpls.header(headerdocument), > htmlfornavbar=tpls.navbar(linksdocuments) > - return/output html from the list function > > Probably the most difficult part is to precompile correctly those > templates, in a way you can use them server side withouth issues. We used a > node.js handlebar precompiler automatically run before 'couchapp push', but > I think it should be possible to use ermouth's ddoc.me to obtain the same > result much more easily. Maybe he can speak for us.. > > Hope this helps, > --Giovanni > Il giorno 19/nov/2015 00:13, "Robin Millette" ha > scritto: > > > Hi, > > > > First time writing to this list, or any couchdb list for that matter. > > Hope to see a brillant couchapp (the concept) revival in 2016! > > > > On Wed, Nov 18, 2015 at 5:53 PM, Nick > wrote: > > > > > Thanks - which indeed implies I am right and there is no solution that > > works > > > without JS on the client... > > > > I also prefer to send complete html responses, for SEO and general > > crawling benefits. > > One strategy I used was to craft views to output different kinds of > > rows. For instance, you've got you main content row, and you can also > > have "block" rows, to pepper your html page with. > > The list fonction can then handle those different row types and > > generate proper html. > > > > > > -- > > Robin > > >
Re: "Couch-served app" vs. couchapp
> the term, which is ’couch+app’, means simply an app stored,running > and delivered from couchdb, nothing more. +1000. Couchapp is an app, stored at and running from CouchDB. Nothing more. Tools, that used to create couchapp-related (d)docs and set up CouchDB, are _not_ couchapps. Unless they are served from CouchDB, certainly, and require no more than CouchDB to run. So, “couchapp builder”, “couchapp tool”, “couchapp IDE” + distinguishable own name is best option if we make tools/utilities. Erica, Ddoc Lab, Smileupps are good as names, one more “couchapp for #anylang” is definitely not. Surely, nobody wants to re-title couchapp for py – you can‘t ask grandpa to change his name. But we can at least be more accurate choosing a name for our own newborn tools. ermouth
Pattern matching lib for couchapps
There exist nice lib for Erlang-style pattern-matching in JS – matches.js ( https://github.com/natefaubion/matches.js). I‘ve prepared gist http://bit.ly/1QqQy4w, that contains bit extended and modified matches.js, see comment under the gist. In short: now ready for ddocs (knows about CouchDB require) and supports recursive syntax. See comment under the gist and original matches.js docs. Note, that compilation of matcher is not lazy, but it‘s nearly indistinguishable comparing to other penalties of typical JS-backed request in CouchDB. Good for new js rewrites (routing) and for (validate_doc_)update update functions (type check). ermouth
Re: Siri integration with CouchDB 1.7.0
I‘ve tried to emit https://www.talater.com/annyang/ into CloudWall – got a lot of fun :) But all these solutions generate significant traffic all the time mic is on. ermouth
Re: CouchDB 1.7.0 Roadmap
Good news. Thanks, @kxepal. We‘ve ported js rewrites to 1.6.1, but not yet checked how it works. Results gonna be next monday, will share here. Think code might be useful. ermouth 2015-11-12 18:55 GMT+03:00 Johs Ensby : > May the force be with you! > Johs > > > > Begin forwarded message: > > > > Since "everyone is busy on 2.0" I'll take care of this. > >
Re: CouchApp build tools
> Do you mean that you completely skipped version control? Since DdocLab source docs are PouchDB/CouchDB/Barrel docs, and have revisions, actually, there exist version control. UI of public version does not give you control over revisions, but it‘s a matter of time. I personally use aside mechanics for it, and this mechanics will be eventually merged inside Ddoc Lab. Task isn‘t so easy, cause we need to be protected from accidental DB compaction. TWIMC, current roadmap for Ddoc Lab is: 1. Create good manual – too many features are now hidden under small UI buttons/checkboxes. 2. Add TAR as a special attach type, able to combine several files into it – to allow `npm install http://localhost:5984/db/_design/ddoc/npmPackage.tar` scenario 3. Add jsondiff component to allow source revisions comparison 4. Add github support (first one way, then two way) 5. Add Erlang editor (to allow Erlang views etc) ermouth ermouth 2015-11-05 13:48 GMT+03:00 Aurélien Bénel : > > the advandage I see (…) is that we skipped Github fir ddocs because it > actually works well in a team too. > > Do you mean that you completely skipped version control? As for us, we > couldn’t do that. > > Don’t take it personally, Ddoc.me is really interesting (and I think I > will use it to teach CouchDB). But there is still a need for a command line > tool for teams that require traceability, visibility or testability. > > > Regards, > > Aurélien
[PROPOSAL] Speed up _list and _update optimizing req object
Both list and update functions are quiet slow. Preliminary investigations showed they are not so slow per se, serious lag is produced by preparation of req object. The format of req object: http://docs.couchdb.org/en/latest/json-structure.html#request-object Subjects for optimiztion: * .info object generation steals 1-2ms even on very powerful machines * .form object generation and transmission via stdio can also take significant time, especially taking in account this field is, in fact, a restructured .body string (read – dupe). As for .info – I‘d propose to eliminate or simplify it, but it can break backward compatibility. So if someone has suggestions how it could be optimized – welcome. If someone uses .info in real life scenarios – please, comment this post and tell a story how you use it. As for .form object, it seems more reasonable to parse .body right inside Spidermonkey. Moreover, it could be done in lazy manner – using getter proxy. It means .form property in .req object should be defined as a getter function, that parses .body only when .form is accessed. ermouth
[PROPOSAL] Upgrade Spidermonkey to 31
Right now CouchDB uses outdated Spidermonkey 1.8.5. To upgrade it we need
to fix one small issue: Spidermonkey 31 can not eval anonymous functions in
global scope.
It means we can not take string representation of ddoc function and convert
it into executable code using eval('function(args){/*code*/}') in
Spidermonkey 31.
Way to fix it is quite straightforward and does not require touching of
existing C code of couch_js (as I see).
We just need to convert anonymous functions into named before sending them
to couch_js.
To achieve it we must replace /^function[\s\r\n\t]*\(/ to `function
someName(`, that is enough.
To generate function name and make it unreacheable for other functions of
the ddoc by name, we can take some quick hash from
(ddocName+fnBranchName+PID).
There exist another possible obstacle – old SMs support extended syntax of
try/catch – but it seems very unlikely someone uses it in real, so it can
be forgotten, since it‘s totally out of JS standards.
ermouth
