Re: Fauxton does not URLencode its links

[Garren Smith]

Not at this stage. It is on the list of things to do if that is any
consolation


On Wed, 25 Jan 2017 at 8:54 PM Garth Gutenberg 
wrote:

> Is there an ETA on when this will be complete?
>
> On Wed, Jan 25, 2017 at 12:07 PM, Garren Smith  wrote:
>
> > Hi Garth,
> >
> > You are right. We have done the encoding for the _all_dbs section but
> > haven't finished the work for the database page.
> >
> > Cheers
> > Garren
> >
> > On Wed, Jan 25, 2017 at 4:45 PM, Garth Gutenberg <
> > garth.gutenb...@gmail.com>
> > wrote:
> >
> > > Hey guys.  Sorry to revive this, but I don't think this fix is fully
> > > working.  In general it seems ok, but it's not applying to Views.
> Steps
> > to
> > > reproduce:
> > >
> > > Create a DB called "a/b".
> > > Create a new View (call it _design/test and "new-view" is fine)
> > > Under Design Documents in the left nav you'll see "test".  Expand it
> and
> > > click on "new-view".  Note that the DB is *not* urlencoded, and
> > navigation
> > > breaks.
> > >
> > > On Thu, Dec 1, 2016 at 1:27 PM, Garth Gutenberg <
> > garth.gutenb...@gmail.com
> > > >
> > > wrote:
> > >
> > > > Thanks Robert.  I'll test that out next week.
> > > >
> > > > On Wed, Nov 30, 2016 at 5:04 PM, Robert Kowalski 
> > > wrote:
> > > >
> > > >> Hi Garth,
> > > >>
> > > >> I think it is already fixed on master:
> > > >> https://github.com/apache/couchdb-fauxton/commit/1aa4ca6f34a
> > > >> 718c294a06a1301f39fe05f157a1c
> > > >>
> > > >>
> > > >> On Wed, Nov 30, 2016 at 10:25 PM, Garth Gutenberg
> > > >>  wrote:
> > > >> > Hey folks.  Fauxton is not URL encoding any of its navigation
> links
> > in
> > > >> > Couch 2.0.  It's a pretty major issue for us as all of our
> databases
> > > use
> > > >> > slashes in their names, which makes navigating around Fauxton
> > > >> unbearable.
> > > >> >
> > > >> > I created a ticket for it here https://issues.apache.
> > > >> > org/jira/browse/COUCHDB-3229 .
> > > >> >
> > > >> > Can someone please take a look at this issue?
> > > >>
> > > >
> > > >
> > >
> >
>

Re: How do you know when a new node in a cluster is "synced"?

[Garth Gutenberg]

Great info.  Thanks!

On Wed, Jan 25, 2017 at 12:11 PM, Paul Davis 
wrote:

> Garth,
>
> The way to tell when a cluster is sync'ed is by looking at the
> `internal_replication_jobs` key in the JSON blob returned from the
> _system endpoint on the 5984 port from each node in the cluster. Once
> its zero (or close to) on each node you're done getting data to the
> new node. Though it can lie some times if there's a busy cluster and
> things get wonky. To be extra sure you can run this on each node in
> the cluster:
>
> `mem3_sync:initial_sync(nodes()).`
>
> Basically all that does is queue up every database for internal
> replication. If that doesn't change the count from zero then you
> should be good to go.
>
>
> To your second question, it would be impossible to automatically sync
> a view when rebuilding it via internal replication. Each view depends
> on the order of updates to the shard. With internal replication that
> order is undefined and would change each time during a rebuild as the
> order of updates via internal replication is approximately random.
> Also, it most definitely would not match the order of the source shard
> when documents have been updated.
>
> However, the answer to your specific problem is to use
> maintenance_mode on the node being rebuilt. Before you first boot the
> node you're wanting to rebuild (or before you connect it to the
> cluster) you just need to set the `[couchdb] maintenance_mode = true`
> parameter. This prevents the node from participating in any
> interactive requests which prevents it from responding to view queries
> before its built its views. Then you just need to watch _active_tasks
> for your view to build before setting maintenance_mode back to false
> or deleting it.
>
> You may also want to make a view query to the individual shards over
> the 5986 node local port as well to make sure that there was a build
> triggered for each shard.
>
> Paul
>
> On Wed, Jan 25, 2017 at 11:27 AM, Garth Gutenberg
>  wrote:
> > Kind of a follow-up question to this.  I've found in my testing that
> when a
> > new node comes online in a cluster, it only syncs the raw data, but not
> the
> > views.  Is there a way to enable syncing of views across cluster nodes as
> > well?  Basically I want all the nodes in my cluster to be exact replicas
> of
> > each other.  We have some relatively large DBs (~4GB) whose views take
> > awhile to generate.
> >
> > To expand on the previous scenario, if the downed node comes up without
> any
> > views, and a client hits hit, that client needs to wait for the view to
> be
> > generated - even though it exists on the other nodes in the cluster.  And
> > that wait time can be 15-30 mins in some cases, which really isn't
> > acceptable when the view is already generated, just not on this
> particular
> > node.
> >
> > On Wed, Jan 25, 2017 at 8:59 AM, Garth Gutenberg <
> garth.gutenb...@gmail.com>
> > wrote:
> >
> >> Scenario:
> >>
> >> I have a three node cluster.  One of the nodes goes offline (server
> dies,
> >> whatever).  I bring up a new node with no data and it starts sync'ing
> with
> >> the other nodes in the cluster.
> >>
> >> How do I know when this sync is complete and the new node has all the
> >> data?  I'm dealing with thousands of DBs, so doing a doc count in each
> one
> >> isn't really feasible - at least not in a timely manner.  Is there a
> log or
> >> API somewhere that indicates completion of data synchronization from a
> >> server perspective, not just individual DBs?
> >>
>

Re: Fauxton does not URLencode its links

[Garth Gutenberg]

Is there an ETA on when this will be complete?

On Wed, Jan 25, 2017 at 12:07 PM, Garren Smith  wrote:

> Hi Garth,
>
> You are right. We have done the encoding for the _all_dbs section but
> haven't finished the work for the database page.
>
> Cheers
> Garren
>
> On Wed, Jan 25, 2017 at 4:45 PM, Garth Gutenberg <
> garth.gutenb...@gmail.com>
> wrote:
>
> > Hey guys.  Sorry to revive this, but I don't think this fix is fully
> > working.  In general it seems ok, but it's not applying to Views.  Steps
> to
> > reproduce:
> >
> > Create a DB called "a/b".
> > Create a new View (call it _design/test and "new-view" is fine)
> > Under Design Documents in the left nav you'll see "test".  Expand it and
> > click on "new-view".  Note that the DB is *not* urlencoded, and
> navigation
> > breaks.
> >
> > On Thu, Dec 1, 2016 at 1:27 PM, Garth Gutenberg <
> garth.gutenb...@gmail.com
> > >
> > wrote:
> >
> > > Thanks Robert.  I'll test that out next week.
> > >
> > > On Wed, Nov 30, 2016 at 5:04 PM, Robert Kowalski 
> > wrote:
> > >
> > >> Hi Garth,
> > >>
> > >> I think it is already fixed on master:
> > >> https://github.com/apache/couchdb-fauxton/commit/1aa4ca6f34a
> > >> 718c294a06a1301f39fe05f157a1c
> > >>
> > >>
> > >> On Wed, Nov 30, 2016 at 10:25 PM, Garth Gutenberg
> > >>  wrote:
> > >> > Hey folks.  Fauxton is not URL encoding any of its navigation links
> in
> > >> > Couch 2.0.  It's a pretty major issue for us as all of our databases
> > use
> > >> > slashes in their names, which makes navigating around Fauxton
> > >> unbearable.
> > >> >
> > >> > I created a ticket for it here https://issues.apache.
> > >> > org/jira/browse/COUCHDB-3229 .
> > >> >
> > >> > Can someone please take a look at this issue?
> > >>
> > >
> > >
> >
>

Re: How do you know when a new node in a cluster is "synced"?

[Paul Davis]

Garth,

The way to tell when a cluster is sync'ed is by looking at the
`internal_replication_jobs` key in the JSON blob returned from the
_system endpoint on the 5984 port from each node in the cluster. Once
its zero (or close to) on each node you're done getting data to the
new node. Though it can lie some times if there's a busy cluster and
things get wonky. To be extra sure you can run this on each node in
the cluster:

`mem3_sync:initial_sync(nodes()).`

Basically all that does is queue up every database for internal
replication. If that doesn't change the count from zero then you
should be good to go.


To your second question, it would be impossible to automatically sync
a view when rebuilding it via internal replication. Each view depends
on the order of updates to the shard. With internal replication that
order is undefined and would change each time during a rebuild as the
order of updates via internal replication is approximately random.
Also, it most definitely would not match the order of the source shard
when documents have been updated.

However, the answer to your specific problem is to use
maintenance_mode on the node being rebuilt. Before you first boot the
node you're wanting to rebuild (or before you connect it to the
cluster) you just need to set the `[couchdb] maintenance_mode = true`
parameter. This prevents the node from participating in any
interactive requests which prevents it from responding to view queries
before its built its views. Then you just need to watch _active_tasks
for your view to build before setting maintenance_mode back to false
or deleting it.

You may also want to make a view query to the individual shards over
the 5986 node local port as well to make sure that there was a build
triggered for each shard.

Paul

On Wed, Jan 25, 2017 at 11:27 AM, Garth Gutenberg
 wrote:
> Kind of a follow-up question to this.  I've found in my testing that when a
> new node comes online in a cluster, it only syncs the raw data, but not the
> views.  Is there a way to enable syncing of views across cluster nodes as
> well?  Basically I want all the nodes in my cluster to be exact replicas of
> each other.  We have some relatively large DBs (~4GB) whose views take
> awhile to generate.
>
> To expand on the previous scenario, if the downed node comes up without any
> views, and a client hits hit, that client needs to wait for the view to be
> generated - even though it exists on the other nodes in the cluster.  And
> that wait time can be 15-30 mins in some cases, which really isn't
> acceptable when the view is already generated, just not on this particular
> node.
>
> On Wed, Jan 25, 2017 at 8:59 AM, Garth Gutenberg 
> wrote:
>
>> Scenario:
>>
>> I have a three node cluster.  One of the nodes goes offline (server dies,
>> whatever).  I bring up a new node with no data and it starts sync'ing with
>> the other nodes in the cluster.
>>
>> How do I know when this sync is complete and the new node has all the
>> data?  I'm dealing with thousands of DBs, so doing a doc count in each one
>> isn't really feasible - at least not in a timely manner.  Is there a log or
>> API somewhere that indicates completion of data synchronization from a
>> server perspective, not just individual DBs?
>>

Re: Fauxton does not URLencode its links

[Garren Smith]

Hi Garth,

You are right. We have done the encoding for the _all_dbs section but
haven't finished the work for the database page.

Cheers
Garren

On Wed, Jan 25, 2017 at 4:45 PM, Garth Gutenberg 
wrote:

> Hey guys.  Sorry to revive this, but I don't think this fix is fully
> working.  In general it seems ok, but it's not applying to Views.  Steps to
> reproduce:
>
> Create a DB called "a/b".
> Create a new View (call it _design/test and "new-view" is fine)
> Under Design Documents in the left nav you'll see "test".  Expand it and
> click on "new-view".  Note that the DB is *not* urlencoded, and navigation
> breaks.
>
> On Thu, Dec 1, 2016 at 1:27 PM, Garth Gutenberg  >
> wrote:
>
> > Thanks Robert.  I'll test that out next week.
> >
> > On Wed, Nov 30, 2016 at 5:04 PM, Robert Kowalski 
> wrote:
> >
> >> Hi Garth,
> >>
> >> I think it is already fixed on master:
> >> https://github.com/apache/couchdb-fauxton/commit/1aa4ca6f34a
> >> 718c294a06a1301f39fe05f157a1c
> >>
> >>
> >> On Wed, Nov 30, 2016 at 10:25 PM, Garth Gutenberg
> >>  wrote:
> >> > Hey folks.  Fauxton is not URL encoding any of its navigation links in
> >> > Couch 2.0.  It's a pretty major issue for us as all of our databases
> use
> >> > slashes in their names, which makes navigating around Fauxton
> >> unbearable.
> >> >
> >> > I created a ticket for it here https://issues.apache.
> >> > org/jira/browse/COUCHDB-3229 .
> >> >
> >> > Can someone please take a look at this issue?
> >>
> >
> >
>

Re: How do you know when a new node in a cluster is "synced"?

[Garth Gutenberg]

Kind of a follow-up question to this.  I've found in my testing that when a
new node comes online in a cluster, it only syncs the raw data, but not the
views.  Is there a way to enable syncing of views across cluster nodes as
well?  Basically I want all the nodes in my cluster to be exact replicas of
each other.  We have some relatively large DBs (~4GB) whose views take
awhile to generate.

To expand on the previous scenario, if the downed node comes up without any
views, and a client hits hit, that client needs to wait for the view to be
generated - even though it exists on the other nodes in the cluster.  And
that wait time can be 15-30 mins in some cases, which really isn't
acceptable when the view is already generated, just not on this particular
node.

On Wed, Jan 25, 2017 at 8:59 AM, Garth Gutenberg 
wrote:

> Scenario:
>
> I have a three node cluster.  One of the nodes goes offline (server dies,
> whatever).  I bring up a new node with no data and it starts sync'ing with
> the other nodes in the cluster.
>
> How do I know when this sync is complete and the new node has all the
> data?  I'm dealing with thousands of DBs, so doing a doc count in each one
> isn't really feasible - at least not in a timely manner.  Is there a log or
> API somewhere that indicates completion of data synchronization from a
> server perspective, not just individual DBs?
>

Re: [ANNOUNCEMENT] couch-chakra, a CouchDB Query Server Runtime build with ChakraCore

[Dave Cottlehuber]

> > While it is absolutely possible to link everything into one process,
> > that’s usually not done.
> 
> Actually that's what I tried in the very beginning, writing a NIF to
> wrap the ChackraCore API to Erlang functions. While in theory this
> would be possible it's however heavily discouraged by the Erlang gods
> to write NIFs with non-deterministic timing. So I quickly stepped back
> from the NIF idea and instead implemented couch-chakra.

You might not have read about dirty schedulers[1] which seem to be a
stable
API now, & here's a few links from Steve Vinoski related to NIF handling
[2][3]
and some sample code [4] from Steve and from JLouis [5] using them.

We already have some couchdb NIF code, and if there's a performance gain
it
would be great to ship some more.

A+
Dave

[1]:
https://medium.com/@jlouis666/erlang-dirty-scheduler-overhead-6e1219dcc7#.jgdyewal6
[2]: https://www.youtube.com/watch?v=nw2eIB6bTxY
[3]:
https://github.com/vinoski/bitwise/blob/master/vinoski-schedulers.pdf
[4]: https://github.com/vinoski/bitwise
[5]: https://github.com/jlouis/enacl

How do you know when a new node in a cluster is "synced"?

[Garth Gutenberg]

Scenario:

I have a three node cluster.  One of the nodes goes offline (server dies,
whatever).  I bring up a new node with no data and it starts sync'ing with
the other nodes in the cluster.

How do I know when this sync is complete and the new node has all the
data?  I'm dealing with thousands of DBs, so doing a doc count in each one
isn't really feasible - at least not in a timely manner.  Is there a log or
API somewhere that indicates completion of data synchronization from a
server perspective, not just individual DBs?

Re: Fauxton does not URLencode its links

[Garth Gutenberg]

Hey guys.  Sorry to revive this, but I don't think this fix is fully
working.  In general it seems ok, but it's not applying to Views.  Steps to
reproduce:

Create a DB called "a/b".
Create a new View (call it _design/test and "new-view" is fine)
Under Design Documents in the left nav you'll see "test".  Expand it and
click on "new-view".  Note that the DB is *not* urlencoded, and navigation
breaks.

On Thu, Dec 1, 2016 at 1:27 PM, Garth Gutenberg 
wrote:

> Thanks Robert.  I'll test that out next week.
>
> On Wed, Nov 30, 2016 at 5:04 PM, Robert Kowalski  wrote:
>
>> Hi Garth,
>>
>> I think it is already fixed on master:
>> https://github.com/apache/couchdb-fauxton/commit/1aa4ca6f34a
>> 718c294a06a1301f39fe05f157a1c
>>
>>
>> On Wed, Nov 30, 2016 at 10:25 PM, Garth Gutenberg
>>  wrote:
>> > Hey folks.  Fauxton is not URL encoding any of its navigation links in
>> > Couch 2.0.  It's a pretty major issue for us as all of our databases use
>> > slashes in their names, which makes navigating around Fauxton
>> unbearable.
>> >
>> > I created a ticket for it here https://issues.apache.
>> > org/jira/browse/COUCHDB-3229 .
>> >
>> > Can someone please take a look at this issue?
>>
>
>

Re: [ANNOUNCEMENT] couch-chakra, a CouchDB Query Server Runtime build with ChakraCore

[Dave Cottlehuber]

On Tue, 24 Jan 2017, at 18:24, Daniel Munch wrote:
> Thanks Jan for clearing things up, I couldn't have answered better
> myself! And thanks everybody else for the feedback so far.
> 
> >> That clears it up, One more question if I may. In use would this
> >> QueryServer replacement module be an adjacent process to the CouchDB
> >> process, or is there some linking fu to make CouchDB and CouchChakra one
> >> process?
> >
> > I haven’t looked to closely, but how I understand it, this is a separate
> > process. Just like it with CouchDB today (you have a beam[.smp] process
> > and zero or more couchjs processes).
> 
> It currently takes exactly the same approach as couchjs - one beam
> process, zero or more couchjs processes, that's why I called it a
> drop-in replacement. In theory you could switch out couchjs by
> couch-chakra and everything should work like before.
> 
> > While it is absolutely possible to link everything into one process,
> > that’s usually not done.
> 
> Actually that's what I tried in the very beginning, writing a NIF to
> wrap the ChackraCore API to Erlang functions. While in theory this
> would be possible it's however heavily discouraged by the Erlang gods
> to write NIFs with non-deterministic timing. So I quickly stepped back
> from the NIF idea and instead implemented couch-chakra.
> 
> Also, like Garren said in a mail before, there has been a couple of
> attempts to redesign the Query Server Protocol and the process model
> for the javascript query server. It looks as if there were different
> opinions on this, and it also looks like it could become a lot of
> work. Personally I'd love to see a binary communication protocol
> between couchdb and the query server and thought that BERT and
> BERT-RPC [1] might be a viable option. I'd also love to exploit the
> rental threading model of ChakraCore like it is explained in the
> article on how Chakra is used in DocumentDB [2]: "In other words, a
> runtime only operates on one thread at a time, but its thread affinity
> is free to change from time to time."
> 
> Add libuv to the sauce and we might win the next buzzword-bingo
> contest with distinction, but that's what this project currently
> represents for me: A playground to explore weird ideas and to have
> some fun hacking on in my free-time.
> 
> Best,
> Daniel

I love this!

In particular Daniel's offered some solutions for finally handling
anonymous functions, which is super awesome and long overdue.
 
Being able to run our test suite with different JS engines would be
really cool, as well as shipping some alternatives.

Serialisation uses a lot of CPU in CouchDB and any experiments or
adventures into finding ways to improve this would be welcome. Currently
we do all of the following:

browser/API (javascript native <->json)
couchdb (json <-> erlang terms)
couchjs ( json <-> javascript native)

I'd love to see some incremental/experimental changes or tests in this
space, as previous attempts got side tracked with discussion rather than
shipped contributions. Providing alternate couchjs builds is relatively
low-impact compared to alternative communication protocols, maybe thats
a good place to start.

A+ 
Dave