Re: 100% CPU on only a single node because of couchjs processes

@Victor Thanks Victor, sounds like the right thing to do. Will do. On 26 December 2017 at 09:18, Víctor Torre wrote: > Hi Sinan, > > I recommend you to destroy the server which has been hacked. If someone > wrote a mining program in it, maybe they've persisted some malicious code > as well (root

Re: 100% CPU on only a single node because of couchjs processes

Hi Sinan, I recommend you to destroy the server which has been hacked. If someone wrote a mining program in it, maybe they've persisted some malicious code as well (rootkit, etc) Definetly, If you detect a database node compromised you must destroy it and load a backup. On Thu, Dec 21, 201

Re: 100% CPU on only a single node because of couchjs processes

PS Just want to say that by removing the cron job as described the crypto mining problem is removed and the 100% CPU problem is resolved. (I have had responses from other users experiencing the exact same problem.) $ sudo -i $ su couchdb $ crontab -e Then remove the line in the cron. If you use t

Re: 100% CPU on only a single node because of couchjs processes

I've finally managed to narrow this down to a reproducible set of scripts and have created a GH issue: https://github.com/apache/couchdb/issues/1063 The summary is that I believe there is a resource leak in CouchDB when you abort continuous listening on the _global_changes database. Fortunately, t

Re: 100% CPU on only a single node because of couchjs processes

I just found this, it may solve the problem by removing it. It is a script that starts the crypto mining program as user couchdb. [image: Inline images 1] On 11 December 2017 at 15:04, Sinan Gabel wrote: > > > I have now checked it, and it is as you @Robert perceived, a crypto mining > program.

Re: 100% CPU on only a single node because of couchjs processes

I have now checked it, and it is as you @Robert perceived, a crypto mining program. It is a program "fs-manager" with config.json as given below that is restarted every 3 hours or so (8:41, 11:41 etc.) unless it keeps running. I updated couchdb to latest version and changed all admin passwords bu

Re: 100% CPU on only a single node because of couchjs processes

@Geoffrey Thanks Geoffrey, I will start by checking out the advice of @Robert and see what that brings. I understand now that ./fs-manager has nothing to do with couchdb but it is running as user: couchdb so hacking could be the issue in my case. On 10 December 2017 at 18:14, Geoffrey Cox wrote:

Re: 100% CPU on only a single node because of couchjs processes

@Robert Thanks a lot for the advice, I will check it. On 10 December 2017 at 18:11, Robert Samuel Newson wrote: > fs-manager is not part of couchdb. You should check that you've not been > hacked. See https://justi.cz/security/2017/11/14/couchdb-rce-npm.html < > https://justi.cz/security/2017/11

Re: 100% CPU on only a single node because of couchjs processes

Hi Sinan, I don’t believe we are encountering the same problem as my CouchDB instances run fine with almost no CPU usage until I introduce some activity. And this is fine except the issue is that with a 2-node cluster, the majority of the activity is focused on just one of the nodes. Perhaps in yo

Re: 100% CPU on only a single node because of couchjs processes

fs-manager is not part of couchdb. You should check that you've not been hacked. See https://justi.cz/security/2017/11/14/couchdb-rce-npm.html . I've seen another case where a user's couchdb installation was compromised and a bitcoin mi

Re: 100% CPU on only a single node because of couchjs processes

Same problem with latest couchdb clustered version. What is the process " */.fs-manager*" doing? Googling it returned something about a python package called "fs-manager"!? For now I have killed the *couchdb fs-manager* process, and couchdb still seems to be working fine. On 10 December 2017 at

Re: 100% CPU on only a single node because of couchjs processes

PS I have just now upgraded to latest couchdb clustered version, I will see if that solves the 100% CPU usage problem. On 10 December 2017 at 15:28, Sinan Gabel wrote: > Hi, > > I do not have a solution but also experiencing the 100% CPU usage. Here's > a .png screen shot of the processes runnin

Re: 100% CPU on only a single node because of couchjs processes

Hi, I do not have a solution but also experiencing the 100% CPU usage. Here's a .png screen shot of the processes running (my case): On 10 December 2017 at 02:00, Geoffrey Cox wrote: > Well, I'm back at this and here is the latest info and I think it may be > related to writes and the _global_

Re: 100% CPU on only a single node because of couchjs processes

Well, I'm back at this and here is the latest info and I think it may be related to writes and the _global_changes database: 1. I run my production env and one of my nodes becomes the "workhorse" node with 100% CPU 2. I stop all my production code from generating any more CouchDB reque

Re: 100% CPU on only a single node because of couchjs processes

Interesting, I read somewhere that having a view per ddoc is more efficient. Thanks for clarifying! On Wed, Dec 6, 2017 at 1:31 AM Jan Lehnardt wrote: > > > On 5. Dec 2017, at 21:13, Geoffrey Cox wrote: > > > > Hey Adam, > > > > Attached is my local.ini and the design doc with the view JS. > > >

Re: 100% CPU on only a single node because of couchjs processes

> On 5. Dec 2017, at 21:13, Geoffrey Cox wrote: > > Hey Adam, > > Attached is my local.ini and the design doc with the view JS. > > Please see my responses below: > > Thanks for the help! > > On Tue, Dec 5, 2017 at 8:55 AM Adam Kocoloski wrote: > Hi Geoff, a couple of additional questions:

Re: 100% CPU on only a single node because of couchjs processes

Hi Adam, quick follow-up: is it possible that writes can also be designed to a "primary" node, like the `stale` option for a view? I was originally thinking that the issue is with reading data via a view, but now I'm thinking it may be related to writing data and those writes somehow triggering the

Re: 100% CPU on only a single node because of couchjs processes

Hey Adam, Attached is my local.ini and the design doc with the view JS. Please see my responses below: Thanks for the help! On Tue, Dec 5, 2017 at 8:55 AM Adam Kocoloski wrote: > Hi Geoff, a couple of additional questions: > > 1) Are you making these view requests with stale=ok or stale=updat

Re: 100% CPU on only a single node because of couchjs processes

Hi Geoff, a couple of additional questions: 1) Are you making these view requests with stale=ok or stale=update_after? 2) What are you using for N and Q in the [cluster] configuration settings? 3) Did you take advantage of the (barely-documented) “zones" attribute when defining cluster members? 3

Re: 100% CPU on only a single node because of couchjs processes

Oops, sorry, I got AZs and regions mixed up! Best Jan -- > On 5. Dec 2017, at 13:43, Robert Samuel Newson wrote: > > Sorry to contradict you, but Cloudant deploys clusters across amazon AZ's as > standard. It's fast enough. It's cross-region that you need to avoid. > > B. > >> On 5 Dec 2017,

Re: 100% CPU on only a single node because of couchjs processes

Thanks for the responses, any other thoughts? FYI: I’m trying to work on a very focused test case that I can share with the Dev team, but it is taking a little while to narrow down the exact cause. On Tue, Dec 5, 2017 at 4:43 AM Robert Samuel Newson wrote: > Sorry to contradict you, but Cloudant

Re: 100% CPU on only a single node because of couchjs processes

Sorry to contradict you, but Cloudant deploys clusters across amazon AZ's as standard. It's fast enough. It's cross-region that you need to avoid. B. > On 5 Dec 2017, at 09:11, Jan Lehnardt wrote: > > Heya Geoff, > > a CouchDB cluster is designed to run in the same data center / with local ar

Re: 100% CPU on only a single node because of couchjs processes

> On 4. Dec 2017, at 22:08, Sinan Gabel wrote: > > Hi, > > I am also experiencing 100% CPU usage, not sure why, it happens suddenly > and continues until couchdb is restarted. > CouchDB version being used is also single-node (n:3, q:8) and v. > 2.1.0-6c4def6 on Ubuntu 16.04 2 vCPU's and 4.5 GB

Re: 100% CPU on only a single node because of couchjs processes

Heya Geoff, a CouchDB cluster is designed to run in the same data center / with local are networking latencies. A cluster across AWS Availability Zones won’t work as you see. If you want CouchDB’s in both AZs, use regular replication and keep the clusters local to the AZ. Best Jan -- > On 4.

Re: 100% CPU on only a single node because of couchjs processes

Hi, I am also experiencing 100% CPU usage, not sure why, it happens suddenly and continues until couchdb is restarted. CouchDB version being used is also single-node (n:3, q:8) and v. 2.1.0-6c4def6 on Ubuntu 16.04 2 vCPU's and 4.5 GB memory. On 4 December 2017 at 19:46, Geoffrey Cox wrote: > Hi

100% CPU on only a single node because of couchjs processes

Hi, I've spent days using trial and error to try and figure out why I am getting a very high CPU load on only a single node in my cluster. I'm hoping someone has an idea of what is going on as I'm getting stuck. Here's my configuration: 1. 2 node cluster: 1. Each node is located in a di