Re: db-create with PostgreSQL fails with user "user"
On 5/15/20 9:35 PM, Louis Bertrand wrote: I chose PostgreSQL because a) it's already installed on the eventual target server and b) I'm familiar with it. However, where is the trade-off between SQLite and PostgreSQL? Tens, hundreds, thousands? Number of users, transactions? Etc. In other words, I might have saved myself some trouble by simply accepting the default for a small site. One advantage of supporting multiple databases is that we don't have to choose on behalf of someone else ;-) Also, I guess the load of a "user" might vary with a factor 100 depending on how the system is used. I guess a system with 1000 quite active users easily can have less than 1 write operation per second and a manageable amount of read operations and work fine with a number of worker processes on a single server. In that case I guess sqlite would work fine. I think the biggest difference is in the operations. If a DBA is running the system, he might want a "real" database. There might be more confidence in how PostgreSQL handle scaling and system failures and allow more powerful integrations with other systems. It is very hard to lose critical data in a DVCS hosting system. The important data are in the repos directly in the file system - not in the database. And worst case, all the clients will still have the data they pushed ... and other clients might already have pulled them. We could perhaps put some generic advice in the docs: Start out with a single server with sqlite and reliable local storage (with regular backup of storage and database). If you need failover, make it cold. If users experience delays while the server load is low (especially if the workers are busy serving repo data), add more worker processes. If the server load is high, consider adding more CPU or memory if feasible. If network load is the bottleneck, try solving that. Also consider offloading buildfarm load to a simpler or separate system or make it "smarter" to decrease the load. Then, if necessary or for other reasons desirable, scale up to use something like PostgreSQL (possibly on a separate HA system/cluster ... and potentially higher latency), shared storage (network or SAN), and multiple (physical) worker servers. While different existing setups might have met different limitations, does any existing users have any comments or advice to add to this? /Mads ___ kallithea-general mailing list kallithea-general@sfconservancy.org https://lists.sfconservancy.org/mailman/listinfo/kallithea-general
RE: db-create with PostgreSQL fails with user "user"
-Original Message- From: Mads Kiilerich Sent: May 15, 2020 12:56 PM To: Louis Bertrand ; Thomas De Schampheleire Cc: Kallithea Subject: Re: db-create with PostgreSQL fails with user "user" [EXTERNAL EMAIL] On 5/15/20 10:08 AM, Louis Bertrand wrote: > Hi Thomas, > > Thanks for the prompt reply but it turns out that it's pilot error. > The name 'user' came from my.ini, where the postgresql URL was set to > default. After setting the URL correctly > > sqlalchemy.url = postgresql://kallithea:S3CR3T@localhost/kallithea >(not the real password) > > the database seems to be initialized correctly. Would a more verbose installation overview have helped you? Something like https://kallithea-scm.org/repos/kallithea-incoming/changeset/512aef4199733151ec76490915bca1e6ce7d63e2 ? /Mads Hi Mads, Yes, greatly (except for slight typo s/scheme/schema). I chose PostgreSQL because a) it's already installed on the eventual target server and b) I'm familiar with it. However, where is the trade-off between SQLite and PostgreSQL? Tens, hundreds, thousands? Number of users, transactions? Etc. In other words, I might have saved myself some trouble by simply accepting the default for a small site. Thanks --Louis This message is intended only for the named recipients. This message may contain information that is confidential or exempt from disclosure under applicable law. Any dissemination or copying of this message by anyone other than a named recipient is strictly prohibited. If you are not a named recipient or an employee or agent responsible for delivering this message to a named recipient, please notify us immediately, and permanently destroy this message and any copies you may have. Warning: Email may not be secure unless properly encrypted. ___ kallithea-general mailing list kallithea-general@sfconservancy.org https://lists.sfconservancy.org/mailman/listinfo/kallithea-general
Re: db-create with PostgreSQL fails with user "user"
On 5/15/20 10:08 AM, Louis Bertrand wrote: Hi Thomas, Thanks for the prompt reply but it turns out that it's pilot error. The name 'user' came from my.ini, where the postgresql URL was set to default. After setting the URL correctly sqlalchemy.url = postgresql://kallithea:S3CR3T@localhost/kallithea (not the real password) the database seems to be initialized correctly. Would a more verbose installation overview have helped you? Something like https://kallithea-scm.org/repos/kallithea-incoming/changeset/512aef4199733151ec76490915bca1e6ce7d63e2 ? /Mads ___ kallithea-general mailing list kallithea-general@sfconservancy.org https://lists.sfconservancy.org/mailman/listinfo/kallithea-general
Re: db-create with PostgreSQL fails with user "user"
Hi Thomas, I'm not familiar with how Raspberry Pi or Debian package things, but at first glance the two packages do indeed seem to be coming from different distributions. Thanks anyway --Louis From: Thomas De Schampheleire Sent: Friday, May 15, 2020 8:45 AM To: Louis Bertrand Cc: Kallithea Subject: Re: db-create with PostgreSQL fails with user "user" [EXTERNAL EMAIL] Hi Louis, Ok, great you found the postgreql error. Regarding npm: it seems indeed your nodejs is too new for the npm version. Based on the versions: nodejs (10.19.0~dfsg1-1+rpi1). npm (5.8.0+ds6-4+deb10u1). It seems that the nodejs installation is rpi specific. Is it coming from the same source as the npm package? Normally, a distribution like debian will provide a consistent set of packages that work together, but if packages from several sources are mixed, then such problem could occur. Or of course it could be a problem in Raspbian itself. In any case, this is not something that Kallthea decides or interferes with, so I'm afraid there is not much we can do. Hope this helps, Thomas El vie., 15 may. 2020 a las 10:08, Louis Bertrand () escribió: > > Hi Thomas, > > Thanks for the prompt reply but it turns out that it's pilot error. The name > 'user' came from my.ini, where the postgresql URL was set to default. After > setting the URL correctly > >sqlalchemy.url = postgresql://kallithea:S3CR3T@localhost/kallithea > (not the real password) > > the database seems to be initialized correctly. > > > However the next step crashes on 'npm install' (see text below). > > With apt-get I can confirm that npm and nodejs are at the latest available > versions but that seems to be the problem: > > nodejs is already the newest version (10.19.0~dfsg1-1+rpi1). > > npm is already the newest version (5.8.0+ds6-4+deb10u1). > npm does not recognize version NodeJS 10.x as valid. > > > I should mention here that I am using a Raspberry Pi mostly because it's what > I have on hand. I'm trying to prototype the system so that I can get our > college IT dept to install it on a public facing server for student work. The > college server is running Ubuntu 18.04.4 LTS. The Pi is running Raspbian 10 > (buster). If the Pi is not suitable or supported, I can switch to a VM > running internally (access via VPN). > > > Thanks in advance > > --Louis > > > > Full text of output: > > (kallithea-venv) kallithea@raspberrypi:~/kallithea$ kallithea-cli > front-end-build > Running 'npm install' to install front-end dependencies from package.json > npm WARN npm npm does not support Node.js v10.19.0 > npm WARN npm You should probably upgrade to a newer version of node as we > npm WARN npm can't make any promises that npm will work with this version. > npm WARN npm Supported releases of Node.js are the latest release of 4, 6, 7, > 8, 9. > npm WARN npm You can find the latest version at https://nodejs.org/ > npm WARN notice [SECURITY] bootstrap has the following vulnerability: 1 > moderate. Go here for more details: > https://www.npmjs.com/advisories?search=bootstrap=3.3.7 - Run `npm i > npm@latest -g` to upgrade your npm version, and then `npm audit` to get more > info. > npm WARN tar write after end > npm WARN notice [SECURITY] clean-css has the following vulnerability: 1 low. > Go here for more details: > https://www.npmjs.com/advisories?search=clean-css=3.4.28 - Run `npm i > npm@latest -g` to upgrade your npm version, and then `npm audit` to get more > info. > npm WARN tar write after end > npm WARN tar write after end > npm ERR! cb() never called! > > npm ERR! This is an error with npm itself. Please report this error at: > npm ERR! <https://github.com/npm/npm/issues> > > npm ERR! A complete log of this run can be found in: > npm ERR! /var/lib/kallithea/.npm/_logs/2020-05-15T07_32_43_524Z-debug.log > Traceback (most recent call last): > File "/var/lib/kallithea/kallithea-venv/bin/kallithea-cli", line 11, in > > load_entry_point('Kallithea', 'console_scripts', 'kallithea-cli')() > File > "/var/lib/kallithea/kallithea-venv/lib/python3.7/site-packages/click/core.py", > line 829, in __call__ > return self.main(*args, **kwargs) > File > "/var/lib/kallithea/kallithea-venv/lib/python3.7/site-packages/click/core.py", > line 782, in main > rv = self.invoke(ctx) > File > "/var/lib/kallithea/kallithea-venv/lib/python3.7/site-packages/click/core.py", > line 1259, in invoke > return _process_result(sub_ctx.command.invoke(sub_ctx)) > File > "/var/lib/kallithea/kallithea-venv/lib/python3.7/site-packages/click/core.py", > line 1066, i
Re: db-create with PostgreSQL fails with user "user"
r, > shell=kallithea.is_windows) > File "/usr/lib/python3.7/subprocess.py", line 347, in check_call > raise CalledProcessError(retcode, cmd) > subprocess.CalledProcessError: Command '['npm', 'install']' returned non-zero > exit status 1 > > > > > From: Thomas De Schampheleire > Sent: Friday, May 15, 2020 2:03 AM > To: Louis Bertrand > Cc: Kallithea > Subject: Re: db-create with PostgreSQL fails with user "user" > > [EXTERNAL EMAIL] > Hello, > > Could you provide the full stacktrace please? > > Thanks, > Thomas > > On Fri, May 15, 2020, 04:14 Louis Bertrand > mailto:louis.bertr...@durhamcollege.ca>> > wrote: > Hello, > I get a strange exception when trying to create the database for a fresh > installation of Kallithea. The exception complains about unknown user "user" > even though I specified --user=Kallithea on the command line. > > The full command line, run as user Kallithea: > $ kallithea-cli db-create -c my.ini --user=kallithea --password=blahblah > --email=kallit...@domain.tld --repos=/var/lib/kallithea/repos > > The exception seems to be raised by psycopg2 when connecting to the database, > and called from sqlalchemy: > conn = _connect(dsn, connection_factory=connection_factory, **kwasync) > psycopg2.OperationalError: FATAL: password authentication failed for user > "user" > > The user kallithea exists as a PostgreSQL role and has createdb permission, > but it seems that the user name doesn't get through to sqlalchemy because > it's using "user" instead of the name supplied on the command line. > > Any ideas? Full rollback available if you wish. > > Thanks > --Louis > > > Louis Bertrand, P.Eng. > Professor, School of Science and Engineering Technology > Durham College, Oshawa ON Canada > > > > > This message is intended only for the named recipients. This message may > contain information that is confidential or exempt from disclosure under > applicable law. Any dissemination or copying of this message by anyone other > than a named recipient is strictly prohibited. If you are not a named > recipient or an employee or agent responsible for delivering this message to > a named recipient, please notify us immediately, and permanently destroy this > message and any copies you may have. Warning: Email may not be secure unless > properly encrypted. > ___ > kallithea-general mailing list > kallithea-general@sfconservancy.org<mailto:kallithea-general@sfconservancy.org> > https://lists.sfconservancy.org/mailman/listinfo/kallithea-general > u ___ kallithea-general mailing list kallithea-general@sfconservancy.org https://lists.sfconservancy.org/mailman/listinfo/kallithea-general
Re: db-create with PostgreSQL fails with user "user"
Hi Thomas, Thanks for the prompt reply but it turns out that it's pilot error. The name 'user' came from my.ini, where the postgresql URL was set to default. After setting the URL correctly sqlalchemy.url = postgresql://kallithea:S3CR3T@localhost/kallithea (not the real password) the database seems to be initialized correctly. However the next step crashes on 'npm install' (see text below). With apt-get I can confirm that npm and nodejs are at the latest available versions but that seems to be the problem: nodejs is already the newest version (10.19.0~dfsg1-1+rpi1). npm is already the newest version (5.8.0+ds6-4+deb10u1). npm does not recognize version NodeJS 10.x as valid. I should mention here that I am using a Raspberry Pi mostly because it's what I have on hand. I'm trying to prototype the system so that I can get our college IT dept to install it on a public facing server for student work. The college server is running Ubuntu 18.04.4 LTS. The Pi is running Raspbian 10 (buster). If the Pi is not suitable or supported, I can switch to a VM running internally (access via VPN). Thanks in advance --Louis Full text of output: (kallithea-venv) kallithea@raspberrypi:~/kallithea$ kallithea-cli front-end-build Running 'npm install' to install front-end dependencies from package.json npm WARN npm npm does not support Node.js v10.19.0 npm WARN npm You should probably upgrade to a newer version of node as we npm WARN npm can't make any promises that npm will work with this version. npm WARN npm Supported releases of Node.js are the latest release of 4, 6, 7, 8, 9. npm WARN npm You can find the latest version at https://nodejs.org/ npm WARN notice [SECURITY] bootstrap has the following vulnerability: 1 moderate. Go here for more details: https://www.npmjs.com/advisories?search=bootstrap=3.3.7 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info. npm WARN tar write after end npm WARN notice [SECURITY] clean-css has the following vulnerability: 1 low. Go here for more details: https://www.npmjs.com/advisories?search=clean-css=3.4.28 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info. npm WARN tar write after end npm WARN tar write after end npm ERR! cb() never called! npm ERR! This is an error with npm itself. Please report this error at: npm ERR! <https://github.com/npm/npm/issues> npm ERR! A complete log of this run can be found in: npm ERR! /var/lib/kallithea/.npm/_logs/2020-05-15T07_32_43_524Z-debug.log Traceback (most recent call last): File "/var/lib/kallithea/kallithea-venv/bin/kallithea-cli", line 11, in load_entry_point('Kallithea', 'console_scripts', 'kallithea-cli')() File "/var/lib/kallithea/kallithea-venv/lib/python3.7/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/var/lib/kallithea/kallithea-venv/lib/python3.7/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/var/lib/kallithea/kallithea-venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/var/lib/kallithea/kallithea-venv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/var/lib/kallithea/kallithea-venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/var/lib/kallithea/kallithea/kallithea/bin/kallithea_cli_front_end.py", line 46, in front_end_build subprocess.check_call(['npm', 'install'], cwd=front_end_dir, shell=kallithea.is_windows) File "/usr/lib/python3.7/subprocess.py", line 347, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['npm', 'install']' returned non-zero exit status 1 From: Thomas De Schampheleire Sent: Friday, May 15, 2020 2:03 AM To: Louis Bertrand Cc: Kallithea Subject: Re: db-create with PostgreSQL fails with user "user" [EXTERNAL EMAIL] Hello, Could you provide the full stacktrace please? Thanks, Thomas On Fri, May 15, 2020, 04:14 Louis Bertrand mailto:louis.bertr...@durhamcollege.ca>> wrote: Hello, I get a strange exception when trying to create the database for a fresh installation of Kallithea. The exception complains about unknown user "user" even though I specified --user=Kallithea on the command line. The full command line, run as user Kallithea: $ kallithea-cli db-create -c my.ini --user=kallithea --password=blahblah --email=kallit...@domain.tld --repos=/var/lib/kallithea/repos The exception seems to be raised by psycopg2 when connecting to the database, and called from sqlalchemy: conn = _connect(dsn, connection_factory=connection_factory, **kwasyn
Re: db-create with PostgreSQL fails with user "user"
Hello, Could you provide the full stacktrace please? Thanks, Thomas On Fri, May 15, 2020, 04:14 Louis Bertrand wrote: > Hello, > I get a strange exception when trying to create the database for a fresh > installation of Kallithea. The exception complains about unknown user > "user" even though I specified --user=Kallithea on the command line. > > The full command line, run as user Kallithea: > $ kallithea-cli db-create -c my.ini --user=kallithea --password=blahblah > --email=kallit...@domain.tld --repos=/var/lib/kallithea/repos > > The exception seems to be raised by psycopg2 when connecting to the > database, and called from sqlalchemy: > conn = _connect(dsn, connection_factory=connection_factory, **kwasync) > psycopg2.OperationalError: FATAL: password authentication failed for user > "user" > > The user kallithea exists as a PostgreSQL role and has createdb > permission, but it seems that the user name doesn't get through to > sqlalchemy because it's using "user" instead of the name supplied on the > command line. > > Any ideas? Full rollback available if you wish. > > Thanks > --Louis > > > Louis Bertrand, P.Eng. > Professor, School of Science and Engineering Technology > Durham College, Oshawa ON Canada > > > > > This message is intended only for the named recipients. This message may > contain information that is confidential or exempt from disclosure under > applicable law. Any dissemination or copying of this message by anyone > other than a named recipient is strictly prohibited. If you are not a named > recipient or an employee or agent responsible for delivering this message > to a named recipient, please notify us immediately, and permanently destroy > this message and any copies you may have. Warning: Email may not be secure > unless properly encrypted. > ___ > kallithea-general mailing list > kallithea-general@sfconservancy.org > https://lists.sfconservancy.org/mailman/listinfo/kallithea-general > ___ kallithea-general mailing list kallithea-general@sfconservancy.org https://lists.sfconservancy.org/mailman/listinfo/kallithea-general