Re: [GENERAL] Can't restart Postgres

[Joshua D. Drake]

On 02/15/2017 03:09 PM, Shawn Thomas wrote:

Just wanted to follow up that re-installing Postgres worked (well almost—I did 
have to reset the permissions and ownership on the key and pem file).

Thanks so much for all the help.



That's what we are here for :D

Sincerely,

JD



-Shawn



--
Command Prompt, Inc.  http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Shawn Thomas]

Just wanted to follow up that re-installing Postgres worked (well almost—I did 
have to reset the permissions and ownership on the key and pem file).  

Thanks so much for all the help.

-Shawn

> On Feb 15, 2017, at 9:49 AM, Adrian Klaver  wrote:
> 
> On 02/15/2017 09:45 AM, Shawn Thomas wrote:
>> Which would you recommend?  Leave the data directory in place and
>> re-install PG or copy it to somewhere else, delete it and then
>> re-install PG?
> 
> I would copy the data directory somewhere else for safe keeping leaving the 
> original in place. Then reinstall Postgres, the install should leave the 
> original directory alone and you will be ready to go. Should there be an oops 
> you will have the copy as backup.
> 
>> 
>> -Shawn
>> 
>>> On Feb 15, 2017, at 9:36 AM, Magnus Hagander >> > wrote:
>>> 
>>> On Wed, Feb 15, 2017 at 6:28 PM, Shawn Thomas
>>> > wrote:
>>> 
>>>Well that would make more sense of things.  I had removed and
>>>re-installed the postresql-common package:
>>> 
>>>https://packages.debian.org/jessie/postgresql-common
>>>
>>> 
>>>and thought that it would leave the main PG package in place.  But
>>>perhaps I was wrong.  I’ll follow Tom’s advice and just re-install
>>>everything (saving the old data directory) and hope the new
>>>installation can use the old data data directory.
>>> 
>>> 
>>> If you removed it and then installed it, then the removal would remove
>>> all dependent packages and if you then only intalled that one and not
>>> the dependencies that would explain it.
>>> 
>>> If you had run a reinstall on it, then it would've kept them around.
>>> 
>>> 
>>> 
>>>One question about this approach though:  the Debian package
>>>installation automatically initializes the new data directory and
>>>starts PG.  If I shut it down and copy the old data directory into
>>>the newly installed one, will there be an xlog issue?
>>> 
>>> 
>>> You have to copy the xlog along with the database.
>>> 
>>> Or if you leave it in place where it is, the packages won't initialize
>>> a new data directory.
>>> 
>>> --
>>> Magnus Hagander
>>> Me: http://www.hagander.net/
>>> Work: http://www.redpill-linpro.com/
>> 
> 
> 
> -- 
> Adrian Klaver
> adrian.kla...@aklaver.com



-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Shawn Thomas]

Yes, definitely. 

> On Feb 15, 2017, at 9:49 AM, Adrian Klaver  wrote:
> 
> On 02/15/2017 09:45 AM, Shawn Thomas wrote:
>> Which would you recommend?  Leave the data directory in place and
>> re-install PG or copy it to somewhere else, delete it and then
>> re-install PG?
> 
> I would copy the data directory somewhere else for safe keeping leaving the 
> original in place. Then reinstall Postgres, the install should leave the 
> original directory alone and you will be ready to go. Should there be an oops 
> you will have the copy as backup.
> 
>> 
>> -Shawn
>> 
>>> On Feb 15, 2017, at 9:36 AM, Magnus Hagander >> > wrote:
>>> 
>>> On Wed, Feb 15, 2017 at 6:28 PM, Shawn Thomas
>>> > wrote:
>>> 
>>>Well that would make more sense of things.  I had removed and
>>>re-installed the postresql-common package:
>>> 
>>>https://packages.debian.org/jessie/postgresql-common
>>>
>>> 
>>>and thought that it would leave the main PG package in place.  But
>>>perhaps I was wrong.  I’ll follow Tom’s advice and just re-install
>>>everything (saving the old data directory) and hope the new
>>>installation can use the old data data directory.
>>> 
>>> 
>>> If you removed it and then installed it, then the removal would remove
>>> all dependent packages and if you then only intalled that one and not
>>> the dependencies that would explain it.
>>> 
>>> If you had run a reinstall on it, then it would've kept them around.
>>> 
>>> 
>>> 
>>>One question about this approach though:  the Debian package
>>>installation automatically initializes the new data directory and
>>>starts PG.  If I shut it down and copy the old data directory into
>>>the newly installed one, will there be an xlog issue?
>>> 
>>> 
>>> You have to copy the xlog along with the database.
>>> 
>>> Or if you leave it in place where it is, the packages won't initialize
>>> a new data directory.
>>> 
>>> --
>>> Magnus Hagander
>>> Me: http://www.hagander.net/
>>> Work: http://www.redpill-linpro.com/
>> 
> 
> 
> -- 
> Adrian Klaver
> adrian.kla...@aklaver.com



-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Shawn Thomas]

Yes, sadly it does explain things.  Your insight has been super helpful though.

-Shawn

> On Feb 15, 2017, at 9:38 AM, Adrian Klaver  wrote:
> 
> On 02/15/2017 09:28 AM, Shawn Thomas wrote:
>> Well that would make more sense of things.  I had removed and
>> re-installed the postresql-common package:
>> 
>> https://packages.debian.org/jessie/postgresql-common
> 
> Well that is the glue that holds the pgcluster scheme together. Also when I 
> try it I get:
> 
> sudo apt-get remove postgresql-common
> 
> The following packages will be REMOVED:
>  postgresql-9.4 postgresql-9.6 postgresql-common postgresql-contrib-9.4 
> postgresql-contrib-9.6 postgresql-server-dev-9.4 postgresql-server-dev-9.6
> Do you want to continue? [Y/n]
> 
> Which would explain a lot.
> 
>> 
>> and thought that it would leave the main PG package in place.  But
>> perhaps I was wrong.  I’ll follow Tom’s advice and just re-install
>> everything (saving the old data directory) and hope the new installation
>> can use the old data data directory.
>> 
>> One question about this approach though:  the Debian package
>> installation automatically initializes the new data directory and starts
>> PG.  If I shut it down and copy the old data directory into the newly
>> installed one, will there be an xlog issue?
>> 
>> -Shawn
>> 
>>> On Feb 15, 2017, at 9:09 AM, Magnus Hagander >> > wrote:
>>> 
>>> On Wed, Feb 15, 2017 at 6:03 PM, Shawn Thomas
>>> > wrote:
>>> 
>>>/usr/lib/postgresql/9.4/bin/pg_ctl: No such file or directory
>>> 
>>>postgres@pangaea:/usr/lib/postgresql/9.4/bin$ ls -al
>>>  total 4008
>>>  drwxr-xr-x 2 root root4096 Feb  9 16:17 .
>>>  drwxr-xr-x 3 root root4096 Feb  9 16:17 ..
>>>  -rwxr-xr-x 1 root root   68128 Nov 16 06:53 clusterdb
>>>  -rwxr-xr-x 1 root root   68192 Nov 16 06:53 createdb
>>>  -rwxr-xr-x 1 root root   63920 Nov 16 06:53 createlang
>>>  -rwxr-xr-x 1 root root   72672 Nov 16 06:53 createuser
>>>  -rwxr-xr-x 1 root root   63936 Nov 16 06:53 dropdb
>>>  -rwxr-xr-x 1 root root   63920 Nov 16 06:53 droplang
>>>  -rwxr-xr-x 1 root root   63904 Nov 16 06:53 dropuser
>>>  -rwxr-xr-x 1 root root   68416 Nov 16 06:53 pg_basebackup
>>>  -rwxr-xr-x 1 root root  351904 Nov 16 06:53 pg_dump
>>>  -rwxr-xr-x 1 root root 2186504 Nov 16 06:53 pg_dumpall
>>>  -rwxr-xr-x 1 root root   30992 Nov 16 06:53 pg_isready
>>>  -rwxr-xr-x 1 root root   47600 Nov 16 06:53 pg_receivexlog
>>>  -rwxr-xr-x 1 root root   51928 Nov 16 06:53 pg_recvlogical
>>>  -rwxr-xr-x 1 root root  154944 Nov 16 06:53 pg_restore
>>>  -rwxr-xr-x 1 root root  515320 Nov 16 06:53 psql
>>>  -rwxr-xr-x 1 root root   68160 Nov 16 06:53 reindexdb
>>>  -rwxr-xr-x 1 root root   72384 Nov 16 06:53 vacuumdb
>>> 
>>>As I mentioned, this Debian package removes pg_ctl from the bin
>>>directory and instead attempts to wrap the pg_ctl functionality in
>>>a perl script so that the PG process is integrated with systemd.
>>>I really wish they hadn’t, and it’s part of the reason I’m where
>>>I’m at.
>>> 
>>> 
>>> pg_ctl is normally present in /usr/lib/postgresql//bin on a
>>> debian system. If that is gone, somebody removed it, or you didn't
>>> install the "postgresql-9.4" package which provides it. On a 9.4 system:
>>> 
>>> $ dpkg -S /usr/lib/postgresql/9.4/bin/pg_ctl
>>> postgresql-9.4: /usr/lib/postgresql/9.4/bin/pg_ctl
>>> 
>>> You could try reinstalling the postgresql-9.4 package and see if it
>>> comes back. The rest of the binaries in that directory seems to be
>>> from postgresql-9.4-client though -- have you actually by mistake
>>> uninstalled the server package completely?
>>> 
>>> As in, that directory is supposed to have the "postgres" binary which
>>> is the database server and it's not there. So there is no wonder it's
>>> not starting...
>>> 
>>> --
>>> Magnus Hagander
>>> Me: http://www.hagander.net/
>>> Work: http://www.redpill-linpro.com/
>> 
> 
> 
> -- 
> Adrian Klaver
> adrian.kla...@aklaver.com



-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Adrian Klaver]

On 02/15/2017 09:45 AM, Shawn Thomas wrote:

Which would you recommend?  Leave the data directory in place and
re-install PG or copy it to somewhere else, delete it and then
re-install PG?


I would copy the data directory somewhere else for safe keeping leaving 
the original in place. Then reinstall Postgres, the install should leave 
the original directory alone and you will be ready to go. Should there 
be an oops you will have the copy as backup.




-Shawn


On Feb 15, 2017, at 9:36 AM, Magnus Hagander > wrote:

On Wed, Feb 15, 2017 at 6:28 PM, Shawn Thomas
> wrote:

Well that would make more sense of things.  I had removed and
re-installed the postresql-common package:

https://packages.debian.org/jessie/postgresql-common


and thought that it would leave the main PG package in place.  But
perhaps I was wrong.  I’ll follow Tom’s advice and just re-install
everything (saving the old data directory) and hope the new
installation can use the old data data directory.


If you removed it and then installed it, then the removal would remove
all dependent packages and if you then only intalled that one and not
the dependencies that would explain it.

If you had run a reinstall on it, then it would've kept them around.



One question about this approach though:  the Debian package
installation automatically initializes the new data directory and
starts PG.  If I shut it down and copy the old data directory into
the newly installed one, will there be an xlog issue?


You have to copy the xlog along with the database.

Or if you leave it in place where it is, the packages won't initialize
a new data directory.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/





--
Adrian Klaver
adrian.kla...@aklaver.com


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Shawn Thomas]

Which would you recommend?  Leave the data directory in place and re-install PG 
or copy it to somewhere else, delete it and then re-install PG?

-Shawn

> On Feb 15, 2017, at 9:36 AM, Magnus Hagander  wrote:
> 
> On Wed, Feb 15, 2017 at 6:28 PM, Shawn Thomas  > wrote:
> Well that would make more sense of things.  I had removed and re-installed 
> the postresql-common package:
> 
> https://packages.debian.org/jessie/postgresql-common 
> 
> 
> and thought that it would leave the main PG package in place.  But perhaps I 
> was wrong.  I’ll follow Tom’s advice and just re-install everything (saving 
> the old data directory) and hope the new installation can use the old data 
> data directory.
> 
> If you removed it and then installed it, then the removal would remove all 
> dependent packages and if you then only intalled that one and not the 
> dependencies that would explain it.
> 
> If you had run a reinstall on it, then it would've kept them around.
> 
>  
> One question about this approach though:  the Debian package installation 
> automatically initializes the new data directory and starts PG.  If I shut it 
> down and copy the old data directory into the newly installed one, will there 
> be an xlog issue?
> 
> You have to copy the xlog along with the database.
> 
> Or if you leave it in place where it is, the packages won't initialize a new 
> data directory.
> 
> -- 
>  Magnus Hagander
>  Me: http://www.hagander.net/ 
>  Work: http://www.redpill-linpro.com/ 

Re: [GENERAL] Can't restart Postgres

[Shawn Thomas]

I was responsible for deleting/re-inatalling the cert.  I was attempting to get 
Shibboleth (a federated authentication service that also uses the cert) running 
and didn’t realize that PG relied on it.

The dumpAll file wasn’t actually deleted but was empty.  I’ll look into that 
mystery once I get the database back in place.

-Shawn

> On Feb 15, 2017, at 9:01 AM, Adrian Klaver  wrote:
> 
> On 02/15/2017 08:35 AM, Shawn Thomas wrote:
>> Yes, that’s the correct sequence of scripts.  And no there’s not anything 
>> really helpful in the system logs.
>> 
>> I’m thinking that at this point I need to approach this problem as more of a 
>> disaster recovery.  There was a full pg_dumpall  file that was deleted and 
>> cannot be recovered so I need to recover the data from the 
>> /var/lib/postgresql/9.4/main directory.  I believe this is called a file 
>> level recovery.  I assume I need to use a fully functional, same version PG 
>> (on another machine?) to create a full dump of the data directory.  Once I 
>> have this I can re-install Postgres on the initial server and read the 
>> databases back into it.
> 
> I have to believe that if you cannot get the server to start then the data 
> directory is no shape to recover from. And if the data directory is good and 
> it is the program files that are corrupted then it would be a matter of 
> reinstalling Postgres. In either case the most important thing to do would be 
> to make a copy of the data directory before you do anything else.
> 
> What exactly happened that caused the ssl cert and the pg_dumpall file to 
> deleted?
> 
> In other words what else got deleted?
> 
>> 
>> Any advice on how to best go about this?  The official documentation seems a 
>> bit thin:
>> 
>> https://www.postgresql.org/docs/9.4/static/backup-file.html
>> 
>> I’ve only worked with normal (pg_dump, pg_dumpall) backups in the past.
>> 
>> -Shawn
>> 
>>> On Feb 15, 2017, at 6:35 AM, Adrian Klaver  
>>> wrote:
>>> 
>>> On 02/14/2017 08:47 PM, Shawn Thomas wrote:
 No it doesn’t matter if run with sudo, postgres or even root.  Debian
 actually wraps the command and executes some some initial scripts with
 different privileges but ends up making sure that Postgres ends up
 running under the postgres user.  I get the same output if run with sudo:
 
 sudo systemctl status postgresql@9.4-main.service
  -l
  Error: could not exec   start -D /var/lib/postgresql/9.4/main -l
 /var/log/postgresql/postgresql-9.4-main.log -s -o  -c
 config_file="/etc/postgresql/9.4/main/postgresql.conf”
 
>>> 
>>> 
>>> So you are talking about:
>>> 
>>> /etc/init.d/postgresql
>>> 
>>> which then calls:
>>> 
>>> /usr/share/postgresql-common/init.d-functions
>>> 
>>> Or is there another setup on your system?
>>> 
>>> Any relevant information in the system logs?
>>> 
 Thanks, though.
 
 -Shawn
>>> 
>>> 
>>> --
>>> Adrian Klaver
>>> adrian.kla...@aklaver.com
>> 
> 
> 
> -- 
> Adrian Klaver
> adrian.kla...@aklaver.com



-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Adrian Klaver]

On 02/15/2017 09:28 AM, Shawn Thomas wrote:

Well that would make more sense of things.  I had removed and
re-installed the postresql-common package:

https://packages.debian.org/jessie/postgresql-common


Well that is the glue that holds the pgcluster scheme together. Also 
when I try it I get:


sudo apt-get remove postgresql-common

The following packages will be REMOVED:
  postgresql-9.4 postgresql-9.6 postgresql-common 
postgresql-contrib-9.4 postgresql-contrib-9.6 postgresql-server-dev-9.4 
postgresql-server-dev-9.6

Do you want to continue? [Y/n]

Which would explain a lot.



and thought that it would leave the main PG package in place.  But
perhaps I was wrong.  I’ll follow Tom’s advice and just re-install
everything (saving the old data directory) and hope the new installation
can use the old data data directory.

One question about this approach though:  the Debian package
installation automatically initializes the new data directory and starts
PG.  If I shut it down and copy the old data directory into the newly
installed one, will there be an xlog issue?

-Shawn


On Feb 15, 2017, at 9:09 AM, Magnus Hagander > wrote:

On Wed, Feb 15, 2017 at 6:03 PM, Shawn Thomas
> wrote:

/usr/lib/postgresql/9.4/bin/pg_ctl: No such file or directory

postgres@pangaea:/usr/lib/postgresql/9.4/bin$ ls -al
  total 4008
  drwxr-xr-x 2 root root4096 Feb  9 16:17 .
  drwxr-xr-x 3 root root4096 Feb  9 16:17 ..
  -rwxr-xr-x 1 root root   68128 Nov 16 06:53 clusterdb
  -rwxr-xr-x 1 root root   68192 Nov 16 06:53 createdb
  -rwxr-xr-x 1 root root   63920 Nov 16 06:53 createlang
  -rwxr-xr-x 1 root root   72672 Nov 16 06:53 createuser
  -rwxr-xr-x 1 root root   63936 Nov 16 06:53 dropdb
  -rwxr-xr-x 1 root root   63920 Nov 16 06:53 droplang
  -rwxr-xr-x 1 root root   63904 Nov 16 06:53 dropuser
  -rwxr-xr-x 1 root root   68416 Nov 16 06:53 pg_basebackup
  -rwxr-xr-x 1 root root  351904 Nov 16 06:53 pg_dump
  -rwxr-xr-x 1 root root 2186504 Nov 16 06:53 pg_dumpall
  -rwxr-xr-x 1 root root   30992 Nov 16 06:53 pg_isready
  -rwxr-xr-x 1 root root   47600 Nov 16 06:53 pg_receivexlog
  -rwxr-xr-x 1 root root   51928 Nov 16 06:53 pg_recvlogical
  -rwxr-xr-x 1 root root  154944 Nov 16 06:53 pg_restore
  -rwxr-xr-x 1 root root  515320 Nov 16 06:53 psql
  -rwxr-xr-x 1 root root   68160 Nov 16 06:53 reindexdb
  -rwxr-xr-x 1 root root   72384 Nov 16 06:53 vacuumdb

As I mentioned, this Debian package removes pg_ctl from the bin
directory and instead attempts to wrap the pg_ctl functionality in
a perl script so that the PG process is integrated with systemd.
I really wish they hadn’t, and it’s part of the reason I’m where
I’m at.


pg_ctl is normally present in /usr/lib/postgresql//bin on a
debian system. If that is gone, somebody removed it, or you didn't
install the "postgresql-9.4" package which provides it. On a 9.4 system:

$ dpkg -S /usr/lib/postgresql/9.4/bin/pg_ctl
postgresql-9.4: /usr/lib/postgresql/9.4/bin/pg_ctl

You could try reinstalling the postgresql-9.4 package and see if it
comes back. The rest of the binaries in that directory seems to be
from postgresql-9.4-client though -- have you actually by mistake
uninstalled the server package completely?

As in, that directory is supposed to have the "postgres" binary which
is the database server and it's not there. So there is no wonder it's
not starting...

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/





--
Adrian Klaver
adrian.kla...@aklaver.com


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Magnus Hagander]

On Wed, Feb 15, 2017 at 6:28 PM, Shawn Thomas 
wrote:

> Well that would make more sense of things.  I had removed and re-installed
> the postresql-common package:
>
> https://packages.debian.org/jessie/postgresql-common
>
> and thought that it would leave the main PG package in place.  But perhaps
> I was wrong.  I’ll follow Tom’s advice and just re-install everything
> (saving the old data directory) and hope the new installation can use the
> old data data directory.
>

If you removed it and then installed it, then the removal would remove all
dependent packages and if you then only intalled that one and not the
dependencies that would explain it.

If you had run a reinstall on it, then it would've kept them around.



> One question about this approach though:  the Debian package installation
> automatically initializes the new data directory and starts PG.  If I shut
> it down and copy the old data directory into the newly installed one, will
> there be an xlog issue?
>

You have to copy the xlog along with the database.

Or if you leave it in place where it is, the packages won't initialize a
new data directory.

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Re: [GENERAL] Can't restart Postgres

[Shawn Thomas]

Well that would make more sense of things.  I had removed and re-installed the 
postresql-common package:

https://packages.debian.org/jessie/postgresql-common

and thought that it would leave the main PG package in place.  But perhaps I 
was wrong.  I’ll follow Tom’s advice and just re-install everything (saving the 
old data directory) and hope the new installation can use the old data data 
directory.

One question about this approach though:  the Debian package installation 
automatically initializes the new data directory and starts PG.  If I shut it 
down and copy the old data directory into the newly installed one, will there 
be an xlog issue?

-Shawn

> On Feb 15, 2017, at 9:09 AM, Magnus Hagander  wrote:
> 
> On Wed, Feb 15, 2017 at 6:03 PM, Shawn Thomas  > wrote:
> /usr/lib/postgresql/9.4/bin/pg_ctl: No such file or directory
> 
> postgres@pangaea:/usr/lib/postgresql/9.4/bin$ ls -al
>   total 4008
>   drwxr-xr-x 2 root root4096 Feb  9 16:17 .
>   drwxr-xr-x 3 root root4096 Feb  9 16:17 ..
>   -rwxr-xr-x 1 root root   68128 Nov 16 06:53 clusterdb
>   -rwxr-xr-x 1 root root   68192 Nov 16 06:53 createdb
>   -rwxr-xr-x 1 root root   63920 Nov 16 06:53 createlang
>   -rwxr-xr-x 1 root root   72672 Nov 16 06:53 createuser
>   -rwxr-xr-x 1 root root   63936 Nov 16 06:53 dropdb
>   -rwxr-xr-x 1 root root   63920 Nov 16 06:53 droplang
>   -rwxr-xr-x 1 root root   63904 Nov 16 06:53 dropuser
>   -rwxr-xr-x 1 root root   68416 Nov 16 06:53 pg_basebackup
>   -rwxr-xr-x 1 root root  351904 Nov 16 06:53 pg_dump
>   -rwxr-xr-x 1 root root 2186504 Nov 16 06:53 pg_dumpall
>   -rwxr-xr-x 1 root root   30992 Nov 16 06:53 pg_isready
>   -rwxr-xr-x 1 root root   47600 Nov 16 06:53 pg_receivexlog
>   -rwxr-xr-x 1 root root   51928 Nov 16 06:53 pg_recvlogical
>   -rwxr-xr-x 1 root root  154944 Nov 16 06:53 pg_restore
>   -rwxr-xr-x 1 root root  515320 Nov 16 06:53 psql
>   -rwxr-xr-x 1 root root   68160 Nov 16 06:53 reindexdb
>   -rwxr-xr-x 1 root root   72384 Nov 16 06:53 vacuumdb
> 
> As I mentioned, this Debian package removes pg_ctl from the bin directory and 
> instead attempts to wrap the pg_ctl functionality in a perl script so that 
> the PG process is integrated with systemd.  I really wish they hadn’t, and 
> it’s part of the reason I’m where I’m at.
> 
> pg_ctl is normally present in /usr/lib/postgresql//bin on a debian 
> system. If that is gone, somebody removed it, or you didn't install the 
> "postgresql-9.4" package which provides it. On a 9.4 system:
> 
> $ dpkg -S /usr/lib/postgresql/9.4/bin/pg_ctl
> postgresql-9.4: /usr/lib/postgresql/9.4/bin/pg_ctl
> 
> You could try reinstalling the postgresql-9.4 package and see if it comes 
> back. The rest of the binaries in that directory seems to be from 
> postgresql-9.4-client though -- have you actually by mistake uninstalled the 
> server package completely?
> 
> As in, that directory is supposed to have the "postgres" binary which is the 
> database server and it's not there. So there is no wonder it's not 
> starting... 
> 
> -- 
>  Magnus Hagander
>  Me: http://www.hagander.net/ 
>  Work: http://www.redpill-linpro.com/ 

Re: [GENERAL] Can't restart Postgres

[Adrian Klaver]

On 02/15/2017 09:28 AM, Joshua D. Drake wrote:

On 02/15/2017 09:17 AM, Adrian Klaver wrote:

On 02/15/2017 09:03 AM, Shawn Thomas wrote:

/usr/lib/postgresql/9.4/bin/pg_ctl: No such file or directory



That should have been:

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:Ubuntu 16.04.2 LTS
Release:16.04
Codename:   xenial



This is starting to sound like someone inadvertently executed an rm
somewhere they shouldn't have (outside of just the original ssl file).


Or a defective package(s) upgrade. Either way crucial parts are missing.



JD





--
Adrian Klaver
adrian.kla...@aklaver.com


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Joshua D. Drake]

On 02/15/2017 09:17 AM, Adrian Klaver wrote:

On 02/15/2017 09:03 AM, Shawn Thomas wrote:

/usr/lib/postgresql/9.4/bin/pg_ctl: No such file or directory



That should have been:

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:Ubuntu 16.04.2 LTS
Release:16.04
Codename:   xenial



This is starting to sound like someone inadvertently executed an rm 
somewhere they shouldn't have (outside of just the original ssl file).


JD


--
Command Prompt, Inc.  http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.
Unless otherwise stated, opinions are my own.


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Joshua D. Drake]

On 02/15/2017 09:03 AM, Shawn Thomas wrote:

/usr/lib/postgresql/9.4/bin/pg_ctl: No such file or directory

postgres@pangaea:/usr/lib/postgresql/9.4/bin$ ls -al
  total 4008
  drwxr-xr-x 2 root root4096 Feb  9 16:17 .
  drwxr-xr-x 3 root root4096 Feb  9 16:17 ..
  -rwxr-xr-x 1 root root   68128 Nov 16 06:53 clusterdb
  -rwxr-xr-x 1 root root   68192 Nov 16 06:53 createdb
  -rwxr-xr-x 1 root root   63920 Nov 16 06:53 createlang
  -rwxr-xr-x 1 root root   72672 Nov 16 06:53 createuser
  -rwxr-xr-x 1 root root   63936 Nov 16 06:53 dropdb
  -rwxr-xr-x 1 root root   63920 Nov 16 06:53 droplang
  -rwxr-xr-x 1 root root   63904 Nov 16 06:53 dropuser
  -rwxr-xr-x 1 root root   68416 Nov 16 06:53 pg_basebackup
  -rwxr-xr-x 1 root root  351904 Nov 16 06:53 pg_dump
  -rwxr-xr-x 1 root root 2186504 Nov 16 06:53 pg_dumpall
  -rwxr-xr-x 1 root root   30992 Nov 16 06:53 pg_isready
  -rwxr-xr-x 1 root root   47600 Nov 16 06:53 pg_receivexlog
  -rwxr-xr-x 1 root root   51928 Nov 16 06:53 pg_recvlogical
  -rwxr-xr-x 1 root root  154944 Nov 16 06:53 pg_restore
  -rwxr-xr-x 1 root root  515320 Nov 16 06:53 psql
  -rwxr-xr-x 1 root root   68160 Nov 16 06:53 reindexdb
  -rwxr-xr-x 1 root root   72384 Nov 16 06:53 vacuumdb

As I mentioned, this Debian package removes pg_ctl from the bin directory and 
instead attempts to wrap the pg_ctl functionality in a perl script so that the 
PG process is integrated with systemd.  I really wish they hadn’t, and it’s 
part of the reason I’m where I’m at.


Ugh o.k.. I run Ubuntu so I don't have that problem.

The thread is a little confusing about what has been tried and what 
hasn't but:


1. Disable ssl, try and start again
2. Try as postgres:

pg_ctlcluster 9.4 stop; pg_ctlcluster 9.4 start

3. Tom has a great suggestion if you are comfortable with those actions.

4. Is that directory listing the total of what is in 9.4/bin? If so... I 
don't see a postgres binary?


JD



--
Command Prompt, Inc.  http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.
Unless otherwise stated, opinions are my own.


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Adrian Klaver]

On 02/15/2017 09:03 AM, Shawn Thomas wrote:

/usr/lib/postgresql/9.4/bin/pg_ctl: No such file or directory



That should have been:

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:Ubuntu 16.04.2 LTS
Release:16.04
Codename:   xenial



--
Adrian Klaver
adrian.kla...@aklaver.com


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Adrian Klaver]

On 02/15/2017 09:03 AM, Shawn Thomas wrote:

/usr/lib/postgresql/9.4/bin/pg_ctl: No such file or directory

postgres@pangaea:/usr/lib/postgresql/9.4/bin$ ls -al
  total 4008
  drwxr-xr-x 2 root root4096 Feb  9 16:17 .
  drwxr-xr-x 3 root root4096 Feb  9 16:17 ..
  -rwxr-xr-x 1 root root   68128 Nov 16 06:53 clusterdb
  -rwxr-xr-x 1 root root   68192 Nov 16 06:53 createdb
  -rwxr-xr-x 1 root root   63920 Nov 16 06:53 createlang
  -rwxr-xr-x 1 root root   72672 Nov 16 06:53 createuser
  -rwxr-xr-x 1 root root   63936 Nov 16 06:53 dropdb
  -rwxr-xr-x 1 root root   63920 Nov 16 06:53 droplang
  -rwxr-xr-x 1 root root   63904 Nov 16 06:53 dropuser
  -rwxr-xr-x 1 root root   68416 Nov 16 06:53 pg_basebackup
  -rwxr-xr-x 1 root root  351904 Nov 16 06:53 pg_dump
  -rwxr-xr-x 1 root root 2186504 Nov 16 06:53 pg_dumpall
  -rwxr-xr-x 1 root root   30992 Nov 16 06:53 pg_isready
  -rwxr-xr-x 1 root root   47600 Nov 16 06:53 pg_receivexlog
  -rwxr-xr-x 1 root root   51928 Nov 16 06:53 pg_recvlogical
  -rwxr-xr-x 1 root root  154944 Nov 16 06:53 pg_restore
  -rwxr-xr-x 1 root root  515320 Nov 16 06:53 psql
  -rwxr-xr-x 1 root root   68160 Nov 16 06:53 reindexdb
  -rwxr-xr-x 1 root root   72384 Nov 16 06:53 vacuumdb

As I mentioned, this Debian package removes pg_ctl from the bin directory and 
instead attempts to wrap the pg_ctl functionality in a perl script so that the 
PG process is integrated with systemd.  I really wish they hadn’t, and it’s 
part of the reason I’m where I’m at.


Ubuntu uses the same setup and I see it, see below.

The Perl script just redirects commands to the correct version of 
Postgres and uses that versions binaries.



aklaver@arkansas:~$ uname -a
Linux arkansas 4.8.6-x86_64-linode78 #1 SMP Tue Nov 1 14:51:21 EDT 2016 
x86_64 x86_64 x86_64 GNU/Linux

aklaver@arkansas:~$ l /usr/lib/postgresql/9.4/bin/
total 8388
drwxr-xr-x 2 root root4096 Feb  9 07:24 ./
drwxr-xr-x 4 root root4096 Sep 29  2015 ../
-rwxr-xr-x 1 root root   68096 Feb  8 07:04 clusterdb*
-rwxr-xr-x 1 root root   68160 Feb  8 07:04 createdb*
-rwxr-xr-x 1 root root   63888 Feb  8 07:04 createlang*
-rwxr-xr-x 1 root root   72640 Feb  8 07:04 createuser*
-rwxr-xr-x 1 root root   63904 Feb  8 07:04 dropdb*
-rwxr-xr-x 1 root root   63888 Feb  8 07:04 droplang*
-rwxr-xr-x 1 root root   63872 Feb  8 07:04 dropuser*
-rwxr-xr-x 1 root root  114296 Feb  8 07:04 initdb*
-rwxr-xr-x 1 root root   26624 Feb  8 07:04 oid2name*
-rwxr-xr-x 1 root root   18432 Feb  8 07:04 pg_archivecleanup*
-rwxr-xr-x 1 root root   68416 Feb  8 07:04 pg_basebackup*
-rwxr-xr-x 1 root root   64600 Feb  8 07:04 pgbench*
-rwxr-xr-x 1 root root   30792 Feb  8 07:04 pg_config*
-rwxr-xr-x 1 root root   30720 Feb  8 07:04 pg_controldata*
-rwxr-xr-x 1 root root   43320 Feb  8 07:04 pg_ctl*
-rwxr-xr-x 1 root root  355968 Feb  8 07:04 pg_dump*
-rwxr-xr-x 1 root root   89320 Feb  8 07:04 pg_dumpall*
-rwxr-xr-x 1 root root   30960 Feb  8 07:04 pg_isready*
-rwxr-xr-x 1 root root   47568 Feb  8 07:04 pg_receivexlog*
-rwxr-xr-x 1 root root   51928 Feb  8 07:04 pg_recvlogical*
-rwxr-xr-x 1 root root   38920 Feb  8 07:04 pg_resetxlog*
-rwxr-xr-x 1 root root  154912 Feb  8 07:04 pg_restore*
-rwxr-xr-x 1 root root   22536 Feb  8 07:04 pg_standby*
-rwxr-xr-x 1 root root   22648 Feb  8 07:04 pg_test_fsync*
-rwxr-xr-x 1 root root   14416 Feb  8 07:04 pg_test_timing*
-rwxr-xr-x 1 root root  113168 Feb  8 07:04 pg_upgrade*
-rwxr-xr-x 1 root root   51672 Feb  8 07:04 pg_xlogdump*
-rwxr-xr-x 1 root root 5993920 Feb  8 07:04 postgres*
lrwxrwxrwx 1 root root   8 Feb  8 07:04 postmaster -> postgres*
-rwxr-xr-x 1 root root  519384 Feb  8 07:04 psql*
-rwxr-xr-x 1 root root   68128 Feb  8 07:04 reindexdb*
-rwxr-xr-x 1 root root   72352 Feb  8 07:04 vacuumdb*
-rwxr-xr-x 1 root root   22528 Feb  8 07:04 vacuumlo*



-Shawn




On Feb 15, 2017, at 8:49 AM, Joshua D. Drake  wrote:

On 02/15/2017 08:35 AM, Shawn Thomas wrote:

Yes, that’s the correct sequence of scripts.  And no there’s not anything 
really helpful in the system logs.

I’m thinking that at this point I need to approach this problem as more of a 
disaster recovery.  There was a full pg_dumpall  file that was deleted and 
cannot be recovered so I need to recover the data from the 
/var/lib/postgresql/9.4/main directory.  I believe this is called a file level 
recovery.  I assume I need to use a fully functional, same version PG (on 
another machine?) to create a full dump of the data directory.  Once I have 
this I can re-install Postgres on the initial server and read the databases 
back into it.

Any advice on how to best go about this?  The official documentation seems a 
bit thin:

https://www.postgresql.org/docs/9.4/static/backup-file.html

I’ve only worked with normal (pg_dump, pg_dumpall) backups in the past.


Shawn

As the postgres user:

/usr/lib/postgresql/9.4/bin/pg_ctl -D /var/lib/postgresql/9.4/main start

What returns?

Sincerely,

JD


--
Command Prompt, Inc.  

Re: [GENERAL] Can't restart Postgres

[Steve Atkins]

> On Feb 14, 2017, at 8:47 PM, Shawn Thomas  wrote:
> 
> No it doesn’t matter if run with sudo, postgres or even root.  Debian 
> actually wraps the command and executes some some initial scripts with 
> different privileges but ends up making sure that Postgres ends up running 
> under the postgres user.  I get the same output if run with sudo:
> 
> sudo systemctl status postgresql@9.4-main.service -l
>Error: could not exec   start -D /var/lib/postgresql/9.4/main -l 
> /var/log/postgresql/postgresql-9.4-main.log -s -o  -c 
> config_file="/etc/postgresql/9.4/main/postgresql.conf”

There's a suspicious hole between "exec" and "start" where I'd expect to see 
the full path to the pg_ctl binary. As though a variable were unset in a script 
or config file.

Cheers,
  Steve



-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Tom Lane]

Shawn Thomas  writes:
> I’m thinking that at this point I need to approach this problem as more of a 
> disaster recovery.  There was a full pg_dumpall  file that was deleted and 
> cannot be recovered so I need to recover the data from the 
> /var/lib/postgresql/9.4/main directory.  I believe this is called a file 
> level recovery.  I assume I need to use a fully functional, same version PG 
> (on another machine?) to create a full dump of the data directory.  Once I 
> have this I can re-install Postgres on the initial server and read the 
> databases back into it. 

Seems like the hard way.  Rename the data directory out of the way, delete
and reinstall the Postgres packages, and once you've confirmed it works
again, rename the data directory back into place (while the server is
stopped!)

If renaming the old data directory back into place makes it start failing
again, then you've narrowed down the problem to something about the
permissions or contents of the data directory itself or the configuration
files therein.

regards, tom lane


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Magnus Hagander]

On Wed, Feb 15, 2017 at 6:03 PM, Shawn Thomas 
wrote:

> /usr/lib/postgresql/9.4/bin/pg_ctl: No such file or directory
>
> postgres@pangaea:/usr/lib/postgresql/9.4/bin$ ls -al
>   total 4008
>   drwxr-xr-x 2 root root4096 Feb  9 16:17 .
>   drwxr-xr-x 3 root root4096 Feb  9 16:17 ..
>   -rwxr-xr-x 1 root root   68128 Nov 16 06:53 clusterdb
>   -rwxr-xr-x 1 root root   68192 Nov 16 06:53 createdb
>   -rwxr-xr-x 1 root root   63920 Nov 16 06:53 createlang
>   -rwxr-xr-x 1 root root   72672 Nov 16 06:53 createuser
>   -rwxr-xr-x 1 root root   63936 Nov 16 06:53 dropdb
>   -rwxr-xr-x 1 root root   63920 Nov 16 06:53 droplang
>   -rwxr-xr-x 1 root root   63904 Nov 16 06:53 dropuser
>   -rwxr-xr-x 1 root root   68416 Nov 16 06:53 pg_basebackup
>   -rwxr-xr-x 1 root root  351904 Nov 16 06:53 pg_dump
>   -rwxr-xr-x 1 root root 2186504 Nov 16 06:53 pg_dumpall
>   -rwxr-xr-x 1 root root   30992 Nov 16 06:53 pg_isready
>   -rwxr-xr-x 1 root root   47600 Nov 16 06:53 pg_receivexlog
>   -rwxr-xr-x 1 root root   51928 Nov 16 06:53 pg_recvlogical
>   -rwxr-xr-x 1 root root  154944 Nov 16 06:53 pg_restore
>   -rwxr-xr-x 1 root root  515320 Nov 16 06:53 psql
>   -rwxr-xr-x 1 root root   68160 Nov 16 06:53 reindexdb
>   -rwxr-xr-x 1 root root   72384 Nov 16 06:53 vacuumdb
>
> As I mentioned, this Debian package removes pg_ctl from the bin directory
> and instead attempts to wrap the pg_ctl functionality in a perl script so
> that the PG process is integrated with systemd.  I really wish they hadn’t,
> and it’s part of the reason I’m where I’m at.
>

pg_ctl is normally present in /usr/lib/postgresql//bin on a debian
system. If that is gone, somebody removed it, or you didn't install the
"postgresql-9.4" package which provides it. On a 9.4 system:

$ dpkg -S /usr/lib/postgresql/9.4/bin/pg_ctl
postgresql-9.4: /usr/lib/postgresql/9.4/bin/pg_ctl

You could try reinstalling the postgresql-9.4 package and see if it comes
back. The rest of the binaries in that directory seems to be from
postgresql-9.4-client though -- have you actually by mistake uninstalled
the server package completely?

As in, that directory is supposed to have the "postgres" binary which is
the database server and it's not there. So there is no wonder it's not
starting...

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Re: [GENERAL] Can't restart Postgres

[Shawn Thomas]

/usr/lib/postgresql/9.4/bin/pg_ctl: No such file or directory

postgres@pangaea:/usr/lib/postgresql/9.4/bin$ ls -al
  total 4008
  drwxr-xr-x 2 root root4096 Feb  9 16:17 .
  drwxr-xr-x 3 root root4096 Feb  9 16:17 ..
  -rwxr-xr-x 1 root root   68128 Nov 16 06:53 clusterdb
  -rwxr-xr-x 1 root root   68192 Nov 16 06:53 createdb
  -rwxr-xr-x 1 root root   63920 Nov 16 06:53 createlang
  -rwxr-xr-x 1 root root   72672 Nov 16 06:53 createuser
  -rwxr-xr-x 1 root root   63936 Nov 16 06:53 dropdb
  -rwxr-xr-x 1 root root   63920 Nov 16 06:53 droplang
  -rwxr-xr-x 1 root root   63904 Nov 16 06:53 dropuser
  -rwxr-xr-x 1 root root   68416 Nov 16 06:53 pg_basebackup
  -rwxr-xr-x 1 root root  351904 Nov 16 06:53 pg_dump
  -rwxr-xr-x 1 root root 2186504 Nov 16 06:53 pg_dumpall
  -rwxr-xr-x 1 root root   30992 Nov 16 06:53 pg_isready
  -rwxr-xr-x 1 root root   47600 Nov 16 06:53 pg_receivexlog
  -rwxr-xr-x 1 root root   51928 Nov 16 06:53 pg_recvlogical
  -rwxr-xr-x 1 root root  154944 Nov 16 06:53 pg_restore
  -rwxr-xr-x 1 root root  515320 Nov 16 06:53 psql
  -rwxr-xr-x 1 root root   68160 Nov 16 06:53 reindexdb
  -rwxr-xr-x 1 root root   72384 Nov 16 06:53 vacuumdb

As I mentioned, this Debian package removes pg_ctl from the bin directory and 
instead attempts to wrap the pg_ctl functionality in a perl script so that the 
PG process is integrated with systemd.  I really wish they hadn’t, and it’s 
part of the reason I’m where I’m at.

-Shawn



> On Feb 15, 2017, at 8:49 AM, Joshua D. Drake  wrote:
> 
> On 02/15/2017 08:35 AM, Shawn Thomas wrote:
>> Yes, that’s the correct sequence of scripts.  And no there’s not anything 
>> really helpful in the system logs.
>> 
>> I’m thinking that at this point I need to approach this problem as more of a 
>> disaster recovery.  There was a full pg_dumpall  file that was deleted and 
>> cannot be recovered so I need to recover the data from the 
>> /var/lib/postgresql/9.4/main directory.  I believe this is called a file 
>> level recovery.  I assume I need to use a fully functional, same version PG 
>> (on another machine?) to create a full dump of the data directory.  Once I 
>> have this I can re-install Postgres on the initial server and read the 
>> databases back into it.
>> 
>> Any advice on how to best go about this?  The official documentation seems a 
>> bit thin:
>> 
>> https://www.postgresql.org/docs/9.4/static/backup-file.html
>> 
>> I’ve only worked with normal (pg_dump, pg_dumpall) backups in the past.
> 
> Shawn
> 
> As the postgres user:
> 
> /usr/lib/postgresql/9.4/bin/pg_ctl -D /var/lib/postgresql/9.4/main start
> 
> What returns?
> 
> Sincerely,
> 
> JD
> 
> 
> -- 
> Command Prompt, Inc.  http://the.postgres.company/
>+1-503-667-4564
> PostgreSQL Centered full stack support, consulting and development.
> Everyone appreciates your honesty, until you are honest with them.
> Unless otherwise stated, opinions are my own.



-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Adrian Klaver]

On 02/15/2017 08:35 AM, Shawn Thomas wrote:

Yes, that’s the correct sequence of scripts.  And no there’s not anything 
really helpful in the system logs.

I’m thinking that at this point I need to approach this problem as more of a 
disaster recovery.  There was a full pg_dumpall  file that was deleted and 
cannot be recovered so I need to recover the data from the 
/var/lib/postgresql/9.4/main directory.  I believe this is called a file level 
recovery.  I assume I need to use a fully functional, same version PG (on 
another machine?) to create a full dump of the data directory.  Once I have 
this I can re-install Postgres on the initial server and read the databases 
back into it.


I have to believe that if you cannot get the server to start then the 
data directory is no shape to recover from. And if the data directory is 
good and it is the program files that are corrupted then it would be a 
matter of reinstalling Postgres. In either case the most important thing 
to do would be to make a copy of the data directory before you do 
anything else.


What exactly happened that caused the ssl cert and the pg_dumpall file 
to deleted?


In other words what else got deleted?



Any advice on how to best go about this?  The official documentation seems a 
bit thin:

https://www.postgresql.org/docs/9.4/static/backup-file.html

I’ve only worked with normal (pg_dump, pg_dumpall) backups in the past.

-Shawn


On Feb 15, 2017, at 6:35 AM, Adrian Klaver  wrote:

On 02/14/2017 08:47 PM, Shawn Thomas wrote:

No it doesn’t matter if run with sudo, postgres or even root.  Debian
actually wraps the command and executes some some initial scripts with
different privileges but ends up making sure that Postgres ends up
running under the postgres user.  I get the same output if run with sudo:

sudo systemctl status postgresql@9.4-main.service
 -l
  Error: could not exec   start -D /var/lib/postgresql/9.4/main -l
/var/log/postgresql/postgresql-9.4-main.log -s -o  -c
config_file="/etc/postgresql/9.4/main/postgresql.conf”




So you are talking about:

/etc/init.d/postgresql

which then calls:

/usr/share/postgresql-common/init.d-functions

Or is there another setup on your system?

Any relevant information in the system logs?


Thanks, though.

-Shawn



--
Adrian Klaver
adrian.kla...@aklaver.com





--
Adrian Klaver
adrian.kla...@aklaver.com


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Joshua D. Drake]

On 02/15/2017 08:35 AM, Shawn Thomas wrote:

Yes, that’s the correct sequence of scripts.  And no there’s not anything 
really helpful in the system logs.

I’m thinking that at this point I need to approach this problem as more of a 
disaster recovery.  There was a full pg_dumpall  file that was deleted and 
cannot be recovered so I need to recover the data from the 
/var/lib/postgresql/9.4/main directory.  I believe this is called a file level 
recovery.  I assume I need to use a fully functional, same version PG (on 
another machine?) to create a full dump of the data directory.  Once I have 
this I can re-install Postgres on the initial server and read the databases 
back into it.

Any advice on how to best go about this?  The official documentation seems a 
bit thin:

https://www.postgresql.org/docs/9.4/static/backup-file.html

I’ve only worked with normal (pg_dump, pg_dumpall) backups in the past.


Shawn

As the postgres user:

/usr/lib/postgresql/9.4/bin/pg_ctl -D /var/lib/postgresql/9.4/main start

What returns?

Sincerely,

JD


--
Command Prompt, Inc.  http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.
Unless otherwise stated, opinions are my own.


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Shawn Thomas]

Yes, that’s the correct sequence of scripts.  And no there’s not anything 
really helpful in the system logs.  

I’m thinking that at this point I need to approach this problem as more of a 
disaster recovery.  There was a full pg_dumpall  file that was deleted and 
cannot be recovered so I need to recover the data from the 
/var/lib/postgresql/9.4/main directory.  I believe this is called a file level 
recovery.  I assume I need to use a fully functional, same version PG (on 
another machine?) to create a full dump of the data directory.  Once I have 
this I can re-install Postgres on the initial server and read the databases 
back into it. 

Any advice on how to best go about this?  The official documentation seems a 
bit thin:

https://www.postgresql.org/docs/9.4/static/backup-file.html

I’ve only worked with normal (pg_dump, pg_dumpall) backups in the past.

-Shawn

> On Feb 15, 2017, at 6:35 AM, Adrian Klaver  wrote:
> 
> On 02/14/2017 08:47 PM, Shawn Thomas wrote:
>> No it doesn’t matter if run with sudo, postgres or even root.  Debian
>> actually wraps the command and executes some some initial scripts with
>> different privileges but ends up making sure that Postgres ends up
>> running under the postgres user.  I get the same output if run with sudo:
>> 
>> sudo systemctl status postgresql@9.4-main.service
>>  -l
>>   Error: could not exec   start -D /var/lib/postgresql/9.4/main -l
>> /var/log/postgresql/postgresql-9.4-main.log -s -o  -c
>> config_file="/etc/postgresql/9.4/main/postgresql.conf”
>> 
> 
> 
> So you are talking about:
> 
> /etc/init.d/postgresql
> 
> which then calls:
> 
> /usr/share/postgresql-common/init.d-functions
> 
> Or is there another setup on your system?
> 
> Any relevant information in the system logs?
> 
>> Thanks, though.
>> 
>> -Shawn
> 
> 
> -- 
> Adrian Klaver
> adrian.kla...@aklaver.com



-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Adrian Klaver]

On 02/14/2017 08:47 PM, Shawn Thomas wrote:

No it doesn’t matter if run with sudo, postgres or even root.  Debian
actually wraps the command and executes some some initial scripts with
different privileges but ends up making sure that Postgres ends up
running under the postgres user.  I get the same output if run with sudo:

sudo systemctl status postgresql@9.4-main.service
 -l
   Error: could not exec   start -D /var/lib/postgresql/9.4/main -l
/var/log/postgresql/postgresql-9.4-main.log -s -o  -c
config_file="/etc/postgresql/9.4/main/postgresql.conf”




So you are talking about:

/etc/init.d/postgresql

which then calls:

/usr/share/postgresql-common/init.d-functions

Or is there another setup on your system?

Any relevant information in the system logs?


Thanks, though.

-Shawn



--
Adrian Klaver
adrian.kla...@aklaver.com


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Gmail]

Sent from my iPad

> On Feb 14, 2017, at 9:47 PM, Shawn Thomas  wrote:
> 
> No it doesn’t matter if run with sudo, postgres or even root.  Debian 
> actually wraps the command and executes some some initial scripts with 
> different privileges but ends up making sure that Postgres ends up running 
> under the postgres user.  I get the same output if run with sudo:
> 
> sudo systemctl status postgresql@9.4-main.service -l
>Error: could not exec   start -D /var/lib/postgresql/9.4/main -l 
> /var/log/postgresql/postgresql-9.4-main.log -s -o  -c 
> config_file="/etc/postgresql/9.4/main/postgresql.conf”
> 
> Thanks, though.
> 
> -
which start

Can you run start with -x ?

Re: [GENERAL] Can't restart Postgres

[Shawn Thomas]

No it doesn’t matter if run with sudo, postgres or even root.  Debian actually 
wraps the command and executes some some initial scripts with different 
privileges but ends up making sure that Postgres ends up running under the 
postgres user.  I get the same output if run with sudo:

sudo systemctl status postgresql@9.4-main.service -l
   Error: could not exec   start -D /var/lib/postgresql/9.4/main -l 
/var/log/postgresql/postgresql-9.4-main.log -s -o  -c 
config_file="/etc/postgresql/9.4/main/postgresql.conf”

Thanks, though.

-Shawn

> On Feb 14, 2017, at 5:12 PM, Adrian Klaver  wrote:
> 
> On 02/14/2017 05:00 PM, Adrian Klaver wrote:
>> On 02/14/2017 12:00 PM, Shawn Thomas wrote:
>>> Yes that would be the standard approach.  But the Debian package removes
>>> pg_ctl from it normal place and wraps it with a perl script in a way
>>> that makes it difficult to work with (it doesn’t accept the same
>>> arguments):
>>> 
>>> https://wiki.debian.org/PostgreSql#pg_ctl_replacement
>>> 
>>> @Mangnus, can you give me an example of how I might use pg_lsclusters
>>> and pg_ctlcluster?  I’ve tried:
>>> 
>> 
>> I do not see a sudo below or is it apparent whether you are doing this
>> as the postgres user.
>> 
>>> pg_ctlcluster 9.4 main start
>>> Error: could not exec   start -D /var/lib/postgresql/9.4/main -l
>>> /var/log/postgresql/postgresql-9.4-main.log -s -o  -c
>>> config_file="/etc/postgresql/9.4/main/postgresql.conf”
>> 
>> Not sure how close Debian 8 is to Ubuntu 16.04(something I use), but
>> from your first post they look like they share the same startup scripts.
>> So something like:
>> 
>> sudo systemctl restart postgresql@9.4-main.service
> ^^^
>  Should be  start
>> 
>> 
>>> 
>>> -Shawn
>>> 
 On Feb 14, 2017, at 11:52 AM, Magnus Hagander > wrote:
 
 On Tue, Feb 14, 2017 at 8:47 PM, Joshua D. Drake > wrote:
 
On 02/14/2017 11:43 AM, Shawn Thomas wrote:
 
pangaea:/var/log# systemctl status postgresql
● postgresql.service - PostgreSQL RDBMS
   Loaded: loaded (/lib/systemd/system/postgresql.service;
enabled)
   Active: active (exited) since Tue 2017-02-14 10:48:18 PST;
50min ago
  Process: 28668 ExecStart=/bin/true (code=exited,
status=0/SUCCESS)
 Main PID: 28668 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/postgresql.service
 
 
What about if use pg_ctl as the postgres user? That will give you
a better idea.
 
 
 You don't want ot be doing that on a systemd system, but try a
 combination of pg_lsclusters and pg_ctlcluster. Might be you need to
 shut it down once that way before it realizes it's down,and then start
 it back up.
 
 
 --
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/
>>> 
>> 
>> 
> 
> 
> -- 
> Adrian Klaver
> adrian.kla...@aklaver.com

Re: [GENERAL] Can't restart Postgres

[Adrian Klaver]

On 02/14/2017 05:00 PM, Adrian Klaver wrote:

On 02/14/2017 12:00 PM, Shawn Thomas wrote:

Yes that would be the standard approach.  But the Debian package removes
pg_ctl from it normal place and wraps it with a perl script in a way
that makes it difficult to work with (it doesn’t accept the same
arguments):

https://wiki.debian.org/PostgreSql#pg_ctl_replacement

@Mangnus, can you give me an example of how I might use pg_lsclusters
and pg_ctlcluster?  I’ve tried:



I do not see a sudo below or is it apparent whether you are doing this
as the postgres user.


pg_ctlcluster 9.4 main start
Error: could not exec   start -D /var/lib/postgresql/9.4/main -l
/var/log/postgresql/postgresql-9.4-main.log -s -o  -c
config_file="/etc/postgresql/9.4/main/postgresql.conf”


Not sure how close Debian 8 is to Ubuntu 16.04(something I use), but
from your first post they look like they share the same startup scripts.
So something like:

sudo systemctl restart postgresql@9.4-main.service

 ^^^
  Should be  start





-Shawn


On Feb 14, 2017, at 11:52 AM, Magnus Hagander > wrote:

On Tue, Feb 14, 2017 at 8:47 PM, Joshua D. Drake > wrote:

On 02/14/2017 11:43 AM, Shawn Thomas wrote:

pangaea:/var/log# systemctl status postgresql
● postgresql.service - PostgreSQL RDBMS
   Loaded: loaded (/lib/systemd/system/postgresql.service;
enabled)
   Active: active (exited) since Tue 2017-02-14 10:48:18 PST;
50min ago
  Process: 28668 ExecStart=/bin/true (code=exited,
status=0/SUCCESS)
 Main PID: 28668 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/postgresql.service


What about if use pg_ctl as the postgres user? That will give you
a better idea.


You don't want ot be doing that on a systemd system, but try a
combination of pg_lsclusters and pg_ctlcluster. Might be you need to
shut it down once that way before it realizes it's down,and then start
it back up.


--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/








--
Adrian Klaver
adrian.kla...@aklaver.com


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Adrian Klaver]

On 02/14/2017 12:00 PM, Shawn Thomas wrote:

Yes that would be the standard approach.  But the Debian package removes
pg_ctl from it normal place and wraps it with a perl script in a way
that makes it difficult to work with (it doesn’t accept the same arguments):

https://wiki.debian.org/PostgreSql#pg_ctl_replacement

@Mangnus, can you give me an example of how I might use pg_lsclusters
and pg_ctlcluster?  I’ve tried:



I do not see a sudo below or is it apparent whether you are doing this 
as the postgres user.



pg_ctlcluster 9.4 main start
Error: could not exec   start -D /var/lib/postgresql/9.4/main -l
/var/log/postgresql/postgresql-9.4-main.log -s -o  -c
config_file="/etc/postgresql/9.4/main/postgresql.conf”


Not sure how close Debian 8 is to Ubuntu 16.04(something I use), but 
from your first post they look like they share the same startup scripts. 
So something like:


sudo systemctl restart postgresql@9.4-main.service




-Shawn


On Feb 14, 2017, at 11:52 AM, Magnus Hagander > wrote:

On Tue, Feb 14, 2017 at 8:47 PM, Joshua D. Drake > wrote:

On 02/14/2017 11:43 AM, Shawn Thomas wrote:

pangaea:/var/log# systemctl status postgresql
● postgresql.service - PostgreSQL RDBMS
   Loaded: loaded (/lib/systemd/system/postgresql.service;
enabled)
   Active: active (exited) since Tue 2017-02-14 10:48:18 PST;
50min ago
  Process: 28668 ExecStart=/bin/true (code=exited,
status=0/SUCCESS)
 Main PID: 28668 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/postgresql.service


What about if use pg_ctl as the postgres user? That will give you
a better idea.


You don't want ot be doing that on a systemd system, but try a
combination of pg_lsclusters and pg_ctlcluster. Might be you need to
shut it down once that way before it realizes it's down,and then start
it back up.


--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/





--
Adrian Klaver
adrian.kla...@aklaver.com


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Tom Lane]

Shawn Thomas  writes:
> I inadvertently deleted the ssl-cert-snakeoil.pem out from under a running 
> Postgres instance (9.4) which caused it to shut down. The last line of 
> main.log:
> FATAL:  could not load server certificate file 
> "/etc/ssl/certs/ssl-cert-snakeoil.pem": No such file or directory

There's a bit more to it than that, because simply deleting that file would
not result in a live server shutting down; before v10, the server only
examines its certificate file at startup.

> I've since restored the cert but cannot get Postgres to start back up.

FWIW, I don't see how removing that file would result in a silent exit
without any error messages.  I suspect you did more damage to the PG
installation than you've realized.  As JD mentioned, permissions problems
on the executable are a possibility.  I'm also wondering if you broke
the logging configuration, such that PG tries to write an error log
message but can't, or it's writing it somewhere other than where you
expect.

regards, tom lane


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Shawn Thomas]

Yes that would be the standard approach.  But the Debian package removes pg_ctl 
from it normal place and wraps it with a perl script in a way that makes it 
difficult to work with (it doesn’t accept the same arguments):

https://wiki.debian.org/PostgreSql#pg_ctl_replacement 


@Mangnus, can you give me an example of how I might use pg_lsclusters and 
pg_ctlcluster?  I’ve tried:

pg_ctlcluster 9.4 main start
Error: could not exec   start -D /var/lib/postgresql/9.4/main -l 
/var/log/postgresql/postgresql-9.4-main.log -s -o  -c 
config_file="/etc/postgresql/9.4/main/postgresql.conf” 

-Shawn

> On Feb 14, 2017, at 11:52 AM, Magnus Hagander  wrote:
> 
> On Tue, Feb 14, 2017 at 8:47 PM, Joshua D. Drake  > wrote:
> On 02/14/2017 11:43 AM, Shawn Thomas wrote:
> pangaea:/var/log# systemctl status postgresql
> ● postgresql.service - PostgreSQL RDBMS
>Loaded: loaded (/lib/systemd/system/postgresql.service; enabled)
>Active: active (exited) since Tue 2017-02-14 10:48:18 PST; 50min ago
>   Process: 28668 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
>  Main PID: 28668 (code=exited, status=0/SUCCESS)
>CGroup: /system.slice/postgresql.service
> 
> What about if use pg_ctl as the postgres user? That will give you a better 
> idea.
> 
> You don't want ot be doing that on a systemd system, but try a combination of 
> pg_lsclusters and pg_ctlcluster. Might be you need to shut it down once that 
> way before it realizes it's down,and then start it back up. 
> 
> 
> -- 
>  Magnus Hagander
>  Me: http://www.hagander.net/ 
>  Work: http://www.redpill-linpro.com/ 

Re: [GENERAL] Can't restart Postgres

[Moreno Andreo]

Il 14/02/2017 20:31, Joshua D. Drake ha scritto:

On 02/14/2017 11:17 AM, Shawn Thomas wrote:
I inadvertently deleted the ssl-cert-snakeoil.pem out from under a 
running Postgres instance (9.4) which caused it to shut down. The 
last line of main.log:


FATAL:  could not load server certificate file 
"/etc/ssl/certs/ssl-cert-snakeoil.pem": No such file or directory


I've since restored the cert but cannot get Postgres to start back 
up.  It's the Debian 8 packaged version which complicates the 
debugging and troubleshooting.  There doesn't seem to be a way to do 
anything with Postgres outsided the of Debian's systemd wrappers.  
All I've got to work with is from /var/syslog:


pangaea systemd[1]: Starting PostgreSQL Cluster 9.4-main...
pangaea postgresql@9.4-main[28684]: Error: could not exec  start -D 
/var/lib/postgresql/9.4/main -l 
/var/log/postgresql/postgresql-9.4-main.log -s -o  -c 
config_file="/etc/postgresql/9.4/main/postgresql.conf" :
pangaea systemd[1]: postgresql@9.4-main.service: control process 
exited, code=exited status=1


Any help/advice would be greatly appreciated.


It is likely a permissions issue. What does the systemctl log say?


I'd take a look with a simple ls -las in the certificate directory 
(/etc/ssl/certs)... Not being sure but your postgres user should be at 
least capable of reading it (the certificate), if not being the owner at 
all


Cheers
Moreno



JD



-Shawn









--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Magnus Hagander]

On Tue, Feb 14, 2017 at 8:47 PM, Joshua D. Drake 
wrote:

> On 02/14/2017 11:43 AM, Shawn Thomas wrote:
>
>> pangaea:/var/log# systemctl status postgresql
>> ● postgresql.service - PostgreSQL RDBMS
>>Loaded: loaded (/lib/systemd/system/postgresql.service; enabled)
>>Active: active (exited) since Tue 2017-02-14 10:48:18 PST; 50min ago
>>   Process: 28668 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
>>  Main PID: 28668 (code=exited, status=0/SUCCESS)
>>CGroup: /system.slice/postgresql.service
>>
>
> What about if use pg_ctl as the postgres user? That will give you a better
> idea.


You don't want ot be doing that on a systemd system, but try a combination
of pg_lsclusters and pg_ctlcluster. Might be you need to shut it down once
that way before it realizes it's down,and then start it back up.


-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Re: [GENERAL] Can't restart Postgres

[Joshua D. Drake]

On 02/14/2017 11:43 AM, Shawn Thomas wrote:

pangaea:/var/log# systemctl status postgresql
● postgresql.service - PostgreSQL RDBMS
   Loaded: loaded (/lib/systemd/system/postgresql.service; enabled)
   Active: active (exited) since Tue 2017-02-14 10:48:18 PST; 50min ago
  Process: 28668 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
 Main PID: 28668 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/postgresql.service


What about if use pg_ctl as the postgres user? That will give you a 
better idea.


jD

--
Command Prompt, Inc.  http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.
Unless otherwise stated, opinions are my own.


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Shawn Thomas]

pangaea:/var/log# systemctl status postgresql
● postgresql.service - PostgreSQL RDBMS
   Loaded: loaded (/lib/systemd/system/postgresql.service; enabled)
   Active: active (exited) since Tue 2017-02-14 10:48:18 PST; 50min ago
  Process: 28668 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
 Main PID: 28668 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/postgresql.service

-Shawn

> On Feb 14, 2017, at 11:31 AM, Joshua D. Drake  wrote:
> 
> On 02/14/2017 11:17 AM, Shawn Thomas wrote:
>> I inadvertently deleted the ssl-cert-snakeoil.pem out from under a running 
>> Postgres instance (9.4) which caused it to shut down. The last line of 
>> main.log:
>> 
>> FATAL:  could not load server certificate file 
>> "/etc/ssl/certs/ssl-cert-snakeoil.pem": No such file or directory
>> 
>> I've since restored the cert but cannot get Postgres to start back up.  It's 
>> the Debian 8 packaged version which complicates the debugging and 
>> troubleshooting.  There doesn't seem to be a way to do anything with 
>> Postgres outsided the of Debian's systemd wrappers.  All I've got to work 
>> with is from /var/syslog:
>> 
>> pangaea systemd[1]: Starting PostgreSQL Cluster 9.4-main...
>> pangaea postgresql@9.4-main[28684]: Error: could not exec  start -D 
>> /var/lib/postgresql/9.4/main -l /var/log/postgresql/postgresql-9.4-main.log 
>> -s -o  -c config_file="/etc/postgresql/9.4/main/postgresql.conf" :
>> pangaea systemd[1]: postgresql@9.4-main.service: control process exited, 
>> code=exited status=1
>> 
>> Any help/advice would be greatly appreciated.
> 
> It is likely a permissions issue. What does the systemctl log say?
> 
> JD
> 
>> 
>> -Shawn
>> 
> 
> 
> -- 
> Command Prompt, Inc.  http://the.postgres.company/ 
> 
>+1-503-667-4564
> PostgreSQL Centered full stack support, consulting and development.
> Everyone appreciates your honesty, until you are honest with them.
> Unless otherwise stated, opinions are my own.

Re: [GENERAL] Can't restart Postgres

[Joshua D. Drake]

On 02/14/2017 11:17 AM, Shawn Thomas wrote:

I inadvertently deleted the ssl-cert-snakeoil.pem out from under a running 
Postgres instance (9.4) which caused it to shut down. The last line of main.log:

FATAL:  could not load server certificate file 
"/etc/ssl/certs/ssl-cert-snakeoil.pem": No such file or directory

I've since restored the cert but cannot get Postgres to start back up.  It's 
the Debian 8 packaged version which complicates the debugging and 
troubleshooting.  There doesn't seem to be a way to do anything with Postgres 
outsided the of Debian's systemd wrappers.  All I've got to work with is from 
/var/syslog:

pangaea systemd[1]: Starting PostgreSQL Cluster 9.4-main...
pangaea postgresql@9.4-main[28684]: Error: could not exec  start -D 
/var/lib/postgresql/9.4/main -l /var/log/postgresql/postgresql-9.4-main.log -s -o  -c 
config_file="/etc/postgresql/9.4/main/postgresql.conf" :
pangaea systemd[1]: postgresql@9.4-main.service: control process exited, 
code=exited status=1

Any help/advice would be greatly appreciated.


It is likely a permissions issue. What does the systemctl log say?

JD



-Shawn




--
Command Prompt, Inc.  http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.
Unless otherwise stated, opinions are my own.


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

[GENERAL] Can't restart Postgres

[Shawn Thomas]

I inadvertently deleted the ssl-cert-snakeoil.pem out from under a running 
Postgres instance (9.4) which caused it to shut down. The last line of main.log:

FATAL:  could not load server certificate file 
"/etc/ssl/certs/ssl-cert-snakeoil.pem": No such file or directory

I've since restored the cert but cannot get Postgres to start back up.  It's 
the Debian 8 packaged version which complicates the debugging and 
troubleshooting.  There doesn't seem to be a way to do anything with Postgres 
outsided the of Debian's systemd wrappers.  All I've got to work with is from 
/var/syslog:

pangaea systemd[1]: Starting PostgreSQL Cluster 9.4-main...
pangaea postgresql@9.4-main[28684]: Error: could not exec  start -D 
/var/lib/postgresql/9.4/main -l /var/log/postgresql/postgresql-9.4-main.log -s 
-o  -c config_file="/etc/postgresql/9.4/main/postgresql.conf" :
pangaea systemd[1]: postgresql@9.4-main.service: control process exited, 
code=exited status=1

Any help/advice would be greatly appreciated.

-Shawn

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Thom Brown]

I think I must have only done a reload on the live server as now I've
tried to restart the service and I've got exactly the same error, so
it's no longer a discrepancy between environments.

The script is actually one which came with the Gentoo package.  I can
see it is using both $PGOPTS and $PGDATA, neither which are populated
with anything on either server.  I've assigned $PGDATA to the database
cluster path but it still doesn't start.  I've also checked
/etc/conf.d/postgresql-8.3 which contains correct settings.

Okay, so I've manually tried starting the server now and told it to
output any log to /tmp.  This is telling me that the request for a
shared memory segment is higher than my kernel's SHMMAX parameter.  My
bad, I've put my settings in incorrectly, and as it states in the
config file, changes to that setting require a restart.  I've reset
all values to back to how they were and it is running again.  I didn't
think my changes were that demanding, but obviously there were.  I'll
have to look into it more.

Thanks for the suggestions.

Thom

On Thu, Oct 30, 2008 at 2:19 AM, Tom Lane [EMAIL PROTECTED] wrote:
 Thom Brown [EMAIL PROTECTED] writes:
 Actually I did ps aux | grep post just to cover all bases, but still
 nothing.. except of course the grep itself.

 The overwhelming impression from here is of a seriously brain-dead
 startup script.  It's spending all its effort on being chatty and none
 on actually dealing with unusual cases correctly :-(.  Whose script
 is it anyway?

 My bet is that there's some discrepancy between what the script is
 expecting and what your intended configuration is.  I'm not sure if
 the discrepancy is exactly the PID-file location or if it's more subtle
 than that, but anyway I'd suggest reading through that script carefully
 to see what it's actually doing.

regards, tom lane


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Thomas]

I myself noticed that if a client is still connected to the DB server,
then PgSQL won't restart. Are you sure all your clients are/were
disconnected? I myself have the DB on remote a virtual machine.

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Thom Brown]

Well that can't really be the problem since it isn't running when
trying to start.

But yes, I've noticed that before which I actually find very useful.
It's a shame there isn't a way for postgres to broadcast to clients
that it wants to shutdown so things like pgAdmin III will say Hey,
the server's about to go down.  Do what you need to go and get the
frag outta here!

On Thu, Oct 30, 2008 at 10:42 AM, Thomas [EMAIL PROTECTED] wrote:
 I myself noticed that if a client is still connected to the DB server,
 then PgSQL won't restart. Are you sure all your clients are/were
 disconnected? I myself have the DB on remote a virtual machine.


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Tom Lane]

Thom Brown [EMAIL PROTECTED] writes:
 The script is actually one which came with the Gentoo package.
 ...
 Okay, so I've manually tried starting the server now and told it to
 output any log to /tmp.  This is telling me that the request for a
 shared memory segment is higher than my kernel's SHMMAX parameter.  My
 bad, I've put my settings in incorrectly, and as it states in the
 config file, changes to that setting require a restart.  I've reset
 all values to back to how they were and it is running again.

Yeah, SHMMAX overrun is a pretty common problem.  You really need to
complain to whoever maintains the Gentoo package that their start script
is so utterly, noisily unhelpful in the presence of a postmaster startup
issue.

regards, tom lane

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

[GENERAL] Can't restart Postgres

[Thom Brown]

Hi,

I've got a development virtual server which matches live exactly
except for the fact that Postgres is running on a different port which
is not used by anything else.  Postgres was running fine until I
updated postgresql.conf to enhance logging and make better use of
system resources.

Here's the problem:

# /etc/init.d/postgresql-8.3 restart
 * Stopping PostgreSQL (this can take up to 90 seconds) ...
pg_ctl: PID file /var/lib/postgresql/8.3/data/postmaster.pid does not exist
Is server running?
 * Some clients did not disconnect within 30 seconds.
 * Going to shutdown the server anyway.
pg_ctl: PID file /var/lib/postgresql/8.3/data/postmaster.pid does not exist
Is server running?
 * Shutting down the server gracefully failed.
 * Forcing it to shutdown which leads to a recover-run on next startup.
pg_ctl: PID file /var/lib/postgresql/8.3/data/postmaster.pid does not exist
Is server running?
 * Forced shutdown failed!!! Something is wrong with your system,
please take care of it manually.
   [ ok ]
 * Starting PostgreSQL ...
waiting for server to
start...could
not start server[ !! ]
 * The pid-file doesn't exist but pg_ctl reported a running server.
 * Please check whether there is another server running on the same
port or read the log-file.


If you're curious, the settings I changed in postgresql.conf are as follows:

OLD: shared_buffers = 24MB
NEW: shared_buffers = 128MB

OLD: #log_destination = 'stderr'
NEW: log_destination = 'stderr'

OLD: #logging_collector = off
NEW: logging_collector = on

OLD: #log_directory = 'pg_log'
NEW: log_directory = '/var/log/pg_log'

OLD: #log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'
NEW: log_filename = 'postgresql-%Y-%m-%d.log'

OLD: #log_rotation_age = 1d
NEW: log_rotation_age = 1d

OLD: #log_min_duration_statement = -1
NEW: log_min_duration_statement = 0

OLD: #log_duration = off
NEW: log_duration = on

OLD: #log_line_prefix = ''
NEW: log_line_prefix = '%t [%p]: [%l-1] '

Note that the live and development configs are identical except for
the port number.

Netstat data

# netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address   Foreign Address State
tcp0  0 localhost:mysql *:* LISTEN
tcp0  0 *:sunrpc*:* LISTEN
tcp0  0 *:39571 *:* LISTEN
tcp6   0  0 [::]:http-alt   [::]:*  LISTEN
tcp6   0  0 [::]:http   [::]:*  LISTEN
tcp6   0  0 [::]:ssh[::]:*  LISTEN
tcp6   0  0 [::]:https  [::]:*  LISTEN
tcp6   0732 linode-dev.prehisto:ssh 217.154.203.18:4244 ESTABLISHED
udp0  0 *:779   *:*
udp0  0 *:32781 *:*
udp0  0 *:sunrpc*:*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags   Type   State I-Node Path
unix  2  [ ACC ] STREAM LISTENING 7294 @/tmp/fam-root-
unix  2  [ ACC ] STREAM LISTENING 1745591
/var/run/cgisock.19119
unix  2  [ ] DGRAM179
@/org/kernel/udev/udevd
unix  2  [ ACC ] STREAM LISTENING 6073 /dev/log
unix  2  [ ACC ] STREAM LISTENING 7275
/var/run/fail2ban/fail2ban.sock
unix  2  [ ACC ] STREAM LISTENING 88239
/var/run/mysqld/mysqld.sock
unix  3  [ ] STREAM CONNECTED 1799425  /dev/log
unix  3  [ ] STREAM CONNECTED 1799424
unix  3  [ ] STREAM CONNECTED 1799422
unix  3  [ ] STREAM CONNECTED 1799421
unix  3  [ ] STREAM CONNECTED 1746483  /dev/log
unix  3  [ ] STREAM CONNECTED 1746482
unix  3  [ ] STREAM CONNECTED 1746384  /dev/log
unix  3  [ ] STREAM CONNECTED 1746382
unix  3  [ ] STREAM CONNECTED 7427 /dev/log
unix  3  [ ] STREAM CONNECTED 7426
unix  3  [ ] STREAM CONNECTED 7298 @/tmp/fam-root-
unix  3  [ ] STREAM CONNECTED 7295
unix  3  [ ] STREAM CONNECTED 6511 /dev/log
unix  3  [ ] STREAM CONNECTED 6508

I'm sure I've had this problem before (a few months ago on my home PC)
and never did solve it.

If anyone can offer some insight I'd be grateful.

Thanks

Thom

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Richard Huxton]

Thom Brown wrote:
 Hi,
 
 I've got a development virtual server which matches live exactly
 except for the fact that Postgres is running on a different port

What do you mean by virtual server? And does it affect definitions of
localhost or shared-memory allocation?

 which
 is not used by anything else.  Postgres was running fine until I
 updated postgresql.conf to enhance logging and make better use of
 system resources.
 
 Here's the problem:
 
 # /etc/init.d/postgresql-8.3 restart
  * Stopping PostgreSQL (this can take up to 90 seconds) ...
 pg_ctl: PID file /var/lib/postgresql/8.3/data/postmaster.pid does not exist
 Is server running?

Your system isn't set up the way you think it is - the .pid file is
missing. Is it looking in the right place?

 waiting for server to
 start...could
 not start server[ !! ]
  * The pid-file doesn't exist but pg_ctl reported a running server.

 If you're curious, the settings I changed in postgresql.conf are as follows:
 
 OLD: shared_buffers = 24MB
 NEW: shared_buffers = 128MB

This can cause problems if your kernel doesn't allocate enough
shared-memory, but you should get a different error message.

 Note that the live and development configs are identical except for
 the port number.

Are they reading the right config files?

 Netstat data
 
 # netstat -a
 Active Internet connections (servers and established)
[snip]

I don't see postgresql here at all. Mysql, fam, fail2ban but not PG.

 If anyone can offer some insight I'd be grateful.

If you've got two installations on the same machine having problems then
either:
1. They're *not* running on different ports with different data
directories (check you're using the correct config file for each)

2. They're having problems with shared memory (in which case you should
see a different error message).

-- 
  Richard Huxton
  Archonet Ltd

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Serge Fonville]

Hi,
Did you check permissions?
Do the pid files exist?
What variables are set?

Regards,

Serge Fonville

On Wed, Oct 29, 2008 at 4:43 PM, Thom Brown [EMAIL PROTECTED] wrote:

 Hi,

 I've got a development virtual server which matches live exactly
 except for the fact that Postgres is running on a different port which
 is not used by anything else.  Postgres was running fine until I
 updated postgresql.conf to enhance logging and make better use of
 system resources.

 Here's the problem:

 # /etc/init.d/postgresql-8.3 restart
  * Stopping PostgreSQL (this can take up to 90 seconds) ...
 pg_ctl: PID file /var/lib/postgresql/8.3/data/postmaster.pid does not
 exist
 Is server running?
  * Some clients did not disconnect within 30 seconds.
  * Going to shutdown the server anyway.
 pg_ctl: PID file /var/lib/postgresql/8.3/data/postmaster.pid does not
 exist
 Is server running?
  * Shutting down the server gracefully failed.
  * Forcing it to shutdown which leads to a recover-run on next startup.
 pg_ctl: PID file /var/lib/postgresql/8.3/data/postmaster.pid does not
 exist
 Is server running?
  * Forced shutdown failed!!! Something is wrong with your system,
 please take care of it manually.
   [ ok ]
  * Starting PostgreSQL ...
 waiting for server to
 start...could
 not start server[ !! ]
  * The pid-file doesn't exist but pg_ctl reported a running server.
  * Please check whether there is another server running on the same
 port or read the log-file.


 If you're curious, the settings I changed in postgresql.conf are as
 follows:

 OLD: shared_buffers = 24MB
 NEW: shared_buffers = 128MB

 OLD: #log_destination = 'stderr'
 NEW: log_destination = 'stderr'

 OLD: #logging_collector = off
 NEW: logging_collector = on

 OLD: #log_directory = 'pg_log'
 NEW: log_directory = '/var/log/pg_log'

 OLD: #log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'
 NEW: log_filename = 'postgresql-%Y-%m-%d.log'

 OLD: #log_rotation_age = 1d
 NEW: log_rotation_age = 1d

 OLD: #log_min_duration_statement = -1
 NEW: log_min_duration_statement = 0

 OLD: #log_duration = off
 NEW: log_duration = on

 OLD: #log_line_prefix = ''
 NEW: log_line_prefix = '%t [%p]: [%l-1] '

 Note that the live and development configs are identical except for
 the port number.

 Netstat data

 # netstat -a
 Active Internet connections (servers and established)
 Proto Recv-Q Send-Q Local Address   Foreign Address State
 tcp0  0 localhost:mysql *:* LISTEN
 tcp0  0 *:sunrpc*:* LISTEN
 tcp0  0 *:39571 *:* LISTEN
 tcp6   0  0 [::]:http-alt   [::]:*  LISTEN
 tcp6   0  0 [::]:http   [::]:*  LISTEN
 tcp6   0  0 [::]:ssh[::]:*  LISTEN
 tcp6   0  0 [::]:https  [::]:*  LISTEN
 tcp6   0732 linode-dev.prehisto:ssh 217.154.203.18:4244
 ESTABLISHED
 udp0  0 *:779   *:*
 udp0  0 *:32781 *:*
 udp0  0 *:sunrpc*:*
 Active UNIX domain sockets (servers and established)
 Proto RefCnt Flags   Type   State I-Node Path
 unix  2  [ ACC ] STREAM LISTENING 7294 @/tmp/fam-root-
 unix  2  [ ACC ] STREAM LISTENING 1745591
 /var/run/cgisock.19119
 unix  2  [ ] DGRAM179
 @/org/kernel/udev/udevd
 unix  2  [ ACC ] STREAM LISTENING 6073 /dev/log
 unix  2  [ ACC ] STREAM LISTENING 7275
 /var/run/fail2ban/fail2ban.sock
 unix  2  [ ACC ] STREAM LISTENING 88239
 /var/run/mysqld/mysqld.sock
 unix  3  [ ] STREAM CONNECTED 1799425  /dev/log
 unix  3  [ ] STREAM CONNECTED 1799424
 unix  3  [ ] STREAM CONNECTED 1799422
 unix  3  [ ] STREAM CONNECTED 1799421
 unix  3  [ ] STREAM CONNECTED 1746483  /dev/log
 unix  3  [ ] STREAM CONNECTED 1746482
 unix  3  [ ] STREAM CONNECTED 1746384  /dev/log
 unix  3  [ ] STREAM CONNECTED 1746382
 unix  3  [ ] STREAM CONNECTED 7427 /dev/log
 unix  3  [ ] STREAM CONNECTED 7426
 unix  3  [ ] STREAM CONNECTED 7298 @/tmp/fam-root-
 unix  3  [ ] STREAM CONNECTED 7295
 unix  3  [ ] STREAM CONNECTED 6511 /dev/log
 unix  3  [ ] STREAM CONNECTED 6508

 I'm sure I've had this problem before (a few months ago on my home PC)
 and never did solve it.

 If anyone can offer some insight I'd be grateful.

 Thanks

 Thom

 --
 Sent via pgsql-general mailing list 

Re: [GENERAL] Can't restart Postgres

[Thom Brown]

Permissions are identical to live.  I've checked the /tmp folder for a
PID reference, but doesn't exist in live or dev.

What do you mean by variables?  How can I check?

I only have one postgresql database cluster on each server.

With regards to the config causing memory problems, the specs of both
environments are absolutely identical.  Same kernel, same disk space,
same distro, same memory.  In fact the development environment had to
be rebuilt so we cloned live and changed the necessary settings.  Is
that potentially an issue, bearing in mind it had been running until
now?

By virtual machine I mean the entire systems are running within
virtual machines on different physical machines.  It is completely
unaware of its host and localhost is a true localhost as far as its
concerned.

The port numbers are only different because we also have a staging
virtual machine which is on the default port and to remotely connect
to both we just changed the port on the development one to an unused
one.  This is the port is has been running on until now.

Thanks

Thom

On Wed, Oct 29, 2008 at 3:57 PM, Serge Fonville
[EMAIL PROTECTED] wrote:
 Hi,
 Did you check permissions?
 Do the pid files exist?
 What variables are set?
 Regards,
 Serge Fonville
 On Wed, Oct 29, 2008 at 4:43 PM, Thom Brown [EMAIL PROTECTED] wrote:

 Hi,

 I've got a development virtual server which matches live exactly
 except for the fact that Postgres is running on a different port which
 is not used by anything else.  Postgres was running fine until I
 updated postgresql.conf to enhance logging and make better use of
 system resources.

 Here's the problem:

 # /etc/init.d/postgresql-8.3 restart
  * Stopping PostgreSQL (this can take up to 90 seconds) ...
 pg_ctl: PID file /var/lib/postgresql/8.3/data/postmaster.pid does not
 exist
 Is server running?
  * Some clients did not disconnect within 30 seconds.
  * Going to shutdown the server anyway.
 pg_ctl: PID file /var/lib/postgresql/8.3/data/postmaster.pid does not
 exist
 Is server running?
  * Shutting down the server gracefully failed.
  * Forcing it to shutdown which leads to a recover-run on next startup.
 pg_ctl: PID file /var/lib/postgresql/8.3/data/postmaster.pid does not
 exist
 Is server running?
  * Forced shutdown failed!!! Something is wrong with your system,
 please take care of it manually.
   [ ok ]
  * Starting PostgreSQL ...
 waiting for server to
 start...could
 not start server[ !! ]
  * The pid-file doesn't exist but pg_ctl reported a running server.
  * Please check whether there is another server running on the same
 port or read the log-file.


 If you're curious, the settings I changed in postgresql.conf are as
 follows:

 OLD: shared_buffers = 24MB
 NEW: shared_buffers = 128MB

 OLD: #log_destination = 'stderr'
 NEW: log_destination = 'stderr'

 OLD: #logging_collector = off
 NEW: logging_collector = on

 OLD: #log_directory = 'pg_log'
 NEW: log_directory = '/var/log/pg_log'

 OLD: #log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'
 NEW: log_filename = 'postgresql-%Y-%m-%d.log'

 OLD: #log_rotation_age = 1d
 NEW: log_rotation_age = 1d

 OLD: #log_min_duration_statement = -1
 NEW: log_min_duration_statement = 0

 OLD: #log_duration = off
 NEW: log_duration = on

 OLD: #log_line_prefix = ''
 NEW: log_line_prefix = '%t [%p]: [%l-1] '

 Note that the live and development configs are identical except for
 the port number.

 Netstat data

 # netstat -a
 Active Internet connections (servers and established)
 Proto Recv-Q Send-Q Local Address   Foreign Address State
 tcp0  0 localhost:mysql *:* LISTEN
 tcp0  0 *:sunrpc*:* LISTEN
 tcp0  0 *:39571 *:* LISTEN
 tcp6   0  0 [::]:http-alt   [::]:*  LISTEN
 tcp6   0  0 [::]:http   [::]:*  LISTEN
 tcp6   0  0 [::]:ssh[::]:*  LISTEN
 tcp6   0  0 [::]:https  [::]:*  LISTEN
 tcp6   0732 linode-dev.prehisto:ssh 217.154.203.18:4244
 ESTABLISHED
 udp0  0 *:779   *:*
 udp0  0 *:32781 *:*
 udp0  0 *:sunrpc*:*
 Active UNIX domain sockets (servers and established)
 Proto RefCnt Flags   Type   State I-Node Path
 unix  2  [ ACC ] STREAM LISTENING 7294 @/tmp/fam-root-
 unix  2  [ ACC ] STREAM LISTENING 1745591
 /var/run/cgisock.19119
 unix  2  [ ] DGRAM179
 @/org/kernel/udev/udevd
 unix  2  [ ACC ] STREAM LISTENING 6073 /dev/log
 unix  2  [ ACC ] STREAM LISTENING 7275
 /var/run/fail2ban/fail2ban.sock
 unix  2  [ ACC ] STREAM 

Re: [GENERAL] Can't restart Postgres

[Scott Marlowe]

On Wed, Oct 29, 2008 at 9:43 AM, Thom Brown [EMAIL PROTECTED] wrote:
 Hi,
  * Forcing it to shutdown which leads to a recover-run on next startup.
 pg_ctl: PID file /var/lib/postgresql/8.3/data/postmaster.pid does not exist
 Is server running?
  * Forced shutdown failed!!! Something is wrong with your system,
 please take care of it manually.
   [ ok ]
  * Starting PostgreSQL ...
 waiting for server to
 start...could
 not start server[ !! ]
  * The pid-file doesn't exist but pg_ctl reported a running server.
  * Please check whether there is another server running on the same
 port or read the log-file.

At this point did you do something like:

ps ax|grep postgres

???

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Thom Brown]

Actually I did ps aux | grep post just to cover all bases, but still
nothing.. except of course the grep itself.

On Wed, Oct 29, 2008 at 6:38 PM, Scott Marlowe [EMAIL PROTECTED] wrote:
 On Wed, Oct 29, 2008 at 9:43 AM, Thom Brown [EMAIL PROTECTED] wrote:
 Hi,
  * Forcing it to shutdown which leads to a recover-run on next startup.
 pg_ctl: PID file /var/lib/postgresql/8.3/data/postmaster.pid does not exist
 Is server running?
  * Forced shutdown failed!!! Something is wrong with your system,
 please take care of it manually.
   [ ok ]
  * Starting PostgreSQL ...
 waiting for server to
 start...could
 not start server[ !! ]
  * The pid-file doesn't exist but pg_ctl reported a running server.
  * Please check whether there is another server running on the same
 port or read the log-file.

 At this point did you do something like:

 ps ax|grep postgres

 ???


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Can't restart Postgres

[Tom Lane]

Thom Brown [EMAIL PROTECTED] writes:
 Actually I did ps aux | grep post just to cover all bases, but still
 nothing.. except of course the grep itself.

The overwhelming impression from here is of a seriously brain-dead
startup script.  It's spending all its effort on being chatty and none
on actually dealing with unusual cases correctly :-(.  Whose script
is it anyway?

My bet is that there's some discrepancy between what the script is
expecting and what your intended configuration is.  I'm not sure if
the discrepancy is exactly the PID-file location or if it's more subtle
than that, but anyway I'd suggest reading through that script carefully
to see what it's actually doing.

regards, tom lane

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general