On 24/9/20 22:42, Rene Peinthor wrote:
They are.
And how do you start the controller?
It should start automatically at boot from systemd, but I can start it
manually with:
systemctl restart linstor-controller
In case it is your next question, here is the content of
/etc/systemd/system/multi-user.target.wants/linstor-controller.service
[Unit]
Description=LINSTOR Controller Service
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
ExecStart=/usr/share/linstor-server/bin/Controller
--logs=/var/log/linstor-controller --config-directory=/etc/linstor
# if killed by signal 143 -> SIGTERM, 129 -> SIGHUP
SuccessExitStatus=0 143 129
User=root
PrivateTmp=yes
[Install]
WantedBy=multi-user.target
Regards,
Adam
On Thu, Sep 24, 2020 at 2:19 PM Adam Goryachev
<[email protected]
<mailto:[email protected]>> wrote:
On 24/9/20 15:30, Rene Peinthor wrote:
Sounds weird, I don't think we changed anything with the database
initialization.
Can you post your /etc/linstor/linstor.toml?
Cheers,
Rene
Hi, please see the contents below (I assume these are default
settings):
# Basic linstor configuration toml file
# For more options check documentation
[db]
user = "linstor"
password = "linstor"
connection_url = "jdbc:h2:/var/lib/linstor/linstordb"
Thanks,
Adam
On Wed, Sep 23, 2020 at 6:54 PM Adam Goryachev
<[email protected]
<mailto:[email protected]>> wrote:
Hi,
After upgrading on one of my test nodes I got some errors, so
I thought to upgrade the controller, and now I am unable to
start the controller.
Here is what I get in the system log:
Sep 24 02:33:40 castle systemd[1]: Started LINSTOR Controller
Service.
Sep 24 02:33:41 castle Controller[1093]: LINSTOR, Module
Controller
Sep 24 02:33:41 castle Controller[1093]: Version:
1.9.0 (678acd24a8b9b73a735407cd79ca33a5e95eb2e2)
Sep 24 02:33:41 castle Controller[1093]: Build time:
2020-09-23T10:27:49+00:00
Sep 24 02:33:41 castle Controller[1093]: Java Version: 11
Sep 24 02:33:41 castle Controller[1093]: Java VM:
Debian, Version 11.0.8+10-post-Debian-1deb10u1
Sep 24 02:33:41 castle Controller[1093]: Operating system:
Linux, Version 4.19.0-10-amd64
Sep 24 02:33:41 castle Controller[1093]: Environment:
amd64, 2 processors, 1988 MiB memory reserved for allocations
Sep 24 02:33:41 castle Controller[1093]: System components
initialization in progress
Sep 24 02:33:42 castle Controller[1093]: 02:33:42.355 [main]
INFO LINSTOR/Controller - SYSTEM - ErrorReporter DB version
1 found.
Sep 24 02:33:42 castle Controller[1093]: 02:33:42.359 [main]
INFO LINSTOR/Controller - SYSTEM - Log directory set to:
'/var/log/linstor-controller'
Sep 24 02:33:42 castle Controller[1093]: 02:33:42.410 [main]
WARN io.sentry.dsn.Dsn - *** Couldn't find a suitable DSN,
Sentry operations will do nothing! See documentation:
https://docs.sentry.io/clients/java/
<https://docs.sentry.io/clients/java/> ***
Sep 24 02:33:42 castle Controller[1093]: 02:33:42.434 [Main]
INFO LINSTOR/Controller - SYSTEM - Loading API classes started.
Sep 24 02:33:43 castle Controller[1093]: 02:33:43.046 [Main]
INFO LINSTOR/Controller - SYSTEM - API classes loading
finished: 611ms
Sep 24 02:33:43 castle Controller[1093]: 02:33:43.047 [Main]
INFO LINSTOR/Controller - SYSTEM - Dependency injection started.
Sep 24 02:33:43 castle Controller[1093]: 02:33:43.065 [Main]
INFO LINSTOR/Controller - SYSTEM - Attempting dynamic load
of extension module
"com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule"
Sep 24 02:33:43 castle Controller[1093]: 02:33:43.065 [Main]
INFO LINSTOR/Controller - SYSTEM - Extension module
"com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule"
is not installed
Sep 24 02:33:43 castle Controller[1093]: WARNING: An illegal
reflective access operation has occurred
Sep 24 02:33:43 castle Controller[1093]: WARNING: Illegal
reflective access by
com.google.inject.internal.cglib.core.$ReflectUtils$1
(file:/usr/share/linstor-server/lib/guice-4.2.2.jar) to
method
java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
Sep 24 02:33:43 castle Controller[1093]: WARNING: Please
consider reporting this to the maintainers of
com.google.inject.internal.cglib.core.$ReflectUtils$1
Sep 24 02:33:43 castle Controller[1093]: WARNING: Use
--illegal-access=warn to enable warnings of further illegal
reflective access operations
Sep 24 02:33:43 castle Controller[1093]: WARNING: All illegal
access operations will be denied in a future release
Sep 24 02:33:44 castle Controller[1093]: 02:33:44.962 [Main]
INFO LINSTOR/Controller - SYSTEM - Dependency injection
finished: 1915ms
Sep 24 02:33:45 castle Controller[1093]: 02:33:45.533 [Main]
INFO LINSTOR/Controller - SYSTEM - Initializing
authentication subsystem
Sep 24 02:33:45 castle Controller[1093]: 02:33:45.736 [Main]
INFO LINSTOR/Controller - SYSTEM - Starting service instance
'TimerEventService' of type TimerEventService
Sep 24 02:33:45 castle Controller[1093]: 02:33:45.740 [Main]
INFO LINSTOR/Controller - SYSTEM - Initializing the database
connection pool
Sep 24 02:33:46 castle Controller[1093]: 02:33:46.303 [Main]
ERROR LINSTOR/Controller - SYSTEM - Database initialization
error [Report number 5F6B78E5-00000-000000]
Sep 24 02:33:46 castle Controller[1093]: 02:33:46.343
[Thread-2] INFO LINSTOR/Controller - SYSTEM - Shutdown in
progress
Sep 24 02:33:46 castle Controller[1093]: 02:33:46.344
[Thread-2] INFO LINSTOR/Controller - SYSTEM - Shutting down
service instance 'TaskScheduleService' of type
TaskScheduleService
Sep 24 02:33:46 castle Controller[1093]: 02:33:46.345
[Thread-2] INFO LINSTOR/Controller - SYSTEM - Waiting for
service instance 'TaskScheduleService' to complete shutdown
Sep 24 02:33:46 castle Controller[1093]: 02:33:46.345
[Thread-2] INFO LINSTOR/Controller - SYSTEM - Shutting down
service instance 'DatabaseService' of type DatabaseService
Sep 24 02:33:46 castle Controller[1093]: 02:33:46.349
[Thread-2] INFO LINSTOR/Controller - SYSTEM - Waiting for
service instance 'DatabaseService' to complete shutdown
Sep 24 02:33:46 castle Controller[1093]: 02:33:46.350
[Thread-2] INFO LINSTOR/Controller - SYSTEM - Shutting down
service instance 'TimerEventService' of type TimerEventService
Sep 24 02:33:46 castle Controller[1093]: 02:33:46.351
[Thread-2] INFO LINSTOR/Controller - SYSTEM - Waiting for
service instance 'TimerEventService' to complete shutdown
Sep 24 02:33:46 castle Controller[1093]: 02:33:46.351
[Thread-2] INFO LINSTOR/Controller - SYSTEM - Shutdown complete
Sep 24 02:33:46 castle systemd[1]:
linstor-controller.service: Main process exited, code=exited,
status=20/n/a
Sep 24 02:33:46 castle systemd[1]:
linstor-controller.service: Failed with result 'exit-code'.
Here is
/var/log/linstor-controller/ErrorReport-5F6B78E5-00000-000000.log
ERROR REPORT 5F6B78E5-00000-000000
============================================================
Application: LINBIT® LINSTOR
Module: Controller
Version: 1.9.0
Build ID: 678acd24a8b9b73a735407cd79ca33a5e95eb2e2
Build time: 2020-09-23T10:27:49+00:00
Error time: 2020-09-24 02:33:46
Node: castle
============================================================
Reported error:
===============
Description:
Database initialization error
Category: LinStorException
Class name: SystemServiceStartException
Class canonical name: com.linbit.SystemServiceStartException
Generated at: Method 'initialize',
Source file 'DbConnectionPoolInitializer.java', Line #56
Error message: Database initialization error
Call backtrace:
Method Native Class:Line number
initialize N
com.linbit.linstor.dbcp.DbConnectionPoolInitializer:56
startSystemServices N
com.linbit.linstor.core.ApplicationLifecycleManager:87
start N com.linbit.linstor.core.Controller:337
main N com.linbit.linstor.core.Controller:556
Caused by:
==========
Category: RuntimeException
Class name: LinStorDBRuntimeException
Class canonical name:
com.linbit.linstor.LinStorDBRuntimeException
Generated at: Method
'setTransactionIsolation', Source file
'DbConnectionPool.java', Line #485
Error message: Failed to set transaction
isolation
Call backtrace:
Method Native Class:Line number
setTransactionIsolation N
com.linbit.linstor.dbcp.DbConnectionPool:485
migrate N com.linbit.linstor.dbcp.DbConnectionPool:202
initialize N
com.linbit.linstor.dbcp.DbConnectionPoolInitializer:50
startSystemServices N
com.linbit.linstor.core.ApplicationLifecycleManager:87
start N com.linbit.linstor.core.Controller:337
main N com.linbit.linstor.core.Controller:556
Caused by:
==========
Category: Exception
Class name: JdbcSQLException
Class canonical name: org.h2.jdbc.JdbcSQLException
Generated at: Method
'getJdbcSQLException', Source file 'DbException.java', Line #357
Error message: Wrong user name or
password [28000-197]
Call backtrace:
Method Native Class:Line number
getJdbcSQLException N org.h2.message.DbException:357
get N org.h2.message.DbException:179
get N org.h2.message.DbException:155
get N org.h2.message.DbException:144
validateUserAndPassword N org.h2.engine.Engine:341
createSessionAndValidate N org.h2.engine.Engine:165
createSession N org.h2.engine.Engine:140
createSession N org.h2.engine.Engine:28
connectEmbeddedOrServer N
org.h2.engine.SessionRemote:351
<init> N org.h2.jdbc.JdbcConnection:124
<init> N org.h2.jdbc.JdbcConnection:103
connect N org.h2.Driver:69
getConnection N java.sql.DriverManager:677
getConnection N java.sql.DriverManager:189
createConnection N
org.apache.commons.dbcp2.DriverManagerConnectionFactory:123
makeObject N
org.apache.commons.dbcp2.PoolableConnectionFactory:355
create N org.apache.commons.pool2.impl.GenericObjectPool:889
borrowObject N
org.apache.commons.pool2.impl.GenericObjectPool:424
borrowObject N
org.apache.commons.pool2.impl.GenericObjectPool:349
getConnection N
org.apache.commons.dbcp2.PoolingDataSource:134
setTransactionIsolation N
com.linbit.linstor.dbcp.DbConnectionPool:477
migrate N com.linbit.linstor.dbcp.DbConnectionPool:202
initialize N
com.linbit.linstor.dbcp.DbConnectionPoolInitializer:50
startSystemServices N
com.linbit.linstor.core.ApplicationLifecycleManager:87
start N com.linbit.linstor.core.Controller:337
main N com.linbit.linstor.core.Controller:556
Finally I found another log in
/var/lib/linstor/linstordb.trace.db
2020-09-24 02:33:45 database: wrong user or password; user:
"LINSTOR"
org.h2.message.DbException: Wrong user name or password
[28000-197]
at org.h2.message.DbException.get(DbException.java:179)
at org.h2.message.DbException.get(DbException.java:155)
at org.h2.message.DbException.get(DbException.java:144)
at org.h2.engine.Engine.openSession(Engine.java:107)
at org.h2.engine.Engine.openSession(Engine.java:179)
at
org.h2.engine.Engine.createSessionAndValidate(Engine.java:157)
at org.h2.engine.Engine.createSession(Engine.java:140)
at org.h2.engine.Engine.createSession(Engine.java:28)
at
org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:351)
at
org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:124)
at
org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:103)
at org.h2.Driver.connect(Driver.java:69)
at
java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
at
java.sql/java.sql.DriverManager.getConnection(DriverManager.java:189)
at
org.apache.commons.dbcp2.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:123)
at
org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:355)
at
org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:889)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:424)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:349)
at
org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
at
com.linbit.linstor.dbcp.DbConnectionPool.setTransactionIsolation(DbConnectionPool.java:477)
at
com.linbit.linstor.dbcp.DbConnectionPool.migrate(DbConnectionPool.java:202)
at
com.linbit.linstor.dbcp.DbConnectionPoolInitializer.initialize(DbConnectionPoolInitializer.java:50)
at
com.linbit.linstor.core.ApplicationLifecycleManager.startSystemServices(ApplicationLifecycleManager.java:87)
at
com.linbit.linstor.core.Controller.start(Controller.java:337)
at
com.linbit.linstor.core.Controller.main(Controller.java:556)
Caused by: org.h2.jdbc.JdbcSQLException: Wrong user name or
password [28000-197]
at
org.h2.message.DbException.getJdbcSQLException(DbException.java:357)
... 26 more
Luckily, this is just a test environment, but I'd like to
know if I messed up in my upgrade process somehow or this is
a bug in the new version. I haven't downgraded or anything yet.
Regards,
Adam
On 23/9/20 22:31, Rene Peinthor wrote:
Hi!
Here is a new minor Linstor release, this release contains
more bug fixes than new features.
Notable features are a new auto-unplace option, to reduce
the replica-count of resources
to the value set in the resource-group after a certain time.
And error-reports can now be deleted
from the REST-API (client).
linstor-server 1.9.0
--------------------
* Added auto-"unplace", automatically removes resources if
they exceed the replica count if `DrbdOptions/auto-unplace`
is set
* Added error-report deletion API
* Expose DRBD promotion_score and may_promote via REST-API
and server sent events
* Prometheus metrics are now mostly cached and much faster
to query
* Send SD_NOTIFY READY to systemd if services are ready
* OpenFlex resource are now correctly limited to 32 characters
* Fix etcd modify net-interface (incorrect serialization)
* Fix autoplacer also toggling diskless resources
* Fix snapshot-shipping with more than 1 volume definitions
* Fix zfs create options handling
* Fixed recaching of lvm-filter
* Fix missing client answer if drbd device doesn't become
ready in reasonable time
* REST-API v1.4.0
https://www.linbit.com/downloads/linstor/linstor-server-1.9.0.tar.gz
<https://www.linbit.com/downloads/linstor/linstor-server-1.9.0.tar.gz>
Linstor PPA:
https://launchpad.net/~linbit/+archive/ubuntu/linbit-drbd9-stack
<https://launchpad.net/~linbit/+archive/ubuntu/linbit-drbd9-stack>
Cheers,
Rene
_______________________________________________
Star us on GITHUB:https://github.com/LINBIT <https://github.com/LINBIT>
drbd-user mailing list
[email protected] <mailto:[email protected]>
https://lists.linbit.com/mailman/listinfo/drbd-user
<https://lists.linbit.com/mailman/listinfo/drbd-user>
_______________________________________________
Star us on GITHUB: https://github.com/LINBIT
<https://github.com/LINBIT>
drbd-user mailing list
[email protected] <mailto:[email protected]>
https://lists.linbit.com/mailman/listinfo/drbd-user
<https://lists.linbit.com/mailman/listinfo/drbd-user>
_______________________________________________
Star us on GITHUB: https://github.com/LINBIT
drbd-user mailing list
[email protected]
https://lists.linbit.com/mailman/listinfo/drbd-user
