-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31239/
-----------------------------------------------------------
Review request for Ambari, Nate Cole and Tom Beerbower.
Bugs: AMBARI-9736
https://issues.apache.org/jira/browse/AMBARI-9736
Repository: ambari
Description
-------
When using MySQL, Ambari appears to try to use invalid connections about about
8 hours of use.
EclipseLink does not appear to correctly handle connections that have been
invalidated by MySQL after the default timeout time of 8 hours.
STR:
Set MySQL's timeouts via /etc/my.cnf
[mysqld]
interactive_timeout=120
wait_timeout=120
Verify the connection timeouts are lowered
mysql> SHOW VARIABLES LIKE "%timeout%";
+----------------------------+----------+
| Variable_name | Value |
+----------------------------+----------+
| connect_timeout | 10 |
| delayed_insert_timeout | 300 |
| innodb_lock_wait_timeout | 50 |
| innodb_rollback_on_timeout | OFF |
| interactive_timeout | 120 |
| lock_wait_timeout | 31536000 |
| net_read_timeout | 30 |
| net_write_timeout | 60 |
| slave_net_timeout | 3600 |
| wait_timeout | 120 |
+----------------------------+----------+
10 rows in set (0.00 sec)
Launch Ambari and wait a few minutes. The exceptions will begin and Ambari will
stop working properly.
The fix here was to use c3p0 (already included via Quartz) instead of
EclipseLink's internal pool management. Since we're so close to release, I felt
that only changing mysql was the right option, with the ability to instruct
Ambari to use c3p0 for other databases if needed.
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
495c1e4
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
3b59fed
ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java
85ae85b
ambari-server/src/main/java/org/apache/ambari/server/orm/EclipseLinkSessionCustomizer.java
PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/scheduler/ExecutionSchedulerImpl.java
324ee27
ambari-server/src/main/java/org/apache/ambari/server/upgrade/AbstractUpgradeCatalog.java
71049b2
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog150.java
2ccf16b
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog151.java
72305c5
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog160.java
cd01779
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog161.java
f5afb46
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog170.java
47f4587
ambari-server/src/main/resources/META-INF/persistence.xml d89f5d5
ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java
0f7de36
Diff: https://reviews.apache.org/r/31239/diff/
Testing
-------
Verification that the issue has been resolved in a reproduction environment.
Also, verification that Ambari's pool stays a consistent size:
mysql> show processlist;
+-----+--------+--------------------+--------+---------+------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info
|
+-----+--------+--------------------+--------+---------+------+-------+------------------+
| 340 | ambari | 192.168.64.1:53549 | ambari | Sleep | 66 | | NULL
|
| 341 | ambari | 192.168.64.1:53557 | ambari | Sleep | 30 | | NULL
|
| 346 | ambari | 192.168.64.1:53675 | ambari | Sleep | 2 | | NULL
|
| 347 | ambari | 192.168.64.1:53676 | ambari | Sleep | 26 | | NULL
|
| 348 | ambari | 192.168.64.1:53677 | ambari | Sleep | 1 | | NULL
|
| 349 | ambari | 192.168.64.1:53727 | ambari | Sleep | 2 | | NULL
|
| 350 | root | localhost | NULL | Query | 0 | NULL | show
processlist |
+-----+--------+--------------------+--------+---------+------+-------+------------------+
7 rows in set (0.00 sec)
New tests written to cover configuration options.
Thanks,
Jonathan Hurley
