Re: Allow to connect on a CLI during a soft-stop or a reload
On Fri, Nov 16, 2018 at 05:07:41PM +0100, Willy Tarreau wrote: > On Fri, Nov 16, 2018 at 04:57:19PM +0100, William Lallemand wrote: > > This patch series allows you to connect to a leaving process from the > > master CLI, which might be convenient for debugging purpose :-) > > Now merged, thank you. However when I try it I get an error message > indicating that it cannot find the socket, but I suppose this is > normal at this stage : > > master> show proc > # > 3414 master 0 -1 > 3441 worker 1 0 > 3437 worker 1 1 > > master> @-1 show info > Can't find the target CLI! > > Then it quits. > > Thanks, > Willy > Hi, That's because you should have done `@!3437 show info` -1 is just the number of reloads of the master which is not incremented (yet). -- William Lallemand
Re: Allow to connect on a CLI during a soft-stop or a reload
On Fri, Nov 16, 2018 at 04:57:19PM +0100, William Lallemand wrote: > This patch series allows you to connect to a leaving process from the > master CLI, which might be convenient for debugging purpose :-) Now merged, thank you. However when I try it I get an error message indicating that it cannot find the socket, but I suppose this is normal at this stage : master> show proc # 3414 master 0 -1 3441 worker 1 0 3437 worker 1 1 master> @-1 show info Can't find the target CLI! Then it quits. Thanks, Willy
[PATCH 3/3] MEDIUM: cli: worker socketpair is unstoppable
In master-worker mode, the socketpair CLI listener of the worker is now marked unstoppable, which allows to connect to the CLI of an old process which is in a leaving state, allowing to debug it. --- src/cli.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/cli.c b/src/cli.c index 85e344d30..720a9b451 100644 --- a/src/cli.c +++ b/src/cli.c @@ -2336,7 +2336,7 @@ int mworker_cli_sockpair_new(struct mworker_proc *mworker_proc, int proc) l->backlog = global.stats_fe->backlog; l->accept = session_accept_fd; l->default_target = global.stats_fe->default_target; - l->options |= LI_O_UNLIMITED; + l->options |= (LI_O_UNLIMITED | LI_O_NOSTOP); /* it's a sockpair but we don't want to keep the fd in the master */ l->options &= ~LI_O_INHERITED; l->nice = -64; /* we want to boost priority for local stats */ -- 2.18.1
[PATCH 1/3] MEDIUM: jobs: support unstoppable jobs for soft stop
This patch allows a process to properly quit when some jobs are still
active, this feature is handled by the unstoppable_jobs variable, which
must be atomically incremented.
During each new iteration of run_poll_loop() the break condition of the
loop is now (jobs - unstoppable_jobs) == 0.
The unique usage of this at the moment is to handle the socketpair CLI
of a the worker during the stopping of the process. During the soft
stop, we could mark the CLI listener as an unstoppable job and still
handle new connections till every other jobs are stopped.
---
include/types/global.h | 1 +
include/types/stats.h | 1 +
src/haproxy.c | 3 ++-
src/stats.c| 2 ++
4 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/include/types/global.h b/include/types/global.h
index 344003a91..521032304 100644
--- a/include/types/global.h
+++ b/include/types/global.h
@@ -224,6 +224,7 @@ extern unsigned long pid_bit; /* bit corresponding to the
process id */
extern int actconn;/* # of active sessions */
extern int listeners;
extern int jobs; /* # of active jobs (listeners, sessions, open
devices) */
+extern int unstoppable_jobs; /* # of active jobs that can't be stopped
during a soft stop */
extern int active_peers; /* # of active peers (connection attempts and
successes) */
extern int connected_peers;/* # of really connected peers */
extern THREAD_LOCAL struct buffer trash;
diff --git a/include/types/stats.h b/include/types/stats.h
index 75250447e..85cc906da 100644
--- a/include/types/stats.h
+++ b/include/types/stats.h
@@ -290,6 +290,7 @@ enum info_field {
INF_DESCRIPTION,
INF_STOPPING,
INF_JOBS,
+ INF_UNSTOPPABLE_JOBS,
INF_LISTENERS,
INF_ACTIVE_PEERS,
INF_CONNECTED_PEERS,
diff --git a/src/haproxy.c b/src/haproxy.c
index 494160ad8..f066ec978 100644
--- a/src/haproxy.c
+++ b/src/haproxy.c
@@ -177,6 +177,7 @@ struct activity activity[MAX_THREADS]
__attribute__((aligned(64))) = { };
int stopping; /* non zero means stopping in progress */
int killed;/* non zero means a hard-stop is triggered */
int jobs = 0; /* number of active jobs (conns, listeners, active tasks, ...)
*/
+int unstoppable_jobs = 0; /* number of active jobs that can't be stopped
during a soft stop */
int active_peers = 0; /* number of active peers (connection attempts and
connected) */
int connected_peers = 0; /* number of connected peers (verified ones) */
@@ -2617,7 +2618,7 @@ static void run_poll_loop()
next = wake_expired_tasks();
/* stop when there's nothing left to do */
- if (jobs == 0)
+ if ((jobs - unstoppable_jobs) == 0)
break;
/* expire immediately if events are pending */
diff --git a/src/stats.c b/src/stats.c
index 07a5ef74e..0574260d9 100644
--- a/src/stats.c
+++ b/src/stats.c
@@ -132,6 +132,7 @@ const char *info_field_names[INF_TOTAL_FIELDS] = {
[INF_DESCRIPTION]= "description",
[INF_STOPPING] = "Stopping",
[INF_JOBS] = "Jobs",
+ [INF_UNSTOPPABLE_JOBS] = "Unstoppable Jobs",
[INF_LISTENERS] = "Listeners",
[INF_ACTIVE_PEERS] = "ActivePeers",
[INF_CONNECTED_PEERS]= "ConnectedPeers",
@@ -3300,6 +3301,7 @@ int stats_fill_info(struct field *info, int len)
info[INF_DESCRIPTION]=
mkf_str(FO_CONFIG|FN_OUTPUT|FS_SERVICE, global.desc);
info[INF_STOPPING] = mkf_u32(0, stopping);
info[INF_JOBS] = mkf_u32(0, jobs);
+ info[INF_UNSTOPPABLE_JOBS] = mkf_u32(0, unstoppable_jobs);
info[INF_LISTENERS] = mkf_u32(0, listeners);
info[INF_ACTIVE_PEERS] = mkf_u32(0, active_peers);
info[INF_CONNECTED_PEERS]= mkf_u32(0, connected_peers);
--
2.18.1
Allow to connect on a CLI during a soft-stop or a reload
This patch series allows you to connect to a leaving process from the master CLI, which might be convenient for debugging purpose :-)
[PATCH 2/3] MEDIUM: listeners: support unstoppable listener
An unstoppable listener is a listener which won't be stop during a soft
stop. The unstoppable_jobs variable is incremented and the listener
won't prevent the process to leave properly.
It is not a good idea to use this feature (the LI_O_NOSTOP flag) with a
listener that need to be bind again on another process during a soft
reload.
---
include/types/listener.h | 1 +
src/proxy.c | 11 +--
2 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/include/types/listener.h b/include/types/listener.h
index 50f193684..eac2037f4 100644
--- a/include/types/listener.h
+++ b/include/types/listener.h
@@ -101,6 +101,7 @@ enum li_state {
#define LI_O_ACC_CIP0x1000 /* find the proxied address in the
NetScaler Client IP header */
#define LI_O_INHERITED 0x2000 /* inherited FD from the parent
process (fd@) */
#define LI_O_MWORKER0x4000 /* keep the FD open in the master but
close it in the children */
+#define LI_O_NOSTOP 0x8000 /* keep the listener active even after
a soft stop */
/* Note: if a listener uses LI_O_UNLIMITED, it is highly recommended that it
adds its own
* maxconn setting to the global.maxsock value so that its resources are
reserved.
diff --git a/src/proxy.c b/src/proxy.c
index f3208ef20..763b95de3 100644
--- a/src/proxy.c
+++ b/src/proxy.c
@@ -1110,20 +1110,27 @@ void zombify_proxy(struct proxy *p)
* This function completely stops a proxy and releases its listeners. It has
* to be called when going down in order to release the ports so that another
* process may bind to them. It must also be called on disabled proxies at the
- * end of start-up. When all listeners are closed, the proxy is set to the
+ * end of start-up. If all listeners are closed, the proxy is set to the
* PR_STSTOPPED state.
*/
void stop_proxy(struct proxy *p)
{
struct listener *l;
+ int nostop = 0;
list_for_each_entry(l, >conf.listeners, by_fe) {
+ if (l->options & LI_O_NOSTOP) {
+ HA_ATOMIC_ADD(_jobs, 1);
+ nostop = 1;
+ continue;
+ }
unbind_listener(l);
if (l->state >= LI_ASSIGNED) {
delete_listener(l);
}
}
- p->state = PR_STSTOPPED;
+ if (!nostop)
+ p->state = PR_STSTOPPED;
}
/* This function resumes listening on the specified proxy. It scans all of its
--
2.18.1
