I'm sponsoring this fast-track for Huafeng Lu and the Sun Cluster
team. The timer is set to 11/17/2008, and a draft of the contract is
available in the case directory as "contract-01".
Note: the fast-track materials were presented to me as "open," though
many of the cases related to these hooks are currently closed. The
change in the system that is the subject of this case, though, should
be reviewable independently, so this shouldn't cause problems.
Introduction
"TCP/IP hooks for clustering" (PSARC 1997/314) introduced several
TCP/IP hooks for the Sun Cluster product, and the contract is
1997/314:contract-1.new. These hooks are called by the Solaris
TCP/IP code at certain places. Now, to support client-side shared
address (RFE 6717519), we need to modify one of these hooks,
cl_inet_connect().
Release Binding
A patch/micro release binding is asserted. This will require the
release of coordinated patches for Solaris and Sun Cluster.
Interfaces
Exported Interface Stability Comments
================== ========= ========
cl_inet_connect Removed (was Contracted) PSARC 1997/314
cl_inet_connect2 Contr. Project Private function pointer
cluster_version_supported Contr. Project Private cluster version
supported by Solaris
udp_t.udp_cl_v6lastdst Project Private most recent dest addr
udp_t.udp_cl_lastdstport Project Private most recent dest port
Detailed Description
Currently, a shared address is used only at the server side. Server
applications are started on Cluster nodes, which reply to requests
coming from external clients. This proposal (RFE 6717519) aims to
allow Cluster nodes to use shared addresses as client addresses to
talk to external servers. Note: this proposal only handles TCP and
UDP; SCTP is beyond its scope.
For TCP, several Cluster nodes bind to the same shared address, then
connect to external applications. The cl_inet_connect() hook is
called at connect time (for both outgoing and incoming connection
requests) to check if the 5-tuple is unique. The hook is added with
a return value to indicate whether the check is passed, and an input
parameter to indicate whether the connection is outgoing or
incoming.
The situation for UDP is a bit different. Since there is no real
connection in UDP and the same socket can be used to send to
multiple destinations, the major task is to register the five-tuple
at the cluster level. Such registration is performed at connect(),
sendto() and sendmsg(), which call the cl_inet_connect() hook at
proper time. For performance reasons, sendto()/sendmsg() calls the
hook only when the destination is new. To facilitate this, two new
fields are added to the udp_t struct to store the most recent
destination address and port: in6_addr_t udp_cl_v6lastdst; in_port_t
udp_cl_lastdstport; In a sendto() or sendmsg(), the
cl_inet_connect() hook is called only when the destination is
different from most recent one.
Versioning
In addition to the hook signature changes listed above, the hook
name will also be changed for versioning reasons. Thus, overall the
new signature of the hook is:
int (*cl_inet_connect2)(uint8_t protocol,
boolean_t is_outgoing, sa_family_t addr_family,
uint8_t *laddrp, in_port_t lport,
uint8_t *faddrp, in_port_t fport);
Here's the old signature for comparison. Note: after renaming, the
old name cl_inet_connect is obsolete and will no longer be called
from Solaris.
void (*cl_inet_connect)(uint8_t protocol,
sa_family_t addr_family,
uint8_t *laddrp, in_port_t lport,
uint8_t *faddrp, in_port_t fport);
With changes at both Solaris and Cluster sides, there is a "version
matching" issue. There is no mechanism to check if the Cluster
version matches the Solaris release; the Cluster simply associate
the hook with an implementation, if the hook is defined in Solaris.
Thus, if the Cluster product is not running with the correct Solaris
release, the hook will be associated with the wrong implementation,
and may cause serious problems or even panic.
To solve this problem, the function pointer is renamed as above. To
help with future versioning, a new version string is introduced in
Solaris code:
const char *cluster_version_supported = "1.1";
The new Cluster code checks which Cluster version Solaris supports,
and associates the corresponding hook implementation with the hook.
This is a generic mechanism, but doesn't work for the currently
released Cluster code, which doesn't check the version. As a
workaround, we rename the hook in Solaris so that the current
Cluster version won't be loaded after upgrading ON. This prevents
the user from damaging his system, and release notes will indicate
what the errors mean.
Out of Scope
Note the actual implementation of these hooks are in the Cluster
code, not in the Solaris code. Solaris TCP/IP code only declares
and calls them. For this project, the Cluster team is responsible
for changing the implementation of the cl_inet_connect2() hook in
the Sun Cluster code.
--
James Carlson, Solaris Networking <james.d.carlson at sun.com>
Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677
