this is a proposal for a name service for etch. there is a text document
describing the
name service and rationale, and a proposed etch service idl.
the intention is to define the service interface, implement an etch
scheme which uses it,
and produce a reference implementation of the service which can support
simple
deployments and also test the etch scheme.
i will also post this to the wiki. please review and comment here.
scott out
Introduction
When an client is connecting to an etch service, a uri is used to specify the
details of the connection. Typically this code looks something like this:
RemoteNameServiceServer server = NameServiceHelper.newServer(
"tcp://host:4001?filter=KeepAlive&TcpTransport.reconnectDelay=4000",
null, "factory" );
The uri specifies all the details of the etch connection in a convenient form.
Yet the uri is still hard to manage. With it embedded in code we must recompile
the client to change the connection details. We can load the uri from some
other place, say a configuration file, environment variable, or the command
line, but all we've done is move the problem to yet another hard to manage
place. (Here 'manage' specifically refers to the process of obtaining the uri or
updating it after a change.)
What can we do to make the uri easier to manage? The main issue is that the
service implementation and deployment specifies what the uri should be. Any
change to the service implementation or deployment could trigger a need to
update the uri that clients use to connect to it. So, one fact (connection uri),
one source (service implementation and deployment), but a delivery path which is
not automatic and often includes a human. Can you imagine trying to update
30,000 clients with a new connection uri?
The classic solution applies here and represents the start of the next phase in
etch development. Etch needs a Name Service.
Basics
At its heart a name service is easy. We use them every day without really
thinking about them. Given a simple but abstract identifier a more complicated
but concrete identifier is produced by some sort of lookup. In exchange for the
abstraction, we obtain some independence from the complicated details of the
concrete identifier, allowing the concrete identifier to change as needed by
the environment. For example, apache.org is translated into the internet
protocol v4 address 140.211.11.130 for us by dns. My name, sccomer, is
translated into a uid (1079) by my nearby linux box and used for all sorts of
evil purposes. In both cases I can use the simple name as a substitute for the
more complicated name seamlessly in the environments I work in. Let's call the
abstract identifier the source. Let's call the concrete identifier the
target.
In order for the name service to be useful to me, I have to be able to depend
upon and trust it. That is, the translation process needs to be available when
I need it, and the translations need to be accurate and, most importantly,
secure. By secure I mean that if I'm going to connect to a service and supply
some credentials and use it to do my work, I have an expectation that the
service itself is reliable and trustworthy. That it is not being spoofed. This
translates into access controls limiting who can make changes to the underlying
data used to implement the name service translation.
Requirements
The name service should be defined using etch. Perhaps this is obvious but I'll
say it here for completeness.
The source should be specified in a uri. This allows us to exchange the target
uri for a source uri + name service api to achieve our goal.
To implement the functionality of the name service we need to have these
elements:
1) an api to access the name translations, update them, etc.
2) a mechanism to protect the name service api from unauthorized access.
3) an etch connection scheme which uses the name service api to automate the
translation process for the client.
First the api elements:
1a) format for source.
1b) a method to translate source to target.
1c) a method to list translations matching a query string.
1d) a method to add translation of a source.
1e) a method to remove translation of a source.
1f) a method to associate additional qualities with a particular translation,
which might be used to further qualify the translation for a particular
purpose.
In order to implement replicated name services, we need an ability to receive
notification of changes to translations at a name service instance:
1f) a method to subscribe to changes.
1g) a method to unsubscribe to changes.
1h) a method to receive notification of changes.
1i) a method to differentiate translations from primary or secondary sources.
These mechanisms protect access:
2a) methods that read need to check authorization for reading (1b, 1c, 1f).
2b) methods that update need to check authorization for updating (1d, 1e).
Etch client access to name service:
3a) Etch clients use a uri to specify etch connection parameters, so it is
natural that the way to specify a source requiring name service translation
would be with a uri which uses an alternate scheme along with the requested
source. Some elements of this source uri must be copied over to the
resulting target uri after translation, perhaps overriding elements there.
Source Format
Within a given name service database may be many entries offering the same
(or, essentially the same) api, that is, etch service name (e.g.,
etch.examples.perf.Perf). This is just as it is for any other service
available over the network (nfs file servers, smtp mail servers, jabber im
servers, etc.). This suggests that the api might be useful as an organizational
concept for the name service database.
Suppose we partition the name space into domains based on the api being offered.
Within a given api domain, there may be a number of named instances. The
instance name is used to uniquely identify a running instance of the service
offering the api. There may be several ways to access an instance (called
schemes in etch). The instance might offer tls and soap schemes, for example.
Finally, a given scheme may be available in more than one flavor, depending
upon the capabilities of the client. The preferred scheme should be tried first,
and if it isn't supported a second scheme is tried, third, etc.
This suggests a four part name: api, instance, scheme, and priority. We can
combine the four parts into a single name by using the slash character as a
separator. This gives the name a path-like quality and also allows it to be
easily embedded in a uri:
api/instance/scheme/priority
One obvious way to express api is to use the fully qualified service name of the
etch idl. These names are composed of standard identifiers separated by periods
(e.g., etch.examples.perf.Perf).
The instance name should not contain the slash character. Since we want to embed
these in a uri, the instance name should not contain any other uri significant
characters either. If we stick to the same format as the api, we still have a
large and interesting name space to work with.
Since the scheme corresponds to a uri scheme name, then the same uri scheme
syntax is required. This is pretty much a standard identifier.
Priority is most easily a positive non-zero integer, with 1 being highest
priority.
So, here is a fully specified source for the Perf service named foo with tcp
scheme:
etch.examples.perf.Perf/foo/tcp/1
Since the sequence number is not often needed, it could be defaulted to 1 if
missing. This would be the same as the previous:
etch.examples.perf.Perf/foo/tcp
Name Service Api Details
Please see the accompanying ns.etch file for specific api details and
documentation.
Etch Scheme
An etch scheme is introduced to gain automatic access to the name service. This
removes from most clients any burden relating to name service, and makes using
name server essentially transparent (i.e., no program changes are required).
Here's an example etch scheme:
etch://etch.examples.perf.Perf/foo/tcp
This would connect to a name service, lookup the source, and then connect to
the returned target. The highest priority entry matching the other three
parameters would be used.
Here is another example, this time with modification to the target uri:
etch://etch.examples.perf.Perf/foo/tcp?add:filter=Logger
Logger would be added to the target filter chain.
Less specific source specifications might be used to request alternate
capabilities. Here we request a connection, tls would be preferred but we will
accept tcp:
etch://etch.examples.perf.Perf/foo/tls,tcp
Here we request the nearest service to 78749 within 100 miles:
etch://etch.examples.perf.Perf/./tcp?distance_le=100mi&zip=78749
(to implement this, the name service would need to have location of
services and some sort of function calculating distance based on zip codes.)
Q/A:
What other target uri edits can you make?
This is all kinda twitchy and not well-baked. Speculation. The problem
is,
without edits there is clearly a gap in capability. But edits are messy
and too many puts us back to first base.
set:option=value (adds/replaces the query term option=value)
del:option (deletes the query term option)
rem:option=value (removes value from option's comma separated list)
Which name service is used?
The name service might be configured in a number of ways:
Environment varible
Via resources
Dhcp option (configures all clients within a subnet)
Can name service be partitioned or federated?
Yes. The instance could have structure which suggests a hierarchy of
name
services serving various domains. A request to lookup or register an
instance not in the local domain could be routed to another name
service for
processing.
Can name service be replicated?
Yes. Using the subscribe feature, one name service could replicate the
contents of another (in a publisher/subscriber fashion). Lookup could be
satisfied locally, while register could flow through to the publisher or
the relationship could be more symmetric.
If I connect to a service using the uri "etch://Foo/bar/tcp" and the connection
goes down, how do I reconnect?
This is tricky. I don't know yet. Originally I was thinking that the
name
would be looked up and then the connection made to the first target
which
worked. That is, the actual transport stack would reflect the matched
target.
But if the target uri for that name changes, or a secondary translation
is
used, then the bottom layers of the current stack would have to be
discarded and rebuilt from the new target uri.
module etch.services.ns
/**
* The NameService provides translation from an abstract name for a service to a
* uri that may be used to contact the service. A given service may offer
* several different connection schemes, and a client may only support a subset
* of those. So, when a client wishes to connect to a service, it may query
* the NameService with both abstract name and desired schemes in order to find
* a suitable match.
*
* Some example queries:
*
* All services with servicename='Foo' and instancename='bar' and scheme='tcp'
*
* serviceUri='Foo/bar/tcp'
*
* Same as the query above, but written out longhand:
*
* servicename='Foo' and instancename='bar' and scheme='tcp'
*
* Service Foo instance bar with tls or tcp (prefer tls):
*
* serviceUri='Foo/bar/tls,tcp'
*
* or:
*
* servicename='Foo' and instancename='bar' and scheme='tls,tcp'
*
* Fully qualified serviceUri:
*
* serviceUri='Foo/bar/tcp/1'
*
* Any instance of service Foo located in Austin, TX:
*
* servicename='Foo' and qualities.location='Austin, TX'
*
* Instances of service Foo with capacity >= 400:
*
* servicename='Foo' and qualities.capacity>=400
*
* Relative operations on qualities require that the quality be present and be
* comparable (with mixed types some type promotion is supported). But you might
* be surprised, qualities which are absent will give a negative comparison no
* matter which sense is used: both qualities.x > 50 and qualities.x < 50 are
* false whenever qualities.x is null. Consider this test: not qualities.x < 50.
* This is not the same as qualities.x >= 50. It is the same as qualities.x =
* null or qualities.x >= 50.
*/
@Timeout( 30000 )
service NameService
{
/** An entry describing a service. */
struct Entry
(
/** A service description uri, composed of
* servicename/instancename/scheme[/priority]. */
string serviceUri,
/** Qualities of this service instance. */
Map qualities,
/** Etch service connection uri, for example
* tcp://localhost:9000?filter=KeepAlive */
string targetUri,
/** Lifetime in seconds from last update. */
int ttl,
/** Who created or last update. */
string who,
/** Date / time of create or last update. */
Datetime lastUpdate,
/** Flag indicating entry has been removed. */
boolean removed
)
/**
*
*/
@Authorize( canLookup, source )
Entry lookup( string source )
/**
* Looks up entries by matching them against the query string. Entries
are
* returned in a stable and consistent order, increasing alphabetical by
* servicename, instancename, and scheme, and then increasing numerical
by
* priority. If a scheme search term appears with a list, schemes are
* returned in the specified order (e.g., scheme='tls,tcp').
*
* @param query a sql-like query expression using the elements of the
* serviceUri, targetUri, and qualities. If query is null or the empty
* string, all entries are matched.
*
* @param offset offset in the list of matched items of the first item
to
* return. This parameter and the count parameter are used to step
through
* the result set when there are many results.
*
* @param count number of items to return.
*
* @return entries matching query. If you requested 10 items and 10 are
* returned, there could be more. To get the next batch, add
results.length
* to offset and call find again:
*
* int i = 0;
* int n = 10;
* Entry[] results;
* while ((results = service.find( null, i, n )).length > 0)
* {
* for (Entry e: results)
* processEntry( e );
* if (results.length < n)
* break;
* i += results.length;
* }
*/
@Authorize( canFind, query )
Entry[] find( string query, int offset, int count )
/**
* Adds or updates the specified entry. The given parameters replace any
* existing values in an Entry whose key is serviceUri, whereas who and
* lastUpdate are set to the current user and current date / time,
* respectively.
*
* @param serviceUri the uri describing the service. The uri should be
of
* the form "servicename/instancename/scheme[/priority]", where
servicename
* is a valid fully qualified service name (e.g.,
* etch.services.ns.NameService"), instancename is a valid etch
identifier
* (e.g, fred, alice01), scheme is a valid uri scheme (e.g., tcp, tls),
and
* priority is an integer >= 1. If priority is omitted, it is defaulted
to
* 1.
*
* @param qualities a map which may be used to describe additional
features
* of the entry, such as purpose, licenses, capacity, location, owner,
* whatever. Query strings may test values of qualities using a variety
of
* sql-like operators. Qualities may be null.
*
* @param targetUri the uri describing the contact information for the
* service.
*
* @param ttl the lifetime of the entry specified as seconds. 0 means
* forever, -1 means remove immediately when the connection to the
* NameService is dropped.
*/
@Authorize( canRegister, serviceUri, qualities )
void register( string serviceUri, Map qualities, string targetUri, int
ttl )
/**
* Registers a number of entries all in one operation. Identical to
calling
* register with each entry in turn.
*
* @param entries a sequence of Entry records with serviceUri,
targetUri,
* qualities, and ttl as specified in register() above. Who and
lastUpdate
* fields are ignored.
*/
@Authorize( canRegisterBulk, entries )
void registerBulk( Entry[] entries )
/**
* Removes the specified entry.
*
* @param serviceUri the uri describing the service.
*/
@Authorize( canRegister, serviceUri, null )
void unregister( string serviceUri )
/**
* Adds a request for notification of changes to entries matching the
query.
* The current value of all matching entries is delivered via
entryChanged
* client message, as well as any updates or new entries.
*
* @param query a sql-like query expression using the elements of the
* serviceUri, targetUri, and qualities. If query is null or the empty
* string, all entries are matched.
*/
@Authorize( canFind, query )
void subscribe( string query )
/**
* Removes a request for notification of changes to entries matching the
* query.
*
* @param query a query previously passed to subscribe. The string must
* match exactly.
*/
void unsubscribe( string query )
/**
* Removes all requests for notification of changes to entries. This
* operation is implicitly performed when the connection to the
NameService
* is dropped.
*/
void unsubscribeAll()
////////////////////
// RIGHTS TESTING //
////////////////////
/**
* Tests whether the current user is authorized to lookup the source.
* @param source the complete specification api/instance/scheme.
*/
boolean canLookup( string source )
/**
* Tests whether the current user is authorized to run the query.
*
* @param query a sql-like query expression using the elements of the
* serviceUri, targetUri, and qualities. If query is null or the empty
* string, all entries are matched.
*
* @return true if the current user is authorized to run the query.
*/
boolean canFind( string query )
/**
* Tests whether the current user is authorized to register the service.
*
* @param serviceUri the uri describing the service.
*
* @param qualities a map which may be used to describe additional
features
* of the entry.
*
* @return true if the current user is authorized to register the
service.
*/
boolean canRegister( string serviceUri, Map qualities )
/**
* Tests whether the current user is authorized to register the entries.
* This is the same as:
*
* for (Entry entry: entries)
* if (!canRegister( entry.serviceUri, entry.qualities ))
* return false;
* return true;
*
* @param entries a sequence of Entry records with serviceUri,
targetUri,
* qualities, and ttl as specified in register() above. Who and
lastUpdate
* fields are ignored.
*
* @return true if the current user is authorized to register the
entries.
* This is an all or nothing proposition.
*/
boolean canRegisterBulk( Entry[] entries )
////////////////////
// CLIENT METHODS //
////////////////////
/**
* Notifies the client of a change in an entry. The entry might have
been
* created, updated, or removed.
*
* Note: while this might have normally been an event, we made it a call
* to slow down the processing of what might otherwise be a rather large
* change set.
*
* Note: when keeping track of entries, always keep the one with the
latest
* lastUpdate.
*
* @param query the query which triggered the notification.
*
* @param entry the entry which has changed.
*/
@Direction( client )
void entryChanged( string query, Entry entry )
}
