A couple of call flow diagram would be useful. Also a re-enforcement of how having the name service defined in terms of an Etch API allows the service to be easily consumed by all Etch-enabled languages.
-- james On Mon, Dec 22, 2008 at 2:53 PM, scott comer (sccomer) <[email protected]> wrote: > this is a proposal for a name service for etch. there is a text document > describing the > name service and rationale, and a proposed etch service idl. > > the intention is to define the service interface, implement an etch scheme > which uses it, > and produce a reference implementation of the service which can support > simple > deployments and also test the etch scheme. > > i will also post this to the wiki. please review and comment here. > > scott out > > > Introduction > > When an client is connecting to an etch service, a uri is used to specify > the > details of the connection. Typically this code looks something like this: > > RemoteNameServiceServer server = NameServiceHelper.newServer( > > "tcp://host:4001?filter=KeepAlive&TcpTransport.reconnectDelay=4000", > null, "factory" ); > > The uri specifies all the details of the etch connection in a convenient > form. > Yet the uri is still hard to manage. With it embedded in code we must > recompile > the client to change the connection details. We can load the uri from some > other place, say a configuration file, environment variable, or the command > line, but all we've done is move the problem to yet another hard to manage > place. (Here 'manage' specifically refers to the process of obtaining the > uri or > updating it after a change.) > > What can we do to make the uri easier to manage? The main issue is that the > service implementation and deployment specifies what the uri should be. Any > change to the service implementation or deployment could trigger a need to > update the uri that clients use to connect to it. So, one fact (connection > uri), > one source (service implementation and deployment), but a delivery path > which is > not automatic and often includes a human. Can you imagine trying to update > 30,000 clients with a new connection uri? > > The classic solution applies here and represents the start of the next phase > in > etch development. Etch needs a Name Service. > > Basics > > At its heart a name service is easy. We use them every day without really > thinking about them. Given a simple but abstract identifier a more > complicated > but concrete identifier is produced by some sort of lookup. In exchange for > the > abstraction, we obtain some independence from the complicated details of the > concrete identifier, allowing the concrete identifier to change as needed by > the environment. For example, apache.org is translated into the internet > protocol v4 address 140.211.11.130 for us by dns. My name, sccomer, is > translated into a uid (1079) by my nearby linux box and used for all sorts > of > evil purposes. In both cases I can use the simple name as a substitute for > the > more complicated name seamlessly in the environments I work in. Let's call > the > abstract identifier the source. Let's call the concrete identifier the > target. > > In order for the name service to be useful to me, I have to be able to > depend > upon and trust it. That is, the translation process needs to be available > when > I need it, and the translations need to be accurate and, most importantly, > secure. By secure I mean that if I'm going to connect to a service and > supply > some credentials and use it to do my work, I have an expectation that the > service itself is reliable and trustworthy. That it is not being spoofed. > This > translates into access controls limiting who can make changes to the > underlying > data used to implement the name service translation. > > Requirements > > The name service should be defined using etch. Perhaps this is obvious but > I'll > say it here for completeness. > > The source should be specified in a uri. This allows us to exchange the > target > uri for a source uri + name service api to achieve our goal. > > To implement the functionality of the name service we need to have these > elements: > > 1) an api to access the name translations, update them, etc. > 2) a mechanism to protect the name service api from unauthorized access. > 3) an etch connection scheme which uses the name service api to automate the > translation process for the client. > > First the api elements: > > 1a) format for source. > 1b) a method to translate source to target. > 1c) a method to list translations matching a query string. > 1d) a method to add translation of a source. > 1e) a method to remove translation of a source. > 1f) a method to associate additional qualities with a particular > translation, > which might be used to further qualify the translation for a particular > purpose. > > In order to implement replicated name services, we need an ability to > receive > notification of changes to translations at a name service instance: > > 1f) a method to subscribe to changes. > 1g) a method to unsubscribe to changes. > 1h) a method to receive notification of changes. > 1i) a method to differentiate translations from primary or secondary > sources. > > These mechanisms protect access: > > 2a) methods that read need to check authorization for reading (1b, 1c, 1f). > 2b) methods that update need to check authorization for updating (1d, 1e). > > Etch client access to name service: > > 3a) Etch clients use a uri to specify etch connection parameters, so it is > natural that the way to specify a source requiring name service > translation > would be with a uri which uses an alternate scheme along with the > requested > source. Some elements of this source uri must be copied over to the > resulting target uri after translation, perhaps overriding elements > there. > > Source Format > > Within a given name service database may be many entries offering the same > (or, essentially the same) api, that is, etch service name (e.g., > etch.examples.perf.Perf). This is just as it is for any other service > available over the network (nfs file servers, smtp mail servers, jabber im > servers, etc.). This suggests that the api might be useful as an > organizational > concept for the name service database. > > Suppose we partition the name space into domains based on the api being > offered. > Within a given api domain, there may be a number of named instances. The > instance name is used to uniquely identify a running instance of the service > offering the api. There may be several ways to access an instance (called > schemes in etch). The instance might offer tls and soap schemes, for > example. > Finally, a given scheme may be available in more than one flavor, depending > upon the capabilities of the client. The preferred scheme should be tried > first, > and if it isn't supported a second scheme is tried, third, etc. > > This suggests a four part name: api, instance, scheme, and priority. We can > combine the four parts into a single name by using the slash character as a > separator. This gives the name a path-like quality and also allows it to be > easily embedded in a uri: > > api/instance/scheme/priority > > One obvious way to express api is to use the fully qualified service name of > the > etch idl. These names are composed of standard identifiers separated by > periods > (e.g., etch.examples.perf.Perf). > > The instance name should not contain the slash character. Since we want to > embed > these in a uri, the instance name should not contain any other uri > significant > characters either. If we stick to the same format as the api, we still have > a > large and interesting name space to work with. > > Since the scheme corresponds to a uri scheme name, then the same uri scheme > syntax is required. This is pretty much a standard identifier. > > Priority is most easily a positive non-zero integer, with 1 being highest > priority. > > So, here is a fully specified source for the Perf service named foo with tcp > scheme: > > etch.examples.perf.Perf/foo/tcp/1 > > Since the sequence number is not often needed, it could be defaulted to 1 if > missing. This would be the same as the previous: > > etch.examples.perf.Perf/foo/tcp > > Name Service Api Details > > Please see the accompanying ns.etch file for specific api details and > documentation. > > Etch Scheme > > An etch scheme is introduced to gain automatic access to the name service. > This > removes from most clients any burden relating to name service, and makes > using > name server essentially transparent (i.e., no program changes are required). > > Here's an example etch scheme: > > etch://etch.examples.perf.Perf/foo/tcp > > This would connect to a name service, lookup the source, and then connect to > the returned target. The highest priority entry matching the other three > parameters would be used. > > Here is another example, this time with modification to the target uri: > > etch://etch.examples.perf.Perf/foo/tcp?add:filter=Logger > > Logger would be added to the target filter chain. > > Less specific source specifications might be used to request alternate > capabilities. Here we request a connection, tls would be preferred but we > will > accept tcp: > > etch://etch.examples.perf.Perf/foo/tls,tcp > > Here we request the nearest service to 78749 within 100 miles: > > etch://etch.examples.perf.Perf/./tcp?distance_le=100mi&zip=78749 > > (to implement this, the name service would need to have location of > services and some sort of function calculating distance based on zip codes.) > > Q/A: > > What other target uri edits can you make? > > This is all kinda twitchy and not well-baked. Speculation. The > problem is, > without edits there is clearly a gap in capability. But edits are > messy > and too many puts us back to first base. > > set:option=value (adds/replaces the query term option=value) > del:option (deletes the query term option) > rem:option=value (removes value from option's comma separated list) > > Which name service is used? > > The name service might be configured in a number of ways: > Environment varible > Via resources > Dhcp option (configures all clients within a subnet) > > Can name service be partitioned or federated? > > Yes. The instance could have structure which suggests a hierarchy of > name > services serving various domains. A request to lookup or register an > instance not in the local domain could be routed to another name > service for > processing. > > Can name service be replicated? > > Yes. Using the subscribe feature, one name service could replicate > the > contents of another (in a publisher/subscriber fashion). Lookup could > be > satisfied locally, while register could flow through to the publisher > or > the relationship could be more symmetric. > > If I connect to a service using the uri "etch://Foo/bar/tcp" and the > connection > goes down, how do I reconnect? > > This is tricky. I don't know yet. Originally I was thinking that the > name > would be looked up and then the connection made to the first target > which > worked. That is, the actual transport stack would reflect the matched > target. > But if the target uri for that name changes, or a secondary > translation is > used, then the bottom layers of the current stack would have to be > discarded and rebuilt from the new target uri. > > module etch.services.ns > > /** > * The NameService provides translation from an abstract name for a service > to a > * uri that may be used to contact the service. A given service may offer > * several different connection schemes, and a client may only support a > subset > * of those. So, when a client wishes to connect to a service, it may query > * the NameService with both abstract name and desired schemes in order to > find > * a suitable match. > * > * Some example queries: > * > * All services with servicename='Foo' and instancename='bar' and > scheme='tcp' > * > * serviceUri='Foo/bar/tcp' > * > * Same as the query above, but written out longhand: > * > * servicename='Foo' and instancename='bar' and scheme='tcp' > * > * Service Foo instance bar with tls or tcp (prefer tls): > * > * serviceUri='Foo/bar/tls,tcp' > * > * or: > * > * servicename='Foo' and instancename='bar' and scheme='tls,tcp' > * > * Fully qualified serviceUri: > * > * serviceUri='Foo/bar/tcp/1' > * > * Any instance of service Foo located in Austin, TX: > * > * servicename='Foo' and qualities.location='Austin, TX' > * > * Instances of service Foo with capacity >= 400: > * > * servicename='Foo' and qualities.capacity>=400 > * > * Relative operations on qualities require that the quality be present and > be > * comparable (with mixed types some type promotion is supported). But you > might > * be surprised, qualities which are absent will give a negative comparison > no > * matter which sense is used: both qualities.x > 50 and qualities.x < 50 > are > * false whenever qualities.x is null. Consider this test: not qualities.x < > 50. > * This is not the same as qualities.x >= 50. It is the same as qualities.x > = > * null or qualities.x >= 50. > */ > @Timeout( 30000 ) > service NameService > { > /** An entry describing a service. */ > struct Entry > ( > /** A service description uri, composed of > * servicename/instancename/scheme[/priority]. */ > string serviceUri, > > /** Qualities of this service instance. */ > Map qualities, > > /** Etch service connection uri, for example > * tcp://localhost:9000?filter=KeepAlive */ > string targetUri, > > /** Lifetime in seconds from last update. */ > int ttl, > > /** Who created or last update. */ > string who, > > /** Date / time of create or last update. */ > Datetime lastUpdate, > > /** Flag indicating entry has been removed. */ > boolean removed > ) > > /** > * > */ > @Authorize( canLookup, source ) > Entry lookup( string source ) > > /** > * Looks up entries by matching them against the query string. > Entries are > * returned in a stable and consistent order, increasing alphabetical > by > * servicename, instancename, and scheme, and then increasing > numerical by > * priority. If a scheme search term appears with a list, schemes are > * returned in the specified order (e.g., scheme='tls,tcp'). > * > * @param query a sql-like query expression using the elements of the > * serviceUri, targetUri, and qualities. If query is null or the > empty > * string, all entries are matched. > * > * @param offset offset in the list of matched items of the first > item to > * return. This parameter and the count parameter are used to step > through > * the result set when there are many results. > * > * @param count number of items to return. > * > * @return entries matching query. If you requested 10 items and 10 > are > * returned, there could be more. To get the next batch, add > results.length > * to offset and call find again: > * > * int i = 0; > * int n = 10; > * Entry[] results; > * while ((results = service.find( null, i, n )).length > 0) > * { > * for (Entry e: results) > * processEntry( e ); > * if (results.length < n) > * break; > * i += results.length; > * } > */ > @Authorize( canFind, query ) > Entry[] find( string query, int offset, int count ) > > /** > * Adds or updates the specified entry. The given parameters replace > any > * existing values in an Entry whose key is serviceUri, whereas who > and > * lastUpdate are set to the current user and current date / time, > * respectively. > * > * @param serviceUri the uri describing the service. The uri should > be of > * the form "servicename/instancename/scheme[/priority]", where > servicename > * is a valid fully qualified service name (e.g., > * etch.services.ns.NameService"), instancename is a valid etch > identifier > * (e.g, fred, alice01), scheme is a valid uri scheme (e.g., tcp, > tls), and > * priority is an integer >= 1. If priority is omitted, it is > defaulted to > * 1. > * > * @param qualities a map which may be used to describe additional > features > * of the entry, such as purpose, licenses, capacity, location, > owner, > * whatever. Query strings may test values of qualities using a > variety of > * sql-like operators. Qualities may be null. > * > * @param targetUri the uri describing the contact information for > the > * service. > * > * @param ttl the lifetime of the entry specified as seconds. 0 means > * forever, -1 means remove immediately when the connection to the > * NameService is dropped. > */ > @Authorize( canRegister, serviceUri, qualities ) > void register( string serviceUri, Map qualities, string targetUri, > int ttl ) > > /** > * Registers a number of entries all in one operation. Identical to > calling > * register with each entry in turn. > * > * @param entries a sequence of Entry records with serviceUri, > targetUri, > * qualities, and ttl as specified in register() above. Who and > lastUpdate > * fields are ignored. > */ > @Authorize( canRegisterBulk, entries ) > void registerBulk( Entry[] entries ) > > /** > * Removes the specified entry. > * > * @param serviceUri the uri describing the service. > */ > @Authorize( canRegister, serviceUri, null ) > void unregister( string serviceUri ) > > /** > * Adds a request for notification of changes to entries matching the > query. > * The current value of all matching entries is delivered via > entryChanged > * client message, as well as any updates or new entries. > * > * @param query a sql-like query expression using the elements of the > * serviceUri, targetUri, and qualities. If query is null or the > empty > * string, all entries are matched. > */ > @Authorize( canFind, query ) > void subscribe( string query ) > > /** > * Removes a request for notification of changes to entries matching > the > * query. > * > * @param query a query previously passed to subscribe. The string > must > * match exactly. > */ > void unsubscribe( string query ) > > /** > * Removes all requests for notification of changes to entries. This > * operation is implicitly performed when the connection to the > NameService > * is dropped. > */ > void unsubscribeAll() > > //////////////////// > // RIGHTS TESTING // > //////////////////// > > /** > * Tests whether the current user is authorized to lookup the source. > * @param source the complete specification api/instance/scheme. > */ > boolean canLookup( string source ) > > /** > * Tests whether the current user is authorized to run the query. > * > * @param query a sql-like query expression using the elements of the > * serviceUri, targetUri, and qualities. If query is null or the > empty > * string, all entries are matched. > * > * @return true if the current user is authorized to run the query. > */ > boolean canFind( string query ) > > /** > * Tests whether the current user is authorized to register the > service. > * > * @param serviceUri the uri describing the service. > * > * @param qualities a map which may be used to describe additional > features > * of the entry. > * > * @return true if the current user is authorized to register the > service. > */ > boolean canRegister( string serviceUri, Map qualities ) > > /** > * Tests whether the current user is authorized to register the > entries. > * This is the same as: > * > * for (Entry entry: entries) > * if (!canRegister( entry.serviceUri, entry.qualities )) > * return false; > * return true; > * > * @param entries a sequence of Entry records with serviceUri, > targetUri, > * qualities, and ttl as specified in register() above. Who and > lastUpdate > * fields are ignored. > * > * @return true if the current user is authorized to register the > entries. > * This is an all or nothing proposition. > */ > boolean canRegisterBulk( Entry[] entries ) > > //////////////////// > // CLIENT METHODS // > //////////////////// > > /** > * Notifies the client of a change in an entry. The entry might have > been > * created, updated, or removed. > * > * Note: while this might have normally been an event, we made it a > call > * to slow down the processing of what might otherwise be a rather > large > * change set. > * > * Note: when keeping track of entries, always keep the one with the > latest > * lastUpdate. > * > * @param query the query which triggered the notification. > * > * @param entry the entry which has changed. > */ > @Direction( client ) > void entryChanged( string query, Entry entry ) > } > >
