Re: where the indirection layer belongs
Dear Keith Moore, I do not want to seem to be ganging up on you, but your reply to Mr. Hain just prompts more questions. You are missing something fundamental here - if a TCP connection breaks (isn't closed cleanly) then the two endpoints can get out of sync regarding how much data was delivered. You can't fix this at higher layers without an unacceptable amount of complexity and overhead. This has nothing to do with the app/transport interface being a sacred invariant - it happens any time you try to layer something on top of transport that has to survive breakage of the transport layer. (how many ways do I have to explain this?) You might have to explain this really often and in simple language too. But why do you assert that it will take lots of complexity and overhead? Can you point to some code where they tried this? As far as I know, nobody has really given this an earnest try as yet. At least not with any IP protocols. It's a lot less glue than the L4-L7 approach, and most of it has to deal with authentication that would be needed for any kind of remotely-originated routing update anyway, regardless of what layer it lived at. What specific glue do you believe it requires for the L4 to L7 approach? How does that compare to what is needed in an L3 solution? Yes you can do that but it presumes that the host knows a priori whether or not it needs the stabilization layer. I would make the mechanism used to provide stable addresses a property of the network- either the network provides "reasonably" stable addresses (i.e. plenty of prior notice before changing them) or it provides a stabilization layer. That way, networks that don't need it don't pay the overhead. But I would argue that the host or at least the application's designer knows whether it will need the stabilisation layer. Making the mechanism that provides the stable network addresses a property of the network leaves the question of how. Even if that were achieved though, that still does not completely or effectively address the problem of one application process identifying its peer across the network. Yours sincerely, Robert Honore.
Re: where the indirection layer belongs
an interface or a set of interfaces." While we have regarded the node and its interface as one and the same, it really isn't, even though we can get away with that most of the times. My point is that we haven't even considered the other half of the problem as far as the application is concerned. We also need a higher level object which we can associate with the node itself or the application endpoint and with which we can associate an identifier. Such an association will need to remain valid at least for the duration of the application's run-time, I believe. And the second question is what should be the context of that identifier's validity? The identifier should be unique within the entire Internet. In some cases (e.g. small networks without connectivity to the Internet core) "very likely to be unique within the Internet" would probably suffice. That doesn't mean that you can send to any point from any other point, because there will still be access controls. But if an address is advertised for a host, and you have permission to send to that host, you should be able to use that address to send to that host. If you accept the statement I made above, the minimum context of that association's validity should be the collection of nodes that are currently participating in that application's protocol. I then have no fundamental disagreement with your assertion that such an identifier should be valid within the entire Internet. The first one I explained above in slightly more detail. The second one should be obvious. If an address becomes invalid because of a topology change somewhere distant in the network, how is a layer above layer 4 going to know about it? It doesn't have access to routing protocol updates - those happen at layer 3 and aren't normally propagated to hosts anyway. When you think about it, you probably don't want hosts to have to listen for information about distant state changes in the network - that would place a tremendous burden on small hosts and nets with low-bandwidth connections. The second one becomes obvious only if you restrict the definition of endpoint to the interface. If instead the definition of the endpoint is taken to be the process or object that is supposed to be the final recipient of the data that is forwarded by the IP and the transport layers, then it is no longer so obvious. Sometimes I wonder if we aren't treating together concerns or problems that we should be treating separately. I don't want *application processes* to be needing to listen for information about distant state changes in the network, but I do want my application processess to be able to adapt or recover when something happens that affects its communication with a peer *process* while neither of the cooperating processes has indicated any intention to stop participating in that communication. Yours sincerely, Robert Honore.
Re: where the indirection layer belongs
Dear Folks, Regarding this discussion about an indirection layer, I am thinking we really should propose the formation of some forum for discussion of these issues. It does not seem to fit cleanly into the IPv6 working group, and out here in no-man's-land, I don't think it is getting the kind of attention it needs. I had already made my position clear. We need something. Call it an indirection layer or a stabilisation layer or whatever you want, but we need a forum where we can specify the problem we are trying to solve and to consider the possible solutions for it. Does anybody agree? Yours sincerely, Robert Honore.
Re: where the indirection layer belongs
Dear Folks, I have been trying to understand the problem we are trying to solve in this thread and trying to see if I could produce for myself something resembling a specific description of that problem. I might have mentioned in passing earlier today that perhaps we are trying to treat as a single problem, separate problems that might be better treated as separate. I am becoming more convinced of this as we go on. As such, I can distinguish the following issues as aspects of the problem given all that was mentioned in this thread, the "solving the real problem" thread and the one on the IPv6 mail list about deprecating Site Local addresses and the usage of IPv6 Link Local addresses. They are as far as I can tell the following. * Stable (or reliable) end-point identifiers * Resiliency of application (protocol) in the face of sudden IP address changes * Self-organised networks With respect to stable end-point identifiers, we must in my opinion, still specify what we are calling an end-point and settle once and for all the question of whether an IP address is a suitable candidate for such an identifier. With respect to the resiliency of applications in the face of [sudden] IP address changes, we need to determine if, as Keith Moore argues it cannot be solved in a single layer, or if it can as Tony Hain argues, as well as where best to situate such a layer. The one about the self-organised networks materialises out of the discussion going on in the IPv6 mail list about the deprecation of Site Local IP addresses and the proper usage of IPv6 Link Local addresses. Many of the contributors to those threads were basically arguing that we should retain either or both of Site Local and Link Local addresses because of intermittently connected sites, nodes or subnetworks, as well as being to maintain communications between two locally adjacent when the local subnetwork either connects to - or disconnects from - a larger network. My question following from all that, are two. Is it worth it to attempt a solution to any of the aforementioned problems? If so, which one should we tackle first? Yours sincerely, Robert Honore.
Re: where the indirection layer belongs
Dear Keith Moore, Maybe I read your paper on project SNIPE too quickly, but it was not immediately clear that the problems you mentioned were a specific result of an attempt to make the application resilient against (sudden) changes in IP address. More specifically, it was not clear from that report that the additional complexity did come from the attempt to provide the kind of resilience we are seeking, or from the rather ambitious goals of project SNIPE. I will re-read more slowly and carefully this time. Yours sincerely, Robert Honore. Keith Moore wrote: (regarding the complexity of putting a general-purpose layer to survive address changes between L4 and L7) But why do you assert that it will take lots of complexity and overhead? Can you point to some code where they tried this? As far as I know, nobody has really given this an earnest try as yet. At least not with any IP protocols. I tried this in connection with a project called SNIPE that I worked on several years ago. SNIPE was an attempt to build a very reliable distributed computing environment that supported, among other things, the ability to access a computing resource via multiple addresses (mostly in order to exploit high-bandwidth local networks not necessarily using IP), and the ability of both hosts and processes to migrate to other addresses. It used a DNS-like service similar to RESCAP (for those who remember that) to register the addresses at which a process was accessible, and it attempted to provide TCP-like streams on top of TCP and this registry that would survive changes in those addresses. Basically I found that you can get such a service to work reasonably well and reasonably efficiently if endpoints don't move without adequate notice. OTOH if hosts or processes do move without adequate notice then you end up needing to implement the mechanisms I mentioned earlier, and that involves extra copies and extra overhead. The reason I am preposing that the two problems (changing addresses with adequate notice and changing addresses without adequate notice) be treated separately is that by trying to make a single mechanism serve both purposes you end up with a lot of inefficiency and/or cost that aren't needed in most cases. And that's true (for different reasons) regardless of whether you insert that single layer between L3 and L4 or between L4 and L7. What specific glue do you believe it requires for the L4 to L7 approach? Thought I'd said this already: buffering of data until acknowledged by the peer, generation and processing of acknowledgements, retransmission of lost messages due to broken connections, windowing (so you don't degrade to stop-and-wait), duplicate suppression. You also need to multiplex control and data information over the same stream and to probe for other addresses when you get a failure on the one you were using. How does that compare to what is needed in an L3 solution? If you work on the problem at (or just above) L3, transport protocols already have the code needed to recover from lost messages, so you don't have to reimplement that. You basically need a mechanism by which the layer can realize it needs to start routing packets differently, and do so. You probably need multiple ways that the layer can get that information because the remote address can change for a variety of reasons and in lots of different places in the network. (That's equally true for the L4-L7 layer as for the L3-L4 layer, but the L4-L7 layer isn't in a position to get some of that information. The L3-L4 layer can potentially recover from address changes more quickly but to do that safely it has to be able to authenticate and establish a basis for trust in a wider variety of information sources.) Yes you can do that but it presumes that the host knows a priori whether or not it needs the stabilization layer. I would make the mechanism used to provide stable addresses a property of the network- either the network provides "reasonably" stable addresses (i.e. plenty of prior notice before changing them) or it provides a stabilization layer. That way, networks that don't need it don't pay the overhead. But I would argue that the host or at least the application's designer knows whether it will need the stabilisation layer. It can't know that reliably unless the network without the stabilization layer has well-defined properties - e.g. the network won't change addresses of a network without advertising those changes with a minimum amount of advance notice. If addresses can potentially change at arbitrary times with no assurance of stability then every app needs the stabilization layer (or provide its own means of recovery). Making the mechanism that provides the stable network addresses a property of the network leaves the question of how. Even if that were achieved though, that still does not completely or effe
Re: Solving the right problems ...
Dear Tony Hain, Perhaps this proposal really requires another working group or something. I seem to remember someone making a similar proposal a several years ago on this list and it didn't seem to get a good reception then. For what it is worth, though, I really do think it is an idea whose time has come. It seems everyone has espoused the idea that the application *must* talk directly to the network transport in order to be efficient and/or resilient, but given the capabilities IPv6 is supposed to have, I wonder how much of a good idea it is to cling to that concept. It is not worthwhile to suggest that the direct application/transport interface be eliminated, but it should be possible for an application to specify what kind of service it wants from the (transport, network) combination and something is there to handle that where the requested service is different from that natively provided by the (transport,network) combination. Problem is that right now, every time an application needs such a service, the application programmer must code his own provisions for it, and none of the propositions I have seen discussed so far seems to deal with that. Keith Moore's statement is quite correct, viz: Application developers need reasonably stable and reliable endpoint identifiers. At present, the IP address is the closest available approximation to such an identifier. Two questions arise out of that though, at least as far as Keith Moore's statement is concerned. The first one is how stable is stable? That is, how long is that identifier supposed to remain stable. And the second question is what should be the context of that identifier's validity? Also from Keith Moore: Disagree. The layer needs to be between the current network and transport layers. If the layer were between transport and applications, many of the services provided by transport would end up having to be re-implemented in higher layers. And It would also force routing decisions to be made by layers much more distant from the network layer that maintains reachability and state information than the transport layer. Does Mr. Moore care to explain a little why he believes these two statements? As far as I can tell it needs not necessarily be so. Yours sincerely, Robert Honore. Tony Hain wrote: In the ongoing saga about topology reality vs. application perception of stability, it occurs to me we are not working on the right problem. In short we have established a sacred invariant in the application / transport interface, and the demands on either side of that interface are the root of conflict. Mobile IP, and the multi6 DHT work are attempts to mitigate it through slight of hand at the IP layer, while SCTP attempts to mask the topology reality in the transport layer. (These are probably not the only examples, but they are the ones that come to mind this morning.) Yet none of these really do the job in all cases. We all agree that applications should not be aware of topology. At the same time, application developers insist on the right to pass around incomplete topology information. There are arguments that the IP address is overloaded as an endpoint name. I don't particularly buy that, because the real endpoint is either a protocol, or a specific port of a protocol. From one perspective, when applications use addresses in referrals, they are specifying that routing through a particular point in the topology will reach the desired endpoint named 'transport/port-id'. But that is a semantics and perception discussion which doesn't help reach the goal. In any case, what applications need is a stable reference to other participants in the app. Yet we know that current and projected reality says that topology is not stable, or consistently reachable. This could be due to technology issues of the underlying links, or simple policy. Either way, the network as seen by the transport layer is not, and will never be stable or consistently reachable from everywhere. Given that, the direct interface with the transport layer creates a failure mode for an application expecting stability. Where this leads us is to the sacred invariant, and the need to solve the problem in the right place. I suggest it is time for the IETF to realize that the protocols are no longer being used in the network of 1985, so it is time to insert a formal stabilization layer between application & transport. Such a layer would be responsible for managing intermittent connectivity states and varying attachment points of the transport layers below. Applications that interact with this layer would be insulated from the inconsistencies being experienced by the transport layer. It would also be reasonable for this layer to manage multiple simultaneous transport interactions so that the application perceived a single data path to its peer. With appropriate trust between the stack and the network policy, this would e
Re: where the indirection layer belongs
Dear Masataka Ohta, Masataka Ohta wrote: Keith; (regarding the complexity of putting a general-purpose layer to survive address changes between L4 and L7) It is not merely complex but also useless to have such a layer. Right now I am not fully aware of all of the specifics of the issues in trying to implement such a layer, but the statement you make in the paragraph just below does not seem to support your statement that "It is not merely complex but also useless to have such a layer." I will explain my position below. The basic problem of the approach to have such a layer is that only the application layer has proper knowledge on proper timeout value to try other addresses. If such a layer is useless as you say, then the application could see no benefit to being able to parametrise the transport or network layers with such information as the proper timeout values to try other addresses. It would also be impossible for the application to benefit from being able to find other addresses to try. Another objection I have to your statement: If the application layer is the one that has the proper knowledge of things like timeouts (I suppose there are things), then it should be possible to implement something on the stack that is close to the application wherein it can collect that information and parametrise the behaviour of the rest of the lower layers. If by your statement you are saying that it is impossible to implement such a thing, then you will have to prove it. So, the issue can be solved only involving application layer, though most applications over TCP can be satisfied by a default timeout of TCP. In either case, there is no room to have an additional layer. I agree completely with your first sentence in this paragraph. The problem we have right now is that, save for the application specifying to the transport which peer port it wants to connect with, it has no further control over the behaviour of the transport or the network layers other than to possibly drop the connection. I cannot as yet agree or disagree with your second sentence. I don't know as yet what needs to be built. I will say that I would prefer to minimise or avoid altogether any modifications to the transport or the network protocols. I would also prefer not to modify any of the libraries or implementations of those protocols, lest we break something. I documented it long ago (April 2000) in draft-ohta-e2e-multihoming-*.txt The Architecture of End to End Multihoming (current most version is 05) that: To support the end to end multihoming, no change is necessary on routing protocols. Instead, APIs and applications must be modified to detect and react against the loss of connection. In case of TCP where there is a network wide defact agreement on the semantics (timeout period) of the loss of connectivity, most of the work can be done by the kernel code at the transport layer, though some timing may be adjusted for some application. However, in general, the condition of "loss of connectivity" varies application by application that the multihoming must directly be controlled by application programs. Masataka Ohta Having looked at draft-ohta-e2e-multihoming-05.txt, I am not convinced that it supports your statement that "It is not merely complex but useless to have such a layer", nor do I believe that the presence of such a layer would necessarily violate the end-to-end principle you advocate in draft-ohta-e2e-multihoming-05.txt. In fact, the way I see it, the presence of such a layer could make it easier to achieve the aims of draft-ohta-e2e-multihoming-05.txt even without needing to designate any special kind of IPv6 address, provided we can solve the problem of specifying end-point identifiers and clarifying exactly what they should identify. PS Layering is abstraction and not indirection at all. Agreed. We should use the correct terminology. Yours sincerely, Robert Honore.
Re: where the indirection layer belongs
Dear Pekka Nikander, Please forgive the late reply. Where can I find out more about Dave Crocker's MAST? See the rest of my message embedded among the qoutation of your text. Pekka Nikander wrote: Robert, I like your analysis very much. Thank you for writing it up. However, I also see a kind of causality here: it looks to me that stable end-point identifiers are mainly needed to make applications survive IP address changes. Dave Crocker's MAST is a good example how you can do that without having stable end-point identifiers. There is more to stable end-point identifiers than just the ability to make an application resilient. I will argue that IP addresses *do not* identify end-points, but node interfaces. That is, they only specify objects on the path to the end-points. Applications, users and system administrators need to specify more than node interfaces They try to do so by twisting IP addresses to all kinds of usages for which they were not intended and are ill-suited in my opinion. Indeed it is a feature that IP addresses and their definition as an identifier for a node interface are so flexible that it is possible to contemplate extending their usage for those purposes. On the other hand, security looks to me as a good reason for having stable end-point identifiers. If you can securely recognize an end-point (with emphasis on the *re-* part of re-cognize), you can develop trust. Trust, in turn, is very handy for lowering transaction costs. Indeed it is, and I would add that end-point identifiers (together with node interface identifiers) are the thing that system administrators and system security officers want. Even facing the danger of opening yet another rat hole, in my opinion we should not have a very strict definition for end-point. That is, IMHO end-point should and could be a fuzzy concept, somewhat like the concept of a site is today. From my point of view, an end-point may be a process, a group of processes, a host, or even a server cluster offering services as a unit. To me, it looks like fate sharing and common semantics are the key points here. An end-point should either work or fail, it should not be usual for half of an end-point fail while the other half is continuing. An end-point should also be considered at the application level as a single unit. The question that that raises is, whether it is reasonable to use such an inclusive notion of an end-point, and what is the simplest kind of structure or object we can use to implement an end-point identifier? We might have to restrict the notion of an end-point somewhat. In my opinion, we clearly need solutions to all of these problems. Furthermore, it looks like introducing stable end-point identifiers to the stack almost automatically protect applications from the changes of IP addresses. I also tend to believe that stable end-point identifiers may also help to build self-organized networks. However, IMHO the problem of self-organized networks is more researchy in nature than the other two. With respect to the protection of applications from changes of IP addresses, the effectiveness of that protection (or its robustness) will depend on what the final specification of end-point identifiers is. With respect to the problem of self-organised networks, that might be a research problem, but I would argue that it is a here-and-now research problem. Now, even though I believe that we should solve the problems (and apparently believe that there are sensible solutions), achieving consensus on solutions that require architectural change may take too long. Hence, I also believe that we need some kind of a road map, with some "temporary" or intermediate solutions along the way to a more long-standing set of solutions. I agree, and your statement corresponds to what Keith Moore says about the solutions fitting into a framework that is yet to be specified. I believe specification of that framework begins with our defining what an end-point is. Yours sincerely, Robert Honore.
Re: where the indirection layer belongs
Dear Masataka Ohta, I was a bit impetuous in saying that I would prefer not to modify libraries or implementations etc. However, my aim so far is more to obtain for myself a clear understanding of the problem we are trying to solve, rather than trying to state requirements of the possible solutions to that problem. My apologies for such a stupid statement. I do believe that there are issues to be resolved and they are being currently reflected in threads like the one about deprecating site-local addresses and the need for provider independent addresses that is currently active, as well as the thread "solving the real problem" started by Tony Hain. I believe the issues being discussed there are real. You might not agree that they are. If the issues are real, then we need to specify or formulate them in a way that is amenable to deriving solutions for them. If the issues are not real, then they are likely the artifacts of wrong perceptions or ideas that users and engineers in the IPv6 community have. In that case, we need to replace those wrong perceptions or ideas with correct ones. It seems to me though, that nobody has stated clearly what those wrong perceptions and ideas are, much less to say what is wrong with them and thus replace them with correct perceptions and ideas. Yours sincerely, Robert Honore. Masataka Ohta wrote: Robert Honore; I would also prefer not to modify any of the libraries or implementations of those protocols, lest we break something. It is a wrong requirement wrong in various ways. First of all, we must change something, which means we must change some code. Though the changes should better be simple, whether the changes can be inserted as a module or not is irrelevant, To make the changes simpler, the modification MUST be confined in libraries or kernel. So, we should try to change libraries or kernel to avoid changes on application code. It, of course, is necessary to change application code for UDP cases for address selection that it is necessary to an API to control address selection in IP headers from applications that IP and transport layer code MUST be modified . Secondly, preservation of code is less important to preservation of protocol and protocol is defined on the wire. If you put some code below transport layer to modify packet content visible from the transport layer, which is the case with MAST, you change the transport layer protocol. But, again, we must change something of the protocol. So, the requirements are: to modify existing code to modify existing protocol and any attempt, including but not limited to MAST, to avoid them is not constructive only to constraing the solution space to result in less capable and less efficient solutions. Of course, it is desirable to preserve existing application code whenever possible. But, we must be aware that it is NOT always possible. In addition, it is desirable to preserve interoperability with existing protocols which automatically means "to preserve interoperability with existing code of the peer". In this case, it is doable. Of course, all the freedom to use multiple addresses is lost when the peer is a leagacy one. Masataka Ohta
