On 6/3/2015 2:25 AM, Bruce Jackson wrote:
I’ve now built 2 different systems based on OSGi at scale (over 1M end devices)
where the endpoints needed to communicate together. In both cases, it has
required some ‘server’ infrastructure to help with message passing between
endpoints, but that could be a side effect of the infrastructure we had to use.
However, these server components were again based on OSGi.
The http question (I.e. using the HTTP service) is actually a bit of a red
herring: there is nothing stopping each endpoint implementing an http service
and also being a client of other endpoints to pass messages. My suggestion
would be to use this approach, as it has so many side benefits, including:
* good availability of tools such as Jackson for message serialisation
* open protocols that will allow for different endpoint implementations in
future
* (by and large) avoiding problems that custom socket protocols will run into
with firewall tunnelling etc
…the list here is pretty long.
I would like to discuss the Remote Service specification before
returning to discussing protocols and RS implementations.
The specification explicitly allows the runtime selection of a
distribution provider to be used to export a remote service, via a
standardized service property called the 'config type' (property name:
service.exported.configs). As an example, the 'ecf.generic.server'
distribution provider is the config type used in this tutorial [1].
I feel that one of the most useful aspects of the config type and the
underlying distribution system is that it abstracts away the
implementation specifics, i.e. the transport protocol (e.g. http, jms,
tcp, mqtt, private, etc), as well as the serialization mechanism/format
(json, binary, xml, etc). It does this for both the remote service
creator/implementer and for the remote service consumer.
IMHO, there are there are many advantages of doing this. A big one is
modularity and reuse. Why should it be necessary for every remote
service author to implement a brand new protocol and serialization, in
order to remote their micro service? Another one is flexibility: any
given reasonably sophisticated app has changing requirements over time,
and sometimes these changing requirements require flexibility in the
underlying libraries. Another big advantage is getting access to all
the simplicity, dynamics, versioning, and other capabilities of OSGi
services provided by the OSGi service registry and libs built on top of
it (e.g. DS, etc). There are of course many others.
Of course, there are disadvantages in reusing distribution systems or
any set of libraries: Perhaps the protocol or serialization mechanism
that's widely implemented and popular today doesn't meet your
application's performance needs. Perhaps there are more appropriate or
new protocols, or improved implementations. Perhaps you have unique
security requirements. Perhaps you have timing or reliability
requirements that are difficult to address. There are lots of
application-specific reasons for favoring or disfavoring non-functional
requirements like performance, ease of deployment and management,
security (e.g. firewalls), network reliability/failure handling, etc.
But in my view, the Remote Services specification adopts a nice balance
here, by allowing potentially many distribution providers, down to the
level of the individual service instance set at *service instance
registration time*. Even if one needs to create a new, entire
distribution system, there are still many advantages of using OSGI
Remote Services...e.g. the service registry's graceful handling of
dynamics inherent in a network-based service, versioning, discovery, and
the ability for service creators and perhaps more importantly service
consumers to ignore some of the subtle and difficult aspects of
distribution (e.g. partial failure, timing, etc). I do think that
standardizing these advantages and having many implementations will over
time make it easier to build, use, understand, and manage distributed
applications.
Returning to RS implementations: The approach ECF has taken (since
well before the Distributed OSGi rfc and RS/RSA standardization) has
been protocol neutral. As such we have gone to the trouble of creating
extensible APIs for various types of network communication: e.g.
filetransfer, instant messaging, group messaging, voice over ip, network
service discovery, and remote services distribution (of course). For
each of these types, we have multiple protocol implementations, some
created by us (in open source), some created by others (in open source
or not). This is true, of course, for our OSGi Remote Services
implementation, and it's distribution and discovery subsystems...as I've
said we have a number of providers that are based upon tcp and we have
providers that are based upon http and REST. But more importantly (I
think), we are trying to support the creation of hopefully many
other/new providers, created by others...possibly extending those that
already exist...that meet their needs for app-specific non-functional
requirements, new/popular application areas (like IoT) for distributed
applications.
The way that we do this with a multi-provider/multi-protocol approach is to:
0) Fully implement the latest specs an test for compliance
1) Define stable, extensible, open APIs for distribution and discovery
(e.g. 'remote services api' -> distribution, 'discovery api' ->
discovery), and maintain/enhance these APIs.
2) Re-use existing work to create stable, extensible, open provider
implementations for these types (e.g. ecf generic, http/REST, r-osgi,
jms, mqtt, etc)
3) Do what we can in open cooperation with the community to document and
provide examples for both 1 and 2, as well as document how to create new
providers [2], and analyze the runtime characteristics of existing
providers.
4) Engage the community (commercial and open source) to drive 0-3
forword in the directions that are most needed, and get support for
these efforts via new committers, contributors, collaborators, and
supporters [3].
5) Provide the support for the community in doing all of the above
Are we resource-limited? Yes, unless you consider the community. Is
there more to do? (particularly around documenting individual
providers)? Yes. My comment is that in today's software world I don't
think these answers are particularly uncommon ;-). [4]
Scott
[1]
https://wiki.eclipse.org/Tutorial:_Building_your_first_OSGi_Remote_Service
See section: Service Host: Register and Export the Service
[2]
https://wiki.eclipse.org/Tutorial:_Creating_a_RESTful_Remote_Service_Provider
[3] https://projects.eclipse.org/projects/rt.ecf/who
[4] https://www.eclipse.org/ecf/NewAndNoteworthy_3.10.0.html
On 02/06/2015 16:08, "Scott Lewis" <sle...@composent.com> wrote:
On 6/2/2015 2:16 AM, Simon Kitching wrote:
Hi Scott,
Maybe I shouldn't hijack a thread on the Felix list to ask about ECF,
but we are already talking about remote-services implementations, and
you are here so I'll do it :-)
Link [1] has instructions on using ECF's remote services on Karaf
which lead to this karaf features file:
http://download.eclipse.org/rt/ecf/3.8.0/site.p2/karaf-features.xml.
That file appears to indicate that 49 bundles are needed (the majority
from eclipse.org). Plus bundles to support whatever discovery provider
is desired. That doesn't seem reasonable; other remote-services
implementations have 1-3 bundles. Have I understood correctly?
For the latest the correct URL is:
http://download.eclipse.org/rt/ecf/latest/site.p2/karaf-features.xml
ECF's implementation is quite modular, and so is made up of more
bundles, generally much smaller and finer-grained.
<stuff deleted>
Looking at the commit-log for the osgi code, I see that you are the
only committer since at least the start of 2014. Is this correct?
I have done the majority of the commits from the start of 2014, but am
not the only committer. The source code history is not a very good
indicator of overall contributions for a project in this point in it's
life-cycle. Also, much of the recent coding work on ECF (by myself and
others) has taken place on our github repos[4].
<stuff deleted>
Regarding building from source : I see no Maven, Ant, Groovy or bnd
buildfiles in the sourcecode - just eclipse project files. Could you
point me to instructions on how to build artifacts from the
source-code (ideally outside of the Eclipse IDE)?
We currently use Eclipse + buckminster to build, and are part of the
Eclipse Simultaneous Release (which also uses buckminster). There are
contributors from Red Hat that are working adding to use Tycho to
build. FWIW, we also produce a maven repos as part of our build [A].
And is there any easy way of seeing what the dependencies of the
projects under osgi/bundles are (the kind of info visible in a maven
pom-file)?
From Eclipse, certainly. When Red Hat's contribution is in place you
will be able to do with maven-pom file as well.
Is there a list of the implemented "distribution providers" (network
protocols for the actual remote method invocation) available, with
details on the features and performance of each choice?
No, regrettably we don't have that. I've opened an enhancement request
to that effect [B]. There was some performance analysis work
contributed some time ago, but unfortunately I think it's likely pretty
obsolete by now.
I couldn't find one. In particular, I needed a high-performance
protocol - one that (a) keeps network sockets open rather than opening
a new connection for each call, and (b) uses a high-performance
serialization mechanism, ie not xml. I couldn't find any page with
that kind of information in it. The closest I found was:
*
https://wiki.eclipse.org/Discovery_and_Distribution_Provider_Configuration_Options
*
https://wiki.eclipse.org/Comparison_of_Discovery_and_Distribution_Providers
but neither of those pages are very useful. In particular, "r-osgi"
and "ecf generic" are not described in any way.
For the details about everything there's always 'use the source'. :-)
I can tell you that r-osgi, ecf generic, JMS, MQTT distribution
providers all
1) keep network sockets open as you describe;
2) use java object serialization as the default
I can also tell you that all of those providers, and the distribution
API that they implement (known to us as the 'remote services api'), are
customizable/extensible for both 1 and 2...meaning that if you wish, you
can replace both the serialization as well as the socket-level transport
with something more suitable for your use case without having to
implement a distribution provider from scratch. As a an example, that
is how websockets support was added in the 3.10 release cycle.
I propose moving this to ecf-dev or direct correspondence if you want
more details.
Scott
[A] http://build.ecf-project.org/maven/
[B] https://bugs.eclipse.org/bugs/show_bug.cgi?id=469155
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@felix.apache.org
For additional commands, e-mail: users-h...@felix.apache.org
*** DISCLAIMER *** This message, including attachments, is intended solely for
the addressee indicated in this message and is strictly confidential or
otherwise privileged. If you are not the intended recipient (or responsible for
delivery of the message to such person) : - (1) please immediately (i) notify
the sender by reply email and (ii) delete this message and attachments, - (2)
any use, copy or dissemination of this transmission is strictly prohibited. If
you or your employer does not consent to Internet email messages of this kind,
please advise Myriad Group AG by reply e-mail immediately. Opinions,
conclusions and other information expressed in this message are not given or
endorsed by Myriad Group AG unless otherwise indicated by an authorized
representative independent of this message.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@felix.apache.org
For additional commands, e-mail: users-h...@felix.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@felix.apache.org
For additional commands, e-mail: users-h...@felix.apache.org