++1
--- Luca Burgazzoli On Tue, Jul 31, 2018 at 9:47 AM, Andrea Cosentino <ancosen1...@yahoo.com.invalid> wrote: > This is a wonderful idea and I believe we can improve our community with a > subproject like this. > > So +1 for me. > > -- > Andrea Cosentino > ---------------------------------- > Apache Camel PMC Chair > Apache Karaf Committer > Apache Servicemix PMC Member > Email: ancosen1...@yahoo.com > Twitter: @oscerd2 > Github: oscerd > > > > > > > On Monday, July 30, 2018, 5:49:50 PM GMT+2, Nicola Ferraro > <ni.ferr...@gmail.com> wrote: > > > > > > Hi Cameleers, > it seems from the comments that this "Kamel" subproject is something we > want to start and I think that also the main camel core will benefit from > the new features it will bring. > > I would like to donate the current "Kamel" code to Apache Camel, in order > to have a initial brick to start from. > > From your reactions, the name "Kamel" seems cool and I'd like to keep it. > However if you talk to people about the "Kamel" project, they won't > understand if you're talking about "Kamel with the K" or "Camel with the C". > For this reasons I propose to keep "Kamel", but also use "Camel K" [ˈkæməl > keɪ] as friendly name when talking about it. This is in line with the > repository we should create for the subproject, that needs to start with > "camel-" (it will be "apache/camel-k" in that case). > > These days I've been experimenting different ideas with Luca Burgazzoli. In > particular, there have been some concerns here on the power of a > declarative DSL (a new one or the existing XML one) and we've found a > strategy that will allow us to use also the Java DSL in Kamel integrations. > Without adding too many details here, at integration build time we can run > the user code in a build container and inspect the produced routes to get > metadata for the optimizer. This is one of the first things we should do > next. > > This possibility of running code that produces integrations opened a lot of > unexpected paths, that I've written down in the project roadmap ( > https://github.com/nicolaferraro/integration-operator) and summarized below. > > One nice feature (kudos to Luca) is that we can simplify life for Kamel > users up to the point that they'll just need to write their routes on a > Java file and run them with e.g.: > "kamel run Routes.java" > > Where "kamel" is a binary we release within the project. Kubernetes custom > resources will be used under the hood, but the "kamel" binary is a utility > that will provide a user experience comparable to that of current > serverless platforms (or even better). > > But there's not just that. > > Luca wrote this weekend a prototype for having a polyglot Camel ( > https://github.com/lburgazzoli/camel-routes-loader). > > With that, you will be able to e.g. write integrations in groovy and use > the same Kamel engine: > "kamel run routes.groovy" > > But also JavaScript: > https://github.com/lburgazzoli/camel-routes-loader/blob/ca986541f7c422ee02c21727cdfe4293d64a364e/src/test/resources/ext/camel/routes.js#L2 > > This is a prototype right now, but a similar approach based on GraalVM has > a lot of potential, because users can not only use their preferred language > to write routes and processors, but also bind them to functionalities > available in their preferred libraries. > > How much this will be feasible depends on the adoption of GraalVM, but I'm > seeing many frameworks adding metadata to make GraalVM work with > reflection. We've also done some work in Camel, there are some Jiras for it > and first tests made by Guillaume signal that it's something feasible, at > least for the Camel core and a subset of components. GraalVM would be > important also to reduce memory footprint and improve startup time, as > already said. > > If the GraalVM approach works (it is working for other frameworks), instead > of rewriting a subset of Camel in Go (as the original proposal mentions), > we can just sanitize and recompile our existing codebase: this way we fully > leverage the strength of Apache Camel. > > Another important thing in the roadmap is actually how we'll integrate with > current FaaS platforms. We will evaluate together if it's better to have a > tighter integration with some of them or to leverage knative for some > aspects of Kamel... > > But, let's get started! > > Il 19 lug 2018 10:45, "Nicola Ferraro" <ni.ferr...@gmail.com> ha scritto: > > It's clear to me that we need to add support for our existing XML DSL, that > is powerful. But there are multiple reasons why I'd like to also "add" a > limited yaml/json notation to "Kamel". > > The first one (and simplest) is that json/yaml is the primary encoding for > all resources exchanged with the Kubernetes API server. I'm not saying that > for this reason the Integration resource cannot contain a XML > (/Java/Ballerina) section, but from a pure presentational point of view, > having the possibility to write simple use cases (even a "hello world") in > pure json/yaml is better than always requiring a mix of json and XML. > > The second one is simplicity. Writing a optimizer (the module that > materializes the integration into running code, choosing a specific runtime > and configuration) for a turing-complete language like Java is not easy: > even if you manage to create a good parser, it may be able to do > optimizations only if you write routes in a particular way, without complex > constructs. > XML is ok from this point of view. The thing is that both json/yaml and XML > are just two different ways to serialize object trees, that can be then > statically analyzed. > The point is not XML vs json/yaml, it's more about tailoring a new > minimalistic DSL to the emerging use cases vs proposing "only" our classic > way of writing integrations. I think XML can be the "advanced" way. We can > experiment optimizations easily with the new DSL, and enable them also on > XML if it's worth. > > Scripting should be part of the spec, but I'd try to use programming > languages only for processing/transformation, not for the route definition. > > Third one is performance. Apart from the fact that json parsers are in > general said to be faster than XML parsers... Given the "fast startup" > target that we want to reach, we may think e.g. to translate the new DSL > into Java or Go code, then compiling it. This allows doing parsing at build > time in order to avoid it on startup. This kind of improvements are much > easier with a limited DSL but much more difficult with a existing > fully-fledged DSL.. > > Nicola > > > > On Wed, Jul 18, 2018 at 11:51 PM Pontus Ullgren <ullg...@gmail.com> wrote: > >> I also like the idea but with some comments. >> >> As Hiram Chirino I'm not sure YAML/JSON is the best language for this. >> Perhaps a more fluent DSL like the Camel Java DSL or perhaps something like >> Ballerina language would be better suited ? >> Also in my experience even simple integrations, that is simple real world >> integration and not just hello world demos, requires >> you to add one or more Java class or scripting in addition to the core >> components. So for it to be useful there must be some way to add custom >> code in some way for aggregation/enrichment strategies. >> >> If you go with a GO implementation I would also like some fallback for >> using Java since I find it unlikely that the existing base of components >> will be ported any time soon. >> >> As for the name as other I like the "Kamel" name however a being a newborn >> kotlin fan I know of one pre-existing project with that name >> that is close enough to allow for potential confusion. >> https://github.com/topicusoverheid/kamel >> >> Looking forward to see the result. >> >> // Pontus >> >> >> >> >> >> >> On Mon, 16 Jul 2018 at 15:15 Hiram Chirino <hi...@hiramchirino.com> wrote: >> >> > Love the idea. >> > >> > Personally, I'd keep using the existing Camel XML DSL if possible. You >> can >> > still embed it in the CRD. The operator that deploys it could inspect it >> > and figure whats the most optimized runtime that can support the DSL. >> > Perhaps if it's only using the restricted set of camel components >> supported >> > by camel-go (https://github.com/lburgazzoli/camel-go) then is uses that. >> > Otherwise it falls back to using camel spring boot. >> > >> > For that to work I think we need the different runtime implementations to >> > provide a way to ask them: 'hey do you support running this camel route?' >> > Not a trivial thing to respond to, it might require a build step in there >> > for traditional Camel runtimes. >> > >> > >> > On Fri, Jul 13, 2018 at 4:32 AM Antonin Stefanutti < >> anto...@stefanutti.fr> >> > wrote: >> > >> > > Hi Nicola, >> > > >> > > I love the idea. >> > > >> > > I just wonder whether YAML/JSON is an expressive enough format in the >> > long >> > > term. But as you’ve mentioned, starting simple would enable >> experimenting >> > > some very interesting / promising optimisations. So it seems worth >> taking >> > > that path, instead of trying to embed a complex DSL or the existing XML >> > DSL >> > > into the CRD. >> > > >> > > Definitely +1 >> > > >> > > > On 13 Jul 2018, at 01:30, Nicola Ferraro <ni.ferr...@gmail.com> >> wrote: >> > > > >> > > > Hi Cameleers, >> > > > it's now passed some time since I started thinking about a new >> project >> > > that >> > > > we can begin here at Apache Camel, and I'd like to have your opinion. >> > > > >> > > > We've already been targeting cloud-native applications with Camel, >> > > > especially on top of Kubernetes, that is becoming "the standard" >> cloud >> > > > platform. But writing a Camel integration and running it on >> Kubernetes >> > > > requires some effort: choosing the base platform (spring-boot, karaf, >> > > > simple main?), adding health checks (actuator?), packaging a docker >> > image >> > > > and creating the Kubernetes resources (fabric8-maven-plugin, helm?), >> > > > publishing the image on a docker registry, then finally deploying the >> > > > resources on a Kubernetes cluster. >> > > > >> > > > The resulting integration container is then far from being optimal >> > from a >> > > > resource consumption point of view: it is likely that a Camel >> > Spring-Boot >> > > > application will require at least 200MB of RAM and also some CPU >> shares >> > > > because of polling threads used by many components. >> > > > >> > > > In case people use a CI/CD pipeline, it will take also a long time to >> > get >> > > > from a code update to having a Kubernetes POD up and running. >> > > > Apart from compilation and image push/pull time, also startup time is >> > > often >> > > > ~10 seconds for Camel + Spring-Boot in a container with standard >> limits >> > > on >> > > > resources, making it difficult to propose this combination for >> > > "serverless >> > > > integration" (this term is becoming increasingly more popular). >> > > > >> > > > So, my proposal is to start to investigate a "more cloud-native" >> > approach >> > > > to integration: *making Camel integrations first-class citizens in >> > > > Kubernetes, and making them super fast and lightweight.* >> > > > >> > > > We can base the project on Kubernetes Custom Resource Definitions >> (CRD) >> > > > < >> > > >> > >> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/ >> > > >, >> > > > for example a Integration CRD and have a Kubernetes "operator" >> > > > <https://coreos.com/operators/> taking care of: >> > > > - Optimizing the integration that we want to run >> > > > - Packaging in a container >> > > > - Running it on Kubernetes >> > > > - Managing its entire lifecycle >> > > > >> > > > A Kubernetes-native integration may look like: >> > > > >> > > > ------------------- >> > > > kind: "Integration" >> > > > apiVersion: "camel.apache.org/v1alpha1" >> > > > metadata: >> > > > name: "example" >> > > > spec: >> > > > replicas: 1 >> > > > routes: >> > > > - id: timer >> > > > route: >> > > > - type: endpoint >> > > > uri: timer:tick >> > > > - type: endpoint >> > > > uri: log:info >> > > > ------------------- >> > > > >> > > > For those who are not familiar with Kubernetes resources, this kind >> of >> > > > YAML/JSON resource definitions are really common. >> > > > The example route is embedded in the Kubernetes resource declaration >> > and >> > > > follows a basic "flow DSL". We may start from a basic one and evolve >> it >> > > as >> > > > new requirements arrive from the community. >> > > > >> > > > I've made a very simple (but working) POC here: >> > > > https://github.com/nicolaferraro/integration-operator. >> > > > >> > > > This idea of a "Cloud-Native Camel" on Kubernetes (project codename >> can >> > > be " >> > > > *Kamel*", if you like it :D), will be an enabler for a lot of nice >> > > features. >> > > > >> > > > For example, we can propose "Kamel" as "ideal" platform for >> "serverless >> > > > integration" (I see many people reinventing the wheel out there): the >> > > > operator can reduce resource consumption of a single integration by >> > > > optimizing the runtime and also pause/resume integrations when they >> are >> > > not >> > > > used, that is the basic idea behind "serverless" (e.g. think to >> > > > HTTP-triggered integrations, but not only). >> > > > Focusing on serverless will bring more emphasis on push-based >> > > notifications >> > > > (webhooks, cloud events <https://cloudevents.io/>), that are rarely >> > > used in >> > > > Camel components, that prefer a poll based approach being it simpler >> to >> > > use >> > > > in classic deployments, but not so good in the cloud, where more >> > > resources >> > > > become higher direct costs for the users. >> > > > >> > > > The presence of the simplified DSL enables also experimenting on >> > > "*reduced* >> > > > subsets of Camel" implemented in languages other than Java, for >> example >> > > one >> > > > language that has a reactive approach on thread scheduling and a >> really >> > > low >> > > > memory footprint, like Go. >> > > > >> > > > But apart from this kind of experiments (that are valid IMO), the >> > "Kamel" >> > > > optimizer will have free room to choose the right platform for the >> > > > integration that the user wants to run, including, in the future, >> doing >> > > AOT >> > > > compilation using Graal/VM (less memory, faster startup) if the >> > features >> > > > (components) used in the integration are supporting it (maybe we can >> > add >> > > > AOT compilation in the roadmap for Camel 3). >> > > > A silly optimization: integrations starting from "timer:..." may be >> > > > scheduled directly with Kubernetes CronJobs, so they will consume >> > > resources >> > > > only when actually running. >> > > > >> > > > Being the final integrations lightweight and being the DSL >> > > > language-independent, we may see a increased adoption of Camel also >> as >> > > > agile integration layer for not-only-java applications (both "cloud" >> > and >> > > > "serverless" applications). >> > > > >> > > > I'm the first one that would like to work on a project ilke this. >> I've >> > > > worked on many Kubernetes/Openshift based applications and frameworks >> > in >> > > > the past years, also on operators and CRDs, and I think this way of >> > > > redesigning integrations has a lot of potential. >> > > > >> > > > Integrations will not be necessarily limited to the simplified DSL, >> but >> > > we >> > > > can add extension points for scripting and even custom libraries >> > > (although >> > > > limiting the freedom of the optimizer). >> > > > >> > > > The most important thing: it may become a great project, since it's >> > > driven >> > > > by a great community. >> > > > >> > > > So, what do you think? Is it crazy enough? >> > > > >> > > > Nicola >> > > >> > > >> > >> > -- >> > Hiram Chirino >> > Engineering | Red Hat, Inc. >> > hchir...@redhat.com | fusesource.com | redhat.com >> > skype: hiramchirino | twitter: @hiramchirino >> > >>
