Hi Nicola, I love the idea.
I just wonder whether YAML/JSON is an expressive enough format in the long term. But as you’ve mentioned, starting simple would enable experimenting some very interesting / promising optimisations. So it seems worth taking that path, instead of trying to embed a complex DSL or the existing XML DSL into the CRD. Definitely +1 > On 13 Jul 2018, at 01:30, Nicola Ferraro <ni.ferr...@gmail.com> wrote: > > Hi Cameleers, > it's now passed some time since I started thinking about a new project that > we can begin here at Apache Camel, and I'd like to have your opinion. > > We've already been targeting cloud-native applications with Camel, > especially on top of Kubernetes, that is becoming "the standard" cloud > platform. But writing a Camel integration and running it on Kubernetes > requires some effort: choosing the base platform (spring-boot, karaf, > simple main?), adding health checks (actuator?), packaging a docker image > and creating the Kubernetes resources (fabric8-maven-plugin, helm?), > publishing the image on a docker registry, then finally deploying the > resources on a Kubernetes cluster. > > The resulting integration container is then far from being optimal from a > resource consumption point of view: it is likely that a Camel Spring-Boot > application will require at least 200MB of RAM and also some CPU shares > because of polling threads used by many components. > > In case people use a CI/CD pipeline, it will take also a long time to get > from a code update to having a Kubernetes POD up and running. > Apart from compilation and image push/pull time, also startup time is often > ~10 seconds for Camel + Spring-Boot in a container with standard limits on > resources, making it difficult to propose this combination for "serverless > integration" (this term is becoming increasingly more popular). > > So, my proposal is to start to investigate a "more cloud-native" approach > to integration: *making Camel integrations first-class citizens in > Kubernetes, and making them super fast and lightweight.* > > We can base the project on Kubernetes Custom Resource Definitions (CRD) > <https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/>, > for example a Integration CRD and have a Kubernetes "operator" > <https://coreos.com/operators/> taking care of: > - Optimizing the integration that we want to run > - Packaging in a container > - Running it on Kubernetes > - Managing its entire lifecycle > > A Kubernetes-native integration may look like: > > ------------------- > kind: "Integration" > apiVersion: "camel.apache.org/v1alpha1" > metadata: > name: "example" > spec: > replicas: 1 > routes: > - id: timer > route: > - type: endpoint > uri: timer:tick > - type: endpoint > uri: log:info > ------------------- > > For those who are not familiar with Kubernetes resources, this kind of > YAML/JSON resource definitions are really common. > The example route is embedded in the Kubernetes resource declaration and > follows a basic "flow DSL". We may start from a basic one and evolve it as > new requirements arrive from the community. > > I've made a very simple (but working) POC here: > https://github.com/nicolaferraro/integration-operator. > > This idea of a "Cloud-Native Camel" on Kubernetes (project codename can be " > *Kamel*", if you like it :D), will be an enabler for a lot of nice features. > > For example, we can propose "Kamel" as "ideal" platform for "serverless > integration" (I see many people reinventing the wheel out there): the > operator can reduce resource consumption of a single integration by > optimizing the runtime and also pause/resume integrations when they are not > used, that is the basic idea behind "serverless" (e.g. think to > HTTP-triggered integrations, but not only). > Focusing on serverless will bring more emphasis on push-based notifications > (webhooks, cloud events <https://cloudevents.io/>), that are rarely used in > Camel components, that prefer a poll based approach being it simpler to use > in classic deployments, but not so good in the cloud, where more resources > become higher direct costs for the users. > > The presence of the simplified DSL enables also experimenting on "*reduced* > subsets of Camel" implemented in languages other than Java, for example one > language that has a reactive approach on thread scheduling and a really low > memory footprint, like Go. > > But apart from this kind of experiments (that are valid IMO), the "Kamel" > optimizer will have free room to choose the right platform for the > integration that the user wants to run, including, in the future, doing AOT > compilation using Graal/VM (less memory, faster startup) if the features > (components) used in the integration are supporting it (maybe we can add > AOT compilation in the roadmap for Camel 3). > A silly optimization: integrations starting from "timer:..." may be > scheduled directly with Kubernetes CronJobs, so they will consume resources > only when actually running. > > Being the final integrations lightweight and being the DSL > language-independent, we may see a increased adoption of Camel also as > agile integration layer for not-only-java applications (both "cloud" and > "serverless" applications). > > I'm the first one that would like to work on a project ilke this. I've > worked on many Kubernetes/Openshift based applications and frameworks in > the past years, also on operators and CRDs, and I think this way of > redesigning integrations has a lot of potential. > > Integrations will not be necessarily limited to the simplified DSL, but we > can add extension points for scripting and even custom libraries (although > limiting the freedom of the optimizer). > > The most important thing: it may become a great project, since it's driven > by a great community. > > So, what do you think? Is it crazy enough? > > Nicola
