Re: Manage and monitor a group of routes

[ski n]

Thanks for the clear answer.

Here are some additional remarks:

1. Statistics

The grouped statistics would be very cool.

2. Events

I currently already use the Event notifier to gather events on routes and
exchanges. This is however limited imo.

A. Route Events

Say I have an integration that consists of around 100 routes. I can get the
following events:

RouteAddedEvent
RouteEvent
RouteReloadedEvent
RouteRemovedEvent
RouteStartedEvent
RouteStartingEvent
RouteStoppedEvent
RouteStoppingEvent

Here I miss the following events:

RouteSuspendEvent
RouteResumeEvent

And failures

RouteStartFailureEvent
RouteStartFailureEvent
RouteStopFailureEvent
RouteResumeFailureEvent
RouteResumeFailureEvent
RouteSuspendFailureEvent

So then I would have similar kind of events on a route level like they are
on context level.

Additionally, it would be nice to get the same information not only about
all the 100 routes individually, but also on a higher level (the route
group of 'flow').

B) Exchange events

I am currently also using ExchangeEvents, but this is limited useful in my
use case. For example when using multiple routes connected by the direct
component then there is one ExchangeStarted and one ExchangeCompleted event
(over all the routes). But when using JMS Queues in between every route,
then every route has an ExchangeStarted and ExchangeCompleted event. From
an exchange lifecycle this is logical, but it's hard to create programming
logic on top of it, when it works differently for several cases.

What I am looking for is:

RouteMessageInEvent   (returning the exchange)
RouteMessageOutEvent (returning the exchange)
RouteMessageFailureEvent (returning the exchange)

and

RouteGroupMessageInEvent   (returning the exchange)
RouteGroupMessageOutEvent (returning the exchange)
RouteGroupMessageFailureEvent (returning the exchange)


3. Manage a group of route

In my last post I added some pseudo code, but to be clear I also already
created some real code for it. What it does:

1. Each route in a group (called a flow) is loaded (added & started).
2. For each loaded route, info is added to a load report (a json).
3. An API is available to start/stop/pause a flow (a group of routes)
4. Caller of API gets a JSON report on start back, if it fails the other
routes are stopped/rolled back.

Here example JSON of the load report (including a failure):

 {"flow": {
"stepsLoaded": {
"total": 3,
"successful": 2,
"failed": 1
},
"environment": "",
"name": "ID_627a57f538c74a000e00060a",
"id": "ID_627a57f538c74a000e00060a",
"event": "error",
"message": "Failed to load flow",
"version": "",
"steps": [
{
"id": "3747809ef661",
"type": "route",
"status": "success"
},
{
"id": "3747809ef661",
"type": "route",
"status": "success"
},
{
"errorMessage": "Failed to create route
ID_627a57f538c74a000e00060a-47a06ca1-d05b-11ec-83f5-3747809ef661:
Route(ID_627a57f538c74a000e00060a-47a06ca1-d05b-11ec-83f5-37... because of
No endpoint could be found for:
xdirect://ID_627a57f538c74a000e00060a_test_3ffded10-d05b-111ec-83f5-3747809ef661,
please check your classpath contains the needed Camel component jar.",
"id": "3747809ef661",
"type": "route",
"status": "error"
}
]
}}

Here the code used (11/02/2023):

Load routes (
https://github.com/assimbly/runtime/blob/develop/dil/src/main/java/org/assimbly/dil/loader/FlowLoader.java
)
Load routes report (
https://github.com/assimbly/runtime/blob/develop/dil/src/main/java/org/assimbly/dil/loader/FlowLoaderReport.java
)
Manage routes (
https://github.com/assimbly/runtime/blob/develop/integration/src/main/java/org/assimbly/integration/impl/CamelIntegration.java)
--> check for startFlow / stopFlow methods

So I have three levels

1. integration (= context = one or more flows)
2. flow (= group of routes)
3. step (=route)

So I created a level in between the context and route. In Camel2 this was
more easily to achieve, because you can have multiple CamelContexts, but
since Camel3 there is a bit of a gap between the CamelContext and Routes.

Though this works fairly well for my purposes, I think when a more generic
way in the framework would be implemented, a lot more people would benefit.
Though that's more for the developer mailing list, than the user mailing
list, I guess. This was just to provide some feedback and ideas.

Raymond







On Sat, Feb 11, 2023 at 10:38 AM Claus Ibsen  wrote:

> Hi
>
> The camel-management already captures on
> - context
> - route
> - processor
>
> levels for each statistics. So you have how many messages are processed by
> routes.
> You can associate a route to a route. And you can see the route in the
> statistics, so your monitoring systems can aggregate data together.
>
> However I think it would also be nice if Camel can do that
> 

Re: Manage and monitor a group of routes

[Claus Ibsen]

Hi

The camel-management already captures on
- context
- route
- processor

levels for each statistics. So you have how many messages are processed by
routes.
You can associate a route to a route. And you can see the route in the
statistics, so your monitoring systems can aggregate data together.

However I think it would also be nice if Camel can do that
https://issues.apache.org/jira/browse/CAMEL-19038

That is on a high level with statistics only (numbers).

For actual message tracing then you can use event notifier, route policy,
message history etc.

To start|stop a collection of routes (in a group) is not possible out of
the box, and you need to do that manually.
Like your pseudo code.





On Fri, Feb 10, 2023 at 6:13 PM ski n  wrote:

> Hi Cameleers,
>
> I'm wondering what the best way is to manage and monitor a group of routes
> as an integration ('a unit of work')?
> Through the mailing list, I try to gather some ideas on this topic.
>
>
> Background: Integration vs Routes
>
> Say I have three routes that together form my integration:
>
> http://camel.apache.org/schema/spring;>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
> So, the work processing messages is split up into three routes, but from a
> high-level view it's just one integration.
>
> From a Camel point of view the order doesn't matter, so I can create the
> same routes also like this:
>
> http://camel.apache.org/schema/spring;>
> 
> 
> 
> 
>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
> Because the order doesn't matter for Camel the integration still works the
> same.
> So messages start at the quartz component and then flow through the others
> routes.
>
>
> What I am trying to achieve:
>
> I was wondering how to manage and monitoring such integrations (route
> groups) at once. Here is some pseudo-code how I imagine it:
>
> RoutesGroup myRoutesGroup = context.getRouteGroup("myRoutesGroup");
> myRoutesGroup.suspend(); //suspends all routes in a group
> myRoutesGroup.start(); //starts all routes in a group
> myRoutesGroup.stop(); //stops all routes in a group
> myRoutesGroup.status(); //get the status from group
>
> Besides managing the integration I also like to monitor it as a group. For
> example to get:
>
> 1. Number or Content of messages processed by a group (in and out messages)
> 2. Number or Content of messages processed by a single route in the group
> (in and out message).
>
> With the second, I mean messages on a route level. What message goes into a
> route and what messages go out. This in contrast to exchanges, because
> exchanges create events on a lower level. In this example I am using direct
> component, so that would create the following exchange events:
>
> 1. First route: ExchangeCreated, some ExchangeSending/ExchangeSent events
> 2. Second route: Only sending/sent events
> 3. Third route: Sending/sent events and the completed event.
>
> From an integration point of view, I am only interested in what goes in and
> out the integration and how many messages that are. Or I am interested
> what messages are going into and out a single route (as a processing step).
>
> Is there a way to work with integrations opposed to routes? What are the
> possibilities in Camel?
>
> Raymond
>


-- 
Claus Ibsen
-
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2