DaanHoogland commented on code in PR #14677:
URL: https://github.com/apache/cloudstack/pull/14677#discussion_r4642622469


##########
plugins/event-bus/rabbitmq/src/main/java/org/apache/cloudstack/mom/rabbitmq/RabbitMQEventBus.java:
##########
@@ -177,6 +190,14 @@ public static void setRetryInterval(Integer retryInterval) 
{
         RabbitMQEventBus.retryInterval = retryInterval;
     }
 
+    public static void setBufferSize(Integer bufferSize) {
+        RabbitMQEventBus.bufferSize = bufferSize;
+    }

Review Comment:
   ```suggestion
       /**
        * maximum size to which the event buffer may grow
        */
       public static void setBufferSize(Integer bufferSize) {
           maxEventBufferSize = bufferSize;
       }
   ```



##########
plugins/event-bus/rabbitmq/src/main/java/org/apache/cloudstack/mom/rabbitmq/RabbitMQEventBus.java:
##########
@@ -85,11 +86,23 @@ public static void setUseSsl(String useSsl) {
 
     private static Integer retryInterval;
 
+    // max number of events kept in memory while the AMQP server is 
unreachable, 0 disables buffering
+    private static Integer bufferSize = 1000;
+
+    // time in milliseconds after which a buffered event is dropped if it 
could not be sent
+    private static Integer bufferExpiry = 300000;
+
+    // events (routing key, description, time of buffering) waiting to be 
sent, oldest first
+    private static final ArrayDeque<Ternary<String, String, Long>> s_buffer = 
new ArrayDeque<>();

Review Comment:
   ```suggestion
       /**
        * events, stored as {routing key, description, time of buffering}, 
waiting to be sent, oldest first
        */
       private static final ArrayDeque<Ternary<String, String, Long>> 
eventBuffer = new ArrayDeque<>();
   ```
   
   needs renaming below as well!



##########
plugins/event-bus/rabbitmq/src/main/java/org/apache/cloudstack/mom/rabbitmq/RabbitMQEventBus.java:
##########
@@ -85,11 +86,23 @@ public static void setUseSsl(String useSsl) {
 
     private static Integer retryInterval;
 
+    // max number of events kept in memory while the AMQP server is 
unreachable, 0 disables buffering
+    private static Integer bufferSize = 1000;

Review Comment:
   ```suggestion
       private static Integer maxEventBufferSize = 1000;
   ```



##########
plugins/event-bus/rabbitmq/src/main/java/org/apache/cloudstack/mom/rabbitmq/RabbitMQEventBus.java:
##########
@@ -85,11 +86,23 @@ public static void setUseSsl(String useSsl) {
 
     private static Integer retryInterval;
 
+    // max number of events kept in memory while the AMQP server is 
unreachable, 0 disables buffering
+    private static Integer bufferSize = 1000;
+
+    // time in milliseconds after which a buffered event is dropped if it 
could not be sent
+    private static Integer bufferExpiry = 300000;

Review Comment:
   ```suggestion
       private static Integer eventBufferExpiryTime = 300000;
   ```



##########
plugins/event-bus/rabbitmq/src/main/java/org/apache/cloudstack/mom/rabbitmq/RabbitMQEventBus.java:
##########
@@ -85,11 +86,23 @@ public static void setUseSsl(String useSsl) {
 
     private static Integer retryInterval;
 
+    // max number of events kept in memory while the AMQP server is 
unreachable, 0 disables buffering
+    private static Integer bufferSize = 1000;
+
+    // time in milliseconds after which a buffered event is dropped if it 
could not be sent
+    private static Integer bufferExpiry = 300000;
+
+    // events (routing key, description, time of buffering) waiting to be 
sent, oldest first
+    private static final ArrayDeque<Ternary<String, String, Long>> s_buffer = 
new ArrayDeque<>();
+
+    // events are buffered only after the AMQP server was reachable at least 
once
+    private static volatile boolean s_connectedOnce = false;
+
     // hashmap to book keep the registered subscribers
     private static ConcurrentHashMap<String, Ternary<String, Channel, 
EventSubscriber>> s_subscribers;
 
     // connection to AMQP server,
-    private static Connection s_connection = null;
+    private static volatile Connection s_connection = null;

Review Comment:
   now we're touching it let's rename to `connection`!?



##########
plugins/event-bus/rabbitmq/src/main/java/org/apache/cloudstack/mom/rabbitmq/RabbitMQEventBus.java:
##########
@@ -273,20 +294,106 @@ public void publish(Event event) throws 
EventBusException {
         String routingKey = createRoutingKey(event);
         String eventDescription = event.getDescription();
 
+        // keep the order: nothing is sent directly while older events are 
waiting or the connection is down
+        if (bufferEvent(routingKey, eventDescription, false)) {
+            return;
+        }
+
         try {
-            Connection connection = getConnection();
-            Channel channel = createChannel(connection);
-            createExchange(channel, amqpExchangeName);
-            publishEventToExchange(channel, amqpExchangeName, routingKey, 
eventDescription);
-            channel.close();
+            sendEvent(getConnection(), routingKey, eventDescription);
         } catch (AlreadyClosedException e) {
             closeConnection();
+            if (bufferEvent(routingKey, eventDescription, true)) {
+                return;
+            }
             throw new EventBusException("Failed to publish event to message 
broker as connection to AMQP broker in lost");
+        } catch (IOException e) {
+            if (bufferEvent(routingKey, eventDescription, true)) {
+                return;
+            }
+            throw new EventBusException("Failed to publish event to message 
broker due to " + e.getMessage());
         } catch (Exception e) {
             throw new EventBusException("Failed to publish event to message 
broker due to " + e.getMessage());
         }
     }
 
+    private void sendEvent(Connection connection, String routingKey, String 
eventDescription) throws Exception {
+        Channel channel = createChannel(connection);
+        createExchange(channel, amqpExchangeName);
+        publishEventToExchange(channel, amqpExchangeName, routingKey, 
eventDescription);
+        channel.close();
+    }
+
+    /** adds the event to the buffer, dropping the oldest one if the buffer is 
full
+     *
+     * @param sendFailed false if the event was not tried yet and has to be 
buffered only to keep the order
+     * @return true if the event was buffered, false if the caller has to deal 
with it
+     */
+    private boolean bufferEvent(String routingKey, String eventDescription, 
boolean sendFailed) {
+        if (!s_connectedOnce || bufferSize <= 0) {
+            return false;
+        }
+        synchronized (s_buffer) {
+            if (!sendFailed && s_buffer.isEmpty() && s_connection != null) {
+                return false;
+            }
+            if (s_buffer.isEmpty()) {
+                logger.warn("Unable to send events to AMQP server, buffering 
them until they can be sent");
+            }
+            if (s_buffer.size() >= bufferSize) {
+                logger.warn("Event buffer is full, dropping the oldest event " 
+ s_buffer.poll().first());
+            }
+            s_buffer.add(new Ternary<>(routingKey, eventDescription, 
System.currentTimeMillis()));
+        }
+        return true;
+    }
+
+    // drops the expired events and, if connected, sends the buffered ones in 
the order they were added

Review Comment:
   top of method comment:
   ```suggestion
       /**
        * drops the expired events and, if connected, sends the buffered ones 
in the order they were added
       */
   ```



##########
plugins/event-bus/rabbitmq/src/main/java/org/apache/cloudstack/mom/rabbitmq/RabbitMQEventBus.java:
##########
@@ -540,35 +649,38 @@ public void shutdownCompleted(ShutdownSignalException 
shutdownSignalException) {
                 }
 
                 abortConnection(); // disconnected to AMQP server, so abort 
the connection and channels
+                // the ReconnectionTask started with the event bus takes care 
of the re-connect

Review Comment:
   ```suggestion
   ```



##########
plugins/event-bus/rabbitmq/src/main/java/org/apache/cloudstack/mom/rabbitmq/RabbitMQEventBus.java:
##########
@@ -273,20 +294,106 @@ public void publish(Event event) throws 
EventBusException {
         String routingKey = createRoutingKey(event);
         String eventDescription = event.getDescription();
 
+        // keep the order: nothing is sent directly while older events are 
waiting or the connection is down
+        if (bufferEvent(routingKey, eventDescription, false)) {
+            return;
+        }
+
         try {
-            Connection connection = getConnection();
-            Channel channel = createChannel(connection);
-            createExchange(channel, amqpExchangeName);
-            publishEventToExchange(channel, amqpExchangeName, routingKey, 
eventDescription);
-            channel.close();
+            sendEvent(getConnection(), routingKey, eventDescription);
         } catch (AlreadyClosedException e) {
             closeConnection();
+            if (bufferEvent(routingKey, eventDescription, true)) {
+                return;
+            }
             throw new EventBusException("Failed to publish event to message 
broker as connection to AMQP broker in lost");
+        } catch (IOException e) {
+            if (bufferEvent(routingKey, eventDescription, true)) {
+                return;
+            }
+            throw new EventBusException("Failed to publish event to message 
broker due to " + e.getMessage());
         } catch (Exception e) {
             throw new EventBusException("Failed to publish event to message 
broker due to " + e.getMessage());
         }
     }
 
+    private void sendEvent(Connection connection, String routingKey, String 
eventDescription) throws Exception {
+        Channel channel = createChannel(connection);
+        createExchange(channel, amqpExchangeName);
+        publishEventToExchange(channel, amqpExchangeName, routingKey, 
eventDescription);
+        channel.close();
+    }
+
+    /** adds the event to the buffer, dropping the oldest one if the buffer is 
full
+     *
+     * @param sendFailed false if the event was not tried yet and has to be 
buffered only to keep the order
+     * @return true if the event was buffered, false if the caller has to deal 
with it
+     */
+    private boolean bufferEvent(String routingKey, String eventDescription, 
boolean sendFailed) {
+        if (!s_connectedOnce || bufferSize <= 0) {
+            return false;
+        }
+        synchronized (s_buffer) {
+            if (!sendFailed && s_buffer.isEmpty() && s_connection != null) {
+                return false;
+            }
+            if (s_buffer.isEmpty()) {
+                logger.warn("Unable to send events to AMQP server, buffering 
them until they can be sent");
+            }
+            if (s_buffer.size() >= bufferSize) {
+                logger.warn("Event buffer is full, dropping the oldest event " 
+ s_buffer.poll().first());
+            }
+            s_buffer.add(new Ternary<>(routingKey, eventDescription, 
System.currentTimeMillis()));
+        }
+        return true;
+    }
+
+    // drops the expired events and, if connected, sends the buffered ones in 
the order they were added
+    private void flushBuffer() {
+        int sent = 0;
+        while (true) {
+            Ternary<String, String, Long> bufferedEvent;
+            synchronized (s_buffer) {
+                bufferedEvent = s_buffer.peek();
+            }
+            if (bufferedEvent == null) {
+                break;
+            }
+
+            boolean expired = System.currentTimeMillis() - 
bufferedEvent.third() > bufferExpiry;
+            Connection connection = s_connection;
+            if (!expired && connection == null) {
+                break;
+            }
+
+            if (expired) {
+                logger.warn("Dropping expired event " + bufferedEvent.first() 
+ " as it could not be sent to AMQP server in time");
+            } else {
+                try {
+                    sendEvent(connection, bufferedEvent.first(), 
bufferedEvent.second());
+                    sent++;
+                } catch (AlreadyClosedException e) {
+                    closeConnection();
+                    break;
+                } catch (IOException e) {
+                    break; // retry with the next run
+                } catch (Exception e) {
+                    logger.warn("Dropping buffered event " + 
bufferedEvent.first() + " due to " + e.getMessage());
+                }
+            }
+
+            // the event stays in the buffer while being sent, so that newer 
events queue up behind it
+            synchronized (s_buffer) {
+                if (s_buffer.peek() == bufferedEvent) {
+                    s_buffer.poll();
+                }

Review Comment:
   can you extract this to a separate method? sounds trivial but it makes for 
better reading



##########
plugins/event-bus/rabbitmq/src/main/java/org/apache/cloudstack/mom/rabbitmq/RabbitMQEventBus.java:
##########
@@ -540,35 +649,38 @@ public void shutdownCompleted(ShutdownSignalException 
shutdownSignalException) {
                 }
 
                 abortConnection(); // disconnected to AMQP server, so abort 
the connection and channels
+                // the ReconnectionTask started with the event bus takes care 
of the re-connect
                 logger.warn("Connection has been shutdown by AMQP server. 
Attempting to reconnect.");
-
-                // initiate re-connect process
-                ReconnectionTask reconnect = new ReconnectionTask();
-                executorService.submit(reconnect);
             }
         }
     }
 
-    // retry logic to connect back to AMQP server after loss of connection
+    // runs for the lifetime of the event bus: connects back to AMQP server 
after any loss of connection
+    // and sends the events buffered in the meantime

Review Comment:
   ```suggestion
       /*
        * runs for the lifetime of the event bus: connects back to AMQP server 
after any loss of connection
        * and sends the events buffered in the meantime
       */
   ```



##########
plugins/event-bus/rabbitmq/src/main/java/org/apache/cloudstack/mom/rabbitmq/RabbitMQEventBus.java:
##########
@@ -85,11 +86,23 @@ public static void setUseSsl(String useSsl) {
 
     private static Integer retryInterval;
 
+    // max number of events kept in memory while the AMQP server is 
unreachable, 0 disables buffering
+    private static Integer bufferSize = 1000;
+
+    // time in milliseconds after which a buffered event is dropped if it 
could not be sent
+    private static Integer bufferExpiry = 300000;
+
+    // events (routing key, description, time of buffering) waiting to be 
sent, oldest first
+    private static final ArrayDeque<Ternary<String, String, Long>> s_buffer = 
new ArrayDeque<>();
+
+    // events are buffered only after the AMQP server was reachable at least 
once
+    private static volatile boolean s_connectedOnce = false;

Review Comment:
   this comment confuses more than explaining (for me) why is it private static 
(only one connection possible is a lack of feature I think but that would be a 
reason).
   Please drop the comment or make it truely prozaic.
   
   Also the naming convention of 's_' is obsolete and no reason to keep it up.
   
   It seems this is nnever set back to false so I'd make it something like
   
   ```suggestion
       /**
        * indicates wether a connection has ever been made, which is acted on 
by checking that events are buffered only after the AMQP server was reachable 
at least once
        */
       private static volatile boolean hasBeenConnectedOnce = false;
   ```
   
   needs renaming elsewhere as well.



##########
plugins/event-bus/rabbitmq/src/main/java/org/apache/cloudstack/mom/rabbitmq/RabbitMQEventBus.java:
##########
@@ -177,6 +190,14 @@ public static void setRetryInterval(Integer retryInterval) 
{
         RabbitMQEventBus.retryInterval = retryInterval;
     }
 
+    public static void setBufferSize(Integer bufferSize) {
+        RabbitMQEventBus.bufferSize = bufferSize;
+    }
+
+    public static void setBufferExpiry(Integer bufferExpiry) {
+        RabbitMQEventBus.bufferExpiry = bufferExpiry;
+    }

Review Comment:
   ```suggestion
       /**
        * expriy time to set for events that cannot be send
        * 
        */
       public static void setBufferExpiry(Integer bufferExpiry) {
           eventBufferExpiryTime = bufferExpiry;
       }
   ```



##########
plugins/event-bus/rabbitmq/src/main/java/org/apache/cloudstack/mom/rabbitmq/RabbitMQEventBus.java:
##########
@@ -540,35 +649,38 @@ public void shutdownCompleted(ShutdownSignalException 
shutdownSignalException) {
                 }
 
                 abortConnection(); // disconnected to AMQP server, so abort 
the connection and channels
+                // the ReconnectionTask started with the event bus takes care 
of the re-connect
                 logger.warn("Connection has been shutdown by AMQP server. 
Attempting to reconnect.");
-
-                // initiate re-connect process
-                ReconnectionTask reconnect = new ReconnectionTask();
-                executorService.submit(reconnect);
             }
         }
     }
 
-    // retry logic to connect back to AMQP server after loss of connection
+    // runs for the lifetime of the event bus: connects back to AMQP server 
after any loss of connection
+    // and sends the events buffered in the meantime
     private class ReconnectionTask extends ManagedContextRunnable {
 
-        boolean connected = false;
         Connection connection = null;
 
         @Override
         protected void runInContext() {
 
-            while (!connected) {
+            while (true) {

Review Comment:
   not all of this is on you, @durdin85 , but this is a forever loop with 
nested try blocks and an anonymous class inside. Could you at least put the new 
part (the sync block in a separate method? If you have the energy, feel free to 
simplify it further.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to