Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2502#discussion_r167402753
--- Diff:
storm-client/src/jvm/org/apache/storm/messaging/netty/BackPressureStatus.java
---
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.storm.messaging.netty;
+
+import org.apache.storm.serialization.KryoValuesDeserializer;
+import org.apache.storm.serialization.KryoValuesSerializer;
+import org.jboss.netty.buffer.ChannelBuffer;
+import org.jboss.netty.buffer.ChannelBuffers;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.concurrent.atomic.AtomicLong;
+
+// Instances of this type are sent from NettyWorker to upstream
WorkerTransfer to indicate BackPressure situation
+public class BackPressureStatus implements java.io.Serializable {
--- End diff --
ControlMessage, MessageBatch and SaslMessageToken are handled explicitly by
MessageDecoder and MessageEncoder using the buffer and read methods.
When ControlMessage, SaslMessageToken, or Message Batch writes themselves
out to a buffer they do not use kryo to do it.
https://github.com/apache/storm/blob/aaebc3b237916340156ac3b8dc956d6c62c34983/storm-client/src/jvm/org/apache/storm/messaging/netty/ControlMessage.java#L60-L64
https://github.com/apache/storm/blob/aaebc3b237916340156ac3b8dc956d6c62c34983/storm-client/src/jvm/org/apache/storm/messaging/netty/SaslMessageToken.java#L86-L101
https://github.com/apache/storm/blob/aaebc3b237916340156ac3b8dc956d6c62c34983/storm-client/src/jvm/org/apache/storm/messaging/netty/MessageBatch.java#L80-L93
It is a hand coded protocol (for good or bad).
The messages inside MessageBatch have already been serialized using kryo
elsewhere in the pipeline.
For the load messages we hid them as a tuple inside a MessageBatch. We did
this so we could do a rolling upgrade with it, but it is an ugly hack.
BackPressureStatus is serialized/deserialized using
MessageEncoder/MessageDecoder, but it uses kryo internally to do it.
So please remove Serializable from BackPressureStatus. Register it with
kryo. Do not remove `BackPressureStatus.buffer()` nor
`BackPressureStatus.read()`. I think the changes to KryoTupleSerializer to
send a BackPressureState can be removed assuming no one is calling them
directly.
The simplest way to test this is to run a topology with multiple workers
and `topology.fall.back.on.java.serialization=false`. This is a config that
makes it so kryo does not fall back to java serialization when trying to use
kryo.
For normal tuples/end users you can set
`topology.testing.always.try.serialize=true` and every tuple emitted will be
serialized, even in local mode. This is a way to unit test that you have setup
kryo appropriately, but this BackPressureStatus is a special message so we need
to do it differently.
---
