[
https://issues.apache.org/jira/browse/STORM-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485882#comment-14485882
]
Robert Joseph Evans commented on STORM-760:
-------------------------------------------
Looking at what LocalState is currently used for I am a bit nervous that we
might be in trouble.
The following are some examples of what I found being written to a LocalState
{code}
{local-assignments={1024
#backtype.storm.daemon.supervisor.LocalAssignment{:storm-id
"topologytest-0e9ff9c4-2e3a-403d-b334-0ce687bcee1a-1-0", :executors ([8 8] [12
12] [2 2] [7 7] [3 3] [1 1] [6 6] [9 9] [11 11] [5 5] [10 10] [4 4])}},
approved-workers={"ce732ff5-761c-42b5-ba3b-3f5f3c6e78ad" 1024}}
{worker-heartbeat={:time-secs 1, :storm-id
"topologytest-a497a6c9-1eda-4abb-858f-4c276d82c1ff-1-0", :executors #{[8 8] [12
12] [2 2] [7 7] [3 3] [1 1] [6 6] [9 9] [11 11] [-1 -1] [13 13] [5 5] [10 10]
[4 4]}, :port 1024}}
{code}
Even though LocalState is storing a map. Inside the map are clojure
{persistent maps, symbols, lists, vectors, and sets} There is also a
LocalAssignment (which is a defrecord, that will have a different serialization
id if ever modified). I'm not sure if we want to convert all of this to thrift
too? or if we just want to move LocalAssignment over to be a map. If we think
clojure will not maintain serialized compatability, then we probably want to go
to thrift. If we think that they will maintain compatibility, then we could do
something much more simply.
In our experience at least, when we moved from clojure 1.5 to 1.6 we saw the
following errors during rolling upgrade testing (reading in local state).
{code}
java.lang.RuntimeException: java.io.InvalidClassException:
clojure.lang.APersistentVector; local class incompatible: stream classdesc
serialVersionUID = -2689426146225009349, local class serialVersionUID =
4667575149454420891
{code}
To me this means that clojure is not trying to maintain compatability for
serialized objects, like vector. So I personally would rather move to Thrift
for the serialization of local state if possible. I am going to file a new
JIRA just for this work, as it is large enough I think it should be separated
from this JIRA.
> use JSON for conf serialization
> -------------------------------
>
> Key: STORM-760
> URL: https://issues.apache.org/jira/browse/STORM-760
> Project: Apache Storm
> Issue Type: Bug
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
>
> Now that STORM-634 has gone in, the only real place we still use java
> serialization that is not required by the contract with the end user is when
> nimbus writes out the topology conf, and the worker/supervisor tries to read
> it back in. We already write it out using JSON when submitting a topology,
> we should do the same here to avoid rolling upgrade issues, ideally
> compressed to save space too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
