It isn't the jar file, but something about the topology itself; I have a
submitter program that submits four topologies all from the same jar. Upon
submitting the first topology, the jar is uploaded and topology starts,
then the submitter submits two more topologies whilst "reusing" the
uploaded jar. The broken pipe occurs when trying to submit the fourth
(large) topology. That is why I was assuming the large message was
actually the encoded topology itself. This is reproducible and the errors
are as follows:
nimbus.log:
2014-03-18 18:16:39 o.a.t.s.TNonblockingServer [ERROR] Read a frame size of
> 6644632, which is bigger than the maximum allowable buffer size for ALL
> connections.
storm jar console:
> 2321 [main] INFO backtype.storm.StormSubmitter - Uploading topology jar
>> /Users/adam/git/impl/impl-storm/target/impl-storm-0.0.1-SNAPSHOT.jar to
>> assigned location:
>> /mnt/storm/nimbus/inbox/stormjar-04acf27a-a6d1-4a9e-9231-9f4f5f30fd03.jar
>
> 97762 [main] INFO backtype.storm.StormSubmitter - Successfully uploaded
>> topology jar to assigned location:
>> /mnt/storm/nimbus/inbox/stormjar-04acf27a-a6d1-4a9e-9231-9f4f5f30fd03.jar
>
> 97762 [main] INFO backtype.storm.StormSubmitter - Submitting topology
>> global__topo_forecastRuntime in distributed mode with conf
>> {"topology.fall.back.on.java.serialization":false,"topology.workers":2,"drpc.servers":["10.118.57.229"],"topology.debug":false,"topology.kryo.register":[{"org.joda.time.DateTime":"de.javakaffee.kryoserializers.jodatime.JodaDateTimeSerializer"},{"org.joda.time.Interval":"de.javakaffee.kryoserializers.jodatime.JodaIntervalSerializer"},"com.mycompany.data.Simulated","com.mycompany.data.SomeClass1","com.mycompany.ml.SomeClass2","com.mycompany.model.SomeClass3","com.mycompany.model.SomeClass4",{"com.mycompany.ml.SomeClass4":"com.esotericsoftware.kryo.serializers.DefaultSerializers$EnumSerializer"},{"java.math.BigDecimal":"com.esotericsoftware.kryo.serializers.DefaultSerializers$BigDecimalSerializer"},{"java.sql.Date":"de.javakaffee.kryoserializers.DateSerializer"},{"com.tdunning.math.stats.TDigest":"com.mycompany.trident.tdigest.TDigestSerializer"},{"java.lang.Class":"com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer"},{"java.util.UUID":"de.javakaffee.kryoserializers.UUIDSerializer"},{"com.google.common.collect.RegularImmutableList":"backtype.storm.serialization.SerializableSerializer"}],"topology.max.spout.pending":16,"topology.message.timeout.secs":900,"drpc.request.timeout.secs":45}
>
> java.lang.RuntimeException:
>> org.apache.thrift7.transport.TTransportException: java.net.SocketException:
>> Broken pipe
>
> at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:112)
>
> at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:58)
>
> at
>> com.mycompany.runtime.DeployStormTopologies.main(DeployStormTopologies.java:92)
>
> Caused by: org.apache.thrift7.transport.TTransportException:
>> java.net.SocketException: Broken pipe
>
> at
>> org.apache.thrift7.transport.TIOStreamTransport.write(TIOStreamTransport.java:147)
>
> at
>> org.apache.thrift7.transport.TFramedTransport.flush(TFramedTransport.java:157)
>
> at org.apache.thrift7.TServiceClient.sendBase(TServiceClient.java:65)
>
> at
>> backtype.storm.generated.Nimbus$Client.send_submitTopology(Nimbus.java:156)
>
> at backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:145)
>
> at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:98)
>
> ... 2 more
>
> Caused by: java.net.SocketException: Broken pipe
>
> at java.net.SocketOutputStream.socketWrite0(Native Method)
>
> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
>
> at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
>
> at
>> org.apache.thrift7.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)
>
> ... 7 more
>
>
On Tue, Mar 18, 2014 at 4:12 PM, P. Taylor Goetz <ptgo...@gmail.com> wrote:
> It uploads the file in small (1024*5 bytes) chunks.
>
> Does this happen every time (i.e. reproducible)? What is the size of your
> topology jar?
>
> Can you post the server side message (I want to see the length it output).
>
> - Taylor
>
> On Mar 18, 2014, at 3:40 PM, Adam Lewis <superca...@gmail.com> wrote:
>
> Upon upgrading from 0.9.0.1 to 0.9.1-incubating, I'm exceeding the new
> thrift max buffer size (nicely logged on the server side, although the
> client just gets a broken pipe stack trace form thrift) with an approx 6 MB
> message(!). Increasing the configured limit solves the problem, but I
> would have thought the 1MB default should be enough.
>
> Does the storm submitter encode the entire topology as a single thrift
> message? I'm really surprised that the message is coming out so large, my
> topology isn't exactly small, but it only has about 20 bolts...does anyone
> have any suggestions on how to determine why the message is so large? Is
> this within the realm of what others have seen or am I doing something
> wrong?
>
> Thanks,
> Adam
>
>
>
