With a trident spout, data from all spouts forms a transaction. So each spout should be using the same txid at the same time.
--Tom On Wednesday, July 9, 2014, aka.fe2s <aka.f...@gmail.com> wrote: > Any ideas on the question above? > > > > > On Wed, Jun 18, 2014 at 6:13 PM, aka.fe2s <aka.f...@gmail.com > <javascript:_e(%7B%7D,'cvml','aka.f...@gmail.com');>> wrote: > >> I'm implementing my own ITridentSpout and would like to make it scalable. >> When I set parallelism hint to 3 with pipelining for my spout, I see the >> following >> >> [Thread-16-$spoutcoord-spout0] MyEmitter - initializeTransaction() txid 1 >> prevMetadata null currMetadata null generated metadata 9002537a >> [Thread-26-spout0] MyEmitter - emitBatch() txId 1 attempt id 0 >> coordinatorMeta 9002537a this.hashCode() 45c8d2d9 emitted e5f7a9d3 >> [Thread-30-spout0] MyEmitter - emitBatch txId 1 attempt id 0 >> coordinatorMeta 9002537a this.hashCode() 38ac85a emitted 1f08bdab >> [Thread-28-spout0] MyEmitter - emitBatch txId 1 attempt id 0 >> coordinatorMeta 9002537a this.hashCode() 58ed567b emitted ee35006c >> >> It creates 3 emitter instances and 3 threads as expected. But coordinator >> propagates the same tx id 1 to _all_ emitters, this doesn't make much sense >> since we should emit tuples for given tx id only once. I was expecting to >> get the following >> >> [Thread-16-$spoutcoord-spout0] MyEmitter - initializeTransaction() txid 1 >> .. >> [Thread-26-spout0] MyEmitter - emitBatch() txId 1 ... >> [Thread-16-$spoutcoord-spout0] MyEmitter - initializeTransaction() txid 2 >> .. >> [Thread-30-spout0] MyEmitter - emitBatch txId 2 ... >> [Thread-16-$spoutcoord-spout0] MyEmitter - initializeTransaction() txid 3 >> .. >> [Thread-28-spout0] MyEmitter - emitBatch txId 3 ... >> >> Is there any way to achieve this? >> >> >> >
