Hello all,
I started with a working trident topology in the form of:
stream
.each(new Fields("type", "session", "data"), new CounterKeyGen(), new
Fields("counterKey"))
.groupBy(new Fields("counterKey"))
.persistentAggregate(counterFactory, new Count(), new Fields("count"))
.newValuesStream()
.each(new Fields("counterKey", "count"), new CountPostProcessing(), new
Fields());
Where CounterKeyGen is a Function that maps incoming tuples to 0 or more keys
for aggregation and CountPostProcessing does some stuff based on the final
counts. Inside these Functions were tweakable constants that I wanted to
update "on the fly" using a global config stored in (static) trident state.
Resulting in:
stream
.stateQuery(config, new SnapshotGet(), new Fields("__CONFIG"))
.each(new Fields("type", "session", "data ", "__CONFIG"), new CounterKeyGen(),
new Fields("counterKey"))
.groupBy(new Fields("counterKey"))
.persistentAggregate(counterFactory, new Count(), new Fields("count"))
.newValuesStream()
.stateQuery(config, new SnapshotGet(), new Fields("__CONFIG"))
.each(new Fields("counterKey", "count", "__CONFIG "), new
CountPostProcessing(), new Fields());
This change hangs the topology, it seems the initial stateQuery is waiting for
a commit tuple that never arrives. From debugging I can tell that the partial
topology continues to work up until the second stateQuery is added. Given that
newValuesStream() outputs "counterKey" and "count" I need to re-fetch the
config or concatenate it with the key in order to have it in the second phase
of the topology. Neither is ideal, but since I need to have these counts
persist across config minor config changes, I cannot concatenate the key and
config naively. I would need to modify my backing map to intelligently discard
the config for the persistence but maintain it otherwise.
Is there a viable workaround for the double-stateQuery issue?
Better still, is there a more efficient way to have batch level read only
parameters than to inject them into every tuple?
-Bart
________________________________
This e-mail may contain CONFIDENTIAL AND PROPRIETARY INFORMATION and/or
PRIVILEGED AND CONFIDENTIAL COMMUNICATION intended solely for the recipient
and, therefore, may not be retransmitted to any party outside of the
recipient's organization without the prior written consent of the sender. If
you have received this e-mail in error please notify the sender immediately by
telephone or reply e-mail and destroy the original message without making a
copy. Deep Silver Volition, LLC accepts no liability for any losses or damages
resulting from infected e-mail transmissions and viruses in e-mail attachment.
