Sachin, STORM-969 makes use of tick tuple to periodically ack and flush the tuples so the scenario you mentioned would not happen. The tickTupleInterval is configurable.
- Arun On 9/9/15, 11:47 PM, "Sachin Pasalkar" <[email protected]> wrote: >Hi, > >I was looking at code where HDFSBolt writes the file with tuple coming to it. >I also had a look at JIRA >STORM-969<https://issues.apache.org/jira/browse/STORM-969>, I have following >question to it > >1)Lets say I have setup fileRotation policy at 64 MB, and I have written file >upto 59MB. Now, my worker failed, the file which I was writing to will be >never get rotated to final location. >2) As per 969 Jira, they have added the forceSync way but they are keeping all >tuples in memory and delaying the acks sent back to spout. In our case, to >write 64 MB data we need to store 5,400,000 process messages, which leads to >lot of data in memory. This may lead to unnecessary reply of tuple from >spout(I am aware its at least once and I can increase >TOPOLOGY_MESSAGE_TIMEOUT_SECS to full fill my requirement but is there other >way?) > >Thanks, >Sachin
