Yes if a batch fails the state may be written to one, but not the other. At least not until the batch completes successfully.
- Bobby On Thursday, August 3, 2017, 3:36:51 PM CDT, Geoffrey Holmes <[email protected]> wrote: #yiv7886372101 #yiv7886372101 -- _filtered #yiv7886372101 {font-family:Arial;panose-1:2 11 6 4 2 2 2 2 2 4;} _filtered #yiv7886372101 {panose-1:2 4 5 3 5 4 6 3 2 4;} _filtered #yiv7886372101 {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;} _filtered #yiv7886372101 {panose-1:2 0 5 3 0 0 0 2 0 4;}#yiv7886372101 #yiv7886372101 p.yiv7886372101MsoNormal, #yiv7886372101 li.yiv7886372101MsoNormal, #yiv7886372101 div.yiv7886372101MsoNormal {margin:0in;margin-bottom:.0001pt;font-size:11.0pt;}#yiv7886372101 a:link, #yiv7886372101 span.yiv7886372101MsoHyperlink {color:blue;text-decoration:underline;}#yiv7886372101 a:visited, #yiv7886372101 span.yiv7886372101MsoHyperlinkFollowed {color:purple;text-decoration:underline;}#yiv7886372101 p.yiv7886372101msonormal, #yiv7886372101 li.yiv7886372101msonormal, #yiv7886372101 div.yiv7886372101msonormal {margin-right:0in;margin-left:0in;font-size:11.0pt;}#yiv7886372101 p.yiv7886372101msochpdefault, #yiv7886372101 li.yiv7886372101msochpdefault, #yiv7886372101 div.yiv7886372101msochpdefault {margin-right:0in;margin-left:0in;font-size:11.0pt;}#yiv7886372101 span.yiv7886372101msohyperlink {}#yiv7886372101 span.yiv7886372101msohyperlinkfollowed {}#yiv7886372101 span.yiv7886372101emailstyle17 {}#yiv7886372101 span.yiv7886372101msoins {}#yiv7886372101 p.yiv7886372101msonormal1, #yiv7886372101 li.yiv7886372101msonormal1, #yiv7886372101 div.yiv7886372101msonormal1 {margin:0in;margin-bottom:.0001pt;font-size:12.0pt;}#yiv7886372101 span.yiv7886372101msohyperlink1 {color:#0563C1;text-decoration:underline;}#yiv7886372101 span.yiv7886372101msohyperlinkfollowed1 {color:#954F72;text-decoration:underline;}#yiv7886372101 span.yiv7886372101emailstyle171 {color:windowtext;font-weight:normal;font-style:normal;}#yiv7886372101 span.yiv7886372101msoins1 {color:teal;text-decoration:underline;}#yiv7886372101 p.yiv7886372101msochpdefault1, #yiv7886372101 li.yiv7886372101msochpdefault1, #yiv7886372101 div.yiv7886372101msochpdefault1 {margin-right:0in;margin-left:0in;font-size:11.0pt;}#yiv7886372101 span.yiv7886372101EmailStyle29 {color:windowtext;font-weight:normal;font-style:normal;}#yiv7886372101 span.yiv7886372101msoIns {text-decoration:underline;color:teal;}#yiv7886372101 .yiv7886372101MsoChpDefault {font-size:10.0pt;} _filtered #yiv7886372101 {margin:1.0in 1.0in 1.0in 1.0in;}#yiv7886372101 div.yiv7886372101WordSection1 {}#yiv7886372101 Thanks. That helps answer my question. What if a batch fails? Could records in that batch get written to disk by one HDFS state but not another? From:Bobby Evans <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Thursday, August 3, 2017 at 1:27 PM To: "[email protected]" <[email protected]> Subject: Re: HDFS state and commits and only-once semantics Writing to a state in storm in not atomic. Storm guarantees that once the batch completes that the data is written out to all of the states that expect to receive it. The HDFS state guarantees that the dat will have been flushed out the the data nodes when a batch completes and if the topology keeps running eventually the files will be rotated and made available for others to process, but there is no guarantee that the files will rotate at the same time or anything like that. - Bobby On Thursday, August 3, 2017, 1:17:39 PM CDT, Geoffrey Holmes <[email protected]> wrote: I read STORM-837 (https://issues.apache.org/jira/browse/STORM-837) and have a question. How does this work if I have more than one HDFS state in my Trident topology? Can I ensure that a record ends up written to both HDFS states or none but not just one or the other?
