viirya commented on code in PR #59327:
URL: https://github.com/apache/spark/pull/59327#discussion_r4240555518
##########
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:
##########
@@ -1772,6 +1761,60 @@ private[history] class FsHistoryProvider(conf:
SparkConf, clock: Clock)
private[spark] object FsHistoryProvider {
+ /**
+ * Positions `in` at the start of the first line that begins at or after
byte `target`,
Review Comment:
Updated in cf13cc0fa2f along the lines you suggested. The scaladoc now says
that the returned stream starts at the first line beginning at or after
`target`, and that `in` is consumed and must not be read afterwards, since the
bytes read past the boundary are prepended to the returned stream rather than
pushed back into `in`.
##########
core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala:
##########
@@ -1248,6 +1248,142 @@ abstract class FsHistoryProviderSuite extends
SparkFunSuite with Matchers with P
}
}
+ test("end event reparse when the skip offset lands inside a multi-byte
character") {
+ val log = newLogFile("end-event-utf8", None, inProgress = false)
+ val bytes = writeRawLog(log, endEventTestLines("end-event-utf8",
+ jobStartLine(1, 0x4e2d.toChar.toString * 200)))
+ // The job description is written as raw UTF-8: skip to the second byte of
a character.
+ val leadByte = bytes.indexWhere(b => (b & 0xF0) == 0xE0)
+ assert(leadByte > 0)
+ val conf = createTestConf().set(END_EVENT_REPARSE_CHUNK_SIZE,
(bytes.length - leadByte - 1L))
Review Comment:
Good catch, thanks. Changed in cf13cc0fa2f: the target is now the third byte
of U+4E2D, so the raw scan starts on the continuation byte B8. I also added
cases to the `skipToLineStart` test with the target on each byte of a 3-byte
character and on the following `\n`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
