Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/16276 )
Change subject: [KUDU-3177] Added kudu.snapshotTimestampMicros to kudu spark readOptions as optional property Added property snapshotTimestampMs to spark read options which will allow consistant scans when timestamp is set before the first dataFrame read. ...................................................................... Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/16276/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16276/2//COMMIT_MSG@8 PS2, Line 8: nit: add a blank line between subject and description. http://gerrit.cloudera.org:8080/#/c/16276/1/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala File java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala: http://gerrit.cloudera.org:8080/#/c/16276/1/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala@71 PS1, Line 71: if (options.scanLocality == ReplicaSelection.CLOSEST_REPLICA) { > Currently the only way to set READ_AT_SNAPSHOT as the readMode is by using I think you could set READ_AT_SNAPSHOT if `options.scanLocality == ReplicaSelection.CLOSEST_REPLICA || options.snapshotTimestampMicros.isDefined` and then move setting snapshotTimestampMicros outside of this if statement. http://gerrit.cloudera.org:8080/#/c/16276/1/java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala File java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala: http://gerrit.cloudera.org:8080/#/c/16276/1/java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala@773 PS1, Line 773: val timestamp = System.currentTimeMillis() * 1000 > It appears that this is an encoded value. Is there a good way to decode int You can use the HybridTimeUtil class to convert as needed: https://github.com/apache/kudu/blob/master/java/kudu-client/src/main/java/org/apache/kudu/util/HybridTimeUtil.java You can see it used in the BackupRDD here: https://github.com/apache/kudu/blob/master/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackupRDD.scala#L71-L77 -- To view, visit http://gerrit.cloudera.org:8080/16276 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I00862c0e174a964efc6cab0b8141b1ac5a1bebc0 Gerrit-Change-Number: 16276 Gerrit-PatchSet: 1 Gerrit-Owner: Kevin McCarthy <[email protected]> Gerrit-Reviewer: Grant Henke <[email protected]> Gerrit-Reviewer: Kevin McCarthy <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Comment-Date: Mon, 03 Aug 2020 19:43:38 +0000 Gerrit-HasComments: Yes
