gene-db commented on code in PR #47473:
URL: https://github.com/apache/spark/pull/47473#discussion_r1691812112
##########
common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java:
##########
@@ -120,6 +120,12 @@ public class VariantUtil {
// Long string value. The content is (4-byte little-endian unsigned integer
representing the
// string size) + (size bytes of string content).
public static final int LONG_STR = 16;
+ // year-month interval value. The content is one byte representing the start
and end field values
+ // (1 bit each starting at least significant bits) and a 4-byte
little-endian signed integer
+ public static final int YEAR_MONTH_INTERVAL = 19;
+ // day-time interval value. The content is one byte representing the start
and end field values
+ // (2 bits each starting at least significant bits) and an 8-byte
little-endian signed integer
Review Comment:
Similar to the README, we should document what 0, 1, 2, 3 represent for the
start and end fields.
##########
common/variant/src/main/java/org/apache/spark/types/variant/Variant.java:
##########
@@ -88,6 +91,16 @@ public long getLong() {
return VariantUtil.getLong(value, pos);
}
+ // Get the start and end fields of a year-month interval from the variant.
+ public IntervalFields getYearMonthIntervalFields() {
Review Comment:
What uses these apis?
##########
common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java:
##########
@@ -377,11 +405,52 @@ public static long getLong(byte[] value, int pos) {
case TIMESTAMP:
case TIMESTAMP_NTZ:
return readLong(value, pos + 1, 8);
+ case YEAR_MONTH_INTERVAL:
+ return readLong(value, pos + 2, 4);
+ case DAY_TIME_INTERVAL:
+ return readLong(value, pos + 2, 8);
default:
throw new IllegalStateException(exceptionMessage);
}
}
+ // Class used to pass around start and end fields of year-month and day-time
interval values.
+ public static class IntervalFields {
+ public IntervalFields(byte startField, byte endField) {
+ this.startField = startField;
+ this.endField = endField;
+ }
+
+ public final byte startField;
+ public final byte endField;
+ }
+
+ // Get the start and end fields of a variant value representing a year-month
interval value. The
+ // returned array contains the start field at the zeroth index and the end
field at the first
+ // index.
+ public static IntervalFields getYearMonthIntervalFields(byte[] value, int
pos) {
+ long fieldInfo = readLong(value, pos + 1, 1);
+ IntervalFields intervalFields = new IntervalFields((byte) (fieldInfo &
0x1),
+ (byte) ((fieldInfo >> 1) & 0x1));
+ if (intervalFields.endField < intervalFields.startField) {
Review Comment:
Do we need to check the type to be year-month interval, and the length? Some
other functions call `checkIndex`.
##########
common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java:
##########
@@ -355,9 +380,12 @@ public static boolean getBoolean(byte[] value, int pos) {
// Get a long value from variant value `value[pos...]`.
// It is only legal to call it if `getType` returns one of
`Type.LONG/DATE/TIMESTAMP/
- // TIMESTAMP_NTZ`. If the type is `DATE`, the return value is guaranteed to
fit into an int and
- // represents the number of days from the Unix epoch. If the type is
`TIMESTAMP/TIMESTAMP_NTZ`,
- // the return value represents the number of microseconds from the Unix
epoch.
+ // TIMESTAMP_NTZ/YEAR_MONTH_INTERVAL/DAY_TIME_INTERVAL`. If the type is
`DATE`, the return value
+ // is guaranteed to fit into an int and represents the number of days from
the Unix epoch.
+ // If the type is `TIMESTAMP/TIMESTAMP_NTZ`, the return value represents the
number of
+ // microseconds from the Unix epoch. If the type is `YEAR_MONTH_INTERVAL`,
the return value
Review Comment:
We should also mention that the year-month one is guaranteed to fit into an
int.
##########
common/variant/src/main/java/org/apache/spark/types/variant/Variant.java:
##########
@@ -113,6 +126,11 @@ public String getString() {
return VariantUtil.getString(value, pos);
}
+ // Get the type info bits from a variant value.
+ public int getTypeInfo() {
Review Comment:
I can't easily tell, but what uses this api?
##########
common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java:
##########
@@ -120,6 +120,12 @@ public class VariantUtil {
// Long string value. The content is (4-byte little-endian unsigned integer
representing the
// string size) + (size bytes of string content).
public static final int LONG_STR = 16;
+ // year-month interval value. The content is one byte representing the start
and end field values
+ // (1 bit each starting at least significant bits) and a 4-byte
little-endian signed integer
Review Comment:
Similar to the README, we should document what 0 and 1 represent for the
start and end fields.
##########
common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java:
##########
@@ -233,6 +245,13 @@ public enum Type {
TIMESTAMP_NTZ,
FLOAT,
BINARY,
+ YEAR_MONTH_INTERVAL,
+ DAY_TIME_INTERVAL,
+ }
+
+ public static int getTypeInfo(byte[] value, int pos) {
Review Comment:
Similar to an earlier question, what uses this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
