This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury.git
The following commit(s) were added to refs/heads/main by this push:
new bab86273 feat(doc): add type mapping for xlang serialization (#1476)
bab86273 is described below
commit bab86273b69d248d60272118b43337f7512aa34f
Author: Shawn Yang <[email protected]>
AuthorDate: Sat Apr 13 13:59:46 2024 +0800
feat(doc): add type mapping for xlang serialization (#1476)
add type mapping for xlang serialization
---------
Co-authored-by: LiangliangSui
<[email protected]>
---
docs/guide/DEVELOPMENT.md | 2 +-
docs/guide/xlang_type_mapping.md | 88 +++++++++++++++++++++++++-
docs/specification/xlang_serialization_spec.md | 3 +-
3 files changed, 89 insertions(+), 4 deletions(-)
diff --git a/docs/guide/DEVELOPMENT.md b/docs/guide/DEVELOPMENT.md
index 5d71166a..bfab5417 100644
--- a/docs/guide/DEVELOPMENT.md
+++ b/docs/guide/DEVELOPMENT.md
@@ -1,6 +1,6 @@
---
title: Development
-sidebar_position: 6
+sidebar_position: 7
id: development
---
diff --git a/docs/guide/xlang_type_mapping.md b/docs/guide/xlang_type_mapping.md
index f529d4ef..4baa4553 100644
--- a/docs/guide/xlang_type_mapping.md
+++ b/docs/guide/xlang_type_mapping.md
@@ -4,4 +4,90 @@ sidebar_position: 3
id: xlang_type_mapping
---
-Coming soon.
+Note:
+
+- For type definition, see [Type Systems in
Spec](../specification/xlang_serialization_spec.md#type-systems)
+- `int16_t[n]/vector<T>` indicates `int16_t[n]/vector<int16_t>`
+- The cross-language serialization is not stable, do not use it in your
production environment.
+
+# Type Mapping
+
+| Fury Type | Fury Type ID | Java | Python |
Javascript | C++ | Golang | Rust
|
+|--------------------|--------------|-----------------|----------------------|-----------------|--------------------------------|------------------|------------------|
+| bool | 1 | bool/Boolean | bool |
Boolean | bool | bool | bool
|
+| int8 | 2 | byte/Byte | int/pyfury.Int8 |
Type.int8() | int8_t | int8 | i8
|
+| int16 | 3 | short/Short | int/pyfury.Int16 |
Type.int16() | int16_t | int16 | i6
|
+| int32 | 4 | int/Integer | int/pyfury.Int32 |
Type.int32() | int32_t | int32 | i32
|
+| var_int32 | 5 | int/Integer | int/pyfury.VarInt32 |
Type.varint32() | fury::varint32_t | fury.varint32 |
fury::varint32 |
+| int64 | 6 | long/Long | int/pyfury.Int64 |
Type.int64() | int64_t | int64 | i64
|
+| var_int64 | 7 | long/Long | int/pyfury.VarInt64 |
Type.varint64() | fury::varint64_t | fury.varint64 |
fury::varint64 |
+| sli_int64 | 8 | long/Long | int/pyfury.SliInt64 |
Type.sliint64() | fury::sliint64_t | fury.sliint64 |
fury::sliint64 |
+| float16 | 9 | float/Float | float/pyfury.Float16 |
Type.float16() | fury::float16_t | fury.float16 | fury::f16
|
+| float32 | 10 | float/Float | float/pyfury.Float32 |
Type.float32() | float | float32 | f32
|
+| float64 | 11 | double/Double | float/pyfury.Float64 |
Type.float64() | double | float64 | f64
|
+| string | 12 | String | str |
String | string | string |
String/str |
+| enum | 13 | Enum subclasses | enum subclasses |
/ | enum | / | enum
|
+| list | 14 | List/Collection | list/tuple |
array | vector | slice | Vec
|
+| set | 15 | Set | set |
/ | set | fury.Set | Set
|
+| map | 16 | Map | dict |
Map | unordered_map | map | HashMap
|
+| duration | 17 | Duration | timedelta |
Number | duration | Duration | Duration
|
+| timestamp | 18 | Instant | datetime |
Number | std::chrono::nanoseconds | Time | DateTime
|
+| decimal | 19 | BigDecimal | Decimal |
bigint | / | / | /
|
+| binary | 20 | byte[] | bytes |
/ | `uint8_t[n]/vector<T>` | `[n]uint8/[]T` |
`Vec<uint8_t>` |
+| array | 21 | array | np.ndarray |
/ | / | array/slice | Vec
|
+| bool_array | 22 | bool[] | ndarray(np.bool_) |
/ | `bool[n]` | `[n]bool/[]T` |
`Vec<bool>` |
+| int8_array | 23 | byte[] | ndarray(int8) |
/ | `int8_t[n]/vector<T>` | `[n]int8/[]T` |
`Vec<i18>` |
+| int16_array | 24 | short[] | ndarray(int16) |
/ | `int16_t[n]/vector<T>` | `[n]int16/[]T` |
`Vec<i16>` |
+| int32_array | 25 | int[] | ndarray(int32) |
/ | `int32_t[n]/vector<T>` | `[n]int32/[]T` |
`Vec<i32>` |
+| int64_array | 26 | long[] | ndarray(int64) |
/ | `int64_t[n]/vector<T>` | `[n]int64/[]T` |
`Vec<i64>` |
+| float16_array | 27 | float[] | ndarray(float16) |
/ | `fury::float16_t[n]/vector<T>` | `[n]float16/[]T` |
`Vec<fury::f16>` |
+| float32_array | 28 | float[] | ndarray(float32) |
/ | `float[n]/vector<T>` | `[n]float32/[]T` |
`Vec<f32>` |
+| float64_array | 29 | double[] | ndarray(float64) |
/ | `double[n]/vector<T>` | `[n]float64/[]T` |
`Vec<f64>` |
+| tensor | 30 | / | / |
/ | / | / | /
|
+| sparse tensor | 31 | / | / |
/ | / | / | /
|
+| arrow record batch | 32 | / | / |
/ | / | / | /
|
+| arrow table | 33 | / | / |
/ | / | / | /
|
+
+# Type info(not implemented currently)
+
+Due to differences between type systems of languages, those types can't be
mapped one-to-one between languages.
+
+If the user notices that one type on a language corresponds to multiple types
in Fury type systems, for example, `long`
+in java has type `int64/varint64/sliint64`, it means the language lacks some
types, and the user must provide extra type
+info when using Fury.
+
+## Type annotation
+
+If the type is a field of another class, users can provide meta hints for
fields of a type, or for the whole type.
+Such information can be provided in other languages too:
+
+- java: use annotation.
+- cpp: use macro and template.
+- golang: use struct tag.
+- python: use typehint.
+- rust: use macro.
+
+Here is en example:
+
+- Java:
+ ```java
+ class Foo {
+ @Int32Type(varint = true)
+ int f1;
+ List<@Int32Type(varint = true) Integer> f2;
+ }
+ ```
+- Python:
+
+ ```python
+ class Foo:
+ f1: Int32Type(varint=True)
+ f2: List[Int32Type(varint=True)]
+ ```
+
+## Type wrapper
+
+If the type is not a field of a class, the user must wrap this type with a
Fury type to pass the extra type info.
+
+For example, suppose Fury Java provide a `VarInt64` type, when a user invoke
`fury.serialize(long_value)`, he need to
+invoke like `fury.serialize(new VarInt64(long_value))`.
diff --git a/docs/specification/xlang_serialization_spec.md
b/docs/specification/xlang_serialization_spec.md
index e01ec921..15094c52 100644
--- a/docs/specification/xlang_serialization_spec.md
+++ b/docs/specification/xlang_serialization_spec.md
@@ -29,11 +29,9 @@ also introduce more complexities compared to static
serialization frameworks. So
- int16: a 16-bit signed integer.
- int32: a 32-bit signed integer.
- var_int32: a 32-bit signed integer which use fury var_int32 encoding.
-- fixed_int32: a 32-bit signed integer which use two's complement encoding.
- int64: a 64-bit signed integer.
- var_int64: a 64-bit signed integer which use fury PVL encoding.
- sli_int64: a 64-bit signed integer which use fury SLI encoding.
-- fixed_int64: a 64-bit signed integer which use two's complement encoding.
- float16: a 16-bit floating point number.
- float32: a 32-bit floating point number.
- float64: a 64-bit floating point number including NaN and Infinity.
@@ -53,6 +51,7 @@ also introduce more complexities compared to static
serialization frameworks. So
interoperability between array and list.
- array: multidimensional array which every sub-array can have different
sizes but all have same type.
- bool_array: one dimensional int16 array.
+ - int8_array: one dimensional int8 array.
- int16_array: one dimensional int16 array.
- int32_array: one dimensional int32 array.
- int64_array: one dimensional int64 array.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
