[PR] optimize unsafePutPositiveVarInt bytecode size [incubator-fury]
pandalee99 opened a new pull request, #1489: URL: https://github.com/apache/incubator-fury/pull/1489 ## optimize unsafePutPositiveVarInt bytecode size i think,the code can be reduce something memory the bytecode reduced from 208 to 192 https://github.com/apache/incubator-fury/assets/46820719/a6a2a6ad-dca9-442f-ac3a-f28c95329bef";> #1462 has been reduced from three variables to two variables, and #1466 is already very few variables, I don't think it can be reduced any more Variable-length coding requires at least one variable determination and one variable storage ## Related issues #1466 ## Does this PR introduce any user-facing change? - [x] Does this PR introduce any public API change? - [x] Does this PR introduce any binary protocol compatibility change? ## Benchmark pre https://github.com/apache/incubator-fury/assets/46820719/56273187-c8cb-4dec-adc8-002437d0841a";> after https://github.com/apache/incubator-fury/assets/46820719/a6a2a6ad-dca9-442f-ac3a-f28c95329bef";> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" is working again!
The GitHub Actions job "Fury CI" on incubator-fury.git has succeeded. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 3eb1338237643f353dba89a6a230aaef06d38585 / chaokunyang lint code Report URL: https://github.com/apache/incubator-fury/actions/runs/8642037249 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(JavaScript): implement xlang protocol [incubator-fury]
chaokunyang commented on code in PR #1487:
URL: https://github.com/apache/incubator-fury/pull/1487#discussion_r1560471277
##
javascript/packages/fury/lib/gen/datetime.ts:
##
@@ -72,5 +72,5 @@ class DateSerializerGenerator extends BaseSerializerGenerator
{
}
}
-CodegenRegistry.register(InternalSerializerType.DATE, DateSerializerGenerator);
+CodegenRegistry.register(InternalSerializerType.DURATION,
DateSerializerGenerator);
Review Comment:
```suggestion
CodegenRegistry.register(InternalSerializerType.DURATION,
DurationSerializerGenerator);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 3eb1338237643f353dba89a6a230aaef06d38585 / chaokunyang lint code Report URL: https://github.com/apache/incubator-fury/actions/runs/8642037249 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[PR] chore(java): add jmh benchmark params to doc [incubator-fury]
chaokunyang opened a new pull request, #1488: URL: https://github.com/apache/incubator-fury/pull/1488 ## What does this PR do? This PR add jmh params to doc, so one can select which benchmark to run easily for performance optimization ## Related issues ## Does this PR introduce any user-facing change? - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(JavaScript): implement xlang protocol [incubator-fury]
chaokunyang commented on code in PR #1487:
URL: https://github.com/apache/incubator-fury/pull/1487#discussion_r1560411086
##
javascript/packages/fury/lib/meta.ts:
##
@@ -85,6 +85,12 @@ export const getMeta = (description: TypeDescription, fury:
Fury): Meta => {
needToWriteRef: Boolean(fury.config.refTracking) && false,
type,
};
+case InternalSerializerType.BINARY:
+ return {
+fixedSize: 8,
+needToWriteRef: Boolean(fury.config.refTracking) && true,
Review Comment:
```suggestion
needToWriteRef: Boolean(fury.config.refTracking),
```
seems `&& true` is unnecessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(doc): add type mapping for xlang serialization [incubator-fury]
chaokunyang commented on code in PR #1476: URL: https://github.com/apache/incubator-fury/pull/1476#discussion_r1560408744 ## docs/guide/xlang_type_mapping.md: ## @@ -4,4 +4,90 @@ sidebar_position: 3 id: xlang_type_mapping --- -Coming soon. +Note: + +- For type definition, see [Type Systems in Spec](../specification/xlang_serialization_spec.md#type-systems) +- `int16_t[n]/vector` indicates `int16_t[n]/vector` +- The cross-language serialization is not stable, do not use it in your production environment. + +# Type Mapping + +| Type | Type ID | Java| Python | Javascript | C++| Golang | Rust | +||-|-|--|-||--|--| +| bool | 1 | bool/Boolean| bool | Boolean | bool | bool | bool | +| int8 | 2 | byte/Byte | int/pyfury.Int8 | Type.int8() | int8_t | int8 | i8 | +| int16 | 3 | short/Short | int/pyfury.Int16 | Type.int16()| int16_t| int16| i6 | +| int32 | 4 | int/Integer | int/pyfury.Int32 | Type.int32()| int32_t| int32| i32 | +| var_int32 | 5 | int/Integer | int/pyfury.VarInt32 | Type.varint32() | fury::varint32_t | fury.varint32| fury::varint32 | +| int64 | 6 | long/Long | int/pyfury.Int64 | Type.int64()| int64_t| int64| i64 | +| var_int64 | 7 | long/Long | int/pyfury.VarInt64 | Type.varint64() | fury::varint64_t | fury.varint64| fury::varint64 | +| sli_int64 | 8 | long/Long | int/pyfury.SliInt64 | Type.sliint64() | fury::sliint64_t | fury.sliint64| fury::sliint64 | +| float16| 9 | float/Float | float/pyfury.Float16 | Type.float16() | fury::float16_t| fury.float16 | fury::f16 | +| float32| 10 | float/Float | float/pyfury.Float32 | Type.float32() | float | float32 | f32 | +| float64| 11 | double/Double | float/pyfury.Float64 | Type.float64() | double | float64 | f64 | +| string | 12 | String | str | String | string | string | String/str | +| enum | 13 | Enum subclasses | enum subclasses | / | enum | /| enum | +| list | 14 | List/Collection | list/tuple | array | vector | slice| Vec | +| set| 15 | Set | set | / | set| fury.Set | Set | +| map| 16 | Map | dict | Map | unordered_map | map | HashMap | +| duration | 17 | Duration| timedelta| Number | duration | Duration | Duration | +| timestamp | 18 | Instant | datetime | Number | std::chrono::nanoseconds | Time | DateTime | +| decimal| 19 | BigDecimal | Decimal | bigint | / | /| / | +| binary | 20 | byte[] | bytes| / | `uint8_t[n]/vector` | `[n]uint8/[]T` | `Vec` | +| array | 21 | array | np.ndarray | / | / | array/slice | Vec | +| bool_array | 22 | bool[] | ndarray(np.bool_)| / | `bool[n]` | `[n]bool/[]T`| `Vec` | +| int8_array | 23 | byte[] | ndarray(int8)| / | `int8_t[n]/vector` | `[n]int8/[]T`| `Vec` | +| int16_array| 24 | short[] | ndarray(int16) | / | `int16_t[n]/vector` | `[n]int16/[]T` | `Vec` | +| int32_array| 25 | int[] | ndarray(int32) | / | `int32_t[n]/vector` | `[n]int32/[]T` | `Vec` | +| int
Re: [PR] feat(doc): add type mapping for xlang serialization [incubator-fury]
LiangliangSui commented on code in PR #1476: URL: https://github.com/apache/incubator-fury/pull/1476#discussion_r1560350333 ## docs/guide/xlang_type_mapping.md: ## @@ -4,4 +4,90 @@ sidebar_position: 3 id: xlang_type_mapping --- -Coming soon. +Note: + +- For type definition, see [Type Systems in Spec](../specification/xlang_serialization_spec.md#type-systems) +- `int16_t[n]/vector` indicates `int16_t[n]/vector` +- The cross-language serialization is not stable, do not use it in your production environment. + +# Type Mapping + +| Type | Type ID | Java| Python | Javascript | C++| Golang | Rust | Review Comment: ```suggestion | Fury Type | Fury Type ID | Java| Python | Javascript | C++| Golang | Rust | ``` ## docs/guide/xlang_type_mapping.md: ## @@ -4,4 +4,90 @@ sidebar_position: 3 id: xlang_type_mapping --- -Coming soon. +Note: + +- For type definition, see [Type Systems in Spec](../specification/xlang_serialization_spec.md#type-systems) +- `int16_t[n]/vector` indicates `int16_t[n]/vector` +- The cross-language serialization is not stable, do not use it in your production environment. + +# Type Mapping + +| Type | Type ID | Java| Python | Javascript | C++| Golang | Rust | +||-|-|--|-||--|--| +| bool | 1 | bool/Boolean| bool | Boolean | bool | bool | bool | +| int8 | 2 | byte/Byte | int/pyfury.Int8 | Type.int8() | int8_t | int8 | i8 | +| int16 | 3 | short/Short | int/pyfury.Int16 | Type.int16()| int16_t| int16| i6 | +| int32 | 4 | int/Integer | int/pyfury.Int32 | Type.int32()| int32_t| int32| i32 | +| var_int32 | 5 | int/Integer | int/pyfury.VarInt32 | Type.varint32() | fury::varint32_t | fury.varint32| fury::varint32 | +| int64 | 6 | long/Long | int/pyfury.Int64 | Type.int64()| int64_t| int64| i64 | +| var_int64 | 7 | long/Long | int/pyfury.VarInt64 | Type.varint64() | fury::varint64_t | fury.varint64| fury::varint64 | +| sli_int64 | 8 | long/Long | int/pyfury.SliInt64 | Type.sliint64() | fury::sliint64_t | fury.sliint64| fury::sliint64 | +| float16| 9 | float/Float | float/pyfury.Float16 | Type.float16() | fury::float16_t| fury.float16 | fury::f16 | +| float32| 10 | float/Float | float/pyfury.Float32 | Type.float32() | float | float32 | f32 | +| float64| 11 | double/Double | float/pyfury.Float64 | Type.float64() | double | float64 | f64 | +| string | 12 | String | str | String | string | string | String/str | +| enum | 13 | Enum subclasses | enum subclasses | / | enum | /| enum | +| list | 14 | List/Collection | list/tuple | array | vector | slice| Vec | +| set| 15 | Set | set | / | set| fury.Set | Set | +| map| 16 | Map | dict | Map | unordered_map | map | HashMap | +| duration | 17 | Duration| timedelta| Number | duration | Duration | Duration | +| timestamp | 18 | Instant | datetime | Number | std::chrono::nanoseconds | Time | DateTime | +| decimal| 19 | BigDecimal | Decimal | bigint | / | /| / | +| binary | 20 | byte[] | bytes| / | `uint8_t[n]/vector` | `[n]uint8/[]T` | `Ve
Re: [PR] feat(doc): add type mapping for xlang serialization [incubator-fury]
chaokunyang commented on code in PR #1476: URL: https://github.com/apache/incubator-fury/pull/1476#discussion_r1559519656 ## docs/guide/xlang_type_mapping.md: ## @@ -4,4 +4,76 @@ sidebar_position: 3 id: xlang_type_mapping --- -Coming soon. +Note: + +- For type definition, see [Type Systems in Spec](../specification/xlang_serialization_spec.md#type-systems) +- `int16_t[n]/vector` indicates `int16_t[n]/vector` +- The cross-language serialization is not stable, do not use it in your production environment. + +# Type Mapping + +| Type | Type ID | Java| Python | Javascript | C++| Golang | Rust | +||-|-|--|-||--|--| +| bool | 1 | bool/Boolean| bool | Boolean | bool | bool | bool | +| int8 | 2 | byte/Byte | int/pyfury.Int8 | Type.int8() | int8_t | int8 | i8 | +| int16 | 3 | short/Short | int/pyfury.Int16 | Type.int16()| int16_t| int16| i6 | +| int32 | 4 | int/Integer | int/pyfury.Int32 | Type.int32()| int32_t| int32| i32 | +| var_int32 | 5 | int/Integer | int/pyfury.VarInt32 | Type.varint32() | fury::varint32_t | fury.varint32| fury::varint32 | +| int64 | 6 | long/Long | int/pyfury.Int64 | Type.int64()| int64_t| int64| i64 | +| var_int64 | 7 | long/Long | int/pyfury.VarInt64 | Type.varint64() | fury::varint64_t | fury.varint64| fury::varint64 | +| sli_int64 | 8 | long/Long | int/pyfury.SliInt64 | Type.sliint64() | fury::sliint64_t | fury.sliint64| fury::sliint64 | +| float16| 9 | float/Float | float/pyfury.Float16 | Type.float16() | fury::float16_t| fury.float16 | fury::f16 | +| float32| 10 | float/Float | float/pyfury.Float32 | Type.float32() | float | float32 | f32 | +| float64| 11 | double/Double | float/pyfury.Float64 | Type.float64() | double | float64 | f64 | +| string | 12 | String | str | String | string | string | String/str | +| enum | 13 | Enum subclasses | enum subclasses | / | enum | /| enum | +| list | 14 | List/Collection | list/tuple | array | vector | slice| Vec | +| set| 15 | Set | set | / | set| fury.Set | Set | +| map| 16 | Map | dict | Map | unordered_map | map | HashMap | +| duration | 17 | Duration| timedelta| Number | duration | Duration | Duration | +| timestamp | 18 | Instant | datetime | Number | std::chrono::nanoseconds | Time | DateTime | +| decimal| 19 | BigDecimal | Decimal | bigint | / | /| / | +| binary | 20 | byte[] | bytes| / | `int8_t[n]/vector` | `[n]int8/[]T`| `Vec` | +| array | 20 | array | np.ndarray | / | / | array/slice | Vec | Review Comment: > We can serialize a bytes to int8 array and deserialize int8 array to bytes. But it would introduce some confusion. since it is so, we keep it. but should assign a new Typeid to binary to differentiate it from the array. Yes , I forgot to update the type Id. I will update it later -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] feat(doc): add type mapping for xlang serialization [incubator-fury]
theweipeng commented on code in PR #1476: URL: https://github.com/apache/incubator-fury/pull/1476#discussion_r1559489133 ## docs/guide/xlang_type_mapping.md: ## @@ -4,4 +4,76 @@ sidebar_position: 3 id: xlang_type_mapping --- -Coming soon. +Note: + +- For type definition, see [Type Systems in Spec](../specification/xlang_serialization_spec.md#type-systems) +- `int16_t[n]/vector` indicates `int16_t[n]/vector` +- The cross-language serialization is not stable, do not use it in your production environment. + +# Type Mapping + +| Type | Type ID | Java| Python | Javascript | C++| Golang | Rust | +||-|-|--|-||--|--| +| bool | 1 | bool/Boolean| bool | Boolean | bool | bool | bool | +| int8 | 2 | byte/Byte | int/pyfury.Int8 | Type.int8() | int8_t | int8 | i8 | +| int16 | 3 | short/Short | int/pyfury.Int16 | Type.int16()| int16_t| int16| i6 | +| int32 | 4 | int/Integer | int/pyfury.Int32 | Type.int32()| int32_t| int32| i32 | +| var_int32 | 5 | int/Integer | int/pyfury.VarInt32 | Type.varint32() | fury::varint32_t | fury.varint32| fury::varint32 | +| int64 | 6 | long/Long | int/pyfury.Int64 | Type.int64()| int64_t| int64| i64 | +| var_int64 | 7 | long/Long | int/pyfury.VarInt64 | Type.varint64() | fury::varint64_t | fury.varint64| fury::varint64 | +| sli_int64 | 8 | long/Long | int/pyfury.SliInt64 | Type.sliint64() | fury::sliint64_t | fury.sliint64| fury::sliint64 | +| float16| 9 | float/Float | float/pyfury.Float16 | Type.float16() | fury::float16_t| fury.float16 | fury::f16 | +| float32| 10 | float/Float | float/pyfury.Float32 | Type.float32() | float | float32 | f32 | +| float64| 11 | double/Double | float/pyfury.Float64 | Type.float64() | double | float64 | f64 | +| string | 12 | String | str | String | string | string | String/str | +| enum | 13 | Enum subclasses | enum subclasses | / | enum | /| enum | +| list | 14 | List/Collection | list/tuple | array | vector | slice| Vec | +| set| 15 | Set | set | / | set| fury.Set | Set | +| map| 16 | Map | dict | Map | unordered_map | map | HashMap | +| duration | 17 | Duration| timedelta| Number | duration | Duration | Duration | +| timestamp | 18 | Instant | datetime | Number | std::chrono::nanoseconds | Time | DateTime | +| decimal| 19 | BigDecimal | Decimal | bigint | / | /| / | +| binary | 20 | byte[] | bytes| / | `int8_t[n]/vector` | `[n]int8/[]T`| `Vec` | +| array | 20 | array | np.ndarray | / | / | array/slice | Vec | Review Comment: > Thrift also has a binary type. And we don't provide uint8 type for now. The int8 array can be a different type Yes I have noticed that we can not distinguish 255 from -1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@
Re: [PR] feat(doc): add type mapping for xlang serialization [incubator-fury]
theweipeng commented on code in PR #1476: URL: https://github.com/apache/incubator-fury/pull/1476#discussion_r1559476930 ## docs/guide/xlang_type_mapping.md: ## @@ -4,4 +4,76 @@ sidebar_position: 3 id: xlang_type_mapping --- -Coming soon. +Note: + +- For type definition, see [Type Systems in Spec](../specification/xlang_serialization_spec.md#type-systems) +- `int16_t[n]/vector` indicates `int16_t[n]/vector` +- The cross-language serialization is not stable, do not use it in your production environment. + +# Type Mapping + +| Type | Type ID | Java| Python | Javascript | C++| Golang | Rust | +||-|-|--|-||--|--| +| bool | 1 | bool/Boolean| bool | Boolean | bool | bool | bool | +| int8 | 2 | byte/Byte | int/pyfury.Int8 | Type.int8() | int8_t | int8 | i8 | +| int16 | 3 | short/Short | int/pyfury.Int16 | Type.int16()| int16_t| int16| i6 | +| int32 | 4 | int/Integer | int/pyfury.Int32 | Type.int32()| int32_t| int32| i32 | +| var_int32 | 5 | int/Integer | int/pyfury.VarInt32 | Type.varint32() | fury::varint32_t | fury.varint32| fury::varint32 | +| int64 | 6 | long/Long | int/pyfury.Int64 | Type.int64()| int64_t| int64| i64 | +| var_int64 | 7 | long/Long | int/pyfury.VarInt64 | Type.varint64() | fury::varint64_t | fury.varint64| fury::varint64 | +| sli_int64 | 8 | long/Long | int/pyfury.SliInt64 | Type.sliint64() | fury::sliint64_t | fury.sliint64| fury::sliint64 | +| float16| 9 | float/Float | float/pyfury.Float16 | Type.float16() | fury::float16_t| fury.float16 | fury::f16 | +| float32| 10 | float/Float | float/pyfury.Float32 | Type.float32() | float | float32 | f32 | +| float64| 11 | double/Double | float/pyfury.Float64 | Type.float64() | double | float64 | f64 | +| string | 12 | String | str | String | string | string | String/str | +| enum | 13 | Enum subclasses | enum subclasses | / | enum | /| enum | +| list | 14 | List/Collection | list/tuple | array | vector | slice| Vec | +| set| 15 | Set | set | / | set| fury.Set | Set | +| map| 16 | Map | dict | Map | unordered_map | map | HashMap | +| duration | 17 | Duration| timedelta| Number | duration | Duration | Duration | +| timestamp | 18 | Instant | datetime | Number | std::chrono::nanoseconds | Time | DateTime | +| decimal| 19 | BigDecimal | Decimal | bigint | / | /| / | +| binary | 20 | byte[] | bytes| / | `int8_t[n]/vector` | `[n]int8/[]T`| `Vec` | +| array | 20 | array | np.ndarray | / | / | array/slice | Vec | Review Comment: We can serialize a bytes to int8 array and deserialize int8 array to bytes. But it would introduce some confusion. since it is so, we keep it. but should assign a new Typeid to binary to differentiate it from the array. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To
Re: [PR] feat(doc): add type mapping for xlang serialization [incubator-fury]
chaokunyang commented on code in PR #1476: URL: https://github.com/apache/incubator-fury/pull/1476#discussion_r1559454151 ## docs/guide/xlang_type_mapping.md: ## @@ -4,4 +4,76 @@ sidebar_position: 3 id: xlang_type_mapping --- -Coming soon. +Note: + +- For type definition, see [Type Systems in Spec](../specification/xlang_serialization_spec.md#type-systems) +- `int16_t[n]/vector` indicates `int16_t[n]/vector` +- The cross-language serialization is not stable, do not use it in your production environment. + +# Type Mapping + +| Type | Type ID | Java| Python | Javascript | C++| Golang | Rust | +||-|-|--|-||--|--| +| bool | 1 | bool/Boolean| bool | Boolean | bool | bool | bool | +| int8 | 2 | byte/Byte | int/pyfury.Int8 | Type.int8() | int8_t | int8 | i8 | +| int16 | 3 | short/Short | int/pyfury.Int16 | Type.int16()| int16_t| int16| i6 | +| int32 | 4 | int/Integer | int/pyfury.Int32 | Type.int32()| int32_t| int32| i32 | +| var_int32 | 5 | int/Integer | int/pyfury.VarInt32 | Type.varint32() | fury::varint32_t | fury.varint32| fury::varint32 | +| int64 | 6 | long/Long | int/pyfury.Int64 | Type.int64()| int64_t| int64| i64 | +| var_int64 | 7 | long/Long | int/pyfury.VarInt64 | Type.varint64() | fury::varint64_t | fury.varint64| fury::varint64 | +| sli_int64 | 8 | long/Long | int/pyfury.SliInt64 | Type.sliint64() | fury::sliint64_t | fury.sliint64| fury::sliint64 | +| float16| 9 | float/Float | float/pyfury.Float16 | Type.float16() | fury::float16_t| fury.float16 | fury::f16 | +| float32| 10 | float/Float | float/pyfury.Float32 | Type.float32() | float | float32 | f32 | +| float64| 11 | double/Double | float/pyfury.Float64 | Type.float64() | double | float64 | f64 | +| string | 12 | String | str | String | string | string | String/str | +| enum | 13 | Enum subclasses | enum subclasses | / | enum | /| enum | +| list | 14 | List/Collection | list/tuple | array | vector | slice| Vec | +| set| 15 | Set | set | / | set| fury.Set | Set | +| map| 16 | Map | dict | Map | unordered_map | map | HashMap | +| duration | 17 | Duration| timedelta| Number | duration | Duration | Duration | +| timestamp | 18 | Instant | datetime | Number | std::chrono::nanoseconds | Time | DateTime | +| decimal| 19 | BigDecimal | Decimal | bigint | / | /| / | +| binary | 20 | byte[] | bytes| / | `int8_t[n]/vector` | `[n]int8/[]T`| `Vec` | +| array | 20 | array | np.ndarray | / | / | array/slice | Vec | Review Comment: Thrift also has a binary type. And we don't provide uint8 type for now. The int8 array can be a different type -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.
Re: [PR] feat(doc): add type mapping for xlang serialization [incubator-fury]
chaokunyang commented on code in PR #1476: URL: https://github.com/apache/incubator-fury/pull/1476#discussion_r1559450427 ## docs/guide/xlang_type_mapping.md: ## @@ -4,4 +4,76 @@ sidebar_position: 3 id: xlang_type_mapping --- -Coming soon. +Note: + +- For type definition, see [Type Systems in Spec](../specification/xlang_serialization_spec.md#type-systems) +- `int16_t[n]/vector` indicates `int16_t[n]/vector` +- The cross-language serialization is not stable, do not use it in your production environment. + +# Type Mapping + +| Type | Type ID | Java| Python | Javascript | C++| Golang | Rust | +||-|-|--|-||--|--| +| bool | 1 | bool/Boolean| bool | Boolean | bool | bool | bool | +| int8 | 2 | byte/Byte | int/pyfury.Int8 | Type.int8() | int8_t | int8 | i8 | +| int16 | 3 | short/Short | int/pyfury.Int16 | Type.int16()| int16_t| int16| i6 | +| int32 | 4 | int/Integer | int/pyfury.Int32 | Type.int32()| int32_t| int32| i32 | +| var_int32 | 5 | int/Integer | int/pyfury.VarInt32 | Type.varint32() | fury::varint32_t | fury.varint32| fury::varint32 | +| int64 | 6 | long/Long | int/pyfury.Int64 | Type.int64()| int64_t| int64| i64 | +| var_int64 | 7 | long/Long | int/pyfury.VarInt64 | Type.varint64() | fury::varint64_t | fury.varint64| fury::varint64 | +| sli_int64 | 8 | long/Long | int/pyfury.SliInt64 | Type.sliint64() | fury::sliint64_t | fury.sliint64| fury::sliint64 | +| float16| 9 | float/Float | float/pyfury.Float16 | Type.float16() | fury::float16_t| fury.float16 | fury::f16 | +| float32| 10 | float/Float | float/pyfury.Float32 | Type.float32() | float | float32 | f32 | +| float64| 11 | double/Double | float/pyfury.Float64 | Type.float64() | double | float64 | f64 | +| string | 12 | String | str | String | string | string | String/str | +| enum | 13 | Enum subclasses | enum subclasses | / | enum | /| enum | +| list | 14 | List/Collection | list/tuple | array | vector | slice| Vec | +| set| 15 | Set | set | / | set| fury.Set | Set | +| map| 16 | Map | dict | Map | unordered_map | map | HashMap | +| duration | 17 | Duration| timedelta| Number | duration | Duration | Duration | +| timestamp | 18 | Instant | datetime | Number | std::chrono::nanoseconds | Time | DateTime | +| decimal| 19 | BigDecimal | Decimal | bigint | / | /| / | +| binary | 20 | byte[] | bytes| / | `int8_t[n]/vector` | `[n]int8/[]T`| `Vec` | +| array | 20 | array | np.ndarray | / | / | array/slice | Vec | Review Comment: There are many languages which have a dedicated type for binary instead of array. For example, python uses `bytes` for binary type. Binary us not used as an array in many cases. I suggest we keep it. There is also a binary type in Arrow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" is working again!
The GitHub Actions job "Fury CI" on incubator-fury.git has succeeded. Run started by GitHub user theweipeng (triggered by theweipeng). Head commit for run: 86685441961266f174c3a884b13c32def847a98e / weipeng add license header Report URL: https://github.com/apache/incubator-fury/actions/runs/8631463805 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user theweipeng (triggered by theweipeng). Head commit for run: 05e3ee850fb4b8bdfd3a1443b5a2b9f45948c8c8 / weipeng lint fix Report URL: https://github.com/apache/incubator-fury/actions/runs/8631413148 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user theweipeng (triggered by theweipeng). Head commit for run: c68d56fb4eece68ed2dd5000592e871595257963 / weipeng remove the binary type, uint8 array is enough Report URL: https://github.com/apache/incubator-fury/actions/runs/8631373919 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Lint PR" is working again!
The GitHub Actions job "Lint PR" on incubator-fury.git has succeeded. Run started by GitHub user theweipeng (triggered by theweipeng). Head commit for run: 05e3ee850fb4b8bdfd3a1443b5a2b9f45948c8c8 / weipeng lint fix Report URL: https://github.com/apache/incubator-fury/actions/runs/8631427771 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Lint PR" failed!
The GitHub Actions job "Lint PR" on incubator-fury.git has failed. Run started by GitHub user theweipeng (triggered by theweipeng). Head commit for run: 05e3ee850fb4b8bdfd3a1443b5a2b9f45948c8c8 / weipeng lint fix Report URL: https://github.com/apache/incubator-fury/actions/runs/8631413046 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user theweipeng (triggered by theweipeng). Head commit for run: 3fae0196bd8a5720008a68019c424e4205a7694c / weipeng Merge branch 'main' into xlang_javascript Report URL: https://github.com/apache/incubator-fury/actions/runs/8631354358 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user theweipeng (triggered by theweipeng). Head commit for run: f4b90eafeeb6ba2a1de43b1b3d1292491d759bd1 / weipeng test Report URL: https://github.com/apache/incubator-fury/actions/runs/8631350758 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(JavaScript) implement xlang protocol [incubator-fury]
theweipeng commented on PR #1487: URL: https://github.com/apache/incubator-fury/pull/1487#issuecomment-2047438880 @LiangliangSui @chaokunyang Hi, This PR addresses the numeric part of the xlang protocol; could you please help review it? The other parts of the protocol will be submitted in another PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Lint PR" failed!
The GitHub Actions job "Lint PR" on incubator-fury.git has failed. Run started by GitHub user theweipeng (triggered by theweipeng). Head commit for run: c68d56fb4eece68ed2dd5000592e871595257963 / weipeng remove the binary type, uint8 array is enough Report URL: https://github.com/apache/incubator-fury/actions/runs/8631373497 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Lint PR" failed!
The GitHub Actions job "Lint PR" on incubator-fury.git has failed. Run started by GitHub user theweipeng (triggered by theweipeng). Head commit for run: 3fae0196bd8a5720008a68019c424e4205a7694c / weipeng Merge branch 'main' into xlang_javascript Report URL: https://github.com/apache/incubator-fury/actions/runs/8631354083 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Lint PR" failed!
The GitHub Actions job "Lint PR" on incubator-fury.git has failed. Run started by GitHub user theweipeng (triggered by theweipeng). Head commit for run: f4b90eafeeb6ba2a1de43b1b3d1292491d759bd1 / weipeng test Report URL: https://github.com/apache/incubator-fury/actions/runs/8631350748 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(doc): add type mapping for xlang serialization [incubator-fury]
theweipeng commented on code in PR #1476: URL: https://github.com/apache/incubator-fury/pull/1476#discussion_r1559314911 ## docs/guide/xlang_type_mapping.md: ## @@ -4,4 +4,76 @@ sidebar_position: 3 id: xlang_type_mapping --- -Coming soon. +Note: + +- For type definition, see [Type Systems in Spec](../specification/xlang_serialization_spec.md#type-systems) +- `int16_t[n]/vector` indicates `int16_t[n]/vector` +- The cross-language serialization is not stable, do not use it in your production environment. + +# Type Mapping + +| Type | Type ID | Java| Python | Javascript | C++| Golang | Rust | +||-|-|--|-||--|--| +| bool | 1 | bool/Boolean| bool | Boolean | bool | bool | bool | +| int8 | 2 | byte/Byte | int/pyfury.Int8 | Type.int8() | int8_t | int8 | i8 | +| int16 | 3 | short/Short | int/pyfury.Int16 | Type.int16()| int16_t| int16| i6 | +| int32 | 4 | int/Integer | int/pyfury.Int32 | Type.int32()| int32_t| int32| i32 | +| var_int32 | 5 | int/Integer | int/pyfury.VarInt32 | Type.varint32() | fury::varint32_t | fury.varint32| fury::varint32 | +| int64 | 6 | long/Long | int/pyfury.Int64 | Type.int64()| int64_t| int64| i64 | +| var_int64 | 7 | long/Long | int/pyfury.VarInt64 | Type.varint64() | fury::varint64_t | fury.varint64| fury::varint64 | +| sli_int64 | 8 | long/Long | int/pyfury.SliInt64 | Type.sliint64() | fury::sliint64_t | fury.sliint64| fury::sliint64 | +| float16| 9 | float/Float | float/pyfury.Float16 | Type.float16() | fury::float16_t| fury.float16 | fury::f16 | +| float32| 10 | float/Float | float/pyfury.Float32 | Type.float32() | float | float32 | f32 | +| float64| 11 | double/Double | float/pyfury.Float64 | Type.float64() | double | float64 | f64 | +| string | 12 | String | str | String | string | string | String/str | +| enum | 13 | Enum subclasses | enum subclasses | / | enum | /| enum | +| list | 14 | List/Collection | list/tuple | array | vector | slice| Vec | +| set| 15 | Set | set | / | set| fury.Set | Set | +| map| 16 | Map | dict | Map | unordered_map | map | HashMap | +| duration | 17 | Duration| timedelta| Number | duration | Duration | Duration | +| timestamp | 18 | Instant | datetime | Number | std::chrono::nanoseconds | Time | DateTime | +| decimal| 19 | BigDecimal | Decimal | bigint | / | /| / | +| binary | 20 | byte[] | bytes| / | `int8_t[n]/vector` | `[n]int8/[]T`| `Vec` | +| array | 20 | array | np.ndarray | / | / | array/slice | Vec | Review Comment: The `binary` type can be perfectly replaced by the `array` type, as they are assigned the same `typeId`. I suggest we remove the `binary` type to simplify our protocol. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@
(incubator-fury) branch main updated: feat(java): channel stream reader (#1483)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury.git
The following commit(s) were added to refs/heads/main by this push:
new 71c5b76f feat(java): channel stream reader (#1483)
71c5b76f is described below
commit 71c5b76fa6affd7453be27709ae2dfcc90eae52d
Author: Nikita Ivchenko
AuthorDate: Wed Apr 10 10:44:02 2024 +0300
feat(java): channel stream reader (#1483)
Co-authored-by: Shawn Yang
---
.../src/main/java/org/apache/fury/BaseFury.java| 9 ++
.../src/main/java/org/apache/fury/Fury.java| 32 ++
.../main/java/org/apache/fury/ThreadLocalFury.java | 21
.../java/org/apache/fury/io/FuryInputStream.java | 1 -
.../org/apache/fury/io/FuryReadableChannel.java| 126 ++---
.../java/org/apache/fury/io/FuryStreamReader.java | 13 +++
.../java/org/apache/fury/memory/MemoryBuffer.java | 20 +++-
.../java/org/apache/fury/pool/ThreadPoolFury.java | 21
.../src/test/java/org/apache/fury/StreamTest.java | 51 +
9 files changed, 272 insertions(+), 22 deletions(-)
diff --git a/java/fury-core/src/main/java/org/apache/fury/BaseFury.java
b/java/fury-core/src/main/java/org/apache/fury/BaseFury.java
index 1bcb31ee..91a62cc4 100644
--- a/java/fury-core/src/main/java/org/apache/fury/BaseFury.java
+++ b/java/fury-core/src/main/java/org/apache/fury/BaseFury.java
@@ -21,6 +21,7 @@ package org.apache.fury;
import java.io.OutputStream;
import org.apache.fury.io.FuryInputStream;
+import org.apache.fury.io.FuryReadableChannel;
import org.apache.fury.memory.MemoryBuffer;
import org.apache.fury.serializer.BufferCallback;
import org.apache.fury.serializer.Serializer;
@@ -114,6 +115,10 @@ public interface BaseFury {
Object deserialize(FuryInputStream inputStream, Iterable
outOfBandBuffers);
+ Object deserialize(FuryReadableChannel channel);
+
+ Object deserialize(FuryReadableChannel channel, Iterable
outOfBandBuffers);
+
/**
* Serialize java object without class info, deserialization should use
{@link
* #deserializeJavaObject}.
@@ -142,6 +147,8 @@ public interface BaseFury {
T deserializeJavaObject(FuryInputStream inputStream, Class cls);
+ T deserializeJavaObject(FuryReadableChannel channel, Class cls);
+
byte[] serializeJavaObjectAndClass(Object obj);
void serializeJavaObjectAndClass(MemoryBuffer buffer, Object obj);
@@ -153,4 +160,6 @@ public interface BaseFury {
Object deserializeJavaObjectAndClass(MemoryBuffer buffer);
Object deserializeJavaObjectAndClass(FuryInputStream inputStream);
+
+ Object deserializeJavaObjectAndClass(FuryReadableChannel channel);
}
diff --git a/java/fury-core/src/main/java/org/apache/fury/Fury.java
b/java/fury-core/src/main/java/org/apache/fury/Fury.java
index 63421ada..e1a8bbd5 100644
--- a/java/fury-core/src/main/java/org/apache/fury/Fury.java
+++ b/java/fury-core/src/main/java/org/apache/fury/Fury.java
@@ -38,6 +38,7 @@ import org.apache.fury.config.Language;
import org.apache.fury.config.LongEncoding;
import org.apache.fury.exception.DeserializationException;
import org.apache.fury.io.FuryInputStream;
+import org.apache.fury.io.FuryReadableChannel;
import org.apache.fury.logging.Logger;
import org.apache.fury.logging.LoggerFactory;
import org.apache.fury.memory.MemoryBuffer;
@@ -771,6 +772,17 @@ public final class Fury implements BaseFury {
}
}
+ @Override
+ public Object deserialize(FuryReadableChannel channel) {
+return deserialize(channel, null);
+ }
+
+ @Override
+ public Object deserialize(FuryReadableChannel channel,
Iterable outOfBandBuffers) {
+MemoryBuffer buf = channel.getBuffer();
+return deserialize(buf, outOfBandBuffers);
+ }
+
private RuntimeException handleReadFailed(Throwable t) {
if (refResolver instanceof MapRefResolver) {
ObjectArray readObjects = ((MapRefResolver)
refResolver).getReadObjects();
@@ -1092,6 +1104,16 @@ public final class Fury implements BaseFury {
}
}
+ /**
+ * Deserialize java object from binary channel by passing class info,
serialization should use
+ * {@link #serializeJavaObject}.
+ */
+ @Override
+ public T deserializeJavaObject(FuryReadableChannel channel, Class
cls) {
+MemoryBuffer buf = channel.getBuffer();
+return deserializeJavaObject(buf, cls);
+ }
+
/**
* Deserialize java object from binary by passing class info, serialization
should use {@link
* #deserializeJavaObjectAndClass}.
@@ -1181,6 +1203,16 @@ public final class Fury implements BaseFury {
}
}
+ /**
+ * Deserialize class info and java object from binary channel, serialization
should use {@link
+ * #serializeJavaObjectAndClass}.
+ */
+ @Override
+ public Object deserializeJavaObjectAndClass(FuryReadableChannel channel) {
+MemoryBuffer buf = channel.getBuffer();
+return deserializeJavaObjectAndClass(buf);
+ }
Re: [PR] feat(java): channel stream reader [incubator-fury]
chaokunyang merged PR #1483: URL: https://github.com/apache/incubator-fury/pull/1483 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(java): channel stream reader [incubator-fury]
Munoon commented on code in PR #1483:
URL: https://github.com/apache/incubator-fury/pull/1483#discussion_r1558960787
##
java/fury-core/src/main/java/org/apache/fury/io/FuryReadableChannel.java:
##
@@ -128,21 +128,21 @@ public void readToUnsafe(Object target, long
targetPointer, int numBytes) {
@Override
public void readToByteBuffer(ByteBuffer dst, int length) {
-readToByteBuffer0(dst, length);
+MemoryBuffer buf = memoryBuffer;
+int remaining = buf.remaining();
+if (remaining < length) {
+ remaining += fillBuffer(length - remaining);
+}
+buf.read(dst, remaining);
}
@Override
public int readToByteBuffer(ByteBuffer dst) {
-return readToByteBuffer0(dst, dst.remaining());
- }
-
- private int readToByteBuffer0(ByteBuffer dst, int length) {
MemoryBuffer buf = memoryBuffer;
int remaining = buf.remaining();
-if (remaining < length) {
- remaining += fillBuffer(length - remaining);
+if (remaining > 0) {
+ buf.read(dst, remaining);
Review Comment:
Fine, so we are expecting the the `ByteBuffer dst` have already set up the
correct limit before invoking this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
