Re: [PR] docs: Use perf type when pr has performance impact. [incubator-fury]
LiangliangSui commented on PR #1474: URL: https://github.com/apache/incubator-fury/pull/1474#issuecomment-2041312743 > LGTM, We may need to update our pr template too The pr template already contains [perf type](https://github.com/apache/incubator-fury/blob/main/.github/workflows/pr-lint.yml#L48). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury) branch main updated: docs: Use perf type when pr has performance impact. (#1474)
This is an automated email from the ASF dual-hosted git repository. chaokunyang pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-fury.git The following commit(s) were added to refs/heads/main by this push: new f892802f docs: Use perf type when pr has performance impact. (#1474) f892802f is described below commit f892802f6d7b6d28134ebc77c47f887f9089bf36 Author: LiangliangSui <116876207+liangliang...@users.noreply.github.com> AuthorDate: Sun Apr 7 12:33:09 2024 +0800 docs: Use perf type when pr has performance impact. (#1474) When PR has an impact on Fury performance, it is recommended to use perf type and provide benchmark data in the PR description. Signed-off-by: LiangliangSui --- CONTRIBUTING.md | 4 1 file changed, 4 insertions(+) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 22115e0d..b7558aff 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -18,6 +18,10 @@ For example, here are good PR titles: - fix(c++): blablabla - chore(python): remove useless yyy file +If the submitted PR affects the performance of Fury, we strongly recommend using the perf type, +and need to provide benchmark data in the PR description. For how to run the benchmark, +please check [Fury Java Benchmark](https://github.com/apache/incubator-fury/blob/main/java/benchmark/README.md). + For more details, please check [pr-lint.yml](./.github/workflows/pr-lint.yml). ## Testing - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] docs: Use perf type when pr has performance impact. [incubator-fury]
chaokunyang merged PR #1474: URL: https://github.com/apache/incubator-fury/pull/1474 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[PR] docs: Use perf type when pr has performance impact. [incubator-fury]
LiangliangSui opened a new pull request, #1474: URL: https://github.com/apache/incubator-fury/pull/1474 When PR has an impact on Fury performance, it is recommended to use perf type and provide benchmark data in the PR description. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] [Java] FuryBuilder supports registered classes [incubator-fury]
LiangliangSui commented on issue #1459: URL: https://github.com/apache/incubator-fury/issues/1459#issuecomment-2041296983 I agree with you, `Fury` and `ClassRegister` have a `has-a` relationship. If we don't implement some interfaces in `FuryBuilder`, then we can only repeatedly define the registerXXX function, or do you have some other good suggestions? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] [Java] FuryBuilder supports registered classes [incubator-fury]
chaokunyang commented on issue #1459: URL: https://github.com/apache/incubator-fury/issues/1459#issuecomment-2041287630 I perfer not. `Fury` is not a `ClassRegister`, there is no such a straightforward `is-a` relationship between these two classes. And it's not that intuitive for a builder class to implement some interface. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [D] At least two owners approve before they can be merged PR. [incubator-fury]
GitHub user chaokunyang added a comment to the discussion: At least two owners approve before they can be merged PR. That's what I want to do too. But our community is still in early stage, not all code owner are always online, and unfortunately we don't have enough owners at this time. So I suggest to defer this to next month, hopefully we can have a stronger community. GitHub link: https://github.com/apache/incubator-fury/discussions/1473#discussioncomment-9033275 This is an automatically sent email for commits@fury.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@fury.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [D] At least two owners approve before they can be merged PR. [incubator-fury]
GitHub user chaokunyang edited a comment on the discussion: At least two owners approve before they can be merged PR. @LiangliangSui That's what I want to do too. But our community is still in early stage, not all code owner are always online, and unfortunately we don't have enough owners at this time. So I suggest to defer this to next month, hopefully we can have a stronger community. GitHub link: https://github.com/apache/incubator-fury/discussions/1473#discussioncomment-9033275 This is an automatically sent email for commits@fury.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@fury.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury) branch main updated: chore(java): add comments for loop unrolling. (#1450)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury.git
The following commit(s) were added to refs/heads/main by this push:
new d28b14da chore(java): add comments for loop unrolling. (#1450)
d28b14da is described below
commit d28b14dabaa6eccedf53353a8de2ca6925ffd48e
Author: LiangliangSui <116876207+liangliang...@users.noreply.github.com>
AuthorDate: Sun Apr 7 01:31:46 2024 +0800
chore(java): add comments for loop unrolling. (#1450)
N/A
-
Signed-off-by: LiangliangSui
---
.../src/main/java/org/apache/fury/memory/MemoryBuffer.java | 9 +
1 file changed, 9 insertions(+)
diff --git
a/java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java
b/java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java
index 025ceb28..011b31a7 100644
--- a/java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java
+++ b/java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java
@@ -1239,6 +1239,9 @@ public final class MemoryBuffer {
}
private int readPositiveVarIntSlow() {
+// Note:
+// Loop are not used here to improve performance,
+// we manually unroll the loop for better performance.
int b = readByte();
int result = b & 0x7F;
if ((b & 0x80) != 0) {
@@ -1468,6 +1471,9 @@ public final class MemoryBuffer {
// Mask first 6 bits,
// bit 8 `set` indicates have next data bytes.
int result = b & 0x3F;
+// Note:
+// Loop are not used here to improve performance.
+// We manually unroll the loop for better performance.
if ((b & 0x80) != 0) { // has 2nd byte
b = UNSAFE.getByte(heapMemory, pos++);
result |= (b & 0x3F) << 6;
@@ -1768,6 +1774,9 @@ public final class MemoryBuffer {
private long readPositiveVarLongSlow() {
long b = readByte();
long result = b & 0x7F;
+// Note:
+// Loop are not used here to improve performance.
+// We manually unroll the loop for better performance.
if ((b & 0x80) != 0) {
b = readByte();
result |= (b & 0x7F) << 7;
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] chore(java): add comments for loop unrolling. [incubator-fury]
chaokunyang merged PR #1450: URL: https://github.com/apache/incubator-fury/pull/1450 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(java): optimize read float/double for jvm jit inline [incubator-fury]
chaokunyang commented on code in PR #1472:
URL: https://github.com/apache/incubator-fury/pull/1472#discussion_r1554641365
##
java/fury-core/src/test/java/org/apache/fury/serializer/PrimitiveSerializersTest.java:
##
@@ -54,4 +59,62 @@ public void testUint16Serializer() {
assertThrows(IllegalArgumentException.class, () ->
serializer.xwrite(buffer, -1));
assertThrows(IllegalArgumentException.class, () ->
serializer.xwrite(buffer, 65536));
}
+
+ @Data
+ @AllArgsConstructor
+ public static class PrimitiveStruct {
Review Comment:
Private will be better. Fury don't support private class codegen previously,
so I used the public class in tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(java): optimize read float/double for jvm jit inline [incubator-fury]
chaokunyang commented on PR #1472: URL: https://github.com/apache/incubator-fury/pull/1472#issuecomment-2041145222 Here is the benchmark result: ``` With This PR: Benchmark (bufferType) (objectType) (references) Mode CntScoreError Units UserTypeDeserializeSuite.fury_deserialize array Struct false thrpt 10 4466936.113 ± 386617.618 ops/s UserTypeDeserializeSuite.fury_deserialize array Struct true thrpt 10 3963160.097 ± 178393.730 ops/s UserTypeDeserializeSuite.fury_deserialize directBuffer Struct false thrpt 10 4879409.124 ± 79998.330 ops/s UserTypeDeserializeSuite.fury_deserialize directBuffer Struct true thrpt 10 4140782.389 ± 95492.701 ops/s Before this PR: Benchmark (bufferType) (objectType) (references) Mode CntScoreError Units UserTypeDeserializeSuite.fury_deserialize array Struct false thrpt 10 4135698.223 ± 344998.039 ops/s UserTypeDeserializeSuite.fury_deserialize array Struct true thrpt 10 4062397.644 ± 94063.642 ops/s UserTypeDeserializeSuite.fury_deserialize directBuffer Struct false thrpt 10 4311519.019 ± 251129.086 ops/s UserTypeDeserializeSuite.fury_deserialize directBuffer Struct true thrpt 10 4056600.395 ± 414134.048 ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(java): optimize read float/double for jvm jit inline [incubator-fury]
chaokunyang commented on PR #1472: URL: https://github.com/apache/incubator-fury/pull/1472#issuecomment-2041140777 > Hi @chaokunyang , I have a suggestion. For this kind of optimization PR, we should force the submitter to provide a benchmark data in the PR, so that we can ensure that the optimization is beneficial to the codebase. > > Then for the PR of the optimization category, we should force the PR title to indicate that it is an optimization PR (standardizing the Fury community). We can add a new category `opt` in [pr-lint.yml](https://github.com/apache/incubator-fury/blob/main/.github/workflows/pr-lint.yml), and I will submit a PR to do this. How abouting add a `perf` category, as described in https://platform.uno/docs/articles/uno-development/git-conventional-commits.html. `opt` can represent many things, like `refactor` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(java): optimize read float/double for jvm jit inline [incubator-fury]
chaokunyang commented on code in PR #1472:
URL: https://github.com/apache/incubator-fury/pull/1472#discussion_r1554641365
##
java/fury-core/src/test/java/org/apache/fury/serializer/PrimitiveSerializersTest.java:
##
@@ -54,4 +59,62 @@ public void testUint16Serializer() {
assertThrows(IllegalArgumentException.class, () ->
serializer.xwrite(buffer, -1));
assertThrows(IllegalArgumentException.class, () ->
serializer.xwrite(buffer, 65536));
}
+
+ @Data
+ @AllArgsConstructor
+ public static class PrimitiveStruct {
Review Comment:
Private will work here too, but the generate code would be different.
Codegen work best for public classes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(java): optimize read float/double for jvm jit inline [incubator-fury]
chaokunyang commented on code in PR #1472:
URL: https://github.com/apache/incubator-fury/pull/1472#discussion_r1554641104
##
java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java:
##
@@ -2318,6 +2205,37 @@ public float readFloat() {
}
}
+ // Reduce method body for better inline in the caller.
+ @CodegenInvoke
+ public float readFloatOnLE() {
+int readerIdx = readerIndex;
+// use subtract to avoid overflow
+int remaining = size - readerIdx;
+if (remaining < 4) {
+ throw new IndexOutOfBoundsException(
Review Comment:
This is temprary , it will be replaced by `streamReader.fillBuffer` in
#1451. The exception will be trown in `DoundChecker`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] chore(java): Reuse unsafePutPositiveVarInt in unsafeWritePositiveVarInt [incubator-fury]
chaokunyang commented on code in PR #1434:
URL: https://github.com/apache/incubator-fury/pull/1434#discussion_r1554640857
##
java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java:
##
@@ -1233,49 +1233,9 @@ public int readVarInt() {
* to avoid using two memory operations.
*/
public int unsafeWritePositiveVarInt(int v) {
-// The encoding algorithm are based on kryo UnsafeMemoryOutput.writeVarInt
-// varint are written using little endian byte order.
-// This version should have better performance since it remove an index
update.
-long value = v;
-final int writerIndex = this.writerIndex;
-long varInt = (value & 0x7F);
-value >>>= 7;
-if (value == 0) {
- UNSAFE.putByte(heapMemory, address + writerIndex, (byte) varInt);
- this.writerIndex = writerIndex + 1;
- return 1;
-}
-// bit 8 `set` indicates have next data bytes.
-varInt |= 0x80;
-varInt |= ((value & 0x7F) << 8);
-value >>>= 7;
-if (value == 0) {
- unsafePutInt(writerIndex, (int) varInt);
- this.writerIndex = writerIndex + 2;
- return 2;
-}
-varInt |= (0x80 << 8);
-varInt |= ((value & 0x7F) << 16);
-value >>>= 7;
-if (value == 0) {
- unsafePutInt(writerIndex, (int) varInt);
- this.writerIndex = writerIndex + 3;
- return 3;
-}
-varInt |= (0x80 << 16);
-varInt |= ((value & 0x7F) << 24);
-value >>>= 7;
-if (value == 0) {
- unsafePutInt(writerIndex, (int) varInt);
- this.writerIndex = writerIndex + 4;
- return 4;
-}
-varInt |= (0x80L << 24);
-varInt |= ((value & 0x7F) << 32);
-varInt &= 0xFL;
-unsafePutLong(writerIndex, varInt);
-this.writerIndex = writerIndex + 5;
-return 5;
+int varintBytes = unsafePutPositiveVarInt(writerIndex, v);
Review Comment:
This looks good, does this have an influence in a end-to-end benchmark? JVM
inline has max depth limit, the current case shouldn't be influenced by this
optimization. Could you run fury benchmark for MediaContent object?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(java): native streaming mode deserialization [incubator-fury]
chaokunyang commented on PR #1451: URL: https://github.com/apache/incubator-fury/pull/1451#issuecomment-2041128594 New readBytesString are too large for inline:  Old version:  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] [Java] FuryBuilder supports registered classes [incubator-fury]
LiangliangSui commented on issue #1459: URL: https://github.com/apache/incubator-fury/issues/1459#issuecomment-2041122588 > One concern is that BaseFury already supports register classes/serializers, do we still need to add this API to FuryBuilder? We can extract all `register(...)` methods in `BaseFury` into an interface, for example named `ClassRegister`, and then let `BaseFury` extends `ClassRegister`, and `FuryBuilder` can also implements `ClassRegister`. `registerSerializer(...)` is similar, we can extract an interface named `SerializerRegister`. WDYT? @chaokunyang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(spec): add xlang type mapping [incubator-fury]
LiangliangSui commented on code in PR #1468: URL: https://github.com/apache/incubator-fury/pull/1468#discussion_r1554628240 ## docs/guide/xlang_type_mapping.md: ## @@ -4,4 +4,75 @@ sidebar_position: 3 id: xlang_type_mapping --- -Coming soon. +Note: + +- For type definition, see [Type Systems in Spec](../specification/xlang_serialization_spec.md#type-systems) +- `int16_t[x]/vector<~>` indicates `int16_t[x]/vector` +- The cross-language serialization is not stable, do not use it in your production environment. + +# Type Mapping + +| Type | Type ID | Java| Python | Javascript | C++| Golang | Rust | +||-|-|--|-||--|--| +| bool | 1 | bool/Boolean| bool | Boolean | bool | bool | bool | +| int8 | 2 | byte/Byte | int/pyfury.Int8 | Type.int8() | int8_t | int8 | i8 | +| int16 | 3 | short/Short | int/pyfury.Int16 | Type.int16()| int16_t| int16| i6 | +| int32 | 4 | int/Integer | int/pyfury.Int32 | Type.int32()| int32_t| int32| i32 | +| var_int32 | 5 | int/Integer | int/pyfury.VarInt32 | Type.varint32() | fury::varint32_t | fury.varint32| fury::varint32 | +| int64 | 6 | long/Long | int/pyfury.Int64 | Type.int64()| int64_t| int64| i64 | +| var_int64 | 7 | long/Long | int/pyfury.VarInt64 | Type.varint64() | fury::varint64_t | fury.varint64| fury::varint64 | +| sli_int64 | 8 | long/Long | int/pyfury.SliInt64 | Type.sliint64() | fury::sliint64_t | fury.sliint64| fury::sliint64 | +| float16| 9 | float/Float | float/pyfury.Float16 | Type.float16() | fury::float16_t| fury.float16 | fury::f16 | +| float32| 10 | float/Float | float/pyfury.Float32 | Type.float32() | float | float32 | f32 | +| float64| 11 | double/Double | float/pyfury.Float64 | Type.float64() | double | float64 | f64 | +| string | 12 | String | str | String | string | string | String/str | +| enum | 13 | Enum subclasses | enum subclasses | / | enum | /| enum | +| list | 14 | List/Collection | list/tuple | array | vector | slice| Vec | +| set| 15 | Set | set | / | set| fury.Set | Set | +| map| 16 | Map | dict | Map | unordered_map | map | HashMap | +| duration | 17 | Duration| timedelta| Number | duration | Duration | Duration | +| timestamp | 18 | Instant | datetime | Number | std::chrono::nanoseconds | Time | DateTime | +| decimal| 19 | BigDecimal | Decimal | bigint | / | /| / | +| array | 20 | array | np.ndarray | / | / | array/slice | Vec | +| bool_array | 21 | bool[] | ndarray(np.bool_)| / | `bool[x]` | `[x]bool/[]~`| `Vec` | +| int16_array| 22 | short[] | ndarray(int16) | / | `int16_t[x]/vector<~>` | `[x]int16/[]~` | `Vec` | +| int32_array| 23 | int[] | ndarray(int32) | / | `int32_t[x]/vector<~>` | `[x]int32/[]~` | `Vec` | +| int64_array| 24 | long[] | ndarray(int64) | / | `int64_t[x]/vector<~>` | `[x]int64/[]~` | `Vec` | +| float16_array | 25 | float[] | ndarray(float16) | / | `fury::float16_t[x]/vector<~>` | `[x]float16/[]~` |
Re: [PR] feat(spec): add xlang type mapping [incubator-fury]
LiangliangSui commented on code in PR #1468: URL: https://github.com/apache/incubator-fury/pull/1468#discussion_r1554628240 ## docs/guide/xlang_type_mapping.md: ## @@ -4,4 +4,75 @@ sidebar_position: 3 id: xlang_type_mapping --- -Coming soon. +Note: + +- For type definition, see [Type Systems in Spec](../specification/xlang_serialization_spec.md#type-systems) +- `int16_t[x]/vector<~>` indicates `int16_t[x]/vector` +- The cross-language serialization is not stable, do not use it in your production environment. + +# Type Mapping + +| Type | Type ID | Java| Python | Javascript | C++| Golang | Rust | +||-|-|--|-||--|--| +| bool | 1 | bool/Boolean| bool | Boolean | bool | bool | bool | +| int8 | 2 | byte/Byte | int/pyfury.Int8 | Type.int8() | int8_t | int8 | i8 | +| int16 | 3 | short/Short | int/pyfury.Int16 | Type.int16()| int16_t| int16| i6 | +| int32 | 4 | int/Integer | int/pyfury.Int32 | Type.int32()| int32_t| int32| i32 | +| var_int32 | 5 | int/Integer | int/pyfury.VarInt32 | Type.varint32() | fury::varint32_t | fury.varint32| fury::varint32 | +| int64 | 6 | long/Long | int/pyfury.Int64 | Type.int64()| int64_t| int64| i64 | +| var_int64 | 7 | long/Long | int/pyfury.VarInt64 | Type.varint64() | fury::varint64_t | fury.varint64| fury::varint64 | +| sli_int64 | 8 | long/Long | int/pyfury.SliInt64 | Type.sliint64() | fury::sliint64_t | fury.sliint64| fury::sliint64 | +| float16| 9 | float/Float | float/pyfury.Float16 | Type.float16() | fury::float16_t| fury.float16 | fury::f16 | +| float32| 10 | float/Float | float/pyfury.Float32 | Type.float32() | float | float32 | f32 | +| float64| 11 | double/Double | float/pyfury.Float64 | Type.float64() | double | float64 | f64 | +| string | 12 | String | str | String | string | string | String/str | +| enum | 13 | Enum subclasses | enum subclasses | / | enum | /| enum | +| list | 14 | List/Collection | list/tuple | array | vector | slice| Vec | +| set| 15 | Set | set | / | set| fury.Set | Set | +| map| 16 | Map | dict | Map | unordered_map | map | HashMap | +| duration | 17 | Duration| timedelta| Number | duration | Duration | Duration | +| timestamp | 18 | Instant | datetime | Number | std::chrono::nanoseconds | Time | DateTime | +| decimal| 19 | BigDecimal | Decimal | bigint | / | /| / | +| array | 20 | array | np.ndarray | / | / | array/slice | Vec | +| bool_array | 21 | bool[] | ndarray(np.bool_)| / | `bool[x]` | `[x]bool/[]~`| `Vec` | +| int16_array| 22 | short[] | ndarray(int16) | / | `int16_t[x]/vector<~>` | `[x]int16/[]~` | `Vec` | +| int32_array| 23 | int[] | ndarray(int32) | / | `int32_t[x]/vector<~>` | `[x]int32/[]~` | `Vec` | +| int64_array| 24 | long[] | ndarray(int64) | / | `int64_t[x]/vector<~>` | `[x]int64/[]~` | `Vec` | +| float16_array | 25 | float[] | ndarray(float16) | / | `fury::float16_t[x]/vector<~>` | `[x]float16/[]~` |
Re: [PR] chore(java): Reuse unsafePutPositiveVarInt in unsafeWritePositiveVarInt [incubator-fury]
LiangliangSui commented on code in PR #1434:
URL: https://github.com/apache/incubator-fury/pull/1434#discussion_r1554626407
##
java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java:
##
@@ -1233,49 +1233,9 @@ public int readVarInt() {
* to avoid using two memory operations.
*/
public int unsafeWritePositiveVarInt(int v) {
-// The encoding algorithm are based on kryo UnsafeMemoryOutput.writeVarInt
-// varint are written using little endian byte order.
-// This version should have better performance since it remove an index
update.
-long value = v;
-final int writerIndex = this.writerIndex;
-long varInt = (value & 0x7F);
-value >>>= 7;
-if (value == 0) {
- UNSAFE.putByte(heapMemory, address + writerIndex, (byte) varInt);
- this.writerIndex = writerIndex + 1;
- return 1;
-}
-// bit 8 `set` indicates have next data bytes.
-varInt |= 0x80;
-varInt |= ((value & 0x7F) << 8);
-value >>>= 7;
-if (value == 0) {
- unsafePutInt(writerIndex, (int) varInt);
- this.writerIndex = writerIndex + 2;
- return 2;
-}
-varInt |= (0x80 << 8);
-varInt |= ((value & 0x7F) << 16);
-value >>>= 7;
-if (value == 0) {
- unsafePutInt(writerIndex, (int) varInt);
- this.writerIndex = writerIndex + 3;
- return 3;
-}
-varInt |= (0x80 << 16);
-varInt |= ((value & 0x7F) << 24);
-value >>>= 7;
-if (value == 0) {
- unsafePutInt(writerIndex, (int) varInt);
- this.writerIndex = writerIndex + 4;
- return 4;
-}
-varInt |= (0x80L << 24);
-varInt |= ((value & 0x7F) << 32);
-varInt &= 0xFL;
-unsafePutLong(writerIndex, varInt);
-this.writerIndex = writerIndex + 5;
-return 5;
+int varintBytes = unsafePutPositiveVarInt(writerIndex, v);
Review Comment:
After passing the benchmark, it can still be inlined and there is no
performance degradation.
The difference between the two is 155266, which should be within the error.
In addition, I saw in the log that `unsafePutPositiveVarInt` can be inlined
in the benchmark process by using `-XX:+PrintCompilation
-XX:+UnlockDiagnosticVMOptions -XX:+PrintInlinin`
```
@ 152 java.lang.Integer::intValue (5 bytes) accessor
@ 155 org.apache.fury.memory.MemoryBuffer::writeVarInt (17 bytes) inline
(hot)
@ 8 org.apache.fury.memory.MemoryBuffer::ensure (37 bytes) inline (hot)
@ 13 org.apache.fury.memory.MemoryBuffer::unsafeWriteVarInt (15 bytes)
inline (hot)
@ 11 org.apache.fury.memory.MemoryBuffer::unsafeWritePositiveVarInt
(22 bytes) inline (hot)
@ 6 org.apache.fury.memory.MemoryBuffer::unsafePutPositiveVarInt
(208 bytes) inline (hot)
@ 39 sun.misc.Unsafe::putByte (11 bytes) force inline by
annotation
@ 7 jdk.internal.misc.Unsafe::putByte (0 bytes) (intrinsic)
@ 81 org.apache.fury.memory.MemoryBuffer::unsafePutInt (45 bytes)
inline (hot)
@ 23 sun.misc.Unsafe::putInt (11 bytes) force inline by
annotation
@ 7 jdk.internal.misc.Unsafe::putInt (0 bytes) (intrinsic)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] chore(java): Reuse unsafePutPositiveVarInt in unsafeWritePositiveVarInt [incubator-fury]
LiangliangSui commented on code in PR #1434:
URL: https://github.com/apache/incubator-fury/pull/1434#discussion_r1554626407
##
java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java:
##
@@ -1233,49 +1233,9 @@ public int readVarInt() {
* to avoid using two memory operations.
*/
public int unsafeWritePositiveVarInt(int v) {
-// The encoding algorithm are based on kryo UnsafeMemoryOutput.writeVarInt
-// varint are written using little endian byte order.
-// This version should have better performance since it remove an index
update.
-long value = v;
-final int writerIndex = this.writerIndex;
-long varInt = (value & 0x7F);
-value >>>= 7;
-if (value == 0) {
- UNSAFE.putByte(heapMemory, address + writerIndex, (byte) varInt);
- this.writerIndex = writerIndex + 1;
- return 1;
-}
-// bit 8 `set` indicates have next data bytes.
-varInt |= 0x80;
-varInt |= ((value & 0x7F) << 8);
-value >>>= 7;
-if (value == 0) {
- unsafePutInt(writerIndex, (int) varInt);
- this.writerIndex = writerIndex + 2;
- return 2;
-}
-varInt |= (0x80 << 8);
-varInt |= ((value & 0x7F) << 16);
-value >>>= 7;
-if (value == 0) {
- unsafePutInt(writerIndex, (int) varInt);
- this.writerIndex = writerIndex + 3;
- return 3;
-}
-varInt |= (0x80 << 16);
-varInt |= ((value & 0x7F) << 24);
-value >>>= 7;
-if (value == 0) {
- unsafePutInt(writerIndex, (int) varInt);
- this.writerIndex = writerIndex + 4;
- return 4;
-}
-varInt |= (0x80L << 24);
-varInt |= ((value & 0x7F) << 32);
-varInt &= 0xFL;
-unsafePutLong(writerIndex, varInt);
-this.writerIndex = writerIndex + 5;
-return 5;
+int varintBytes = unsafePutPositiveVarInt(writerIndex, v);
Review Comment:
After passing the benchmark, it can still be inlined and there is no
performance regression.
The difference between the two is 155266, which should be within the error.
In addition, I saw in the log that `unsafePutPositiveVarInt` can be inlined
in the benchmark process by using `-XX:+PrintCompilation
-XX:+UnlockDiagnosticVMOptions -XX:+PrintInlinin`
```
@ 152 java.lang.Integer::intValue (5 bytes) accessor
@ 155 org.apache.fury.memory.MemoryBuffer::writeVarInt (17 bytes) inline
(hot)
@ 8 org.apache.fury.memory.MemoryBuffer::ensure (37 bytes) inline (hot)
@ 13 org.apache.fury.memory.MemoryBuffer::unsafeWriteVarInt (15 bytes)
inline (hot)
@ 11 org.apache.fury.memory.MemoryBuffer::unsafeWritePositiveVarInt
(22 bytes) inline (hot)
@ 6 org.apache.fury.memory.MemoryBuffer::unsafePutPositiveVarInt
(208 bytes) inline (hot)
@ 39 sun.misc.Unsafe::putByte (11 bytes) force inline by
annotation
@ 7 jdk.internal.misc.Unsafe::putByte (0 bytes) (intrinsic)
@ 81 org.apache.fury.memory.MemoryBuffer::unsafePutInt (45 bytes)
inline (hot)
@ 23 sun.misc.Unsafe::putInt (11 bytes) force inline by
annotation
@ 7 jdk.internal.misc.Unsafe::putInt (0 bytes) (intrinsic)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(java): optimize read float/double for jvm jit inline [incubator-fury]
LiangliangSui commented on PR #1472: URL: https://github.com/apache/incubator-fury/pull/1472#issuecomment-204294 Hi @chaokunyang , I have a suggestion. For this kind of optimization PR, we should force the submitter to provide a benchmark data in the PR, so that we can ensure that the optimization is beneficial to the codebase. Then for the PR of the optimization category, we should force the PR title to indicate that it is an optimization PR (standardizing the Fury community). We can add a new category `opt` in [pr-lint.yml](https://github.com/apache/incubator-fury/blob/main/.github/workflows/pr-lint.yml), and I will submit a PR to do this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(java): optimize read float/double for jvm jit inline [incubator-fury]
LiangliangSui commented on code in PR #1472:
URL: https://github.com/apache/incubator-fury/pull/1472#discussion_r1554623191
##
java/fury-core/src/test/java/org/apache/fury/serializer/PrimitiveSerializersTest.java:
##
@@ -54,4 +59,62 @@ public void testUint16Serializer() {
assertThrows(IllegalArgumentException.class, () ->
serializer.xwrite(buffer, -1));
assertThrows(IllegalArgumentException.class, () ->
serializer.xwrite(buffer, 65536));
}
+
+ @Data
+ @AllArgsConstructor
+ public static class PrimitiveStruct {
+byte byte1;
+byte byte2;
+char char1;
+char char2;
+short short1;
+short short2;
+int int1;
+int int2;
+long long1;
+long long2;
+float float1;
+float float2;
+double double1;
+double double2;
+ }
+
+ @Test(dataProvider = "compressNumberAndCodeGen")
+ public void testPrimitiveStruct(boolean compressNumber, boolean codegen) {
+PrimitiveStruct struct =
+new PrimitiveStruct(
+Byte.MIN_VALUE,
+Byte.MIN_VALUE,
+Character.MIN_VALUE,
+Character.MIN_VALUE,
+Short.MIN_VALUE,
+Short.MIN_VALUE,
+Integer.MIN_VALUE,
+Integer.MIN_VALUE,
+Long.MIN_VALUE,
+Long.MIN_VALUE,
+Float.MIN_VALUE,
+Float.MIN_VALUE,
+Double.MIN_VALUE,
+Double.MIN_VALUE);
+if (compressNumber) {
+ FuryBuilder builder =
Review Comment:
Nit: We can move the common part of FuryBuilder before the if branch, so
that if and else can share this part
##
java/fury-core/src/test/java/org/apache/fury/serializer/PrimitiveSerializersTest.java:
##
@@ -54,4 +59,62 @@ public void testUint16Serializer() {
assertThrows(IllegalArgumentException.class, () ->
serializer.xwrite(buffer, -1));
assertThrows(IllegalArgumentException.class, () ->
serializer.xwrite(buffer, 65536));
}
+
+ @Data
+ @AllArgsConstructor
+ public static class PrimitiveStruct {
Review Comment:
Should be private.
```java
private static class PrimitiveStruct
```
##
java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java:
##
@@ -2318,6 +2205,37 @@ public float readFloat() {
}
}
+ // Reduce method body for better inline in the caller.
+ @CodegenInvoke
+ public float readFloatOnLE() {
+int readerIdx = readerIndex;
+// use subtract to avoid overflow
+int remaining = size - readerIdx;
+if (remaining < 4) {
+ throw new IndexOutOfBoundsException(
Review Comment:
Nit: we can reuse
[`throwIndexOutOfBoundsException`](https://github.com/apache/incubator-fury/blob/main/java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java#L1853-L1857).
`throwIndexOutOfBoundsException` is not executed frequently and should have no
impact on inlining.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] chore(java): add comments for loop unrolling. [incubator-fury]
LiangliangSui commented on PR #1450: URL: https://github.com/apache/incubator-fury/pull/1450#issuecomment-2041108710 > @LiangliangSui There are some conflict between your branch and main branch, could you resolve the conflict? Updated, ptal. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" is working again!
The GitHub Actions job "Fury CI" on incubator-fury.git has succeeded. Run started by GitHub user LiangliangSui (triggered by LiangliangSui). Head commit for run: 63750ec67139b4a6a48af455ae144ae96a0d0516 / LiangliangSui Rerun ci. Signed-off-by: LiangliangSui Report URL: https://github.com/apache/incubator-fury/actions/runs/8581874158 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user LiangliangSui (triggered by LiangliangSui). Head commit for run: 3a31459121599362de7e480fd351d86b52cfe05a / LiangliangSui chore(java): Add comments for loop unrolling Signed-off-by: LiangliangSui Report URL: https://github.com/apache/incubator-fury/actions/runs/8581822392 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(java): native streaming mode deserialization [incubator-fury]
chaokunyang commented on PR #1451: URL: https://github.com/apache/incubator-fury/pull/1451#issuecomment-2041104708 Method spilit threshold: - 15 ``` Benchmark (bufferType) (objectType) (references) Mode CntScoreError Units UserTypeDeserializeSuite.fury_deserialize arraySTRUCT false thrpt 10 4333464.647 ± 247025.147 ops/s UserTypeDeserializeSuite.fury_deserialize arraySTRUCT true thrpt 10 3643508.209 ± 470967.165 ops/s UserTypeDeserializeSuite.fury_deserialize directBufferSTRUCT false thrpt 10 3907185.065 ± 267600.435 ops/s UserTypeDeserializeSuite.fury_deserialize directBufferSTRUCT true thrpt 10 3961404.504 ± 415918.311 ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 4932c70708687df07c1c0ff0ea98fa0e639c9f57 / chaokunyang import platform Report URL: https://github.com/apache/incubator-fury/actions/runs/8581614856 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: e30effa9c5b549b03a2a0e59a33700d7ce3ba3fa / chaokunyang import platform Report URL: https://github.com/apache/incubator-fury/actions/runs/8581611969 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 98a028a55e7053c1200b6b9720281ab593a8b03c / chaokunyang fix merge conflict Report URL: https://github.com/apache/incubator-fury/actions/runs/8581383218 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 3c0fb9dd2a1df4cb5752e7db48d95d9c59c28f1f / chaokunyang Merge remote-tracking branch 'ant/main' into streaming_deserialization Report URL: https://github.com/apache/incubator-fury/actions/runs/8581255200 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury) branch main updated: feat(java): optimize read float/double for jvm jit inline (#1472)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury.git
The following commit(s) were added to refs/heads/main by this push:
new 1d302ef1 feat(java): optimize read float/double for jvm jit inline
(#1472)
1d302ef1 is described below
commit 1d302ef1112de7c684af65fd9b0589715129f91c
Author: Shawn Yang
AuthorDate: Sat Apr 6 20:43:06 2024 +0800
feat(java): optimize read float/double for jvm jit inline (#1472)
- optimize read float jvm jit inline by separate little/big endian
methods
- optimize read double jvm jit inline by separate little/big endian
methods
- generate unsafe get float code online
- generate unsafe get double code online
---
.../java/org/apache/fury/builder/CodecBuilder.java | 36 +++--
.../java/org/apache/fury/memory/MemoryBuffer.java | 175 -
.../test/java/org/apache/fury/FuryTestBase.java| 5 +
.../org/apache/fury/memory/MemoryBufferTest.java | 6 +-
.../fury/serializer/PrimitiveSerializersTest.java | 65 +++-
5 files changed, 158 insertions(+), 129 deletions(-)
diff --git
a/java/fury-core/src/main/java/org/apache/fury/builder/CodecBuilder.java
b/java/fury-core/src/main/java/org/apache/fury/builder/CodecBuilder.java
index 746a698d..9c69757e 100644
--- a/java/fury-core/src/main/java/org/apache/fury/builder/CodecBuilder.java
+++ b/java/fury-core/src/main/java/org/apache/fury/builder/CodecBuilder.java
@@ -572,18 +572,13 @@ public abstract class CodecBuilder {
return new StaticInvoke(MemoryBuffer.class, "unsafePutDouble", base, pos,
value);
}
- /**
- * Build unsafeGet operation.
- *
- * @see MemoryBuffer#unsafeGet(Object, long)
- */
+ /** Build unsafeGet operation. */
protected Expression unsafeGet(Expression base, Expression pos) {
-return new StaticInvoke(MemoryBuffer.class, "unsafeGet",
PRIMITIVE_BYTE_TYPE, base, pos);
+return new StaticInvoke(Platform.class, "getByte", PRIMITIVE_BYTE_TYPE,
base, pos);
}
protected Expression unsafeGetBoolean(Expression base, Expression pos) {
-return new StaticInvoke(
-MemoryBuffer.class, "unsafeGetBoolean", PRIMITIVE_BOOLEAN_TYPE, base,
pos);
+return new StaticInvoke(Platform.class, "getBoolean",
PRIMITIVE_BOOLEAN_TYPE, base, pos);
}
protected Expression unsafeGetChar(Expression base, Expression pos) {
@@ -614,18 +609,25 @@ public abstract class CodecBuilder {
protected Expression unsafeGetLong(Expression base, Expression pos) {
StaticInvoke expr = new StaticInvoke(Platform.class, "getLong",
PRIMITIVE_LONG_TYPE, base, pos);
if (!Platform.IS_LITTLE_ENDIAN) {
- expr = new StaticInvoke(Long.class, "reverseBytes", PRIMITIVE_INT_TYPE,
expr.inline());
+ expr = new StaticInvoke(Long.class, "reverseBytes", PRIMITIVE_LONG_TYPE,
expr.inline());
}
return expr;
}
protected Expression unsafeGetFloat(Expression base, Expression pos) {
-return new StaticInvoke(MemoryBuffer.class, "unsafeGetFloat",
PRIMITIVE_FLOAT_TYPE, base, pos);
+StaticInvoke expr = new StaticInvoke(Platform.class, "getInt",
PRIMITIVE_INT_TYPE, base, pos);
+if (!Platform.IS_LITTLE_ENDIAN) {
+ expr = new StaticInvoke(Integer.class, "reverseBytes",
PRIMITIVE_INT_TYPE, expr.inline());
+}
+return new StaticInvoke(Float.class, "intBitsToFloat",
PRIMITIVE_FLOAT_TYPE, expr.inline());
}
protected Expression unsafeGetDouble(Expression base, Expression pos) {
-return new StaticInvoke(
-MemoryBuffer.class, "unsafeGetDouble", PRIMITIVE_DOUBLE_TYPE, base,
pos);
+StaticInvoke expr = new StaticInvoke(Platform.class, "getLong",
PRIMITIVE_LONG_TYPE, base, pos);
+if (!Platform.IS_LITTLE_ENDIAN) {
+ expr = new StaticInvoke(Long.class, "reverseBytes", PRIMITIVE_LONG_TYPE,
expr.inline());
+}
+return new StaticInvoke(Double.class, "longBitsToDouble",
PRIMITIVE_DOUBLE_TYPE, expr.inline());
}
protected Expression readChar(Expression buffer) {
@@ -655,4 +657,14 @@ public abstract class CodecBuilder {
public static String readLongFunc() {
return Platform.IS_LITTLE_ENDIAN ? "readLongOnLE" : "readLongOnBE";
}
+
+ protected Expression readFloat(Expression buffer) {
+String func = Platform.IS_LITTLE_ENDIAN ? "readFloatOnLE" :
"readFloatOnBE";
+return new Invoke(buffer, func, PRIMITIVE_FLOAT_TYPE);
+ }
+
+ protected Expression readDouble(Expression buffer) {
+String func = Platform.IS_LITTLE_ENDIAN ? "readDoubleOnLE" :
"readDoubleOnBE";
+return new Invoke(buffer, func, PRIMITIVE_DOUBLE_TYPE);
+ }
}
diff --git
a/java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java
b/java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java
index 155b8df6..025ceb28 100644
--- a/java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java
+++
Re: [PR] feat(java): optimize read float/double for jvm jit inline [incubator-fury]
chaokunyang merged PR #1472: URL: https://github.com/apache/incubator-fury/pull/1472 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" is working again!
The GitHub Actions job "Fury CI" on incubator-fury.git has succeeded. Run started by GitHub user theweipeng (triggered by theweipeng). Head commit for run: 959796233fdd25a3bf47bc2dd75c6d60e5d7d08e / Shawn Yang feat(spec): add xlang type mapping (#1468) Report URL: https://github.com/apache/incubator-fury/actions/runs/8581145099 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" is working again!
The GitHub Actions job "Fury CI" on incubator-fury.git has succeeded. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: c42cdd4592802183e79342d7fbf8c85c115f327e / chaokunyang fix unsafeGetDouble Report URL: https://github.com/apache/incubator-fury/actions/runs/8581126580 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 93940b91be52e9048a5d57ebb1715f78d3659b22 / chaokunyang fix unsafeGetDouble Report URL: https://github.com/apache/incubator-fury/actions/runs/8581114693 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury-site) branch deploy updated: deploy: 4deb175d6ce38b37bd3a84bbca9f36e4285af8d0
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch deploy
in repository https://gitbox.apache.org/repos/asf/incubator-fury-site.git
The following commit(s) were added to refs/heads/deploy by this push:
new 0ca9a20 deploy: 4deb175d6ce38b37bd3a84bbca9f36e4285af8d0
0ca9a20 is described below
commit 0ca9a206b3b5eec7b64f61e2dcea357782d28919
Author: chaokunyang
AuthorDate: Sat Apr 6 12:32:12 2024 +
deploy: 4deb175d6ce38b37bd3a84bbca9f36e4285af8d0
---
404.html | 4 +--
assets/js/52d4d5f6.32778d20.js | 1 -
assets/js/52d4d5f6.a504027f.js | 1 +
assets/js/72b486c3.1de42ed6.js | 1 +
assets/js/72b486c3.2f73747b.js | 1 -
assets/js/7b95f1e0.23f8195a.js | 1 -
assets/js/7b95f1e0.47d51ef6.js | 1 +
.../{935f2afb.87230918.js => 935f2afb.99626f54.js} | 2 +-
assets/js/adfc56cb.319bb751.js | 1 -
assets/js/adfc56cb.cd381916.js | 1 +
assets/js/e20a7859.27395eae.js | 1 +
assets/js/e20a7859.a8768b5c.js | 1 -
assets/js/{main.5863021d.js => main.2a8cd0f2.js} | 4 +--
...js.LICENSE.txt => main.2a8cd0f2.js.LICENSE.txt} | 0
...e~main.e1125dd7.js => runtime~main.c8fe22c9.js} | 2 +-
blog/archive/index.html| 4 +--
blog/fury_0_1_0_release/index.html | 4 +--
blog/fury_0_1_1_release/index.html | 4 +--
blog/fury_0_1_2_release/index.html | 4 +--
blog/fury_0_2_0_release/index.html | 4 +--
blog/fury_0_2_1_release/index.html | 4 +--
blog/fury_0_3_0_release/index.html | 4 +--
blog/fury_0_3_1_release/index.html | 4 +--
blog/fury_0_4_0_release/index.html | 4 +--
blog/fury_0_4_1_release/index.html | 4 +--
.../index.html | 4 +--
blog/index.html| 4 +--
blog/tags/fury/index.html | 4 +--
blog/tags/index.html | 4 +--
docs/guide/development/index.html | 8 ++---
docs/guide/graalvm_guide/index.html| 8 ++---
docs/guide/java_object_graph_guide/index.html | 6 ++--
docs/guide/row_format_guide/index.html | 6 ++--
docs/guide/scala_guide/index.html | 8 ++---
docs/guide/xlang_object_graph_guide/index.html | 6 ++--
docs/guide/xlang_type_mapping/index.html | 34 +++---
docs/introduction/benchmark/index.html | 4 +--
docs/introduction/features/index.html | 4 +--
docs/introduction/index.html | 4 +--
.../fury_java_serialization_spec/index.html| 4 +--
docs/specification/fury_row_format_spec/index.html | 4 +--
.../fury_xlang_serialization_spec/index.html | 6 ++--
docs/start/install/index.html | 4 +--
docs/start/usage/index.html| 4 +--
index.html | 4 +--
lunr-index-1712102951567.json | 1 -
lunr-index-1712406700847.json | 1 +
lunr-index.json| 2 +-
markdown-page/index.html | 4 +--
search-doc-1712102951567.json | 1 -
search-doc-1712406700847.json | 1 +
search-doc.json| 2 +-
52 files changed, 114 insertions(+), 90 deletions(-)
diff --git a/404.html b/404.html
index 8da09af..1c248b8 100644
--- a/404.html
+++ b/404.html
@@ -5,8 +5,8 @@
Page Not Found | Apache Fury (incubating)https://fury.apache.org/404.html;>
-
-
+
+
!function(){function
t(t){document.documentElement.setAttribute("data-theme",t)}var
e=function(){try{return new
URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}}()||function(){try{return
localStorage.getItem("theme")}catch(t){}}();t(null!==e?e:"light")}(),function(){try{const
c=new URLSearchParams(window.location.search).entries();for(var[t,e]of
c)if(t.startsWith("docusaurus-data-")){var
a=t.replace("docusaurus-data-","data-");document.documentElement.se [...]
diff --git a/assets/js/52d4d5f6.32778d20.js b/assets/js/52d4d5f6.32778d20.js
deleted file mode 100644
index 9c7c7cd..000
--- a/assets/js/52d4d5f6.32778d20.js
+++ /dev/null
@@ -1 +0,0 @@
-"use
strict";(self.webpackChunkfury_site=self.webpackChunkfury_site||[]).push([[4404],{1888:(e,r,n)=>{n.r(r),n.d(r,{assets:()=>o,contentTitle:()=>t,default:()=>u,frontMatter:()=>s,metadata:()=>l,toc:()=>c});var
a=n(5893),i=n(1151);const s={title:"GraalVM
Guide",sidebar_position:6,id:"graalvm_guide"},t="GraalVM Native
Re: [PR] feat(spec): add xlang type mapping [incubator-fury]
theweipeng commented on code in PR #1468: URL: https://github.com/apache/incubator-fury/pull/1468#discussion_r1554556705 ## docs/guide/xlang_type_mapping.md: ## @@ -4,4 +4,75 @@ sidebar_position: 3 id: xlang_type_mapping --- -Coming soon. +Note: + +- For type definition, see [Type Systems in Spec](../specification/xlang_serialization_spec.md#type-systems) +- `int16_t[x]/vector<~>` indicates `int16_t[x]/vector` +- The cross-language serialization is not stable, do not use it in your production environment. + +# Type Mapping + +| Type | Type ID | Java| Python | Javascript | C++| Golang | Rust | +||-|-|--|-||--|--| +| bool | 1 | bool/Boolean| bool | Boolean | bool | bool | bool | +| int8 | 2 | byte/Byte | int/pyfury.Int8 | Type.int8() | int8_t | int8 | i8 | +| int16 | 3 | short/Short | int/pyfury.Int16 | Type.int16()| int16_t| int16| i6 | +| int32 | 4 | int/Integer | int/pyfury.Int32 | Type.int32()| int32_t| int32| i32 | +| var_int32 | 5 | int/Integer | int/pyfury.VarInt32 | Type.varint32() | fury::varint32_t | fury.varint32| fury::varint32 | +| int64 | 6 | long/Long | int/pyfury.Int64 | Type.int64()| int64_t| int64| i64 | +| var_int64 | 7 | long/Long | int/pyfury.VarInt64 | Type.varint64() | fury::varint64_t | fury.varint64| fury::varint64 | +| sli_int64 | 8 | long/Long | int/pyfury.SliInt64 | Type.sliint64() | fury::sliint64_t | fury.sliint64| fury::sliint64 | +| float16| 9 | float/Float | float/pyfury.Float16 | Type.float16() | fury::float16_t| fury.float16 | fury::f16 | +| float32| 10 | float/Float | float/pyfury.Float32 | Type.float32() | float | float32 | f32 | +| float64| 11 | double/Double | float/pyfury.Float64 | Type.float64() | double | float64 | f64 | +| string | 12 | String | str | String | string | string | String/str | +| enum | 13 | Enum subclasses | enum subclasses | / | enum | /| enum | +| list | 14 | List/Collection | list/tuple | array | vector | slice| Vec | +| set| 15 | Set | set | / | set| fury.Set | Set | +| map| 16 | Map | dict | Map | unordered_map | map | HashMap | +| duration | 17 | Duration| timedelta| Number | duration | Duration | Duration | +| timestamp | 18 | Instant | datetime | Number | std::chrono::nanoseconds | Time | DateTime | +| decimal| 19 | BigDecimal | Decimal | bigint | / | /| / | +| array | 20 | array | np.ndarray | / | / | array/slice | Vec | +| bool_array | 21 | bool[] | ndarray(np.bool_)| / | `bool[x]` | `[x]bool/[]~`| `Vec` | +| int16_array| 22 | short[] | ndarray(int16) | / | `int16_t[x]/vector<~>` | `[x]int16/[]~` | `Vec` | +| int32_array| 23 | int[] | ndarray(int32) | / | `int32_t[x]/vector<~>` | `[x]int32/[]~` | `Vec` | +| int64_array| 24 | long[] | ndarray(int64) | / | `int64_t[x]/vector<~>` | `[x]int64/[]~` | `Vec` | +| float16_array | 25 | float[] | ndarray(float16) | / | `fury::float16_t[x]/vector<~>` | `[x]float16/[]~` |
(incubator-fury-site) branch main updated (d6f914d -> 4deb175)
This is an automated email from the ASF dual-hosted git repository. chaokunyang pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/incubator-fury-site.git from d6f914d add fury serialization specification tab (#108) new 45472f0 synced local 'docs/guide/' with remote 'docs/guide/' new 4deb175 synced local 'docs/specification/' with remote 'docs/specification/' The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/guide/DEVELOPMENT.md | 2 +- docs/guide/xlang_type_mapping.md | 73 +- docs/specification/xlang_serialization_spec.md | 2 - 3 files changed, 73 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: cd2390b3257d5e4040b6593662d098e2f44fb20b / chaokunyang fix merge conflict Report URL: https://github.com/apache/incubator-fury/actions/runs/8581092524 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury-site) 01/02: synced local 'docs/guide/' with remote 'docs/guide/'
This is an automated email from the ASF dual-hosted git repository. chaokunyang pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-fury-site.git commit 45472f09d1474dba9615705be6b1cc480e0370dd Author: chaokunyang AuthorDate: Sat Apr 6 12:31:02 2024 + synced local 'docs/guide/' with remote 'docs/guide/' --- docs/guide/DEVELOPMENT.md| 2 +- docs/guide/xlang_type_mapping.md | 73 +++- 2 files changed, 73 insertions(+), 2 deletions(-) diff --git a/docs/guide/DEVELOPMENT.md b/docs/guide/DEVELOPMENT.md index 5d71166..bfab541 100644 --- a/docs/guide/DEVELOPMENT.md +++ b/docs/guide/DEVELOPMENT.md @@ -1,6 +1,6 @@ --- title: Development -sidebar_position: 6 +sidebar_position: 7 id: development --- diff --git a/docs/guide/xlang_type_mapping.md b/docs/guide/xlang_type_mapping.md index f529d4e..bec2295 100644 --- a/docs/guide/xlang_type_mapping.md +++ b/docs/guide/xlang_type_mapping.md @@ -4,4 +4,75 @@ sidebar_position: 3 id: xlang_type_mapping --- -Coming soon. +Note: + +- For type definition, see [Type Systems in Spec](../specification/xlang_serialization_spec.md#type-systems) +- `int16_t[x]/vector<~>` indicates `int16_t[x]/vector` +- The cross-language serialization is not stable, do not use it in your production environment. + +# Type Mapping + +| Type | Type ID | Java| Python | Javascript | C++| Golang | Rust | +||-|-|--|-||--|--| +| bool | 1 | bool/Boolean| bool | Boolean | bool | bool | bool | +| int8 | 2 | byte/Byte | int/pyfury.Int8 | Type.int8() | int8_t | int8 | i8 | +| int16 | 3 | short/Short | int/pyfury.Int16 | Type.int16()| int16_t| int16| i6 | +| int32 | 4 | int/Integer | int/pyfury.Int32 | Type.int32()| int32_t| int32| i32 | +| var_int32 | 5 | int/Integer | int/pyfury.VarInt32 | Type.varint32() | fury::varint32_t | fury.varint32| fury::varint32 | +| int64 | 6 | long/Long | int/pyfury.Int64 | Type.int64()| int64_t| int64| i64 | +| var_int64 | 7 | long/Long | int/pyfury.VarInt64 | Type.varint64() | fury::varint64_t | fury.varint64| fury::varint64 | +| sli_int64 | 8 | long/Long | int/pyfury.SliInt64 | Type.sliint64() | fury::sliint64_t | fury.sliint64| fury::sliint64 | +| float16| 9 | float/Float | float/pyfury.Float16 | Type.float16() | fury::float16_t| fury.float16 | fury::f16 | +| float32| 10 | float/Float | float/pyfury.Float32 | Type.float32() | float | float32 | f32 | +| float64| 11 | double/Double | float/pyfury.Float64 | Type.float64() | double | float64 | f64 | +| string | 12 | String | str | String | string | string | String/str | +| enum | 13 | Enum subclasses | enum subclasses | / | enum | /| enum | +| list | 14 | List/Collection | list/tuple | array | vector | slice| Vec | +| set| 15 | Set | set | / | set| fury.Set | Set | +| map| 16 | Map | dict | Map | unordered_map | map | HashMap | +| duration | 17 | Duration| timedelta| Number | duration | Duration | Duration | +| timestamp | 18 | Instant | datetime | Number | std::chrono::nanoseconds | Time | DateTime | +| decimal| 19 | BigDecimal | Decimal | bigint | / | /| / | +| array | 20 | array | np.ndarray | / | / | array/slice |
(incubator-fury-site) 02/02: synced local 'docs/specification/' with remote 'docs/specification/'
This is an automated email from the ASF dual-hosted git repository. chaokunyang pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-fury-site.git commit 4deb175d6ce38b37bd3a84bbca9f36e4285af8d0 Author: chaokunyang AuthorDate: Sat Apr 6 12:31:02 2024 + synced local 'docs/specification/' with remote 'docs/specification/' --- docs/specification/xlang_serialization_spec.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/specification/xlang_serialization_spec.md b/docs/specification/xlang_serialization_spec.md index e01ec92..7838f78 100644 --- a/docs/specification/xlang_serialization_spec.md +++ b/docs/specification/xlang_serialization_spec.md @@ -29,11 +29,9 @@ also introduce more complexities compared to static serialization frameworks. So - int16: a 16-bit signed integer. - int32: a 32-bit signed integer. - var_int32: a 32-bit signed integer which use fury var_int32 encoding. -- fixed_int32: a 32-bit signed integer which use two's complement encoding. - int64: a 64-bit signed integer. - var_int64: a 64-bit signed integer which use fury PVL encoding. - sli_int64: a 64-bit signed integer which use fury SLI encoding. -- fixed_int64: a 64-bit signed integer which use two's complement encoding. - float16: a 16-bit floating point number. - float32: a 32-bit floating point number. - float64: a 64-bit floating point number including NaN and Infinity. - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury) branch main updated: feat(spec): add xlang type mapping (#1468)
This is an automated email from the ASF dual-hosted git repository. wangweipeng pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-fury.git The following commit(s) were added to refs/heads/main by this push: new 95979623 feat(spec): add xlang type mapping (#1468) 95979623 is described below commit 959796233fdd25a3bf47bc2dd75c6d60e5d7d08e Author: Shawn Yang AuthorDate: Sat Apr 6 20:30:50 2024 +0800 feat(spec): add xlang type mapping (#1468) --- docs/guide/DEVELOPMENT.md | 2 +- docs/guide/xlang_type_mapping.md | 73 +- docs/specification/xlang_serialization_spec.md | 2 - 3 files changed, 73 insertions(+), 4 deletions(-) diff --git a/docs/guide/DEVELOPMENT.md b/docs/guide/DEVELOPMENT.md index 5d71166a..bfab5417 100644 --- a/docs/guide/DEVELOPMENT.md +++ b/docs/guide/DEVELOPMENT.md @@ -1,6 +1,6 @@ --- title: Development -sidebar_position: 6 +sidebar_position: 7 id: development --- diff --git a/docs/guide/xlang_type_mapping.md b/docs/guide/xlang_type_mapping.md index f529d4ef..bec22957 100644 --- a/docs/guide/xlang_type_mapping.md +++ b/docs/guide/xlang_type_mapping.md @@ -4,4 +4,75 @@ sidebar_position: 3 id: xlang_type_mapping --- -Coming soon. +Note: + +- For type definition, see [Type Systems in Spec](../specification/xlang_serialization_spec.md#type-systems) +- `int16_t[x]/vector<~>` indicates `int16_t[x]/vector` +- The cross-language serialization is not stable, do not use it in your production environment. + +# Type Mapping + +| Type | Type ID | Java| Python | Javascript | C++| Golang | Rust | +||-|-|--|-||--|--| +| bool | 1 | bool/Boolean| bool | Boolean | bool | bool | bool | +| int8 | 2 | byte/Byte | int/pyfury.Int8 | Type.int8() | int8_t | int8 | i8 | +| int16 | 3 | short/Short | int/pyfury.Int16 | Type.int16()| int16_t| int16| i6 | +| int32 | 4 | int/Integer | int/pyfury.Int32 | Type.int32()| int32_t| int32| i32 | +| var_int32 | 5 | int/Integer | int/pyfury.VarInt32 | Type.varint32() | fury::varint32_t | fury.varint32| fury::varint32 | +| int64 | 6 | long/Long | int/pyfury.Int64 | Type.int64()| int64_t| int64| i64 | +| var_int64 | 7 | long/Long | int/pyfury.VarInt64 | Type.varint64() | fury::varint64_t | fury.varint64| fury::varint64 | +| sli_int64 | 8 | long/Long | int/pyfury.SliInt64 | Type.sliint64() | fury::sliint64_t | fury.sliint64| fury::sliint64 | +| float16| 9 | float/Float | float/pyfury.Float16 | Type.float16() | fury::float16_t| fury.float16 | fury::f16 | +| float32| 10 | float/Float | float/pyfury.Float32 | Type.float32() | float | float32 | f32 | +| float64| 11 | double/Double | float/pyfury.Float64 | Type.float64() | double | float64 | f64 | +| string | 12 | String | str | String | string | string | String/str | +| enum | 13 | Enum subclasses | enum subclasses | / | enum | /| enum | +| list | 14 | List/Collection | list/tuple | array | vector | slice| Vec | +| set| 15 | Set | set | / | set| fury.Set | Set | +| map| 16 | Map | dict | Map | unordered_map | map | HashMap | +| duration | 17 | Duration| timedelta| Number | duration | Duration | Duration | +| timestamp | 18 | Instant | datetime | Number | std::chrono::nanoseconds | Time | DateTime | +| decimal| 19 | BigDecimal | Decimal | bigint | /
Re: [PR] feat(spec): add xlang type mapping [incubator-fury]
theweipeng merged PR #1468: URL: https://github.com/apache/incubator-fury/pull/1468 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[PR] feat(java): optimize read float/double for jvm jit inline [incubator-fury]
chaokunyang opened a new pull request, #1472: URL: https://github.com/apache/incubator-fury/pull/1472 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury) branch main updated: feat(java): optimize read char/short jvm jit inline (#1471)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury.git
The following commit(s) were added to refs/heads/main by this push:
new 4c60eb97 feat(java): optimize read char/short jvm jit inline (#1471)
4c60eb97 is described below
commit 4c60eb9780ec51ffe1aac3ee2da594c0762799e0
Author: Shawn Yang
AuthorDate: Sat Apr 6 19:53:53 2024 +0800
feat(java): optimize read char/short jvm jit inline (#1471)
- optimize read char jvm jit inline by separate little/big endian
methods
- optimize read short jvm jit inline by separate little/big endian
methods
- generate unsafe get short code online
- generate unsafe get char code online
---
.../fury/builder/BaseObjectCodecBuilder.java | 5 +-
.../java/org/apache/fury/builder/CodecBuilder.java | 23 -
.../java/org/apache/fury/memory/MemoryBuffer.java | 111 +++--
.../org/apache/fury/memory/MemoryBufferTest.java | 2 +-
4 files changed, 84 insertions(+), 57 deletions(-)
diff --git
a/java/fury-core/src/main/java/org/apache/fury/builder/BaseObjectCodecBuilder.java
b/java/fury-core/src/main/java/org/apache/fury/builder/BaseObjectCodecBuilder.java
index a27abbab..f92e076e 100644
---
a/java/fury-core/src/main/java/org/apache/fury/builder/BaseObjectCodecBuilder.java
+++
b/java/fury-core/src/main/java/org/apache/fury/builder/BaseObjectCodecBuilder.java
@@ -40,7 +40,6 @@ import static
org.apache.fury.type.TypeUtils.PRIMITIVE_BYTE_TYPE;
import static org.apache.fury.type.TypeUtils.PRIMITIVE_DOUBLE_TYPE;
import static org.apache.fury.type.TypeUtils.PRIMITIVE_FLOAT_TYPE;
import static org.apache.fury.type.TypeUtils.PRIMITIVE_INT_TYPE;
-import static org.apache.fury.type.TypeUtils.PRIMITIVE_SHORT_TYPE;
import static org.apache.fury.type.TypeUtils.PRIMITIVE_VOID_TYPE;
import static org.apache.fury.type.TypeUtils.SET_TYPE;
import static org.apache.fury.type.TypeUtils.getElementType;
@@ -1131,9 +1130,9 @@ public abstract class BaseObjectCodecBuilder extends
CodecBuilder {
} else if (cls == boolean.class || cls == Boolean.class) {
return new Invoke(buffer, "readBoolean", PRIMITIVE_BOOLEAN_TYPE);
} else if (cls == char.class || cls == Character.class) {
-return new Invoke(buffer, "readChar", TypeToken.of(char.class));
+return readChar(buffer);
} else if (cls == short.class || cls == Short.class) {
-return new Invoke(buffer, "readShort", PRIMITIVE_SHORT_TYPE);
+return readShort(buffer);
} else if (cls == int.class || cls == Integer.class) {
return fury.compressInt() ? readVarInt(buffer) : readInt(buffer);
} else if (cls == long.class || cls == Long.class) {
diff --git
a/java/fury-core/src/main/java/org/apache/fury/builder/CodecBuilder.java
b/java/fury-core/src/main/java/org/apache/fury/builder/CodecBuilder.java
index 121c6ecb..746a698d 100644
--- a/java/fury-core/src/main/java/org/apache/fury/builder/CodecBuilder.java
+++ b/java/fury-core/src/main/java/org/apache/fury/builder/CodecBuilder.java
@@ -587,11 +587,20 @@ public abstract class CodecBuilder {
}
protected Expression unsafeGetChar(Expression base, Expression pos) {
-return new StaticInvoke(MemoryBuffer.class, "unsafeGetChar",
PRIMITIVE_CHAR_TYPE, base, pos);
+StaticInvoke expr = new StaticInvoke(Platform.class, "getChar",
PRIMITIVE_CHAR_TYPE, base, pos);
+if (!Platform.IS_LITTLE_ENDIAN) {
+ expr = new StaticInvoke(Character.class, "reverseBytes",
PRIMITIVE_CHAR_TYPE, expr.inline());
+}
+return expr;
}
protected Expression unsafeGetShort(Expression base, Expression pos) {
-return new StaticInvoke(MemoryBuffer.class, "unsafeGetShort",
PRIMITIVE_SHORT_TYPE, base, pos);
+StaticInvoke expr =
+new StaticInvoke(Platform.class, "getShort", PRIMITIVE_SHORT_TYPE,
base, pos);
+if (!Platform.IS_LITTLE_ENDIAN) {
+ expr = new StaticInvoke(Short.class, "reverseBytes",
PRIMITIVE_SHORT_TYPE, expr.inline());
+}
+return expr;
}
protected Expression unsafeGetInt(Expression base, Expression pos) {
@@ -619,6 +628,16 @@ public abstract class CodecBuilder {
MemoryBuffer.class, "unsafeGetDouble", PRIMITIVE_DOUBLE_TYPE, base,
pos);
}
+ protected Expression readChar(Expression buffer) {
+String func = Platform.IS_LITTLE_ENDIAN ? "readCharOnLE" : "readCharOnBE";
+return new Invoke(buffer, func, PRIMITIVE_CHAR_TYPE);
+ }
+
+ protected Expression readShort(Expression buffer) {
+String func = Platform.IS_LITTLE_ENDIAN ? "readShortOnLE" :
"readShortOnBE";
+return new Invoke(buffer, func, PRIMITIVE_SHORT_TYPE);
+ }
+
protected Expression readInt(Expression buffer) {
String func = Platform.IS_LITTLE_ENDIAN ? "readIntOnLE" : "readIntOnBE";
return new Invoke(buffer, func, PRIMITIVE_INT_TYPE);
diff --git
Re: [PR] feat(java): optimize read char/short jvm jit inline [incubator-fury]
chaokunyang merged PR #1471: URL: https://github.com/apache/incubator-fury/pull/1471 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[PR] feat(java): optimize read char/short jvm jit inline [incubator-fury]
chaokunyang opened a new pull request, #1471: URL: https://github.com/apache/incubator-fury/pull/1471 - optimize read char jvm jit inline by separate little/big endian methods - optimize read short jvm jit inline by separate little/big endian methods - generate unsafe get short code online - generate unsafe get char code online -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" is working again!
The GitHub Actions job "Fury CI" on incubator-fury.git has succeeded. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: f6e9c732a0701c7f59444282ff33c234d8548e5d / Shawn Yang feat(java): reduce code size of read long to optimize jvm jit inline (#1470) This PR reduced code size of read long for jvm jit inline: - Reduced readVarLong code size by separating little/big endian - Reduced readLong code size by separating little/big endian - Reduced unsafeGetLong code size by separating little/big endian and generate online Report URL: https://github.com/apache/incubator-fury/actions/runs/8580814302 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 3801cfa6980fe65d163f37d3063fe41cf4eef77e / chaokunyang fix readSliLongLE/BE naming Report URL: https://github.com/apache/incubator-fury/actions/runs/8580776370 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(java): reduce code size of read long to optimize jvm jit inline [incubator-fury]
chaokunyang merged PR #1470: URL: https://github.com/apache/incubator-fury/pull/1470 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury) branch main updated: feat(java): reduce code size of read long to optimize jvm jit inline (#1470)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury.git
The following commit(s) were added to refs/heads/main by this push:
new f6e9c732 feat(java): reduce code size of read long to optimize jvm jit
inline (#1470)
f6e9c732 is described below
commit f6e9c732a0701c7f59444282ff33c234d8548e5d
Author: Shawn Yang
AuthorDate: Sat Apr 6 19:29:42 2024 +0800
feat(java): reduce code size of read long to optimize jvm jit inline (#1470)
This PR reduced code size of read long for jvm jit inline:
- Reduced readVarLong code size by separating little/big endian
- Reduced readLong code size by separating little/big endian
- Reduced unsafeGetLong code size by separating little/big endian and
generate online
---
.../java/org/apache/fury/builder/CodecBuilder.java | 14 +-
.../fury/builder/CompatibleCodecBuilder.java | 10 +-
.../java/org/apache/fury/memory/MemoryBuffer.java | 227 +
.../fury/serializer/PrimitiveSerializers.java | 7 +-
4 files changed, 213 insertions(+), 45 deletions(-)
diff --git
a/java/fury-core/src/main/java/org/apache/fury/builder/CodecBuilder.java
b/java/fury-core/src/main/java/org/apache/fury/builder/CodecBuilder.java
index 457d41bf..121c6ecb 100644
--- a/java/fury-core/src/main/java/org/apache/fury/builder/CodecBuilder.java
+++ b/java/fury-core/src/main/java/org/apache/fury/builder/CodecBuilder.java
@@ -603,7 +603,11 @@ public abstract class CodecBuilder {
}
protected Expression unsafeGetLong(Expression base, Expression pos) {
-return new StaticInvoke(MemoryBuffer.class, "unsafeGetLong",
PRIMITIVE_LONG_TYPE, base, pos);
+StaticInvoke expr = new StaticInvoke(Platform.class, "getLong",
PRIMITIVE_LONG_TYPE, base, pos);
+if (!Platform.IS_LITTLE_ENDIAN) {
+ expr = new StaticInvoke(Long.class, "reverseBytes", PRIMITIVE_INT_TYPE,
expr.inline());
+}
+return expr;
}
protected Expression unsafeGetFloat(Expression base, Expression pos) {
@@ -624,4 +628,12 @@ public abstract class CodecBuilder {
String func = Platform.IS_LITTLE_ENDIAN ? "readVarIntOnLE" :
"readVarIntOnBE";
return new Invoke(buffer, func, PRIMITIVE_INT_TYPE);
}
+
+ protected Expression readLong(Expression buffer) {
+return new Invoke(buffer, readLongFunc(), PRIMITIVE_LONG_TYPE);
+ }
+
+ public static String readLongFunc() {
+return Platform.IS_LITTLE_ENDIAN ? "readLongOnLE" : "readLongOnBE";
+ }
}
diff --git
a/java/fury-core/src/main/java/org/apache/fury/builder/CompatibleCodecBuilder.java
b/java/fury-core/src/main/java/org/apache/fury/builder/CompatibleCodecBuilder.java
index 50117db0..02be7621 100644
---
a/java/fury-core/src/main/java/org/apache/fury/builder/CompatibleCodecBuilder.java
+++
b/java/fury-core/src/main/java/org/apache/fury/builder/CompatibleCodecBuilder.java
@@ -718,7 +718,7 @@ public class CompatibleCodecBuilder extends
BaseObjectCodecBuilder {
setFieldValue(bean, descriptor, tryInlineCast(expr,
descriptor.getTypeToken(;
return new ListExpression(
deserializeAction,
-new Assign(partFieldInfo, inlineInvoke(buffer, "readLong",
PRIMITIVE_LONG_TYPE)));
+new Assign(partFieldInfo, inlineInvoke(buffer, readLongFunc(),
PRIMITIVE_LONG_TYPE)));
}
private void readSeparateTypesHashFields(
@@ -866,7 +866,7 @@ public class CompatibleCodecBuilder extends
BaseObjectCodecBuilder {
fieldInfo.getEncodedFieldInfo());
return new ListExpression(
new Expression.ForceEvaluate(readAction),
-new Assign(partFieldInfo, inlineInvoke(buffer, "readLong",
PRIMITIVE_LONG_TYPE)));
+new Assign(partFieldInfo, inlineInvoke(buffer, readLongFunc(),
PRIMITIVE_LONG_TYPE)));
}
protected Expression getFinalClassInfo(Class cls) {
@@ -939,7 +939,8 @@ public class CompatibleCodecBuilder extends
BaseObjectCodecBuilder {
partFieldInfo),
endTagLiteral),
returnEndTag ? new Return(endTagLiteral) : new
Return(bean)),
-new Assign(partFieldInfo, inlineInvoke(buffer, "readLong",
PRIMITIVE_LONG_TYPE;
+new Assign(
+partFieldInfo, inlineInvoke(buffer, readLongFunc(),
PRIMITIVE_LONG_TYPE;
}
private Expression skipField8End(
@@ -958,7 +959,8 @@ public class CompatibleCodecBuilder extends
BaseObjectCodecBuilder {
partFieldInfo),
endTagLiteral),
new Return(bean)),
-new Assign(partFieldInfo, inlineInvoke(buffer, "readLong",
PRIMITIVE_LONG_TYPE;
+new Assign(
+partFieldInfo, inlineInvoke(buffer, readLongFunc(),
PRIMITIVE_LONG_TYPE;
}
private Comparator isEmbedType(Expression partFieldInfo, int flagBits, byte
[GH] (incubator-fury): Workflow run "Fury CI" is working again!
The GitHub Actions job "Fury CI" on incubator-fury.git has succeeded. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 9a72a88c8831e0eb3036a5da18aed1308d1e9e4a / chaokunyang add readSliLongLE/BE method Report URL: https://github.com/apache/incubator-fury/actions/runs/8580802637 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: fdce72d5794842ea46b6be70ba0bd94796de8556 / chaokunyang generate unsafeGetLong online Report URL: https://github.com/apache/incubator-fury/actions/runs/8580762062 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: d5f4745994c652a618a0e1704bb5c0a991ce17a2 / chaokunyang optimize readVarLong code size Report URL: https://github.com/apache/incubator-fury/actions/runs/8580752193 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[PR] feat(java): reduce code size of read long for jvm jit inline [incubator-fury]
chaokunyang opened a new pull request, #1470: URL: https://github.com/apache/incubator-fury/pull/1470 This PR reduced code size of read long for jvm jit inline: - Reduced readVarLong code size by separating little/big endian - Reduced readLong code size by separating little/big endian - Reduced unsafeGetLong code size by separating little/big endian and generate online -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury) branch main updated: feat(java): reduce readInt/readVarInt code size for for jvm jit inline (#1469)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury.git
The following commit(s) were added to refs/heads/main by this push:
new d73c3d9f feat(java): reduce readInt/readVarInt code size for for jvm
jit inline (#1469)
d73c3d9f is described below
commit d73c3d9f8c0c750ffe718a1553cd10c45e605c57
Author: Shawn Yang
AuthorDate: Sat Apr 6 18:59:32 2024 +0800
feat(java): reduce readInt/readVarInt code size for for jvm jit inline
(#1469)
This PR optimizes readVarInt/readInt performance by reduce code size of
readVarInt for better jvm jit
---
.../fury/builder/BaseObjectCodecBuilder.java | 3 +-
.../java/org/apache/fury/builder/CodecBuilder.java | 16 ++-
.../apache/fury/builder/ObjectCodecBuilder.java| 2 +-
.../java/org/apache/fury/memory/MemoryBuffer.java | 118 -
4 files changed, 133 insertions(+), 6 deletions(-)
diff --git
a/java/fury-core/src/main/java/org/apache/fury/builder/BaseObjectCodecBuilder.java
b/java/fury-core/src/main/java/org/apache/fury/builder/BaseObjectCodecBuilder.java
index 0b0b7594..a27abbab 100644
---
a/java/fury-core/src/main/java/org/apache/fury/builder/BaseObjectCodecBuilder.java
+++
b/java/fury-core/src/main/java/org/apache/fury/builder/BaseObjectCodecBuilder.java
@@ -1135,8 +1135,7 @@ public abstract class BaseObjectCodecBuilder extends
CodecBuilder {
} else if (cls == short.class || cls == Short.class) {
return new Invoke(buffer, "readShort", PRIMITIVE_SHORT_TYPE);
} else if (cls == int.class || cls == Integer.class) {
-String func = fury.compressInt() ? "readVarInt" : "readInt";
-return new Invoke(buffer, func, PRIMITIVE_INT_TYPE);
+return fury.compressInt() ? readVarInt(buffer) : readInt(buffer);
} else if (cls == long.class || cls == Long.class) {
return LongSerializer.readLong(buffer, fury.longEncoding());
} else if (cls == float.class || cls == Float.class) {
diff --git
a/java/fury-core/src/main/java/org/apache/fury/builder/CodecBuilder.java
b/java/fury-core/src/main/java/org/apache/fury/builder/CodecBuilder.java
index 3ce014bc..457d41bf 100644
--- a/java/fury-core/src/main/java/org/apache/fury/builder/CodecBuilder.java
+++ b/java/fury-core/src/main/java/org/apache/fury/builder/CodecBuilder.java
@@ -595,7 +595,11 @@ public abstract class CodecBuilder {
}
protected Expression unsafeGetInt(Expression base, Expression pos) {
-return new StaticInvoke(MemoryBuffer.class, "unsafeGetInt",
PRIMITIVE_INT_TYPE, base, pos);
+StaticInvoke expr = new StaticInvoke(Platform.class, "getInt",
PRIMITIVE_INT_TYPE, base, pos);
+if (!Platform.IS_LITTLE_ENDIAN) {
+ expr = new StaticInvoke(Integer.class, "reverseBytes",
PRIMITIVE_INT_TYPE, expr.inline());
+}
+return expr;
}
protected Expression unsafeGetLong(Expression base, Expression pos) {
@@ -610,4 +614,14 @@ public abstract class CodecBuilder {
return new StaticInvoke(
MemoryBuffer.class, "unsafeGetDouble", PRIMITIVE_DOUBLE_TYPE, base,
pos);
}
+
+ protected Expression readInt(Expression buffer) {
+String func = Platform.IS_LITTLE_ENDIAN ? "readIntOnLE" : "readIntOnBE";
+return new Invoke(buffer, func, PRIMITIVE_INT_TYPE);
+ }
+
+ protected Expression readVarInt(Expression buffer) {
+String func = Platform.IS_LITTLE_ENDIAN ? "readVarIntOnLE" :
"readVarIntOnBE";
+return new Invoke(buffer, func, PRIMITIVE_INT_TYPE);
+ }
}
diff --git
a/java/fury-core/src/main/java/org/apache/fury/builder/ObjectCodecBuilder.java
b/java/fury-core/src/main/java/org/apache/fury/builder/ObjectCodecBuilder.java
index 37365f9d..6731c313 100644
---
a/java/fury-core/src/main/java/org/apache/fury/builder/ObjectCodecBuilder.java
+++
b/java/fury-core/src/main/java/org/apache/fury/builder/ObjectCodecBuilder.java
@@ -695,7 +695,7 @@ public class ObjectCodecBuilder extends
BaseObjectCodecBuilder {
compressStarted = true;
addIncReaderIndexExpr(groupExpressions, buffer, acc);
}
-fieldValue = new Invoke(buffer, "readVarInt", PRIMITIVE_INT_TYPE);
+fieldValue = readVarInt(buffer);
}
} else if (clz == long.class) {
if (!fury.compressLong()) {
diff --git
a/java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java
b/java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java
index 6c933318..97285991 100644
--- a/java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java
+++ b/java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java
@@ -1223,9 +1223,94 @@ public final class MemoryBuffer {
}
/** Reads the 1-5 byte int part of a varint. */
+ @CodegenInvoke
public int readVarInt() {
-int r = readPositiveVarInt();
-return (r >>> 1) ^ -(r & 1);
+if (LITTLE_ENDIAN) {
+ return
Re: [PR] feat(java): reduce readInt/readVarInt code size for for jvm jit inline [incubator-fury]
chaokunyang merged PR #1469: URL: https://github.com/apache/incubator-fury/pull/1469 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(java): optimize readVarInt performance for jvm jit [incubator-fury]
chaokunyang commented on PR #1469: URL: https://github.com/apache/incubator-fury/pull/1469#issuecomment-2041042694 Before:  After:  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] [Java] remove guava dependency [incubator-fury]
chaokunyang commented on issue #1113: URL: https://github.com/apache/incubator-fury/issues/1113#issuecomment-2041036344 > OK, I'll separate those by different PRs starting from `TypeRef` implementation, which looks like the most hard one, so it'll take some time... Yep, the `TypeRef` is the most compilciated one, and the `TypeToken` in guava has some performance issue, so we did some optimization in `org.apache.fury.type.TypeUtils`. If we have our own implementation, such helper can be simplified a lot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] [Java] remove guava dependency [incubator-fury]
Munoon commented on issue #1113: URL: https://github.com/apache/incubator-fury/issues/1113#issuecomment-2041035399 OK, I'll separate those by different PRs starting from `TypeRef` implementation, which looks like the most hard one, so it'll take some time... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(spec): add xlang type mapping [incubator-fury]
chaokunyang commented on PR #1468: URL: https://github.com/apache/incubator-fury/pull/1468#issuecomment-2041027971 BYW, I was thinking whether we should make all type id starts from 0, and let users to register type id from 64. In this way, we don't have to add ID by 64, and there won't be two kinds of ID to understand. One is the registed ID, the other is internal ID. But the drawback is that users must understand the concept of 64 and the pre-registered ID. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury) branch main updated: feat(java): extract public Fury methods to BaseFury (#1467)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury.git
The following commit(s) were added to refs/heads/main by this push:
new ca996d4e feat(java): extract public Fury methods to BaseFury (#1467)
ca996d4e is described below
commit ca996d4ed7cfedf8370ca993bc429972fbacf518
Author: Nikita Ivchenko
AuthorDate: Sat Apr 6 10:22:18 2024 +0300
feat(java): extract public Fury methods to BaseFury (#1467)
Extract public Fury method to the `BaseFury` interface and implement
them in `ThreadPoolFury` and `ThreadLocalFury`, so end users can avoid
using `execute()` and other utility methods.
Off topic: further steps might be changing `BaseFury` interface name to
`Fury` while existing class renaming to something like `FuryImpl`.
`FuryBuilder` should return the new `Fury` interface, so we will hide
system methods like `xwriteRef` and others. WDYT?
---
.../org/apache/fury/AbstractThreadSafeFury.java| 11 ++
.../src/main/java/org/apache/fury/BaseFury.java| 42
.../src/main/java/org/apache/fury/Fury.java| 20 +++-
.../main/java/org/apache/fury/ThreadLocalFury.java | 97 ++
.../java/org/apache/fury/pool/ThreadPoolFury.java | 114 +
5 files changed, 282 insertions(+), 2 deletions(-)
diff --git
a/java/fury-core/src/main/java/org/apache/fury/AbstractThreadSafeFury.java
b/java/fury-core/src/main/java/org/apache/fury/AbstractThreadSafeFury.java
index 9f60a191..4085e7d9 100644
--- a/java/fury-core/src/main/java/org/apache/fury/AbstractThreadSafeFury.java
+++ b/java/fury-core/src/main/java/org/apache/fury/AbstractThreadSafeFury.java
@@ -21,6 +21,7 @@ package org.apache.fury;
import java.util.function.Consumer;
import org.apache.fury.serializer.Serializer;
+import org.apache.fury.serializer.SerializerFactory;
public abstract class AbstractThreadSafeFury implements ThreadSafeFury {
@Override
@@ -48,5 +49,15 @@ public abstract class AbstractThreadSafeFury implements
ThreadSafeFury {
processCallback(fury -> fury.registerSerializer(type, serializerClass));
}
+ @Override
+ public void registerSerializer(Class type, Serializer serializer) {
+processCallback(fury -> fury.registerSerializer(type, serializer));
+ }
+
+ @Override
+ public void setSerializerFactory(SerializerFactory serializerFactory) {
+processCallback(fury -> fury.setSerializerFactory(serializerFactory));
+ }
+
protected abstract void processCallback(Consumer callback);
}
diff --git a/java/fury-core/src/main/java/org/apache/fury/BaseFury.java
b/java/fury-core/src/main/java/org/apache/fury/BaseFury.java
index 2fc943cf..e268dfb5 100644
--- a/java/fury-core/src/main/java/org/apache/fury/BaseFury.java
+++ b/java/fury-core/src/main/java/org/apache/fury/BaseFury.java
@@ -19,8 +19,12 @@
package org.apache.fury;
+import java.io.InputStream;
+import java.io.OutputStream;
import org.apache.fury.memory.MemoryBuffer;
+import org.apache.fury.serializer.BufferCallback;
import org.apache.fury.serializer.Serializer;
+import org.apache.fury.serializer.SerializerFactory;
import org.apache.fury.serializer.Serializers;
/** All Fury’s basic interface, including Fury’s basic methods. */
@@ -64,9 +68,16 @@ public interface BaseFury {
*/
void registerSerializer(Class type, Class
serializerClass);
+ void registerSerializer(Class type, Serializer serializer);
+
+ void setSerializerFactory(SerializerFactory serializerFactory);
+
/** Return serialized obj as a byte array. */
byte[] serialize(Object obj);
+ /** Return serialized obj as a byte array. */
+ byte[] serialize(Object obj, BufferCallback callback);
+
/**
* Serialize obj to a off-heap buffer specified by
address and
* size.
@@ -76,9 +87,18 @@ public interface BaseFury {
/** Serialize data into buffer. */
MemoryBuffer serialize(MemoryBuffer buffer, Object obj);
+ /** Serialize obj to a buffer. */
+ MemoryBuffer serialize(MemoryBuffer buffer, Object obj, BufferCallback
callback);
+
+ void serialize(OutputStream outputStream, Object obj);
+
+ void serialize(OutputStream outputStream, Object obj, BufferCallback
callback);
+
/** Deserialize obj from a byte array. */
Object deserialize(byte[] bytes);
+ Object deserialize(byte[] bytes, Iterable outOfBandBuffers);
+
/**
* Deserialize obj from a off-heap buffer specified by
address and
* size.
@@ -88,6 +108,12 @@ public interface BaseFury {
/** Deserialize obj from a buffer. */
Object deserialize(MemoryBuffer buffer);
+ Object deserialize(MemoryBuffer buffer, Iterable
outOfBandBuffers);
+
+ Object deserialize(InputStream inputStream);
+
+ Object deserialize(InputStream inputStream, Iterable
outOfBandBuffers);
+
/**
* Serialize java object without class info, deserialization should use
{@link
* #deserializeJavaObject}.
Re: [PR] feat(java): extract public Fury methods to BaseFury [incubator-fury]
chaokunyang merged PR #1467: URL: https://github.com/apache/incubator-fury/pull/1467 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] [Java] remove guava dependency [incubator-fury]
chaokunyang commented on issue #1113: URL: https://github.com/apache/incubator-fury/issues/1113#issuecomment-2041000105 > > @Munoon That would be really great. Actually I am on the work of first release of Fury under asf and don't have enough time for this feature > > Can you please specify which exactly help you need? To complete @nandakumar131 PR or to do some further steps? Here are the things which may need to be finished: - Implement a `TypeRef` to replace `TypeToken` in guava, which is work that @nandakumar131 doing before. - Remove guava Cache which Fury used for weak key and soft values map - Remove guava `FinalizableWeakReference` which fury used to build a multi-key weak map - Remove guava `ListeningExecutorService` which fury used to compile classes and register callback - Remove guava `com.google.common.collect` usage in the src/main/java dir except `org.apache.fury.serializer.collection.GuavaCollectionSerializers` - Make `org.apache.fury.serializer.collection.GuavaCollectionSerializers` load stub classes, so even guava jar are not in classpath, teh fury init won't throw exception, like we did in org.apache.fury.serializer.collection.ImmutableCollectionSerializers - Add a script to check `com.google.` are only used in `GuavaCollectionSerializers` and test classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] [Java] optimize map performance for reference tracking and serializer dispatch [incubator-fury]
chaokunyang commented on issue #1409:
URL:
https://github.com/apache/incubator-fury/issues/1409#issuecomment-2040992654
Hi @tommyettinger , thanks for this hard work.
Fury has a `src/main/java/org/apache/fury/benchmark/MapSuite.java` which did
some benchmarks for IdentityMap.
For end-to-end tests,
`src/main/java/org/apache/fury/benchmark/UserTypeSerializeSuite.java` can be
used. You can use following config:
```java
public static void main(String[] args) throws IOException {
if (args.length == 0) {
String commandLine =
"org.apache.fury.*UserTypeSerializeSuite.fury* -f 3 -wi 10 -i 10
-t 1 -w 2s -r 2s -rf csv";
System.out.println(commandLine);
args = commandLine.split(" ");
}
Main.main(args);
}
```
`org.apache.fury.benchmark.state.BenchmarkState#references` should be set to
true only, otherwise referecne tracking won't be enabled:
```java
@Param({"true"})
public boolean references;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] chore(java): Add comments for loop unrolling [incubator-fury]
chaokunyang commented on PR #1450: URL: https://github.com/apache/incubator-fury/pull/1450#issuecomment-2040991342 @LiangliangSui There are some conflict between your branch and main branch, could you resolve the conflict? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] chore(java): Add comments for loop unrolling [incubator-fury]
chaokunyang commented on code in PR #1450:
URL: https://github.com/apache/incubator-fury/pull/1450#discussion_r1554535283
##
java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java:
##
@@ -1316,24 +1316,15 @@ public int readPositiveVarInt() {
}
private int readPositiveVarIntSlow() {
-int b = readByte();
-int result = b & 0x7F;
-if ((b & 0x80) != 0) {
+int shift = 0;
Review Comment:
I don't think it's necessary, only rolling code will be used in the release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] [Java] optimize map performance for reference tracking and serializer dispatch [incubator-fury]
tommyettinger commented on issue #1409: URL: https://github.com/apache/incubator-fury/issues/1409#issuecomment-2040990074 I'm making progress on the rest of the map types now; IdentityMap seems done and ObjectIntMap might be done or close to it. Hybridizing the two shouldn't be hard at all. Is there a particular benchmark that you think would best test the performance of these map types? It looks like running all benchmarks would take well over a day. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
