Re: [I] [JavaScript] Implement meta string encoding algorithm for JavaScript [incubator-fury]
chaokunyang commented on issue #1542: URL: https://github.com/apache/incubator-fury/issues/1542#issuecomment-2144401868 > Hello @chaokunyang , I am Forcha Pearl from Cameroon . I am very good bit manipulation, endianess, encoding DataStructures and Algorithm. I also experienced with Java, python and javascript. I am interested in participting in OSPP 2024 under Apache furry this year. Please has this issue ben reserved for the OSPP contribution period. Hi @Forchapeatl , this is not an issue reserved for the OSPP. If it's not taken by someone else in this month, we will take it over. The issues for OSPP can be found at #1670 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user LiangliangSui (triggered by LiangliangSui). Head commit for run: c57e1f882d1e7ed75c0a5ceb24a862c4267b7bf5 / LiangliangSui fix ci error Signed-off-by: LiangliangSui Report URL: https://github.com/apache/incubator-fury/actions/runs/9345489747 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user LiangliangSui (triggered by LiangliangSui). Head commit for run: e449e5b174cab2243c767ac1aee0dcefbfaac75b / LiangliangSui add PACKAGE_ENCODER Signed-off-by: LiangliangSui Report URL: https://github.com/apache/incubator-fury/actions/runs/9345275019 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] perf(java): Add ClassInfo ClassBytes generation conditions. [incubator-fury]
LiangliangSui commented on code in PR #1667: URL: https://github.com/apache/incubator-fury/pull/1667#discussion_r1623834201 ## java/fury-core/src/main/java/org/apache/fury/resolver/ClassInfo.java: ## @@ -86,14 +85,19 @@ public class ClassInfo { } else { this.fullClassNameBytes = null; } +// When `classId == ClassResolver.REPLACE_STUB_ID` was established, +// means only classes are serialized, not the instance. If we +// serialize such class only, we need to write classname bytes. if (cls != null -&& (classId == ClassResolver.NO_CLASS_ID || classId == ClassResolver.REPLACE_STUB_ID)) { +&& ((classId == ClassResolver.NO_CLASS_ID +&& !classResolver.getFury().getConfig().isMetaShareEnabled()) +|| classId == ClassResolver.REPLACE_STUB_ID)) { // REPLACE_STUB_ID for write replace class in `ClassSerializer`. Tuple2 tuple2 = Encoders.encodePkgAndClass(cls); this.packageNameBytes = - metaStringResolver.getOrCreateMetaStringBytes(PACKAGE_ENCODER.encode(tuple2.f0)); + metaStringResolver.getOrCreateMetaStringBytes(Encoders.encodePackage(tuple2.f0)); this.classNameBytes = - metaStringResolver.getOrCreateMetaStringBytes(TYPE_NAME_ENCODER.encode(tuple2.f1)); + metaStringResolver.getOrCreateMetaStringBytes(Encoders.encodeTypeName(tuple2.f1)); Review Comment: `MetaStringBytes#hashCode & 0b111` only controls the range of the encoding set, not the encoding set to be used. > The inconsistency may introduce some error Could you elaborate on this? I'm not sure what will cause the inconsistency. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[I] Implement metadatascoped share mode [incubator-fury]
chaokunyang opened a new issue, #1673: URL: https://github.com/apache/incubator-fury/issues/1673 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[I] [Javascript] Type Forward/Backward compatible object serialization for Fury JavaScript [incubator-fury]
chaokunyang opened a new issue, #1670: URL: https://github.com/apache/incubator-fury/issues/1670 ## Is your feature request related to a problem? Please describe. ## Describe the solution you'd like ## Additional context -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] [scala]Can furry support serializing Seq collections and case classes in Scala, and then deserializing the resulting objects into corresponding Java objects in Java? [incubator-fury]
chaokunyang commented on issue #1669: URL: https://github.com/apache/incubator-fury/issues/1669#issuecomment-2144299591 Hi @xiaxianggo , you can register a customized serializer for this: ```java class SeqSerializer extends AbstractCollectionSerializer { @Override public Collection onCollectionWrite(MemoryBuffer buffer, Object value) { } @Override public Object read(MemoryBuffer buffer) { } @Override public Collection newCollection(MemoryBuffer buffer) { int numElements = buffer.readVarUint32Small7(); setNumElements(numElements); ArrayList arrayList = new ArrayList(numElements); fury.getRefResolver().reference(arrayList); return arrayList; } @Override public Object onCollectionRead(Collection collection) { } } ``` Take `org.apache.fury.serializer.collection.CollectionSerializers.ArrayListSerializer` as an example -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] perf(java): Add ClassInfo ClassBytes generation conditions. [incubator-fury]
chaokunyang commented on code in PR #1667: URL: https://github.com/apache/incubator-fury/pull/1667#discussion_r1623791643 ## java/fury-core/src/main/java/org/apache/fury/resolver/ClassResolver.java: ## @@ -1701,7 +1701,7 @@ public Class xreadClass(MemoryBuffer buffer) { Class cls = classNameBytes2Class.get(byteString); if (cls == null) { Preconditions.checkNotNull(byteString); - String className = byteString.decode('.', '_'); + String className = byteString.decode(Encoders.PACKAGE_DECODER); Review Comment: How about changing it to `GENERIC_ENCODER` introduced in this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] perf(java): Add ClassInfo ClassBytes generation conditions. [incubator-fury]
chaokunyang commented on code in PR #1667: URL: https://github.com/apache/incubator-fury/pull/1667#discussion_r1623791156 ## java/fury-core/src/main/java/org/apache/fury/resolver/ClassInfo.java: ## @@ -86,14 +85,19 @@ public class ClassInfo { } else { this.fullClassNameBytes = null; } +// When `classId == ClassResolver.REPLACE_STUB_ID` was established, +// means only classes are serialized, not the instance. If we +// serialize such class only, we need to write classname bytes. if (cls != null -&& (classId == ClassResolver.NO_CLASS_ID || classId == ClassResolver.REPLACE_STUB_ID)) { +&& ((classId == ClassResolver.NO_CLASS_ID +&& !classResolver.getFury().getConfig().isMetaShareEnabled()) +|| classId == ClassResolver.REPLACE_STUB_ID)) { // REPLACE_STUB_ID for write replace class in `ClassSerializer`. Tuple2 tuple2 = Encoders.encodePkgAndClass(cls); this.packageNameBytes = - metaStringResolver.getOrCreateMetaStringBytes(PACKAGE_ENCODER.encode(tuple2.f0)); + metaStringResolver.getOrCreateMetaStringBytes(Encoders.encodePackage(tuple2.f0)); this.classNameBytes = - metaStringResolver.getOrCreateMetaStringBytes(TYPE_NAME_ENCODER.encode(tuple2.f1)); + metaStringResolver.getOrCreateMetaStringBytes(Encoders.encodeTypeName(tuple2.f1)); Review Comment: The encoding set in `MetaStringBytes#hashCode & 0x111` use whole `org.apache.fury.meta.MetaString.Encoding.values()`. But the `Encoders.encodePackage` only use `UTF_8, ALL_TO_LOWER_SPECIAL, LOWER_UPPER_DIGIT_SPECIAL`. The inconsistency may introduce some error -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury) branch main updated: fix(java): Fix header offset issue in MetaStringBytes hashcode (#1668)
This is an automated email from the ASF dual-hosted git repository. chaokunyang pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-fury.git The following commit(s) were added to refs/heads/main by this push: new a2515a93 fix(java): Fix header offset issue in MetaStringBytes hashcode (#1668) a2515a93 is described below commit a2515a936b439129d93eb22acc5c63a23285f23b Author: LiangliangSui <116876207+liangliang...@users.noreply.github.com> AuthorDate: Mon Jun 3 13:07:58 2024 +0800 fix(java): Fix header offset issue in MetaStringBytes hashcode (#1668) ## What does this PR do? MetaStringBytes `hashcode & 0xff`, that is, header, represents the encoding ## Related issues ## Does this PR introduce any user-facing change? - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark - Signed-off-by: LiangliangSui --- .../src/main/java/org/apache/fury/resolver/MetaStringBytes.java | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/java/fury-core/src/main/java/org/apache/fury/resolver/MetaStringBytes.java b/java/fury-core/src/main/java/org/apache/fury/resolver/MetaStringBytes.java index 8867a9e7..f14cbce8 100644 --- a/java/fury-core/src/main/java/org/apache/fury/resolver/MetaStringBytes.java +++ b/java/fury-core/src/main/java/org/apache/fury/resolver/MetaStringBytes.java @@ -28,6 +28,7 @@ import org.apache.fury.util.MurmurHash3; @Internal final class MetaStringBytes { static final short DEFAULT_DYNAMIC_WRITE_STRING_ID = -1; + private static final int HEADER_MASK = 0xff; final byte[] bytes; final long hashCode; @@ -55,7 +56,7 @@ final class MetaStringBytes { hashCode += 256; // last byte is reserved for header. } hashCode &= 0xff00L; -int header = metaString.getEncoding().getValue(); +int header = metaString.getEncoding().getValue() & HEADER_MASK; this.hashCode = hashCode | header; } @@ -64,9 +65,8 @@ final class MetaStringBytes { } public String decode(MetaStringDecoder decoder) { -int header = (int) (hashCode & 0xff); -int encodingFlags = header & 0b111; -MetaString.Encoding encoding = MetaString.Encoding.values()[encodingFlags]; +int header = (int) (hashCode & HEADER_MASK); +MetaString.Encoding encoding = MetaString.Encoding.values()[header]; return decoder.decode(bytes, encoding); } - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] fix(java): Fix header offset issue in MetaStringBytes hashcode [incubator-fury]
chaokunyang merged PR #1668: URL: https://github.com/apache/incubator-fury/pull/1668 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" is working again!
The GitHub Actions job "Fury CI" on incubator-fury.git has succeeded. Run started by GitHub user LiangliangSui (triggered by LiangliangSui). Head commit for run: 8cdb114a2abf417470434c1aaefcf5efad22a78f / LiangliangSui lint code Signed-off-by: LiangliangSui Report URL: https://github.com/apache/incubator-fury/actions/runs/9343074991 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[I] [scala] When serializing collections and case classes in Scala, can furry support deserializing the resulting objects into corresponding Java objects in Java? [incubator-fury]
xiaxianggo opened a new issue, #1669: URL: https://github.com/apache/incubator-fury/issues/1669 Can furry support serializing Seq collections and case classes in Scala, and then deserializing the resulting objects into corresponding Java objects in Java? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user LiangliangSui (triggered by LiangliangSui). Head commit for run: 448695a1e2dfdcb2d0e9474ba735b5967ba74b90 / LiangliangSui fix(java): Fix header offset issue in MetaStringBytes hashcode Signed-off-by: LiangliangSui Report URL: https://github.com/apache/incubator-fury/actions/runs/9338774779 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[PR] fix(java): Fix header offset issue in MetaStringBytes hashcode [incubator-fury]
LiangliangSui opened a new pull request, #1668: URL: https://github.com/apache/incubator-fury/pull/1668 ## What does this PR do? MetaStringBytes `hashcode & 0xff`, that is, header, represents the encoding ## Related issues ## Does this PR introduce any user-facing change? - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] perf(java): Add ClassInfo ClassBytes generation conditions. [incubator-fury]
LiangliangSui commented on code in PR #1667: URL: https://github.com/apache/incubator-fury/pull/1667#discussion_r1623416961 ## java/fury-core/src/main/java/org/apache/fury/resolver/ClassInfo.java: ## @@ -86,14 +85,19 @@ public class ClassInfo { } else { this.fullClassNameBytes = null; } +// When `classId == ClassResolver.REPLACE_STUB_ID` was established, +// means only classes are serialized, not the instance. If we +// serialize such class only, we need to write classname bytes. if (cls != null -&& (classId == ClassResolver.NO_CLASS_ID || classId == ClassResolver.REPLACE_STUB_ID)) { +&& ((classId == ClassResolver.NO_CLASS_ID +&& !classResolver.getFury().getConfig().isMetaShareEnabled()) +|| classId == ClassResolver.REPLACE_STUB_ID)) { // REPLACE_STUB_ID for write replace class in `ClassSerializer`. Tuple2 tuple2 = Encoders.encodePkgAndClass(cls); this.packageNameBytes = - metaStringResolver.getOrCreateMetaStringBytes(PACKAGE_ENCODER.encode(tuple2.f0)); + metaStringResolver.getOrCreateMetaStringBytes(Encoders.encodePackage(tuple2.f0)); this.classNameBytes = - metaStringResolver.getOrCreateMetaStringBytes(TYPE_NAME_ENCODER.encode(tuple2.f1)); + metaStringResolver.getOrCreateMetaStringBytes(Encoders.encodeTypeName(tuple2.f1)); Review Comment: And `MetaStringEncoder` supports UTF8 by default, even if my `EncodingSet` does not specify UTF8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] perf(java): Add ClassInfo ClassBytes generation conditions. [incubator-fury]
LiangliangSui commented on code in PR #1667: URL: https://github.com/apache/incubator-fury/pull/1667#discussion_r1623412799 ## java/fury-core/src/main/java/org/apache/fury/resolver/ClassResolver.java: ## @@ -1701,7 +1701,7 @@ public Class xreadClass(MemoryBuffer buffer) { Class cls = classNameBytes2Class.get(byteString); if (cls == null) { Preconditions.checkNotNull(byteString); - String className = byteString.decode('.', '_'); + String className = byteString.decode(Encoders.PACKAGE_DECODER); Review Comment: I saw that the [ClassInfo constructor](https://github.com/apache/incubator-fury/blob/main/java/fury-core/src/main/java/org/apache/fury/resolver/ClassInfo.java#L84-L85) also uses PACKAGE_ENCODER encoding `Class#getName()` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] perf(java): Add ClassInfo ClassBytes generation conditions. [incubator-fury]
LiangliangSui commented on code in PR #1667: URL: https://github.com/apache/incubator-fury/pull/1667#discussion_r1623412375 ## java/fury-core/src/main/java/org/apache/fury/resolver/ClassInfo.java: ## @@ -86,14 +85,19 @@ public class ClassInfo { } else { this.fullClassNameBytes = null; } +// When `classId == ClassResolver.REPLACE_STUB_ID` was established, +// means only classes are serialized, not the instance. If we +// serialize such class only, we need to write classname bytes. if (cls != null -&& (classId == ClassResolver.NO_CLASS_ID || classId == ClassResolver.REPLACE_STUB_ID)) { +&& ((classId == ClassResolver.NO_CLASS_ID +&& !classResolver.getFury().getConfig().isMetaShareEnabled()) +|| classId == ClassResolver.REPLACE_STUB_ID)) { // REPLACE_STUB_ID for write replace class in `ClassSerializer`. Tuple2 tuple2 = Encoders.encodePkgAndClass(cls); this.packageNameBytes = - metaStringResolver.getOrCreateMetaStringBytes(PACKAGE_ENCODER.encode(tuple2.f0)); + metaStringResolver.getOrCreateMetaStringBytes(Encoders.encodePackage(tuple2.f0)); this.classNameBytes = - metaStringResolver.getOrCreateMetaStringBytes(TYPE_NAME_ENCODER.encode(tuple2.f1)); + metaStringResolver.getOrCreateMetaStringBytes(Encoders.encodeTypeName(tuple2.f1)); Review Comment: I think we should not update `MetaStringBytes#decode` for the following reasons: `Encoders.encodePackage` supports UTF-8 encoding by default, so even if the classname contains unicode characters, it can be encoded successfully. ```java static final Encoding[] pkgEncodings = new Encoding[] {UTF_8, ALL_TO_LOWER_SPECIAL, LOWER_UPPER_DIGIT_SPECIAL}; static final Encoding[] typeNameEncodings = new Encoding[] { UTF_8, LOWER_UPPER_DIGIT_SPECIAL, FIRST_TO_LOWER_SPECIAL, ALL_TO_LOWER_SPECIAL }; ``` I coded a unit test, as follows, which pass ```java @Test public void testEncoding() { String str = "你好,世界"; MetaString metaString = Encoders.encodePackage(str); MetaStringBytes metaStringBytes = new MetaStringBytes(metaString); String decode = metaStringBytes.decode(Encoders.PACKAGE_DECODER); Assert.assertEquals(str, decode); } ``` > And the key here is that MetaStringBytes can't distinguish whether the data is package or classname, so the decode need use the whole encoding set `MetaStringBytes#decode` uses the encoding stored in `MetaStringBytes#hashCode & 0x111`, which means that if the encoding is successful, the decoding will be successful. It should have nothing to do with the encoding set. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org