Re: [PR] feat(java): support user context for serialize global data(#1595) [incubator-fury]
MrChang0 commented on code in PR #1596:
URL: https://github.com/apache/incubator-fury/pull/1596#discussion_r1591807170
##
java/fury-core/src/main/java/org/apache/fury/resolver/UserContextResolver.java:
##
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.fury.resolver;
+
+import org.apache.fury.Fury;
+import org.apache.fury.memory.MemoryBuffer;
+
+/** write/read user custom global data after metaContext data. */
+public abstract class UserContextResolver {
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(java): support user context for serialize global data(#1595) [incubator-fury]
MrChang0 commented on code in PR #1596:
URL: https://github.com/apache/incubator-fury/pull/1596#discussion_r1591785196
##
java/fury-core/src/main/java/org/apache/fury/resolver/UserContextResolver.java:
##
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.fury.resolver;
+
+import org.apache.fury.Fury;
+import org.apache.fury.memory.MemoryBuffer;
+
+/** write/read user custom global data after metaContext data. */
+public abstract class UserContextResolver {
+
+ protected final Fury fury;
+
+ public UserContextResolver(Fury fury) {
+this.fury = fury;
+ }
+
+ public abstract void write(MemoryBuffer buffer);
+
+ public abstract void read(MemoryBuffer buffer);
+
+ /** write/read end should clear user data. */
+ public abstract void reset();
Review Comment:
I would like context used without state, context is global in a fury and
reset every write/read. if we do it like meta context, we need store contexts
and fury. so I think context design should be same and I wander your advice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury-site) branch deploy updated: deploy: 80cc38d91582c87c48f9166c89fe5713fd596fc5
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch deploy in repository https://gitbox.apache.org/repos/asf/incubator-fury-site.git The following commit(s) were added to refs/heads/deploy by this push: new c8b3ef5 deploy: 80cc38d91582c87c48f9166c89fe5713fd596fc5 c8b3ef5 is described below commit c8b3ef5238af50a36f2b2ab02de3476557b41880 Author: chaokunyang AuthorDate: Tue May 7 02:33:11 2024 + deploy: 80cc38d91582c87c48f9166c89fe5713fd596fc5 --- 404.html | 4 ++-- assets/js/1e879fa7.b7e54c13.js | 1 + assets/js/1e879fa7.d316a53d.js | 1 - assets/js/814f3328.d60b23ce.js | 1 + assets/js/814f3328.de5520b6.js | 1 - assets/js/b2f554cd.29ae74c4.js | 1 - assets/js/b2f554cd.4a851524.js | 1 + assets/js/b486c30c.c2feec5d.js | 1 - assets/js/b486c30c.de3d3e10.js | 1 + assets/js/cc296a15.295f3a9f.js | 1 + assets/js/cc296a15.b59ee70f.js | 1 - assets/js/f1aad19d.634ce94b.js | 1 + assets/js/f1aad19d.da5e5582.js | 1 - assets/js/main.9025f89e.js | 2 -- assets/js/main.a8a3f08e.js | 2 ++ ...5f89e.js.LICENSE.txt => main.a8a3f08e.js.LICENSE.txt} | 0 ...runtime~main.36ff1eee.js => runtime~main.4c4d0244.js} | 2 +- blog/archive/index.html | 6 +++--- blog/atom.xml| 16 blog/fury_0_1_0_release/index.html | 6 +++--- blog/fury_0_1_1_release/index.html | 6 +++--- blog/fury_0_1_2_release/index.html | 6 +++--- blog/fury_0_2_0_release/index.html | 6 +++--- blog/fury_0_2_1_release/index.html | 6 +++--- blog/fury_0_3_0_release/index.html | 6 +++--- blog/fury_0_3_1_release/index.html | 6 +++--- blog/fury_0_4_0_release/index.html | 6 +++--- blog/fury_0_4_1_release/index.html | 6 +++--- blog/fury_0_5_0_release/index.html | 8 .../index.html | 6 +++--- .../index.html | 8 blog/index.html | 6 +++--- blog/page/2/index.html | 6 +++--- blog/rss.xml | 16 blog/tags/fury/index.html| 6 +++--- blog/tags/fury/page/2/index.html | 6 +++--- blog/tags/index.html | 6 +++--- docs/guide/development/index.html| 4 ++-- docs/guide/graalvm_guide/index.html | 4 ++-- docs/guide/java_object_graph_guide/index.html| 4 ++-- docs/guide/row_format_guide/index.html | 4 ++-- docs/guide/scala_guide/index.html| 4 ++-- docs/guide/xlang_object_graph_guide/index.html | 4 ++-- docs/guide/xlang_type_mapping/index.html | 4 ++-- docs/introduction/benchmark/index.html | 4 ++-- docs/introduction/features/index.html| 4 ++-- docs/introduction/index.html | 4 ++-- .../fury_java_serialization_spec/index.html | 4 ++-- docs/specification/fury_row_format_spec/index.html | 4 ++-- .../fury_xlang_serialization_spec/index.html | 4 ++-- docs/start/install/index.html| 4 ++-- docs/start/usage/index.html | 4 ++-- download/index.html | 4 ++-- index.html | 4 ++-- lunr-index-1715048363964.json| 1 - lunr-index-1715049161377.json| 1 + lunr-index.json | 2 +- markdown-page/index.html | 4 ++-- search-doc-1715048363964.json| 1 - search-doc-1715049161377.json| 1 + search-doc.json | 2 +- sitemap.xml | 2 +- 62 files changed, 124 insertions(+), 124 deletions(-) diff --git a/404.html b/404.html index 1c73849..a5c24fc 100644 --- a/404.html +++ b/404.html @@ -5,8 +5,8 @@ Page Not Found | Apache Fury (incubating)https://fury.apache.org/404.html";> - - + + !fu
(incubator-fury-site) branch main updated: fix metastring link (#122)
This is an automated email from the ASF dual-hosted git repository. chaokunyang pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-fury-site.git The following commit(s) were added to refs/heads/main by this push: new 80cc38d fix metastring link (#122) 80cc38d is described below commit 80cc38d91582c87c48f9166c89fe5713fd596fc5 Author: Shawn Yang AuthorDate: Tue May 7 10:32:11 2024 +0800 fix metastring link (#122) --- blog/2024-05-06-metastring-space-efficient_encoding_for_string.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blog/2024-05-06-metastring-space-efficient_encoding_for_string.md b/blog/2024-05-06-metastring-space-efficient_encoding_for_string.md index 4129247..550bfdd 100644 --- a/blog/2024-05-06-metastring-space-efficient_encoding_for_string.md +++ b/blog/2024-05-06-metastring-space-efficient_encoding_for_string.md @@ -1,5 +1,5 @@ --- -slug: fury +slug: fury_meta_string_37_5_percent_space_efficient_encoding_than_utf8 title: 'Meta String: A 37.5% space efficient string encoding than UTF-8 in Fury serialization' authors: [chaokunyang] tags: [fury] - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] fix metastring link [incubator-fury-site]
chaokunyang merged PR #122: URL: https://github.com/apache/incubator-fury-site/pull/122 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[PR] fix metastring link [incubator-fury-site]
chaokunyang opened a new pull request, #122: URL: https://github.com/apache/incubator-fury-site/pull/122 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury-site) branch deploy updated: deploy: 4e3a34c34b4a1238e126112e0529676160f100c1
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch deploy
in repository https://gitbox.apache.org/repos/asf/incubator-fury-site.git
The following commit(s) were added to refs/heads/deploy by this push:
new 5b31f6a deploy: 4e3a34c34b4a1238e126112e0529676160f100c1
5b31f6a is described below
commit 5b31f6ab7ae96aff29685250f510cef770800622
Author: chaokunyang
AuthorDate: Tue May 7 02:19:52 2024 +
deploy: 4e3a34c34b4a1238e126112e0529676160f100c1
---
404.html | 4 ++--
assets/js/b2f554cd.29ae74c4.js | 1 +
assets/js/b2f554cd.867754a2.js | 1 -
assets/js/cc296a15.580a3b9a.js | 1 -
assets/js/cc296a15.b59ee70f.js | 1 +
assets/js/f1aad19d.8fd0635e.js | 1 -
assets/js/f1aad19d.da5e5582.js | 1 +
assets/js/{main.e04cc605.js => main.9025f89e.js} | 4 ++--
...js.LICENSE.txt => main.9025f89e.js.LICENSE.txt} | 0
...e~main.f673c4ba.js => runtime~main.36ff1eee.js} | 2 +-
blog/archive/index.html| 4 ++--
blog/atom.xml | 22 +-
blog/fury/index.html | 26 +++---
blog/fury_0_1_0_release/index.html | 4 ++--
blog/fury_0_1_1_release/index.html | 4 ++--
blog/fury_0_1_2_release/index.html | 4 ++--
blog/fury_0_2_0_release/index.html | 4 ++--
blog/fury_0_2_1_release/index.html | 4 ++--
blog/fury_0_3_0_release/index.html | 4 ++--
blog/fury_0_3_1_release/index.html | 4 ++--
blog/fury_0_4_0_release/index.html | 4 ++--
blog/fury_0_4_1_release/index.html | 4 ++--
blog/fury_0_5_0_release/index.html | 4 ++--
.../index.html | 4 ++--
blog/index.html| 26 +++---
blog/page/2/index.html | 4 ++--
blog/rss.xml | 22 +-
blog/tags/fury/index.html | 26 +++---
blog/tags/fury/page/2/index.html | 4 ++--
blog/tags/index.html | 4 ++--
docs/guide/development/index.html | 4 ++--
docs/guide/graalvm_guide/index.html| 4 ++--
docs/guide/java_object_graph_guide/index.html | 4 ++--
docs/guide/row_format_guide/index.html | 4 ++--
docs/guide/scala_guide/index.html | 4 ++--
docs/guide/xlang_object_graph_guide/index.html | 4 ++--
docs/guide/xlang_type_mapping/index.html | 4 ++--
docs/introduction/benchmark/index.html | 4 ++--
docs/introduction/features/index.html | 4 ++--
docs/introduction/index.html | 4 ++--
.../fury_java_serialization_spec/index.html| 4 ++--
docs/specification/fury_row_format_spec/index.html | 4 ++--
.../fury_xlang_serialization_spec/index.html | 4 ++--
docs/start/install/index.html | 4 ++--
docs/start/usage/index.html| 4 ++--
download/index.html| 4 ++--
index.html | 4 ++--
lunr-index-1715045432718.json | 1 -
lunr-index-1715048363964.json | 1 +
lunr-index.json| 2 +-
markdown-page/index.html | 4 ++--
search-doc-1715045432718.json | 1 -
search-doc-1715048363964.json | 1 +
search-doc.json| 2 +-
54 files changed, 139 insertions(+), 139 deletions(-)
diff --git a/404.html b/404.html
index eaa3e44..1c73849 100644
--- a/404.html
+++ b/404.html
@@ -5,8 +5,8 @@
Page Not Found | Apache Fury (incubating)https://fury.apache.org/404.html";>
-
-
+
+
!function(){function
t(t){document.documentElement.setAttribute("data-theme",t)}var
e=function(){try{return new
URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}}()||function(){try{return
localStorage.getItem("theme")}catch(t){}}();t(null!==e?e:"light")}(),function(){try{const
c=new URLSearchParams(window.location.search).entries();for(var[t,e]of
c)if(t.startsWith("docusaurus-data-")){var
a=t.replace("docusaurus-data-","data-");document.documentElement.se [...]
diff --git a/assets/js/b2f554cd.29ae74c4.js b/assets/js/b2f554cd.29ae74c4.js
new file mode 100644
index 000..380a5d8
--- /dev/null
+++ b/assets/js/b2f554cd.29ae74c4.js
@@ -0,0 +1 @@
+"use
strict";(self.webpackChunkfury_site=self.webpackChunkfury_site||[]).push([[1477],{10:a=>{a.exports=JSON.parse('{"blogPosts":[{"id":"fury"
[PR] refine meta string blog [incubator-fury-site]
chaokunyang opened a new pull request, #121: URL: https://github.com/apache/incubator-fury-site/pull/121 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury-site) branch main updated: refine meta string blog (#121)
This is an automated email from the ASF dual-hosted git repository. chaokunyang pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-fury-site.git The following commit(s) were added to refs/heads/main by this push: new 4e3a34c refine meta string blog (#121) 4e3a34c is described below commit 4e3a34c34b4a1238e126112e0529676160f100c1 Author: Shawn Yang AuthorDate: Tue May 7 10:18:55 2024 +0800 refine meta string blog (#121) --- ...tastring-space-efficient_encoding_for_string.md | 22 +++--- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/blog/2024-05-06-metastring-space-efficient_encoding_for_string.md b/blog/2024-05-06-metastring-space-efficient_encoding_for_string.md index 8d75c52..4129247 100644 --- a/blog/2024-05-06-metastring-space-efficient_encoding_for_string.md +++ b/blog/2024-05-06-metastring-space-efficient_encoding_for_string.md @@ -7,16 +7,16 @@ tags: [fury] ## Background -In rpc/serialization systems, we often need to send **`namespace/path/filename/fieldName/packageName/moduleName/className/enumValue`** between processes. +In rpc/serialization systems, we often need to send **`namespace/path/filename/fieldName/packageName/moduleName/className/enumValue`** string between processes. -Those strings are mostly ascii strings. In order to transfer between processes, we often encode such strings using utf-8 encodings. Such encoding +Those strings are mostly ascii strings. In order to transfer between processes, we encode such strings using utf-8 encodings. Such encoding will take one byte for every char, which is not space efficient actually. -If we take a deeper look, we will found that most chars are **lower chars plus `.`, `$` and `_`**, which can be expressed in a much -smaller range **`0~32`**, and one byte can represent range `0~255`, the significant bits are wasted. And the cost is not ignorable, in a dynamic serialization +If we take a deeper look, we will found that most chars are **lowercase chars, `.`, `$` and `_`**, which can be expressed in a much +smaller range **`0~32`**. But one byte can represent range `0~255`, the significant bits are wasted, and this cost is not ignorable. In a dynamic serialization framework, such meta will take considerable cost compared to real data. -So we proposed a new string encoding algorithm which we called **meta string encoding**. It will encode most chars using less bits instead of `8` bits in utf-8 encoding. +So we proposed a new string encoding algorithm which we called **meta string encoding** in Fury. It will encode most chars using `5` bits instead of `8` bits in utf-8 encoding, which can bring **37.5% space cost savings** compared to utf-8 encoding. ## Meta String Introduction @@ -36,9 +36,10 @@ String binary encoding algorithm: | LOWER_SPECIAL | `a-z._$\|`| every char is written using 5 bits, `a-z`: `0b0~0b11001`, `._$\|`: `0b11010~0b11101`, prepend one bit at the start to indicate whether strip last char since last byte may have 7 redundant bits(1 indicates strip last char) | | LOWER_UPPER_DIGIT_SPECIAL | `a-zA-Z0~9._` | every char is written using 6 bits, `a-z`: `0b0~0b11001`, `A-Z`: `0b11010~0b110011`, `0~9`: `0b110100~0b01`, `._`: `0b10~0b11`, prepend one bit at the start to indicate whether strip last char since last byte may have 7 redundant bits(1 indicates strip last char) | | UTF-8 | any chars | UTF-8 encoding | -If we use `LOWER_SPECIAL/LOWER_UPPER_DIGIT_SPECIAL`, we must add a strip last char flag in encoded data. This is because every char will be encoded using `5/6` bits, and the last char may have `1~7` bits which are unused by encoding, such bits may cause an extra char read, which we must strip off. -Encoding code snippet in java, see [`org.apache.fury.meta.MetaStringEncoder#encodeGeneric(char[], int)`](https://github.com/apache/incubator-fury/blob/93800888595065b2690fec093ab0cbfd6ac7dedc/java/fury-core/src/main/java/org/apache/fury/meta/MetaStringEncoder.java#L235) for more detailed: +If we use `LOWER_SPECIAL/LOWER_UPPER_DIGIT_SPECIAL`, we must add a strip last char flag in encoded data. This is because every char will be encoded using `5/6` bits, and the last char may have `1~7` bits which are unused by encoding, such bits may cause an extra char to be read, which we must strip off. + +Here is encoding code snippet in java, see [`org.apache.fury.meta.MetaStringEncoder#encodeGeneric(char[], int)`](https://github.com/apache/incubator-fury/blob/93800888595065b2690fec093ab0cbfd6ac7dedc/java/fury-core/src/main
Re: [PR] refine meta string blog [incubator-fury-site]
chaokunyang merged PR #121: URL: https://github.com/apache/incubator-fury-site/pull/121 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] add meta string blog [incubator-fury-site]
chaokunyang merged PR #120: URL: https://github.com/apache/incubator-fury-site/pull/120 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury-site) branch main updated: add meta string blog (#120)
This is an automated email from the ASF dual-hosted git repository. chaokunyang pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-fury-site.git The following commit(s) were added to refs/heads/main by this push: new 091fb42 add meta string blog (#120) 091fb42 is described below commit 091fb42928a47fa73e6704310c365b9c2472321c Author: Shawn Yang AuthorDate: Tue May 7 09:30:01 2024 +0800 add meta string blog (#120) --- ...tastring-space-efficient_encoding_for_string.md | 178 + 1 file changed, 178 insertions(+) diff --git a/blog/2024-05-06-metastring-space-efficient_encoding_for_string.md b/blog/2024-05-06-metastring-space-efficient_encoding_for_string.md new file mode 100644 index 000..8d75c52 --- /dev/null +++ b/blog/2024-05-06-metastring-space-efficient_encoding_for_string.md @@ -0,0 +1,178 @@ +--- +slug: fury +title: 'Meta String: A 37.5% space efficient string encoding than UTF-8 in Fury serialization' +authors: [chaokunyang] +tags: [fury] +--- + +## Background + +In rpc/serialization systems, we often need to send **`namespace/path/filename/fieldName/packageName/moduleName/className/enumValue`** between processes. + +Those strings are mostly ascii strings. In order to transfer between processes, we often encode such strings using utf-8 encodings. Such encoding +will take one byte for every char, which is not space efficient actually. + +If we take a deeper look, we will found that most chars are **lower chars plus `.`, `$` and `_`**, which can be expressed in a much +smaller range **`0~32`**, and one byte can represent range `0~255`, the significant bits are wasted. And the cost is not ignorable, in a dynamic serialization +framework, such meta will take considerable cost compared to real data. + +So we proposed a new string encoding algorithm which we called **meta string encoding**. It will encode most chars using less bits instead of `8` bits in utf-8 encoding. + +## Meta String Introduction + +Meta string encoding algorithm is mainly used to encode meta strings such as **field names, namespace, packageName, className, path and filename**. +Such a string is enumerated and limited, so the encoding performance is not important since we can cache the encoding result. + +Meta string encoding uses `5/6` bits instead of `8` bits in utf-8 encoding for every chars. Since it uses less bits than utf8, it can bring +**37.5% space cost savings** compared to utf-8 and has a smaller encoded binary size, which uses less storage and makes the network transfer faster. + + +## Encoding Algorithms + +String binary encoding algorithm: + +| Algorithm | Pattern | Description | +|---|---|--| +| LOWER_SPECIAL | `a-z._$\|`| every char is written using 5 bits, `a-z`: `0b0~0b11001`, `._$\|`: `0b11010~0b11101`, prepend one bit at the start to indicate whether strip last char since last byte may have 7 redundant bits(1 indicates strip last char) | +| LOWER_UPPER_DIGIT_SPECIAL | `a-zA-Z0~9._` | every char is written using 6 bits, `a-z`: `0b0~0b11001`, `A-Z`: `0b11010~0b110011`, `0~9`: `0b110100~0b01`, `._`: `0b10~0b11`, prepend one bit at the start to indicate whether strip last char since last byte may have 7 redundant bits(1 indicates strip last char) | +| UTF-8 | any chars | UTF-8 encoding | +If we use `LOWER_SPECIAL/LOWER_UPPER_DIGIT_SPECIAL`, we must add a strip last char flag in encoded data. This is because every char will be encoded using `5/6` bits, and the last char may have `1~7` bits which are unused by encoding, such bits may cause an extra char read, which we must strip off. + +Encoding code snippet in java, see [`org.apache.fury.meta.MetaStringEncoder#encodeGeneric(char[], int)`](https://github.com/apache/incubator-fury/blob/93800888595065b2690fec093ab0cbfd6ac7dedc/java/fury-core/src/main/java/org/apache/fury/meta/MetaStringEncoder.java#L235) for more detailed: +```java +private byte[] encodeGeneric(char[] chars, int bitsPerCh
[PR] add meta string blog [incubator-fury-site]
chaokunyang opened a new pull request, #120: URL: https://github.com/apache/incubator-fury-site/pull/120 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] No type mapping from class java.lang.String to class java.lang.String [incubator-fury]
chaokunyang closed issue #1604: No type mapping from class java.lang.String to class java.lang.String URL: https://github.com/apache/incubator-fury/issues/1604 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] fix(java): fix TypeRef getSubType [incubator-fury]
chaokunyang merged PR #1608: URL: https://github.com/apache/incubator-fury/pull/1608 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury) branch main updated: fix(java): fix TypeRef getSubType (#1608)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury.git
The following commit(s) were added to refs/heads/main by this push:
new 72fdfd59 fix(java): fix TypeRef getSubType (#1608)
72fdfd59 is described below
commit 72fdfd5946b1e76a9708d25ea4d70db1433ca798
Author: Shawn Yang
AuthorDate: Tue May 7 00:29:36 2024 +0800
fix(java): fix TypeRef getSubType (#1608)
## What does this PR do?
fix TypeRef getSubType
## Related issues
Closes #1604
## Does this PR introduce any user-facing change?
- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?
## Benchmark
---
.../org/apache/fury/benchmark/state/JsonTest.java | 57 ++
.../main/java/org/apache/fury/reflect/TypeRef.java | 13 ++---
.../java/org/apache/fury/reflect/TypeRefTest.java | 42
3 files changed, 102 insertions(+), 10 deletions(-)
diff --git
a/java/benchmark/src/test/java/org/apache/fury/benchmark/state/JsonTest.java
b/java/benchmark/src/test/java/org/apache/fury/benchmark/state/JsonTest.java
new file mode 100644
index ..f96ea6a1
--- /dev/null
+++ b/java/benchmark/src/test/java/org/apache/fury/benchmark/state/JsonTest.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.fury.benchmark.state;
+
+import com.alibaba.fastjson2.JSONObject;
+import org.apache.fury.Fury;
+import org.apache.fury.config.CompatibleMode;
+import org.apache.fury.config.Language;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+public class JsonTest {
+ public static class DemoResponse {
+private JSONObject json;
+
+public DemoResponse(JSONObject json) {
+ this.json = json;
+}
+ }
+
+ @Test
+ public void testSerializeJson() {
+// For issue: https://github.com/apache/incubator-fury/issues/1604
+JSONObject jsonObject = new JSONObject();
+jsonObject.put("k1", "v1");
+jsonObject.put("k2", "v2");
+DemoResponse resp = new DemoResponse(jsonObject);
+Fury fury =
+Fury.builder()
+.withLanguage(Language.JAVA)
+.requireClassRegistration(false)
+.withRefTracking(true)
+.registerGuavaTypes(false)
+.withCompatibleMode(CompatibleMode.COMPATIBLE)
+.build();
+byte[] serialized = fury.serialize(resp);
+DemoResponse o = (DemoResponse) fury.deserialize(serialized);
+Assert.assertEquals(o.json, jsonObject);
+ }
+}
diff --git a/java/fury-core/src/main/java/org/apache/fury/reflect/TypeRef.java
b/java/fury-core/src/main/java/org/apache/fury/reflect/TypeRef.java
index 9a41fa13..b984bb32 100644
--- a/java/fury-core/src/main/java/org/apache/fury/reflect/TypeRef.java
+++ b/java/fury-core/src/main/java/org/apache/fury/reflect/TypeRef.java
@@ -498,15 +498,6 @@ public class TypeRef {
for (int i = 0; i < fromArgs.length; i++) {
populateTypeMappings(mappings, fromArgs[i], toArgs[i]);
}
-} else if (supertypeWithArgsFromSubtype instanceof Class) {
- if (toType instanceof WildcardType) {
-return; // Okay to say Foo is
- }
- // Can't map from a raw class to anything other than itself or a
wildcard.
- // You can't say "assuming String is Integer".
- // And we don't support "assuming String is T"; user has to say
"assuming T is String".
- throw new IllegalArgumentException(
- "No type mapping from " + supertypeWithArgsFromSubtype + " to " +
toType);
} else if (supertypeWithArgsFromSubtype instanceof GenericArrayType) {
if (toType instanceof WildcardType) {
return; // Okay to say A[] is
@@ -516,7 +507,9 @@ public class TypeRef {
((GenericArrayType)
supertypeWithArgsFromSubtype).getGenericComponentType();
populateTypeMappings(mappings, fromComponentType, componentType);
} else {
- throw new AssertionError("Unknown type: " + toType);
+ if (!(supertypeWithArgsFromSubtype instanceof Class)) {
+
Re: [I] [All] Clear serialization language flag [incubator-fury]
LiangliangSui commented on issue #1607: URL: https://github.com/apache/incubator-fury/issues/1607#issuecomment-2096397927 Yes, we can also keep it and add an explanation in xlang_spec. This will create an extra byte in the Fury Header. I'll update the spec later -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" is working again!
The GitHub Actions job "Fury CI" on incubator-fury.git has succeeded. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 22357d60409aca484d4afe71c8962b3ac9107352 / chaokunyang fix license header Report URL: https://github.com/apache/incubator-fury/actions/runs/8971950872 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] fix(java): fix TypeRef getSubType [incubator-fury]
Munoon commented on PR #1608: URL: https://github.com/apache/incubator-fury/pull/1608#issuecomment-2096345605 Hi, @chaokunyang. LGTM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury-site) branch deploy updated: deploy: 496044d788a0cf6e3440f5d1494dca42664a498b
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch deploy
in repository https://gitbox.apache.org/repos/asf/incubator-fury-site.git
The following commit(s) were added to refs/heads/deploy by this push:
new 65be28a deploy: 496044d788a0cf6e3440f5d1494dca42664a498b
65be28a is described below
commit 65be28a87f16b432d258c5db6530e4fe84a170ec
Author: chaokunyang
AuthorDate: Mon May 6 15:43:42 2024 +
deploy: 496044d788a0cf6e3440f5d1494dca42664a498b
---
404.html | 4 ++--
assets/js/18bd1185.a837fe3c.js| 1 -
assets/js/1e879fa7.ed696249.js| 1 +
assets/js/b2f554cd.3b60ec58.js| 1 -
assets/js/b2f554cd.6705ce25.js| 1 +
assets/js/{4b17afb5.c73ec818.js => b486c30c.e7f77dd8.js} | 2 +-
assets/js/main.4cbab5a4.js| 2 ++
...{main.a67b7167.js.LICENSE.txt => main.4cbab5a4.js.LICENSE.txt} | 0
assets/js/main.a67b7167.js| 2 --
assets/js/runtime~main.c6fe829c.js| 1 +
assets/js/runtime~main.f6f7d1b6.js| 1 -
blog/archive/index.html | 6 +++---
blog/atom.xml | 4 ++--
blog/fury_0_1_0_release/index.html| 4 ++--
blog/fury_0_1_1_release/index.html| 4 ++--
blog/fury_0_1_2_release/index.html| 4 ++--
blog/fury_0_2_0_release/index.html| 4 ++--
blog/fury_0_2_1_release/index.html| 4 ++--
blog/fury_0_3_0_release/index.html| 4 ++--
blog/fury_0_3_1_release/index.html| 4 ++--
blog/fury_0_4_0_release/index.html| 4 ++--
blog/fury_0_4_1_release/index.html| 4 ++--
blog/fury_0_5_0_release/index.html| 8
.../index.html| 4 ++--
blog/index.html | 6 +++---
blog/page/2/index.html| 4 ++--
blog/rss.xml | 4 ++--
blog/tags/fury/index.html | 6 +++---
blog/tags/fury/page/2/index.html | 4 ++--
blog/tags/index.html | 4 ++--
docs/guide/development/index.html | 4 ++--
docs/guide/graalvm_guide/index.html | 4 ++--
docs/guide/java_object_graph_guide/index.html | 4 ++--
docs/guide/row_format_guide/index.html| 4 ++--
docs/guide/scala_guide/index.html | 4 ++--
docs/guide/xlang_object_graph_guide/index.html| 4 ++--
docs/guide/xlang_type_mapping/index.html | 4 ++--
docs/introduction/benchmark/index.html| 4 ++--
docs/introduction/features/index.html | 4 ++--
docs/introduction/index.html | 4 ++--
docs/specification/fury_java_serialization_spec/index.html| 4 ++--
docs/specification/fury_row_format_spec/index.html| 4 ++--
docs/specification/fury_xlang_serialization_spec/index.html | 4 ++--
docs/start/install/index.html | 4 ++--
docs/start/usage/index.html | 4 ++--
download/index.html | 4 ++--
index.html| 4 ++--
lunr-index-1715007432436.json | 1 -
lunr-index-1715010191748.json | 1 +
lunr-index.json | 2 +-
markdown-page/index.html | 4 ++--
search-doc-1715007432436.json | 1 -
search-doc-1715010191748.json | 1 +
search-doc.json | 2 +-
54 files changed, 91 insertions(+), 91 deletions(-)
diff --git a/404.html b/404.html
index 68c48da..cd5690e 100644
--- a/404.html
+++ b/404.html
@@ -5,8 +5,8 @@
Page Not Found | Apache Fury (incubating)https://fury.apache.org/404.html";>
-
-
+
+
!function(){function
t(t){document.documentElement.setAttribute("data-theme",t)}var
e=function(){try{return new
URLSearchParams(window.location.sear
(incubator-fury-site) branch main updated: fix 0.5.0 release date (#119)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury-site.git
The following commit(s) were added to refs/heads/main by this push:
new 496044d fix 0.5.0 release date (#119)
496044d is described below
commit 496044d788a0cf6e3440f5d1494dca42664a498b
Author: Shawn Yang
AuthorDate: Mon May 6 23:42:40 2024 +0800
fix 0.5.0 release date (#119)
---
...{2024-05-13-fury_0_5_0_release.md => 2024-05-03-fury_0_5_0_release.md} | 0
1 file changed, 0 insertions(+), 0 deletions(-)
diff --git a/blog/2024-05-13-fury_0_5_0_release.md
b/blog/2024-05-03-fury_0_5_0_release.md
similarity index 100%
rename from blog/2024-05-13-fury_0_5_0_release.md
rename to blog/2024-05-03-fury_0_5_0_release.md
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] fix 0.5.0 release date [incubator-fury-site]
chaokunyang merged PR #119: URL: https://github.com/apache/incubator-fury-site/pull/119 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[PR] fix 0.5.0 release date [incubator-fury-site]
chaokunyang opened a new pull request, #119: URL: https://github.com/apache/incubator-fury-site/pull/119 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 06835dd3872abd0a761e66cc9819f35053c2942e / chaokunyang fix TypeRef getSubType Report URL: https://github.com/apache/incubator-fury/actions/runs/8971696227 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] fix(java): fix TypeRef getSubType [incubator-fury]
chaokunyang commented on PR #1608: URL: https://github.com/apache/incubator-fury/pull/1608#issuecomment-2096307662 Hi @Munoon , could you take a look at this PR? I can reproduce the error in #1606 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[PR] fix(java): fix TypeRef getSubType [incubator-fury]
chaokunyang opened a new pull request, #1608: URL: https://github.com/apache/incubator-fury/pull/1608 ## What does this PR do? fix TypeRef getSubType ## Related issues Closes #1604 ## Does this PR introduce any user-facing change? - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury-site) branch deploy updated: deploy: 831f16b2221720b76beb938998e23c486b9b1daa
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch deploy
in repository https://gitbox.apache.org/repos/asf/incubator-fury-site.git
The following commit(s) were added to refs/heads/deploy by this push:
new de9f45e deploy: 831f16b2221720b76beb938998e23c486b9b1daa
de9f45e is described below
commit de9f45eb4212e0762539ce083d091d3bc773a6f9
Author: chaokunyang
AuthorDate: Mon May 6 14:57:43 2024 +
deploy: 831f16b2221720b76beb938998e23c486b9b1daa
---
404.html | 4 +--
assets/js/72b486c3.4e7c1641.js | 1 +
assets/js/72b486c3.90b2843e.js | 1 -
assets/js/{main.fe563657.js => main.a67b7167.js} | 4 +--
...js.LICENSE.txt => main.a67b7167.js.LICENSE.txt} | 0
...e~main.00c45358.js => runtime~main.f6f7d1b6.js} | 2 +-
blog/archive/index.html| 4 +--
blog/fury_0_1_0_release/index.html | 4 +--
blog/fury_0_1_1_release/index.html | 4 +--
blog/fury_0_1_2_release/index.html | 4 +--
blog/fury_0_2_0_release/index.html | 4 +--
blog/fury_0_2_1_release/index.html | 4 +--
blog/fury_0_3_0_release/index.html | 4 +--
blog/fury_0_3_1_release/index.html | 4 +--
blog/fury_0_4_0_release/index.html | 4 +--
blog/fury_0_4_1_release/index.html | 4 +--
blog/fury_0_5_0_release/index.html | 4 +--
.../index.html | 4 +--
blog/index.html| 4 +--
blog/page/2/index.html | 4 +--
blog/tags/fury/index.html | 4 +--
blog/tags/fury/page/2/index.html | 4 +--
blog/tags/index.html | 4 +--
docs/guide/development/index.html | 4 +--
docs/guide/graalvm_guide/index.html| 4 +--
docs/guide/java_object_graph_guide/index.html | 4 +--
docs/guide/row_format_guide/index.html | 4 +--
docs/guide/scala_guide/index.html | 4 +--
docs/guide/xlang_object_graph_guide/index.html | 4 +--
docs/guide/xlang_type_mapping/index.html | 4 +--
docs/introduction/benchmark/index.html | 4 +--
docs/introduction/features/index.html | 4 +--
docs/introduction/index.html | 4 +--
.../fury_java_serialization_spec/index.html| 4 +--
docs/specification/fury_row_format_spec/index.html | 4 +--
.../fury_xlang_serialization_spec/index.html | 38 --
docs/start/install/index.html | 4 +--
docs/start/usage/index.html| 4 +--
download/index.html| 4 +--
index.html | 4 +--
lunr-index-1714962340768.json | 1 -
lunr-index-1715007432436.json | 1 +
lunr-index.json| 2 +-
markdown-page/index.html | 4 +--
search-doc-1714962340768.json | 1 -
search-doc-1715007432436.json | 1 +
search-doc.json| 2 +-
47 files changed, 99 insertions(+), 95 deletions(-)
diff --git a/404.html b/404.html
index 41ec908..68c48da 100644
--- a/404.html
+++ b/404.html
@@ -5,8 +5,8 @@
Page Not Found | Apache Fury (incubating)https://fury.apache.org/404.html";>
-
-
+
+
!function(){function
t(t){document.documentElement.setAttribute("data-theme",t)}var
e=function(){try{return new
URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}}()||function(){try{return
localStorage.getItem("theme")}catch(t){}}();t(null!==e?e:"light")}(),function(){try{const
c=new URLSearchParams(window.location.search).entries();for(var[t,e]of
c)if(t.startsWith("docusaurus-data-")){var
a=t.replace("docusaurus-data-","data-");document.documentElement.se [...]
diff --git a/assets/js/72b486c3.4e7c1641.js b/assets/js/72b486c3.4e7c1641.js
new file mode 100644
index 000..de7e94b
--- /dev/null
+++ b/assets/js/72b486c3.4e7c1641.js
@@ -0,0 +1 @@
+"use
strict";(self.webpackChunkfury_site=self.webpackChunkfury_site||[]).push([[2452],{1196:(e,n,i)=>{i.r(n),i.d(n,{assets:()=>d,contentTitle:()=>l,default:()=>h,frontMatter:()=>r,metadata:()=>a,toc:()=>c});var
t=i(5893),s=i(1151);const r={title:"Fury Xlang Serialization
Format",sidebar_position:0,id:"fury_xlang_serialization_spec"},l="Cross-language
Serialization
Specification",a={id:"specification/fury_xlang_serialization_spec",title:"Fury
Xlang Serialization Format",description:"Forma [...]
\ No newline at end of file
diff --git a/assets/js/72b486c3.90b2843e.js b/assets/js/72b486c3.90b2843e.js
deleted file mode 100644
index 698eea0..000
--- a/ass
Re: [I] [All] Clear serialization language flag [incubator-fury]
LiangliangSui commented on issue #1607: URL: https://github.com/apache/incubator-fury/issues/1607#issuecomment-2096236881 cc @chaokunyang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[I] [All] Clear serialization language flag [incubator-fury]
LiangliangSui opened a new issue, #1607: URL: https://github.com/apache/incubator-fury/issues/1607 ## Is your feature request related to a problem? Please describe. The current Fury implementation (Java/golang/javascript, etc.), the language used for serialization will be added to the Fury Header(Contradicts the current xlang_spec). Fury Header looks like the following format. ``` |2 bytes | 4 bits| 1 bit | 1 bit | 1 bit | 1 bit | 1 byte | optional 4 bytes | +--+---+---+---++---+--+-+ | magic number | reserved bits | oob | xlang | endian | null | language | unsigned int for meta start offset | ``` code show as below   ## Describe the solution you'd like Do we need to remove these redundant language identifiers? ## Additional context -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury-site) branch main updated: 🔄 synced local 'docs/specification/' with remote 'docs/specification/'
This is an automated email from the ASF dual-hosted git repository. chaokunyang pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-fury-site.git The following commit(s) were added to refs/heads/main by this push: new 831f16b 🔄 synced local 'docs/specification/' with remote 'docs/specification/' 831f16b is described below commit 831f16b2221720b76beb938998e23c486b9b1daa Author: chaokunyang AuthorDate: Mon May 6 14:56:41 2024 + 🔄 synced local 'docs/specification/' with remote 'docs/specification/' --- docs/specification/xlang_serialization_spec.md | 34 ++ 1 file changed, 19 insertions(+), 15 deletions(-) diff --git a/docs/specification/xlang_serialization_spec.md b/docs/specification/xlang_serialization_spec.md index 0583e7e..7b46b55 100644 --- a/docs/specification/xlang_serialization_spec.md +++ b/docs/specification/xlang_serialization_spec.md @@ -288,23 +288,27 @@ Meta header is a 64 bits number value encoded in little endian order. fields, then use fields info in meta for deserializing compatible fields. - type id: the registered id for the current type, which will be written as an unsigned varint. - field info: -- Header(8 bits): -- Format: -- `reserved 1 bit + 3 bits field name encoding + polymorphism flag + nullability flag + ref tracking flag + tag id flag`. -- Users can use annotation to provide that info. -- tag id: when set to 1, the field name will be written by an unsigned varint tag id. -- ref tracking: when set to 0, ref tracking will be disabled for this field. -- nullability: when set to 0, this field won't be null. -- polymorphism: when set to 1, the actual type of field will be the declared field type even the type if - not `final`. -- 3 bits field name encoding will be set to meta string encoding flags when tag id is not set. -- Type id: +- header(8 + bits): `3 bits size + 2 bits field name encoding + polymorphism flag + nullability flag + ref tracking flag`. + Users can use annotation to provide those info. +- 2 bits field name encoding: +- encoding: `UTF8/ALL_TO_LOWER_SPECIAL/LOWER_UPPER_DIGIT_SPECIAL/TAG_ID` +- If tag id is used, i.e. field name is written by an unsigned varint tag id. 2 bits encoding will be `11`. +- size of field name: +- The `3 bits size: 0~7` will be used to indicate length `1~7`, the value `7` indicates to read more bytes, + the encoding will encode `size - 7` as a varint next. +- If encoding is `TAG_ID`, then num_bytes of field name will be used to store tag id. +- ref tracking: when set to 1, ref tracking will be enabled for this field. +- nullability: when set to 1, this field can be null. +- polymorphism: when set to 1, the actual type of field will be the declared field type even the type if + not `final`. +- field name: If tag id is set, tag id will be used instead. Otherwise meta string encoding `[length]` and data will + be written instead. +- type id: - For registered type-consistent classes, it will be the registered type id. - Otherwise it will be encoded as `OBJECT_ID` if it isn't `final` and `FINAL_OBJECT_ID` if it's `final`. The meta for such types is written separately instead of inlining here is to reduce meta space cost if object of - this type is serialized in the current object graph multiple times, and the field value may be null too. -- Field name: If tag id is set, tag id will be used instead. Otherwise meta string encoding length and data will - be written instead. + this type is serialized in current object graph multiple times, and the field value may be null too. Field order are left as implementation details, which is not exposed to specification, the deserialization need to resort fields based on Fury field comparator. In this way, fury can compute statistics for field names or types and @@ -473,7 +477,7 @@ which will be encoded by elements header, each use one bit: By default, all bits are unset, which means all elements won't track ref, all elements are same type, not null and the actual element is the declared type in the custom type field. -The implementation can generate different deserialization code based read header, and look up the generated code from +The implementation can generate different deserialization code based read header, and look up the generated code from a linear map/list. elements data - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(sepc): update type meta field info spec [incubator-fury]
chaokunyang merged PR #1603: URL: https://github.com/apache/incubator-fury/pull/1603 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury) branch main updated: feat(sepc): update type meta field info spec (#1603)
This is an automated email from the ASF dual-hosted git repository. chaokunyang pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-fury.git The following commit(s) were added to refs/heads/main by this push: new 93800888 feat(sepc): update type meta field info spec (#1603) 93800888 is described below commit 93800888595065b2690fec093ab0cbfd6ac7dedc Author: Shawn Yang AuthorDate: Mon May 6 22:56:27 2024 +0800 feat(sepc): update type meta field info spec (#1603) ## What does this PR do? Update type meta field info spec: ``` - field info: - header(8 bits): `3 bits size + 2 bits field name encoding + polymorphism flag + nullability flag + ref tracking flag`. Users can use annotation to provide those info. - 2 bits field name encoding: - encoding: `UTF8/ALL_TO_LOWER_SPECIAL/LOWER_UPPER_DIGIT_SPECIAL/TAG_ID` - If tag id is used, i.e. field name is written by an unsigned varint tag id. 2 bits encoding will be `11`. - size of field name: - The `3 bits size: 0~7` will be used to indicate length `1~7`, the value `7` indicates to read more bytes, the encoding will encode `size - 7` as a varint next. - If encoding is `TAG_ID`, then num_bytes of field name will be used to store tag id. - ref tracking: when set to 1, ref tracking will be enabled for this field. - nullability: when set to 1, this field can be null. - polymorphism: when set to 1, the actual type of field will be the declared field type even the type if not `final`. - field name: If tag id is set, tag id will be used instead. Otherwise meta string encoding `[length]` and data will be written instead. ``` ## Related issues #1556 ## Does this PR introduce any user-facing change? - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark --- docs/specification/xlang_serialization_spec.md | 34 ++ 1 file changed, 19 insertions(+), 15 deletions(-) diff --git a/docs/specification/xlang_serialization_spec.md b/docs/specification/xlang_serialization_spec.md index 0583e7e0..7b46b556 100644 --- a/docs/specification/xlang_serialization_spec.md +++ b/docs/specification/xlang_serialization_spec.md @@ -288,23 +288,27 @@ Meta header is a 64 bits number value encoded in little endian order. fields, then use fields info in meta for deserializing compatible fields. - type id: the registered id for the current type, which will be written as an unsigned varint. - field info: -- Header(8 bits): -- Format: -- `reserved 1 bit + 3 bits field name encoding + polymorphism flag + nullability flag + ref tracking flag + tag id flag`. -- Users can use annotation to provide that info. -- tag id: when set to 1, the field name will be written by an unsigned varint tag id. -- ref tracking: when set to 0, ref tracking will be disabled for this field. -- nullability: when set to 0, this field won't be null. -- polymorphism: when set to 1, the actual type of field will be the declared field type even the type if - not `final`. -- 3 bits field name encoding will be set to meta string encoding flags when tag id is not set. -- Type id: +- header(8 + bits): `3 bits size + 2 bits field name encoding + polymorphism flag + nullability flag + ref tracking flag`. + Users can use annotation to provide those info. +- 2 bits field name encoding: +- encoding: `UTF8/ALL_TO_LOWER_SPECIAL/LOWER_UPPER_DIGIT_SPECIAL/TAG_ID` +- If tag id is used, i.e. field name is written by an unsigned varint tag id. 2 bits encoding will be `11`. +- size of field name: +- The `3 bits size: 0~7` will be used to indicate length `1~7`, the value `7` indicates to read more bytes, + the encoding will encode `size - 7` as a varint next. +- If encoding is `TAG_ID`, then num_bytes of field name will be used to store tag id. +- ref tracking: when set to 1, ref tracking will be enabled for this field. +- nullability: when set to 1, this field can be null. +- polymorphism: when set to 1, the actual type of field will be the declared field type even the type if + not `final`. +- field name: If tag id is set, tag id will be used instead. Otherwise meta string encoding `[length]` and data will + be written instead. +- type id: - For registered type-consistent classes, it will be the registered type id. - Otherwise it will be encoded as `OBJECT_ID` if it isn't `final` and `FINAL_OBJECT_ID` i
Re: [PR] feat(sepc): update type meta field info spec [incubator-fury]
LiangliangSui commented on code in PR #1603: URL: https://github.com/apache/incubator-fury/pull/1603#discussion_r1591136077 ## docs/specification/xlang_serialization_spec.md: ## @@ -288,23 +288,27 @@ Meta header is a 64 bits number value encoded in little endian order. fields, then use fields info in meta for deserializing compatible fields. - type id: the registered id for the current type, which will be written as an unsigned varint. - field info: -- Header(8 bits): -- Format: -- `reserved 1 bit + 3 bits field name encoding + polymorphism flag + nullability flag + ref tracking flag + tag id flag`. -- Users can use annotation to provide that info. -- tag id: when set to 1, the field name will be written by an unsigned varint tag id. -- ref tracking: when set to 0, ref tracking will be disabled for this field. -- nullability: when set to 0, this field won't be null. -- polymorphism: when set to 1, the actual type of field will be the declared field type even the type if - not `final`. -- 3 bits field name encoding will be set to meta string encoding flags when tag id is not set. -- Type id: +- header(8 + bits): `3 bits size + 2 bits field name encoding + polymorphism flag + nullability flag + ref tracking flag`. + Users can use annotation to provide those info. +- 2 bits field name encoding: +- encoding: `UTF8/ALL_TO_LOWER_SPECIAL/LOWER_UPPER_DIGIT_SPECIAL/TAG_ID` +- If tag id is used, i.e. field name is written by an unsigned varint tag id. 2 bits encoding will be `11`. +- size of field name: +- The `3 bits size: 0~7` will be used to indicate length `1~7`, the value `7` indicates to read more bytes, + the encoding will encode `size - 7` as a varint next. Review Comment: Got it, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury) branch main updated: fix(java): clear serializer for collection/map (#1606)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury.git
The following commit(s) were added to refs/heads/main by this push:
new 7d912ca1 fix(java): clear serializer for collection/map (#1606)
7d912ca1 is described below
commit 7d912ca161d7e8f75176ce4d25fdcf2aac61a74a
Author: Shawn Yang
AuthorDate: Mon May 6 22:54:33 2024 +0800
fix(java): clear serializer for collection/map (#1606)
## What does this PR do?
Some collectionSerializer may overwrite write/read method, then clear
element serializer may not got invoked.
This PR clears serializer for collection/map to avoid container use
wrong serializer for nested elements.
## Related issues
#1558
https://github.com/apache/incubator-fury/issues/1455,
https://github.com/apache/incubator-fury/issues/1325 and
https://github.com/apache/incubator-fury/issues/1176.
## Does this PR introduce any user-facing change?
- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?
## Benchmark
---
.../org/apache/fury/resolver/FieldResolver.java| 91 +-
.../fury/serializer/CompatibleSerializer.java | 43 +++---
2 files changed, 107 insertions(+), 27 deletions(-)
diff --git
a/java/fury-core/src/main/java/org/apache/fury/resolver/FieldResolver.java
b/java/fury-core/src/main/java/org/apache/fury/resolver/FieldResolver.java
index b7651179..047a2af1 100644
--- a/java/fury-core/src/main/java/org/apache/fury/resolver/FieldResolver.java
+++ b/java/fury-core/src/main/java/org/apache/fury/resolver/FieldResolver.java
@@ -568,29 +568,54 @@ public class FieldResolver {
ClassInfo elementClassInfo = classResolver.readClassInfo(buffer,
classInfoHolder);
ClassInfo classInfo = classResolver.readClassInfo(buffer,
classInfoHolder);
CollectionSerializer collectionSerializer = (CollectionSerializer)
classInfo.getSerializer();
-
collectionSerializer.setElementSerializer(elementClassInfo.getSerializer());
- o = collectionSerializer.read(buffer);
+ try {
+
collectionSerializer.setElementSerializer(elementClassInfo.getSerializer());
+o = collectionSerializer.read(buffer);
+ } finally {
+// Some collectionSerializer may overwrite write/read method, then
clear element serializer
+// may not got invoked.
+collectionSerializer.setElementSerializer(null);
+ }
} else if (fieldType == FieldTypes.MAP_KV_FINAL) {
ClassInfo keyClassInfo = classResolver.readClassInfo(buffer,
classInfoHolder);
ClassInfo valueClassInfo = classResolver.readClassInfo(buffer,
classInfoHolder);
ClassInfo classInfo = classResolver.readClassInfo(buffer,
classInfoHolder);
MapSerializer mapSerializer = (MapSerializer) classInfo.getSerializer();
- mapSerializer.setKeySerializer(keyClassInfo.getSerializer());
- mapSerializer.setValueSerializer(valueClassInfo.getSerializer());
- o = mapSerializer.read(buffer);
+ try {
+mapSerializer.setKeySerializer(keyClassInfo.getSerializer());
+mapSerializer.setValueSerializer(valueClassInfo.getSerializer());
+o = mapSerializer.read(buffer);
+ } finally {
+// Some mmapSerializer may overwrite write/read method, then clear
serializer
+// may not got invoked.
+mapSerializer.setKeySerializer(null);
+mapSerializer.setValueSerializer(null);
+ }
} else if (fieldType == FieldTypes.MAP_KEY_FINAL) {
ClassInfo keyClassInfo = classResolver.readClassInfo(buffer,
classInfoHolder);
ClassInfo classInfo = classResolver.readClassInfo(buffer,
classInfoHolder);
MapSerializer mapSerializer = (MapSerializer) classInfo.getSerializer();
- mapSerializer.setKeySerializer(keyClassInfo.getSerializer());
- o = mapSerializer.read(buffer);
+ try {
+mapSerializer.setKeySerializer(keyClassInfo.getSerializer());
+o = mapSerializer.read(buffer);
+ } finally {
+// Some mmapSerializer may overwrite write/read method, then clear
serializer
+// may not got invoked.
+mapSerializer.setKeySerializer(null);
+ }
} else {
Preconditions.checkArgument(fieldType == FieldTypes.MAP_VALUE_FINAL);
ClassInfo valueClassInfo = classResolver.readClassInfo(buffer,
classInfoHolder);
ClassInfo classInfo = classResolver.readClassInfo(buffer,
classInfoHolder);
MapSerializer mapSerializer = (MapSerializer) classInfo.getSerializer();
- mapSerializer.setValueSerializer(valueClassInfo.getSerializer());
- o = mapSerializer.read(buffer);
+ try {
+mapSerializer.setValueSerializer(valueClassInfo.getSerializer());
+o = mapSerializer.read(buffer);
+ } fina
Re: [PR] fix(java): clear serializer for collection/map [incubator-fury]
chaokunyang merged PR #1606: URL: https://github.com/apache/incubator-fury/pull/1606 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(sepc): update type meta field info spec [incubator-fury]
chaokunyang commented on code in PR #1603: URL: https://github.com/apache/incubator-fury/pull/1603#discussion_r1591128731 ## docs/specification/xlang_serialization_spec.md: ## @@ -288,23 +288,27 @@ Meta header is a 64 bits number value encoded in little endian order. fields, then use fields info in meta for deserializing compatible fields. - type id: the registered id for the current type, which will be written as an unsigned varint. - field info: -- Header(8 bits): -- Format: -- `reserved 1 bit + 3 bits field name encoding + polymorphism flag + nullability flag + ref tracking flag + tag id flag`. -- Users can use annotation to provide that info. -- tag id: when set to 1, the field name will be written by an unsigned varint tag id. -- ref tracking: when set to 0, ref tracking will be disabled for this field. -- nullability: when set to 0, this field won't be null. -- polymorphism: when set to 1, the actual type of field will be the declared field type even the type if - not `final`. -- 3 bits field name encoding will be set to meta string encoding flags when tag id is not set. -- Type id: +- header(8 + bits): `3 bits size + 2 bits field name encoding + polymorphism flag + nullability flag + ref tracking flag`. + Users can use annotation to provide those info. +- 2 bits field name encoding: +- encoding: `UTF8/ALL_TO_LOWER_SPECIAL/LOWER_UPPER_DIGIT_SPECIAL/TAG_ID` +- If tag id is used, i.e. field name is written by an unsigned varint tag id. 2 bits encoding will be `11`. +- size of field name: +- The `3 bits size: 0~7` will be used to indicate length `1~7`, the value `7` indicates to read more bytes, + the encoding will encode `size - 7` as a varint next. Review Comment: Yes, that's what it means -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(sepc): update type meta field info spec [incubator-fury]
LiangliangSui commented on code in PR #1603: URL: https://github.com/apache/incubator-fury/pull/1603#discussion_r1591107937 ## docs/specification/xlang_serialization_spec.md: ## @@ -288,23 +288,27 @@ Meta header is a 64 bits number value encoded in little endian order. fields, then use fields info in meta for deserializing compatible fields. - type id: the registered id for the current type, which will be written as an unsigned varint. - field info: -- Header(8 bits): -- Format: -- `reserved 1 bit + 3 bits field name encoding + polymorphism flag + nullability flag + ref tracking flag + tag id flag`. -- Users can use annotation to provide that info. -- tag id: when set to 1, the field name will be written by an unsigned varint tag id. -- ref tracking: when set to 0, ref tracking will be disabled for this field. -- nullability: when set to 0, this field won't be null. -- polymorphism: when set to 1, the actual type of field will be the declared field type even the type if - not `final`. -- 3 bits field name encoding will be set to meta string encoding flags when tag id is not set. -- Type id: +- header(8 + bits): `3 bits size + 2 bits field name encoding + polymorphism flag + nullability flag + ref tracking flag`. + Users can use annotation to provide those info. +- 2 bits field name encoding: +- encoding: `UTF8/ALL_TO_LOWER_SPECIAL/LOWER_UPPER_DIGIT_SPECIAL/TAG_ID` +- If tag id is used, i.e. field name is written by an unsigned varint tag id. 2 bits encoding will be `11`. +- size of field name: +- The `3 bits size: 0~7` will be used to indicate length `1~7`, the value `7` indicates to read more bytes, + the encoding will encode `size - 7` as a varint next. Review Comment: So `0(0b000)~6(0b110)` means `1~7`, and `7 (0b111)` means there are more bytes. Can I understand it this way? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[PR] fix(java): clear serializer for collection/map [incubator-fury]
chaokunyang opened a new pull request, #1606: URL: https://github.com/apache/incubator-fury/pull/1606 ## What does this PR do? Some collectionSerializer may overwrite write/read method, then clear element serializer may not got invoked. This PRclear serializer for collection/map to avoid container use wrong serializer for nested elements. ## Related issues #1558 https://github.com/apache/incubator-fury/issues/1455, https://github.com/apache/incubator-fury/issues/1325 and https://github.com/apache/incubator-fury/issues/1176. ## Does this PR introduce any user-facing change? - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(sepc): update type meta field info spec [incubator-fury]
chaokunyang commented on code in PR #1603: URL: https://github.com/apache/incubator-fury/pull/1603#discussion_r1591044192 ## docs/specification/xlang_serialization_spec.md: ## @@ -288,23 +288,27 @@ Meta header is a 64 bits number value encoded in little endian order. fields, then use fields info in meta for deserializing compatible fields. - type id: the registered id for the current type, which will be written as an unsigned varint. - field info: -- Header(8 bits): -- Format: -- `reserved 1 bit + 3 bits field name encoding + polymorphism flag + nullability flag + ref tracking flag + tag id flag`. -- Users can use annotation to provide that info. -- tag id: when set to 1, the field name will be written by an unsigned varint tag id. -- ref tracking: when set to 0, ref tracking will be disabled for this field. -- nullability: when set to 0, this field won't be null. -- polymorphism: when set to 1, the actual type of field will be the declared field type even the type if - not `final`. -- 3 bits field name encoding will be set to meta string encoding flags when tag id is not set. -- Type id: +- header(8 + bits): `3 bits size + 2 bits field name encoding + polymorphism flag + nullability flag + ref tracking flag`. + Users can use annotation to provide those info. +- 2 bits field name encoding: +- encoding: `UTF8/ALL_TO_LOWER_SPECIAL/LOWER_UPPER_DIGIT_SPECIAL/TAG_ID` +- If tag id is used, i.e. field name is written by an unsigned varint tag id. 2 bits encoding will be `11`. +- size of field name: +- The `3 bits size: 0~7` will be used to indicate length `1~7`, the value `7` indicates to read more bytes, + the encoding will encode `size - 7` as a varint next. Review Comment: The field name won't be empty, so value `0` can be used to indicate length `1`. In this way, we can have wider range -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(sepc): update type meta field info spec [incubator-fury]
LiangliangSui commented on code in PR #1603: URL: https://github.com/apache/incubator-fury/pull/1603#discussion_r1591007924 ## docs/specification/xlang_serialization_spec.md: ## @@ -288,23 +288,27 @@ Meta header is a 64 bits number value encoded in little endian order. fields, then use fields info in meta for deserializing compatible fields. - type id: the registered id for the current type, which will be written as an unsigned varint. - field info: -- Header(8 bits): -- Format: -- `reserved 1 bit + 3 bits field name encoding + polymorphism flag + nullability flag + ref tracking flag + tag id flag`. -- Users can use annotation to provide that info. -- tag id: when set to 1, the field name will be written by an unsigned varint tag id. -- ref tracking: when set to 0, ref tracking will be disabled for this field. -- nullability: when set to 0, this field won't be null. -- polymorphism: when set to 1, the actual type of field will be the declared field type even the type if - not `final`. -- 3 bits field name encoding will be set to meta string encoding flags when tag id is not set. -- Type id: +- header(8 + bits): `3 bits size + 2 bits field name encoding + polymorphism flag + nullability flag + ref tracking flag`. + Users can use annotation to provide those info. +- 2 bits field name encoding: +- encoding: `UTF8/ALL_TO_LOWER_SPECIAL/LOWER_UPPER_DIGIT_SPECIAL/TAG_ID` +- If tag id is used, i.e. field name is written by an unsigned varint tag id. 2 bits encoding will be `11`. +- size of field name: +- The `3 bits size: 0~7` will be used to indicate length `1~7`, the value `7` indicates to read more bytes, + the encoding will encode `size - 7` as a varint next. Review Comment: > The `3 bits size: 0~7` will be used to indicate length `1~7`, the value `7` indicates to read more bytes, Do we give up using 0? I don't understand this paragraph very thoroughly. Could you help explain it in depth? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
