svn commit: r68815 - in /dev/incubator/fury/0.5.0-rc4: ./ apache-fury-0.5.0-incubating-src.tar.gz apache-fury-0.5.0-incubating-src.tar.gz.asc apache-fury-0.5.0-incubating-src.tar.gz.sha512
Author: chaokunyang Date: Sat Apr 27 06:52:28 2024 New Revision: 68815 Log: Prepare for fury 0.5.0-rc4 Added: dev/incubator/fury/0.5.0-rc4/ dev/incubator/fury/0.5.0-rc4/apache-fury-0.5.0-incubating-src.tar.gz (with props) dev/incubator/fury/0.5.0-rc4/apache-fury-0.5.0-incubating-src.tar.gz.asc dev/incubator/fury/0.5.0-rc4/apache-fury-0.5.0-incubating-src.tar.gz.sha512 Added: dev/incubator/fury/0.5.0-rc4/apache-fury-0.5.0-incubating-src.tar.gz == Binary file - no diff available. Propchange: dev/incubator/fury/0.5.0-rc4/apache-fury-0.5.0-incubating-src.tar.gz -- svn:mime-type = application/octet-stream Added: dev/incubator/fury/0.5.0-rc4/apache-fury-0.5.0-incubating-src.tar.gz.asc == --- dev/incubator/fury/0.5.0-rc4/apache-fury-0.5.0-incubating-src.tar.gz.asc (added) +++ dev/incubator/fury/0.5.0-rc4/apache-fury-0.5.0-incubating-src.tar.gz.asc Sat Apr 27 06:52:28 2024 @@ -0,0 +1,16 @@ +-BEGIN PGP SIGNATURE- + +iQIzBAABCAAdFiEEHiza5MCK19aU0csTnXvo5F5YC6QFAmYsnxwACgkQnXvo5F5Y +C6Q/YA/+KNJ3/3QfJ/GysudnASyUttU9lDMBN6op3khPB2aPuS9inNVAgUN08Sl5 +1rB9+eI2i/bxMBkLjb5cPmGtnWSDjK52GGr9kXkzg+O8Ew+xYu+PbuNA6WQ1dZO/ +RfYBgwl0dVhAGFUJPnbAWXwFppZKI0+ISqYmQvvpWC6aYNm0tfztOks1SU2GUido +P1/Z27senDXOsAfFsLhugi3vpPKnwb24OdktU/VBuPjUhE8AUSiBFnkzkF9bnz+V +0c4t/xpe/Dm7pX1hG0HeLsGprjEsiZEh+EHQe/GeFh6QAK6tTC+7fJp34e9F3gWR +tpwSsR081Q9gO5rDIXQ3TXJB/92EMhd8aec8RN9SlrdtWY/1wSGSSCj31esHDzhv +RsZGGYcswIwRCIc1UVIUekWy5kR2vSFep30NJiPbZzH6MscjHoFLMaVet+zmq43S +dfaXSwCQ3Uj7Zg8eFLEPIn11WuqIAzRh+KUKncU3ewsg6OSp17ZBA2tUGDGnWd+U +F5tt8Y0OUxO/GZ6zfuNLG+oE25h+npN9TGfxhlIy02esWEyrsyMOn5gZJ3onoBt/ +dPbTl88Xkc/9MTb42eXuMmdyhMOyjlduOhlHUhvyU4MM0RUDO60mtTZ2cA2S7vdl +QUmiCA6bOAkn0kVNtsBSXwhDzkEgV+GORe2EXc7Q2+VOplR7S5Y= +=wEwd +-END PGP SIGNATURE- Added: dev/incubator/fury/0.5.0-rc4/apache-fury-0.5.0-incubating-src.tar.gz.sha512 == --- dev/incubator/fury/0.5.0-rc4/apache-fury-0.5.0-incubating-src.tar.gz.sha512 (added) +++ dev/incubator/fury/0.5.0-rc4/apache-fury-0.5.0-incubating-src.tar.gz.sha512 Sat Apr 27 06:52:28 2024 @@ -0,0 +1 @@ +638b4ab603105934148fcaf67db0e2cb7fbc6daae7ed71ec70cfbe9f05081bfea118a968fcf6f289157c7f71d492fd2cf86c0c7e1477623bdb0b032edb5add8b apache-fury-0.5.0-incubating-src.tar.gz - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] chore: prepare release for 0.5.0 rc4 [incubator-fury]
chaokunyang merged PR #1589: URL: https://github.com/apache/incubator-fury/pull/1589 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" is working again!
The GitHub Actions job "Fury CI" on incubator-fury.git has succeeded. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 8d6aae56b10f8354a93a9937b00c9d4a70fbaedb / chaokunyang fix merge Report URL: https://github.com/apache/incubator-fury/actions/runs/8857930748 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: bbd9d91a04a629807cf0509afcb02d58e5326e41 / chaokunyang fix platform util pkg Report URL: https://github.com/apache/incubator-fury/actions/runs/8857921486 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 860e3da308bd6477e4e096b124bc9b83cbb61023 / chaokunyang Merge remote-tracking branch 'ant/main' into prepare_0.5.0_rc4 Report URL: https://github.com/apache/incubator-fury/actions/runs/8857763052 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[PR] chore: prepare 0.5.0 rc4 [incubator-fury]
chaokunyang opened a new pull request, #1589: URL: https://github.com/apache/incubator-fury/pull/1589 ## What does this PR do? This PR sync latest main into release 0.5.0 branch ## Related issues ## Does this PR introduce any user-facing change? - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[I] Remove commented content in git commit message [incubator-fury]
chaokunyang opened a new issue, #1588: URL: https://github.com/apache/incubator-fury/issues/1588 ## Is your feature request related to a problem? Please describe. Currently there are many comments contents in PR. ``` ## What does this PR do? ## Related issues ## Does this PR introduce any user-facing change? - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark ``` If contributor doesn't remove it, it will be included in the commited message, which make the git history kinds of messy. ## Describe the solution you'd like We should remove such conents automatically before merge PR ## Additional context -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury) branch main updated: fix(java): meta string strip last char && check empty encoded string (#1587)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury.git
The following commit(s) were added to refs/heads/main by this push:
new 6ad2ca57 fix(java): meta string strip last char && check empty encoded
string (#1587)
6ad2ca57 is described below
commit 6ad2ca578630a105d2791f48f80488167340d29c
Author: qingoba <51332889+qing...@users.noreply.github.com>
AuthorDate: Sat Apr 27 13:45:48 2024 +0800
fix(java): meta string strip last char && check empty encoded string (#1587)
## What does this PR do?
Fix two bugs:
+ fix method `MetaString.stripLastChar()` logical error
+ add empty encoded bytes check in `MetaStringDecoder.decode()`
## Related issues
- #1565
## Does this PR introduce any user-facing change?
- [No] Does this PR introduce any public API change?
- [No] Does this PR introduce any binary protocol compatibility change?
---
.gitignore | 2 ++
.../main/java/org/apache/fury/meta/MetaString.java | 2 +-
.../org/apache/fury/meta/MetaStringDecoder.java| 3 +++
.../java/org/apache/fury/meta/MetaStringTest.java | 26 ++
4 files changed, 32 insertions(+), 1 deletion(-)
diff --git a/.gitignore b/.gitignore
index c438485d..53cfc24d 100644
--- a/.gitignore
+++ b/.gitignore
@@ -36,3 +36,5 @@ javascript/**/node_modules/
javascript/**/build
MODULE.bazel
MODULE.bazel.lock
+.DS_Store
+**/.DS_Store
diff --git a/java/fury-core/src/main/java/org/apache/fury/meta/MetaString.java
b/java/fury-core/src/main/java/org/apache/fury/meta/MetaString.java
index ec164048..3c2be21b 100644
--- a/java/fury-core/src/main/java/org/apache/fury/meta/MetaString.java
+++ b/java/fury-core/src/main/java/org/apache/fury/meta/MetaString.java
@@ -79,7 +79,7 @@ public class MetaString {
this.bytes = bytes;
if (encoding != Encoding.UTF_8) {
Preconditions.checkArgument(bytes.length > 0);
- this.stripLastChar = (bytes[0] & 0b1) != 0;
+ this.stripLastChar = (bytes[0] & 0x80) != 0;
} else {
this.stripLastChar = false;
}
diff --git
a/java/fury-core/src/main/java/org/apache/fury/meta/MetaStringDecoder.java
b/java/fury-core/src/main/java/org/apache/fury/meta/MetaStringDecoder.java
index 345663af..85df1086 100644
--- a/java/fury-core/src/main/java/org/apache/fury/meta/MetaStringDecoder.java
+++ b/java/fury-core/src/main/java/org/apache/fury/meta/MetaStringDecoder.java
@@ -48,6 +48,9 @@ public class MetaStringDecoder {
* @return Decoded string.
*/
public String decode(byte[] encodedData, Encoding encoding) {
+if (encodedData.length == 0) {
+ return "";
+}
switch (encoding) {
case LOWER_SPECIAL:
return decodeLowerSpecial(encodedData);
diff --git
a/java/fury-core/src/test/java/org/apache/fury/meta/MetaStringTest.java
b/java/fury-core/src/test/java/org/apache/fury/meta/MetaStringTest.java
index 4fa84d75..f85d5e15 100644
--- a/java/fury-core/src/test/java/org/apache/fury/meta/MetaStringTest.java
+++ b/java/fury-core/src/test/java/org/apache/fury/meta/MetaStringTest.java
@@ -20,7 +20,9 @@
package org.apache.fury.meta;
import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
import static org.testng.Assert.assertNotSame;
+import static org.testng.Assert.assertTrue;
import static org.testng.AssertJUnit.assertSame;
import org.apache.fury.util.StringUtils;
@@ -188,4 +190,28 @@ public class MetaStringTest {
decoder.decode(encodedMetaString.getBytes(),
encodedMetaString.getEncoding());
assertEquals(decodedString, testString);
}
+
+ @Test
+ public void testStripLastChar() {
+String testString = "abc"; // encoded as 1|0|00, 001|00010, exactly
two bytes
+MetaStringEncoder encoder = new MetaStringEncoder('_', '$');
+MetaString encodedMetaString = encoder.encode(testString);
+assertFalse(encodedMetaString.stripLastChar());
+
+testString =
+"abcde"; // encoded as 1|0|00, 001|00010, 00011|001, 00xx,
stripped last char
+encodedMetaString = encoder.encode(testString);
+assertTrue(encodedMetaString.stripLastChar());
+ }
+
+ @Test
+ public void testEmptyString() {
+MetaStringEncoder encoder = new MetaStringEncoder('_', '$');
+MetaString metaString = encoder.encode("");
+assertEquals(metaString.getBytes(), new byte[0]);
+
+MetaStringDecoder decoder = new MetaStringDecoder('_', '$');
+String decoded = decoder.decode(metaString.getBytes(),
metaString.getEncoding());
+assertEquals(decoded, "");
+ }
}
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] fix(java): meta string strip last char && check empty encoded string [incubator-fury]
chaokunyang merged PR #1587: URL: https://github.com/apache/incubator-fury/pull/1587 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" is working again!
The GitHub Actions job "Fury CI" on incubator-fury.git has succeeded. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 1885a7689f87fd1919ebd66cba191a03fcdfbbb2 / chaokunyang lint code Report URL: https://github.com/apache/incubator-fury/actions/runs/8857706987 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] deserialize fails with java.lang.NullPointerException [incubator-fury]
chaokunyang commented on issue #1455: URL: https://github.com/apache/incubator-fury/issues/1455#issuecomment-2080373092 Hi @andyczerwonka , could you reproduce this error locally now? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] deserialize fails with java.lang.NullPointerException [incubator-fury]
chaokunyang commented on issue #1455: URL: https://github.com/apache/incubator-fury/issues/1455#issuecomment-2080372923 > Can we improve the log system and insert some info and debug logs into the code? For this situation where reproducible code cannot be provided but is reproducible stably, users can dynamically configure the log level through ENV, and then upload the collected log, we can restore the problem through these logs. > > But I'm also worried that too many insert points in the code will cause performance degradation. > > WDYT? @chaokunyang The log will be in critical path. If the error happens occasionally, the users may won't enable such log level. If the error happens frequently, we may not need the log to restore the error -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 3ce7b4c683ba9f55fe29136f14518f537e7326f6 / chaokunyang Merge remote-tracking branch 'ant/main' into new_meta_encoding_in_java Report URL: https://github.com/apache/incubator-fury/actions/runs/8857660302 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] Sporadic java.lang.ArrayIndexOutOfBoundsException/Can't deserialize lambda if RuntimeException is deserialized [incubator-fury]
chaokunyang commented on issue #1558: URL: https://github.com/apache/incubator-fury/issues/1558#issuecomment-2080372088 Hi @johannschenkl . Is this issue still persisits? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" is working again!
The GitHub Actions job "Fury CI" on incubator-fury.git has succeeded. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 18feb0e99a1df258adbe3018fbdf4314cf5233c1 / Shawn Yang feat(java): revive soft reference (#1585) ## What does this PR do? This PR revives soft reference to try to make it has a longer lifetime ## Related issues ## Does this PR introduce any user-facing change? - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark Report URL: https://github.com/apache/incubator-fury/actions/runs/8857646772 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] fix(java): meta string strip last char && check empty encoded string [incubator-fury]
chaokunyang commented on PR #1587: URL: https://github.com/apache/incubator-fury/pull/1587#issuecomment-2080371445 Code lint error:  Could you use `mvn spotless:apply` to format java code? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user qingoba (triggered by qingoba). Head commit for run: 0b030a5de041ee9d530d8da5a336df62d5e0a261 / Yuqing Liu fix(java): meta string strip last char & check empty eecoded string Report URL: https://github.com/apache/incubator-fury/actions/runs/8857634107 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury) branch main updated: feat(java): revive soft reference (#1585)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury.git
The following commit(s) were added to refs/heads/main by this push:
new 18feb0e9 feat(java): revive soft reference (#1585)
18feb0e9 is described below
commit 18feb0e99a1df258adbe3018fbdf4314cf5233c1
Author: Shawn Yang
AuthorDate: Sat Apr 27 13:28:17 2024 +0800
feat(java): revive soft reference (#1585)
## What does this PR do?
This PR revives soft reference to try to make it has a longer lifetime
## Related issues
## Does this PR introduce any user-facing change?
- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?
## Benchmark
---
.../src/main/java/org/apache/fury/util/DelayedRef.java| 11 +--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/java/fury-core/src/main/java/org/apache/fury/util/DelayedRef.java
b/java/fury-core/src/main/java/org/apache/fury/util/DelayedRef.java
index c2254ace..ecb4585c 100644
--- a/java/fury-core/src/main/java/org/apache/fury/util/DelayedRef.java
+++ b/java/fury-core/src/main/java/org/apache/fury/util/DelayedRef.java
@@ -34,7 +34,7 @@ public class DelayedRef {
// If other components doesn't strong hold the referent, this is used
// to cache and get the referent. But the reference will be set to null
// when there is a memory pressure.
- private final SoftReference softRef;
+ private SoftReference softRef;
public DelayedRef(T o) {
weakRef = new WeakReference<>(o);
@@ -43,6 +43,13 @@ public class DelayedRef {
public T get() {
T t = weakRef.get();
-return t != null ? t : softRef.get();
+if (t != null) {
+ if (softRef.get() == null) {
+// may be null when a full gc happened.
+softRef = new SoftReference<>(t);
+ }
+ return t;
+}
+return softRef.get();
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(java): revive soft reference [incubator-fury]
chaokunyang merged PR #1585: URL: https://github.com/apache/incubator-fury/pull/1585 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] [Java] Cannot use `CompatibleMode.COMPATIBLE` with native-image when registering classes with `createSerializer: true` [incubator-fury]
chaokunyang closed issue #1555: [Java] Cannot use `CompatibleMode.COMPATIBLE` with native-image when registering classes with `createSerializer: true` URL: https://github.com/apache/incubator-fury/issues/1555 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury) branch main updated: feat(java): support compatible mode for graalvm (#1586)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury.git
The following commit(s) were added to refs/heads/main by this push:
new 007e8dfa feat(java): support compatible mode for graalvm (#1586)
007e8dfa is described below
commit 007e8dfa06c0ff615ad24d335d3ecaca0fedf662
Author: Shawn Yang
AuthorDate: Sat Apr 27 13:25:53 2024 +0800
feat(java): support compatible mode for graalvm (#1586)
## What does this PR do?
This PR supports compatible mode for graalvm by using `MethodHandle` to
geenrate accessor at build time instead of using `Unsafe`
## Related issues
Closes #1555
## Does this PR introduce any user-facing change?
- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?
## Benchmark
---
.../graalvm/{Main.java => CompatibleExample.java} | 28
...{Main.java => CompatibleThreadSafeExample.java} | 34 ++
.../main/java/org/apache/fury/graalvm/Example.java | 7 ++-
.../main/java/org/apache/fury/graalvm/Main.java| 4 ++
.../org/apache/fury/graalvm/ThreadSafeExample.java | 7 ++-
...rdExample.java => CompatibleRecordExample.java} | 24 --
.../apache/fury/graalvm/record/RecordExample.java | 7 ++-
.../apache/fury/graalvm/record/RecordExample2.java | 1 +
.../graalvm_tests/native-image.properties | 6 ++-
.../org/apache/fury/builder/AccessorHelper.java| 3 --
.../org/apache/fury/reflect/FieldAccessor.java | 53 +++---
.../org/apache/fury/util/function/Functions.java | 3 --
.../fury-core/native-image.properties | 11 +
.../org/apache/fury/reflect/FieldAccessorTest.java | 51 +
14 files changed, 180 insertions(+), 59 deletions(-)
diff --git
a/integration_tests/graalvm_tests/src/main/java/org/apache/fury/graalvm/Main.java
b/integration_tests/graalvm_tests/src/main/java/org/apache/fury/graalvm/CompatibleExample.java
similarity index 62%
copy from
integration_tests/graalvm_tests/src/main/java/org/apache/fury/graalvm/Main.java
copy to
integration_tests/graalvm_tests/src/main/java/org/apache/fury/graalvm/CompatibleExample.java
index d5b403ae..4a7100b7 100644
---
a/integration_tests/graalvm_tests/src/main/java/org/apache/fury/graalvm/Main.java
+++
b/integration_tests/graalvm_tests/src/main/java/org/apache/fury/graalvm/CompatibleExample.java
@@ -19,16 +19,24 @@
package org.apache.fury.graalvm;
-import org.apache.fury.graalvm.record.RecordExample;
-import org.apache.fury.graalvm.record.RecordExample2;
+import org.apache.fury.Fury;
+import org.apache.fury.config.CompatibleMode;
-public class Main {
- public static void main(String[] args) throws Throwable {
-Example.main(args);
-RecordExample.main(args);
-RecordExample2.main(args);
-ThreadSafeExample.main(args);
-ProxyExample.main(args);
-Benchmark.main(args);
+public class CompatibleExample {
+ static Fury fury;
+
+ static {
+fury =
+Fury.builder()
+.requireClassRegistration(true)
+.withCompatibleMode(CompatibleMode.COMPATIBLE)
+.build();
+// register and generate serializer code.
+fury.register(Foo.class, true);
+ }
+
+ public static void main(String[] args) {
+Example.test(fury);
+System.out.println("CompatibleExample succeed");
}
}
diff --git
a/integration_tests/graalvm_tests/src/main/java/org/apache/fury/graalvm/Main.java
b/integration_tests/graalvm_tests/src/main/java/org/apache/fury/graalvm/CompatibleThreadSafeExample.java
similarity index 52%
copy from
integration_tests/graalvm_tests/src/main/java/org/apache/fury/graalvm/Main.java
copy to
integration_tests/graalvm_tests/src/main/java/org/apache/fury/graalvm/CompatibleThreadSafeExample.java
index d5b403ae..9ddb4892 100644
---
a/integration_tests/graalvm_tests/src/main/java/org/apache/fury/graalvm/Main.java
+++
b/integration_tests/graalvm_tests/src/main/java/org/apache/fury/graalvm/CompatibleThreadSafeExample.java
@@ -19,16 +19,32 @@
package org.apache.fury.graalvm;
-import org.apache.fury.graalvm.record.RecordExample;
-import org.apache.fury.graalvm.record.RecordExample2;
+import org.apache.fury.Fury;
+import org.apache.fury.ThreadLocalFury;
+import org.apache.fury.ThreadSafeFury;
+import org.apache.fury.config.CompatibleMode;
+
+public class CompatibleThreadSafeExample {
+ static ThreadSafeFury fury;
+
+ static {
+fury =
+new ThreadLocalFury(
+classLoader -> {
+ Fury f =
+ Fury.builder()
+ .requireClassRegistration(true)
+ .withCompatibleMode(CompatibleMode.COMPATIBLE)
+ .build();
+ // register and generate serializer code.
+ f.registe
Re: [PR] feat(java): support compatible mode for graalvm [incubator-fury]
chaokunyang merged PR #1586: URL: https://github.com/apache/incubator-fury/pull/1586 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" is working again!
The GitHub Actions job "Fury CI" on incubator-fury.git has succeeded. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 10cb4b08b2007e66162c02f3e98cee7baec8a9ff / chaokunyang add license header Report URL: https://github.com/apache/incubator-fury/actions/runs/8857611344 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 3999043a427ef09485d5cfc321f8b6414eabecde / chaokunyang support compatible mode for graalvm Report URL: https://github.com/apache/incubator-fury/actions/runs/8857594187 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[PR] feat(java): support compatible mode for graalvm [incubator-fury]
chaokunyang opened a new pull request, #1586: URL: https://github.com/apache/incubator-fury/pull/1586 ## What does this PR do? This PR supports compatible mode for graalvm ## Related issues Closes #1555 ## Does this PR introduce any user-facing change? - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" is working again!
The GitHub Actions job "Fury CI" on incubator-fury.git has succeeded. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 73f9e9bfba56cb9c907a11e7159549e3cd4bdfb2 / Shawn Yang fix(java): grow buffer to interger.max (#1582) ## What does this PR do? grow buffer to interger.max ## Related issues #1576 ## Does this PR introduce any user-facing change? - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark Report URL: https://github.com/apache/incubator-fury/actions/runs/8857251573 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: ab97bbeba16482ce51efcdfab12c376a765cb711 / chaokunyang revive soft reference Report URL: https://github.com/apache/incubator-fury/actions/runs/8857249751 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[PR] feat(java): revive soft reference [incubator-fury]
chaokunyang opened a new pull request, #1585: URL: https://github.com/apache/incubator-fury/pull/1585 ## What does this PR do? This PR revives soft reference to try to make it has a longer lifetime ## Related issues ## Does this PR introduce any user-facing change? - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] fix(java): grow buffer to interger.max [incubator-fury]
chaokunyang merged PR #1582: URL: https://github.com/apache/incubator-fury/pull/1582 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury) branch main updated: fix(java): grow buffer to interger.max (#1582)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury.git
The following commit(s) were added to refs/heads/main by this push:
new 73f9e9bf fix(java): grow buffer to interger.max (#1582)
73f9e9bf is described below
commit 73f9e9bfba56cb9c907a11e7159549e3cd4bdfb2
Author: Shawn Yang
AuthorDate: Sat Apr 27 12:15:04 2024 +0800
fix(java): grow buffer to interger.max (#1582)
## What does this PR do?
grow buffer to interger.max
## Related issues
#1576
## Does this PR introduce any user-facing change?
- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?
## Benchmark
---
.../java/org/apache/fury/io/FuryInputStream.java | 9 -
.../java/org/apache/fury/io/FuryReadableChannel.java | 4 +++-
.../java/org/apache/fury/io/FuryStreamReader.java| 1 -
.../java/org/apache/fury/memory/MemoryBuffer.java| 20
4 files changed, 23 insertions(+), 11 deletions(-)
diff --git
a/java/fury-core/src/main/java/org/apache/fury/io/FuryInputStream.java
b/java/fury-core/src/main/java/org/apache/fury/io/FuryInputStream.java
index 7096bd21..3dce088a 100644
--- a/java/fury-core/src/main/java/org/apache/fury/io/FuryInputStream.java
+++ b/java/fury-core/src/main/java/org/apache/fury/io/FuryInputStream.java
@@ -77,11 +77,10 @@ public class FuryInputStream extends InputStream implements
FuryStreamReader {
private static byte[] growBuffer(int minFillSize, MemoryBuffer buffer) {
int newSize;
int targetSize = buffer.size() + minFillSize;
-if (targetSize < BUFFER_GROW_STEP_THRESHOLD) {
- newSize = targetSize << 2;
-} else {
- newSize = (int) (targetSize * 1.5);
-}
+newSize =
+targetSize < MemoryBuffer.BUFFER_GROW_STEP_THRESHOLD
+? targetSize << 2
+: (int) Math.min(targetSize * 1.5d, Integer.MAX_VALUE);
byte[] newBuffer = new byte[newSize];
byte[] heapMemory = buffer.getHeapMemory();
System.arraycopy(heapMemory, 0, newBuffer, 0, buffer.size());
diff --git
a/java/fury-core/src/main/java/org/apache/fury/io/FuryReadableChannel.java
b/java/fury-core/src/main/java/org/apache/fury/io/FuryReadableChannel.java
index 293620b7..245a34b4 100644
--- a/java/fury-core/src/main/java/org/apache/fury/io/FuryReadableChannel.java
+++ b/java/fury-core/src/main/java/org/apache/fury/io/FuryReadableChannel.java
@@ -56,7 +56,9 @@ public class FuryReadableChannel implements FuryStreamReader,
ReadableByteChanne
int newLimit = position + minFillSize;
if (newLimit > byteBuf.capacity()) {
int newSize =
-newLimit < BUFFER_GROW_STEP_THRESHOLD ? newLimit << 2 : (int)
(newLimit * 1.5);
+newLimit < MemoryBuffer.BUFFER_GROW_STEP_THRESHOLD
+? newLimit << 2
+: (int) Math.min(newLimit * 1.5d, Integer.MAX_VALUE);
ByteBuffer newByteBuf = ByteBuffer.allocateDirect(newSize);
byteBuf.position(0);
newByteBuf.put(byteBuf);
diff --git
a/java/fury-core/src/main/java/org/apache/fury/io/FuryStreamReader.java
b/java/fury-core/src/main/java/org/apache/fury/io/FuryStreamReader.java
index e02d4efb..4d4c6d5e 100644
--- a/java/fury-core/src/main/java/org/apache/fury/io/FuryStreamReader.java
+++ b/java/fury-core/src/main/java/org/apache/fury/io/FuryStreamReader.java
@@ -26,7 +26,6 @@ import org.apache.fury.memory.MemoryBuffer;
/** A streaming reader to make {@link MemoryBuffer} to support streaming
reading. */
public interface FuryStreamReader {
- int BUFFER_GROW_STEP_THRESHOLD = 100 * 1024 * 1024;
/**
* Read stream and fill the data to underlying {@link MemoryBuffer}, which
is also the buffer
diff --git
a/java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java
b/java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java
index f5b1eea4..d57009eb 100644
--- a/java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java
+++ b/java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java
@@ -60,6 +60,7 @@ import sun.misc.Unsafe;
* DesiredMethodLimit,MaxRecursiveInlineLevel,FreqInlineSize,MaxInlineSize
*/
public final class MemoryBuffer {
+ public static final int BUFFER_GROW_STEP_THRESHOLD = 100 * 1024 * 1024;
private static final Unsafe UNSAFE = Platform.UNSAFE;
private static final boolean LITTLE_ENDIAN = (ByteOrder.nativeOrder() ==
ByteOrder.LITTLE_ENDIAN);
@@ -1204,18 +1205,29 @@ public final class MemoryBuffer {
/** For off-heap buffer, this will make a heap buffer internally. */
public void grow(int neededSize) {
-ensure(writerIndex + neededSize);
+int length = writerIndex + neededSize;
+if (length > size) {
+ growBuffer(length);
+}
}
/** For off-heap buff
(incubator-fury) branch main updated: chore(java): move ignore method into Exception util (#1584)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury.git
The following commit(s) were added to refs/heads/main by this push:
new e6646154 chore(java): move ignore method into Exception util (#1584)
e6646154 is described below
commit e6646154a9df32eee83473410ab10cdf2636791f
Author: Shawn Yang
AuthorDate: Sat Apr 27 12:11:01 2024 +0800
chore(java): move ignore method into Exception util (#1584)
## What does this PR do?
move exception ignore method into Exception util
## Related issues
## Does this PR introduce any user-facing change?
- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?
## Benchmark
---
.../main/java/org/apache/fury/builder/MetaSharedCodecBuilder.java | 4 ++--
.../java/org/apache/fury/serializer/ObjectStreamSerializer.java | 4 ++--
.../src/main/java/org/apache/fury/serializer/Serializers.java | 6 +++---
.../src/main/java/org/apache/fury/serializer/StructSerializer.java | 4 ++--
.../apache/fury/serializer/collection/SynchronizedSerializers.java | 4 ++--
.../apache/fury/serializer/collection/UnmodifiableSerializers.java | 4 ++--
.../src/main/java/org/apache/fury/util/ExceptionUtils.java | 2 ++
java/fury-core/src/main/java/org/apache/fury/util/Utils.java| 5 +
.../src/main/java/org/apache/fury/util/unsafe/_JDKAccess.java | 4 ++--
9 files changed, 18 insertions(+), 19 deletions(-)
diff --git
a/java/fury-core/src/main/java/org/apache/fury/builder/MetaSharedCodecBuilder.java
b/java/fury-core/src/main/java/org/apache/fury/builder/MetaSharedCodecBuilder.java
index 8879f567..074c7c2c 100644
---
a/java/fury-core/src/main/java/org/apache/fury/builder/MetaSharedCodecBuilder.java
+++
b/java/fury-core/src/main/java/org/apache/fury/builder/MetaSharedCodecBuilder.java
@@ -40,9 +40,9 @@ import org.apache.fury.serializer.Serializer;
import org.apache.fury.serializer.Serializers;
import org.apache.fury.type.Descriptor;
import org.apache.fury.type.DescriptorGrouper;
+import org.apache.fury.util.ExceptionUtils;
import org.apache.fury.util.Preconditions;
import org.apache.fury.util.StringUtils;
-import org.apache.fury.util.Utils;
import org.apache.fury.util.record.RecordComponent;
import org.apache.fury.util.record.RecordUtils;
@@ -174,7 +174,7 @@ public class MetaSharedCodecBuilder extends
ObjectCodecBuilder {
// Note that the field value shouldn't be an inlined value, otherwise
field value read may
// be ignored.
// Add an ignored call here to make expression type to void.
- return new Expression.StaticInvoke(Utils.class, "ignore", value);
+ return new Expression.StaticInvoke(ExceptionUtils.class, "ignore",
value);
}
return super.setFieldValue(bean, descriptor, value);
}
diff --git
a/java/fury-core/src/main/java/org/apache/fury/serializer/ObjectStreamSerializer.java
b/java/fury-core/src/main/java/org/apache/fury/serializer/ObjectStreamSerializer.java
index 65a1c268..137d3c94 100644
---
a/java/fury-core/src/main/java/org/apache/fury/serializer/ObjectStreamSerializer.java
+++
b/java/fury-core/src/main/java/org/apache/fury/serializer/ObjectStreamSerializer.java
@@ -59,8 +59,8 @@ import org.apache.fury.resolver.ClassInfo;
import org.apache.fury.resolver.ClassResolver;
import org.apache.fury.resolver.FieldResolver;
import org.apache.fury.resolver.FieldResolver.ClassField;
+import org.apache.fury.util.ExceptionUtils;
import org.apache.fury.util.Preconditions;
-import org.apache.fury.util.Utils;
import org.apache.fury.util.unsafe._JDKAccess;
/**
@@ -296,7 +296,7 @@ public class ObjectStreamSerializer extends Serializer {
_JDKAccess.makeJDKConsumer(lookup,
lookup.unreflect(readObjectNoData));
}
} catch (Exception e) {
-Utils.ignore(e);
+ExceptionUtils.ignore(e);
}
this.writeObjectFunc = writeObjectFunc;
this.readObjectFunc = readObjectFunc;
diff --git
a/java/fury-core/src/main/java/org/apache/fury/serializer/Serializers.java
b/java/fury-core/src/main/java/org/apache/fury/serializer/Serializers.java
index e0b84f31..7e1bab63 100644
--- a/java/fury-core/src/main/java/org/apache/fury/serializer/Serializers.java
+++ b/java/fury-core/src/main/java/org/apache/fury/serializer/Serializers.java
@@ -49,9 +49,9 @@ import org.apache.fury.memory.Platform;
import org.apache.fury.reflect.ReflectionUtils;
import org.apache.fury.resolver.ClassResolver;
import org.apache.fury.type.Type;
+import org.apache.fury.util.ExceptionUtils;
import org.apache.fury.util.GraalvmSupport;
import org.apache.fury.util.Preconditions;
-import org.apache.fury.util.Utils;
import org.apache.fury.util.unsafe._JDKAccess;
/** Serialization utils and common seri
Re: [PR] chore(java): move ignore method into Exception util [incubator-fury]
chaokunyang merged PR #1584: URL: https://github.com/apache/incubator-fury/pull/1584 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" is working again!
The GitHub Actions job "Fury CI" on incubator-fury.git has succeeded. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 049a6376782984ea8a7cf352afeaa1d6ee6b905c / chaokunyang mv ignore into ExceptionUtils Report URL: https://github.com/apache/incubator-fury/actions/runs/8857207175 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] fix(java): grow buffer to interger.max [incubator-fury]
chaokunyang commented on code in PR #1582:
URL: https://github.com/apache/incubator-fury/pull/1582#discussion_r1581726847
##
java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java:
##
@@ -1204,18 +1204,27 @@ public void writePrimitiveArray(Object arr, int offset,
int numBytes) {
/** For off-heap buffer, this will make a heap buffer internally. */
public void grow(int neededSize) {
-ensure(writerIndex + neededSize);
+int length = writerIndex + neededSize;
+if (length > size) {
+ growBuffer(length);
+}
}
/** For off-heap buffer, this will make a heap buffer internally. */
public void ensure(int length) {
if (length > size) {
- byte[] data = new byte[length * 2];
- copyToUnsafe(0, data, Platform.BYTE_ARRAY_OFFSET, size());
- initHeapBuffer(data, 0, data.length);
+ growBuffer(length);
}
}
+ private void growBuffer(int length) {
+int newSize =
+length < 104857600 ? length << 2 : (int) Math.min(length * 1.5d,
Integer.MAX_VALUE);
Review Comment:
good suggestion, updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 396162b5b3ec2d8a3bae3a2f1348e18024675635 / chaokunyang lint code Report URL: https://github.com/apache/incubator-fury/actions/runs/8857180910 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] fix(java): grow buffer to interger.max [incubator-fury]
LiangliangSui commented on code in PR #1582:
URL: https://github.com/apache/incubator-fury/pull/1582#discussion_r1581726542
##
java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java:
##
@@ -1204,18 +1204,27 @@ public void writePrimitiveArray(Object arr, int offset,
int numBytes) {
/** For off-heap buffer, this will make a heap buffer internally. */
public void grow(int neededSize) {
-ensure(writerIndex + neededSize);
+int length = writerIndex + neededSize;
+if (length > size) {
+ growBuffer(length);
+}
}
/** For off-heap buffer, this will make a heap buffer internally. */
public void ensure(int length) {
if (length > size) {
- byte[] data = new byte[length * 2];
- copyToUnsafe(0, data, Platform.BYTE_ARRAY_OFFSET, size());
- initHeapBuffer(data, 0, data.length);
+ growBuffer(length);
}
}
+ private void growBuffer(int length) {
+int newSize =
+length < 104857600 ? length << 2 : (int) Math.min(length * 1.5d,
Integer.MAX_VALUE);
Review Comment:
Move `org.apache.fury.io.FuryStreamReader#BUFFER_GROW_STEP_THRESHOLD` to
`MemoryBuffer` and reuse it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] fix(java): grow buffer to interger.max [incubator-fury]
chaokunyang commented on code in PR #1582:
URL: https://github.com/apache/incubator-fury/pull/1582#discussion_r1581725946
##
java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java:
##
@@ -1204,18 +1204,27 @@ public void writePrimitiveArray(Object arr, int offset,
int numBytes) {
/** For off-heap buffer, this will make a heap buffer internally. */
public void grow(int neededSize) {
-ensure(writerIndex + neededSize);
+int length = writerIndex + neededSize;
+if (length > size) {
+ growBuffer(length);
+}
}
/** For off-heap buffer, this will make a heap buffer internally. */
public void ensure(int length) {
if (length > size) {
- byte[] data = new byte[length * 2];
- copyToUnsafe(0, data, Platform.BYTE_ARRAY_OFFSET, size());
- initHeapBuffer(data, 0, data.length);
+ growBuffer(length);
}
}
+ private void growBuffer(int length) {
+int newSize =
+length < 104857600 ? length << 2 : (int) Math.min(length * 1.5d,
Integer.MAX_VALUE);
Review Comment:
It' just 100 * 1024 * 1024
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury) branch main updated: refactor(java): move reflect utils into reflect package (#1583)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury.git
The following commit(s) were added to refs/heads/main by this push:
new 91170b5b refactor(java): move reflect utils into reflect package
(#1583)
91170b5b is described below
commit 91170b5bea055b98f303b3b19a588681a6793448
Author: Shawn Yang
AuthorDate: Sat Apr 27 12:00:35 2024 +0800
refactor(java): move reflect utils into reflect package (#1583)
## What does this PR do?
This RP move reflect utils into reflect package.
## Related issues
## Does this PR introduce any user-facing change?
- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?
## Benchmark
---
.../main/java/org/apache/fury/benchmark/NewJava11StringSuite.java| 2 +-
.../src/main/java/org/apache/fury/benchmark/NewStringSuite.java | 2 +-
.../src/main/java/org/apache/fury/builder/AccessorHelper.java| 5 +++--
.../main/java/org/apache/fury/builder/BaseObjectCodecBuilder.java| 2 +-
.../src/main/java/org/apache/fury/builder/CodecBuilder.java | 2 +-
.../main/java/org/apache/fury/builder/CompatibleCodecBuilder.java| 2 +-
java/fury-core/src/main/java/org/apache/fury/builder/Generated.java | 2 +-
.../src/main/java/org/apache/fury/codegen/CodeGenerator.java | 2 +-
.../src/main/java/org/apache/fury/codegen/CodegenContext.java| 2 +-
java/fury-core/src/main/java/org/apache/fury/codegen/Expression.java | 2 +-
.../src/main/java/org/apache/fury/codegen/ExpressionVisitor.java | 2 +-
.../fury-core/src/main/java/org/apache/fury/codegen/JaninoUtils.java | 2 +-
java/fury-core/src/main/java/org/apache/fury/meta/ClassDef.java | 2 +-
.../main/java/org/apache/fury/{util => reflect}/FieldAccessor.java | 3 ++-
.../main/java/org/apache/fury/{util => reflect}/ReflectionUtils.java | 5 +++--
.../java/org/apache/fury/{util => reflect}/UnsafeFieldAccessor.java | 3 ++-
java/fury-core/src/main/java/org/apache/fury/resolver/ClassInfo.java | 2 +-
.../src/main/java/org/apache/fury/resolver/ClassResolver.java| 2 +-
.../src/main/java/org/apache/fury/resolver/FieldResolver.java| 4 ++--
.../src/main/java/org/apache/fury/serializer/ArraySerializers.java | 2 +-
.../main/java/org/apache/fury/serializer/CompatibleSerializer.java | 4 ++--
.../java/org/apache/fury/serializer/ExternalizableSerializer.java| 2 +-
.../src/main/java/org/apache/fury/serializer/JdkProxySerializer.java | 2 +-
.../src/main/java/org/apache/fury/serializer/LambdaSerializer.java | 2 +-
.../main/java/org/apache/fury/serializer/MetaSharedSerializer.java | 4 ++--
.../src/main/java/org/apache/fury/serializer/ObjectSerializer.java | 4 ++--
.../main/java/org/apache/fury/serializer/ObjectStreamSerializer.java | 2 +-
.../java/org/apache/fury/serializer/ReplaceResolveSerializer.java| 2 +-
.../src/main/java/org/apache/fury/serializer/Serializers.java| 2 +-
.../src/main/java/org/apache/fury/serializer/StringSerializer.java | 2 +-
.../src/main/java/org/apache/fury/serializer/StructSerializer.java | 2 +-
.../fury/serializer/collection/AbstractCollectionSerializer.java | 2 +-
.../org/apache/fury/serializer/collection/AbstractMapSerializer.java | 2 +-
.../apache/fury/serializer/collection/ChildContainerSerializers.java | 2 +-
.../org/apache/fury/serializer/collection/CollectionSerializers.java | 2 +-
.../java/org/apache/fury/serializer/collection/MapSerializers.java | 2 +-
.../src/main/java/org/apache/fury/type/DescriptorGrouper.java| 2 +-
java/fury-core/src/main/java/org/apache/fury/type/GenericType.java | 2 +-
java/fury-core/src/main/java/org/apache/fury/type/ScalaTypes.java| 2 +-
java/fury-core/src/main/java/org/apache/fury/type/TypeUtils.java | 2 +-
.../fury-core/src/main/java/org/apache/fury/util/ExceptionUtils.java | 1 +
.../src/main/java/org/apache/fury/util/function/Functions.java | 2 +-
.../native-image/org.apache.fury/fury-core/native-image.properties | 4 ++--
java/fury-core/src/test/java/org/apache/fury/FuryTestBase.java | 2 +-
java/fury-core/src/test/java/org/apache/fury/StreamTest.java | 2 +-
java/fury-core/src/test/java/org/apache/fury/TestUtils.java | 4 ++--
.../src/test/java/org/apache/fury/codegen/ExpressionVisitorTest.java | 2 +-
.../src/test/java/org/apache/fury/codegen/JaninoUtilsTest.java | 2 +-
.../test/java/org/apache/fury/serializer/ArraySerializersTest.java | 2 +-
.../src/test/java/org/apache/fury/serializer/ClassUtils.java | 2 +-
.../org/apache/fury/serializer/CodegenCompatibleSerializerTest.java | 2 +-
.../java/org/apache/fury/serializer/CompatibleSerializerTest.java| 2 +-
.../java/org/apache/fury/serializer/MetaSharedCompat
Re: [PR] refactor(java): move reflect utils into reflect package [incubator-fury]
chaokunyang merged PR #1583: URL: https://github.com/apache/incubator-fury/pull/1583 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] fix(java): grow buffer to interger.max [incubator-fury]
LiangliangSui commented on code in PR #1582:
URL: https://github.com/apache/incubator-fury/pull/1582#discussion_r1581725642
##
java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java:
##
@@ -1204,18 +1204,27 @@ public void writePrimitiveArray(Object arr, int offset,
int numBytes) {
/** For off-heap buffer, this will make a heap buffer internally. */
public void grow(int neededSize) {
-ensure(writerIndex + neededSize);
+int length = writerIndex + neededSize;
+if (length > size) {
+ growBuffer(length);
+}
}
/** For off-heap buffer, this will make a heap buffer internally. */
public void ensure(int length) {
if (length > size) {
- byte[] data = new byte[length * 2];
- copyToUnsafe(0, data, Platform.BYTE_ARRAY_OFFSET, size());
- initHeapBuffer(data, 0, data.length);
+ growBuffer(length);
}
}
+ private void growBuffer(int length) {
+int newSize =
+length < 104857600 ? length << 2 : (int) Math.min(length * 1.5d,
Integer.MAX_VALUE);
Review Comment:
How is the value `104857600` calculated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" is working again!
The GitHub Actions job "Fury CI" on incubator-fury.git has succeeded. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 807ea97cc5094a0c08422a557c776a5bc2380d06 / chaokunyang move ignore into Exception util Report URL: https://github.com/apache/incubator-fury/actions/runs/8857126538 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Lint PR" is working again!
The GitHub Actions job "Lint PR" on incubator-fury.git has succeeded. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 807ea97cc5094a0c08422a557c776a5bc2380d06 / chaokunyang move ignore into Exception util Report URL: https://github.com/apache/incubator-fury/actions/runs/8857131976 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 8fc58c569fa29712009ba22baa81b59a17e11901 / chaokunyang move reflect utils into refllect package Report URL: https://github.com/apache/incubator-fury/actions/runs/8857117043 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Lint PR" failed!
The GitHub Actions job "Lint PR" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 807ea97cc5094a0c08422a557c776a5bc2380d06 / chaokunyang move ignore into Exception util Report URL: https://github.com/apache/incubator-fury/actions/runs/8857128010 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Lint PR" failed!
The GitHub Actions job "Lint PR" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 807ea97cc5094a0c08422a557c776a5bc2380d06 / chaokunyang move ignore into Exception util Report URL: https://github.com/apache/incubator-fury/actions/runs/8857127075 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Lint PR" failed!
The GitHub Actions job "Lint PR" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 807ea97cc5094a0c08422a557c776a5bc2380d06 / chaokunyang move ignore into Exception util Report URL: https://github.com/apache/incubator-fury/actions/runs/8857126486 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[PR] chore(java) : move ignore into Exception util [incubator-fury]
chaokunyang opened a new pull request, #1584: URL: https://github.com/apache/incubator-fury/pull/1584 ## What does this PR do? move ignore into Exception util ## Related issues ## Does this PR introduce any user-facing change? - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(spec/java): add strip flag in meta string encoding spec [incubator-fury]
qingoba commented on PR #1565: URL: https://github.com/apache/incubator-fury/pull/1565#issuecomment-2080347221 I can fix this now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 90e2cb3af3a8761fad300c644f102582f1b8d0d8 / chaokunyang make growBuffer in a separate method Report URL: https://github.com/apache/incubator-fury/actions/runs/8857100821 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[PR] refactor(java): move reflect utils into reflect package [incubator-fury]
chaokunyang opened a new pull request, #1583: URL: https://github.com/apache/incubator-fury/pull/1583 ## What does this PR do? This RP move reflect utils into reflect package. ## Related issues ## Does this PR introduce any user-facing change? - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 5e04e6619bd79bd0bb0138aeb82a26fd0f556910 / chaokunyang grow buffer to interger.max Report URL: https://github.com/apache/incubator-fury/actions/runs/8857087653 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[PR] fix(java): grow buffer to interger.max [incubator-fury]
chaokunyang opened a new pull request, #1582: URL: https://github.com/apache/incubator-fury/pull/1582 ## What does this PR do? grow buffer to interger.max ## Related issues #1576 ## Does this PR introduce any user-facing change? - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(spec/java): add strip flag in meta string encoding spec [incubator-fury]
chaokunyang commented on PR #1565:
URL: https://github.com/apache/incubator-fury/pull/1565#issuecomment-2080343071
> There is another question. If input string is empty, encoded bytes is also
empty:
>
> ```java
> public MetaString encode(String input, Encoding encoding) {
> Preconditions.checkArgument(
> input.length() < Short.MAX_VALUE, "Long meta string than 32767 is
not allowed");
> if (input.isEmpty()) {
> return new MetaString(input, Encoding.UTF_8, specialChar1,
specialChar2, new byte[0]);
> }
> }
> ```
>
> However, decode procedure doesn't check empty. It will access `bytes[0]`
directly:
>
> ```java
> private String decodeLowerSpecial(byte[] data) {
> StringBuilder decoded = new StringBuilder();
> int totalBits = data.length * 8; // Total number of bits in the data
> boolean stripLastChar = (data[0] & 0x80) != 0; // Check the first bit of
the first byte
> }
> ```
>
> So decode procedure should check empty.
Hi @qingoba , do you have time to fix this today? We plan to start release
fury [v0.5.0](#1389) today and include this fix. If you don't have time, I can
take over it and submit a fix later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury) branch main updated: fix(java): add guava derive comments (#1581)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury.git
The following commit(s) were added to refs/heads/main by this push:
new 108c92f3 fix(java): add guava derive comments (#1581)
108c92f3 is described below
commit 108c92f3b981fd65859c4322944f20a510b02e4b
Author: Shawn Yang
AuthorDate: Sat Apr 27 11:31:17 2024 +0800
fix(java): add guava derive comments (#1581)
## What does this PR do?
add guava derive comments
## Related issues
#1553
## Does this PR introduce any user-facing change?
- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?
## Benchmark
---
.../src/main/java/org/apache/fury/reflect/TypeParameter.java| 2 ++
java/fury-core/src/main/java/org/apache/fury/reflect/TypeRef.java | 6 ++
java/fury-core/src/main/java/org/apache/fury/reflect/Types.java | 2 ++
3 files changed, 6 insertions(+), 4 deletions(-)
diff --git
a/java/fury-core/src/main/java/org/apache/fury/reflect/TypeParameter.java
b/java/fury-core/src/main/java/org/apache/fury/reflect/TypeParameter.java
index 2823cbd7..1612e65c 100644
--- a/java/fury-core/src/main/java/org/apache/fury/reflect/TypeParameter.java
+++ b/java/fury-core/src/main/java/org/apache/fury/reflect/TypeParameter.java
@@ -19,6 +19,8 @@ import java.lang.reflect.Type;
import java.lang.reflect.TypeVariable;
import java.util.Objects;
+// Derived from Guava 32.1.2 com.google.common.reflect.TypeParameter
+//
https://github.com/google/guava/blob/9f6a3840/guava/src/com/google/common/reflect/TypeParameter.java
public abstract class TypeParameter {
final TypeVariable typeVariable;
diff --git a/java/fury-core/src/main/java/org/apache/fury/reflect/TypeRef.java
b/java/fury-core/src/main/java/org/apache/fury/reflect/TypeRef.java
index 9dae933f..9a41fa13 100644
--- a/java/fury-core/src/main/java/org/apache/fury/reflect/TypeRef.java
+++ b/java/fury-core/src/main/java/org/apache/fury/reflect/TypeRef.java
@@ -35,10 +35,8 @@ import java.util.stream.Collectors;
import java.util.stream.Stream;
import org.apache.fury.type.TypeUtils;
-/**
- * Mostly derived from Guava 32.1.2 com.google.common.reflect.TypeToken
- *
https://github.com/google/guava/blob/master/guava/src/com/google/common/reflect/TypeToken.java
- */
+// Mostly derived from Guava 32.1.2 com.google.common.reflect.TypeToken
+//
https://github.com/google/guava/blob/9f6a3840/guava/src/com/google/common/reflect/TypeToken.java
public class TypeRef {
private final Type type;
diff --git a/java/fury-core/src/main/java/org/apache/fury/reflect/Types.java
b/java/fury-core/src/main/java/org/apache/fury/reflect/Types.java
index 1f357863..cbfc7cdb 100644
--- a/java/fury-core/src/main/java/org/apache/fury/reflect/Types.java
+++ b/java/fury-core/src/main/java/org/apache/fury/reflect/Types.java
@@ -27,6 +27,8 @@ import java.util.Arrays;
import java.util.Objects;
import javax.annotation.CheckForNull;
+// Derived from Guava 32.1.2 com.google.common.reflect.Types
+//
https://github.com/google/guava/blob/9f6a3840/guava/src/com/google/common/reflect/Types.java
class Types {
public static Type newArrayType(Type componentType) {
if (componentType instanceof WildcardType) {
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] fix(java): add guava derive comments [incubator-fury]
chaokunyang merged PR #1581: URL: https://github.com/apache/incubator-fury/pull/1581 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(golang): meta string encoding algorithm in golang [incubator-fury]
chaokunyang merged PR #1566: URL: https://github.com/apache/incubator-fury/pull/1566 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury) branch main updated: feat(golang): meta string encoding algorithm in golang (#1566)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury.git
The following commit(s) were added to refs/heads/main by this push:
new f907f7f0 feat(golang): meta string encoding algorithm in golang (#1566)
f907f7f0 is described below
commit f907f7f0ccbbebdf90152fae6d3a3d5a63354ecc
Author: qingoba <51332889+qing...@users.noreply.github.com>
AuthorDate: Sat Apr 27 11:28:23 2024 +0800
feat(golang): meta string encoding algorithm in golang (#1566)
## What does this PR do?
This PR implements meta string encoding described in [xlang
serialization
spec](https://fury.apache.org/docs/specification/fury_xlang_serialization_spec#meta-string)
## Related issues
- #1240
- #1413
- #1514
## Does this PR introduce any user-facing change?
- [No] Does this PR introduce any public API change?
- [No] Does this PR introduce any binary protocol compatibility change?
-
Co-authored-by: Shawn Yang
---
go/fury/meta/meta_string.go | 56
go/fury/meta/meta_string_decoder.go | 169
go/fury/meta/meta_string_encoder.go | 253
go/fury/meta/meta_string_test.go| 82
4 files changed, 560 insertions(+)
diff --git a/go/fury/meta/meta_string.go b/go/fury/meta/meta_string.go
new file mode 100644
index ..429e6e58
--- /dev/null
+++ b/go/fury/meta/meta_string.go
@@ -0,0 +1,56 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package meta
+
+// Encoding Algorithms Flags
+type Encoding uint8
+
+const (
+ UTF_8 Encoding = 0x00
+ LOWER_SPECIAL Encoding = 0x01
+ LOWER_UPPER_DIGIT_SPECIAL Encoding = 0x02
+ FIRST_TO_LOWER_SPECIALEncoding = 0x03
+ ALL_TO_LOWER_SPECIAL Encoding = 0x04
+)
+
+// MetaString saves the serialized data
+type MetaString struct {
+ inputString string
+ encoding Encoding // encoding flag
+ specialChar1 byte
+ specialChar2 byte
+ encodedBytes []byte // serialized data
+}
+
+func (ms *MetaString) GetInputString() string { return ms.inputString }
+
+func (ms *MetaString) GetEncoding() Encoding { return ms.encoding }
+
+func (ms *MetaString) GetSpecialChar1() byte { return ms.specialChar1 }
+
+func (ms *MetaString) GetSpecialChar2() byte { return ms.specialChar2 }
+
+func (ms *MetaString) GetEncodedBytes() []byte { return ms.encodedBytes }
+
+// StripLastChar return true if last char should be stripped
+func (ms *MetaString) StripLastChar() bool {
+ if ms.encoding == UTF_8 || ms.encodedBytes == nil {
+ return false
+ }
+ return (ms.encodedBytes[0] & 0x80) > 0
+}
diff --git a/go/fury/meta/meta_string_decoder.go
b/go/fury/meta/meta_string_decoder.go
new file mode 100644
index ..59273b03
--- /dev/null
+++ b/go/fury/meta/meta_string_decoder.go
@@ -0,0 +1,169 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package meta
+
+import (
+ "fmt"
+)
+
+type Decoder struct {
+ specialChar1 byte
+ specialChar2 byte
+}
+
+func NewDecoder(specialCh1 byte, specialCh2 byte) *Decoder {
+ return &Decoder{
+ specialChar1: specialCh1,
+ specialChar2: specialCh2,
+ }
+}
+
+// Decode
+// Accept an
Re: [PR] fix(java): add guava derive comments [incubator-fury]
Munoon commented on PR #1581: URL: https://github.com/apache/incubator-fury/pull/1581#issuecomment-2079712282 LGTM Perhaps it's just worth adding more permanent links? For example, one that indicates the state of a file in a particular commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury) branch main updated: fix(java): fix bytebuffer no such method error (#1580)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury.git
The following commit(s) were added to refs/heads/main by this push:
new 124bf0ec fix(java): fix bytebuffer no such method error (#1580)
124bf0ec is described below
commit 124bf0ec3fbe6138e2e2c5aa7f1bc14c3d3b8055
Author: Shawn Yang
AuthorDate: Sat Apr 27 00:19:11 2024 +0800
fix(java): fix bytebuffer no such method error (#1580)
## What does this PR do?
This PR add a bytebuffer util and fix bytebuffer no such method error
## Related issues
Closes #1577
#1579
## Does this PR introduce any user-facing change?
- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?
## Benchmark
---
.../fury/benchmark/UserTypeDeserializeSuite.java | 4 +-
.../fury/benchmark/UserTypeSerializeSuite.java | 4 +-
.../org/apache/fury/benchmark/ZeroCopySuite.java | 6 +-
.../fury/benchmark/state/FlatBuffersState.java | 4 +-
.../org/apache/fury/benchmark/state/FstState.java | 6 +-
.../apache/fury/benchmark/state/JsonbState.java| 8 +-
.../org/apache/fury/io/FuryReadableChannel.java| 3 +-
.../org/apache/fury/memory/ByteBufferUtil.java | 108 +
.../java/org/apache/fury/memory/MemoryBuffer.java | 18 ++--
.../main/java/org/apache/fury/memory/Platform.java | 81
.../src/test/java/org/apache/fury/StreamTest.java | 28 --
.../org/apache/fury/memory/MemoryBufferTest.java | 7 +-
.../fury/serializer/BufferSerializersTest.java | 5 +-
.../java/org/apache/fury/util/PlatformTest.java| 11 ++-
14 files changed, 148 insertions(+), 145 deletions(-)
diff --git
a/java/benchmark/src/main/java/org/apache/fury/benchmark/UserTypeDeserializeSuite.java
b/java/benchmark/src/main/java/org/apache/fury/benchmark/UserTypeDeserializeSuite.java
index 94d24784..f18acc5a 100644
---
a/java/benchmark/src/main/java/org/apache/fury/benchmark/UserTypeDeserializeSuite.java
+++
b/java/benchmark/src/main/java/org/apache/fury/benchmark/UserTypeDeserializeSuite.java
@@ -30,7 +30,7 @@ import org.apache.fury.benchmark.state.KryoState;
import org.apache.fury.benchmark.state.ObjectType;
import org.apache.fury.benchmark.state.ProtoBuffersState;
import org.apache.fury.benchmark.state.ProtostuffState;
-import org.apache.fury.memory.Platform;
+import org.apache.fury.memory.ByteBufferUtil;
import org.openjdk.jmh.Main;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
@@ -125,7 +125,7 @@ public class UserTypeDeserializeSuite {
@Benchmark
public Object
flatbuffers_deserialize(FlatBuffersState.FlatBuffersUserTypeState state) {
-Platform.clearBuffer(state.deserializedData);
+ByteBufferUtil.clearBuffer(state.deserializedData);
if (state.objectType == ObjectType.SAMPLE) {
return FlatBuffersState.deserializeSample(state.deserializedData);
} else {
diff --git
a/java/benchmark/src/main/java/org/apache/fury/benchmark/UserTypeSerializeSuite.java
b/java/benchmark/src/main/java/org/apache/fury/benchmark/UserTypeSerializeSuite.java
index c563f099..b6697025 100644
---
a/java/benchmark/src/main/java/org/apache/fury/benchmark/UserTypeSerializeSuite.java
+++
b/java/benchmark/src/main/java/org/apache/fury/benchmark/UserTypeSerializeSuite.java
@@ -33,7 +33,7 @@ import org.apache.fury.benchmark.state.KryoState;
import org.apache.fury.benchmark.state.ObjectType;
import org.apache.fury.benchmark.state.ProtoBuffersState;
import org.apache.fury.benchmark.state.ProtostuffState;
-import org.apache.fury.memory.Platform;
+import org.apache.fury.memory.ByteBufferUtil;
import org.openjdk.jmh.Main;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
@@ -133,7 +133,7 @@ public class UserTypeSerializeSuite {
@Benchmark
public Object
flatbuffers_serialize(FlatBuffersState.FlatBuffersUserTypeState state) {
-Platform.clearBuffer(state.directBuffer);
+ByteBufferUtil.clearBuffer(state.directBuffer);
if (state.objectType == ObjectType.SAMPLE) {
return FlatBuffersState.serializeSample((Sample) state.object,
state.directBuffer);
} else {
diff --git
a/java/benchmark/src/main/java/org/apache/fury/benchmark/ZeroCopySuite.java
b/java/benchmark/src/main/java/org/apache/fury/benchmark/ZeroCopySuite.java
index 9861519c..0e06f450 100644
--- a/java/benchmark/src/main/java/org/apache/fury/benchmark/ZeroCopySuite.java
+++ b/java/benchmark/src/main/java/org/apache/fury/benchmark/ZeroCopySuite.java
@@ -41,9 +41,9 @@ import org.apache.fury.benchmark.data.ComparableByteArray;
import org.apache.fury.benchmark.data.SerializableByteBuffer;
import org.apache.fury.benchmark.state.BufferType;
import org.apache
Re: [I] serialize ByteBuffer report error [incubator-fury]
chaokunyang closed issue #1577: serialize ByteBuffer report error URL: https://github.com/apache/incubator-fury/issues/1577 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] fix(java): fix bytebuffer no such method error [incubator-fury]
chaokunyang merged PR #1580: URL: https://github.com/apache/incubator-fury/pull/1580 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" is working again!
The GitHub Actions job "Fury CI" on incubator-fury.git has succeeded. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 4dc4c266c830bcc98a7ed1f69fe3ba042befbc33 / chaokunyang add guava derive comments Report URL: https://github.com/apache/incubator-fury/actions/runs/8850999189 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] fix(java): add guava derive comments [incubator-fury]
chaokunyang commented on PR #1581: URL: https://github.com/apache/incubator-fury/pull/1581#issuecomment-2079669051 Hi @Munoon , could you take a look at this PR? We'd bettern add derive for copied guava code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[PR] fix(java): add guava derive comments [incubator-fury]
chaokunyang opened a new pull request, #1581: URL: https://github.com/apache/incubator-fury/pull/1581 ## What does this PR do? add guava derive comments ## Related issues #1553 ## Does this PR introduce any user-facing change? - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 431332fb5d283eeedb9980d5c2f38a53b3358601 / chaokunyang Merge remote-tracking branch 'ant/main' into new_meta_encoding_in_java Report URL: https://github.com/apache/incubator-fury/actions/runs/8850886371 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] fix(java): update pom.xml to use javac 8 [incubator-fury]
chaokunyang commented on code in PR #1579: URL: https://github.com/apache/incubator-fury/pull/1579#discussion_r1581206299 ## java/fury-core/pom.xml: ## @@ -127,6 +127,11 @@ org.apache.maven.plugins maven-compiler-plugin + Review Comment: I did a test, seems not work? Could you test it locally, you can compile it with java9+, then select to use java8 to run. And do we need to add this config to parent pom? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[PR] fix(java): fix bytebuffer no such method error [incubator-fury]
chaokunyang opened a new pull request, #1580: URL: https://github.com/apache/incubator-fury/pull/1580 ## What does this PR do? This PR add a bytebuffer util and fix bytebuffer no such method error ## Related issues Closes #1577 #1579 ## Does this PR introduce any user-facing change? - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] fix(java): update pom.xml to use javac 8 [incubator-fury]
chaokunyang commented on PR #1579: URL: https://github.com/apache/incubator-fury/pull/1579#issuecomment-2079607764 > Hi @stardustman , thanks for your contribution. Could you add a ByteBuffer ut for this ? Seems this is not easy to add a test, we must compile use one java version, and run it with another java version. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] refactor(java): replace Guava's TypeToken with self-made [incubator-fury]
chaokunyang merged PR #1553: URL: https://github.com/apache/incubator-fury/pull/1553 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] refactor(java): replace Guava's TypeToken with self-made [incubator-fury]
chaokunyang commented on PR #1553: URL: https://github.com/apache/incubator-fury/pull/1553#issuecomment-2079531803 > > LGTM, do you have any further updates? @Munoon > > I didn't start yet processing this comment: [#1553 (comment)](https://github.com/apache/incubator-fury/pull/1553#discussion_r1577925050) But if you think, that it is not required - lets go as it is not to break anything. We can optimize this in a further PRs. > > Also, I'm not sure, but I think that generics are used there when using maps and list. The compiler used by Fury doesn't generics, so Fury ignore generics in many places. Let's merge this first, we can optimize other parts in further PRs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] refactor(java): replace Guava's TypeToken with self-made [incubator-fury]
Munoon commented on PR #1553: URL: https://github.com/apache/incubator-fury/pull/1553#issuecomment-2079494233 > LGTM, do you have any further updates? @Munoon I didn't start yet processing this comment: https://github.com/apache/incubator-fury/pull/1553#discussion_r1577925050 But if you think, that it is not required - lets go as it is not to break anything. We can optimize this in a further PRs. Also, I'm not sure, but I think that generics are used there when using maps and list. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] [Java] Memory issue about FuryInputStream [incubator-fury]
chaokunyang commented on issue #1576: URL: https://github.com/apache/incubator-fury/issues/1576#issuecomment-2079268630 Would you like to help optimize this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] [Java] Memory issue about FuryInputStream [incubator-fury]
chaokunyang commented on issue #1576: URL: https://github.com/apache/incubator-fury/issues/1576#issuecomment-2079267981 Yes, that is possible. We don't release buffer before we finished the deserialization. One case I can imagine is that when buffer close to int max size, we can't multiple 1.5. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury) branch main updated: chore(java): move Platform class to memory package (#1578)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury.git
The following commit(s) were added to refs/heads/main by this push:
new c6bc9960 chore(java): move Platform class to memory package (#1578)
c6bc9960 is described below
commit c6bc9960f361e93604720ef62b8f25ff375819ea
Author: Shawn Yang
AuthorDate: Fri Apr 26 17:40:33 2024 +0800
chore(java): move Platform class to memory package (#1578)
## What does this PR do?
This PR moves Platform class to memory package, since it's most ly for
memory related operations
## Related issues
## Does this PR introduce any user-facing change?
- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?
## Benchmark
---
LICENSE| 2 +-
.../integration_tests/JDKCompatibilityTest.java| 2 +-
.../apache/fury/benchmark/CompressStringSuite.java | 2 +-
.../org/apache/fury/benchmark/Identity2IdMap.java | 2 +-
.../org/apache/fury/benchmark/MemorySuite.java | 2 +-
.../fury/benchmark/NewJava11StringSuite.java | 2 +-
.../org/apache/fury/benchmark/NewStringSuite.java | 2 +-
.../fury/benchmark/UserTypeDeserializeSuite.java | 2 +-
.../fury/benchmark/UserTypeSerializeSuite.java | 2 +-
.../org/apache/fury/benchmark/ZeroCopySuite.java | 2 +-
.../fury/benchmark/state/FlatBuffersState.java | 2 +-
.../org/apache/fury/benchmark/state/FstState.java | 2 +-
.../apache/fury/benchmark/state/JsonbState.java| 2 +-
.../src/main/java/org/apache/fury/Fury.java| 2 +-
.../fury/builder/BaseObjectCodecBuilder.java | 2 +-
.../java/org/apache/fury/builder/CodecBuilder.java | 2 +-
.../fury/builder/CompatibleCodecBuilder.java | 2 +-
.../java/org/apache/fury/builder/JITContext.java | 2 +-
.../apache/fury/builder/ObjectCodecBuilder.java| 2 +-
.../java/org/apache/fury/codegen/Expression.java | 2 +-
.../org/apache/fury/codegen/ExpressionVisitor.java | 2 +-
.../java/org/apache/fury/config/FuryBuilder.java | 2 +-
.../java/org/apache/fury/io/FuryInputStream.java | 2 +-
.../org/apache/fury/io/FuryReadableChannel.java| 2 +-
.../java/org/apache/fury/memory/LittleEndian.java | 2 --
.../java/org/apache/fury/memory/MemoryBuffer.java | 1 -
.../java/org/apache/fury/memory/MemoryUtils.java | 1 -
.../org/apache/fury/{util => memory}/Platform.java | 3 ++-
.../main/java/org/apache/fury/meta/ClassDef.java | 2 +-
.../org/apache/fury/resolver/ClassResolver.java| 2 +-
.../apache/fury/serializer/ArraySerializers.java | 2 +-
.../fury/serializer/CompatibleSerializer.java | 2 +-
.../fury/serializer/ExternalizableSerializer.java | 2 +-
.../apache/fury/serializer/ForwardSerializer.java | 2 +-
.../org/apache/fury/serializer/JavaSerializer.java | 2 +-
.../apache/fury/serializer/JdkProxySerializer.java | 2 +-
.../fury/serializer/MetaSharedSerializer.java | 2 +-
.../apache/fury/serializer/ObjectSerializer.java | 2 +-
.../fury/serializer/ObjectStreamSerializer.java| 2 +-
.../fury/serializer/PrimitiveSerializers.java | 2 +-
.../fury/serializer/ReplaceResolveSerializer.java | 2 +-
.../org/apache/fury/serializer/Serializers.java| 2 +-
.../apache/fury/serializer/StringSerializer.java | 2 +-
.../apache/fury/serializer/StructSerializer.java | 2 +-
.../org/apache/fury/serializer/URLSerializer.java | 2 +-
.../collection/CollectionSerializers.java | 2 +-
.../collection/GuavaCollectionSerializers.java | 2 +-
.../collection/ImmutableCollectionSerializers.java | 2 +-
.../fury/serializer/collection/MapSerializers.java | 2 +-
.../collection/SynchronizedSerializers.java| 2 +-
.../collection/UnmodifiableSerializers.java| 2 +-
.../scala/SingletonCollectionSerializer.java | 2 +-
.../serializer/scala/SingletonMapSerializer.java | 2 +-
.../scala/SingletonObjectSerializer.java | 2 +-
.../main/java/org/apache/fury/type/Descriptor.java | 2 +-
.../java/org/apache/fury/util/FieldAccessor.java | 1 +
.../java/org/apache/fury/util/ReflectionUtils.java | 1 +
.../org/apache/fury/util/UnsafeFieldAccessor.java | 1 +
.../org/apache/fury/util/unsafe/DefineClass.java | 2 +-
.../org/apache/fury/util/unsafe/_Collections.java | 2 +-
.../org/apache/fury/util/unsafe/_JDKAccess.java| 2 +-
java/fury-core/src/main/resources/META-INF/LICENSE | 2 +-
.../fury-core/native-image.properties | 2 +-
.../src/test/java/org/apache/fury/FuryTest.java| 2 +-
.../test/java/org/apache/fury/FuryTestBase.java| 2 +-
.../src/test/java/org/apache/fury/StreamTest.java | 28 ++
.../org/apache/fury/codegen/JaninoUtilsTest.java | 2 +-
Re: [PR] chore(java): move Platform class to memory package [incubator-fury]
chaokunyang merged PR #1578: URL: https://github.com/apache/incubator-fury/pull/1578 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] fix(java): update pom.xml to use javac 8 [incubator-fury]
chaokunyang commented on PR #1579: URL: https://github.com/apache/incubator-fury/pull/1579#issuecomment-2079018988 Hi @stardustman , thanks for your contribution. Could you add a ByteBuffer ut for this ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" is working again!
The GitHub Actions job "Fury CI" on incubator-fury.git has succeeded. Run started by GitHub user stardustman (triggered by stardustman). Head commit for run: d919f96050a0ca7ee5c236bf61b3727ad5c2e5c3 / stardust Update pom.xml Report URL: https://github.com/apache/incubator-fury/actions/runs/8846044375 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: b021c53aa0f75fae48f282eab714bc91a0567082 / chaokunyang fix license Report URL: https://github.com/apache/incubator-fury/actions/runs/8846069445 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Lint PR" is working again!
The GitHub Actions job "Lint PR" on incubator-fury.git has succeeded. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: b021c53aa0f75fae48f282eab714bc91a0567082 / chaokunyang fix license Report URL: https://github.com/apache/incubator-fury/actions/runs/8846069240 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Lint PR" failed!
The GitHub Actions job "Lint PR" on incubator-fury.git has failed. Run started by GitHub user stardustman (triggered by stardustman). Head commit for run: d919f96050a0ca7ee5c236bf61b3727ad5c2e5c3 / stardust Update pom.xml Report URL: https://github.com/apache/incubator-fury/actions/runs/8846044153 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" is working again!
The GitHub Actions job "Fury CI" on incubator-fury.git has succeeded. Run started by GitHub user stardustman (triggered by stardustman). Head commit for run: 195c8f3f6bc9fedce1b307f1b51a633c489e8a25 / stardust Update pom.xml fix using high version jdk compile to low version such as jdk8 using --release instead of -target Report URL: https://github.com/apache/incubator-fury/actions/runs/8845977899 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Lint PR" failed!
The GitHub Actions job "Lint PR" on incubator-fury.git has failed. Run started by GitHub user stardustman (triggered by stardustman). Head commit for run: 195c8f3f6bc9fedce1b307f1b51a633c489e8a25 / stardust Update pom.xml fix using high version jdk compile to low version such as jdk8 using --release instead of -target Report URL: https://github.com/apache/incubator-fury/actions/runs/8845977682 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[PR] Update pom.xml [incubator-fury]
stardustman opened a new pull request, #1579: URL: https://github.com/apache/incubator-fury/pull/1579 fix using high version jdk compile to low version such as jdk8 using --release instead of -target solve the https://github.com/apache/incubator-fury/issues/1577 > Starting JDK 9, the javac executable can accept the --release option to specify against which Java SE release you want to build the project. For example, you have JDK 11 installed and used by Maven, but you want to build the project against Java 8. The --release option ensures that the code is compiled following the rules of the programming language of the specified release, and that generated classes target the release as well as the public API of that release. This means that, unlike the [-source and -target options](https://maven.apache.org/plugins/maven-compiler-plugin/examples/set-compiler-source-and-target.html), the compiler will detect and generate an error when using APIs that don't exist in previous releases of Java SE. https://maven.apache.org/plugins/maven-compiler-plugin/examples/set-compiler-release.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: b6025e30f52d5422a744e629df8645e8e07b669e / chaokunyang lint code Report URL: https://github.com/apache/incubator-fury/actions/runs/8845926039 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 56e27a01cf98509b746eff9bcf14965aca726e94 / chaokunyang move Platform class to memory package Report URL: https://github.com/apache/incubator-fury/actions/runs/8845791304 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[PR] chore(java): move Platform class to memory package [incubator-fury]
chaokunyang opened a new pull request, #1578: URL: https://github.com/apache/incubator-fury/pull/1578 ## What does this PR do? This PR moves Platform class to memory package, since it's most ly for memory related operations ## Related issues ## Does this PR introduce any user-facing change? - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] refactor(java): replace Guava's TypeToken with self-made [incubator-fury]
chaokunyang commented on code in PR #1553:
URL: https://github.com/apache/incubator-fury/pull/1553#discussion_r1580701962
##
java/fury-core/src/main/java/org/apache/fury/reflect/TypeRef.java:
##
@@ -0,0 +1,828 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.fury.reflect;
+
+import static org.apache.fury.reflect.Types.asTypeVariableKeyOrNull;
+import static org.apache.fury.reflect.Types.newArrayType;
+import static org.apache.fury.reflect.Types.typeVariablesEquals;
+
+import java.lang.reflect.Array;
+import java.lang.reflect.GenericArrayType;
+import java.lang.reflect.Modifier;
+import java.lang.reflect.ParameterizedType;
+import java.lang.reflect.Type;
+import java.lang.reflect.TypeVariable;
+import java.lang.reflect.WildcardType;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Stream;
+import org.apache.fury.type.TypeUtils;
+
+public class TypeRef {
+
+ private final Type type;
+
+ /**
+ * Constructs a new type token of {@code T}.
+ *
+ * Clients create an empty anonymous subclass. This embeds the type
parameter in the anonymous
+ * class's type hierarchy, so we can reconstitute it at runtime despite
erasure.
+ *
+ * For example:
+ *
+ * {@code
+ * TypeToken> t = new TypeToken>() {};
+ * }
+ */
+ protected TypeRef() {
+this.type = capture();
+ }
+
+ private TypeRef(Class declaringClass) {
+this.type = declaringClass;
+ }
+
+ private TypeRef(Type type) {
+this.type = type;
+ }
+
+ /** Returns an instance of type token that wraps {@code type}. */
+ public static TypeRef of(Class clazz) {
+return new TypeRef<>(clazz);
+ }
+
+ /** Returns an instance of type token that wraps {@code type}. */
+ public static TypeRef of(Type type) {
+return new TypeRef<>(type);
+ }
+
+ /** Returns the captured type. */
+ private Type capture() {
+final Type superclass = getClass().getGenericSuperclass();
+if (!(superclass instanceof ParameterizedType)) {
+ throw new IllegalArgumentException(superclass + " isn't parameterized");
+}
+return ((ParameterizedType) superclass).getActualTypeArguments()[0];
+ }
+
+ /** Returns the represented type. */
+ public Type getType() {
+return type;
+ }
+
+ /**
+ * Returns the raw type of {@code T}. Formally speaking, if {@code T} is
returned by {@link
+ * java.lang.reflect.Method#getGenericReturnType}, the raw type is what's
returned by {@link
+ * java.lang.reflect.Method#getReturnType} of the same method object.
Specifically:
+ *
+ *
+ * If {@code T} is a {@code Class} itself, {@code T} itself is
returned.
+ * If {@code T} is a {@link ParameterizedType}, the raw type of the
parameterized type is
+ * returned.
+ * If {@code T} is a {@link GenericArrayType}, the returned type is
the corresponding array
+ * class. For example: {@code List[] => List[]}.
+ * If {@code T} is a type variable or a wildcard type, the raw type of
the first upper bound
+ * is returned. For example: {@code => Foo}.
+ *
+ */
+ public Class getRawType() {
+@SuppressWarnings("unchecked")
+Class result = (Class) TypeUtils.getRawType(type);
+return result;
+ }
+
+ private static Stream> getRawTypes(Type... types) {
+return Arrays.stream(types)
+.flatMap(
+type -> {
+ if (type instanceof TypeVariable) {
+return getRawTypes(((TypeVariable) type).getBounds());
+ } else if (type instanceof WildcardType) {
+return getRawTypes(((WildcardType) type).getUpperBounds());
+ } else if (type instanceof ParameterizedType) {
+return Stream.of((Class) ((ParameterizedType)
type).getRawType());
+ } else if (type instanceof Class) {
+return Stream.of((Class) type);
+ } else if (type instanceof GenericArrayType) {
+Class rawType =
+getArrayClass(
+of(((GenericArrayType)
type).getGenericComponentType()).getRawType());
+
(incubator-fury) branch main updated: feat(java): support deserialize unexisted array/enum classes (#1569)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury.git
The following commit(s) were added to refs/heads/main by this push:
new d59c57c3 feat(java): support deserialize unexisted array/enum classes
(#1569)
d59c57c3 is described below
commit d59c57c33e5d4b2f5fc1c1af466113abbb1a8457
Author: Shawn Yang
AuthorDate: Fri Apr 26 16:48:48 2024 +0800
feat(java): support deserialize unexisted array/enum classes (#1569)
## What does this PR do?
This PR support sdeserialize unexisted array/enum object.
For type compatible serialization, deserialization process may not have
the class in serialization peer. In such cases, we should support skip
object data of unexisted array/enum
## Related issues
Closes #1564
## Does this PR introduce any user-facing change?
- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?
## Benchmark
---
.../java/org/apache/fury/builder/CodecBuilder.java | 4 +-
.../java/org/apache/fury/codegen/JaninoUtils.java | 10 ++
.../org/apache/fury/meta/MetaStringEncoder.java| 5 +
.../java/org/apache/fury/resolver/ClassInfo.java | 39 --
.../org/apache/fury/resolver/ClassResolver.java| 136 +-
.../org/apache/fury/resolver/MetaStringBytes.java | 6 +-
.../apache/fury/serializer/ArraySerializers.java | 152 -
.../apache/fury/serializer/CodegenSerializer.java | 7 +-
.../fury/serializer/UnexistedClassSerializers.java | 18 +++
.../main/java/org/apache/fury/type/TypeUtils.java | 8 ++
.../java/org/apache/fury/util/ReflectionUtils.java | 30 +++-
.../java/org/apache/fury/util/StringUtils.java | 8 ++
.../serializer/UnexistedClassSerializersTest.java | 130 ++
13 files changed, 505 insertions(+), 48 deletions(-)
diff --git
a/java/fury-core/src/main/java/org/apache/fury/builder/CodecBuilder.java
b/java/fury-core/src/main/java/org/apache/fury/builder/CodecBuilder.java
index d8ebc77c..a9753fec 100644
--- a/java/fury-core/src/main/java/org/apache/fury/builder/CodecBuilder.java
+++ b/java/fury-core/src/main/java/org/apache/fury/builder/CodecBuilder.java
@@ -345,7 +345,9 @@ public abstract class CodecBuilder {
if (duplicatedFields.contains(fieldName) ||
!sourcePublicAccessible(beanClass)) {
return unsafeSetField(bean, d, value);
}
-if (!d.isFinalField() && Modifier.isPublic(d.getModifiers())) {
+if (!d.isFinalField()
+&& Modifier.isPublic(d.getModifiers())
+&& Modifier.isPublic(d.getRawType().getModifiers())) {
return new Expression.SetField(bean, fieldName, value);
} else if (d.getWriteMethod() != null &&
Modifier.isPublic(d.getWriteMethod().getModifiers())) {
return new Invoke(bean, d.getWriteMethod().getName(), value);
diff --git
a/java/fury-core/src/main/java/org/apache/fury/codegen/JaninoUtils.java
b/java/fury-core/src/main/java/org/apache/fury/codegen/JaninoUtils.java
index 90b8cc3c..bddd900a 100644
--- a/java/fury-core/src/main/java/org/apache/fury/codegen/JaninoUtils.java
+++ b/java/fury-core/src/main/java/org/apache/fury/codegen/JaninoUtils.java
@@ -51,6 +51,16 @@ import org.codehaus.janino.util.ClassFile;
public class JaninoUtils {
private static final Logger LOG = LoggerFactory.getLogger(JaninoUtils.class);
+ public static Class compileClass(
+ ClassLoader loader, String pkg, String className, String code) {
+ByteArrayClassLoader classLoader = compile(loader, new CompileUnit(pkg,
className, code));
+try {
+ return classLoader.loadClass(StringUtils.isBlank(pkg) ? className : pkg
+ "." + className);
+} catch (ClassNotFoundException e) {
+ throw new RuntimeException(e);
+}
+ }
+
public static ByteArrayClassLoader compile(
ClassLoader parentClassLoader, CompileUnit... compileUnits) {
final Map classes = toBytecode(parentClassLoader,
compileUnits);
diff --git
a/java/fury-core/src/main/java/org/apache/fury/meta/MetaStringEncoder.java
b/java/fury-core/src/main/java/org/apache/fury/meta/MetaStringEncoder.java
index fe6796a7..6c8d69c2 100644
--- a/java/fury-core/src/main/java/org/apache/fury/meta/MetaStringEncoder.java
+++ b/java/fury-core/src/main/java/org/apache/fury/meta/MetaStringEncoder.java
@@ -25,6 +25,11 @@ import org.apache.fury.util.Preconditions;
/** Encodes plain text strings into MetaString objects with specified encoding
mechanisms. */
public class MetaStringEncoder {
+ public static final MetaStringEncoder PACKAGE_ENCODER = new
MetaStringEncoder('.', '_');
+ public static final MetaStringDecoder PACKAGE_DECODER = new
MetaStringDecoder('.', '_');
+ public static final MetaStringEncoder TYPE_NAME_ENCODER = new
MetaStringEncoder('$', '_');
+ public static f
Re: [PR] feat(java): support deserialize unexisted array/enum classes [incubator-fury]
chaokunyang merged PR #1569: URL: https://github.com/apache/incubator-fury/pull/1569 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" is working again!
The GitHub Actions job "Fury CI" on incubator-fury.git has succeeded. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: d59c57c33e5d4b2f5fc1c1af466113abbb1a8457 / Shawn Yang feat(java): support deserialize unexisted array/enum classes (#1569) ## What does this PR do? This PR support sdeserialize unexisted array/enum object. For type compatible serialization, deserialization process may not have the class in serialization peer. In such cases, we should support skip object data of unexisted array/enum ## Related issues Closes #1564 ## Does this PR introduce any user-facing change? - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark Report URL: https://github.com/apache/incubator-fury/actions/runs/8845640013 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] 【JAVA】deserialize class not found USE withRefTracking(true).requireClassRegistration(false).withCompatibleMode(CompatibleMode.COMPATIBLE).buildThreadSafeFury() [incubator-fury]
chaokunyang closed issue #1564: 【JAVA】deserialize class not found USE withRefTracking(true).requireClassRegistration(false).withCompatibleMode(CompatibleMode.COMPATIBLE).buildThreadSafeFury() URL: https://github.com/apache/incubator-fury/issues/1564 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(golang): meta string encoding algorithm in golang [incubator-fury]
chaokunyang commented on code in PR #1566:
URL: https://github.com/apache/incubator-fury/pull/1566#discussion_r1580673646
##
go/fury/meta/meta_string_test.go:
##
@@ -0,0 +1,82 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package meta
+
+import (
+ "github.com/stretchr/testify/require"
+ "testing"
+)
+
+func TestEncodeAndDecodeMetaString(t *testing.T) {
+ var data MetaString
+ var dst string
+ var err error
+
+ str2bits := map[string]int{
+ // "abc_def" should be encoded as 0|0|00, 001|00010|,
11011|000, 11|00100|0, 0101
+ "abc_def":5,
+ "org.apache.fury.benchmark.data": 5,
+ "HelloWorld__123.2024": 6,
+ "MediaContent": 5,
+ "Apple_banana": 5,
+ "你好，世界": 0, // not used
Review Comment:
@qingoba Could you use golang format?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(golang): meta string encoding algorithm in golang [incubator-fury]
LiangliangSui commented on code in PR #1566:
URL: https://github.com/apache/incubator-fury/pull/1566#discussion_r1580654079
##
go/fury/meta/meta_string_test.go:
##
@@ -0,0 +1,82 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package meta
+
+import (
+ "github.com/stretchr/testify/require"
+ "testing"
+)
+
+func TestEncodeAndDecodeMetaString(t *testing.T) {
+ var data MetaString
+ var dst string
+ var err error
+
+ str2bits := map[string]int{
+ // "abc_def" should be encoded as 0|0|00, 001|00010|,
11011|000, 11|00100|0, 0101
+ "abc_def":5,
+ "org.apache.fury.benchmark.data": 5,
+ "HelloWorld__123.2024": 6,
+ "MediaContent": 5,
+ "Apple_banana": 5,
+ "你好，世界": 0, // not used
Review Comment:
Nit: format
##
go/fury/meta/meta_string_test.go:
##
@@ -0,0 +1,82 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package meta
+
+import (
+ "github.com/stretchr/testify/require"
+ "testing"
+)
+
+func TestEncodeAndDecodeMetaString(t *testing.T) {
+ var data MetaString
+ var dst string
+ var err error
+
+ str2bits := map[string]int{
+ // "abc_def" should be encoded as 0|0|00, 001|00010|,
11011|000, 11|00100|0, 0101
+ "abc_def":5,
+ "org.apache.fury.benchmark.data": 5,
+ "HelloWorld__123.2024": 6,
+ "MediaContent": 5,
+ "Apple_banana": 5,
+ "你好，世界": 0, // not used
+ }
+ str2encoding := map[string]Encoding{
+ "abc_def":LOWER_SPECIAL,
+ "org.apache.fury.benchmark.data": LOWER_SPECIAL,
+ "MediaContent": ALL_TO_LOWER_SPECIAL,
+ "HelloWorld__123.2024": LOWER_UPPER_DIGIT_SPECIAL,
+ "Apple_banana": FIRST_TO_LOWER_SPECIAL,
+ "你好，世界": UTF_8,
Review Comment:
Nit: format
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(golang): meta string encoding algorithm in golang [incubator-fury]
chaokunyang commented on PR #1566: URL: https://github.com/apache/incubator-fury/pull/1566#issuecomment-2078874906 @qingoba Ci has some failures. Could you take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] [Java] Memory issue about FuryInputStream [incubator-fury]
yihwang6 commented on issue #1576: URL: https://github.com/apache/incubator-fury/issues/1576#issuecomment-2078845580 Ok, but there may still be a large buffer size when deserializing a large object. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[I] serialize ByteBuffer report error [incubator-fury]
stardustman opened a new issue, #1577:
URL: https://github.com/apache/incubator-fury/issues/1577
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/incubator-fury/issues) and found no similar
issues.
### Version
1. jdk1.8.0_241
2. fury v0.4.1
### Component(s)
Java
### Minimal reproduce step
test code as follows:
```java
public class FurySerde {
// Note that Fury instances should be reused between
// multiple serializations of different objects.
private static final ThreadSafeFury fury = new
ThreadLocalFury(classLoader -> {
Fury f = Fury.builder()
.withLanguage(Language.JAVA)
.withClassLoader(classLoader)
.requireClassRegistration(false)
.build();
return f;
});
public static byte[] encoder(Object object) {
return fury.serialize(object);
}
public static T decoder(byte[] bytes) {
return (T) fury.deserialize(bytes);
}
public static void main(String[] args) {
byte len = 10;
ByteBuffer byteBuffer = ByteBuffer.allocate(len);
for (int i = 0; i < len; i++) {
byteBuffer.put((byte) i);
}
System.out.println(FurySerde.encoder(byteBuffer).length);
}
}
```
### What did you expect to see?
get the size of serialization byte array
### What did you see instead?
Exception in thread "main" java.lang.NoSuchMethodError:
java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
at io.fury.memory.MemoryBuffer.put(MemoryBuffer.java:404)
at io.fury.memory.MemoryBuffer.write(MemoryBuffer.java:1935)
at io.fury.memory.MemoryBuffer.write(MemoryBuffer.java:1928)
at
io.fury.serializer.BufferObject$ByteBufferBufferObject.writeTo(BufferObject.java:59)
at io.fury.Fury.writeBufferObject(Fury.java:571)
at
io.fury.serializer.BufferSerializers$ByteBufferSerializer.write(BufferSerializers.java:45)
at
io.fury.serializer.BufferSerializers$ByteBufferSerializer.write(BufferSerializers.java:36)
at io.fury.Fury.writeData(Fury.java:551)
at io.fury.Fury.writeRef(Fury.java:344)
at io.fury.Fury.write(Fury.java:319)
at io.fury.Fury.serialize(Fury.java:255)
at io.fury.Fury.serialize(Fury.java:221)
at io.fury.ThreadLocalFury.serialize(ThreadLocalFury.java:67)
at benchmarks.serde.FurySerde.encoder(FurySerde.java:37)
at benchmarks.serde.FurySerde.main(FurySerde.java:51)
### Anything Else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user qingoba (triggered by qingoba). Head commit for run: 28646b0aeee8a1cc27d48c561f03ad0dbd759825 / qingoba <51332889+qing...@users.noreply.github.com> Update go/fury/meta/meta_string_test.go Co-authored-by: Shawn Yang Report URL: https://github.com/apache/incubator-fury/actions/runs/8844827650 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user qingoba (triggered by qingoba). Head commit for run: c89d7e709301e8204945949cf8da2bb7b35a63df / qingoba <51332889+qing...@users.noreply.github.com> Update go/fury/meta/meta_string_test.go Co-authored-by: Shawn Yang Report URL: https://github.com/apache/incubator-fury/actions/runs/8844829129 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] [Java] Memory issue about FuryInputStream [incubator-fury]
chaokunyang commented on issue #1576: URL: https://github.com/apache/incubator-fury/issues/1576#issuecomment-2078811375 `org.apache.fury.io.FuryInputStream#shrinkBuffer` will be invoked after the deserialziation finished -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] feat(golang): meta string encoding algorithm in golang [incubator-fury]
chaokunyang commented on code in PR #1566:
URL: https://github.com/apache/incubator-fury/pull/1566#discussion_r1580608168
##
go/fury/meta/meta_string_test.go:
##
@@ -0,0 +1,82 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package meta
+
+import (
+ "github.com/stretchr/testify/require"
+ "testing"
+)
+
+func TestEncodeAndDecodeMetaString(t *testing.T) {
+ var data MetaString
+ var dst string
+ var err error
+
+ str2bits := map[string]int{
+ // "abc_def" should be encoded as 0|0|00, 001|00010|,
11011|000, 11|00100|0, 0101
+ "abc_def":5,
+ "org.apache.fury.benchmark.data": 5,
+ "HelloWorld__123.2024": 6,
+ "MediaContent": 5,
+ "Apple_banana": 5,
+ "欲海回狂": 0, // not used
Review Comment:
How about replacing it to "你好，世界"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
