Re: [PR] chore(java): Add comments for loop unrolling [incubator-fury]
LiangliangSui commented on code in PR #1450:
URL: https://github.com/apache/incubator-fury/pull/1450#discussion_r1549004814
##
java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java:
##
@@ -1316,24 +1316,15 @@ public int readPositiveVarInt() {
}
private int readPositiveVarIntSlow() {
-int b = readByte();
-int result = b & 0x7F;
-if ((b & 0x80) != 0) {
+int shift = 0;
Review Comment:
Updated.
BTW, do we need to add the benchmark code (sample code above) that uses loop
and does not use loop in the `MemoryBuffer#readPositiveVarIntSlow` method to
the benchmark directory?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] [Java] Does Fury preregister common Java collection classes, or do I need to register them manually? [incubator-fury]
LiangliangSui commented on issue #1456: URL: https://github.com/apache/incubator-fury/issues/1456#issuecomment-2033562567 Should we indicate in https://github.com/apache/incubator-fury/blob/main/docs/guide/java_object_graph_guide.md which classes have been registered and which Serializers have been registered? Currently these things are in the documentation It's rather vague. @chaokunyang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] [Java] Does Fury preregister common Java collection classes, or do I need to register them manually? [incubator-fury]
LiangliangSui commented on issue #1456: URL: https://github.com/apache/incubator-fury/issues/1456#issuecomment-2033558042 @cdxf Details can be viewed at https://github.com/apache/incubator-fury/blob/main/java/fury-core/src/main/java/org/apache/fury/resolver/ClassResolver.java#L262 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] [Java] Does Fury preregister common Java collection classes, or do I need to register them manually? [incubator-fury]
chaokunyang commented on issue #1456: URL: https://github.com/apache/incubator-fury/issues/1456#issuecomment-2033554622 They are all registered by fury, you can access ClassResolver first a detailed registered list -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[I] [Java] Does Fury preregister common Java collection classes, or do I need to register them manually? [incubator-fury]
cdxf opened a new issue, #1456: URL: https://github.com/apache/incubator-fury/issues/1456 I would like to know if Fury automatically handles the registration of these common Java collection classes: ```ArrayList```, ```HashMap``` or classes that are not part of the public API (which I can't access directly), like: ```ImmutableCollections.Map1```, ```ImmutableCollections.List12```, ```ImmutableCollections.ListN```... Or if I need to explicitly register them myself when using Fury? And what about Interface class, do I need to register the Interface class also (for example ```List.class```) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] fix(license): fix license issue [incubator-fury]
chaokunyang commented on PR #1446: URL: https://github.com/apache/incubator-fury/pull/1446#issuecomment-2033504154 Hi @pjfanning , could you help review this pr? For move license section from notice to license, we can resolve it in another pr -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] deserialize failing with java.lang.OutOfMemoryError: Java heap space [incubator-fury]
chaokunyang commented on issue #1454: URL: https://github.com/apache/incubator-fury/issues/1454#issuecomment-2033435465 > > Could you disable sync compilation and try it again? There are some inconsistentency between compile mode and interpretor mode > > Assuming you want me to `withAsyncCompilation(false)`, yes? Yes, that's what I mean. Does it still fail? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] deserialize fails with java.lang.NullPointerException [incubator-fury]
andyczerwonka commented on issue #1455: URL: https://github.com/apache/incubator-fury/issues/1455#issuecomment-2033356454 > Hi @andyczerwonka , could you provide reproduction code here? I'll debug into your code to find out what happened Unfortunately we're in the same situation as we are [here](https://github.com/apache/incubator-fury/issues/1454#issuecomment-2033319371). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] deserialize failing with java.lang.OutOfMemoryError: Java heap space [incubator-fury]
andyczerwonka commented on issue #1454: URL: https://github.com/apache/incubator-fury/issues/1454#issuecomment-2033354605 > Could you disable sync compilation and try it again? There are some inconsistentency between compile mode and interpretor mode Assuming you want me to `withAsyncCompilation(false)`, yes? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] deserialize failing with java.lang.OutOfMemoryError: Java heap space [incubator-fury]
chaokunyang commented on issue #1454: URL: https://github.com/apache/incubator-fury/issues/1454#issuecomment-209872 Could you disable sync compilation and try it again? There are some inconsistentency between compile mode and interpretor mode -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] deserialize failing with java.lang.OutOfMemoryError: Java heap space [incubator-fury]
andyczerwonka commented on issue #1454: URL: https://github.com/apache/incubator-fury/issues/1454#issuecomment-2033319371 > Hi @andyczerwonka , can I reproduce this error in my local environment? Unfortunately not, because the code is proprietary and I cannot share it. I was hoping the stack tracewould give us some insight into where it's failing so I could create a sample model. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] deserialize failing with java.lang.OutOfMemoryError: Java heap space [incubator-fury]
andyczerwonka commented on issue #1454: URL: https://github.com/apache/incubator-fury/issues/1454#issuecomment-2033317854 > And could you provide your Fury creation code here too? ```scala private val fury = Fury .builder() .withLanguage(Language.JAVA) .requireClassRegistration(false) .withScalaOptimizationEnabled(true) .withRefTracking(true) .withStringCompressed(true) .withLongCompressed(true) .withIntCompressed(true) .withAsyncCompilation(true) .buildThreadSafeFury() ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] deserialize failing with java.lang.OutOfMemoryError: Java heap space [incubator-fury]
chaokunyang commented on issue #1454: URL: https://github.com/apache/incubator-fury/issues/1454#issuecomment-2033312961 And could you provide your Fury creation code here too? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] deserialize failing with java.lang.OutOfMemoryError: Java heap space [incubator-fury]
chaokunyang commented on issue #1454: URL: https://github.com/apache/incubator-fury/issues/1454#issuecomment-2033312511 Hi @andyczerwonka , can I reproduce this error in my local environment? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] deserialize fails with java.lang.NullPointerException [incubator-fury]
chaokunyang commented on issue #1455: URL: https://github.com/apache/incubator-fury/issues/1455#issuecomment-2033309968 Hi @andyczerwonka , could you provide reproduction code? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury-site) branch deploy updated: deploy: d6f914d6b2da8473949c4b42ff1f8a303f968117
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch deploy
in repository https://gitbox.apache.org/repos/asf/incubator-fury-site.git
The following commit(s) were added to refs/heads/deploy by this push:
new 54d46ac deploy: d6f914d6b2da8473949c4b42ff1f8a303f968117
54d46ac is described below
commit 54d46ac58bff4841fcd86977ce9311ded9175711
Author: theweipeng
AuthorDate: Wed Apr 3 00:09:46 2024 +
deploy: d6f914d6b2da8473949c4b42ff1f8a303f968117
---
404.html | 6 +++---
assets/js/6ba6e33f.12094b45.js| 1 -
assets/js/6ba6e33f.27132e1c.js| 1 +
assets/js/72b486c3.2f73747b.js| 1 +
assets/js/72b486c3.6813874e.js| 1 -
assets/js/81ff565a.8c365ffc.js| 1 -
assets/js/81ff565a.b791fcff.js| 1 +
assets/js/935f2afb.4cc32fe6.js| 1 -
assets/js/935f2afb.87230918.js| 1 +
assets/js/main.4e829a60.js| 2 --
assets/js/main.5863021d.js| 2 ++
...{main.4e829a60.js.LICENSE.txt => main.5863021d.js.LICENSE.txt} | 0
assets/js/{runtime~main.3f343efb.js => runtime~main.e1125dd7.js} | 2 +-
blog/archive/index.html | 6 +++---
blog/fury_0_1_0_release/index.html| 6 +++---
blog/fury_0_1_1_release/index.html| 6 +++---
blog/fury_0_1_2_release/index.html| 6 +++---
blog/fury_0_2_0_release/index.html| 6 +++---
blog/fury_0_2_1_release/index.html| 6 +++---
blog/fury_0_3_0_release/index.html| 6 +++---
blog/fury_0_3_1_release/index.html| 6 +++---
blog/fury_0_4_0_release/index.html| 6 +++---
blog/fury_0_4_1_release/index.html| 6 +++---
.../index.html| 6 +++---
blog/index.html | 6 +++---
blog/tags/fury/index.html | 6 +++---
blog/tags/index.html | 6 +++---
docs/guide/development/index.html | 6 +++---
docs/guide/graalvm_guide/index.html | 6 +++---
docs/guide/java_object_graph_guide/index.html | 6 +++---
docs/guide/row_format_guide/index.html| 6 +++---
docs/guide/scala_guide/index.html | 6 +++---
docs/guide/xlang_object_graph_guide/index.html| 6 +++---
docs/guide/xlang_type_mapping/index.html | 6 +++---
docs/introduction/benchmark/index.html| 6 +++---
docs/introduction/features/index.html | 6 +++---
docs/introduction/index.html | 6 +++---
docs/specification/fury_java_serialization_spec/index.html| 8
docs/specification/fury_row_format_spec/index.html| 8
docs/specification/fury_xlang_serialization_spec/index.html | 8
docs/start/install/index.html | 6 +++---
docs/start/usage/index.html | 6 +++---
index.html| 6 +++---
lunr-index-1712078575649.json | 1 -
lunr-index-1712102951567.json | 1 +
lunr-index.json | 2 +-
markdown-page/index.html | 6 +++---
search-doc-1712078575649.json => search-doc-1712102951567.json| 2 +-
search-doc.json | 2 +-
49 files changed, 110 insertions(+), 110 deletions(-)
diff --git a/404.html b/404.html
index 0acc54b..8da09af 100644
--- a/404.html
+++ b/404.html
@@ -5,11 +5,11 @@
Page Not Found | Apache Fury (incubating)https://fury.apache.org/404.html";>
-
-
+
+
-!function(){function
t(t){document.documentElement.setAttribute("data-theme",t)}var
e=function(){try{return new
URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}}()||function(){try{return
localStorage.getItem("theme")}catch(t){}}();t(null!==e?e:"light")}(),function(){try{const
c=new URLSearchParams(window.location.search).entries();for(var[t,e]of
c)if(t.startsWith("docusaurus-data-")){var
a=t.replace("docusaurus-data-","data-");docum
(incubator-fury-site) branch main updated: add fury serialization specification tab (#108)
This is an automated email from the ASF dual-hosted git repository.
wangweipeng pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury-site.git
The following commit(s) were added to refs/heads/main by this push:
new d6f914d add fury serialization specification tab (#108)
d6f914d is described below
commit d6f914d6b2da8473949c4b42ff1f8a303f968117
Author: Shawn Yang
AuthorDate: Wed Apr 3 08:08:34 2024 +0800
add fury serialization specification tab (#108)
---
docusaurus.config.ts | 6 ++
sidebars.ts | 1 +
2 files changed, 7 insertions(+)
diff --git a/docusaurus.config.ts b/docusaurus.config.ts
index 72f9f42..a51d9e6 100644
--- a/docusaurus.config.ts
+++ b/docusaurus.config.ts
@@ -78,6 +78,12 @@ const config: Config = {
position: 'right',
label: 'Guide',
},
+{
+ type: 'docSidebar',
+ sidebarId: 'specificationSidebar',
+ position: 'right',
+ label: 'Specification',
+},
{to: '/blog', label: 'Blog', position: 'right'},
{
type: 'dropdown',
diff --git a/sidebars.ts b/sidebars.ts
index b4b97e3..d539b14 100644
--- a/sidebars.ts
+++ b/sidebars.ts
@@ -15,6 +15,7 @@ const sidebars: SidebarsConfig = {
// tutorialSidebar: [{type: 'autogenerated', dirName: '.'}],
startSidebar: [{type: 'autogenerated', dirName: 'start'}],
introductionSidebar: [{type: 'autogenerated', dirName: 'introduction'}],
+ specificationSidebar: [{type: 'autogenerated', dirName: 'specification'}],
guideSidebar: [{type: 'autogenerated', dirName: 'guide'}],
// But you can create a sidebar manually
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] add fury serialization specification tab [incubator-fury-site]
theweipeng merged PR #108: URL: https://github.com/apache/incubator-fury-site/pull/108 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[I] deserialize fails with java.lang.NullPointerException [incubator-fury]
andyczerwonka opened a new issue, #1455: URL: https://github.com/apache/incubator-fury/issues/1455 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-fury/issues) and found no similar issues. ### Version 0.5.0-rc2 ### Component(s) Java ### Minimal reproduce step We are attempting to replace the latest snapshot with rc2, and we're getting the following exception ``` org.apache.fury.exception.DeserializationException: Deserialize failed, read objects are: [, null, scala.collection.generic.DefaultSerializationProxy@5d2828c9, scala.collection.IterableFactory$ToFactory@3a082ff4, null] at org.apache.fury.Fury.handleReadFailed(Fury.java:784) at org.apache.fury.Fury.deserialize(Fury.java:755) at org.apache.fury.Fury.deserialize(Fury.java:685) at org.apache.fury.ThreadLocalFury.deserialize(ThreadLocalFury.java:99) at io.citrine.mithril.modules.predictors.Predictor$.$anonfun$decode$1(Predictor.scala:111) at scala.util.Try$.apply(Try.scala:210) at io.citrine.mithril.modules.predictors.Predictor$.decode(Predictor.scala:107) at io.citrine.mithrilworkbench.Main$.$anonfun$tryToDecodeThisPredictor$1(Main.scala:334) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scribe.LoggerSupport.elapsed(LoggerSupport.scala:67) at scribe.LoggerSupport.elapsed$(LoggerSupport.scala:62) at scribe.package$.elapsed(package.scala:8) at io.citrine.mithrilworkbench.Main$.tryToDecodeThisPredictor(Main.scala:323) at io.citrine.mithrilworkbench.Main$.$anonfun$main$5(Main.scala:351) at io.citrine.mithrilworkbench.Main$.$anonfun$main$5$adapted(Main.scala:351) at mainargs.Invoker$.$anonfun$invoke0$3(Invoker.scala:63) at mainargs.Result.flatMap(Result.scala:13) at mainargs.Result.flatMap$(Result.scala:12) at mainargs.Result$Success.flatMap(Result.scala:23) at mainargs.Invoker$.invoke0(Invoker.scala:62) at mainargs.Invoker$.invoke(Invoker.scala:72) at mainargs.Invoker$.$anonfun$runMains$1(Invoker.scala:92) at mainargs.Result.flatMap(Result.scala:13) at mainargs.Result.flatMap$(Result.scala:12) at mainargs.Result$Success.flatMap(Result.scala:23) at mainargs.Invoker$.groupArgs$1(Invoker.scala:92) at mainargs.Invoker$.runMains(Invoker.scala:107) at mainargs.ParserForMethods.runRaw0(Parser.scala:170) at mainargs.ParserForMethods.runEither(Parser.scala:108) at mainargs.ParserForMethods.runEither(Parser.scala:151) at mainargs.ParserForMethods.runOrExit(Parser.scala:56) at io.citrine.mithrilworkbench.Main$.main(Main.scala:352) at io.citrine.mithrilworkbench.Main.main(Main.scala) Caused by: java.lang.NullPointerException at scala.collection.IterableFactory$ToFactory.newBuilder(Factory.scala:275) at scala.collection.generic.DefaultSerializationProxy.readObject(DefaultSerializationProxy.scala:48) at org.apache.fury.serializer.ObjectStreamSerializer.read(ObjectStreamSerializer.java:221) at org.apache.fury.serializer.ReplaceResolveSerializer.readObject(ReplaceResolveSerializer.java:316) at org.apache.fury.serializer.ReplaceResolveSerializer.read(ReplaceResolveSerializer.java:305) at org.apache.fury.Fury.readData(Fury.java:884) at org.apache.fury.serializer.ReplaceResolveSerializer.read(ReplaceResolveSerializer.java:284) at org.apache.fury.serializer.collection.CollectionSerializers$JDKCompatibleCollectionSerializer.read(CollectionSerializers.java:574) at org.apache.fury.Fury.readDataInternal(Fury.java:916) at org.apache.fury.Fury.readRef(Fury.java:831) at org.apache.fury.serializer.ObjectSerializer.readOtherFieldValue(ObjectSerializer.java:438) at org.apache.fury.serializer.ObjectSerializer.readAndSetFields(ObjectSerializer.java:382) at org.apache.fury.serializer.ObjectSerializer.read(ObjectSerializer.java:316) at org.apache.fury.Fury.readDataInternal(Fury.java:916) at org.apache.fury.Fury.readRef(Fury.java:831) at org.apache.fury.serializer.ObjectSerializer.readOtherFieldValue(ObjectSerializer.java:438) at org.apache.fury.serializer.ObjectSerializer.readAndSetFields(ObjectSerializer.java:382) at org.apache.fury.serializer.ObjectSerializer.read(ObjectSerializer.java:316) at org.apache.fury.Fury.readDataInternal(Fury.java:916) at org.apache.fury.Fury.readRef(Fury.java:818) at org.apache.fury.serializer.ObjectStreamSerializer$FuryObjectInputStream.readObjectOverride(ObjectStreamSerializer.java:693) at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:480) at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:447)
[I] Deserialize failing with java.lang.OutOfMemoryError: Java heap space [incubator-fury]
andyczerwonka opened a new issue, #1454:
URL: https://github.com/apache/incubator-fury/issues/1454
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/incubator-fury/issues) and found no similar
issues.
### Version
latest 0.5.0 snapshot
### Component(s)
Java
### Minimal reproduce step
Here's the exception we're seeing.
```
org.apache.fury.exception.DeserializationException: Deserialize failed, read
objects are: [PredictorStorageData(null,null,null),
ConfigurationMetadata(Perfume SMILES price-process,Default Graph Model
generated from data inspection.,null), Some(null), {}]
at org.apache.fury.Fury.handleReadFailed(Fury.java:784)
at org.apache.fury.Fury.deserialize(Fury.java:755)
at org.apache.fury.Fury.deserialize(Fury.java:685)
at org.apache.fury.ThreadLocalFury.deserialize(ThreadLocalFury.java:99)
at
io.citrine.mithril.modules.predictors.Predictor$.$anonfun$decode$1(Predictor.scala:111)
at scala.util.Try$.apply(Try.scala:210)
at
io.citrine.mithril.modules.predictors.Predictor$.decode(Predictor.scala:107)
at
io.citrine.mithrilworkbench.Main$.$anonfun$tryToDecodeThisPredictor$1(Main.scala:330)
at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at scribe.LoggerSupport.elapsed(LoggerSupport.scala:67)
at scribe.LoggerSupport.elapsed$(LoggerSupport.scala:62)
at scribe.package$.elapsed(package.scala:8)
at
io.citrine.mithrilworkbench.Main$.tryToDecodeThisPredictor(Main.scala:323)
at io.citrine.mithrilworkbench.Main$.$anonfun$main$5(Main.scala:340)
at
io.citrine.mithrilworkbench.Main$.$anonfun$main$5$adapted(Main.scala:340)
at mainargs.Invoker$.$anonfun$invoke0$3(Invoker.scala:63)
at mainargs.Result.flatMap(Result.scala:13)
at mainargs.Result.flatMap$(Result.scala:12)
at mainargs.Result$Success.flatMap(Result.scala:23)
at mainargs.Invoker$.invoke0(Invoker.scala:62)
at mainargs.Invoker$.invoke(Invoker.scala:72)
at mainargs.Invoker$.$anonfun$runMains$1(Invoker.scala:92)
at mainargs.Result.flatMap(Result.scala:13)
at mainargs.Result.flatMap$(Result.scala:12)
at mainargs.Result$Success.flatMap(Result.scala:23)
at mainargs.Invoker$.groupArgs$1(Invoker.scala:92)
at mainargs.Invoker$.runMains(Invoker.scala:107)
at mainargs.ParserForMethods.runRaw0(Parser.scala:170)
at mainargs.ParserForMethods.runEither(Parser.scala:108)
at mainargs.ParserForMethods.runEither(Parser.scala:151)
at mainargs.ParserForMethods.runOrExit(Parser.scala:56)
at io.citrine.mithrilworkbench.Main$.main(Main.scala:341)
at io.citrine.mithrilworkbench.Main.main(Main.scala)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.base/java.lang.reflect.Array.multiNewArray(Native Method)
at java.base/java.lang.reflect.Array.newInstance(Array.java:114)
at
org.apache.fury.serializer.ArraySerializers$ObjectArraySerializer.newArray(ArraySerializers.java:177)
at
org.apache.fury.serializer.ArraySerializers$ObjectArraySerializer.read(ArraySerializers.java:126)
at
org.apache.fury.serializer.ArraySerializers$ObjectArraySerializer.read(ArraySerializers.java:43)
at org.apache.fury.Fury.readDataInternal(Fury.java:916)
at org.apache.fury.Fury.readRef(Fury.java:831)
at
org.apache.fury.serializer.collection.AbstractMapSerializer.generalJavaRead(AbstractMapSerializer.java:568)
at
org.apache.fury.serializer.collection.AbstractMapSerializer.genericJavaRead(AbstractMapSerializer.java:443)
at
org.apache.fury.serializer.collection.AbstractMapSerializer.readElements(AbstractMapSerializer.java:435)
at
org.apache.fury.serializer.collection.MapSerializer.read(MapSerializer.java:46)
at
org.apache.fury.serializer.collection.MapSerializer.read(MapSerializer.java:27)
at org.apache.fury.Fury.readDataInternal(Fury.java:916)
at org.apache.fury.Fury.readRef(Fury.java:831)
at
org.apache.fury.serializer.ObjectSerializer.readOtherFieldValue(ObjectSerializer.java:438)
at
org.apache.fury.serializer.ObjectSerializer.readAndSetFields(ObjectSerializer.java:382)
at
org.apache.fury.serializer.ObjectSerializer.read(ObjectSerializer.java:316)
at org.apache.fury.Fury.readDataInternal(Fury.java:916)
at org.apache.fury.Fury.readRef(Fury.java:831)
at
org.apache.fury.serializer.ObjectSerializer.readOtherFieldValue(ObjectSerializer.java:438)
at
org.apache.fury.serializer.ObjectSerializer.readAndSetFields(ObjectSerializer.java:382)
at
org.apache.fury.serializer.ObjectSerializer.read(ObjectSerializer.java:316)
at org.apache.fury.Fury.readDataInternal(Fury.java:916)
at org.apache.fury.Fury.readRef(Fury.ja
Re: [I] Fury fails when deserializing from InputStream [incubator-fury]
chaokunyang commented on issue #1435: URL: https://github.com/apache/incubator-fury/issues/1435#issuecomment-2032670945 In those days, objects are getting bigger and bigger, the steaming is important to reduce latency for big objects, especially for big data and tensor computing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" is working again!
The GitHub Actions job "Fury CI" on incubator-fury.git has succeeded. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 700181ae578166bc08a54181cd5fc8e76cbcd3e3 / LiangliangSui <116876207+liangliang...@users.noreply.github.com> chore(java): Replace io.* in the benchmark with org.apache.fury (#1453) Signed-off-by: LiangliangSui Report URL: https://github.com/apache/incubator-fury/actions/runs/8526898763 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury-site): Workflow run "Deploy" is working again!
The GitHub Actions job "Deploy" on incubator-fury-site.git has succeeded. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 3215ec1cd4bdb158dabb2b3e60ccd1da2fe98420 / chaokunyang 🔄 created local 'docs/specification/' from remote 'docs/specification/' Report URL: https://github.com/apache/incubator-fury-site/actions/runs/8526898420 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user chaokunyang (triggered by chaokunyang). Head commit for run: 997c84cd1c35afca1d86770306f6f74dd5bbcaae / chaokunyang make fury support streaming deserialization Report URL: https://github.com/apache/incubator-fury/actions/runs/8526821473 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
(incubator-fury) branch main updated: chore(doc): sync specification doc (#1452)
This is an automated email from the ASF dual-hosted git repository. chaokunyang pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-fury.git The following commit(s) were added to refs/heads/main by this push: new 62607949 chore(doc): sync specification doc (#1452) 62607949 is described below commit 6260794926ee13de02fadeb7bc5817883bba8f8e Author: Shawn Yang AuthorDate: Wed Apr 3 01:22:06 2024 +0800 chore(doc): sync specification doc (#1452) Since our serialization spec has been formulized, we can publish it to our website now. This PR sync specification files to incubator-fury-website - Co-authored-by: LiangliangSui <116876207+liangliang...@users.noreply.github.com> --- .github/sync.yml | 2 ++ README.md | 34 ++ docs/README.md | 15 ++ docs/protocols/README.md | 4 --- docs/protocols/row_format.md | 2 -- .../java_serialization_spec.md}| 5 docs/specification/row_format_spec.md | 8 + .../xlang_serialization_spec.md} | 8 - 8 files changed, 48 insertions(+), 30 deletions(-) diff --git a/.github/sync.yml b/.github/sync.yml index bed00023..432f85c5 100644 --- a/.github/sync.yml +++ b/.github/sync.yml @@ -18,5 +18,7 @@ apache/incubator-fury-site@main: - source: docs/guide/ dest: docs/guide/ + - source: docs/specification/ +dest: docs/specification/ - source: docs/benchmarks/ dest: static/img/benchmarks/ diff --git a/README.md b/README.md index 11bb9c23..24543cbd 100644 --- a/README.md +++ b/README.md @@ -25,9 +25,9 @@ https://fury.apache.org In addition to cross-language serialization, Fury also features at: -- Drop-in replace Java serialization frameworks such as JDK/Kryo/Hessian, but 100x faster at most, which can greatly improve +- Drop-in replace Java serialization frameworks such as JDK/Kryo/Hessian, but 100x faster at most, which can greatly improve the efficiency of high-performance RPC calls, data transfer, and object persistence. -- **100% compatible** with JDK serialization API with much faster implementation: supporting JDK `writeObject`/`readObject`/`writeReplace`/`readResolve`/`readObjectNoData`/`Externalizable` API. +- **100% compatible** with JDK serialization API with much faster implementation: supporting JDK `writeObject`/`readObject`/`writeReplace`/`readResolve`/`readObjectNoData`/`Externalizable` API. - Supports **Java 8~21**, Java 17+ `record` is supported too. - Supports [AOT compilation serialization](docs/guide/graalvm_guide.md) for **GraalVM native image**, and no reflection/serialization json config are needed. - Supports shared and circular reference object serialization for golang. @@ -35,19 +35,17 @@ In addition to cross-language serialization, Fury also features at: - Supports automatic object serialization for golang. ## Protocols -Different scenarios have different serialization requirements. Fury designed and implemented -multiple binary protocols for those requirements: -- **Cross-language object graph protocol**: - - Cross-language serialize any object automatically, no need for IDL definition, schema compilation and object to/from protocol -conversion. - - Support shared reference and circular reference, no duplicate data or recursion error. +Fury designed and implemented multiple binary protocols for different scenarios: +- **[xlang serialization format](docs/specification/xlang_serialization_spec.md)**: + - Cross-language serialize any object automatically, no need for IDL definition, schema compilation and object to/from +protocol conversion. + - Support optional shared reference and circular reference, no duplicate data or recursion error. - Support object polymorphism. -- **Native java/python object graph protocol**: Highly-optimized based on type system of the language. -- **Row format protocol**: A cache-friendly binary random access format, supports skipping serialization and partial serialization, - and can convert to column-format automatically. +- **[Java serialization format](docs/specification/java_serialization_spec.md)**: Highly-optimized and drop-in replacement for Java serialization. +- **[Row format format](docs/specification/row_format_spec.md)**: A cache-friendly binary random access format, supports skipping serialization and + partial serialization, and can convert to column-format automatically. -New protocols can be easily added based on Fury existing buffer, encoding, meta, codegen and other capabilities. All of those share the same codebase, and the optimization for one protocol -can be reused by another protocol. +New protocols can be easily added based on Fury existing buffer, encoding, meta, codegen and ot
(incubator-fury) branch main updated: chore(java): Replace io.* in the benchmark with org.apache.fury (#1453)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury.git
The following commit(s) were added to refs/heads/main by this push:
new 700181ae chore(java): Replace io.* in the benchmark with
org.apache.fury (#1453)
700181ae is described below
commit 700181ae578166bc08a54181cd5fc8e76cbcd3e3
Author: LiangliangSui <116876207+liangliang...@users.noreply.github.com>
AuthorDate: Wed Apr 3 01:22:25 2024 +0800
chore(java): Replace io.* in the benchmark with org.apache.fury (#1453)
Signed-off-by: LiangliangSui
---
java/benchmark/README.md | 4 ++--
.../benchmark/src/main/java/org/apache/fury/benchmark/ArraySuite.java | 2 +-
java/benchmark/src/main/java/org/apache/fury/benchmark/Benchmark.java | 3 ++-
.../src/main/java/org/apache/fury/benchmark/CollectionSuite.java | 3 ++-
.../src/main/java/org/apache/fury/benchmark/CompressStringSuite.java | 2 +-
java/benchmark/src/main/java/org/apache/fury/benchmark/MapSuite.java | 2 +-
.../src/main/java/org/apache/fury/benchmark/MemorySuite.java | 3 ++-
.../src/main/java/org/apache/fury/benchmark/NewJava11StringSuite.java | 3 ++-
.../src/main/java/org/apache/fury/benchmark/NewStringSuite.java | 3 ++-
java/benchmark/src/main/java/org/apache/fury/benchmark/RowSuite.java | 2 +-
.../main/java/org/apache/fury/benchmark/UserTypeDeserializeSuite.java | 2 +-
.../main/java/org/apache/fury/benchmark/UserTypeSerializeSuite.java | 2 +-
.../src/main/java/org/apache/fury/benchmark/ZeroCopySuite.java| 3 ++-
13 files changed, 20 insertions(+), 14 deletions(-)
diff --git a/java/benchmark/README.md b/java/benchmark/README.md
index 8e205b68..3c3b5c9a 100644
--- a/java/benchmark/README.md
+++ b/java/benchmark/README.md
@@ -29,7 +29,7 @@ cd ../java && mvn install -DskipTests -Dcheckstyle.skip
-Dlicense.skip -Dmaven.j
mvn package -Pjmh
# run benchmark
nohup java -jar target/benchmarks.jar -f 5 -wi 3 -i 5 -t 1 -w 3s -r 5s -rf csv
>bench.log 2>&1 &
-java -jar target/benchmarks.jar "io.*\.deserialize$" -f 1 -wi 1 -i 3 -t 1 -w
2s -r 2s -rf csv
+java -jar target/benchmarks.jar "org.apache.fury.*\.deserialize$" -f 1 -wi 1
-i 3 -t 1 -w 2s -r 2s -rf csv
```
Generate Protobuf/Flatbuffers code manually:
@@ -75,7 +75,7 @@ Using `async-profiler` to generate flame graph.
```bash
export pic=s1.html
-nohup java -jar target/benchmarks.jar 'io.*Fury.*deserialize*' -f 1 -wi 1 -i 1
-t 1 -w 1s -r 35s -rf csv &
+nohup java -jar target/benchmarks.jar 'org.apache.fury.*Fury.*deserialize*' -f
1 -wi 1 -i 1 -t 1 -w 1s -r 35s -rf csv &
profiler.sh -d 30 -f $pic `jps | grep ForkedMain | awk '{print $1}'`
```
diff --git
a/java/benchmark/src/main/java/org/apache/fury/benchmark/ArraySuite.java
b/java/benchmark/src/main/java/org/apache/fury/benchmark/ArraySuite.java
index cbd0c5b9..a57c879e 100644
--- a/java/benchmark/src/main/java/org/apache/fury/benchmark/ArraySuite.java
+++ b/java/benchmark/src/main/java/org/apache/fury/benchmark/ArraySuite.java
@@ -233,7 +233,7 @@ public class ArraySuite {
// ops/s
public static void main(String[] args) throws Exception {
if (args.length == 0) {
- String commandLine = "io.*ArraySuite.* -f 1 -wi 3 -i 3 -t 1 -w 2s -r 2s
-rf csv";
+ String commandLine = "org.apache.fury.*ArraySuite.* -f 1 -wi 3 -i 3 -t 1
-w 2s -r 2s -rf csv";
System.out.println(commandLine);
args = commandLine.split(" ");
}
diff --git
a/java/benchmark/src/main/java/org/apache/fury/benchmark/Benchmark.java
b/java/benchmark/src/main/java/org/apache/fury/benchmark/Benchmark.java
index 4130a273..b0089254 100644
--- a/java/benchmark/src/main/java/org/apache/fury/benchmark/Benchmark.java
+++ b/java/benchmark/src/main/java/org/apache/fury/benchmark/Benchmark.java
@@ -29,7 +29,8 @@ public class Benchmark {
// the compile process.
public static void main(String[] args) throws Exception {
if (args.length == 0) {
- String commandLine = "io.*_deserialize$ -f 3 -wi 5 -i 3 -t 1 -w 3s -r 3s
-rf csv";
+ String commandLine =
+ "org.apache.fury.*_deserialize$ -f 3 -wi 5 -i 3 -t 1 -w 3s -r 3s -rf
csv";
System.out.println(commandLine);
args = commandLine.split(" ");
}
diff --git
a/java/benchmark/src/main/java/org/apache/fury/benchmark/CollectionSuite.java
b/java/benchmark/src/main/java/org/apache/fury/benchmark/CollectionSuite.java
index b06ed919..597e5a83 100644
---
a/java/benchmark/src/main/java/org/apache/fury/benchmark/CollectionSuite.java
+++
b/java/benchmark/src/main/java/org/apache/fury/benchmark/CollectionSuite.java
@@ -33,7 +33,8 @@ public class CollectionSuite {
public static void main(String[] args) throws Exception {
if (args.length == 0) {
- String commandLine = "io.*CollectionSuite.* -f 3 -wi 5 -i 5 -t 1 -w 2s
-r 2s -rf csv";
+ String commandLine =
+ "org.a
(incubator-fury-site) branch main updated: 🔄 created local 'docs/specification/' from remote 'docs/specification/'
This is an automated email from the ASF dual-hosted git repository. chaokunyang pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-fury-site.git The following commit(s) were added to refs/heads/main by this push: new 3215ec1 🔄 created local 'docs/specification/' from remote 'docs/specification/' 3215ec1 is described below commit 3215ec1cd4bdb158dabb2b3e60ccd1da2fe98420 Author: chaokunyang AuthorDate: Tue Apr 2 17:22:23 2024 + 🔄 created local 'docs/specification/' from remote 'docs/specification/' --- docs/specification/java_serialization_spec.md | 548 ++ docs/specification/row_format_spec.md | 8 + docs/specification/xlang_serialization_spec.md | 757 + 3 files changed, 1313 insertions(+) diff --git a/docs/specification/java_serialization_spec.md b/docs/specification/java_serialization_spec.md new file mode 100644 index 000..a5d0bec --- /dev/null +++ b/docs/specification/java_serialization_spec.md @@ -0,0 +1,548 @@ +--- +title: Fury Java Serialization Format +sidebar_position: 1 +id: fury_java_serialization_spec +--- +# Fury Java Serialization Specification + +## Spec overview + +Fury Java Serialization is an automatic object serialization framework that supports reference and polymorphism. Fury +will +convert an object from/to fury java serialization binary format. Fury has two core concepts for java serialization: + +- **Fury Java Binary format** +- **Framework to convert object to/from Fury Java Binary format** + +The serialization format is a dynamic binary format. The dynamics and reference/polymorphism support make Fury flexible, +much more easy to use, but +also introduce more complexities compared to static serialization frameworks. So the format will be more complex. + +Here is the overall format: + +``` +| fury header | object ref meta | object class meta | object value data | +``` + +The data are serialized using little endian byte order overall. If bytes swap is costly for some object, +Fury will write the byte order for that object into the data instead of converting it to little endian. + +## Fury header + +Fury header consists starts one byte: + +``` +| 4 bits| 1 bit | 1 bit | 1 bit | 1 bit | optional 4 bytes | ++---+---+---++---++ +| reserved bits | oob | xlang | endian | null | unsigned int for meta start offset | +``` + +- null flag: 1 when object is null, 0 otherwise. If an object is null, other bits won't be set. +- endian flag: 1 when data is encoded by little endian, 0 for big endian. +- xlang flag: 1 when serialization uses xlang format, 0 when serialization uses Fury java format. +- oob flag: 1 when passed `BufferCallback` is not null, 0 otherwise. + +If meta share mode is enabled, an uncompressed unsigned int is appended to indicate the start offset of metadata. + +## Reference Meta + +Reference tracking handles whether the object is null, and whether to track reference for the object by writing +corresponding flags and maintaining internal state. + +Reference flags: + +| Flag| Byte Value | Description | +|-||-| +| NULL FLAG | `-3` | This flag indicates the object is a null value. We don't use another byte to indicate REF, so that we can save one byte. | +| REF FLAG| `-2` | This flag indicates the object is already serialized previously, and fury will write a ref id with unsigned varint format instead of serialize it again | +| NOT_NULL VALUE FLAG | `-1` | This flag indicates the object is a non-null value and fury doesn't track ref for this type of object. | +| REF VALUE FLAG | `0`| This flag indicates the object is referencable and the first time to serialize. | + +When reference tracking is disabled globally or for specific types, or for certain types within a particular +context(e.g., a field of a class), only the `NULL` and `NOT_NULL VALUE` flags will be used for reference meta. + +## Class Meta + +Fury supports to register class by an optional id, the registration can be used for security check and class +identification. +If a class is registered, it will have a user-provided or an auto-growing unsigned int i.e. `class_id`. + +Depending on whether meta share mode and registration is enabled for current class, Fury will write class meta +differently. + +### Schema consistent + +If schema consistent mo
Re: [PR] chore(java): Replace io.* in the benchmark with org.apache.fury [incubator-fury]
chaokunyang merged PR #1453: URL: https://github.com/apache/incubator-fury/pull/1453 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] chore(doc): sync specification doc [incubator-fury]
chaokunyang merged PR #1452: URL: https://github.com/apache/incubator-fury/pull/1452 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] Fury fails when deserializing from InputStream [incubator-fury]
chaokunyang commented on issue #1435: URL: https://github.com/apache/incubator-fury/issues/1435#issuecomment-2032622165 > Fury seems like an interesting project to me, so I'd like to contribute, actually. If you think, that this feature worth implementing - I'd like to make a try. > > Also, I think that MemoryBuffer could become interface now, where different implementations holds different data sources (native address, byte array, `InputStream`, channel, etc) That would be very great, I try to implement the native streaming deserialization in #1451 . It's a little complicated due to we don't employ a polymorphic design for `MemoryBuffer`. The streaming reading for `InputStream` are basiclly finished, but the support for `java.nio.Channel` are not included in that PR. Maybe it can be taken as an example to implement the streaming channel deserializaiton. I thought about make `MemoryBuffer` polymorphic when I create fury several years ago, but polymorphism will make the invocation and other performance optimization such as bound checks elimination infeasible. And considering most cases are small object deserializaiton, the streaming are unnecessary, so I made it as a final classes in the end. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] chore(java): Replace io.* in the benchmark with org.apache.fury [incubator-fury]
LiangliangSui commented on PR #1453: URL: https://github.com/apache/incubator-fury/pull/1453#issuecomment-2032084325 CI has been passed, please help to merge it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" is working again!
The GitHub Actions job "Fury CI" on incubator-fury.git has succeeded. Run started by GitHub user LiangliangSui (triggered by LiangliangSui). Head commit for run: e8c1976ad450a8638243f2fe412423618f29f288 / LiangliangSui trim tab Signed-off-by: LiangliangSui Report URL: https://github.com/apache/incubator-fury/actions/runs/8523797671 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user LiangliangSui (triggered by LiangliangSui). Head commit for run: c41cbabadb2fb9b21f441ef3dd892d6f753bd7e7 / LiangliangSui lint code Signed-off-by: LiangliangSui Report URL: https://github.com/apache/incubator-fury/actions/runs/8523649380 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] Fury fails when deserializing from InputStream [incubator-fury]
Munoon commented on issue #1435: URL: https://github.com/apache/incubator-fury/issues/1435#issuecomment-2032048074 Fury seems like an interesting project to me, so I'd like to contribute, actually. If you think, that this feature worth implementing - I'd like to make a try. Also, I think that MemoryBuffer could become interface now, where different implementations holds different data sources (native address, byte array, `InputStream`, channel, etc) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [I] [Java] remove guava dependency [incubator-fury]
doom369 commented on issue #1113: URL: https://github.com/apache/incubator-fury/issues/1113#issuecomment-2032037260 Hello. It seems like all dependencies are removed, so you can now remove the guava from the required dependencies during the build? Currently, the fury jar is 2.3 mb and guava brings an additional 3 mb. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[GH] (incubator-fury): Workflow run "Fury CI" failed!
The GitHub Actions job "Fury CI" on incubator-fury.git has failed. Run started by GitHub user LiangliangSui (triggered by LiangliangSui). Head commit for run: c0eed11027fed242a596342a781f83bba28a0838 / LiangliangSui chore(java): Replace io.* in the benchmark with org.apache.fury Signed-off-by: LiangliangSui Report URL: https://github.com/apache/incubator-fury/actions/runs/8523491159 With regards, GitHub Actions via GitBox - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
[PR] chore(java): Replace io.* in the benchmark with org.apache.fury [incubator-fury]
LiangliangSui opened a new pull request, #1453: URL: https://github.com/apache/incubator-fury/pull/1453 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] chore(java): Reuse unsafePutPositiveVarInt in unsafeWritePositiveVarInt [incubator-fury]
chaokunyang commented on code in PR #1434:
URL: https://github.com/apache/incubator-fury/pull/1434#discussion_r1547828485
##
java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java:
##
@@ -1233,49 +1233,9 @@ public int readVarInt() {
* to avoid using two memory operations.
*/
public int unsafeWritePositiveVarInt(int v) {
-// The encoding algorithm are based on kryo UnsafeMemoryOutput.writeVarInt
-// varint are written using little endian byte order.
-// This version should have better performance since it remove an index
update.
-long value = v;
-final int writerIndex = this.writerIndex;
-long varInt = (value & 0x7F);
-value >>>= 7;
-if (value == 0) {
- UNSAFE.putByte(heapMemory, address + writerIndex, (byte) varInt);
- this.writerIndex = writerIndex + 1;
- return 1;
-}
-// bit 8 `set` indicates have next data bytes.
-varInt |= 0x80;
-varInt |= ((value & 0x7F) << 8);
-value >>>= 7;
-if (value == 0) {
- unsafePutInt(writerIndex, (int) varInt);
- this.writerIndex = writerIndex + 2;
- return 2;
-}
-varInt |= (0x80 << 8);
-varInt |= ((value & 0x7F) << 16);
-value >>>= 7;
-if (value == 0) {
- unsafePutInt(writerIndex, (int) varInt);
- this.writerIndex = writerIndex + 3;
- return 3;
-}
-varInt |= (0x80 << 16);
-varInt |= ((value & 0x7F) << 24);
-value >>>= 7;
-if (value == 0) {
- unsafePutInt(writerIndex, (int) varInt);
- this.writerIndex = writerIndex + 4;
- return 4;
-}
-varInt |= (0x80L << 24);
-varInt |= ((value & 0x7F) << 32);
-varInt &= 0xFL;
-unsafePutLong(writerIndex, varInt);
-this.writerIndex = writerIndex + 5;
-return 5;
+int varintBytes = unsafePutPositiveVarInt(writerIndex, v);
Review Comment:
You may need to update fury creation to use LongEncoding.PVL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] chore(java): code clean [incubator-fury]
chaokunyang commented on code in PR #1450:
URL: https://github.com/apache/incubator-fury/pull/1450#discussion_r1547826521
##
java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java:
##
@@ -1316,24 +1316,15 @@ public int readPositiveVarInt() {
}
private int readPositiveVarIntSlow() {
-int b = readByte();
-int result = b & 0x7F;
-if ((b & 0x80) != 0) {
+int shift = 0;
Review Comment:
Yes, could you add some comments in this pr? I think we'd better add
comments to similar methods
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] chore(java): Reuse unsafePutPositiveVarInt in unsafeWritePositiveVarInt [incubator-fury]
LiangliangSui commented on code in PR #1434:
URL: https://github.com/apache/incubator-fury/pull/1434#discussion_r1547817704
##
java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java:
##
@@ -1233,49 +1233,9 @@ public int readVarInt() {
* to avoid using two memory operations.
*/
public int unsafeWritePositiveVarInt(int v) {
-// The encoding algorithm are based on kryo UnsafeMemoryOutput.writeVarInt
-// varint are written using little endian byte order.
-// This version should have better performance since it remove an index
update.
-long value = v;
-final int writerIndex = this.writerIndex;
-long varInt = (value & 0x7F);
-value >>>= 7;
-if (value == 0) {
- UNSAFE.putByte(heapMemory, address + writerIndex, (byte) varInt);
- this.writerIndex = writerIndex + 1;
- return 1;
-}
-// bit 8 `set` indicates have next data bytes.
-varInt |= 0x80;
-varInt |= ((value & 0x7F) << 8);
-value >>>= 7;
-if (value == 0) {
- unsafePutInt(writerIndex, (int) varInt);
- this.writerIndex = writerIndex + 2;
- return 2;
-}
-varInt |= (0x80 << 8);
-varInt |= ((value & 0x7F) << 16);
-value >>>= 7;
-if (value == 0) {
- unsafePutInt(writerIndex, (int) varInt);
- this.writerIndex = writerIndex + 3;
- return 3;
-}
-varInt |= (0x80 << 16);
-varInt |= ((value & 0x7F) << 24);
-value >>>= 7;
-if (value == 0) {
- unsafePutInt(writerIndex, (int) varInt);
- this.writerIndex = writerIndex + 4;
- return 4;
-}
-varInt |= (0x80L << 24);
-varInt |= ((value & 0x7F) << 32);
-varInt &= 0xFL;
-unsafePutLong(writerIndex, varInt);
-this.writerIndex = writerIndex + 5;
-return 5;
+int varintBytes = unsafePutPositiveVarInt(writerIndex, v);
Review Comment:
Okay, I will do this check later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
Re: [PR] chore(java): code clean [incubator-fury]
LiangliangSui commented on code in PR #1450:
URL: https://github.com/apache/incubator-fury/pull/1450#discussion_r1547815390
##
java/fury-core/src/main/java/org/apache/fury/memory/MemoryBuffer.java:
##
@@ -1316,24 +1316,15 @@ public int readPositiveVarInt() {
}
private int readPositiveVarIntSlow() {
-int b = readByte();
-int result = b & 0x7F;
-if ((b & 0x80) != 0) {
+int shift = 0;
Review Comment:
Use the following code to run the benchmark
```java
package org.example;
import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.results.format.ResultFormatType;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;
import java.util.concurrent.TimeUnit;
@BenchmarkMode(Mode.Throughput)
@OutputTimeUnit(TimeUnit.SECONDS)
@Fork(5)
@Warmup(iterations = 3, time = 2, timeUnit = TimeUnit.SECONDS)
@Measurement(iterations = 3, time = 2, timeUnit = TimeUnit.SECONDS)
@State(Scope.Benchmark)
@CompilerControl(CompilerControl.Mode.INLINE)
public class Main {
public static void main(String[] args) throws RunnerException {
Options opt = new OptionsBuilder()
.include(Main.class.getName())
.resultFormat(ResultFormatType.CSV)
.build();
new Runner(opt).run();
}
private byte getBytes(int shift) {
if (shift == 4) {
return 0x75;
}
return (byte) 0x85;
}
@Benchmark
public int readPositiveVarIntSlowBefore() {
int b = getBytes(0);
int result = b & 0x7F;
if ((b & 0x80) != 0) {
b = getBytes(1);
result |= (b & 0x7F) << 7;
if ((b & 0x80) != 0) {
b = getBytes(2);
result |= (b & 0x7F) << 14;
if ((b & 0x80) != 0) {
b = getBytes(3);
result |= (b & 0x7F) << 21;
if ((b & 0x80) != 0) {
b = getBytes(4);
result |= (b & 0x7F) << 28;
}
}
}
}
return result;
}
@Benchmark
public int readPositiveVarIntSlowAfter() {
int shift = 0;
int result = 0;
byte b;
do {
b = getBytes(shift);
result |= (b & 0x7f) << (shift * 7);
shift++;
} while ((b & 0x80) != 0);
return result;
}
}
```
The performance before modification is far better than the performance after
modification.
But this coding method affects the readability of the code. Should we add
some comments to explain why we do this, so that we can also convey the purpose
of this behavior to subsequent developers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@fury.apache.org
For additional commands, e-mail: commits-h...@fury.apache.org
