Re: [protobuf] Serialize BufferedImage with Proto
What are the alternatives? One way or another, you have to create a representation of the object... enough to be able to create a new one on deserialization. On Wed, Feb 26, 2014 at 10:34 PM, jCoder wrote: > I was afraid of that answer. > I feel like sending the whole object to a byte[] would cost just as much if > not more time then it currently takes. > > On Wednesday, February 26, 2014 10:22:20 PM UTC-5, Ilia Mirkin wrote: >> >> On Wed, Feb 26, 2014 at 10:13 PM, jCoder wrote: >> > I was able to change the BufferedImage to an ImageIcon which is >> > Serializable. >> > This along with using the FST Serialization class/jar I was able to >> > improve >> > the over all write time to about 10 seconds for a total write size of >> > about >> > 140 MB >> > Which is still only 14MB/s write speed when I know I can write at >> > 150+MB/s >> > >> > My question is can ProtoBuf handle serializing an ImageIcon, Color, or >> > any >> > Java objects besides basic Strings and Ints? and is it faster the FST? >> >> Protobuf does not handle serialization of Java objects. You can give >> it a byte[] array (or an int/float/etc or a String). Hence my >> suggestion for the benchmark. I suspect that the majority of the time >> is going to computing the byte[] array representing your object, which >> you'd have to do for protobuf anyways. >> >> > >> > Also one of my custom object classes has a nested object class inside of >> > it, >> > will this slow down serialization? >> > >> > >> > On Wednesday, February 26, 2014 9:10:24 PM UTC-5, Ilia Mirkin wrote: >> >> >> >> On Tue, Feb 25, 2014 at 3:03 PM, jCoder wrote: >> >> > I was wondering if anyone had any success implementing this with a >> >> > HashMap >> >> > that has a pointer to a BufferedImage. >> >> > Example: >> >> > Map thumb = new HashMap(); >> >> > >> >> > I am currently using Serialization with a custom writeObject() and >> >> > readObject() to turn the BufferedImage into a byte[] and back again. >> >> > >> >> > However this process takes roughly 25 second+ for approximately >> >> > 18,000 >> >> > BufferedImages (size: 16pixels x 12pixels) previously loaded into >> >> > memory >> >> > to >> >> > be serialized into a file (resulting size 9,408 KB). >> >> > Please note this all happening on a SSD (so disk write speed should >> >> > not >> >> > be >> >> > an issue). >> >> > >> >> > There has be a faster way to do this perhaps Protocol Buffers can >> >> > help, >> >> > I am >> >> > just not sure the best way to handle a BufferedImage with it would >> >> > be. >> >> > >> >> > Any help would be greatly appreciated. >> >> >> >> Try writing a benchmark that simply converts the BufferedImages to >> >> byte[] and throws away the results. That's a lower-bound on your >> >> overall serialization speed (without switching away from BufferedImage >> >> to something else). >> >> >> >> -ilia >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "Protocol Buffers" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> > an >> > email to protobuf+u...@googlegroups.com. >> > To post to this group, send email to prot...@googlegroups.com. >> > Visit this group at http://groups.google.com/group/protobuf. >> > For more options, visit https://groups.google.com/groups/opt_out. > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to protobuf+unsubscr...@googlegroups.com. > To post to this group, send email to protobuf@googlegroups.com. > Visit this group at http://groups.google.com/group/protobuf. > For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Serialize BufferedImage with Proto
I was afraid of that answer. I feel like sending the whole object to a byte[] would cost just as much if not more time then it currently takes. On Wednesday, February 26, 2014 10:22:20 PM UTC-5, Ilia Mirkin wrote: > > On Wed, Feb 26, 2014 at 10:13 PM, jCoder > > wrote: > > I was able to change the BufferedImage to an ImageIcon which is > > Serializable. > > This along with using the FST Serialization class/jar I was able to > improve > > the over all write time to about 10 seconds for a total write size of > about > > 140 MB > > Which is still only 14MB/s write speed when I know I can write at > 150+MB/s > > > > My question is can ProtoBuf handle serializing an ImageIcon, Color, or > any > > Java objects besides basic Strings and Ints? and is it faster the FST? > > Protobuf does not handle serialization of Java objects. You can give > it a byte[] array (or an int/float/etc or a String). Hence my > suggestion for the benchmark. I suspect that the majority of the time > is going to computing the byte[] array representing your object, which > you'd have to do for protobuf anyways. > > > > > Also one of my custom object classes has a nested object class inside of > it, > > will this slow down serialization? > > > > > > On Wednesday, February 26, 2014 9:10:24 PM UTC-5, Ilia Mirkin wrote: > >> > >> On Tue, Feb 25, 2014 at 3:03 PM, jCoder wrote: > >> > I was wondering if anyone had any success implementing this with a > >> > HashMap > >> > that has a pointer to a BufferedImage. > >> > Example: > >> > Map thumb = new HashMap(); > >> > > >> > I am currently using Serialization with a custom writeObject() and > >> > readObject() to turn the BufferedImage into a byte[] and back again. > >> > > >> > However this process takes roughly 25 second+ for approximately > 18,000 > >> > BufferedImages (size: 16pixels x 12pixels) previously loaded into > memory > >> > to > >> > be serialized into a file (resulting size 9,408 KB). > >> > Please note this all happening on a SSD (so disk write speed should > not > >> > be > >> > an issue). > >> > > >> > There has be a faster way to do this perhaps Protocol Buffers can > help, > >> > I am > >> > just not sure the best way to handle a BufferedImage with it would > be. > >> > > >> > Any help would be greatly appreciated. > >> > >> Try writing a benchmark that simply converts the BufferedImages to > >> byte[] and throws away the results. That's a lower-bound on your > >> overall serialization speed (without switching away from BufferedImage > >> to something else). > >> > >> -ilia > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "Protocol Buffers" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to protobuf+u...@googlegroups.com . > > To post to this group, send email to prot...@googlegroups.com. > > > Visit this group at http://groups.google.com/group/protobuf. > > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Serialize BufferedImage with Proto
On Wed, Feb 26, 2014 at 10:13 PM, jCoder wrote: > I was able to change the BufferedImage to an ImageIcon which is > Serializable. > This along with using the FST Serialization class/jar I was able to improve > the over all write time to about 10 seconds for a total write size of about > 140 MB > Which is still only 14MB/s write speed when I know I can write at 150+MB/s > > My question is can ProtoBuf handle serializing an ImageIcon, Color, or any > Java objects besides basic Strings and Ints? and is it faster the FST? Protobuf does not handle serialization of Java objects. You can give it a byte[] array (or an int/float/etc or a String). Hence my suggestion for the benchmark. I suspect that the majority of the time is going to computing the byte[] array representing your object, which you'd have to do for protobuf anyways. > > Also one of my custom object classes has a nested object class inside of it, > will this slow down serialization? > > > On Wednesday, February 26, 2014 9:10:24 PM UTC-5, Ilia Mirkin wrote: >> >> On Tue, Feb 25, 2014 at 3:03 PM, jCoder wrote: >> > I was wondering if anyone had any success implementing this with a >> > HashMap >> > that has a pointer to a BufferedImage. >> > Example: >> > Map thumb = new HashMap(); >> > >> > I am currently using Serialization with a custom writeObject() and >> > readObject() to turn the BufferedImage into a byte[] and back again. >> > >> > However this process takes roughly 25 second+ for approximately 18,000 >> > BufferedImages (size: 16pixels x 12pixels) previously loaded into memory >> > to >> > be serialized into a file (resulting size 9,408 KB). >> > Please note this all happening on a SSD (so disk write speed should not >> > be >> > an issue). >> > >> > There has be a faster way to do this perhaps Protocol Buffers can help, >> > I am >> > just not sure the best way to handle a BufferedImage with it would be. >> > >> > Any help would be greatly appreciated. >> >> Try writing a benchmark that simply converts the BufferedImages to >> byte[] and throws away the results. That's a lower-bound on your >> overall serialization speed (without switching away from BufferedImage >> to something else). >> >> -ilia > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to protobuf+unsubscr...@googlegroups.com. > To post to this group, send email to protobuf@googlegroups.com. > Visit this group at http://groups.google.com/group/protobuf. > For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Serialize BufferedImage with Proto
I was able to change the BufferedImage to an ImageIcon which is Serializable. This along with using the FST Serialization class/jar I was able to improve the over all write time to about 10 seconds for a total write size of about 140 MB Which is still only 14MB/s write speed when I know I can write at 150+MB/s My question is can ProtoBuf handle serializing an ImageIcon, Color, or any Java objects besides basic Strings and Ints? and is it faster the FST? Also one of my custom object classes has a nested object class inside of it, will this slow down serialization? On Wednesday, February 26, 2014 9:10:24 PM UTC-5, Ilia Mirkin wrote: > > On Tue, Feb 25, 2014 at 3:03 PM, jCoder > > wrote: > > I was wondering if anyone had any success implementing this with a > HashMap > > that has a pointer to a BufferedImage. > > Example: > > Map thumb = new HashMap(); > > > > I am currently using Serialization with a custom writeObject() and > > readObject() to turn the BufferedImage into a byte[] and back again. > > > > However this process takes roughly 25 second+ for approximately 18,000 > > BufferedImages (size: 16pixels x 12pixels) previously loaded into memory > to > > be serialized into a file (resulting size 9,408 KB). > > Please note this all happening on a SSD (so disk write speed should not > be > > an issue). > > > > There has be a faster way to do this perhaps Protocol Buffers can help, > I am > > just not sure the best way to handle a BufferedImage with it would be. > > > > Any help would be greatly appreciated. > > Try writing a benchmark that simply converts the BufferedImages to > byte[] and throws away the results. That's a lower-bound on your > overall serialization speed (without switching away from BufferedImage > to something else). > > -ilia > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Serialize BufferedImage with Proto
On Tue, Feb 25, 2014 at 3:03 PM, jCoder wrote: > I was wondering if anyone had any success implementing this with a HashMap > that has a pointer to a BufferedImage. > Example: > Map thumb = new HashMap(); > > I am currently using Serialization with a custom writeObject() and > readObject() to turn the BufferedImage into a byte[] and back again. > > However this process takes roughly 25 second+ for approximately 18,000 > BufferedImages (size: 16pixels x 12pixels) previously loaded into memory to > be serialized into a file (resulting size 9,408 KB). > Please note this all happening on a SSD (so disk write speed should not be > an issue). > > There has be a faster way to do this perhaps Protocol Buffers can help, I am > just not sure the best way to handle a BufferedImage with it would be. > > Any help would be greatly appreciated. Try writing a benchmark that simply converts the BufferedImages to byte[] and throws away the results. That's a lower-bound on your overall serialization speed (without switching away from BufferedImage to something else). -ilia -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
