Oh, that's perfect. I'm very, very close. Do you mind taking at a short snippet of my code I put into this test file:
https://gist.github.com/anonymous/8b748d8a3810515be8d9349768eb92e4 So, I replicated most of the logic found in BlockFileLoader.java. Rather than reading from input stream, I am trying to keep and index to iterate through the actual bytes themselves. Line 66 yields the error: NegativeArraySizeException at org.bitcoinj.core.Message.readBytes. I'm guessing that my index is wrong at some point. Can't spot the bug, however. Keep in mind, I want to build from the bytes array, because in another situation I do not want to use the FileInputStream. Any ideas? Thank you so much. On 25 February 2017 at 16:34, Jameson Lopp <jameson.l...@gmail.com> wrote: > Aha, if you can somehow get access to the bytes then you should be able to > duplicate the byte logic here: https://github.com/bitcoinj/bitcoinj/blob/ > 3177bd52a2bfa491c5902e95b8840030e1a31159/core/src/main/java/ > org/bitcoinj/utils/BlockFileLoader.java#L129 > > Basically, you need to do some low level byte parsing so that you can > eventually read in what the size of the next Block is, then you can take > the next N bytes and pass them to the deserializer in order to actually > construct the Block object: https://github.com/bitcoinj/bitcoinj/blob/ > 3177bd52a2bfa491c5902e95b8840030e1a31159/core/src/main/java/ > org/bitcoinj/utils/BlockFileLoader.java#L152 > > - Jameson > > On Sat, Feb 25, 2017 at 5:20 PM, Nishil Shah <nishil.sha...@gmail.com> > wrote: > >> Figured it out. I'm using hadoop and the files are stored on the >> distributed file system. Need to find a workaround. If I read all the bytes >> from the file, can I construct a block with it by passing it in to the >> Block constructor? What exactly is the "payloadBytes" parameter? >> >> Thanks. >> >> On 22 February 2017 at 16:49, Jameson Lopp <jameson.l...@gmail.com> >> wrote: >> >>> That's quite odd; it sounds like you didn't actually add the File(s) to >>> the List. Did you do a .exists() check on them like it does here? >>> https://github.com/bitcoinj/bitcoinj/blob/3177bd52a2bf >>> a491c5902e95b8840030e1a31159/core/src/main/java/org/bitcoinj >>> /utils/BlockFileLoader.java#L67 >>> >>> I don't think the different styles of getting the params should matter. >>> Feel free to share your code. >>> >>> On Wed, Feb 22, 2017 at 3:14 PM, Nishil Shah <nishil.sha...@gmail.com> >>> wrote: >>> >>>> Ah, I can't check right now, but I remember I was using new >>>> MainNetParams() as opposed to MainNetParams.get(). Could that be the >>>> difference? >>>> >>>> On 22 February 2017 at 12:53, Nishil Shah <nishil.sha...@gmail.com> >>>> wrote: >>>> >>>>> next() just complains that there is no next item in the iterator. Does >>>>> this mean the I/O failed? According to the docs, the BlockFileLoader >>>>> swallows I/O exceptions, so I can't see the I/O error. >>>>> >>>>> On 21 February 2017 at 10:31, Jameson Lopp <jameson.l...@gmail.com> >>>>> wrote: >>>>> >>>>>> You should definitely be able to build your own List<File> list that >>>>>> you pass to the loader - I've done this myself in the past. >>>>>> >>>>>> What error are you seeing? >>>>>> >>>>>> On Mon, Feb 20, 2017 at 4:20 PM, Nishil Shah <nishil.sha...@gmail.com >>>>>> > wrote: >>>>>> >>>>>>> I would use that List to initialize the block file loader. >>>>>>> >>>>>>> On 20 February 2017 at 15:19, Nishil Shah <nishil.sha...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> I see, thanks. One last question: So shouldn't I be able to simply >>>>>>>> build a List<File> and append paths to my block files >>>>>>>> "/path/to/blk00000.dat"? I tried this and when I do >>>>>>>> blockFileLoader.next() >>>>>>>> I get an error. Don't think this works. >>>>>>>> >>>>>>>> On 19 February 2017 at 03:25, Jameson Lopp <jameson.l...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Right, getReferenceClientList >>>>>>>>> <https://github.com/bitcoinj/bitcoinj/blob/3177bd52a2bfa491c5902e95b8840030e1a31159/core/src/main/java/org/bitcoinj/utils/BlockFileLoader.java#L53> >>>>>>>>> just tries to be smart and figure out the location of your block files >>>>>>>>> automatically. If it doesn't work for your system, you can model its >>>>>>>>> logic >>>>>>>>> to create your own function that builds a list of your block files. >>>>>>>>> >>>>>>>>> For your purposes, I'm assuming you're analyzing Bitcoin on >>>>>>>>> mainnet, so you'd pass MainNetParams.get(). The only other thing for >>>>>>>>> you to >>>>>>>>> figure out then would be your data storage, which would be your >>>>>>>>> "chain" >>>>>>>>> object or whatever function to which you pass the blocks to be >>>>>>>>> processed. >>>>>>>>> >>>>>>>>> - Jameson >>>>>>>>> >>>>>>>>> On Sun, Feb 19, 2017 at 2:00 AM, Nishil Shah < >>>>>>>>> nishil.sha...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> It doesn't take in a file name? >>>>>>>>>> >>>>>>>>>> Sent from my iPhone >>>>>>>>>> >>>>>>>>>> On Feb 19, 2017, at 12:33 AM, Jameson Lopp < >>>>>>>>>> jameson.l...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>> I think what you want is the BlockFileLoader, as seen here in the >>>>>>>>>> BlockImporter: https://github.com/bitcoinj/bitcoinj/blob/mas >>>>>>>>>> ter/tools/src/main/java/org/bitcoinj/tools/BlockImporter.java#L67 >>>>>>>>>> >>>>>>>>>> - Jameson >>>>>>>>>> >>>>>>>>>> On Sat, Feb 18, 2017 at 11:51 PM, Nishil Shah < >>>>>>>>>> nishil.sha...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Is there anyway to import blk?????.dat files individually and >>>>>>>>>>> parse them into Block objects? I'm not trying to use any network >>>>>>>>>>> connections. I want to get everything from disk if possible to make >>>>>>>>>>> my >>>>>>>>>>> computations/processing as fast as possible because I want parse >>>>>>>>>>> many files >>>>>>>>>>> in parallel. I've read about some "block importer" tool but can't >>>>>>>>>>> really >>>>>>>>>>> see how to use it in my situation. Thanks. >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "bitcoinj" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to bitcoinj+unsubscr...@googlegroups.com. >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to a topic >>>>>>>>>> in the Google Groups "bitcoinj" group. >>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>> https://groups.google.com/d/topic/bitcoinj/4wB6-i7ys8E/unsub >>>>>>>>>> scribe. >>>>>>>>>> To unsubscribe from this group and all its topics, send an email >>>>>>>>>> to bitcoinj+unsubscr...@googlegroups.com. >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "bitcoinj" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to bitcoinj+unsubscr...@googlegroups.com. >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>>> the Google Groups "bitcoinj" group. >>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>> https://groups.google.com/d/topic/bitcoinj/4wB6-i7ys8E/unsubscribe >>>>>>>>> . >>>>>>>>> To unsubscribe from this group and all its topics, send an email >>>>>>>>> to bitcoinj+unsubscr...@googlegroups.com. >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "bitcoinj" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to bitcoinj+unsubscr...@googlegroups.com. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to a topic in >>>>>> the Google Groups "bitcoinj" group. >>>>>> To unsubscribe from this topic, visit https://groups.google.com/d/to >>>>>> pic/bitcoinj/4wB6-i7ys8E/unsubscribe. >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> bitcoinj+unsubscr...@googlegroups.com. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "bitcoinj" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to bitcoinj+unsubscr...@googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "bitcoinj" group. >>> To unsubscribe from this topic, visit https://groups.google.com/d/to >>> pic/bitcoinj/4wB6-i7ys8E/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> bitcoinj+unsubscr...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "bitcoinj" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to bitcoinj+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "bitcoinj" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/bitcoinj/4wB6-i7ys8E/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > bitcoinj+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "bitcoinj" group. To unsubscribe from this group and stop receiving emails from it, send an email to bitcoinj+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
