Re: [Factor-talk] Any way of making sense of what's in the boot image?
Quick question about parse-file.Is the quotation that is returned on the stack as a result of this word a parse tree, as discussed in the documentation's article on parsing? This would be a parse tree as in "Tokens are appended to the parse tree, the top level of which is a quotation returned by the original parser invocation.", right? There could also potentially be nested elements in the tree contributed by other parse words encountered in the course of satisfying the original parser invocation. Am I understanding this correctly? Thanks. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Factor-talk mailing list Factor-talk@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/factor-talk
Re: [Factor-talk] Any way of making sense of what's in the boot image?
Much appreciated, thanks. I had actually already discovered some significant portion of what you explain below, which is why I'm going to hold off on the baby-step questions so that I can save the limited Joe Groff bandwidth that the universe is going to provide me for more meaty important stuff. > Date: Thu, 9 Aug 2012 06:26:54 -0700 > From: arc...@gmail.com > To: factor-talk@lists.sourceforge.net > Subject: Re: [Factor-talk] Any way of making sense of what's in the boot > image? > > On Wed, Aug 8, 2012 at 9:56 PM, Michael Clagett wrote: > > Hi Joe -- > > > > Thank you so much for your guidance. I'm assuming that it is the > > basis\bootstrap\image\image.factor and the core\bootstrap\primitives.factor, > > stage1.factor and syntax.factor that you are referring to? > > > > One question just to help me get oriented before I go into heavy > > head-twisting mode. I notice that all these files have significant numbers > > of pre-existing vocabularies that they use. If this is what is used to > > generate the early stage1 bootstrapping stuff, then am I correct to assume > > that this is done from some version of factor where all this stuff has built > > already from scratch (that would be the implication, if all these files are > > able to use those vocabularies)? Or is there some other mechanism at work > > here that I should know about, and if so, is it written up anywhere that you > > know about. > > > > I have just read Slava's January 2010 post on the bootstrapping mechanism, > > so I understand the mechanism a bit better now, but am still thinking there > > has to be some Factor system somewhere that was built without a bootstrap > > image in order to have the vocabularies available that are used to generate > > the bootstrapping. Is there anything anywhere documenting that aspect of > > the picture. The reason I ask is that I would like to get a sense of how > > extensive that core fundamental set of vocabularies is that is needed to > > process the primitives and stage1 factor files. > > > > Here's the USING statement for primitives.factor: > > > > USING: alien alien.strings arrays byte-arrays generic hashtables > > hashtables.private io io.encodings.ascii kernel math > > math.private math.order namespaces make parser sequences strings > > vectors words quotations assocs layouts classes classes.private > > classes.builtin classes.singleton classes.tuple > > classes.tuple.private kernel.private vocabs vocabs.loader > > source-files definitions slots classes.union > > classes.intersection classes.predicate compiler.units > > bootstrap.image.private io.files accessors combinators ; > > IN: bootstrap.primitives > > > > It's got a lot of stuff there. Is a lot of this built in the C++ code that > > constitutes the VM? Or is some fundamental processing capability put into > > place that allows the processing of factor files to build these > > vocabularies? And if so, is the factor source used also included in the > > downloaded platform? Or was this a special primordial code base that was > > used once and never really needed again? > > Although core/bootstrap/primitives.factor is in core/, it's not > actually loaded as-is into the boot image; it's really a script > designed to run as part of `make-image` in the host image. All of > those modules are loaded and run in the host image. While there of > course had to have been a primordial Factor that could boot itself, I > think it would be too different now to resemble current Factor. > Factor's bootstrapping loop evolved somewhat organically. > > You can tell whether a word is a C++ primitive by `see`ing it in the > listener. Primitive words show up as `PRIMITIVE:` when prettyprinted. > > At a high level, `make-image` has two stages: First, it executes > `core/bootstrap/stage1.factor`, which runs 'bootstrap/syntax.factor` > to transfer the basic syntax from the host image to the boot image, > `bootstrap/primitives.factor` to set up primitive words corresponding > to VM primitives, and a CPU-specific `basis/cpu/*/bootstrap.factor` > script to assemble "sub-primitives" which are blobs of assembly > language code composed by the VM's compiler. It also composes a > startup quotation to begin the second stage of bootstrap when the boot > image is started. Second, after the bootstrapping elements have been > loaded, it serializes the data structures loaded by the bootstrap > scripts into a boot image. Again, it does this without help from the > VM, though it does borrow some knowledge of layouts from the compiler > and other Factor libraries. > > > > P.S. Incidentally, while I am on the subject, I find myself able to trace > > through the VM startup code in Visual Studio with no problem at all, until I > > get to the factor_vm::c_to_factor, which wraps a c-to-factor sub-primitive > > inside of a callback stub and then invokes that function. On both a Windows > > 7 platform and a Windows Server 2008 platform, my Visual Studio blows up >
Re: [Factor-talk] Any way of making sense of what's in the boot image?
On Wed, Aug 8, 2012 at 9:56 PM, Michael Clagett wrote: > Hi Joe -- > > Thank you so much for your guidance. I'm assuming that it is the > basis\bootstrap\image\image.factor and the core\bootstrap\primitives.factor, > stage1.factor and syntax.factor that you are referring to? > > One question just to help me get oriented before I go into heavy > head-twisting mode. I notice that all these files have significant numbers > of pre-existing vocabularies that they use. If this is what is used to > generate the early stage1 bootstrapping stuff, then am I correct to assume > that this is done from some version of factor where all this stuff has built > already from scratch (that would be the implication, if all these files are > able to use those vocabularies)? Or is there some other mechanism at work > here that I should know about, and if so, is it written up anywhere that you > know about. > > I have just read Slava's January 2010 post on the bootstrapping mechanism, > so I understand the mechanism a bit better now, but am still thinking there > has to be some Factor system somewhere that was built without a bootstrap > image in order to have the vocabularies available that are used to generate > the bootstrapping. Is there anything anywhere documenting that aspect of > the picture. The reason I ask is that I would like to get a sense of how > extensive that core fundamental set of vocabularies is that is needed to > process the primitives and stage1 factor files. > > Here's the USING statement for primitives.factor: > > USING: alien alien.strings arrays byte-arrays generic hashtables > hashtables.private io io.encodings.ascii kernel math > math.private math.order namespaces make parser sequences strings > vectors words quotations assocs layouts classes classes.private > classes.builtin classes.singleton classes.tuple > classes.tuple.private kernel.private vocabs vocabs.loader > source-files definitions slots classes.union > classes.intersection classes.predicate compiler.units > bootstrap.image.private io.files accessors combinators ; > IN: bootstrap.primitives > > It's got a lot of stuff there. Is a lot of this built in the C++ code that > constitutes the VM? Or is some fundamental processing capability put into > place that allows the processing of factor files to build these > vocabularies? And if so, is the factor source used also included in the > downloaded platform? Or was this a special primordial code base that was > used once and never really needed again? Although core/bootstrap/primitives.factor is in core/, it's not actually loaded as-is into the boot image; it's really a script designed to run as part of `make-image` in the host image. All of those modules are loaded and run in the host image. While there of course had to have been a primordial Factor that could boot itself, I think it would be too different now to resemble current Factor. Factor's bootstrapping loop evolved somewhat organically. You can tell whether a word is a C++ primitive by `see`ing it in the listener. Primitive words show up as `PRIMITIVE:` when prettyprinted. At a high level, `make-image` has two stages: First, it executes `core/bootstrap/stage1.factor`, which runs 'bootstrap/syntax.factor` to transfer the basic syntax from the host image to the boot image, `bootstrap/primitives.factor` to set up primitive words corresponding to VM primitives, and a CPU-specific `basis/cpu/*/bootstrap.factor` script to assemble "sub-primitives" which are blobs of assembly language code composed by the VM's compiler. It also composes a startup quotation to begin the second stage of bootstrap when the boot image is started. Second, after the bootstrapping elements have been loaded, it serializes the data structures loaded by the bootstrap scripts into a boot image. Again, it does this without help from the VM, though it does borrow some knowledge of layouts from the compiler and other Factor libraries. > > P.S. Incidentally, while I am on the subject, I find myself able to trace > through the VM startup code in Visual Studio with no problem at all, until I > get to the factor_vm::c_to_factor, which wraps a c-to-factor sub-primitive > inside of a callback stub and then invokes that function. On both a Windows > 7 platform and a Windows Server 2008 platform, my Visual Studio blows up > when the callback address loaded into EDX is called. Anybody ever > encountered this and any idea how to get around it? Factor code doesn't follow the C calling convention so it's unlikely that a debugger will be able to walk a stack with Factor frames. In recent versions of Factor you can trigger Factor's own low-level debugger with ^C, which will let you backtrace and step through Factor frames. -Joe -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Di
Re: [Factor-talk] Any way of making sense of what's in the boot image?
Thank you, Joe, once again for your help. I have settled into a nice rythym of virtually tracing through the code by looking at the source (both vm and factor files), augmented by the Help system in the listener. That should keep me busy and out of your hair for a couple months (or couple years? :( ). Let me hold off on asking any more questions until I've had a chance to absorb some of this. Maybe one little question. Am I correct in suspecting that there is a high-level Factor word I can execute on demand and trace through the exectution of via the Factor debugger to get a more dynamic picture of how this all unfolds? Would it be make-image? Can you just call that directly or is it only called in the context of a sequence of operations that have already set other things up? > Date: Wed, 8 Aug 2012 21:12:23 -0700 > From: arc...@gmail.com > To: factor-talk@lists.sourceforge.net > Subject: Re: [Factor-talk] Any way of making sense of what's in the boot > image? > > On Wed, Aug 8, 2012 at 8:23 PM, Joe Groff wrote: > > > > The code in the bootstrap.image module outputs the bootstrap image as > > a binary blob. You can get a sense for the format of the bootstrap > > image and its contents by looking at that module. > > To clarify, by "output as a binary blob" I mean the code in > bootstrap.image doesn't have any special help from the VM—it just > generates a byte array whose contents resemble a Factor image and > outputs that to a file. It could be transliterated to another language > and still generate a valid boot image. > > -Joe > > -- > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > ___ > Factor-talk mailing list > Factor-talk@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/factor-talk -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Factor-talk mailing list Factor-talk@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/factor-talk
