Re: [Factor-talk] New "Modern" Factor parser

[Alexander Ilin]

Hello, Doug! As you know, I'm a newbie in Factor, so if I ask some dumb questions, please have patience or postpone your answer until you can contain your rage : )) Basically, I did not understand most of what you wrote in that epochal e-mail about the locals-and-roots branch, and some of the things I imagine I did understand I'd like you to comment on, just to make sure I have too vivid an imagination. 04.07.2016, 05:18, "Doug Coleman" :There's a branch called locals-and-roots that has locals/fry/macros/memoize/stack-checker in core Does that mean that the minimal size of the executable is growing again? A) Goals of the new parser: 1) Simplify/regularize the Factor syntax2) Remove parsing words' ability to take over the parser3) Keep the parsed text for refactoring4) Allow out of order definitions and maybe circular vocabulary dependencies Ouch! Are you sure about this? Is there a need that outweighs the headache? Or is there no headache? 5) Easier syntax for DSLs6) Support the same syntax everywhere7) Allow custom lexer/file extension associations8) Load all code on every platform (but not run) How will 3) and 8) affect the minimum memory footprint?What about the image file size? 9) Add strings with arbitrary payloads What's an arbitrary payload? An example or a use case would be nice. 10) Make most valid-looking syntax lex, but not necessarily compile B) Syntax cheat sheet: TAG: objs ;TAG< FOO: ... ; BAR: ... ; TAG>tag[ objs ] tag{ objs } tag( objs )tag"text" tag[[text]] tag{{text}} tag((text))tag`text tag``text`` tag```text``` In the line above - does the first piece mean that tag`text does not have to end with a closing `?That seems weird and inconsistent. Is there a justification? Do I have to use double ticks to include a space? tag\ text In the line above can text be "a quoted text with spaces" or is it limited to a spaceless token?If it can be quoted, can it be tagged as well? tag1\ tag2\ tag3[text tag3]? tag:: objobj1: obj2 ! { obj1 obj2 }-- ! splits a sequence, e.g. ( a -- b ) -> ( { a } { b } )tag! text til end of line  C) More syntax explanation: Top level definitions are:  FOO: ... ; or FOO: ... FOO; Delimiters close like:V{ 1 2 3 } or V{ 1 2 3 V}asdf[ 1 2 3 ] or asdf[ 1 2 3 asdf]  What about tag"some string tag"? Is the second instance of "tag" a part of the tagged string, or is it a repeated closing tag? Nesting definitions inside others:PRIVATE<  FOO: ... ;  BAR: ... ;PRIVATE>  Does this mean that the old syntax of  is no longer valid, or is it no longer idiomatic, or both?I kind of like the  syntax, and find it easily understandable. Will it be possible to define words like cm>inch, or will that open a scope that has to end with cm<, or will it trigger the syntax error of "missing the starting cm< tag"? Syntax itself is concatenative -- you can do things like this:V{ 1 2 3 }[ 0 ]  ! should desugar to C-style array access Even without the explicit nth word? I don't understand this. Isn't [ 0 ] a quotation that pushes 0 onto stack? D) Implementation: All syntax starts with tags (text without delimiters) and tokens end with whitespace or a delimiter. Could you define "delimiter", please? A lexing rule is some self-contained rule that parses text according to the delimiter encountered. These rules are used to group tokens together, so at the end of parsing a .factor file, there is a sequence of standalone definitions grouped with their decorators, like inline/foldable/flushable/etc. In theory, you could at this point ``randomize`` the top-level definitions, rewrite them to disk, then reparse and have the same code, even repeating any number of times.  1) single-line-lexer - like a ! comment right now The tags for these rules should just be for metadata. I want to change the comment character to # eventually instead of ! (thoughts?) Examples:author! erg! No tag, plain comment  While I have no objection to # for a comment starter, I don't necessarily like the idea of a tagged comment.But even if there is a good use for such things, to me, a comment starts with a designated character and continues to EOL. It can't eat a word to the left of it.Same as with 

[Factor-talk] New "Modern" Factor parser

[Doug Coleman]

Hi All,

I wrote a new parser after bouncing a ton of emails off John (thanks!).
It's ready enough to be worked on by other people. There's a branch called
locals-and-roots that has locals/fry/macros/memoize/stack-checker in core,
has a new vocab roots layout for demos/language/libs/ffi/webapps/apps (not
sure I like it), and has the new parser alongside the existing parser in
core/modern/*.

For the most part, everything looks the same. I'm still working on $example
syntax, functors, compilation, refactoring tools, fry in arrays,
stack-checker improvements, etc. You're welcome to help with ideas or code!

Repository:
git://factorcode.org/git/factor.git

Branch:
locals-and-roots

Boot images:
http://downloads.factorcode.org/images/boot.unix-x86.64.image.locals-and-roots
http://downloads.factorcode.org/images/boot.windows-x86.64.image.locals-and-roots

Source:
http://gitweb.factorcode.org/gitweb.cgi?p=factor.git;a=blob;f=core/modern/modern.factor;h=d244d774cfdc7eb1e7f52ccd0091a94f56944a25;hb=refs/heads/locals-and-roots

http://gitweb.factorcode.org/gitweb.cgi?p=factor.git;a=blob;f=core/modern/modern-tests.factor;h=9bdf7183e74ef084c2188f10d5ef01c893e82363;hb=refs/heads/locals-and-roots


A) Goals of the new parser:

1) Simplify/regularize the Factor syntax
2) Remove parsing words' ability to take over the parser
3) Keep the parsed text for refactoring
4) Allow out of order definitions and maybe circular vocabulary dependencies
5) Easier syntax for DSLs
6) Support the same syntax everywhere
7) Allow custom lexer/file extension associations
8) Load all code on every platform (but not run)
9) Add strings with arbitrary payloads
10) Make most valid-looking syntax lex, but not necessarily compile



B) Syntax cheat sheet:

TAG: objs ;
TAG< FOO: ... ; BAR: ... ; TAG>
tag[ objs ] tag{ objs } tag( objs )
tag"text" tag[[text]] tag{{text}} tag((text))
tag`text tag``text`` tag```text```
tag\ text
tag:: obj
obj1: obj2 ! { obj1 obj2 }
-- ! splits a sequence, e.g. ( a -- b ) -> ( { a } { b } )
tag! text til end of line


C) More syntax explanation:

Top level definitions are:
  FOO: ... ; or FOO: ... FOO;

Delimiters close like:
V{ 1 2 3 } or V{ 1 2 3 V}
asdf[ 1 2 3 ] or asdf[ 1 2 3 asdf]

Nesting definitions inside others:
PRIVATE<
  FOO: ... ;
  BAR: ... ;
PRIVATE>

Nested definitions where ``PROTOCOL<`` has arity 1 (one fixed argument
spot):
PROTOCOL< sequence
  GENERIC: set-nth ( obj n seq -- )
  GENERIC: nth ( n seq -- obj )
PROTOCOL>

Semi-colons are optional if you declare an arity:
ARITY: \ CONSTANT: 2
CONSTANT: a 1 ;
CONSTANT: b 2

Top-level definitions and definition-groups interrupt a colon-definition:
: add ( a b -- c ) +   ! error because : has no arity declaration
   ! and semi-colon is required
PRIVATE<
..
PRIVATE>

Definition groups nest:
PRIVATE<
STUFF<
   THING1: .. ;
   THING2: .. ;
STUFF>
PRIVATE>

The -- from stack effects can generalize to a way to split a sequence in
two without needing extra delimiters. So ( a -- b ) becomes ( { a } { b }
), or H{ 1 2 -- 3 4 } is H{ { 1 2 } { 3 4 } }.

Another pattern we use is ``a: b``, like in stack effects, which makes a
pair.
( a quot: ( b -- c ) -- d )
( a { quot ( b -- c ) } -- d ) ! after pair rule
( { a { quot ( b -- c ) } } { d } ) ! after -- rule
( { a { quot { { b } { c } } } } { d } ) ! after -- rule

Colon pairs means that H{ 1: 2 3: 4 } would work as well. This is slightly
tricky/needs more thought.


Syntax itself is concatenative -- you can do things like this:
V{ 1 2 3 }[ 0 ]  ! should desugar to C-style array access

Of course, you have to compile the new forms or throw an exception.


D) Implementation:

All syntax starts with tags (text without delimiters) and tokens end with
whitespace or a delimiter. A lexing rule is some self-contained rule that
parses text according to the delimiter encountered. These rules are used to
group tokens together, so at the end of parsing a .factor file, there is a
sequence of standalone definitions grouped with their decorators, like
inline/foldable/flushable/etc.

In theory, you could at this point ``randomize`` the top-level definitions,
rewrite them to disk, then reparse and have the same code, even repeating
any number of times.


1) single-line-lexer - like a ! comment right now

The tags for these rules should just be for metadata. I want to change the
comment character to # eventually instead of ! (thoughts?)

Examples:
author! erg
! No tag, plain comment


2)  backtick-lexer - single backtick parses til whitespace, multiple parses
til matching number of backticks.

Examples:
char`a char``a`` char```a```
fixnum`3 fixnum``345``


3) backslash-lexer - a \ turns off the parsing rules for the next token

If you don't escape a lexing form, then it will execute its rule. So this
is how you turn that behavior off.

Examples:
SYNTAX: \ url" ... ;
\ {

! some words in the smalltalk vocabulary end in
! colon, need to escape them like this to call them
! or rename them
execute\ 

[Factor-talk] New to Factor

[graham telfer]

Hi there,

I have just downloaded Factor and look forward to learning it. I used to 
program using Forth and it is good to see that one of the major gripes I always 
had about Forth: its lack of data structures and the intoned mantra , Forth 
doesn't provide data structures because it is so easy to design and build your 
own has finally been sunk.

I have a question to ask. One of my interests is constraint programming. Has 
any work been done in Factor towards developing constraint solvers? 
  --
Using storage to extend the benefits of virtualization and iSCSI
Virtualization increases hardware utilization and delivers a new level of
agility. Learn what those decisions are and how to modernize your storage 
and backup environments for virtualization.
http://www.accelacomm.com/jaw/sfnl/114/51434361/___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk