Re: [dev] Preprocessor

[Adrian Grigore]

In HTML5, attribute quotes are optional so I'm going to remove all sanitization.

On Thu, May 2, 2019 at 3:44 AM Adrian Grigore
 wrote:
>
> In the code itself as in compile your own `pp.c`.
>
> On Thu, May 2, 2019 at 2:45 AM Adrian Grigore
>  wrote:
> >
> > I sanitized double-quotes because they have a high frequency in HTML files 
> > and
> > you would have to do lots of escaping. I was also playing with HTML files 
> > while
> > developing.
> >
> > I sanitized backticks before "$()" are the POSIX preferred method for 
> > command
> > substitution. Rest are normal POSIX sh(1) strings.
> >
> > I may remove all form of sanitization and people could add it themselves in 
> > the
> > code itself as and if necessary.
> >
> > What do you think? What's your preference?
> >
> > On Wed, May 1, 2019 at 10:57 PM Teodoro Santoni  wrote:
> > >
> > > Hi,
> > >
> > > 2019-05-01 18:21 GMT, Adrian Grigore :
> > > > I think it's done:
> > > >
> > > > http://adi.tilde.institute/tmp/pp.c
> > > >
> > > >
> > > >
> > > > On Thu, Apr 25, 2019 at 6:44 AM Adrian Grigore
> > > >  wrote:
> > > >>
> > > >> Ok, refactored.
> > > >>
> > > >> http://adi.tilde.institute/tmp/pp.c
> > > >>
> > > >> Sorry, was experimenting before.
> > > >>
> > > >> On Wed, Apr 24, 2019 at 11:58 PM opal hart  wrote:
> > > >> >
> > > >> > On Tue, 23 Apr 2019 12:26:04 -0400
> > > >> > Cág  wrote:
> > > >> > > http://porkmail.org/era/unix/award.html
> > > >> >
> > > >> > `ls | cat` is actually useful to combat the differences between
> > > >> > implementations of ls(1) and ensure a one-column, uncoloured list
> > > >> >
> > > >> > --
> > > >> > wowaname 
> > > >> >
> > > >>
> > > >>
> > > >> --
> > > >> Thank you,
> > > >> Adi,
> > > >> Freelance Developer
> > > >> http://adi.tilde.institute/
> > > >
> > > >
> > > >
> > > > --
> > > > Thank you,
> > > > Adi,
> > > > Freelance Developer
> > > > http://adi.tilde.institute/
> > > >
> > > >
> > >
> > > Since you've ditched cat with heredocs for echo, and started to
> > > sanitize double-quotes and backticks running down the PHP path of
> > > doom, you may want to think about whether to sanitize or not dollar
> > > ($) symbols, too.
> > > It's a very good idea this one, I prefer this way to the m4 quirkies.
> > >
> >
> >
> > --
> > Thank you,
> > Adi,
> > Freelance Developer
> > http://adi.tilde.institute/
>
>
>
> --
> Thank you,
> Adi,
> Freelance Developer
> http://adi.tilde.institute/



-- 
Thank you,
Adi,
Freelance Developer
http://adi.tilde.institute/

Re: [dev] Preprocessor

[Piotr Oleskiewicz]

Hi,

I really like the project's scope & execution!

> Or that it's better done with a few lines of awk.

It has been done actually, see http://werc.cat-v.org/docs/rc-template-lang 
(implementation: 
https://code.9front.org/hg/werc/file/92f7463dac1a/bin/template.awk).

Best wishes,

Piotr

Re: [dev] Preprocessor

[Sijmen J. Mulder]

Hi Adrian,

Adrian Grigore  wrote:
> I built a preprocessor. It allows embedding shell code in any type of files.

A tool like this has been on my mind for a while but in a different
form: a more generic template-to-code tool that turns the literal parts
of a template into print statements:

 
   <% printf("Hello, %s!", "World!); %>
   Lorem ipsum dolor sit amet ...
 

becomes:

  ...
  int main() {
puts("[...]");
printf("Hello, %s!", "World!);
puts("[...]");
return 0;
  }

It would need to know a bit about the target language.

For your shell example it could look like:

  
  <% for p in "$1"/*; do %>

  <% echo "$p"; %>

  <% done %>
  

I'm not sure of the best way to deal with the multi language aspect, to
have it built in or configurable in some way.

But mostly I'm sure it must have been done before better than I could.
Or that it's better done with a few lines of awk.

Sijmen

Re: [dev] Preprocessor

[Adrian Grigore]

In the code itself as in compile your own `pp.c`.

On Thu, May 2, 2019 at 2:45 AM Adrian Grigore
 wrote:
>
> I sanitized double-quotes because they have a high frequency in HTML files and
> you would have to do lots of escaping. I was also playing with HTML files 
> while
> developing.
>
> I sanitized backticks before "$()" are the POSIX preferred method for command
> substitution. Rest are normal POSIX sh(1) strings.
>
> I may remove all form of sanitization and people could add it themselves in 
> the
> code itself as and if necessary.
>
> What do you think? What's your preference?
>
> On Wed, May 1, 2019 at 10:57 PM Teodoro Santoni  wrote:
> >
> > Hi,
> >
> > 2019-05-01 18:21 GMT, Adrian Grigore :
> > > I think it's done:
> > >
> > > http://adi.tilde.institute/tmp/pp.c
> > >
> > >
> > >
> > > On Thu, Apr 25, 2019 at 6:44 AM Adrian Grigore
> > >  wrote:
> > >>
> > >> Ok, refactored.
> > >>
> > >> http://adi.tilde.institute/tmp/pp.c
> > >>
> > >> Sorry, was experimenting before.
> > >>
> > >> On Wed, Apr 24, 2019 at 11:58 PM opal hart  wrote:
> > >> >
> > >> > On Tue, 23 Apr 2019 12:26:04 -0400
> > >> > Cág  wrote:
> > >> > > http://porkmail.org/era/unix/award.html
> > >> >
> > >> > `ls | cat` is actually useful to combat the differences between
> > >> > implementations of ls(1) and ensure a one-column, uncoloured list
> > >> >
> > >> > --
> > >> > wowaname 
> > >> >
> > >>
> > >>
> > >> --
> > >> Thank you,
> > >> Adi,
> > >> Freelance Developer
> > >> http://adi.tilde.institute/
> > >
> > >
> > >
> > > --
> > > Thank you,
> > > Adi,
> > > Freelance Developer
> > > http://adi.tilde.institute/
> > >
> > >
> >
> > Since you've ditched cat with heredocs for echo, and started to
> > sanitize double-quotes and backticks running down the PHP path of
> > doom, you may want to think about whether to sanitize or not dollar
> > ($) symbols, too.
> > It's a very good idea this one, I prefer this way to the m4 quirkies.
> >
>
>
> --
> Thank you,
> Adi,
> Freelance Developer
> http://adi.tilde.institute/



-- 
Thank you,
Adi,
Freelance Developer
http://adi.tilde.institute/

Re: [dev] Preprocessor

[Adrian Grigore]

I sanitized double-quotes because they have a high frequency in HTML files and
you would have to do lots of escaping. I was also playing with HTML files while
developing.

I sanitized backticks before "$()" are the POSIX preferred method for command
substitution. Rest are normal POSIX sh(1) strings.

I may remove all form of sanitization and people could add it themselves in the
code itself as and if necessary.

What do you think? What's your preference?

On Wed, May 1, 2019 at 10:57 PM Teodoro Santoni  wrote:
>
> Hi,
>
> 2019-05-01 18:21 GMT, Adrian Grigore :
> > I think it's done:
> >
> > http://adi.tilde.institute/tmp/pp.c
> >
> >
> >
> > On Thu, Apr 25, 2019 at 6:44 AM Adrian Grigore
> >  wrote:
> >>
> >> Ok, refactored.
> >>
> >> http://adi.tilde.institute/tmp/pp.c
> >>
> >> Sorry, was experimenting before.
> >>
> >> On Wed, Apr 24, 2019 at 11:58 PM opal hart  wrote:
> >> >
> >> > On Tue, 23 Apr 2019 12:26:04 -0400
> >> > Cág  wrote:
> >> > > http://porkmail.org/era/unix/award.html
> >> >
> >> > `ls | cat` is actually useful to combat the differences between
> >> > implementations of ls(1) and ensure a one-column, uncoloured list
> >> >
> >> > --
> >> > wowaname 
> >> >
> >>
> >>
> >> --
> >> Thank you,
> >> Adi,
> >> Freelance Developer
> >> http://adi.tilde.institute/
> >
> >
> >
> > --
> > Thank you,
> > Adi,
> > Freelance Developer
> > http://adi.tilde.institute/
> >
> >
>
> Since you've ditched cat with heredocs for echo, and started to
> sanitize double-quotes and backticks running down the PHP path of
> doom, you may want to think about whether to sanitize or not dollar
> ($) symbols, too.
> It's a very good idea this one, I prefer this way to the m4 quirkies.
>


-- 
Thank you,
Adi,
Freelance Developer
http://adi.tilde.institute/

Re: [dev] Preprocessor

[Teodoro Santoni]

Hi,

2019-05-01 18:21 GMT, Adrian Grigore :
> I think it's done:
>
> http://adi.tilde.institute/tmp/pp.c
>
>
>
> On Thu, Apr 25, 2019 at 6:44 AM Adrian Grigore
>  wrote:
>>
>> Ok, refactored.
>>
>> http://adi.tilde.institute/tmp/pp.c
>>
>> Sorry, was experimenting before.
>>
>> On Wed, Apr 24, 2019 at 11:58 PM opal hart  wrote:
>> >
>> > On Tue, 23 Apr 2019 12:26:04 -0400
>> > Cág  wrote:
>> > > http://porkmail.org/era/unix/award.html
>> >
>> > `ls | cat` is actually useful to combat the differences between
>> > implementations of ls(1) and ensure a one-column, uncoloured list
>> >
>> > --
>> > wowaname 
>> >
>>
>>
>> --
>> Thank you,
>> Adi,
>> Freelance Developer
>> http://adi.tilde.institute/
>
>
>
> --
> Thank you,
> Adi,
> Freelance Developer
> http://adi.tilde.institute/
>
>

Since you've ditched cat with heredocs for echo, and started to
sanitize double-quotes and backticks running down the PHP path of
doom, you may want to think about whether to sanitize or not dollar
($) symbols, too.
It's a very good idea this one, I prefer this way to the m4 quirkies.

Re: [dev] Preprocessor

[Adrian Grigore]

I think it's done:

http://adi.tilde.institute/tmp/pp.c



On Thu, Apr 25, 2019 at 6:44 AM Adrian Grigore
 wrote:
>
> Ok, refactored.
>
> http://adi.tilde.institute/tmp/pp.c
>
> Sorry, was experimenting before.
>
> On Wed, Apr 24, 2019 at 11:58 PM opal hart  wrote:
> >
> > On Tue, 23 Apr 2019 12:26:04 -0400
> > Cág  wrote:
> > > http://porkmail.org/era/unix/award.html
> >
> > `ls | cat` is actually useful to combat the differences between
> > implementations of ls(1) and ensure a one-column, uncoloured list
> >
> > --
> > wowaname 
> >
>
>
> --
> Thank you,
> Adi,
> Freelance Developer
> http://adi.tilde.institute/



--
Thank you,
Adi,
Freelance Developer
http://adi.tilde.institute/

Re: [dev] Preprocessor

[Adrian Grigore]

Ok, refactored.

http://adi.tilde.institute/tmp/pp.c

Sorry, was experimenting before.

On Wed, Apr 24, 2019 at 11:58 PM opal hart  wrote:
>
> On Tue, 23 Apr 2019 12:26:04 -0400
> Cág  wrote:
> > http://porkmail.org/era/unix/award.html
>
> `ls | cat` is actually useful to combat the differences between
> implementations of ls(1) and ensure a one-column, uncoloured list
>
> --
> wowaname 
>


-- 
Thank you,
Adi,
Freelance Developer
http://adi.tilde.institute/

Re: [dev] Preprocessor

[opal hart]

On Tue, 23 Apr 2019 12:26:04 -0400
Cág  wrote:
> http://porkmail.org/era/unix/award.html

`ls | cat` is actually useful to combat the differences between
implementations of ls(1) and ensure a one-column, uncoloured list

-- 
wowaname 

Re: [dev] Preprocessor

[Hadrien Lacour]

On Tue, Apr 23, 2019 at 12:44:57PM -0700, Evan Gates wrote:
> On Tue, Apr 23, 2019 at 12:42 PM Hadrien Lacour
>  wrote:
> > That was just shitposting. I use `find` to avoid most of the UNIX 
> > braindamage
> > in this case.
>
> Which is good as long as you use -exec correctly. Or if you're going
> to use xargs make sure to use nul separated lists. Xargs without the
> nul option is broken by design.
>
Yeah, I know. Always baffled me that POSIX allows '\n' in filenames without
standardizing the various '\0' options (read -0, find -print0, xargs -0,
sed -Z, etc...).
If only ASCII FS-US had been used as intended.

Re: [dev] Preprocessor

[Adrian Grigore]

Ignore the shell stuff, I know it's bad (I also usually use find(1)
for these cases), good comments however.
It's good you mentioned it so anybody reading this would know it's
bad, thank you!

On Tue, Apr 23, 2019 at 11:07 PM Peter Nagy  wrote:
>
> This was a question about the preprocessor, please take your sh (off-)topic 
> to a different thread.
>
> That main, mostly the switch part, is huge! Having more than 3 levels of 
> nesting of loops+conditionals (if, for, while, switch) is a code smell; it is 
> hard to read, hard to reason about and easy to get wrong.
>
> Bonus points for not using lex/yacc etc, writing a parser by hand needn't be 
> hard. Still, some refactoring to create smaller, readable and maintainable 
> functions would go a long way. Maybe trying out to write out the states the 
> parser can be in as a table would help to see if the parser can be simplified 
> or the token choices changed. Toying with an FSM implementation might be fun 
> too :)
>

I'm VERY aware of that, I'm still toying with it, I'll probably
abstract those just not sure about the abstractions yet so I
kept them there in my face.

I'd rather not change the token as "#!" reminds me of the shell.
Changing the token however would remove some code.

>
> Why are you building this tool? Fun, education, prototype? What is the end 
> goal? Who is the audience?
>

Generating static web sites would be an idea or could be embedded in
Markdown files.

My doubts are mostly about error handling and undefined behaviour.

I updated it, still no refactoring tho.

Speeds are comparable to PHP for simple IO code.

http://adi.tilde.institute/tmp/pp.c

Thanks for the feedback!

> Good luck with your project!
> --
> Peter Nagy
>
>  - To reach a goal one has to enjoy the journey
>
> On April 23, 2019 7:44:57 PM UTC, Evan Gates  wrote:
> >On Tue, Apr 23, 2019 at 12:42 PM Hadrien Lacour
> > wrote:
> >> That was just shitposting. I use `find` to avoid most of the UNIX
> >braindamage
> >> in this case.
> >
> >Which is good as long as you use -exec correctly. Or if you're going
> >to use xargs make sure to use nul separated lists. Xargs without the
> >nul option is broken by design.
>


-- 
Thank you,
Adi

Re: [dev] Preprocessor

[Peter Nagy]

This was a question about the preprocessor, please take your sh (off-)topic to 
a different thread.

That main, mostly the switch part, is huge! Having more than 3 levels of 
nesting of loops+conditionals (if, for, while, switch) is a code smell; it is 
hard to read, hard to reason about and easy to get wrong.

Bonus points for not using lex/yacc etc, writing a parser by hand needn't be 
hard. Still, some refactoring to create smaller, readable and maintainable 
functions would go a long way. Maybe trying out to write out the states the 
parser can be in as a table would help to see if the parser can be simplified 
or the token choices changed. Toying with an FSM implementation might be fun 
too :)


Why are you building this tool? Fun, education, prototype? What is the end 
goal? Who is the audience? 

Good luck with your project!
--
Peter Nagy

 - To reach a goal one has to enjoy the journey

On April 23, 2019 7:44:57 PM UTC, Evan Gates  wrote:
>On Tue, Apr 23, 2019 at 12:42 PM Hadrien Lacour
> wrote:
>> That was just shitposting. I use `find` to avoid most of the UNIX
>braindamage
>> in this case.
>
>Which is good as long as you use -exec correctly. Or if you're going
>to use xargs make sure to use nul separated lists. Xargs without the
>nul option is broken by design.

Re: [dev] Preprocessor

[Evan Gates]

On Tue, Apr 23, 2019 at 12:42 PM Hadrien Lacour
 wrote:
> That was just shitposting. I use `find` to avoid most of the UNIX braindamage
> in this case.

Which is good as long as you use -exec correctly. Or if you're going
to use xargs make sure to use nul separated lists. Xargs without the
nul option is broken by design.

Re: [dev] Preprocessor

[Hadrien Lacour]

On Tue, Apr 23, 2019 at 09:30:14AM -0700, Evan Gates wrote:
> On Tue, Apr 23, 2019 at 9:24 AM Hadrien Lacour
>  wrote:
> > What if "$1" is empty? POSIX sh doesn't have the nullglob shop, you know.
>
> [ "$1" ] || exit # add a message if you want
> [ -d "$1" ] || exit # if you want to check that the directory exists
> for p in "$1"/*; do [ -e "$p" ] || continue; ... # if you want to make
> sure the glob expanded and files exist
>
> Another option is:
>
> [ "$1" ] || exit
> set -- "$1"/*
> [ -e "$1" ] || exit
> for p do ...; done
>
> And in reality use something to print an error message, not just exit. E.g.:
>
> die() { printf %s\\n "$*" >&2; exit 1; }
> [ "$1" ] || die Argument is empty
> [ -d "$1" ] || die "Directory does not exist: $1"
>
> etc.
>

That was just shitposting. I use `find` to avoid most of the UNIX braindamage
in this case.

Re: [dev] Preprocessor

[Cág]

Evan Gates wrote:


Not sure about the preprocessor stuff, but this right here
is terrible practice. Use a for loop and glob. Assuming
that "$1" is a directory:

for p in "$1"/*; do ...

http://mywiki.wooledge.org/ParsingLs


http://porkmail.org/era/unix/award.html

--
caóc

Re: [dev] Preprocessor

[Evan Gates]

On Tue, Apr 23, 2019 at 9:24 AM Hadrien Lacour
 wrote:
> What if "$1" is empty? POSIX sh doesn't have the nullglob shop, you know.

[ "$1" ] || exit # add a message if you want
[ -d "$1" ] || exit # if you want to check that the directory exists
for p in "$1"/*; do [ -e "$p" ] || continue; ... # if you want to make
sure the glob expanded and files exist

Another option is:

[ "$1" ] || exit
set -- "$1"/*
[ -e "$1" ] || exit
for p do ...; done

And in reality use something to print an error message, not just exit. E.g.:

die() { printf %s\\n "$*" >&2; exit 1; }
[ "$1" ] || die Argument is empty
[ -d "$1" ] || die "Directory does not exist: $1"

etc.

Re: [dev] Preprocessor

[Hadrien Lacour]

On Mon, Apr 22, 2019 at 01:35:02PM -0700, Evan Gates wrote:
> On Mon, Apr 22, 2019 at 1:25 PM Adrian Grigore
>  wrote:
> >
> > ls -1 "$1" | while IFS= read -r p
> > do
>
> Not sure about the preprocessor stuff, but this right here is terrible
> practice. Use a for loop and glob. Assuming that "$1" is a directory:
>
> for p in "$1"/*; do ...
>
> http://mywiki.wooledge.org/ParsingLs
>

What if "$1" is empty? POSIX sh doesn't have the nullglob shop, you know.

Re: [dev] Preprocessor

[Evan Gates]

On Mon, Apr 22, 2019 at 1:25 PM Adrian Grigore
 wrote:
>
> ls -1 "$1" | while IFS= read -r p
> do

Not sure about the preprocessor stuff, but this right here is terrible
practice. Use a for loop and glob. Assuming that "$1" is a directory:

for p in "$1"/*; do ...

http://mywiki.wooledge.org/ParsingLs

[dev] Preprocessor

[Adrian Grigore]

Hi,

I built a preprocessor. It allows embedding shell code in any type of files.
Can you guys give some tips or a code review?

The file format it accepts is the following:

#!/home/adi/src/pp/pp




#!
ls -1 "$1" | while IFS= read -r p
do
#!
   

  $p

   
#!
done
#!



Here's the code:

http://adi.tilde.institute/tmp/pp.c

--
Thank you,
Adi,