Re: Cleaning imported javascript headlines

[Edward K. Ream]

On Friday, February 23, 2018 at 11:35:42 AM UTC-6, Edward K. Ream wrote:

> Afaik, there are few if any problems with other languages.

Just for the record, there is a generic problem with all importers that has 
no real solution, namely constructs that look like section references.  
Recognizing such things isn't the problem.

Rather, the problem is how to alter the original code.  Any of Leo's 
importers will generate an @ignore directive for such files because the 
"missing" section reference will cause the perfect import checks to fail.  
It will then be up to the user to change the sources.  Imo, this is a 
reasonable approach.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Cleaning imported javascript headlines

[vitalije]

many (most?) problems with imports can be fixed by hand in the generated 
> @clean tree.

Yes, I am aware of this. But it turned out quite often for me that amount 
of work needed for that manual fixing exceeds the amount of work needed to 
split file by hand in the first place.

I would not argue that files from some (possible many) other languages can 
be imported properly. But there is not so small number of languages like 
javascript which can't be imported satisfactorily. Javascript itself has a 
number of dialects, scala is another example. Groovy also comes to my mind. 
They have so rich syntax that I really doubt that Leo in its current form 
can handle them very well. I haven't tested it, but every time I used 
import I ended up doing lot of manual reshaping and quite often I abandoned 
the whole process and started all over again with manual splitting file.

Vitalije

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Cleaning imported javascript headlines

[Edward K. Ream]

On Friday, February 23, 2018 at 10:22:55 AM UTC-6, vitalije wrote:

Cleaning generated headlines is just a part of the problem. More serious 
> problem is choosing what should go in to single node.
>

I agree completely.
 

> As far as I am concerned, I would much prefer to have a toolbox with 
> several commands to help me importing files on my own. I wrote about that 
> idea here 
> . I 
> don't know when (and if) I would have time to write such a plugin. Solving 
> this serious problem (* importing external source files) in fully automatic 
> but satisfactory way IMHO must involve writing a proper lexer/parser 
> functions at least for .js .jsx, .ts .tsx files. That seems to be a lot of 
> work, so I believe more likely is that I would pursue this simpler idea - a 
> plugin with handful of universal functions for manipulating source code, 
> that user can use to reshape code any way he/she likes. That would work for 
> all kinds of source files.
>

Remember that all importers have a post pass in which they are free to 
alter the already generated nodes.

Javascript is probably unique in all commonly-used languages in not having 
a proper, *always used*, syntax for generating classes, functions and 
methods.  This makes the problem *much* harder for js than it is for all 
other languages.

Afaik, there are few if any problems with other languages.  Once in a while 
the python importer has troubles with if statements at the top level 
interacting with decorators.  That's about it.  Tweaking the python post 
pass should fix this, but it's a low priority item.

So yes, have at it if you will.  But remember that many (most?) problems 
with imports can be fixed by hand in the generated @clean tree.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Cleaning imported javascript headlines

[vitalije]

  There are problems assigning lines to nodes.
>
> Edward
>
Precisely  

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Cleaning imported javascript headlines

[Edward K. Ream]

On Friday, February 23, 2018 at 10:32:59 AM UTC-6, Edward K. Ream wrote:

I am working on improvements [to .  Please hold suggestions for a bit.
>

Rev 0deba52 completes the present round of work on this project. There are 
no problems that I can see with the *headlines* when importing leovue/src.  
There are problems assigning lines to nodes.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Cleaning imported javascript headlines

[Edward K. Ream]

On Friday, February 23, 2018 at 9:42:24 AM UTC-6, Edward K. Ream wrote:

> The javascript importer now defines [clean_headline] this way:

I am working on improvements.  Please hold suggestions for a bit.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Cleaning imported javascript headlines

[vitalije]

Cleaning generated headlines is just a part of the problem. More serious 
problem is choosing what should go in to single node. As far as I am 
concerned, I would much prefer to have a toolbox with several commands to 
help me importing files on my own. I wrote about that idea here 
. I 
don't know when (and if) I would have time to write such a plugin. Solving 
this serious problem (* importing external source files) in fully automatic 
but satisfactory way IMHO must involve writing a proper lexer/parser 
functions at least for .js .jsx, .ts .tsx files. That seems to be a lot of 
work, so I believe more likely is that I would pursue this simpler idea - a 
plugin with handful of universal functions for manipulating source code, 
that user can use to reshape code any way he/she likes. That would work for 
all kinds of source files.
Vitalije

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Cleaning imported javascript headlines

[Edward K. Ream]

Leo's import system has many strengths.  The most important is that code 
will almost always import correctly even if the resulting nodes aren't even 
close to optimal.  This is particularly important for javascript, where 
coding styles vary so widely.

After generating nodes, the import system calls the clean_headline method.  
The base Importer class defines a clean_headline as follows:

def clean_headline(self, s, p=None):
'''
Return the cleaned version headline s.
Will typically be overridden in subclasses.
'''
return s.strip()

The javascript importer now defines this method this way:

clean_regex_list1 = [
re.compile(r'\s*\(?(function\b\s*[\w]*)\s*\('),
re.compile(r'\s*([\w]+\:\s*\(*\s*function\s*\()'),
re.compile(r'\s*(?:const|let|var)\s*(\w+\s*(?:=\s*.*)=>)'),
]
clean_regex_list2 = [
re.compile(r'(.*)\((\s*function)'),
re.compile(r'(.*\=)(\s*function)'),
]

def clean_headline(self, s, p=None):
'''Return a cleaned up headline s.'''
s = s.strip()
# Don't clean a headline twice.
if s.endswith('>>') and s.startswith('<<'):
return s
for ch in '{(=':
if s.endswith(ch):
s = s[:-1].strip()
# First regex cleanup.
for pattern in self.clean_regex_list1:
m = pattern.match(s)
if m:
s = m.group(1)
break
# Second regex cleanup.
for pattern in self.clean_regex_list2:
m = pattern.match(s)
if m:
s = m.group(1) + m.group(2)
break
s = s.replace('  ', ' ')
return g.truncate(s, 100)

This isn't perfect, but is much better than previous versions.  I encourage 
Vitalije or anyone else to suggest improvements.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.