Want to help DMD bugfixing? Write a simple utility.
Here's the task:
Given a .d source file, strip out all of the unittest {} blocks,
including everything inside them.
Strip out all comments as well.
Print out the resulting file.
Motivation: Bug reports frequently come with very large test cases.
Even ones which look small often import from Phobos.
Reducing the test case is the first step in fixing the bug, and it's
frequently ~30% of the total time required. Stripping out the unit tests
is the most time-consuming and error-prone part of reducing the test case.
This should be a good task if you're relatively new to D but would like
to do something really useful.
-Don
Re: Want to help DMD bugfixing? Write a simple utility.
On 3/20/11 1:11 AM, Don wrote:
Here's the task:
Given a .d source file, strip out all of the unittest {} blocks,
including everything inside them.
Strip out all comments as well.
Print out the resulting file.
I realize that you asked for a very specific utility, but in several
instances, http://delta.tigris.org/ worked fine for me for reducing
large test cases.
Parts of it are tailored to C/C++ though, so a port/adaption for D would
be a nice project as well.
David
Re: Want to help DMD bugfixing? Write a simple utility.
On Saturday 19 March 2011 17:11:56 Don wrote:
> Here's the task:
> Given a .d source file, strip out all of the unittest {} blocks,
> including everything inside them.
> Strip out all comments as well.
> Print out the resulting file.
>
> Motivation: Bug reports frequently come with very large test cases.
> Even ones which look small often import from Phobos.
> Reducing the test case is the first step in fixing the bug, and it's
> frequently ~30% of the total time required. Stripping out the unit tests
> is the most time-consuming and error-prone part of reducing the test case.
>
> This should be a good task if you're relatively new to D but would like
> to do something really useful.
Unfortunately, to do that 100% correctly, you need to actually have a working D
lexer (and possibly parser). You might be able to get something close enough to
work in most cases, but it doesn't take all that much to throw off a basic
implementation of this sort of thing if you don't lex/parse it with something
which properly understands D.
- Jonathan M Davis
Re: Want to help DMD bugfixing? Write a simple utility.
Jonathan M Davis wrote:
On Saturday 19 March 2011 17:11:56 Don wrote:
Here's the task:
Given a .d source file, strip out all of the unittest {} blocks,
including everything inside them.
Strip out all comments as well.
Print out the resulting file.
Motivation: Bug reports frequently come with very large test cases.
Even ones which look small often import from Phobos.
Reducing the test case is the first step in fixing the bug, and it's
frequently ~30% of the total time required. Stripping out the unit tests
is the most time-consuming and error-prone part of reducing the test case.
This should be a good task if you're relatively new to D but would like
to do something really useful.
Unfortunately, to do that 100% correctly, you need to actually have a working D
lexer (and possibly parser). You might be able to get something close enough to
work in most cases, but it doesn't take all that much to throw off a basic
implementation of this sort of thing if you don't lex/parse it with something
which properly understands D.
- Jonathan M Davis
I didn't say it needs 100% accuracy. You can assume, for example, that
"unittest" always occurs at the start of a line. The only other things
you need to lex are {}, string literals, and comments.
BTW, the immediate motivation for this is std.datetime in Phobos. The
sheer number of unittests in there is an absolute catastrophe for
tracking down bugs. It makes a tool like this MANDATORY.
Re: Want to help DMD bugfixing? Write a simple utility.
On Saturday 19 March 2011 18:04:57 Don wrote:
> Jonathan M Davis wrote:
> > On Saturday 19 March 2011 17:11:56 Don wrote:
> >> Here's the task:
> >> Given a .d source file, strip out all of the unittest {} blocks,
> >> including everything inside them.
> >> Strip out all comments as well.
> >> Print out the resulting file.
> >>
> >> Motivation: Bug reports frequently come with very large test cases.
> >> Even ones which look small often import from Phobos.
> >> Reducing the test case is the first step in fixing the bug, and it's
> >> frequently ~30% of the total time required. Stripping out the unit tests
> >> is the most time-consuming and error-prone part of reducing the test
> >> case.
> >>
> >> This should be a good task if you're relatively new to D but would like
> >> to do something really useful.
> >
> > Unfortunately, to do that 100% correctly, you need to actually have a
> > working D lexer (and possibly parser). You might be able to get
> > something close enough to work in most cases, but it doesn't take all
> > that much to throw off a basic implementation of this sort of thing if
> > you don't lex/parse it with something which properly understands D.
> >
> > - Jonathan M Davis
>
> I didn't say it needs 100% accuracy. You can assume, for example, that
> "unittest" always occurs at the start of a line. The only other things
> you need to lex are {}, string literals, and comments.
>
> BTW, the immediate motivation for this is std.datetime in Phobos. The
> sheer number of unittests in there is an absolute catastrophe for
> tracking down bugs. It makes a tool like this MANDATORY.
I tried to create a similar tool before and gave up because I couldn't make it
100% accurate and was running into problems with it. If someone wants to take a
shot at it though, that's fine.
As for the unit tests in std.datetime making it hard to track down bugs, that
only makes sense to me if you're trying to look at the whole thing at once and
track down a compiler bug which happens _somewhere_ in the code, but you don't
know where. Other than a problem like that, I don't really see how the unit
tests get in the way of tracking down bugs. Is it that you need to compile in a
version of std.datetime which doesn't have any unit tests compiled in but you
still need to compile with -unittest for other stuff?
I _am_ working on streamlining the unit tests in std.datetime so that they take
up fewer lines of code without reducing how well they cover the code, so
depending on your problem with the amount of unit test code, that could help,
but I expect that whatever your core problem with the number of unit tests is,
that won't fix it.
- Jonathan M Davis
Re: Want to help DMD bugfixing? Write a simple utility.
On 2011-03-19 20:41:09 -0400, Jonathan M Davis said:
On Saturday 19 March 2011 17:11:56 Don wrote:
Here's the task:
Given a .d source file, strip out all of the unittest {} blocks,
including everything inside them.
Strip out all comments as well.
Print out the resulting file.
Motivation: Bug reports frequently come with very large test cases.
Even ones which look small often import from Phobos.
Reducing the test case is the first step in fixing the bug, and it's
frequently ~30% of the total time required. Stripping out the unit tests
is the most time-consuming and error-prone part of reducing the test case.
This should be a good task if you're relatively new to D but would like
to do something really useful.
Unfortunately, to do that 100% correctly, you need to actually have a working D
lexer (and possibly parser). You might be able to get something close enough to
work in most cases, but it doesn't take all that much to throw off a basic
implementation of this sort of thing if you don't lex/parse it with something
which properly understands D.
Well, I made simple lexer for D strings, comments, identifiers and a
few other tokens which should be up to that task. It's what I use to
parse files and detect dependencies in D for Xcode. Unfortunately, it's
written in Objective-C++ (but half of it is plain C)...
Very short unit test:
--
Michel Fortin
michel.for...@michelf.com
http://michelf.com/
Re: Want to help DMD bugfixing? Write a simple utility.
Jonathan M Davis wrote:
On Saturday 19 March 2011 18:04:57 Don wrote:
Jonathan M Davis wrote:
On Saturday 19 March 2011 17:11:56 Don wrote:
Here's the task:
Given a .d source file, strip out all of the unittest {} blocks,
including everything inside them.
Strip out all comments as well.
Print out the resulting file.
Motivation: Bug reports frequently come with very large test cases.
Even ones which look small often import from Phobos.
Reducing the test case is the first step in fixing the bug, and it's
frequently ~30% of the total time required. Stripping out the unit tests
is the most time-consuming and error-prone part of reducing the test
case.
This should be a good task if you're relatively new to D but would like
to do something really useful.
Unfortunately, to do that 100% correctly, you need to actually have a
working D lexer (and possibly parser). You might be able to get
something close enough to work in most cases, but it doesn't take all
that much to throw off a basic implementation of this sort of thing if
you don't lex/parse it with something which properly understands D.
- Jonathan M Davis
I didn't say it needs 100% accuracy. You can assume, for example, that
"unittest" always occurs at the start of a line. The only other things
you need to lex are {}, string literals, and comments.
BTW, the immediate motivation for this is std.datetime in Phobos. The
sheer number of unittests in there is an absolute catastrophe for
tracking down bugs. It makes a tool like this MANDATORY.
I tried to create a similar tool before and gave up because I couldn't make it
100% accurate and was running into problems with it. If someone wants to take a
shot at it though, that's fine.
As for the unit tests in std.datetime making it hard to track down bugs, that
only makes sense to me if you're trying to look at the whole thing at once and
track down a compiler bug which happens _somewhere_ in the code, but you don't
know where. Other than a problem like that, I don't really see how the unit
tests get in the way of tracking down bugs. Is it that you need to compile in a
version of std.datetime which doesn't have any unit tests compiled in but you
still need to compile with -unittest for other stuff?
No. All you know there's a bug that's being triggered somewhere in
Phobos (with -unittest). It's probably not in std.datetime.
But Phobos is a horrible ball of mud where everything imports everything
else, and std.datetime is near the centre of that ball. What you have to
do is reduce the amount of code, and especially the number of modules,
as rapidly as possible; this means getting rid of imports.
To do this, you need to remove large chunks of code from the files. This
is pretty simple; comment out half of the file, if it still works, then
delete it. Normally this works well because typically only about a dozen
lines are actually being used. After doing this about three or four
times it's small enough that you can usually get rid of most of the imports.
Unittests foul this up because they use functions/classes from inside
the file.
In the case of std.datetime it's even worse because the signal-to-noise
ratio is so incredibly poor; it's really difficult to find the few lines
of code that are actually being used by other Phobos modules.
My experience (obviously only over the last month or so) has been that
if the reduction of a bug is non-obvious, more than 10% of the total
time taken to fix that bug is the time taken to cut down std.datetime.
Re: Want to help DMD bugfixing? Write a simple utility.
> Jonathan M Davis wrote:
> > On Saturday 19 March 2011 18:04:57 Don wrote:
> >> Jonathan M Davis wrote:
> >>> On Saturday 19 March 2011 17:11:56 Don wrote:
> Here's the task:
> Given a .d source file, strip out all of the unittest {} blocks,
> including everything inside them.
> Strip out all comments as well.
> Print out the resulting file.
>
> Motivation: Bug reports frequently come with very large test cases.
> Even ones which look small often import from Phobos.
> Reducing the test case is the first step in fixing the bug, and it's
> frequently ~30% of the total time required. Stripping out the unit
> tests is the most time-consuming and error-prone part of reducing the
> test case.
>
> This should be a good task if you're relatively new to D but would
> like to do something really useful.
> >>>
> >>> Unfortunately, to do that 100% correctly, you need to actually have a
> >>> working D lexer (and possibly parser). You might be able to get
> >>> something close enough to work in most cases, but it doesn't take all
> >>> that much to throw off a basic implementation of this sort of thing if
> >>> you don't lex/parse it with something which properly understands D.
> >>>
> >>> - Jonathan M Davis
> >>
> >> I didn't say it needs 100% accuracy. You can assume, for example, that
> >> "unittest" always occurs at the start of a line. The only other things
> >> you need to lex are {}, string literals, and comments.
> >>
> >> BTW, the immediate motivation for this is std.datetime in Phobos. The
> >> sheer number of unittests in there is an absolute catastrophe for
> >> tracking down bugs. It makes a tool like this MANDATORY.
> >
> > I tried to create a similar tool before and gave up because I couldn't
> > make it 100% accurate and was running into problems with it. If someone
> > wants to take a shot at it though, that's fine.
> >
> > As for the unit tests in std.datetime making it hard to track down bugs,
> > that only makes sense to me if you're trying to look at the whole thing
> > at once and track down a compiler bug which happens _somewhere_ in the
> > code, but you don't know where. Other than a problem like that, I don't
> > really see how the unit tests get in the way of tracking down bugs. Is
> > it that you need to compile in a version of std.datetime which doesn't
> > have any unit tests compiled in but you still need to compile with
> > -unittest for other stuff?
>
> No. All you know there's a bug that's being triggered somewhere in
> Phobos (with -unittest). It's probably not in std.datetime.
> But Phobos is a horrible ball of mud where everything imports everything
> else, and std.datetime is near the centre of that ball. What you have to
> do is reduce the amount of code, and especially the number of modules,
> as rapidly as possible; this means getting rid of imports.
>
> To do this, you need to remove large chunks of code from the files. This
> is pretty simple; comment out half of the file, if it still works, then
> delete it. Normally this works well because typically only about a dozen
> lines are actually being used. After doing this about three or four
> times it's small enough that you can usually get rid of most of the
> imports. Unittests foul this up because they use functions/classes from
> inside the file.
>
> In the case of std.datetime it's even worse because the signal-to-noise
> ratio is so incredibly poor; it's really difficult to find the few lines
> of code that are actually being used by other Phobos modules.
>
> My experience (obviously only over the last month or so) has been that
> if the reduction of a bug is non-obvious, more than 10% of the total
> time taken to fix that bug is the time taken to cut down std.datetime.
Hmmm. I really don't know what could be done to fix that (other than making it
easier to rip out the unittest blocks). And enough of std.datetime depends on
other parts of std.datetime that trimming it down isn't (and can't be) exactly
easy. In general, SysTime is the most likely type to be used, and it depends
on Date, TimeOfDay, and DateTime, and all 4 of those depend on most of the
free functions in the module. It's not exactly designed in a manner which
allows you to cut out large chunks and still have it compile. And I don't
think that it _could_ be designed that way and still have the functionality
that it has.
I guess that this sort of problem is one that would pop up mainly when dealing
with compiler bugs. I have a hard time seeing it popping up with your typical
bug in Phobos itself. So, I guess that this is the sort of thing that you'd
run into and I likely wouldn't.
I really don't know how the situation could be improved though other than
making it easier to cut out the unit tests.
- Jonathan M Davis
Re: Want to help DMD bugfixing? Write a simple utility.
Jonathan M Davis wrote:
Jonathan M Davis wrote:
On Saturday 19 March 2011 18:04:57 Don wrote:
Jonathan M Davis wrote:
On Saturday 19 March 2011 17:11:56 Don wrote:
Here's the task:
Given a .d source file, strip out all of the unittest {} blocks,
including everything inside them.
Strip out all comments as well.
Print out the resulting file.
Motivation: Bug reports frequently come with very large test cases.
Even ones which look small often import from Phobos.
Reducing the test case is the first step in fixing the bug, and it's
frequently ~30% of the total time required. Stripping out the unit
tests is the most time-consuming and error-prone part of reducing the
test case.
This should be a good task if you're relatively new to D but would
like to do something really useful.
Unfortunately, to do that 100% correctly, you need to actually have a
working D lexer (and possibly parser). You might be able to get
something close enough to work in most cases, but it doesn't take all
that much to throw off a basic implementation of this sort of thing if
you don't lex/parse it with something which properly understands D.
- Jonathan M Davis
I didn't say it needs 100% accuracy. You can assume, for example, that
"unittest" always occurs at the start of a line. The only other things
you need to lex are {}, string literals, and comments.
BTW, the immediate motivation for this is std.datetime in Phobos. The
sheer number of unittests in there is an absolute catastrophe for
tracking down bugs. It makes a tool like this MANDATORY.
I tried to create a similar tool before and gave up because I couldn't
make it 100% accurate and was running into problems with it. If someone
wants to take a shot at it though, that's fine.
As for the unit tests in std.datetime making it hard to track down bugs,
that only makes sense to me if you're trying to look at the whole thing
at once and track down a compiler bug which happens _somewhere_ in the
code, but you don't know where. Other than a problem like that, I don't
really see how the unit tests get in the way of tracking down bugs. Is
it that you need to compile in a version of std.datetime which doesn't
have any unit tests compiled in but you still need to compile with
-unittest for other stuff?
No. All you know there's a bug that's being triggered somewhere in
Phobos (with -unittest). It's probably not in std.datetime.
But Phobos is a horrible ball of mud where everything imports everything
else, and std.datetime is near the centre of that ball. What you have to
do is reduce the amount of code, and especially the number of modules,
as rapidly as possible; this means getting rid of imports.
To do this, you need to remove large chunks of code from the files. This
is pretty simple; comment out half of the file, if it still works, then
delete it. Normally this works well because typically only about a dozen
lines are actually being used. After doing this about three or four
times it's small enough that you can usually get rid of most of the
imports. Unittests foul this up because they use functions/classes from
inside the file.
In the case of std.datetime it's even worse because the signal-to-noise
ratio is so incredibly poor; it's really difficult to find the few lines
of code that are actually being used by other Phobos modules.
My experience (obviously only over the last month or so) has been that
if the reduction of a bug is non-obvious, more than 10% of the total
time taken to fix that bug is the time taken to cut down std.datetime.
Hmmm. I really don't know what could be done to fix that (other than making it
easier to rip out the unittest blocks). And enough of std.datetime depends on
other parts of std.datetime that trimming it down isn't (and can't be) exactly
easy. In general, SysTime is the most likely type to be used, and it depends
on Date, TimeOfDay, and DateTime, and all 4 of those depend on most of the
free functions in the module. It's not exactly designed in a manner which
allows you to cut out large chunks and still have it compile. And I don't
think that it _could_ be designed that way and still have the functionality
that it has.
The problem is purely the large fraction of the module which is devoted
to unit tests. That's all.
I guess that this sort of problem is one that would pop up mainly when dealing
with compiler bugs. I have a hard time seeing it popping up with your typical
bug in Phobos itself. So, I guess that this is the sort of thing that you'd
run into and I likely wouldn't.
Yes.
I really don't know how the situation could be improved though other than
making it easier to cut out the unit tests.
- Jonathan M Davis
Hence the motivation for this utility. The problem exists in all
modules, but in std.datetime it's such an obvious time-waster that I
can't keep ignoring it.
Re: Want to help DMD bugfixing? Write a simple utility.
> Jonathan M Davis wrote:
> >> Jonathan M Davis wrote:
> >>> On Saturday 19 March 2011 18:04:57 Don wrote:
> Jonathan M Davis wrote:
> > On Saturday 19 March 2011 17:11:56 Don wrote:
> >> Here's the task:
> >> Given a .d source file, strip out all of the unittest {} blocks,
> >> including everything inside them.
> >> Strip out all comments as well.
> >> Print out the resulting file.
> >>
> >> Motivation: Bug reports frequently come with very large test cases.
> >> Even ones which look small often import from Phobos.
> >> Reducing the test case is the first step in fixing the bug, and it's
> >> frequently ~30% of the total time required. Stripping out the unit
> >> tests is the most time-consuming and error-prone part of reducing
> >> the test case.
> >>
> >> This should be a good task if you're relatively new to D but would
> >> like to do something really useful.
> >
> > Unfortunately, to do that 100% correctly, you need to actually have a
> > working D lexer (and possibly parser). You might be able to get
> > something close enough to work in most cases, but it doesn't take all
> > that much to throw off a basic implementation of this sort of thing
> > if you don't lex/parse it with something which properly understands
> > D.
> >
> > - Jonathan M Davis
>
> I didn't say it needs 100% accuracy. You can assume, for example, that
> "unittest" always occurs at the start of a line. The only other things
> you need to lex are {}, string literals, and comments.
>
> BTW, the immediate motivation for this is std.datetime in Phobos. The
> sheer number of unittests in there is an absolute catastrophe for
> tracking down bugs. It makes a tool like this MANDATORY.
> >>>
> >>> I tried to create a similar tool before and gave up because I couldn't
> >>> make it 100% accurate and was running into problems with it. If someone
> >>> wants to take a shot at it though, that's fine.
> >>>
> >>> As for the unit tests in std.datetime making it hard to track down
> >>> bugs, that only makes sense to me if you're trying to look at the
> >>> whole thing at once and track down a compiler bug which happens
> >>> _somewhere_ in the code, but you don't know where. Other than a
> >>> problem like that, I don't really see how the unit tests get in the
> >>> way of tracking down bugs. Is it that you need to compile in a version
> >>> of std.datetime which doesn't have any unit tests compiled in but you
> >>> still need to compile with -unittest for other stuff?
> >>
> >> No. All you know there's a bug that's being triggered somewhere in
> >> Phobos (with -unittest). It's probably not in std.datetime.
> >> But Phobos is a horrible ball of mud where everything imports everything
> >> else, and std.datetime is near the centre of that ball. What you have to
> >> do is reduce the amount of code, and especially the number of modules,
> >> as rapidly as possible; this means getting rid of imports.
> >>
> >> To do this, you need to remove large chunks of code from the files. This
> >> is pretty simple; comment out half of the file, if it still works, then
> >> delete it. Normally this works well because typically only about a dozen
> >> lines are actually being used. After doing this about three or four
> >> times it's small enough that you can usually get rid of most of the
> >> imports. Unittests foul this up because they use functions/classes from
> >> inside the file.
> >>
> >> In the case of std.datetime it's even worse because the signal-to-noise
> >> ratio is so incredibly poor; it's really difficult to find the few lines
> >> of code that are actually being used by other Phobos modules.
> >>
> >> My experience (obviously only over the last month or so) has been that
> >> if the reduction of a bug is non-obvious, more than 10% of the total
> >> time taken to fix that bug is the time taken to cut down std.datetime.
> >
> > Hmmm. I really don't know what could be done to fix that (other than
> > making it easier to rip out the unittest blocks). And enough of
> > std.datetime depends on other parts of std.datetime that trimming it
> > down isn't (and can't be) exactly easy. In general, SysTime is the most
> > likely type to be used, and it depends on Date, TimeOfDay, and DateTime,
> > and all 4 of those depend on most of the free functions in the module.
> > It's not exactly designed in a manner which allows you to cut out large
> > chunks and still have it compile. And I don't think that it _could_ be
> > designed that way and still have the functionality that it has.
>
> The problem is purely the large fraction of the module which is devoted
> to unit tests. That's all.
>
> > I guess that this sort of problem is one that would pop up mainly when
> > dealing with compiler bugs. I have a hard time seeing it popping up with
> > your typical bug in Phobos itself. So, I guess that this i
Re: Want to help DMD bugfixing? Write a simple utility.
Not very elegant but this should get the job done:
000 module strip;
001 import std.algoritm : countUntil;
002 import std.array: strip;
003 import std.file : read;
004 import std.string : splitlines;
005 import std.stdio: writeln;
006
007 void main(string[] args)
008 {
009 bool start = false;
010 bool end = false;
011
012 bool comment = false;
013 bool nested = false;
014
015 int lBrace, rBrace;
016
017 auto f = cast(char[]) read(args[1]);
018 auto file = splitlines(f);
019
020 foreach(ref ndx, line; file)
021 {
022 if(countUntil(strip(line), "//") == 0)
023 {
024 continue;
025 }
026
027 if(!comment && countUntil(line, "/+", != -1)
028 {
029 nested = true;
030
031 if(countUntil(line, "+/") != -1)
032 nested = false;
033
034 continue;
035 }
036
037 while(nested)
038 {
039 if(countUntil(file[ndx], "+/") != -1)
040 {
041 nested = false;
042 end = true;
043 goto endTest;
044 }
045 ndx++;
046 }
047
048 if(!nested && countUntil(line, "/*", != -1)
049 {
050 comment = true;
051
052 if(countUntil(line, "*/") != -1)
053 comment = false;
054
055 continue;
056 }
057
058 while(comment)
059 {
060 if(countUntil(file[ndx], "*/") != -1)
061 {
062 comment = false;
063 end = true;
064 goto endTest;
065 }
066 ndx++;
067 }
068
069 if(!end && countUntil(line, "unittest") != -1)
070 {
071 start = true;
072 }
073
074 if(!nested && start)
075 {
076 if(countUntil(line, "{") != -1)
077 {
078 lBrace++;
079 }
080
081 if(countUntil(line, "}") != -1)
082 {
083 rBrace++;
084 if(rBrace > 0 && lBrace == rBrace)
085 {
086 end = true;
087 lBrace = rBrace = 0;
088 }
089 }
090 }
091
092 if(!start)
093 writeln(line);
094
095 endTest:;
096 if(end)
097 {
098 start = false;
099 end = false;
100 }
101 }
102 }
cheers.
Re: Want to help DMD bugfixing? Write a simple utility.
The following patch addresses the following issues:
1) fixed improper handling of nested and multiline comments that
do not take up a complete line.
2) eliminate extra blank lines where unit tests and comments are
removed.
Replace lines 31 & 32 with:
# auto n = countUntil(line, "+/");
# if(n != -1 && n < line.lenght - 2)
# {
# nested = false;
# goto output;
# }
Replace lines 52 & 53 with:
# auto n = countUntil(line, "*/");
# if(n != -1 && n < line.lenght - 2)
# {
# comment = false;
# goto output;
# }
Replace lines 92 - 100 with;
# output:;
# if(!start)
# {
# if(line.length == 0 && !blankLine)
# {
# blankLine = true;
# writeln(line);
# }
# else if(line.length == 0 && blankLine)
# {
# goto endTest;
# }
#
# endTest:;
# if(end)
# {
# start = false;
# end = false;
# blankLine = true;
# }
# }
Re: Want to help DMD bugfixing? Write a simple utility.
Messed that up again: see embeded change. Wish I could just copy
and pase but that's not possible with my current setup.
== Quote from Tyro[a.c.edwards] (nos...@home.com)'s article
> The following patch addresses the following issues:
> 1) fixed improper handling of nested and multiline comments that
> do not take up a complete line.
> 2) eliminate extra blank lines where unit tests and comments are
> removed.
> Replace lines 31 & 32 with:
> # auto n = countUntil(line, "+/");
> # if(n != -1 && n < line.lenght - 2)
> # {
> # nested = false;
> # goto output;
> # }
> Replace lines 52 & 53 with:
> # auto n = countUntil(line, "*/");
> # if(n != -1 && n < line.lenght - 2)
> # {
> # comment = false;
> # goto output;
> # }
> Replace lines 92 - 100 with;
> # output:;
> # if(!start)
> # {
> # if(line.length == 0 && !blankLine)
> # {
> # blankLine = true;
> # writeln(line);
> # }
> # else if(line.length == 0 && blankLine)
> # {
> # goto endTest;
> # }
#else
#{
#writeln(line);
#}
> #
> # endTest:;
> # if(end)
> # {
> # start = false;
> # end = false;
> # blankLine = true;
> # }
> # }
Re: Want to help DMD bugfixing? Write a simple utility.
On 03/19/2011 06:11 PM, Don wrote:
Here's the task:
Given a .d source file, strip out all of the unittest {} blocks,
including everything inside them.
Strip out all comments as well.
Print out the resulting file.
Motivation: Bug reports frequently come with very large test cases.
Even ones which look small often import from Phobos.
Reducing the test case is the first step in fixing the bug, and it's
frequently ~30% of the total time required. Stripping out the unit tests
is the most time-consuming and error-prone part of reducing the test case.
This should be a good task if you're relatively new to D but would like
to do something really useful.
-Don
Is there a copy of the official D grammar somewhere online? I wrote a
lexer for my Compiler class and would love to try and apply it to
another grammar.
-Kai Meyer
Re: Want to help DMD bugfixing? Write a simple utility.
On 20/03/2011 19:55, Kai Meyer wrote:
On 03/19/2011 06:11 PM, Don wrote:
Here's the task:
Given a .d source file, strip out all of the unittest {} blocks,
including everything inside them.
Strip out all comments as well.
Print out the resulting file.
Motivation: Bug reports frequently come with very large test cases.
Even ones which look small often import from Phobos.
Reducing the test case is the first step in fixing the bug, and it's
frequently ~30% of the total time required. Stripping out the unit tests
is the most time-consuming and error-prone part of reducing the test
case.
This should be a good task if you're relatively new to D but would like
to do something really useful.
-Don
Is there a copy of the official D grammar somewhere online? I wrote a
lexer for my Compiler class and would love to try and apply it to
another grammar.
-Kai Meyer
As far as I know the documentation /is/ the official grammar
http://digitalmars.com/d/2.0/lex.html
Re: Want to help DMD bugfixing? Write a simple utility.
On 3/19/11 9:11 PM, Don wrote:
Here's the task:
Given a .d source file, strip out all of the unittest {} blocks,
including everything inside them.
Strip out all comments as well.
Print out the resulting file.
Motivation: Bug reports frequently come with very large test cases.
Even ones which look small often import from Phobos.
Reducing the test case is the first step in fixing the bug, and it's
frequently ~30% of the total time required. Stripping out the unit tests
is the most time-consuming and error-prone part of reducing the test case.
This should be a good task if you're relatively new to D but would like
to do something really useful.
-Don
Can it be done in Ruby? Or you need it in D?
Re: Want to help DMD bugfixing? Write a simple utility.
On Mon, 21 Mar 2011 01:52:45 +0100, Ary Manzana
wrote:
On 3/19/11 9:11 PM, Don wrote:
Here's the task:
Given a .d source file, strip out all of the unittest {} blocks,
including everything inside them.
Strip out all comments as well.
Print out the resulting file.
Motivation: Bug reports frequently come with very large test cases.
Even ones which look small often import from Phobos.
Reducing the test case is the first step in fixing the bug, and it's
frequently ~30% of the total time required. Stripping out the unit tests
is the most time-consuming and error-prone part of reducing the test
case.
This should be a good task if you're relatively new to D but would like
to do something really useful.
-Don
Can it be done in Ruby? Or you need it in D?
Part of the idea was that someone use it to learn D. However, the important
part is that it's done. Doing it in D would be preferable, but not a
requisite.
--
Simen
Re: Want to help DMD bugfixing? Write a simple utility.
On Sun, 20 Mar 2011 07:50:10 -, Jonathan M Davis
wrote:
Jonathan M Davis wrote:
> On Saturday 19 March 2011 18:04:57 Don wrote:
>> Jonathan M Davis wrote:
>>> On Saturday 19 March 2011 17:11:56 Don wrote:
Here's the task:
Given a .d source file, strip out all of the unittest {} blocks,
including everything inside them.
Strip out all comments as well.
Print out the resulting file.
Motivation: Bug reports frequently come with very large test cases.
Even ones which look small often import from Phobos.
Reducing the test case is the first step in fixing the bug, and
it's
frequently ~30% of the total time required. Stripping out the unit
tests is the most time-consuming and error-prone part of reducing
the
test case.
This should be a good task if you're relatively new to D but would
like to do something really useful.
>>>
>>> Unfortunately, to do that 100% correctly, you need to actually have
a
>>> working D lexer (and possibly parser). You might be able to get
>>> something close enough to work in most cases, but it doesn't take
all
>>> that much to throw off a basic implementation of this sort of thing
if
>>> you don't lex/parse it with something which properly understands D.
>>>
>>> - Jonathan M Davis
>>
>> I didn't say it needs 100% accuracy. You can assume, for example,
that
>> "unittest" always occurs at the start of a line. The only other
things
>> you need to lex are {}, string literals, and comments.
>>
>> BTW, the immediate motivation for this is std.datetime in Phobos. The
>> sheer number of unittests in there is an absolute catastrophe for
>> tracking down bugs. It makes a tool like this MANDATORY.
>
> I tried to create a similar tool before and gave up because I couldn't
> make it 100% accurate and was running into problems with it. If
someone
> wants to take a shot at it though, that's fine.
>
> As for the unit tests in std.datetime making it hard to track down
bugs,
> that only makes sense to me if you're trying to look at the whole
thing
> at once and track down a compiler bug which happens _somewhere_ in the
> code, but you don't know where. Other than a problem like that, I
don't
> really see how the unit tests get in the way of tracking down bugs. Is
> it that you need to compile in a version of std.datetime which doesn't
> have any unit tests compiled in but you still need to compile with
> -unittest for other stuff?
No. All you know there's a bug that's being triggered somewhere in
Phobos (with -unittest). It's probably not in std.datetime.
But Phobos is a horrible ball of mud where everything imports everything
else, and std.datetime is near the centre of that ball. What you have to
do is reduce the amount of code, and especially the number of modules,
as rapidly as possible; this means getting rid of imports.
To do this, you need to remove large chunks of code from the files. This
is pretty simple; comment out half of the file, if it still works, then
delete it. Normally this works well because typically only about a dozen
lines are actually being used. After doing this about three or four
times it's small enough that you can usually get rid of most of the
imports. Unittests foul this up because they use functions/classes from
inside the file.
In the case of std.datetime it's even worse because the signal-to-noise
ratio is so incredibly poor; it's really difficult to find the few lines
of code that are actually being used by other Phobos modules.
My experience (obviously only over the last month or so) has been that
if the reduction of a bug is non-obvious, more than 10% of the total
time taken to fix that bug is the time taken to cut down std.datetime.
Hmmm. I really don't know what could be done to fix that (other than
making it
easier to rip out the unittest blocks). And enough of std.datetime
depends on
other parts of std.datetime that trimming it down isn't (and can't be)
exactly
easy. In general, SysTime is the most likely type to be used, and it
depends
on Date, TimeOfDay, and DateTime, and all 4 of those depend on most of
the
free functions in the module. It's not exactly designed in a manner which
allows you to cut out large chunks and still have it compile. And I don't
think that it _could_ be designed that way and still have the
functionality
that it has.
I guess that this sort of problem is one that would pop up mainly when
dealing
with compiler bugs. I have a hard time seeing it popping up with your
typical
bug in Phobos itself. So, I guess that this is the sort of thing that
you'd
run into and I likely wouldn't.
I really don't know how the situation could be improved though other than
making it easier to cut out the unit tests.
I was just thinking .. if we get a list of the symbols the linker is
including, then write an app to take that list, and strip everything else
out of the source .. would that work. The Q's are how hard is
Re: Want to help DMD bugfixing? Write a simple utility.
> On Sun, 20 Mar 2011 07:50:10 -, Jonathan M Davis
>
> wrote:
> >> Jonathan M Davis wrote:
> >> > On Saturday 19 March 2011 18:04:57 Don wrote:
> >> >> Jonathan M Davis wrote:
> >> >>> On Saturday 19 March 2011 17:11:56 Don wrote:
> >> Here's the task:
> >> Given a .d source file, strip out all of the unittest {} blocks,
> >> including everything inside them.
> >> Strip out all comments as well.
> >> Print out the resulting file.
> >>
> >> Motivation: Bug reports frequently come with very large test cases.
> >> Even ones which look small often import from Phobos.
> >> Reducing the test case is the first step in fixing the bug, and
> >>
> >> it's
> >>
> >> frequently ~30% of the total time required. Stripping out the unit
> >> tests is the most time-consuming and error-prone part of reducing
> >>
> >> the
> >>
> >> test case.
> >>
> >> This should be a good task if you're relatively new to D but would
> >> like to do something really useful.
> >> >>>
> >> >>> Unfortunately, to do that 100% correctly, you need to actually have
> >>
> >> a
> >>
> >> >>> working D lexer (and possibly parser). You might be able to get
> >> >>> something close enough to work in most cases, but it doesn't take
> >>
> >> all
> >>
> >> >>> that much to throw off a basic implementation of this sort of thing
> >>
> >> if
> >>
> >> >>> you don't lex/parse it with something which properly understands D.
> >> >>>
> >> >>> - Jonathan M Davis
> >> >>
> >> >> I didn't say it needs 100% accuracy. You can assume, for example,
> >>
> >> that
> >>
> >> >> "unittest" always occurs at the start of a line. The only other
> >>
> >> things
> >>
> >> >> you need to lex are {}, string literals, and comments.
> >> >>
> >> >> BTW, the immediate motivation for this is std.datetime in Phobos. The
> >> >> sheer number of unittests in there is an absolute catastrophe for
> >> >> tracking down bugs. It makes a tool like this MANDATORY.
> >> >
> >> > I tried to create a similar tool before and gave up because I couldn't
> >> > make it 100% accurate and was running into problems with it. If
> >>
> >> someone
> >>
> >> > wants to take a shot at it though, that's fine.
> >> >
> >> > As for the unit tests in std.datetime making it hard to track down
> >>
> >> bugs,
> >>
> >> > that only makes sense to me if you're trying to look at the whole
> >>
> >> thing
> >>
> >> > at once and track down a compiler bug which happens _somewhere_ in the
> >> > code, but you don't know where. Other than a problem like that, I
> >>
> >> don't
> >>
> >> > really see how the unit tests get in the way of tracking down bugs. Is
> >> > it that you need to compile in a version of std.datetime which doesn't
> >> > have any unit tests compiled in but you still need to compile with
> >> > -unittest for other stuff?
> >>
> >> No. All you know there's a bug that's being triggered somewhere in
> >> Phobos (with -unittest). It's probably not in std.datetime.
> >> But Phobos is a horrible ball of mud where everything imports everything
> >> else, and std.datetime is near the centre of that ball. What you have to
> >> do is reduce the amount of code, and especially the number of modules,
> >> as rapidly as possible; this means getting rid of imports.
> >>
> >> To do this, you need to remove large chunks of code from the files. This
> >> is pretty simple; comment out half of the file, if it still works, then
> >> delete it. Normally this works well because typically only about a dozen
> >> lines are actually being used. After doing this about three or four
> >> times it's small enough that you can usually get rid of most of the
> >> imports. Unittests foul this up because they use functions/classes from
> >> inside the file.
> >>
> >> In the case of std.datetime it's even worse because the signal-to-noise
> >> ratio is so incredibly poor; it's really difficult to find the few lines
> >> of code that are actually being used by other Phobos modules.
> >>
> >> My experience (obviously only over the last month or so) has been that
> >> if the reduction of a bug is non-obvious, more than 10% of the total
> >> time taken to fix that bug is the time taken to cut down std.datetime.
> >
> > Hmmm. I really don't know what could be done to fix that (other than
> > making it
> > easier to rip out the unittest blocks). And enough of std.datetime
> > depends on
> > other parts of std.datetime that trimming it down isn't (and can't be)
> > exactly
> > easy. In general, SysTime is the most likely type to be used, and it
> > depends
> > on Date, TimeOfDay, and DateTime, and all 4 of those depend on most of
> > the
> > free functions in the module. It's not exactly designed in a manner which
> > allows you to cut out large chunks and still have it compile. And I don't
> > think that it _could_ be designed that way and still have the
> > functionality
> > that it has.
> >
> > I guess that this sort
Re: Want to help DMD bugfixing? Write a simple utility.
On Wed, 23 Mar 2011 15:16:46 -, Jonathan M Davis wrote: I was just thinking .. if we get a list of the symbols the linker is including, then write an app to take that list, and strip everything else out of the source .. would that work. The Q's are how hard is it to get the symbols from the linker and then how hard is it to match those to source. IIRC there are functions in phobos to convert to/from symbol names, so if the app had sufficient lexing and parsing capability it could match on those. That would require a full-blown D lexer and parser. - Jonathan M Davis Yeah, I thought as much. I wonder if the new guy "Ilya" who just posted on digitalmars.D would find this interesting.. -- Using Opera's revolutionary email client: http://www.opera.com/mail/
Re: Want to help DMD bugfixing? Write a simple utility.
On 3/23/11, Jonathan M Davis wrote: > That would require a full-blown D lexer and parser. > > - Jonathan M Davis > Isn't DDMD written in D? I'm not sure about how finished it is though.
Re: Want to help DMD bugfixing? Write a simple utility.
On 03/23/2011 09:16 AM, Jonathan M Davis wrote:
On Sun, 20 Mar 2011 07:50:10 -, Jonathan M Davis
wrote:
Jonathan M Davis wrote:
On Saturday 19 March 2011 18:04:57 Don wrote:
Jonathan M Davis wrote:
On Saturday 19 March 2011 17:11:56 Don wrote:
Here's the task:
Given a .d source file, strip out all of the unittest {} blocks,
including everything inside them.
Strip out all comments as well.
Print out the resulting file.
Motivation: Bug reports frequently come with very large test cases.
Even ones which look small often import from Phobos.
Reducing the test case is the first step in fixing the bug, and
it's
frequently ~30% of the total time required. Stripping out the unit
tests is the most time-consuming and error-prone part of reducing
the
test case.
This should be a good task if you're relatively new to D but would
like to do something really useful.
Unfortunately, to do that 100% correctly, you need to actually have
a
working D lexer (and possibly parser). You might be able to get
something close enough to work in most cases, but it doesn't take
all
that much to throw off a basic implementation of this sort of thing
if
you don't lex/parse it with something which properly understands D.
- Jonathan M Davis
I didn't say it needs 100% accuracy. You can assume, for example,
that
"unittest" always occurs at the start of a line. The only other
things
you need to lex are {}, string literals, and comments.
BTW, the immediate motivation for this is std.datetime in Phobos. The
sheer number of unittests in there is an absolute catastrophe for
tracking down bugs. It makes a tool like this MANDATORY.
I tried to create a similar tool before and gave up because I couldn't
make it 100% accurate and was running into problems with it. If
someone
wants to take a shot at it though, that's fine.
As for the unit tests in std.datetime making it hard to track down
bugs,
that only makes sense to me if you're trying to look at the whole
thing
at once and track down a compiler bug which happens _somewhere_ in the
code, but you don't know where. Other than a problem like that, I
don't
really see how the unit tests get in the way of tracking down bugs. Is
it that you need to compile in a version of std.datetime which doesn't
have any unit tests compiled in but you still need to compile with
-unittest for other stuff?
No. All you know there's a bug that's being triggered somewhere in
Phobos (with -unittest). It's probably not in std.datetime.
But Phobos is a horrible ball of mud where everything imports everything
else, and std.datetime is near the centre of that ball. What you have to
do is reduce the amount of code, and especially the number of modules,
as rapidly as possible; this means getting rid of imports.
To do this, you need to remove large chunks of code from the files. This
is pretty simple; comment out half of the file, if it still works, then
delete it. Normally this works well because typically only about a dozen
lines are actually being used. After doing this about three or four
times it's small enough that you can usually get rid of most of the
imports. Unittests foul this up because they use functions/classes from
inside the file.
In the case of std.datetime it's even worse because the signal-to-noise
ratio is so incredibly poor; it's really difficult to find the few lines
of code that are actually being used by other Phobos modules.
My experience (obviously only over the last month or so) has been that
if the reduction of a bug is non-obvious, more than 10% of the total
time taken to fix that bug is the time taken to cut down std.datetime.
Hmmm. I really don't know what could be done to fix that (other than
making it
easier to rip out the unittest blocks). And enough of std.datetime
depends on
other parts of std.datetime that trimming it down isn't (and can't be)
exactly
easy. In general, SysTime is the most likely type to be used, and it
depends
on Date, TimeOfDay, and DateTime, and all 4 of those depend on most of
the
free functions in the module. It's not exactly designed in a manner which
allows you to cut out large chunks and still have it compile. And I don't
think that it _could_ be designed that way and still have the
functionality
that it has.
I guess that this sort of problem is one that would pop up mainly when
dealing
with compiler bugs. I have a hard time seeing it popping up with your
typical
bug in Phobos itself. So, I guess that this is the sort of thing that
you'd
run into and I likely wouldn't.
I really don't know how the situation could be improved though other than
making it easier to cut out the unit tests.
I was just thinking .. if we get a list of the symbols the linker is
including, then write an app to take that list, and strip everything else
out of the source .. would that work. The Q's are how hard is it to get
the symbols from the linker and then how hard is it to match those to
source. IIRC there are functions
Re: Want to help DMD bugfixing? Write a simple utility.
> On 03/23/2011 09:16 AM, Jonathan M Davis wrote:
> >> On Sun, 20 Mar 2011 07:50:10 -, Jonathan M
> >> Davis
> >>
> >> wrote:
> Jonathan M Davis wrote:
> > On Saturday 19 March 2011 18:04:57 Don wrote:
> >> Jonathan M Davis wrote:
> >>> On Saturday 19 March 2011 17:11:56 Don wrote:
> Here's the task:
> Given a .d source file, strip out all of the unittest {} blocks,
> including everything inside them.
> Strip out all comments as well.
> Print out the resulting file.
>
> Motivation: Bug reports frequently come with very large test
> cases. Even ones which look small often import from Phobos.
> Reducing the test case is the first step in fixing the bug, and
>
> it's
>
> frequently ~30% of the total time required. Stripping out the unit
> tests is the most time-consuming and error-prone part of reducing
>
> the
>
> test case.
>
> This should be a good task if you're relatively new to D but would
> like to do something really useful.
> >>>
> >>> Unfortunately, to do that 100% correctly, you need to actually have
>
> a
>
> >>> working D lexer (and possibly parser). You might be able to get
> >>> something close enough to work in most cases, but it doesn't take
>
> all
>
> >>> that much to throw off a basic implementation of this sort of thing
>
> if
>
> >>> you don't lex/parse it with something which properly understands D.
> >>>
> >>> - Jonathan M Davis
> >>
> >> I didn't say it needs 100% accuracy. You can assume, for example,
>
> that
>
> >> "unittest" always occurs at the start of a line. The only other
>
> things
>
> >> you need to lex are {}, string literals, and comments.
> >>
> >> BTW, the immediate motivation for this is std.datetime in Phobos.
> >> The sheer number of unittests in there is an absolute catastrophe
> >> for tracking down bugs. It makes a tool like this MANDATORY.
> >
> > I tried to create a similar tool before and gave up because I
> > couldn't make it 100% accurate and was running into problems with
> > it. If
>
> someone
>
> > wants to take a shot at it though, that's fine.
> >
> > As for the unit tests in std.datetime making it hard to track down
>
> bugs,
>
> > that only makes sense to me if you're trying to look at the whole
>
> thing
>
> > at once and track down a compiler bug which happens _somewhere_ in
> > the code, but you don't know where. Other than a problem like that,
> > I
>
> don't
>
> > really see how the unit tests get in the way of tracking down bugs.
> > Is it that you need to compile in a version of std.datetime which
> > doesn't have any unit tests compiled in but you still need to
> > compile with -unittest for other stuff?
>
> No. All you know there's a bug that's being triggered somewhere in
> Phobos (with -unittest). It's probably not in std.datetime.
> But Phobos is a horrible ball of mud where everything imports
> everything else, and std.datetime is near the centre of that ball.
> What you have to do is reduce the amount of code, and especially the
> number of modules, as rapidly as possible; this means getting rid of
> imports.
>
> To do this, you need to remove large chunks of code from the files.
> This is pretty simple; comment out half of the file, if it still
> works, then delete it. Normally this works well because typically
> only about a dozen lines are actually being used. After doing this
> about three or four times it's small enough that you can usually get
> rid of most of the imports. Unittests foul this up because they use
> functions/classes from inside the file.
>
> In the case of std.datetime it's even worse because the
> signal-to-noise ratio is so incredibly poor; it's really difficult to
> find the few lines of code that are actually being used by other
> Phobos modules.
>
> My experience (obviously only over the last month or so) has been that
> if the reduction of a bug is non-obvious, more than 10% of the total
> time taken to fix that bug is the time taken to cut down std.datetime.
> >>>
> >>> Hmmm. I really don't know what could be done to fix that (other than
> >>> making it
> >>> easier to rip out the unittest blocks). And enough of std.datetime
> >>> depends on
> >>> other parts of std.datetime that trimming it down isn't (and can't be)
> >>> exactly
> >>> easy. In general, SysTime is the most likely type to be used, and it
> >>> depends
> >>> on Date, TimeOfDay, and DateTime, and all 4 of those depend on most of
> >>> the
> >>> free f
Re: Want to help DMD bugfixing? Write a simple utility.
> On 3/23/11, Jonathan M Davis wrote: > > That would require a full-blown D lexer and parser. > > > > - Jonathan M Davis > > Isn't DDMD written in D? I'm not sure about how finished it is though. Yes, but the lexer and parser in ddmd are not only GPL (which would be a problem for some stuff but not others - for something like Don's utility, it wouldn't be a problem), and more importantly, it is tied to the compiler code. It's not designed to be used by an arbitrary program. For that, you would need a lexer and parser which were designed with an API such that an arbitrary D program could use them. For instance, the lexer could produce a range of tokens to be processed, and a program which wants to use the lexer can then process that range. - Jonathan M Davis
Re: Want to help DMD bugfixing? Write a simple utility.
On 3/23/11, Jonathan M Davis wrote: >> On 3/23/11, Jonathan M Davis wrote: >> > That would require a full-blown D lexer and parser. >> > >> > - Jonathan M Davis >> >> Isn't DDMD written in D? I'm not sure about how finished it is though. > > Yes, but the lexer and parser in ddmd are not only GPL (which would be a > problem for some stuff but not others - for something like Don's utility, it > wouldn't be a problem), and more importantly, it is tied to the compiler > code. > It's not designed to be used by an arbitrary program. For that, you would > need > a lexer and parser which were designed with an API such that an arbitrary D > program could use them. For instance, the lexer could produce a range of > tokens to be processed, and a program which wants to use the lexer can then > process that range. > > - Jonathan M Davis > I didn't even know it was GPL. It doesn't come with a license file.
Re: Want to help DMD bugfixing? Write a simple utility.
What about the artistic license, the front-end can be used with that license. Is that less restrictive than GPL?
Re: Want to help DMD bugfixing? Write a simple utility.
> What about the artistic license, the front-end can be used with that > license. Is that less restrictive than GPL? I don't know what the exact licensing situation is. However, as I understand it, the C++ front-end is under the GPL, and therefore because ddmd is based on the C++ front-end, it is also under the GPL. If that's not the case, I don't know what the licensing situation really is. And I don't know what the artistic license says exactly, so I don't know what its restrictions are. - Jonathan M Davis
Re: Want to help DMD bugfixing? Write a simple utility.
> Currently, as far as I know, there are only two lexers and two parsers for > D: the C++ front end which dmd, gdc, and ldc use and the D front end which > ddmd uses and which is based on the C++ front end. Both of those are under > the GPL (which makes them useless for a lot of stuff) and both of them are > tied to compilers. Being able to lex D code and get the list of tokens in > a D program and being able to parse D code and get the resultant abstract > syntax tree would be very useful for a number of programs. There is a third one: http://code.google.com/p/dil/. The main page says that the lexer and the parser are fully implemented for both D1 and D2. But the license is also the GPL.
Re: Want to help DMD bugfixing? Write a simple utility.
On 03/24/2011 08:53 AM, Alexey Prokhin wrote: Currently, as far as I know, there are only two lexers and two parsers for D: the C++ front end which dmd, gdc, and ldc use and the D front end which ddmd uses and which is based on the C++ front end. Both of those are under the GPL (which makes them useless for a lot of stuff) and both of them are tied to compilers. Being able to lex D code and get the list of tokens in a D program and being able to parse D code and get the resultant abstract syntax tree would be very useful for a number of programs. I fully support this. We desperately need it, I guess, working and maintained along language evolution. This is the whole purpose of the GSOC proposal "D tools in D": http://prowiki.org/wiki4d/wiki.cgi?GSOC_2011_Ideas#DtoolsinD Semantic analysis, introduced step by step, would be a huge plus. Denis -- _ vita es estrany spir.wikidot.com
Re: Want to help DMD bugfixing? Write a simple utility.
Is there a copy of the official D grammar somewhere online? I wrote a lexer for my Compiler class and would love to try and apply it to another grammar. The official D grammar is spread among the specification. But I recall that someone compiled a complete grammar for D1 some time ago.
Re: Want to help DMD bugfixing? Write a simple utility.
On Wed, 23 Mar 2011 21:16:02 -, Jonathan M Davis
wrote:
There are tasks for which you need to be able to lex and parse D code.
To 100% correctly remove unit tests would be one such task.
Is that last bit true? You definitely need to be able to lex it, but
instead of actually parsing it you just count { and } and remove
'unittest' plus { plus } plus everything in between right?
--
Using Opera's revolutionary email client: http://www.opera.com/mail/
Re: Want to help DMD bugfixing? Write a simple utility.
On 03/25/2011 12:08 PM, Regan Heath wrote:
On Wed, 23 Mar 2011 21:16:02 -, Jonathan M Davis
wrote:
There are tasks for which you need to be able to lex and parse D code. To
100% correctly remove unit tests would be one such task.
Is that last bit true? You definitely need to be able to lex it, but instead of
actually parsing it you just count { and } and remove 'unittest' plus { plus }
plus everything in between right?
At first sight, you're both wrong: you'd need to count { } levels. Also, I
think true lexing is not really needed: you'd only need to put apart strings
and comments that could hold non-code { & }.
(But these are only very superficial notes.)
Denis
--
_
vita es estrany
spir.wikidot.com
Re: Want to help DMD bugfixing? Write a simple utility.
spir wrote:
On 03/25/2011 12:08 PM, Regan Heath wrote:
On Wed, 23 Mar 2011 21:16:02 -, Jonathan M Davis
wrote:
There are tasks for which you need to be able to lex and parse D
code. To
100% correctly remove unit tests would be one such task.
Is that last bit true? You definitely need to be able to lex it, but
instead of
actually parsing it you just count { and } and remove 'unittest' plus
{ plus }
plus everything in between right?
At first sight, you're both wrong: you'd need to count { } levels. Also,
I think true lexing is not really needed: you'd only need to put apart
strings and comments that could hold non-code { & }.
(But these are only very superficial notes.)
Denis
Yes, exactly: you just need to lex strings (including q{}), comments
(which you remove),
unittest, and count levels of {.
You need to worry about backslashes in comments, but that's about it.
I even did this in a CTFE function once, I know it isn't complicated.
Should be possible in < 50 lines of code.
I just didn't want to have to do it myself.
In fact, it would be adequate to replace:
unittest
{
blah...
}
with:
unittest{}
Then you don't need to worry about special cases like:
version(XXX)
unittest
{
...
}
Re: Want to help DMD bugfixing? Write a simple utility.
"Regan Heath" wrote in message
news:op.vswbv8qj54x...@puck.auriga.bhead.co.uk...
> On Wed, 23 Mar 2011 21:16:02 -, Jonathan M Davis
> wrote:
>> There are tasks for which you need to be able to lex and parse D code.
>> To 100% correctly remove unit tests would be one such task.
>
> Is that last bit true? You definitely need to be able to lex it, but
> instead of actually parsing it you just count { and } and remove
> 'unittest' plus { plus } plus everything in between right?
>
No, to do it 100% reliably, you do need lexing/parsing, and also the
semantics stage. Example:
string makeATest(string str)
{
return "unit"~"test { "~str~" }";
}
mixin(makeATest(q{
// Do tests
}));
Re: Want to help DMD bugfixing? Write a simple utility.
"Alexey Prokhin" wrote in message news:mailman.2713.1300954193.4748.digitalmars-d-le...@puremagic.com... >> Currently, as far as I know, there are only two lexers and two parsers >> for >> D: the C++ front end which dmd, gdc, and ldc use and the D front end >> which >> ddmd uses and which is based on the C++ front end. Both of those are >> under >> the GPL (which makes them useless for a lot of stuff) and both of them >> are >> tied to compilers. Being able to lex D code and get the list of tokens in >> a D program and being able to parse D code and get the resultant abstract >> syntax tree would be very useful for a number of programs. > There is a third one: http://code.google.com/p/dil/. The main page says > that > the lexer and the parser are fully implemented for both D1 and D2. But the > license is also the GPL. The nearly-done v0.4 of my Goldie parsing system (zlib/libpng license) comes with a mostly-complete lexing-only grammar for D2. http://www.dsource.org/projects/goldie/browser/trunk/lang/dlex.grm The limitations of it right now: - Doesn't do nested comments. That requires a feature (that's going to be introduced in the related tool GOLD Parsing System v4.2) that I haven't had a chance to add into Goldie just yet. - It's possible there might be some edge-case bugs regarding either the ".." operator and/or float literals. - It's ASCII-only. Goldie supports Unicode, but character set optimization isn't implemented yet, so unicode grammars are technically possible but impractical ATM (this will be the top priority after I get v0.4 released).
Re: Want to help DMD bugfixing? Write a simple utility.
"Andrej Mitrovic" wrote in message news:mailman.2696.1300895928.4748.digitalmars-d-le...@puremagic.com... > On 3/23/11, Jonathan M Davis wrote: >> That would require a full-blown D lexer and parser. >> >> - Jonathan M Davis >> > Isn't DDMD written in D? I'm not sure about how finished it is though. I've done a little bit of playing around with DDMD for a (still only just barely-started) project, and it seems to be fairly well up to the task of building an AST and running semantics. It is still based on a somewhat older version of D2, though, and my understanding is that actually building a real-world program with it is still impractical (though I haven't tried).
Re: Want to help DMD bugfixing? Write a simple utility.
"Nick Sabalausky" wrote in message news:imivp7$2fu$1...@digitalmars.com... > "Alexey Prokhin" wrote in message > news:mailman.2713.1300954193.4748.digitalmars-d-le...@puremagic.com... >>> Currently, as far as I know, there are only two lexers and two parsers >>> for >>> D: the C++ front end which dmd, gdc, and ldc use and the D front end >>> which >>> ddmd uses and which is based on the C++ front end. Both of those are >>> under >>> the GPL (which makes them useless for a lot of stuff) and both of them >>> are >>> tied to compilers. Being able to lex D code and get the list of tokens >>> in >>> a D program and being able to parse D code and get the resultant >>> abstract >>> syntax tree would be very useful for a number of programs. >> There is a third one: http://code.google.com/p/dil/. The main page says >> that >> the lexer and the parser are fully implemented for both D1 and D2. But >> the >> license is also the GPL. > > The nearly-done v0.4 of my Goldie parsing system (zlib/libpng license) > comes with a mostly-complete lexing-only grammar for D2. > > http://www.dsource.org/projects/goldie/browser/trunk/lang/dlex.grm > > The limitations of it right now: > > - Doesn't do nested comments. That requires a feature (that's going to be > introduced in the related tool GOLD Parsing System v4.2) that I haven't > had a chance to add into Goldie just yet. > Note that this probably isn't a big of a problem as it sounds: For one thing, it still recognizes "/+" and "+/" as tokens. It'll just try to lex everything in between too. And when Goldie is used to just lex, you still get the entire source lexed even if it has errors, and the lex-error tokens get included in the resulting token array. So it would be pretty easy to just call Goldie's lex function, and then step through the token array removing balanced /+ and +/ sections manually.
Re: Want to help DMD bugfixing? Write a simple utility.
"Jonathan M Davis" wrote in message news:mailman.2700.1300915109.4748.digitalmars-d-le...@puremagic.com... >> On 3/23/11, Jonathan M Davis wrote: >> > That would require a full-blown D lexer and parser. >> > >> > - Jonathan M Davis >> >> Isn't DDMD written in D? I'm not sure about how finished it is though. > > Yes, but the lexer and parser in ddmd are not only GPL (which would be a > problem for some stuff but not others - for something like Don's utility, > it > wouldn't be a problem), and more importantly, it is tied to the compiler > code. > It's not designed to be used by an arbitrary program. For that, you would > need > a lexer and parser which were designed with an API such that an arbitrary > D > program could use them. For instance, the lexer could produce a range of > tokens to be processed, and a program which wants to use the lexer can > then > process that range. > I don't know about the license issues, but I don't think the API is a big deal. I'm in the early stages of a DDMD-based project to compile D code down to Haxe, and all I really had to do was comment out the backend-related section at the end of main(), inject my AST-walking/processing functions into the AST classes (though, admittedly, there is 1.5 metric fuckton of these AST classes), and then add a little bit of code at the end of main() to launch my AST-traversal. The main() function could easily be converted to a non-main one. The only real difficultly is the fact that the AST isn't really documented, except for what little exists on one particular Wiki4D page (sorry, don't have the link ATM). Hmm, although, depending what you're doing with it, you may also want to hook DDMD's stdout/stderr output, or at least the error/warning functions.
