Re: [classlib] Preprocessor - CHECKPOINT

[Geir Magnusson Jr.]

Tim Ellison wrote:

Etienne Gagnon wrote:

I'm proposing to simply use "precise svn url/version identification" as
markup:



As long as we are *sure* that the releasetarget code was built from
precisely that url/version (without any "local workspace" modification)
then it should be as precise as having markups everywhere in the source
code. :-)


Ah! penny has dropped! -- cool, yes.  The immutability of the SVN
revision means that you don't need to reverse the releasetarget, you can
go back directly to the [canonical|master|stored] form and see exactly
where the releasetarget lines came from.  I like it.


Of course, this adds a (small?) dependence on Subversion, but the gains
in markup noise reduction seems to be worth the trouble.


Yep -- it keeps the release code squeaky clean, and it's not like we
don't depend on Subversion being there already for fixing bugs reported
in released code.


Right - we already have similar information in patches, although we 
don't rely on it.


Index: index.xml
===
--- index.xml   (revision 470768)
+++ index.xml   (working copy)




Regards,
Tim

Re: [classlib] Preprocessor - CHECKPOINT

[Tim Ellison]

Etienne Gagnon wrote:
> I'm proposing to simply use "precise svn url/version identification" as
> markup:

> As long as we are *sure* that the releasetarget code was built from
> precisely that url/version (without any "local workspace" modification)
> then it should be as precise as having markups everywhere in the source
> code. :-)

Ah! penny has dropped! -- cool, yes.  The immutability of the SVN
revision means that you don't need to reverse the releasetarget, you can
go back directly to the [canonical|master|stored] form and see exactly
where the releasetarget lines came from.  I like it.

> Of course, this adds a (small?) dependence on Subversion, but the gains
> in markup noise reduction seems to be worth the trouble.

Yep -- it keeps the release code squeaky clean, and it's not like we
don't depend on Subversion being there already for fixing bugs reported
in released code.

Regards,
Tim

-- 

Tim Ellison ([EMAIL PROTECTED])
IBM Java technology centre, UK.

Re: [classlib] Preprocessor - CHECKPOINT

[Etienne Gagnon]

Tim Ellison wrote:
>>>Agreed.  It would be interesting to determine the most effective
>>>location for those markers (measured by reverse mapping accuracy vs.
>>>number of markers).
>>
>>I am a fan of accuracy...  Yet, maybe it would be simple enough if a
>>release was always based on a specific svn URL, then the mark could be
>>totally exact using svn keywords... :-)
> 
> 
> You lost me here.  I'm trying to define tie-points between the
> 'releasetarget' source and the canonical form.  AIUI this will require
> structured comments in the 'releasetarget', right?

Heh!  I'm proposing another "unusual" markup that would be least intrusive.

I'm proposing to simply use "precise svn url/version identification" as
markup:


file:XXX.java
-
/*
 * @ProcessIdentification(j2se,release)
 * @ProcessURL(https://svn.apache.org/repos/harmony/somepath/XXX.java)
 * @ProcessRevision(144)
 *
 * This tells us that this file is the result of:
 * process(svn cat -r revision url, j2serelease)
 */

/* releasetarget code, WITHOUT any additional markup:-) */
...
...


As long as we are *sure* that the releasetarget code was built from
precisely that url/version (without any "local workspace" modification)
then it should be as precise as having markups everywhere in the source
code. :-)

Of course, this adds a (small?) dependence on Subversion, but the gains
in markup noise reduction seems to be worth the trouble.

Etienne

-- 
Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/
SableVM:   http://www.sablevm.org/
SableCC:   http://www.sablecc.org/


signature.asc
Description: OpenPGP digital signature

Re: [classlib] Preprocessor - CHECKPOINT

[Geir Magnusson Jr.]

So this has been a long and exciting thread, and it's still not clear 
that we understand each other.  There are at least two different models 
here.


I'll go re-read and try and clarify the difference, and I suppose the 
next step for Etienne would be an example of how his approach works.  It 
will be fun to play with.


geir

Geir Magnusson Jr. wrote:



Tim Ellison wrote:

Geir Magnusson Jr. wrote:

Etienne Gagnon wrote:

Geir Magnusson Jr. wrote:

The "communication" aspect of 2 can be quite helpful when doing
system-wide changes.  Just think about the effect of simply doing a
system-wide reindentation of source code; this is a nightmare when
developing using branches, as diff/merge tools are line-based, not
syntax-based.  So, if your new indenter moves things across lines,
you're in hell.


So here's where I'm starting to believe I'm missing something
important :

Can you give me an example of the "communication" that you imagine
taking place here?  That's the part I'm just not grokking here.

How about: j2se5 developers, seeing that his planned modification to
some code will lead to utterly broken nearby j2se6 (commented) code
fragments, steps on this mailing list and asks j2se6 developers to help
him find the most harmless way to do the change (or, at least,warn them
about the problem)?

Ok - we do that now.  I thought you were saying that your tool added
somehow to communications.


I am not saying that using the tool should replace using branches for
all situations, but there are situations where it would be more
effective not to use branches when the code is mostly identical.  

I want to avoid branches at all costs.

[somewhat off-topic]  I think that we should consider tools/approaches
on their merit.  Sometimes, branches are the right solution...  You
wouldn't want to use syntax-processing instead of branches for managing
sandboxes. ;-)

I'm trying to figure out the merit here.


I believe that there will still be a role for branches, both for
sandboxes to experiment as Etienne said, and for the maintenance streams
of our releases.


Of course we aren't going to ban branches.  The context is branching for 
mainline code.  Lets stay focused...





Clearly there's some confusion here.  My goal is to

1) Avoid branches at all costs so we can share as much code, and 
get as
much benefit for collaboration between different versions, and 
different

platforms, if that happens.

Agreed.


2) make it simple to work in either the 'master' code, or the 'target'
code through tooling, including standard IDE activities like debugging

Agreed.


3) make it easy for users to report bugs based on 'target' code,
including patches and line numbers


Ah!  That's something to add to my requirements.  Fine!  I hadn't
included the "patches" thing into account.  It doesn't break what I've
exposed so far; it simply adds to it.

It has to be the case - we'll do snapshots and distributions of
"src.jar" and when a developer goes into the debugger, they need to see
normal Java SE 5 code.


Agreed -- our src.jar will be 'normal' Java code without multi-target
mark-ups.



That's neat.  I like it.  Yet, we would encourage developers to work 
and

submit patches using devtarget code, instead of releasetarget code.

I don't know this terminology.  I was using "master" being the code in
SVN, and "target", being the code Y, so the map is :

  master == devtarget
  target == releasetarget

right?  Ok.


There are multiple 'devtarget's -- each contains all the source code
marked-up for every target being developed, but each is distinct by
having different targets uncommented.  Since each devtarget contains the
entire source code it is possible to accurately transform from any one
devtarget to any other devtarget.  One of the devtargets is actually
stored in svn, and that is the canonical form of the devtarget code.


So that's 'master'. and everythign else is a transformed master, right?





I want to work in the master code w/ an IDE plugin that lets me think
I'm in target (and lets me flip back to master).  No preprocessing of
the tree is required to develop.

However, end-users - people who take our JDK and work with it, will
report bugs with stacktraces and line numbers that are from
target/releasetarget code.


Screech -- the releasetarget, by definition, does not contain the source
for all possible targets so that it is consumable by end-users.


That is correct.

The only code that contains all code is master.





So what to do with a patch?

I can either

  patch -p0 << reverse(patch.file)


I don't understand how you will ensure there is enough information in
patch.file to make the reverse() function exact?  I proposed that our
releasetarget code contains tie-points where it is out of sync with the
closest devtarget by some number of lines.


I don't understand why you'd bother.  If you know that patch P creates 
Y' from a given Y, and you can always create Y as you know everything to 
do the xform from master sourc

Re: [classlib] Preprocessor - CHECKPOINT

[Geir Magnusson Jr.]

Tim Ellison wrote:

Geir Magnusson Jr. wrote:

Etienne Gagnon wrote:

Geir Magnusson Jr. wrote:

The "communication" aspect of 2 can be quite helpful when doing
system-wide changes.  Just think about the effect of simply doing a
system-wide reindentation of source code; this is a nightmare when
developing using branches, as diff/merge tools are line-based, not
syntax-based.  So, if your new indenter moves things across lines,
you're in hell.


So here's where I'm starting to believe I'm missing something
important :

Can you give me an example of the "communication" that you imagine
taking place here?  That's the part I'm just not grokking here.

How about: j2se5 developers, seeing that his planned modification to
some code will lead to utterly broken nearby j2se6 (commented) code
fragments, steps on this mailing list and asks j2se6 developers to help
him find the most harmless way to do the change (or, at least,warn them
about the problem)?

Ok - we do that now.  I thought you were saying that your tool added
somehow to communications.


I am not saying that using the tool should replace using branches for
all situations, but there are situations where it would be more
effective not to use branches when the code is mostly identical.  

I want to avoid branches at all costs.

[somewhat off-topic]  I think that we should consider tools/approaches
on their merit.  Sometimes, branches are the right solution...  You
wouldn't want to use syntax-processing instead of branches for managing
sandboxes. ;-)

I'm trying to figure out the merit here.


I believe that there will still be a role for branches, both for
sandboxes to experiment as Etienne said, and for the maintenance streams
of our releases.


Of course we aren't going to ban branches.  The context is branching for 
mainline code.  Lets stay focused...





Clearly there's some confusion here.  My goal is to

1) Avoid branches at all costs so we can share as much code, and get as
much benefit for collaboration between different versions, and different
platforms, if that happens.

Agreed.


2) make it simple to work in either the 'master' code, or the 'target'
code through tooling, including standard IDE activities like debugging

Agreed.


3) make it easy for users to report bugs based on 'target' code,
including patches and line numbers


Ah!  That's something to add to my requirements.  Fine!  I hadn't
included the "patches" thing into account.  It doesn't break what I've
exposed so far; it simply adds to it.

It has to be the case - we'll do snapshots and distributions of
"src.jar" and when a developer goes into the debugger, they need to see
normal Java SE 5 code.


Agreed -- our src.jar will be 'normal' Java code without multi-target
mark-ups.




That's neat.  I like it.  Yet, we would encourage developers to work and
submit patches using devtarget code, instead of releasetarget code.

I don't know this terminology.  I was using "master" being the code in
SVN, and "target", being the code Y, so the map is :

  master == devtarget
  target == releasetarget

right?  Ok.


There are multiple 'devtarget's -- each contains all the source code
marked-up for every target being developed, but each is distinct by
having different targets uncommented.  Since each devtarget contains the
entire source code it is possible to accurately transform from any one
devtarget to any other devtarget.  One of the devtargets is actually
stored in svn, and that is the canonical form of the devtarget code.


So that's 'master'. and everythign else is a transformed master, right?





I want to work in the master code w/ an IDE plugin that lets me think
I'm in target (and lets me flip back to master).  No preprocessing of
the tree is required to develop.

However, end-users - people who take our JDK and work with it, will
report bugs with stacktraces and line numbers that are from
target/releasetarget code.


Screech -- the releasetarget, by definition, does not contain the source
for all possible targets so that it is consumable by end-users.


That is correct.

The only code that contains all code is master.





So what to do with a patch?

I can either

  patch -p0 << reverse(patch.file)


I don't understand how you will ensure there is enough information in
patch.file to make the reverse() function exact?  I proposed that our
releasetarget code contains tie-points where it is out of sync with the
closest devtarget by some number of lines.


I don't understand why you'd bother.  If you know that patch P creates 
Y' from a given Y, and you can always create Y as you know everything to 
do the xform from master source and the platform name, everything is known.





on the main code or use patching facilities in an IDE to patch the
transform view

Man, this tooling is going to be fancy! :/



In other words, here's how I see the distribution(s):

 - Binary j2se5 Harmony release:  includes j2se5release src.jar for
   end-developers using debuggers to step through API code

Re: [classlib] Preprocessor - CHECKPOINT

[Tim Ellison]

Geir Magnusson Jr. wrote:
> Etienne Gagnon wrote:
>> Geir Magnusson Jr. wrote:
 The "communication" aspect of 2 can be quite helpful when doing
 system-wide changes.  Just think about the effect of simply doing a
 system-wide reindentation of source code; this is a nightmare when
 developing using branches, as diff/merge tools are line-based, not
 syntax-based.  So, if your new indenter moves things across lines,
 you're in hell.

>>> So here's where I'm starting to believe I'm missing something
>>> important :
>>>
>>> Can you give me an example of the "communication" that you imagine
>>> taking place here?  That's the part I'm just not grokking here.
>>
>> How about: j2se5 developers, seeing that his planned modification to
>> some code will lead to utterly broken nearby j2se6 (commented) code
>> fragments, steps on this mailing list and asks j2se6 developers to help
>> him find the most harmless way to do the change (or, at least,warn them
>> about the problem)?
> 
> Ok - we do that now.  I thought you were saying that your tool added
> somehow to communications.
> 
>>
 I am not saying that using the tool should replace using branches for
 all situations, but there are situations where it would be more
 effective not to use branches when the code is mostly identical.  
>>> I want to avoid branches at all costs.
>>
>> [somewhat off-topic]  I think that we should consider tools/approaches
>> on their merit.  Sometimes, branches are the right solution...  You
>> wouldn't want to use syntax-processing instead of branches for managing
>> sandboxes. ;-)
> 
> I'm trying to figure out the merit here.

I believe that there will still be a role for branches, both for
sandboxes to experiment as Etienne said, and for the maintenance streams
of our releases.

>>> Clearly there's some confusion here.  My goal is to
>>>
>>> 1) Avoid branches at all costs so we can share as much code, and get as
>>> much benefit for collaboration between different versions, and different
>>> platforms, if that happens.
>>
>> Agreed.
>>
>>> 2) make it simple to work in either the 'master' code, or the 'target'
>>> code through tooling, including standard IDE activities like debugging
>>
>> Agreed.
>>
>>> 3) make it easy for users to report bugs based on 'target' code,
>>> including patches and line numbers
>>
>>
>> Ah!  That's something to add to my requirements.  Fine!  I hadn't
>> included the "patches" thing into account.  It doesn't break what I've
>> exposed so far; it simply adds to it.
> 
> It has to be the case - we'll do snapshots and distributions of
> "src.jar" and when a developer goes into the debugger, they need to see
> normal Java SE 5 code.

Agreed -- our src.jar will be 'normal' Java code without multi-target
mark-ups.



>> That's neat.  I like it.  Yet, we would encourage developers to work and
>> submit patches using devtarget code, instead of releasetarget code.
> 
> I don't know this terminology.  I was using "master" being the code in
> SVN, and "target", being the code Y, so the map is :
> 
>   master == devtarget
>   target == releasetarget
> 
> right?  Ok.

There are multiple 'devtarget's -- each contains all the source code
marked-up for every target being developed, but each is distinct by
having different targets uncommented.  Since each devtarget contains the
entire source code it is possible to accurately transform from any one
devtarget to any other devtarget.  One of the devtargets is actually
stored in svn, and that is the canonical form of the devtarget code.

> I want to work in the master code w/ an IDE plugin that lets me think
> I'm in target (and lets me flip back to master).  No preprocessing of
> the tree is required to develop.
> 
> However, end-users - people who take our JDK and work with it, will
> report bugs with stacktraces and line numbers that are from
> target/releasetarget code.

Screech -- the releasetarget, by definition, does not contain the source
for all possible targets so that it is consumable by end-users.

> So what to do with a patch?
> 
> I can either
> 
>   patch -p0 << reverse(patch.file)

I don't understand how you will ensure there is enough information in
patch.file to make the reverse() function exact?  I proposed that our
releasetarget code contains tie-points where it is out of sync with the
closest devtarget by some number of lines.

> on the main code or use patching facilities in an IDE to patch the
> transform view
> 
> Man, this tooling is going to be fancy! :/
> 
> 
>>
>> In other words, here's how I see the distribution(s):
>>
>>  - Binary j2se5 Harmony release:  includes j2se5release src.jar for
>>end-developers using debuggers to step through API code.
>>
>>  - Source j2se5 Harmony release: API are in j2se5dev form.  The build
>>process uses the processing tool to generate j2se5release src.jar.
>>
>>  - Binary j2se6 Harmony release:  includes j2se6release src.jar for
>>end-developers using debuggers to ste

Re: [classlib] Preprocessor - CHECKPOINT

[Tim Ellison]

Etienne Gagnon wrote:
> Geir Magnusson Jr. wrote:
>>> To release code, one would apply:
>>>
>>>  process(X, release-target) => Y
>>>
>>> Now, it is important to understand that Y, in this case, is NOT suitable
>>> for doing any modification as
>>>
>>>  revert(Y) => Kaboom!  (The tool will simply report that it can't do it;
>>> it won't crash.)
>>> ...
>> This is what I thought we were talking about all along - basically
>> starting w/ the full source, and pre-process to the "canonical" source
>> for the target version.
>>
>> However, I don't understand why I can't go backwards, modulo some manual
>> merging if needed.
>>
>> For example, if I have X and release-version, I should be able to take a
>> Y' and resolve back to X'.  There are problematic cases. [...]
> 
> It's the problematic cases I am trying to get entirely rid of "by
> design", by having an "exact" transformation process, in contrast with
> the usual branch/merge "inexact" process.

Yes, this is the key point for me, that the transforms are exactly
reversible so you loose nothing by switching back and forth.

> Of course, this means that development happens using "complete" source
> code (this is either the canonical form or a development target form,
> but not a release form).

Ack, and I agree with the comment made elsewhere that it is useful for
each processed form's developer to see their changes in the context of
all potential processed forms.

> For applying user-provided patches, then it can still be done the good
> old way, using these "little" patches made from release code, but
> applying them directly to the canonical or dev-target code, resolving
> conflicts manually.  [The smaller the patch, the easier it is to apply
> it on files with target-specific code.]

How so?  With a customized patch tool?  Because the end-user's patch
will have line numbers relative to the 'releasetarget' code.

> The idea is that manual processing is never required when coding using
> canonical and dev-target modes.  No inexact modifications, no difficult
> merges, etc.

Hurray.

> And, as a bonus, communication between parrallel target
> developers.  I know, it's an "unusual" development processing tool, but,
> hey, why shouldn't Harmony innovate? :-)

...continue to innovate ;-)

Regards,
Tim

-- 

Tim Ellison ([EMAIL PROTECTED])
IBM Java technology centre, UK.

Re: [classlib] Preprocessor - CHECKPOINT

[Tim Ellison]

Etienne Gagnon wrote:
> Tim Ellison wrote:
>>>   For example, if class X of the main version is not part of j2me,
>>>   "process(j2me)" would move this file to a subdirectory ".streams/".
>> Why would you move the files rather than exclude them?
>>
>> I was assuming that the processor would generate a whole new source tree
>> for each process() target, so that you could work on the original
>> checked-out file in it's 'canonicalized form' for Big Java work, or
>> process("jme") into a new location and work in the non-canonical form
>> your Little Java spectacles on.  [...]
> 
> Neat idea!
> 
> I would put the following restriction, though:  one should NOT modify
> more than one target at a time.

Fair enough -- at least for now.

> So, you would probably need some way to prevent parallel modifications
> in "spectacle views".

Perhaps we should agree to use your 'devtarget' term for this :-)

> One way to achieve this:
> 
> process(X,target, destination) =>  Xtarget in a distinct location
>  and X files are changed to read-only and some tracking file F tells us
>  about Xtarget's location.

Sure, should be simple enough if there is a well-known location for this
lock file.  As you say, then developers would work on a single devtarget
at a time in their local workspace, but potentially go back to the
canonical form and switch to another devtarget to check that before
checking in the canonical form.

> So, you would also need:
> 
> release(F)  =>  asks all kind of questions (want to lose changes? delete
> files? etc.)
> 
> Of course:
> 
> revert(F?|destination?) => makes X read-write.
> 
> 
> [Hoping this was clear enough...  It's not a very good explanation...]

Yep, I got it; and it seems quite simple with these restrictions.

>> Agreed.  It would be interesting to determine the most effective
>> location for those markers (measured by reverse mapping accuracy vs.
>> number of markers).
> 
> I am a fan of accuracy...  Yet, maybe it would be simple enough if a
> release was always based on a specific svn URL, then the mark could be
> totally exact using svn keywords... :-)

You lost me here.  I'm trying to define tie-points between the
'releasetarget' source and the canonical form.  AIUI this will require
structured comments in the 'releasetarget', right?

Regards,
Tim

-- 

Tim Ellison ([EMAIL PROTECTED])
IBM Java technology centre, UK.

Re: [classlib] Preprocessor - CHECKPOINT

[Geir Magnusson Jr.]

BTW, I'm more than happy if you just do a quick sketch of what you're 
thinking, and we resume the convo from there.


I just wanted to clear up some confusion I had.

geir


Geir Magnusson Jr. wrote:



Etienne Gagnon wrote:

Geir Magnusson Jr. wrote:

The "communication" aspect of 2 can be quite helpful when doing
system-wide changes.  Just think about the effect of simply doing a
system-wide reindentation of source code; this is a nightmare when
developing using branches, as diff/merge tools are line-based, not
syntax-based.  So, if your new indenter moves things across lines,
you're in hell.

So here's where I'm starting to believe I'm missing something 
important :


Can you give me an example of the "communication" that you imagine
taking place here?  That's the part I'm just not grokking here.


How about: j2se5 developers, seeing that his planned modification to
some code will lead to utterly broken nearby j2se6 (commented) code
fragments, steps on this mailing list and asks j2se6 developers to help
him find the most harmless way to do the change (or, at least,warn them
about the problem)?


Ok - we do that now.  I thought you were saying that your tool added 
somehow to communications.





I am not saying that using the tool should replace using branches for
all situations, but there are situations where it would be more
effective not to use branches when the code is mostly identical.  

I want to avoid branches at all costs.


[somewhat off-topic]  I think that we should consider tools/approaches
on their merit.  Sometimes, branches are the right solution...  You
wouldn't want to use syntax-processing instead of branches for managing
sandboxes. ;-)


I'm trying to figure out the merit here.




Clearly there's some confusion here.  My goal is to

1) Avoid branches at all costs so we can share as much code, and get as
much benefit for collaboration between different versions, and different
platforms, if that happens.


Agreed.


2) make it simple to work in either the 'master' code, or the 'target'
code through tooling, including standard IDE activities like debugging


Agreed.


3) make it easy for users to report bugs based on 'target' code,
including patches and line numbers



Ah!  That's something to add to my requirements.  Fine!  I hadn't
included the "patches" thing into account.  It doesn't break what I've
exposed so far; it simply adds to it.


It has to be the case - we'll do snapshots and distributions of 
"src.jar" and when a developer goes into the debugger, they need to see 
normal Java SE 5 code.





What I've suggested is  :

  forward(X, platform) -> Y

  reverse(X, platform, Y') -> X'


OK.  I see this in *addition* to:

 forward(X, devtarget) -> Y
 reverse(Y') -> X'


I believe that

  reverse(X, palatform, Y') -> X'

is the same as

  reverse(Y') -> X'

as you need to know at least what platform Y' is, and what it came from. 
 You could always embed as metadata in the code itself in a comment, I 
suppose.  I was just being explicit.





So, to simplify the discussion, let's rewrite your proposal as:

 forward(X, releasetarget) -> Y
 reverse(Y',X,releasetarget) ~~> X' (possibly reporting
 conflicts/problems)

That's neat.  I like it.  Yet, we would encourage developers to work and
submit patches using devtarget code, instead of releasetarget code.


I don't know this terminology.  I was using "master" being the code in 
SVN, and "target", being the code Y, so the map is :


  master == devtarget
  target == releasetarget

right?  Ok.

I want to work in the master code w/ an IDE plugin that lets me think 
I'm in target (and lets me flip back to master).  No preprocessing of 
the tree is required to develop.


However, end-users - people who take our JDK and work with it, will 
report bugs with stacktraces and line numbers that are from 
target/releasetarget code.


So what to do with a patch?

I can either

  patch -p0 << reverse(patch.file)

on the main code or use patching facilities in an IDE to patch the 
transform view


Man, this tooling is going to be fancy! :/




In other words, here's how I see the distribution(s):

 - Binary j2se5 Harmony release:  includes j2se5release src.jar for
   end-developers using debuggers to step through API code.

 - Source j2se5 Harmony release: API are in j2se5dev form.  The build
   process uses the processing tool to generate j2se5release src.jar.

 - Binary j2se6 Harmony release:  includes j2se6release src.jar for
   end-developers using debuggers to step through API code.

 - Source j2se5 Harmony release: API are in j2se6dev form.  The build
   process uses the processing tool to generate j2se6release src.jar.


Yes.



Why?  Because reverse works much better with devtarget than
releasetarget code, and the communications benefit of devtarget that are
lost in releasetarget code (because of stream code erasure).  In
addition, using smart IDEs, there wouldn't be much difference for
developers betwee

Re: [classlib] Preprocessor - CHECKPOINT

[Geir Magnusson Jr.]

Etienne Gagnon wrote:

Geir Magnusson Jr. wrote:

The "communication" aspect of 2 can be quite helpful when doing
system-wide changes.  Just think about the effect of simply doing a
system-wide reindentation of source code; this is a nightmare when
developing using branches, as diff/merge tools are line-based, not
syntax-based.  So, if your new indenter moves things across lines,
you're in hell.


So here's where I'm starting to believe I'm missing something important :

Can you give me an example of the "communication" that you imagine
taking place here?  That's the part I'm just not grokking here.


How about: j2se5 developers, seeing that his planned modification to
some code will lead to utterly broken nearby j2se6 (commented) code
fragments, steps on this mailing list and asks j2se6 developers to help
him find the most harmless way to do the change (or, at least,warn them
about the problem)?


Ok - we do that now.  I thought you were saying that your tool added 
somehow to communications.





I am not saying that using the tool should replace using branches for
all situations, but there are situations where it would be more
effective not to use branches when the code is mostly identical.  

I want to avoid branches at all costs.


[somewhat off-topic]  I think that we should consider tools/approaches
on their merit.  Sometimes, branches are the right solution...  You
wouldn't want to use syntax-processing instead of branches for managing
sandboxes. ;-)


I'm trying to figure out the merit here.




Clearly there's some confusion here.  My goal is to

1) Avoid branches at all costs so we can share as much code, and get as
much benefit for collaboration between different versions, and different
platforms, if that happens.


Agreed.


2) make it simple to work in either the 'master' code, or the 'target'
code through tooling, including standard IDE activities like debugging


Agreed.


3) make it easy for users to report bugs based on 'target' code,
including patches and line numbers



Ah!  That's something to add to my requirements.  Fine!  I hadn't
included the "patches" thing into account.  It doesn't break what I've
exposed so far; it simply adds to it.


It has to be the case - we'll do snapshots and distributions of 
"src.jar" and when a developer goes into the debugger, they need to see 
normal Java SE 5 code.





What I've suggested is  :

  forward(X, platform) -> Y

  reverse(X, platform, Y') -> X'


OK.  I see this in *addition* to:

 forward(X, devtarget) -> Y
 reverse(Y') -> X'


I believe that

  reverse(X, palatform, Y') -> X'

is the same as

  reverse(Y') -> X'

as you need to know at least what platform Y' is, and what it came from. 
 You could always embed as metadata in the code itself in a comment, I 
suppose.  I was just being explicit.





So, to simplify the discussion, let's rewrite your proposal as:

 forward(X, releasetarget) -> Y
 reverse(Y',X,releasetarget) ~~> X' (possibly reporting
 conflicts/problems)

That's neat.  I like it.  Yet, we would encourage developers to work and
submit patches using devtarget code, instead of releasetarget code.


I don't know this terminology.  I was using "master" being the code in 
SVN, and "target", being the code Y, so the map is :


  master == devtarget
  target == releasetarget

right?  Ok.

I want to work in the master code w/ an IDE plugin that lets me think 
I'm in target (and lets me flip back to master).  No preprocessing of 
the tree is required to develop.


However, end-users - people who take our JDK and work with it, will 
report bugs with stacktraces and line numbers that are from 
target/releasetarget code.


So what to do with a patch?

I can either

  patch -p0 << reverse(patch.file)

on the main code or use patching facilities in an IDE to patch the 
transform view


Man, this tooling is going to be fancy! :/




In other words, here's how I see the distribution(s):

 - Binary j2se5 Harmony release:  includes j2se5release src.jar for
   end-developers using debuggers to step through API code.

 - Source j2se5 Harmony release: API are in j2se5dev form.  The build
   process uses the processing tool to generate j2se5release src.jar.

 - Binary j2se6 Harmony release:  includes j2se6release src.jar for
   end-developers using debuggers to step through API code.

 - Source j2se5 Harmony release: API are in j2se6dev form.  The build
   process uses the processing tool to generate j2se6release src.jar.


Yes.



Why?  Because reverse works much better with devtarget than
releasetarget code, and the communications benefit of devtarget that are
lost in releasetarget code (because of stream code erasure).  In
addition, using smart IDEs, there wouldn't be much difference for
developers between the "visible" formats of dev vs release code when
working on it.


I don't think it matters, as I think that most people interested in 
working on the code will just check out of SVN.





for X being

Re: [classlib] Preprocessor - CHECKPOINT

[Etienne Gagnon]

Geir Magnusson Jr. wrote:
>> The "communication" aspect of 2 can be quite helpful when doing
>> system-wide changes.  Just think about the effect of simply doing a
>> system-wide reindentation of source code; this is a nightmare when
>> developing using branches, as diff/merge tools are line-based, not
>> syntax-based.  So, if your new indenter moves things across lines,
>> you're in hell.
>>
> 
> So here's where I'm starting to believe I'm missing something important :
> 
> Can you give me an example of the "communication" that you imagine
> taking place here?  That's the part I'm just not grokking here.

How about: j2se5 developers, seeing that his planned modification to
some code will lead to utterly broken nearby j2se6 (commented) code
fragments, steps on this mailing list and asks j2se6 developers to help
him find the most harmless way to do the change (or, at least,warn them
about the problem)?

>> I am not saying that using the tool should replace using branches for
>> all situations, but there are situations where it would be more
>> effective not to use branches when the code is mostly identical.  
> 
> I want to avoid branches at all costs.

[somewhat off-topic]  I think that we should consider tools/approaches
on their merit.  Sometimes, branches are the right solution...  You
wouldn't want to use syntax-processing instead of branches for managing
sandboxes. ;-)

> Clearly there's some confusion here.  My goal is to
> 
> 1) Avoid branches at all costs so we can share as much code, and get as
> much benefit for collaboration between different versions, and different
> platforms, if that happens.

Agreed.

> 2) make it simple to work in either the 'master' code, or the 'target'
> code through tooling, including standard IDE activities like debugging

Agreed.

> 
> 3) make it easy for users to report bugs based on 'target' code,
> including patches and line numbers


Ah!  That's something to add to my requirements.  Fine!  I hadn't
included the "patches" thing into account.  It doesn't break what I've
exposed so far; it simply adds to it.

> What I've suggested is  :
> 
>   forward(X, platform) -> Y
> 
>   reverse(X, platform, Y') -> X'

OK.  I see this in *addition* to:

 forward(X, devtarget) -> Y
 reverse(Y') -> X'

So, to simplify the discussion, let's rewrite your proposal as:

 forward(X, releasetarget) -> Y
 reverse(Y',X,releasetarget) ~~> X' (possibly reporting
 conflicts/problems)

That's neat.  I like it.  Yet, we would encourage developers to work and
submit patches using devtarget code, instead of releasetarget code.

In other words, here's how I see the distribution(s):

 - Binary j2se5 Harmony release:  includes j2se5release src.jar for
   end-developers using debuggers to step through API code.

 - Source j2se5 Harmony release: API are in j2se5dev form.  The build
   process uses the processing tool to generate j2se5release src.jar.

 - Binary j2se6 Harmony release:  includes j2se6release src.jar for
   end-developers using debuggers to step through API code.

 - Source j2se5 Harmony release: API are in j2se6dev form.  The build
   process uses the processing tool to generate j2se6release src.jar.

Why?  Because reverse works much better with devtarget than
releasetarget code, and the communications benefit of devtarget that are
lost in releasetarget code (because of stream code erasure).  In
addition, using smart IDEs, there wouldn't be much difference for
developers between the "visible" formats of dev vs release code when
working on it.

> for X being a single class.  To produce a complete target source base,
> walk the single source tree as defined by a 'platform descriptor' :

I don't really understand the "platform descriptor" thing, yet I think
that it is, somehow, an orthogonal issue to the discussion above.
[Remember my long message describing 2 levels of processing:
file/directory-level and source-level.]  My first prototype tool would
attack source-level processing (discussed above), to validate the approach.

Maybe you could try rephrasing your file/directory-level (right?)
proposal?  How does it differ from Tim's proposal?

Etienne

-- 
Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/
SableVM:   http://www.sablevm.org/
SableCC:   http://www.sablecc.org/


signature.asc
Description: OpenPGP digital signature

Re: [classlib] Preprocessor - CHECKPOINT

[Geir Magnusson Jr.]

Etienne Gagnon wrote:

[SNIP]



Such changes are usually not dealt with very neatly with "svn merge".


I'm confused.  I don't think anyone has suggested that we use "svn 
merge" for this.



Now, if using the processign tool, you get at least 2 benefits over "svn
merge":
1- No need to do any "svn merge"...  The method was moved AND renamed
even for j2se6.
2- You were aware, when moving the method and changing its name that you
could possibly be making the life of j2se6 developers a little more
difficult.



Right - that's how I thought it would work.


The "communication" aspect of 2 can be quite helpful when doing
system-wide changes.  Just think about the effect of simply doing a
system-wide reindentation of source code; this is a nightmare when
developing using branches, as diff/merge tools are line-based, not
syntax-based.  So, if your new indenter moves things across lines,
you're in hell.



So here's where I'm starting to believe I'm missing something important :

Can you give me an example of the "communication" that you imagine 
taking place here?  That's the part I'm just not grokking here.



Believe me; I've lived through that trying to maintain a "well indented"
GNU Classpath in SableVM's repository.

I am not saying that using the tool should replace using branches for
all situations, but there are situations where it would be more
effective not to use branches when the code is mostly identical.  


I want to avoid branches at all costs.


Using
traditional pre-processing tools, in such cases, is often the solution
adopted by some people; the problems of "simply-mided" traditiona
pre-processing has been discussed earlier in this thread.  I am
proposing, instead, to use an innovative "syntax-based, revertable
processing".  That's all.  [I guess you'll need to see it working to
make sense of it...  I'll work on a prototype.  I've just submitted the
ICLA/ACQ].


Syntax based is fine - we never settled on what kind of pre-processing 
scheme we would use, so whatever seems best, go for it.


Clearly there's some confusion here.  My goal is to

1) Avoid branches at all costs so we can share as much code, and get as 
much benefit for collaboration between different versions, and different 
platforms, if that happens.


2) make it simple to work in either the 'master' code, or the 'target' 
code through tooling, including standard IDE activities like debugging


3) make it easy for users to report bugs based on 'target' code, 
including patches and line numbers


Now, maybe I'm misunderstanding you, but it sounds like your approach is 
one-way, and coarse grained.


What I've suggested is  :

  forward(X, platform) -> Y

  reverse(X, platform, Y') -> X'

for X being a single class.  To produce a complete target source base, 
walk the single source tree as defined by a 'platform descriptor' :


  foreach(file in descriptor) {
 forward(X, platform);
  }

so for a  source tree

 master/
 dir1/
   X1
   X2
 dir2/
   X3
   X4

and a platform descriptor

 java5/
 dir1/
   X1
 dir2/
   X4

then walking the tree gets us :

  java5/
 dir1/
   Y1
 dir2/
   Y4


Does this match your model?  If not, what's different?  I admit the 
above is simple minded.  I can imagine many, many messy corner cases. 
But it's a reasonable starting point?  I'd like to setup this 'master' 
tree in the sandbox.  Can anyone suggest any good examples of diffs 
between java 5 and java 6 so we can put diffs in the classes X1-4, or 
have paltform unique classes in X1-4?


geir

Re: [classlib] Preprocessor - CHECKPOINT

[Mikhail Fursov]

On 11/3/06, Etienne Gagnon <[EMAIL PROTECTED]> wrote:


Mikhail Fursov wrote:
> Good point! I think Java1.2 classes format is a reason to move J2ME
> development into separate branch.
> The only usecase for preprocessor is left (correct me if I miss
something)
> is Java6 changes.
> ?

a) This is "class format"...  AFAIK, you can use generics for
development, but compile with -target=1.2 or something like it, no?

b) If I'm wrong in a), then what if you wanted additional branches of
harmony: j2se1.4, j2me (classformat1.2)?  Then using the proposed tool
would be IMO preferable to using completely distinct svn branches for
these.


Etienne



Yes,
it's possible to write J2ME applications using Java5 syntax with
generics and use
special compiler to compile it to Java1.2 format.
My position here is:
1) Let's ask volunteers/gurus who do JavaXYZ port what is more convenient
for them
2) The idea with Preprocessor (I mean your version with a special 2-way
tool) is very good and scalable until you do not have diffs in almost every
line.
3) Today we have a lot of changes in classlib: ~100 commits per day. Merging
of branches is almost unreal. Maybe in the future situation will change
because we will have more stable
code. Maybe not: we can have 10 times more developers! Let's wait and
will see :)




--
Mikhail Fursov

Re: [classlib] Preprocessor - CHECKPOINT

[Etienne Gagnon]

Geir Magnusson Jr. wrote:
>> I have nothing against this!  But, we have to make sure:
>> 1- we don't lose the "communication" aspect of telling developers about
>>  parrallel development.
> 
> I'm not sure I understand what problem you are trying to solve there. We
> do parallel development on the source base right now - no one ever does
> a blocking checkout - it's optimistic.


What tells you, when you modify method m() of class C that,in the middle
of it, there's different ways of processing something in j2se6?

e.g.

class C
{
  .../* many, many methods */

  private void m()
  {
... /* canonical j2se5 code */
...
// @Process(j2se6)
// { do something additional }
...
...
  }
}


So, if you were, for example, to move m() to become the first method,
and ALSO change its name...


class C
{
  private void m2()
  {
... /* canonical j2se5 code with modifications!! */
...
// @Process(j2se6)
// { do something additional }
...
...
  }

  .../* many, many methods */

}



Such changes are usually not dealt with very neatly with "svn merge".
Now, if using the processign tool, you get at least 2 benefits over "svn
merge":
1- No need to do any "svn merge"...  The method was moved AND renamed
even for j2se6.
2- You were aware, when moving the method and changing its name that you
could possibly be making the life of j2se6 developers a little more
difficult.

The "communication" aspect of 2 can be quite helpful when doing
system-wide changes.  Just think about the effect of simply doing a
system-wide reindentation of source code; this is a nightmare when
developing using branches, as diff/merge tools are line-based, not
syntax-based.  So, if your new indenter moves things across lines,
you're in hell.

Believe me; I've lived through that trying to maintain a "well indented"
GNU Classpath in SableVM's repository.

I am not saying that using the tool should replace using branches for
all situations, but there are situations where it would be more
effective not to use branches when the code is mostly identical.  Using
traditional pre-processing tools, in such cases, is often the solution
adopted by some people; the problems of "simply-mided" traditiona
pre-processing has been discussed earlier in this thread.  I am
proposing, instead, to use an innovative "syntax-based, revertable
processing".  That's all.  [I guess you'll need to see it working to
make sense of it...  I'll work on a prototype.  I've just submitted the
ICLA/ACQ].

Etienne

-- 
Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/
SableVM:   http://www.sablevm.org/
SableCC:   http://www.sablecc.org/


signature.asc
Description: OpenPGP digital signature

Re: [classlib] Preprocessor - CHECKPOINT

[Etienne Gagnon]

Mikhail Fursov wrote:
> Good point! I think Java1.2 classes format is a reason to move J2ME
> development into separate branch.
> The only usecase for preprocessor is left (correct me if I miss something)
> is Java6 changes.
> ?

a) This is "class format"...  AFAIK, you can use generics for
development, but compile with -target=1.2 or something like it, no?

b) If I'm wrong in a), then what if you wanted additional branches of
harmony: j2se1.4, j2me (classformat1.2)?  Then using the proposed tool
would be IMO preferable to using completely distinct svn branches for these.


Etienne

-- 
Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/
SableVM:   http://www.sablevm.org/
SableCC:   http://www.sablecc.org/


signature.asc
Description: OpenPGP digital signature

Re: [classlib] Preprocessor - CHECKPOINT

[Geir Magnusson Jr.]

Etienne Gagnon wrote:

Geir Magnusson Jr. wrote:
[...] 


See, I'm hoping for something a tad different :

1) For building : process() (and revert() for fun) for cmd line use for
the build scripts, so we just do
... 


2) For development : IDE plugin where

  a) I can tell plugin that my project def/configuration is whatever,
 it using metadata in a file, only consider the code that is
 defined (or not excluded - whatever is easier)

  b) I can tell plugin to look at X, know that I want
 "java5" and in situ, it shows me what Z would look like.
 I edit this, but I'm really editing X.  And I want to
 be sure, I push a button or have a split screen that
 shows me what X really looks like.

 I can edit in X, or in "virtual Z", or both at the same
 time, as I'm just really editing X.


I have nothing against this!  But, we have to make sure:
1- we don't lose the "communication" aspect of telling developers about
 parrallel development.


I'm not sure I understand what problem you are trying to solve there. 
We do parallel development on the source base right now - no one ever 
does a blocking checkout - it's optimistic.




2- we have a robust "language" design; this is best achieved by first
 building the command-line based tool, then extend it into a full-blown
 Eclipse plugin... :-)


Right.  The fact it's the same basic transformation code and process. 
The fact that someone embedded it in eclipse doesn't change the 
fundamental process.


geir



Etienne

Re: [classlib] Preprocessor - CHECKPOINT

[Etienne Gagnon]

Sian January wrote:
> I may be totally off track here, but how about just having two copies of
> all
> the files that differ?  I don't believe it would be that many, and it would
> save us from having complicated source files or having to use special tools
> or special IDE plug-ins.  For me the value of having clearly readable
> source
> code and being able to use an IDE out of the box outweighs any extra effort
> there may be with this solution.

I suspect that "two copies" might eventually become "three, four, five"
copies...  Also, having parrallel copies does not help when modifying
"shared" code in these copies.  Actually, it would be even worse than
using the normal branching feature of Subversion, as you wouldn't get
neat "svn merge" help...

Etienne

-- 
Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/
SableVM:   http://www.sablevm.org/
SableCC:   http://www.sablecc.org/


signature.asc
Description: OpenPGP digital signature

Re: [classlib] Preprocessor - CHECKPOINT

[Geir Magnusson Jr.]

Etienne Gagnon wrote:

Geir Magnusson Jr. wrote:

To release code, one would apply:

 process(X, release-target) => Y

Now, it is important to understand that Y, in this case, is NOT suitable
for doing any modification as

 revert(Y) => Kaboom!  (The tool will simply report that it can't do it;
it won't crash.)
...

This is what I thought we were talking about all along - basically
starting w/ the full source, and pre-process to the "canonical" source
for the target version.

However, I don't understand why I can't go backwards, modulo some manual
merging if needed.

For example, if I have X and release-version, I should be able to take a
Y' and resolve back to X'.  There are problematic cases. [...]



It's the problematic cases I am trying to get entirely rid of "by
design", by having an "exact" transformation process, in contrast with
the usual branch/merge "inexact" process.


At what cost though?  We are used to the inexact process now...



Of course, this means that development happens using "complete" source
code (this is either the canonical form or a development target form,
but not a release form).


What's the diff?



For applying user-provided patches, then it can still be done the good
old way, using these "little" patches made from release code, but
applying them directly to the canonical or dev-target code, resolving
conflicts manually.  [The smaller the patch, the easier it is to apply
it on files with target-specific code.]


In which case we have the inexact process anyway.



The idea is that manual processing is never required when coding using
canonical and dev-target modes.  No inexact modifications, no difficult
merges, etc.  And, as a bonus, communication between parrallel target
developers.  I know, it's an "unusual" development processing tool, but,
hey, why shouldn't Harmony innovate? :-)


I'm missing something then.  You just said you can't go backwards. 
There is only one file for class C in SVN.  How do I work on anything 
but that copy if there is no going backwards?


geir

Re: [classlib] Preprocessor - CHECKPOINT

[Etienne Gagnon]

Geir Magnusson Jr. wrote:
>[...] 
> 
> See, I'm hoping for something a tad different :
> 
> 1) For building : process() (and revert() for fun) for cmd line use for
> the build scripts, so we just do
>... 
> 
> 2) For development : IDE plugin where
>
>   a) I can tell plugin that my project def/configuration is whatever,
>  it using metadata in a file, only consider the code that is
>  defined (or not excluded - whatever is easier)
> 
>   b) I can tell plugin to look at X, know that I want
>  "java5" and in situ, it shows me what Z would look like.
>  I edit this, but I'm really editing X.  And I want to
>  be sure, I push a button or have a split screen that
>  shows me what X really looks like.
> 
>  I can edit in X, or in "virtual Z", or both at the same
>  time, as I'm just really editing X.

I have nothing against this!  But, we have to make sure:
1- we don't lose the "communication" aspect of telling developers about
 parrallel development.
2- we have a robust "language" design; this is best achieved by first
 building the command-line based tool, then extend it into a full-blown
 Eclipse plugin... :-)

Etienne

-- 
Etienne M. Gagnon, Ph.D. [EMAIL PROTECTED]
President  http://www.sablevm.org/
J-Meg inc. http://www.sablecc.org/

-- 
Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/
SableVM:   http://www.sablevm.org/
SableCC:   http://www.sablecc.org/


signature.asc
Description: OpenPGP digital signature

Re: [classlib] Preprocessor - CHECKPOINT

[Etienne Gagnon]

Geir Magnusson Jr. wrote:
>> To release code, one would apply:
>>
>>  process(X, release-target) => Y
>>
>> Now, it is important to understand that Y, in this case, is NOT suitable
>> for doing any modification as
>>
>>  revert(Y) => Kaboom!  (The tool will simply report that it can't do it;
>> it won't crash.)
>>...
> 
> This is what I thought we were talking about all along - basically
> starting w/ the full source, and pre-process to the "canonical" source
> for the target version.
> 
> However, I don't understand why I can't go backwards, modulo some manual
> merging if needed.
> 
> For example, if I have X and release-version, I should be able to take a
> Y' and resolve back to X'.  There are problematic cases. [...]


It's the problematic cases I am trying to get entirely rid of "by
design", by having an "exact" transformation process, in contrast with
the usual branch/merge "inexact" process.

Of course, this means that development happens using "complete" source
code (this is either the canonical form or a development target form,
but not a release form).

For applying user-provided patches, then it can still be done the good
old way, using these "little" patches made from release code, but
applying them directly to the canonical or dev-target code, resolving
conflicts manually.  [The smaller the patch, the easier it is to apply
it on files with target-specific code.]

The idea is that manual processing is never required when coding using
canonical and dev-target modes.  No inexact modifications, no difficult
merges, etc.  And, as a bonus, communication between parrallel target
developers.  I know, it's an "unusual" development processing tool, but,
hey, why shouldn't Harmony innovate? :-)

Etienne

-- 
Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/
SableVM:   http://www.sablevm.org/
SableCC:   http://www.sablecc.org/


signature.asc
Description: OpenPGP digital signature

Re: [classlib] Preprocessor - CHECKPOINT

[Etienne Gagnon]

Tim Ellison wrote:
>>   For example, if class X of the main version is not part of j2me,
>>   "process(j2me)" would move this file to a subdirectory ".streams/".
> 
> Why would you move the files rather than exclude them?
> 
> I was assuming that the processor would generate a whole new source tree
> for each process() target, so that you could work on the original
> checked-out file in it's 'canonicalized form' for Big Java work, or
> process("jme") into a new location and work in the non-canonical form
> your Little Java spectacles on.  [...]

Neat idea!

I would put the following restriction, though:  one should NOT modify
more than one target at a time.

So, you would probably need some way to prevent parallel modifications
in "spectacle views".  One way to achieve this:

process(X,target, destination) =>  Xtarget in a distinct location
 and X files are changed to read-only and some tracking file F tells us
 about Xtarget's location.

So, you would also need:

release(F)  =>  asks all kind of questions (want to lose changes? delete
files? etc.)

Of course:

revert(F?|destination?) => makes X read-write.


[Hoping this was clear enough...  It's not a very good explanation...]

> Agreed.  It would be interesting to determine the most effective
> location for those markers (measured by reverse mapping accuracy vs.
> number of markers).

I am a fan of accuracy...  Yet, maybe it would be simple enough if a
release was always based on a specific svn URL, then the mark could be
totally exact using svn keywords... :-)

Etienne

-- 
Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/
SableVM:   http://www.sablevm.org/
SableCC:   http://www.sablecc.org/


signature.asc
Description: OpenPGP digital signature

Re: [classlib] Preprocessor - CHECKPOINT

[Geir Magnusson Jr.]

Sian January wrote:

On 02/11/06, Geir Magnusson Jr. <[EMAIL PROTECTED]> wrote:




Sian January wrote:
> I believe that in CVS when you make a branch there's nothing in it to
> begin with, so if you check out code from the branch it looks the same
> as code in head.  Then if a change is made in a file in head and that
> file hasn't been changed in the branch that change is reflected in the
> branch.  My understanding is that SVN is based on CVS so I think it
> should work the same way.

Nope.

There is really no such thing as a branch or tag in SVN.  It's just a
copy.  I think it's the feature I miss most from CVS.



That's very unfortunate.  I had assumed that SVN was a superset of CVS as
it's more recent, but obviously I was wrong :-(


it's a shame.  I really, really miss that feature.

geir

Re: [classlib] Preprocessor - CHECKPOINT

[Sian January]

On 02/11/06, Geir Magnusson Jr. <[EMAIL PROTECTED]> wrote:




Sian January wrote:
> I believe that in CVS when you make a branch there's nothing in it to
> begin with, so if you check out code from the branch it looks the same
> as code in head.  Then if a change is made in a file in head and that
> file hasn't been changed in the branch that change is reflected in the
> branch.  My understanding is that SVN is based on CVS so I think it
> should work the same way.

Nope.

There is really no such thing as a branch or tag in SVN.  It's just a
copy.  I think it's the feature I miss most from CVS.



That's very unfortunate.  I had assumed that SVN was a superset of CVS as
it's more recent, but obviously I was wrong :-(


So my point was that if it's only a small
> number of classes that are branched then integrating fixes shouldn't be
> that problematic.  Please feel free to correct me if any of those
> assumptions are wrong.
>
> Just thinking about J2ME, I can imagine that some source files are going
> to be very different.  For example there are no Java 5 features in J2ME,
> so any generic classes will have to be almost completely different.  My
> concern is that trying to combine two quite different classes in the
> same file is going be very difficult to read and understand.

Right.  I don't know the first thing about ME.  Maybe we should come up
w/ use cases based on Java 6 (as we're going to do it at some point) and
work from there.



Yes - that seems to make much more sense.

geir


>
> Thanks,
>
> Sian
>
>
> On 02/11/06, *Geir Magnusson Jr.* <[EMAIL PROTECTED]
> > wrote:
>
>
>
> Sian January wrote:
>  > I may be totally off track here, but how about just having two
copies
>  > of all the files that differ?  I don't believe it would be that
many,
>  > and it would save us from having complicated source files or
> having to
>  > use special tools or special IDE plug-ins.  For me the value of
> having
>  > clearly readable source code and being able to use an IDE out of
> the box
>  > outweighs any extra effort there may be with this solution.
>
> Because I think that still means we have separate branches, and thus
> the
> integration problem for fixes.
>
> geir
>
>  >
>  > Regards,
>  >
>  > Sian
>  >
>  >
>  >
>  > On 31/10/06, *Geir Magnusson Jr.* <[EMAIL PROTECTED]
> 
>  > >> wrote:
>  >
>  > So we all agree that it's not an ideal solution.
>  >
>  > Can anyone think of anything else?  No one said this was
> going to be
>  > easy...
>  >
>  > geir
>  >
>  >
>  >
>  >
>  > --
>  > Sian January
>  >
>  > IBM Java Technology Centre, UK
>
>
>
>
> --
> Sian January
>
> IBM Java Technology Centre, UK





--
Sian January

IBM Java Technology Centre, UK

Re: [classlib] Preprocessor - CHECKPOINT

[Mikhail Fursov]

On 11/2/06, Sian January <[EMAIL PROTECTED]> wrote:


Just thinking about J2ME, I can imagine that some source files are going
to be very different.  For example there are no Java 5 features in J2ME,
so
any generic classes will have to be almost completely different.  My
concern
is that trying to combine two quite different classes in the same file is
going be very difficult to read and understand.



Good point! I think Java1.2 classes format is a reason to move J2ME
development into separate branch.
The only usecase for preprocessor is left (correct me if I miss something)
is Java6 changes.
?

--
Mikhail Fursov

Re: [classlib] Preprocessor - CHECKPOINT

[Geir Magnusson Jr.]

Sian January wrote:
I believe that in CVS when you make a branch there's nothing in it to 
begin with, so if you check out code from the branch it looks the same 
as code in head.  Then if a change is made in a file in head and that 
file hasn't been changed in the branch that change is reflected in the 
branch.  My understanding is that SVN is based on CVS so I think it 
should work the same way.  


Nope.

There is really no such thing as a branch or tag in SVN.  It's just a 
copy.  I think it's the feature I miss most from CVS.


So my point was that if it's only a small 
number of classes that are branched then integrating fixes shouldn't be 
that problematic.  Please feel free to correct me if any of those 
assumptions are wrong.
 
Just thinking about J2ME, I can imagine that some source files are going 
to be very different.  For example there are no Java 5 features in J2ME, 
so any generic classes will have to be almost completely different.  My 
concern is that trying to combine two quite different classes in the 
same file is going be very difficult to read and understand.


Right.  I don't know the first thing about ME.  Maybe we should come up 
w/ use cases based on Java 6 (as we're going to do it at some point) and 
work from there.


geir

 
Thanks,
 
Sian



On 02/11/06, *Geir Magnusson Jr.* <[EMAIL PROTECTED] 
> wrote:




Sian January wrote:
 > I may be totally off track here, but how about just having two copies
 > of all the files that differ?  I don't believe it would be that many,
 > and it would save us from having complicated source files or
having to
 > use special tools or special IDE plug-ins.  For me the value of
having
 > clearly readable source code and being able to use an IDE out of
the box
 > outweighs any extra effort there may be with this solution.

Because I think that still means we have separate branches, and thus
the
integration problem for fixes.

geir

 >
 > Regards,
 >
 > Sian
 >
 >
 >
 > On 31/10/06, *Geir Magnusson Jr.* <[EMAIL PROTECTED]

 > >> wrote:
 >
 > So we all agree that it's not an ideal solution.
 >
 > Can anyone think of anything else?  No one said this was
going to be
 > easy...
 >
 > geir
 >
 >
 >
 >
 > --
 > Sian January
 >
 > IBM Java Technology Centre, UK




--
Sian January

IBM Java Technology Centre, UK

Re: [classlib] Preprocessor - CHECKPOINT

[Sian January]

I believe that in CVS when you make a branch there's nothing in it to begin
with, so if you check out code from the branch it looks the same as code in
head.  Then if a change is made in a file in head and that file hasn't been
changed in the branch that change is reflected in the branch.  My
understanding is that SVN is based on CVS so I think it should work the same
way.  So my point was that if it's only a small number of classes that
are branched then integrating fixes shouldn't be that problematic.  Please
feel free to correct me if any of those assumptions are wrong.

Just thinking about J2ME, I can imagine that some source files are going
to be very different.  For example there are no Java 5 features in J2ME, so
any generic classes will have to be almost completely different.  My concern
is that trying to combine two quite different classes in the same file is
going be very difficult to read and understand.

Thanks,

Sian


On 02/11/06, Geir Magnusson Jr. <[EMAIL PROTECTED]> wrote:




Sian January wrote:
> I may be totally off track here, but how about just having two copies
> of all the files that differ?  I don't believe it would be that many,
> and it would save us from having complicated source files or having to
> use special tools or special IDE plug-ins.  For me the value of having
> clearly readable source code and being able to use an IDE out of the box
> outweighs any extra effort there may be with this solution.

Because I think that still means we have separate branches, and thus the
integration problem for fixes.

geir

>
> Regards,
>
> Sian
>
>
>
> On 31/10/06, *Geir Magnusson Jr.* <[EMAIL PROTECTED]
> > wrote:
>
> So we all agree that it's not an ideal solution.
>
> Can anyone think of anything else?  No one said this was going to be
> easy...
>
> geir
>
>
>
>
> --
> Sian January
>
> IBM Java Technology Centre, UK





--
Sian January

IBM Java Technology Centre, UK

Re: [classlib] Preprocessor - CHECKPOINT

[Geir Magnusson Jr.]

Sian January wrote:
I may be totally off track here, but how about just having two copies 
of all the files that differ?  I don't believe it would be that many, 
and it would save us from having complicated source files or having to 
use special tools or special IDE plug-ins.  For me the value of having 
clearly readable source code and being able to use an IDE out of the box 
outweighs any extra effort there may be with this solution.


Because I think that still means we have separate branches, and thus the 
integration problem for fixes.


geir

 
Regards,


Sian
 

 
On 31/10/06, *Geir Magnusson Jr.* <[EMAIL PROTECTED] 
> wrote:


So we all agree that it's not an ideal solution.

Can anyone think of anything else?  No one said this was going to be
easy...

geir




--
Sian January

IBM Java Technology Centre, UK

Re: [classlib] Preprocessor - CHECKPOINT

[Sian January]

I may be totally off track here, but how about just having two copies of all
the files that differ?  I don't believe it would be that many, and it would
save us from having complicated source files or having to use special tools
or special IDE plug-ins.  For me the value of having clearly readable source
code and being able to use an IDE out of the box outweighs any extra effort
there may be with this solution.

Regards,

Sian



On 31/10/06, Geir Magnusson Jr. <[EMAIL PROTECTED]> wrote:


So we all agree that it's not an ideal solution.

Can anyone think of anything else?  No one said this was going to be
easy...

geir





--
Sian January

IBM Java Technology Centre, UK

Re: [classlib] Preprocessor - CHECKPOINT

[Geir Magnusson Jr.]

Cool.  Any Eclipse Plugin gurus here that could create a called in eclipse speak... perspective?> that has two edit panes, one 
with the code w the preproc directives in it, and one that is the result 
of the preproc given some defines?


geir


Nathan Beyer wrote:

I've read a few articles about J2ME development using Antenna, which
has a Java preprocessor task for Ant [1]. The Mobility Pack for
NetBeans includes a Java preprocessor, which it claims is
similar/based on Antenna [2].

I still feel a little dirty about the thought of preprocessing Java
source, but I guess it's done.

-Nathan

[1] http://antenna.sourceforge.net/#preprocess
[2] http://www.netbeans.org/kb/50/quickstart-mobility.html

On 11/1/06, Geir Magnusson Jr. <[EMAIL PROTECTED]> wrote:



Etienne Gagnon wrote:
> Geir Magnusson Jr. wrote:
>> There's caching too, I think.  LogCache4J
>>
>> What I meant was that it didn't seem like we came to a conclusion 
on it

>> - that if we had a general pre-processing solution, we could use that
>> too for logging, rather than have two.
>>
>> The actual use-cases will help figure this out.
>
> Here two typical some use cases, and some proposed solutions:
>
> Problem
> ---
> logging, and other situations where you really don't want to put the
> "additional" source code in the main source files
>
> Solution
> 
> use aspects  (Plug: you might want to give a look at the optimizing abc
> compiler)
>
>
> Problem
> ---
> supporting a different API specifications/versions, such as j2me and
> j2se1.4, in addition to the "main" version (e.g. j2se1.5)
>
>
> Solution
> 
>

Right - these were the main hypotheticals that started this whole thread
off - I was thinking more about a definite example, like a given class C
has a different implementation of method M in Java5 than in Java6.

Even if we walk through this with a clear example simple manual process,
it will be enlightening.

> This is a trickier problem.  We can divide the problem at two main 
levels:

>
> 1- file/directory level
> 2- source-code level
>
> At the file/directory level, each version (e.g. j2me, j2se1.4, ...)
> might include additional files (relative to the main version), and 
might

> not include some files of the main version.  In other words, j2me might
> not contain database APIs.

Yep

>
> Managing file inclusion/exclusion can be done in various ways.
>
>  a) ant based:  use distinct ant (sub-)files for each version.
>
>The problem is that IDEs (e.g. Eclipse) will most likely show some
>of the excluded files in its class/files browser.  This wouldn't be
>very elegant.  It also implies "always" compiling through ant files.
>
>Of course, one could develop Eclipse-specific configuration files to
>mimic the inclusion/exclusion of ant files, but then, this opens the
>door for discrepancies between ant vs eclipse inclusion/exclusion
>lists.  I don't like it.
>

Me neither.

>  b) custom-tool based: the custom processing tool I proposed could also
>parse inclusion/exclusion lists, and use these lists to move files
>around, in addition to processing the content of unmoved files.

I'm not sure anything needs to be moved.

>
>For example, if class X of the main version is not part of j2me,
>"process(j2me)" would move this file to a subdirectory ".streams/".
>
>If a class Y is not in the "main" version (the one used for "svn
>ci"), it resides in subdirectory ".streams" in the trunk.
>"process(j2me)" moves this file into the normal directory.
>
>As for IDEs, now you can configure them to automatically exclude
>".stream/" directories.

This would get messy.  I'd rather just have a plug-in that reads a
description file for version X and just doesn't show me what isn't part
of version X.

>
>Inclusion/exclusion could be managed in two ways:
>1- the processing tool could look for inclusion/exclusion list 
files,

>   such as "j2me.inclusion, j2me.exclusion, main.inclusion,
>   main.exclusion, etc."
>
>   This would lead to the best performance (for process(X)), yet it
>   does require much manual update of inclusion/exclusion lists, 
with

>   all the discrepancies that result from human mistakes while
>   updating these files.
>
>2- (my preferred way), directives, at the beginning of the source
>   code would indicate whether a file is included in a version or
>   not.  Depending on target, the file would be moved to the
>   appropriate directory (normal, ".streams").

Either one - the nice thing about #1 over #2 is that you can actually
look at it to get a summary.  But you can generate the same info w/ a
tree walk, I suppose.

>
>
> Of course, there's also the problem of non-source files, i.e.
> resources.  IMO, resources could be managed using specific
> directories (".main/", ".j2me", ".j2se1.4") and a ".shared/"
> directory with symbolic links in the specific directories.
>

This is getting messy.

>

Re: [classlib] Preprocessor - CHECKPOINT

[Nathan Beyer]

I've read a few articles about J2ME development using Antenna, which
has a Java preprocessor task for Ant [1]. The Mobility Pack for
NetBeans includes a Java preprocessor, which it claims is
similar/based on Antenna [2].

I still feel a little dirty about the thought of preprocessing Java
source, but I guess it's done.

-Nathan

[1] http://antenna.sourceforge.net/#preprocess
[2] http://www.netbeans.org/kb/50/quickstart-mobility.html

On 11/1/06, Geir Magnusson Jr. <[EMAIL PROTECTED]> wrote:



Etienne Gagnon wrote:
> Geir Magnusson Jr. wrote:
>> There's caching too, I think.  LogCache4J
>>
>> What I meant was that it didn't seem like we came to a conclusion on it
>> - that if we had a general pre-processing solution, we could use that
>> too for logging, rather than have two.
>>
>> The actual use-cases will help figure this out.
>
> Here two typical some use cases, and some proposed solutions:
>
> Problem
> ---
> logging, and other situations where you really don't want to put the
> "additional" source code in the main source files
>
> Solution
> 
> use aspects  (Plug: you might want to give a look at the optimizing abc
> compiler)
>
>
> Problem
> ---
> supporting a different API specifications/versions, such as j2me and
> j2se1.4, in addition to the "main" version (e.g. j2se1.5)
>
>
> Solution
> 
>

Right - these were the main hypotheticals that started this whole thread
off - I was thinking more about a definite example, like a given class C
has a different implementation of method M in Java5 than in Java6.

Even if we walk through this with a clear example simple manual process,
it will be enlightening.

> This is a trickier problem.  We can divide the problem at two main levels:
>
> 1- file/directory level
> 2- source-code level
>
> At the file/directory level, each version (e.g. j2me, j2se1.4, ...)
> might include additional files (relative to the main version), and might
> not include some files of the main version.  In other words, j2me might
> not contain database APIs.

Yep

>
> Managing file inclusion/exclusion can be done in various ways.
>
>  a) ant based:  use distinct ant (sub-)files for each version.
>
>The problem is that IDEs (e.g. Eclipse) will most likely show some
>of the excluded files in its class/files browser.  This wouldn't be
>very elegant.  It also implies "always" compiling through ant files.
>
>Of course, one could develop Eclipse-specific configuration files to
>mimic the inclusion/exclusion of ant files, but then, this opens the
>door for discrepancies between ant vs eclipse inclusion/exclusion
>lists.  I don't like it.
>

Me neither.

>  b) custom-tool based: the custom processing tool I proposed could also
>parse inclusion/exclusion lists, and use these lists to move files
>around, in addition to processing the content of unmoved files.

I'm not sure anything needs to be moved.

>
>For example, if class X of the main version is not part of j2me,
>"process(j2me)" would move this file to a subdirectory ".streams/".
>
>If a class Y is not in the "main" version (the one used for "svn
>ci"), it resides in subdirectory ".streams" in the trunk.
>"process(j2me)" moves this file into the normal directory.
>
>As for IDEs, now you can configure them to automatically exclude
>".stream/" directories.

This would get messy.  I'd rather just have a plug-in that reads a
description file for version X and just doesn't show me what isn't part
of version X.

>
>Inclusion/exclusion could be managed in two ways:
>1- the processing tool could look for inclusion/exclusion list files,
>   such as "j2me.inclusion, j2me.exclusion, main.inclusion,
>   main.exclusion, etc."
>
>   This would lead to the best performance (for process(X)), yet it
>   does require much manual update of inclusion/exclusion lists, with
>   all the discrepancies that result from human mistakes while
>   updating these files.
>
>2- (my preferred way), directives, at the beginning of the source
>   code would indicate whether a file is included in a version or
>   not.  Depending on target, the file would be moved to the
>   appropriate directory (normal, ".streams").

Either one - the nice thing about #1 over #2 is that you can actually
look at it to get a summary.  But you can generate the same info w/ a
tree walk, I suppose.

>
>
> Of course, there's also the problem of non-source files, i.e.
> resources.  IMO, resources could be managed using specific
> directories (".main/", ".j2me", ".j2se1.4") and a ".shared/"
> directory with symbolic links in the specific directories.
>

This is getting messy.

>
> As for source-level management, you would use my proposed processing
> tool to view the source files with the right spectacles [as Tim said so
> elegantly:-)].  For "development targets", it is important that:
>
>  revert(process(X, target)) => X
>
> By "development target"

Re: [classlib] Preprocessor - CHECKPOINT

[Geir Magnusson Jr.]

Etienne Gagnon wrote:

Mikhail Fursov wrote:

On 11/1/06, Etienne Gagnon <[EMAIL PROTECTED]> wrote:

For comfortable IDE development, one could imagine that the IDE editor
can reduce to "one-line visible" comments (or better, specially
formatted ones) so that it gives you the impression that you are really
wearing target-specific spectacles.  [I know Eclipse allows for such
things already].
...

Etienne,
What is 'comfortable IDE development' if you can't modify the Y? Am I
missing something here?


Maybe my text was wrongly formatted...  You looked at the wrong example.
 Comfortable development happens only using "development targets".  E.g.

1- process(X, devtarget) -> Z

2- edit Z in IDE using comfortable development, where you see a single
   commented line for every hidden stream code chunk, keeping you aware
   that other streams have related code there [you click on the "+" in
   Eclipse if you want to see the complete chunk].  Of course, you
   should never delete a chunk without consulting other stream
   developers first.
   So:  edit Z -> Z'


See, I'm hoping for something a tad different :

1) For building : process() (and revert() for fun) for cmd line use for 
the build scripts, so we just do


  $ ant -Dtarget=java5

and  what pops out is the java5 classlib jars, natives, includes, etc 
for java5.  And would use the "version definition" metadata to choose 
the fileset, and then use process() to xform the code to the target sources.


2) For development : IDE plugin where

  a) I can tell plugin that my project def/configuration is whatever,
 it using metadata in a file, only consider the code that is
 defined (or not excluded - whatever is easier)

  b) I can tell plugin to look at X, know that I want
 "java5" and in situ, it shows me what Z would look like.
 I edit this, but I'm really editing X.  And I want to
 be sure, I push a button or have a split screen that
 shows me what X really looks like.

 I can edit in X, or in "virtual Z", or both at the same
 time, as I'm just really editing X.

I think it's key that our solution can work in an IDE, not just on a 
command line.



geir

Re: [classlib] Preprocessor - CHECKPOINT

[Geir Magnusson Jr.]

Etienne Gagnon wrote:

Geir Magnusson Jr. wrote:

There's caching too, I think.  LogCache4J

What I meant was that it didn't seem like we came to a conclusion on it
- that if we had a general pre-processing solution, we could use that
too for logging, rather than have two.

The actual use-cases will help figure this out.


Here two typical some use cases, and some proposed solutions:

Problem
---
logging, and other situations where you really don't want to put the
"additional" source code in the main source files

Solution

use aspects  (Plug: you might want to give a look at the optimizing abc
compiler)


Problem
---
supporting a different API specifications/versions, such as j2me and
j2se1.4, in addition to the "main" version (e.g. j2se1.5)


Solution




Right - these were the main hypotheticals that started this whole thread 
off - I was thinking more about a definite example, like a given class C 
has a different implementation of method M in Java5 than in Java6.


Even if we walk through this with a clear example simple manual process, 
it will be enlightening.



This is a trickier problem.  We can divide the problem at two main levels:

1- file/directory level
2- source-code level

At the file/directory level, each version (e.g. j2me, j2se1.4, ...)
might include additional files (relative to the main version), and might
not include some files of the main version.  In other words, j2me might
not contain database APIs.


Yep



Managing file inclusion/exclusion can be done in various ways.

 a) ant based:  use distinct ant (sub-)files for each version.

   The problem is that IDEs (e.g. Eclipse) will most likely show some
   of the excluded files in its class/files browser.  This wouldn't be
   very elegant.  It also implies "always" compiling through ant files.

   Of course, one could develop Eclipse-specific configuration files to
   mimic the inclusion/exclusion of ant files, but then, this opens the
   door for discrepancies between ant vs eclipse inclusion/exclusion
   lists.  I don't like it.



Me neither.


 b) custom-tool based: the custom processing tool I proposed could also
   parse inclusion/exclusion lists, and use these lists to move files
   around, in addition to processing the content of unmoved files.


I'm not sure anything needs to be moved.



   For example, if class X of the main version is not part of j2me,
   "process(j2me)" would move this file to a subdirectory ".streams/".

   If a class Y is not in the "main" version (the one used for "svn
   ci"), it resides in subdirectory ".streams" in the trunk.
   "process(j2me)" moves this file into the normal directory.

   As for IDEs, now you can configure them to automatically exclude
   ".stream/" directories.


This would get messy.  I'd rather just have a plug-in that reads a 
description file for version X and just doesn't show me what isn't part 
of version X.




   Inclusion/exclusion could be managed in two ways:
   1- the processing tool could look for inclusion/exclusion list files,
  such as "j2me.inclusion, j2me.exclusion, main.inclusion,
  main.exclusion, etc."

  This would lead to the best performance (for process(X)), yet it
  does require much manual update of inclusion/exclusion lists, with
  all the discrepancies that result from human mistakes while
  updating these files.

   2- (my preferred way), directives, at the beginning of the source
  code would indicate whether a file is included in a version or
  not.  Depending on target, the file would be moved to the
  appropriate directory (normal, ".streams").


Either one - the nice thing about #1 over #2 is that you can actually 
look at it to get a summary.  But you can generate the same info w/ a 
tree walk, I suppose.





Of course, there's also the problem of non-source files, i.e.
resources.  IMO, resources could be managed using specific
directories (".main/", ".j2me", ".j2se1.4") and a ".shared/"
directory with symbolic links in the specific directories.



This is getting messy.



As for source-level management, you would use my proposed processing
tool to view the source files with the right spectacles [as Tim said so
elegantly:-)].  For "development targets", it is important that:

 revert(process(X, target)) => X

By "development target" I mean a target that is meant for Harmony
developers in prevision of reverting "modified code" to a format
suitable for "svn ci" (i.e. revert to "main" target).

For comfortable IDE development, one could imagine that the IDE editor
can reduce to "one-line visible" comments (or better, specially
formatted ones) so that it gives you the impression that you are really
wearing target-specific spectacles.  [I know Eclipse allows for such
things already].

To release code, one would apply:

 process(X, release-target) => Y

Now, it is important to understand that Y, in this case, is NOT suitable
for doing any modification as

 revert(Y) 

RE: [classlib] Preprocessor - CHECKPOINT

[Fedotov, Alexei A]

> I am really offering to develop this custom tool.  

+1
For me this is a key message. For sure that would be a nice tool. Does
it worth to dig use cases even more? Do we plan to develop j2me here at
Harmony?

Etienne,
I wonder if it worth to have SableCC itself exposed as a part of
Harmony. Say, you post a JSR for compiler generation API and add an
appropriate package to Harmony JRE.


With best regards,
Alexei Fedotov,
Intel Java & XML Engineering

>-Original Message-
>From: Etienne Gagnon [mailto:[EMAIL PROTECTED]
>Sent: Wednesday, November 01, 2006 4:51 PM
>To: harmony-dev@incubator.apache.org
>Subject: Re: [classlib] Preprocessor - CHECKPOINT
>
>Geir Magnusson Jr. wrote:
>> There's caching too, I think.  LogCache4J
>>
>> What I meant was that it didn't seem like we came to a conclusion on
it
>> - that if we had a general pre-processing solution, we could use that
>> too for logging, rather than have two.
>>
>> The actual use-cases will help figure this out.
>
>Here two typical some use cases, and some proposed solutions:
>
>Problem
>---
>logging, and other situations where you really don't want to put the
>"additional" source code in the main source files
>
>Solution
>
>use aspects  (Plug: you might want to give a look at the optimizing abc
>compiler)
>
>
>Problem
>---
>supporting a different API specifications/versions, such as j2me and
>j2se1.4, in addition to the "main" version (e.g. j2se1.5)
>
>
>Solution
>
>
>This is a trickier problem.  We can divide the problem at two main
levels:
>
>1- file/directory level
>2- source-code level
>
>At the file/directory level, each version (e.g. j2me, j2se1.4, ...)
>might include additional files (relative to the main version), and
might
>not include some files of the main version.  In other words, j2me might
>not contain database APIs.
>
>Managing file inclusion/exclusion can be done in various ways.
>
> a) ant based:  use distinct ant (sub-)files for each version.
>
>   The problem is that IDEs (e.g. Eclipse) will most likely show some
>   of the excluded files in its class/files browser.  This wouldn't be
>   very elegant.  It also implies "always" compiling through ant files.
>
>   Of course, one could develop Eclipse-specific configuration files to
>   mimic the inclusion/exclusion of ant files, but then, this opens the
>   door for discrepancies between ant vs eclipse inclusion/exclusion
>   lists.  I don't like it.
>
> b) custom-tool based: the custom processing tool I proposed could also
>   parse inclusion/exclusion lists, and use these lists to move files
>   around, in addition to processing the content of unmoved files.
>
>   For example, if class X of the main version is not part of j2me,
>   "process(j2me)" would move this file to a subdirectory ".streams/".
>
>   If a class Y is not in the "main" version (the one used for "svn
>   ci"), it resides in subdirectory ".streams" in the trunk.
>   "process(j2me)" moves this file into the normal directory.
>
>   As for IDEs, now you can configure them to automatically exclude
>   ".stream/" directories.
>
>   Inclusion/exclusion could be managed in two ways:
>   1- the processing tool could look for inclusion/exclusion list
files,
>  such as "j2me.inclusion, j2me.exclusion, main.inclusion,
>  main.exclusion, etc."
>
>  This would lead to the best performance (for process(X)), yet it
>  does require much manual update of inclusion/exclusion lists,
with
>  all the discrepancies that result from human mistakes while
>  updating these files.
>
>   2- (my preferred way), directives, at the beginning of the source
>  code would indicate whether a file is included in a version or
>  not.  Depending on target, the file would be moved to the
>  appropriate directory (normal, ".streams").
>
>
>Of course, there's also the problem of non-source files, i.e.
>resources.  IMO, resources could be managed using specific
>directories (".main/", ".j2me", ".j2se1.4") and a ".shared/"
>directory with symbolic links in the specific directories.
>
>
>As for source-level management, you would use my proposed processing
>tool to view the source files with the right spectacles [as Tim said so
>elegantly:-)].  For "development targets", it is important that:
>
> revert(process(X, target)) => X
>
>By "development target" I mean a target that is meant for Harmony
>developers in prevision of rev

Re: [classlib] Preprocessor - CHECKPOINT

[Tim Ellison]

Etienne Gagnon wrote:
> Here two typical some use cases, and some proposed solutions:
> 
> Problem
> ---
> logging, and other situations where you really don't want to put the
> "additional" source code in the main source files
> 
> Solution
> 
> use aspects  (Plug: you might want to give a look at the optimizing abc
> compiler)

+1  and/or 'intrinsic' VM-level tracing of the Java code.

> Problem
> ---
> supporting a different API specifications/versions, such as j2me and
> j2se1.4, in addition to the "main" version (e.g. j2se1.5)
> 
> 
> Solution
> 
> 
> This is a trickier problem.  We can divide the problem at two main levels:
> 
> 1- file/directory level
> 2- source-code level
> 
> At the file/directory level, each version (e.g. j2me, j2se1.4, ...)
> might include additional files (relative to the main version), and might
> not include some files of the main version.  In other words, j2me might
> not contain database APIs.
> 
> Managing file inclusion/exclusion can be done in various ways.
> 
>  a) ant based:  use distinct ant (sub-)files for each version.
> 
>The problem is that IDEs (e.g. Eclipse) will most likely show some
>of the excluded files in its class/files browser.  This wouldn't be
>very elegant.  It also implies "always" compiling through ant files.
> 
>Of course, one could develop Eclipse-specific configuration files to
>mimic the inclusion/exclusion of ant files, but then, this opens the
>door for discrepancies between ant vs eclipse inclusion/exclusion
>lists.  I don't like it.
> 
>  b) custom-tool based: the custom processing tool I proposed could also
>parse inclusion/exclusion lists, and use these lists to move files
>around, in addition to processing the content of unmoved files.
> 
>For example, if class X of the main version is not part of j2me,
>"process(j2me)" would move this file to a subdirectory ".streams/".

Why would you move the files rather than exclude them?

I was assuming that the processor would generate a whole new source tree
for each process() target, so that you could work on the original
checked-out file in it's 'canonicalized form' for Big Java work, or
process("jme") into a new location and work in the non-canonical form
your Little Java spectacles on.  That would prevent you accidentally
checking in the Little Java code, since you would need to
reverse-process the code back to replace the checked-out file (with a
test to ensure you are not overwriting changes made to Big Java, or for
extra credit, merging changes in the working copies).  The svn check-in
would then always be in the Big Java canonicalized form.

This idea would loose the ability to check-in directly from the IDE
though for people working with the non-canonicalized source form.

>If a class Y is not in the "main" version (the one used for "svn
>ci"), it resides in subdirectory ".streams" in the trunk.
>"process(j2me)" moves this file into the normal directory.
> 
>As for IDEs, now you can configure them to automatically exclude
>".stream/" directories.
> 
>Inclusion/exclusion could be managed in two ways:
>1- the processing tool could look for inclusion/exclusion list files,
>   such as "j2me.inclusion, j2me.exclusion, main.inclusion,
>   main.exclusion, etc."
> 
>   This would lead to the best performance (for process(X)), yet it
>   does require much manual update of inclusion/exclusion lists, with
>   all the discrepancies that result from human mistakes while
>   updating these files.
> 
>2- (my preferred way), directives, at the beginning of the source
>   code would indicate whether a file is included in a version or
>   not.  Depending on target, the file would be moved to the
>   appropriate directory (normal, ".streams").

Agreed. Since we are requiring the source to be processed for any
deliverable target, we might as well keep the mark-up in-line.

> Of course, there's also the problem of non-source files, i.e.
> resources.  IMO, resources could be managed using specific
> directories (".main/", ".j2me", ".j2se1.4") and a ".shared/"
> directory with symbolic links in the specific directories.
> 
> 
> As for source-level management, you would use my proposed processing
> tool to view the source files with the right spectacles [as Tim said so
> elegantly:-)].  For "development targets", it is important that:
> 
>  revert(process(X, target)) => X
> 
> By "development target" I mean a target that is meant for Harmony
> developers in prevision of reverting "modified code" to a format
> suitable for "svn ci" (i.e. revert to "main" target).
> 
> For comfortable IDE development, one could imagine that the IDE editor
> can reduce to "one-line visible" comments (or better, specially
> formatted ones) so that it gives you the impression that you are really
> wearing target-specific spectacles.  [I know Eclipse allows for such
> things already].
>

Re: [classlib] Preprocessor - CHECKPOINT

[Mikhail Fursov]

On 11/1/06, Etienne Gagnon <[EMAIL PROTECTED]> wrote:


Comfortable development happens only using "development targets".  E.g.

1- process(X, devtarget) -> Z

2- edit Z in IDE using comfortable development, where you see a single
   commented line for every hidden stream code chunk, keeping you aware
   that other streams have related code there [you click on the "+" in
   Eclipse if you want to see the complete chunk].  Of course, you
   should never delete a chunk without consulting other stream
   developers first.
   So:  edit Z -> Z'

3- revert(Z') -> X'   this works, as long as devtarget is a stream code
  preserving target (a "development target").

4- svn ci of X'  :-)



Etienne, thanks! Now I understand  how it works.
Having such a tool seems like a very promising good idea!


--
Mikhail Fursov

Re: [classlib] Preprocessor - CHECKPOINT

[Etienne Gagnon]

Mikhail Fursov wrote:
> On 11/1/06, Etienne Gagnon <[EMAIL PROTECTED]> wrote:
>> For comfortable IDE development, one could imagine that the IDE editor
>> can reduce to "one-line visible" comments (or better, specially
>> formatted ones) so that it gives you the impression that you are really
>> wearing target-specific spectacles.  [I know Eclipse allows for such
>> things already].
>>...
> Etienne,
> What is 'comfortable IDE development' if you can't modify the Y? Am I
> missing something here?

Maybe my text was wrongly formatted...  You looked at the wrong example.
 Comfortable development happens only using "development targets".  E.g.

1- process(X, devtarget) -> Z

2- edit Z in IDE using comfortable development, where you see a single
   commented line for every hidden stream code chunk, keeping you aware
   that other streams have related code there [you click on the "+" in
   Eclipse if you want to see the complete chunk].  Of course, you
   should never delete a chunk without consulting other stream
   developers first.
   So:  edit Z -> Z'

3- revert(Z') -> X'   this works, as long as devtarget is a stream code
  preserving target (a "development target").

4- svn ci of X'  :-)


You wouldn't want to do the same with a "release target".  A release
target is a target where you want to completely remove other streams
source code from the processor output.  This is so your typical J2SE
"end developer" that looks at Harmony's source.zip code wouldn't have
lots of non J2SE commented code in his face.  [This was a concern
expressed earlier in this thread].

The output of process(X, releasetarget) should NOT be used for
development, not within, nor outside an IDE.  It's only role is to give
end developers "clean" source code to look at.  :-)

Etienne
PS:  Note how "revert()" does not expect a target argument.  So, to
switch from j2me to j2xx development, one must: process(revert(Z),j2xx)
[where Z == process(X,j2me)].  This simplifies quite a few things...


-- 
Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/
SableVM:   http://www.sablevm.org/
SableCC:   http://www.sablecc.org/


signature.asc
Description: OpenPGP digital signature

Re: [classlib] Preprocessor - CHECKPOINT

[Mikhail Fursov]

On 11/1/06, Etienne Gagnon <[EMAIL PROTECTED]> wrote:


For comfortable IDE development, one could imagine that the IDE editor
can reduce to "one-line visible" comments (or better, specially
formatted ones) so that it gives you the impression that you are really
wearing target-specific spectacles.  [I know Eclipse allows for such
things already].

To release code, one would apply:

process(X, release-target) => Y

Now, it is important to understand that Y, in this case, is NOT suitable
for doing any modification as

revert(Y) => Kaboom!  (The tool will simply report that it can't do it;
it won't crash.)



Etienne,
What is 'comfortable IDE development' if you can't modify the Y? Am I
missing something here?

--
Mikhail Fursov

Re: [classlib] Preprocessor - CHECKPOINT

[Etienne Gagnon]

Mikhail Fursov wrote:
>> At the file/directory level, each version (e.g. j2me, j2se1.4, ...)
...
> Just my $0.02: IMO it's unreal to support J2SE 1.4 & 1.5 in the same
> source.
> Too many differences in the language due to generics. This example needs
> branches & weekly manual merges (not a big problem imho)

You're absolutely right!  I wasn't thinking about it when I took that
example.  This is really a typical case (j2se 1.4) where using svn
branches is the right solution.

But, for j2me1.5, java for credit cards 1.5 and j2se1.6 (maybe, if they
don't redesign the Java syntax, etc.), I think that a tool along the
lines I was proposing would be best.


Etienne

-- 
Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/
SableVM:   http://www.sablevm.org/
SableCC:   http://www.sablecc.org/


signature.asc
Description: OpenPGP digital signature

Re: [classlib] Preprocessor - CHECKPOINT

[Mikhail Fursov]

On 11/1/06, Etienne Gagnon <[EMAIL PROTECTED]> wrote:


Problem
---
supporting a different API specifications/versions, such as j2me and
j2se1.4, in addition to the "main" version (e.g. j2se1.5)


Solution


This is a trickier problem.  We can divide the problem at two main levels:

1- file/directory level
2- source-code level

At the file/directory level, each version (e.g. j2me, j2se1.4, ...)
might include additional files (relative to the main version), and might
not include some files of the main version.  In other words, j2me might
not contain database APIs.

Managing file inclusion/exclusion can be done in various ways.



Just my $0.02: IMO it's unreal to support J2SE 1.4 & 1.5 in the same source.
Too many differences in the language due to generics. This example needs
branches & weekly manual merges (not a big problem imho)


--
Mikhail Fursov

Re: [classlib] Preprocessor - CHECKPOINT

[Etienne Gagnon]

Geir Magnusson Jr. wrote:
> There's caching too, I think.  LogCache4J
> 
> What I meant was that it didn't seem like we came to a conclusion on it
> - that if we had a general pre-processing solution, we could use that
> too for logging, rather than have two.
> 
> The actual use-cases will help figure this out.

Here two typical some use cases, and some proposed solutions:

Problem
---
logging, and other situations where you really don't want to put the
"additional" source code in the main source files

Solution

use aspects  (Plug: you might want to give a look at the optimizing abc
compiler)


Problem
---
supporting a different API specifications/versions, such as j2me and
j2se1.4, in addition to the "main" version (e.g. j2se1.5)


Solution


This is a trickier problem.  We can divide the problem at two main levels:

1- file/directory level
2- source-code level

At the file/directory level, each version (e.g. j2me, j2se1.4, ...)
might include additional files (relative to the main version), and might
not include some files of the main version.  In other words, j2me might
not contain database APIs.

Managing file inclusion/exclusion can be done in various ways.

 a) ant based:  use distinct ant (sub-)files for each version.

   The problem is that IDEs (e.g. Eclipse) will most likely show some
   of the excluded files in its class/files browser.  This wouldn't be
   very elegant.  It also implies "always" compiling through ant files.

   Of course, one could develop Eclipse-specific configuration files to
   mimic the inclusion/exclusion of ant files, but then, this opens the
   door for discrepancies between ant vs eclipse inclusion/exclusion
   lists.  I don't like it.

 b) custom-tool based: the custom processing tool I proposed could also
   parse inclusion/exclusion lists, and use these lists to move files
   around, in addition to processing the content of unmoved files.

   For example, if class X of the main version is not part of j2me,
   "process(j2me)" would move this file to a subdirectory ".streams/".

   If a class Y is not in the "main" version (the one used for "svn
   ci"), it resides in subdirectory ".streams" in the trunk.
   "process(j2me)" moves this file into the normal directory.

   As for IDEs, now you can configure them to automatically exclude
   ".stream/" directories.

   Inclusion/exclusion could be managed in two ways:
   1- the processing tool could look for inclusion/exclusion list files,
  such as "j2me.inclusion, j2me.exclusion, main.inclusion,
  main.exclusion, etc."

  This would lead to the best performance (for process(X)), yet it
  does require much manual update of inclusion/exclusion lists, with
  all the discrepancies that result from human mistakes while
  updating these files.

   2- (my preferred way), directives, at the beginning of the source
  code would indicate whether a file is included in a version or
  not.  Depending on target, the file would be moved to the
  appropriate directory (normal, ".streams").


Of course, there's also the problem of non-source files, i.e.
resources.  IMO, resources could be managed using specific
directories (".main/", ".j2me", ".j2se1.4") and a ".shared/"
directory with symbolic links in the specific directories.


As for source-level management, you would use my proposed processing
tool to view the source files with the right spectacles [as Tim said so
elegantly:-)].  For "development targets", it is important that:

 revert(process(X, target)) => X

By "development target" I mean a target that is meant for Harmony
developers in prevision of reverting "modified code" to a format
suitable for "svn ci" (i.e. revert to "main" target).

For comfortable IDE development, one could imagine that the IDE editor
can reduce to "one-line visible" comments (or better, specially
formatted ones) so that it gives you the impression that you are really
wearing target-specific spectacles.  [I know Eclipse allows for such
things already].

To release code, one would apply:

 process(X, release-target) => Y

Now, it is important to understand that Y, in this case, is NOT suitable
for doing any modification as

 revert(Y) => Kaboom!  (The tool will simply report that it can't do it;
it won't crash.)

Yet, I think that it would be important that the processing tool leaves
"markers" in Y, so that we could also have a tool to help finding the
canonical source line of a reported bug:

 revertLine(Y, L') => L  (where L' is a line reported by end-developer,
  and L the related line in "svn").

Markers would be short single lines comments, so the end-developer
annoyance would be kept minimal.


What do you think?


I am really offering to develop this custom tool.  Just help me identify
various Harmony needs I might have missed!

Of course, this tool is not the best solution to ALL problems, yet, so
far, I think that it seems t

RE: [classlib] Preprocessor - CHECKPOINT

[Fedotov, Alexei A]

Sorry for spam,
>I
>bet if you could convince some investor that it was a web2.0 thing, you
>could get it funded

I believe I know one investor - just convince G**gle to distribute AJAX
modules for his code search engine which would automatically insert
comments with code sensitive advertisement. 

With best regards,
Alexei Fedotov,
Intel Java & XML Engineering

>-Original Message-
>From: Geir Magnusson Jr. [mailto:[EMAIL PROTECTED]
>Sent: Wednesday, November 01, 2006 4:39 AM
>To: harmony-dev@incubator.apache.org
>Subject: Re: [classlib] Preprocessor - CHECKPOINT
>
>
>
>Etienne Gagnon wrote:
>> Tim Ellison wrote:
>>> IMO it's not ideal that the preprocessed source still contains all
the
>>> streams, albeit in comments.  It wouldn't make the source very
>>> 'consumable' to the Mrs. SE or ME developer.
>>
>> Hmmm...  It's always possible to have a special output mode that puts
>> empty (or advertizing, hehe) comments, instead of other stream code
>> (thus, preserving line numbers).
>>
>> To continue on my earlier example:
>>
>> Java source => "j2se end-developer"
>> ---
>>
>>
>> ...
>>
>> //  Download Harmony[tm] from
>> //
>> //  http://the.nice.harmony.url/download
>> //
>> //  :-)
>>
>>   @Processor(Not in j2me!)
>>   int some_field =
>> some +
>>   initializing()
>> code;
>>
>> ...
>>
>>
>> Or, more likely:
>>
>> Java source => "j2se end-developer"
>> ---
>>
>>
>> ...
>>
>> //  Please  ignore this comment.  It has been
>> //  intentionally left here to preserve line numbers
>> //  for bug reporting purpose.
>> //
>> //  Please report bugs to http://bugs.of.harmony.url/...
>>
>>   @Processor(Not in j2me!)
>>   int some_field =
>> some +
>>   initializing()
>> code;
>>
>> ...
>>
>>
>> So, J2ME & J2SE end-developers are kept happy.
>
>I'm still not quite getting the importance of preserving the line
>numbers like that if we have some minimal tooling to let us work with
it
>in eclipse or IDEA invisibly  users would report bug at line X, and
>we'd either look at the transformed code that they are actually using,
>and translate backwards, or have a plugin that lets us "A/B" between
>transformed and original.
>
>The key is to play with some examples, I guess.
>
>>
>> As a bonu$, you can al$o $tart a nice busine$$ $elling advertizing
$pace
>> in $ource code.  ;-P
>
>It's our idea - you run with it.  let us know how that works out.  (I
>bet if you could convince some investor that it was a web2.0 thing, you
>could get it funded)
>
>
>>
>> Etienne
>>

RE: [classlib] Preprocessor - CHECKPOINT

[Fedotov, Alexei A]

Etienne,
The example is quite interesting. But the idea of selling comment space
for advertising really rocks! :-)

With best regards,
Alexei Fedotov,
Intel Java & XML Engineering

>-Original Message-
>From: Etienne Gagnon [mailto:[EMAIL PROTECTED]
>Sent: Wednesday, November 01, 2006 2:25 AM
>To: harmony-dev@incubator.apache.org
>Subject: Re: [classlib] Preprocessor - CHECKPOINT
>
>Tim Ellison wrote:
>> IMO it's not ideal that the preprocessed source still contains all
the
>> streams, albeit in comments.  It wouldn't make the source very
>> 'consumable' to the Mrs. SE or ME developer.
>
>Hmmm...  It's always possible to have a special output mode that puts
>empty (or advertizing, hehe) comments, instead of other stream code
>(thus, preserving line numbers).
>
>To continue on my earlier example:
>
>Java source => "j2se end-developer"
>---
>
>
>...
>
>//  Download Harmony[tm] from
>//
>//  http://the.nice.harmony.url/download
>//
>//  :-)
>
>  @Processor(Not in j2me!)
>  int some_field =
>some +
>  initializing()
>code;
>
>...
>
>
>Or, more likely:
>
>Java source => "j2se end-developer"
>---
>
>
>...
>
>//  Please  ignore this comment.  It has been
>//  intentionally left here to preserve line numbers
>//  for bug reporting purpose.
>//
>//  Please report bugs to http://bugs.of.harmony.url/...
>
>  @Processor(Not in j2me!)
>  int some_field =
>some +
>  initializing()
>code;
>
>...
>
>
>So, J2ME & J2SE end-developers are kept happy.
>
>As a bonu$, you can al$o $tart a nice busine$$ $elling advertizing
$pace
>in $ource code.  ;-P
>
>Etienne
>
>--
>Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/
>SableVM:   http://www.sablevm.org/
>SableCC:   http://www.sablecc.org/

RE: [classlib] Preprocessor - CHECKPOINT

[Fedotov, Alexei A]

>Regardless, I think we need to settle on our exact requirement first,
>before spending too much time on looking for a solution.

+1
This exactly matches my morning metro thoughts. Nathan, thanks for
catching this point.

With best regards,
Alexei Fedotov,
Intel Java & XML Engineering

>-Original Message-
>From: Nathan Beyer [mailto:[EMAIL PROTECTED]
>Sent: Wednesday, November 01, 2006 4:06 AM
>To: harmony-dev@incubator.apache.org
>Subject: Re: [classlib] Preprocessor - CHECKPOINT
>
>What's the concern about just using the prescribed branching pattern
>for SVN? There are some other nice tricks like "externals" for pulling
>in common files into the working copies of other branches (ala the
>'concurrent' code in 'standard' that's pulled into 'enhanced' on
>checkout). I would propose we at least attempt to go down a path of
>investigating a branching.
>
>Regardless, I think we need to settle on our exact requirement first,
>before spending too much time on looking for a solution. For example,
>if logging is a real requirement, but everyone agrees it can be done
>via instrumentation (AspectJ, java.lang.instrument, etc), then are
>there any other requirements that affect the actual source files
>internally? If not, then could all of the other requirements be
>fulfilled by judicious SCM use?
>
>So, I would suggest we back up a little and just layout all of the
>requirements first, so we can make sure everyone's in agreement about
>the needs.
>
>-Nathan
>
>On 10/31/06, Geir Magnusson Jr. <[EMAIL PROTECTED]> wrote:
>> So we all agree that it's not an ideal solution.
>>
>> Can anyone think of anything else?  No one said this was going to be
>easy...
>>
>> geir
>>

Re: [classlib] Preprocessor - CHECKPOINT

[Geir Magnusson Jr.]

Tim Ellison wrote:

Geir Magnusson Jr. wrote:

Nathan Beyer wrote:

What's the concern about just using the prescribed branching pattern
for SVN? There are some other nice tricks like "externals" for pulling
in common files into the working copies of other branches (ala the
'concurrent' code in 'standard' that's pulled into 'enhanced' on
checkout). 

Even the authors of SVN warn people away from using externals.


Yeah, and a nightmare when trying to 'tag' code -- copying the link to
HEAD is no help.


I would propose we at least attempt to go down a path of
investigating a branching.

We should consider everything, but I'd personally rather keep as few
codelines as possible.


Agreed.


Regardless, I think we need to settle on our exact requirement first,
before spending too much time on looking for a solution. For example,
if logging is a real requirement, but everyone agrees it can be done
via instrumentation (AspectJ, java.lang.instrument, etc), then are
there any other requirements that affect the actual source files
internally? If not, then could all of the other requirements be
fulfilled by judicious SCM use?

So, I would suggest we back up a little and just layout all of the
requirements first, so we can make sure everyone's in agreement about
the needs.


Nathan is right -- this is hypothetical now, unless (for example) we
start on Java 6 development now.


Exactly - we need use cases (and it's not clear that the logging
problems have been resolved w/ aspects yet...)


You're joking, right?  I tease the aspect people that logging is the
only problem that has been solved(*) .  There are lots of references
on how to do that, eg:
  http://www.developer.com/java/other/article.php/3109831


There's caching too, I think.  LogCache4J

What I meant was that it didn't seem like we came to a conclusion on it 
- that if we had a general pre-processing solution, we could use that 
too for logging, rather than have two.


The actual use-cases will help figure this out.

geir





(*) it's not true though, there are a number of tasks that are
well-suited to using aspects.  However, I would use them judiciously.


Like caching :)  (And get your local psychic retainer to tell you what 
the code is doing... ;)



geir



Regards,
Tim

Re: [classlib] Preprocessor - CHECKPOINT

[Tim Ellison]

Etienne Gagnon wrote:
> Tim Ellison wrote:
>> IMO it's not ideal that the preprocessed source still contains all the
>> streams, albeit in comments.  It wouldn't make the source very
>> 'consumable' to the Mrs. SE or ME developer.
> 
> Hmmm...  It's always possible to have a special output mode that puts
> empty (or advertizing, hehe) comments, instead of other stream code
> (thus, preserving line numbers).

Right, but it was the presence of the comments, not the contents of the
comments that I was objecting to.  Using padding comments would be bad
too IMO.



> So, J2ME & J2SE end-developers are kept happy.

As I said before, I like the idea to be able to flip the comments
between different processor targets for the benefit of Harmony
developers.  So, for example, you can work with your SE spectacles on,
and I can work with my ME spectacles on, then we both commit to the
repository in a canonical form.  However, for the end-user (Mrs Java
developer) the processor would strip out the irrelevant streams' code
from the canonical form to produce a clean target source code.

> As a bonu$, you can al$o $tart a nice busine$$ $elling advertizing $pace
> in $ource code.  ;-P

LOL -- hey, now you are talking ;-)

Regards,
Tim

-- 

Tim Ellison ([EMAIL PROTECTED])

Re: [classlib] Preprocessor - CHECKPOINT

[Tim Ellison]

Geir Magnusson Jr. wrote:
> Nathan Beyer wrote:
>> What's the concern about just using the prescribed branching pattern
>> for SVN? There are some other nice tricks like "externals" for pulling
>> in common files into the working copies of other branches (ala the
>> 'concurrent' code in 'standard' that's pulled into 'enhanced' on
>> checkout). 
> 
> Even the authors of SVN warn people away from using externals.

Yeah, and a nightmare when trying to 'tag' code -- copying the link to
HEAD is no help.

>> I would propose we at least attempt to go down a path of
>> investigating a branching.
> 
> We should consider everything, but I'd personally rather keep as few
> codelines as possible.

Agreed.

>> Regardless, I think we need to settle on our exact requirement first,
>> before spending too much time on looking for a solution. For example,
>> if logging is a real requirement, but everyone agrees it can be done
>> via instrumentation (AspectJ, java.lang.instrument, etc), then are
>> there any other requirements that affect the actual source files
>> internally? If not, then could all of the other requirements be
>> fulfilled by judicious SCM use?
>>
>> So, I would suggest we back up a little and just layout all of the
>> requirements first, so we can make sure everyone's in agreement about
>> the needs.

Nathan is right -- this is hypothetical now, unless (for example) we
start on Java 6 development now.

> Exactly - we need use cases (and it's not clear that the logging
> problems have been resolved w/ aspects yet...)

You're joking, right?  I tease the aspect people that logging is the
only problem that has been solved(*) .  There are lots of references
on how to do that, eg:
  http://www.developer.com/java/other/article.php/3109831


(*) it's not true though, there are a number of tasks that are
well-suited to using aspects.  However, I would use them judiciously.

Regards,
Tim

-- 

Tim Ellison ([EMAIL PROTECTED])

Re: [classlib] Preprocessor - CHECKPOINT

[Geir Magnusson Jr.]

Etienne Gagnon wrote:

Tim Ellison wrote:

IMO it's not ideal that the preprocessed source still contains all the
streams, albeit in comments.  It wouldn't make the source very
'consumable' to the Mrs. SE or ME developer.


Hmmm...  It's always possible to have a special output mode that puts
empty (or advertizing, hehe) comments, instead of other stream code
(thus, preserving line numbers).

To continue on my earlier example:

Java source => "j2se end-developer"
---


...

//  Download Harmony[tm] from
//
//  http://the.nice.harmony.url/download
//
//  :-)

  @Processor(Not in j2me!)
  int some_field =
some +
  initializing()
code;

...


Or, more likely:

Java source => "j2se end-developer"
---


...

//  Please  ignore this comment.  It has been
//  intentionally left here to preserve line numbers
//  for bug reporting purpose.
//
//  Please report bugs to http://bugs.of.harmony.url/...

  @Processor(Not in j2me!)
  int some_field =
some +
  initializing()
code;

...


So, J2ME & J2SE end-developers are kept happy.


I'm still not quite getting the importance of preserving the line 
numbers like that if we have some minimal tooling to let us work with it 
in eclipse or IDEA invisibly  users would report bug at line X, and 
we'd either look at the transformed code that they are actually using, 
and translate backwards, or have a plugin that lets us "A/B" between 
transformed and original.


The key is to play with some examples, I guess.



As a bonu$, you can al$o $tart a nice busine$$ $elling advertizing $pace
in $ource code.  ;-P


It's our idea - you run with it.  let us know how that works out.  (I 
bet if you could convince some investor that it was a web2.0 thing, you 
could get it funded)





Etienne

Re: [classlib] Preprocessor - CHECKPOINT

[Geir Magnusson Jr.]

Nathan Beyer wrote:

What's the concern about just using the prescribed branching pattern
for SVN? There are some other nice tricks like "externals" for pulling
in common files into the working copies of other branches (ala the
'concurrent' code in 'standard' that's pulled into 'enhanced' on
checkout). 


Even the authors of SVN warn people away from using externals.


I would propose we at least attempt to go down a path of
investigating a branching.


We should consider everything, but I'd personally rather keep as few 
codelines as possible.




Regardless, I think we need to settle on our exact requirement first,
before spending too much time on looking for a solution. For example,
if logging is a real requirement, but everyone agrees it can be done
via instrumentation (AspectJ, java.lang.instrument, etc), then are
there any other requirements that affect the actual source files
internally? If not, then could all of the other requirements be
fulfilled by judicious SCM use?

So, I would suggest we back up a little and just layout all of the
requirements first, so we can make sure everyone's in agreement about
the needs.


Exactly - we need use cases (and it's not clear that the logging 
problems have been resolved w/ aspects yet...)


geir



-Nathan

On 10/31/06, Geir Magnusson Jr. <[EMAIL PROTECTED]> wrote:

So we all agree that it's not an ideal solution.

Can anyone think of anything else?  No one said this was going to be 
easy...


geir

Re: [classlib] Preprocessor - CHECKPOINT

[Nathan Beyer]

What's the concern about just using the prescribed branching pattern
for SVN? There are some other nice tricks like "externals" for pulling
in common files into the working copies of other branches (ala the
'concurrent' code in 'standard' that's pulled into 'enhanced' on
checkout). I would propose we at least attempt to go down a path of
investigating a branching.

Regardless, I think we need to settle on our exact requirement first,
before spending too much time on looking for a solution. For example,
if logging is a real requirement, but everyone agrees it can be done
via instrumentation (AspectJ, java.lang.instrument, etc), then are
there any other requirements that affect the actual source files
internally? If not, then could all of the other requirements be
fulfilled by judicious SCM use?

So, I would suggest we back up a little and just layout all of the
requirements first, so we can make sure everyone's in agreement about
the needs.

-Nathan

On 10/31/06, Geir Magnusson Jr. <[EMAIL PROTECTED]> wrote:

So we all agree that it's not an ideal solution.

Can anyone think of anything else?  No one said this was going to be easy...

geir

Re: [classlib] Preprocessor - CHECKPOINT

[Etienne Gagnon]

Tim Ellison wrote:
> IMO it's not ideal that the preprocessed source still contains all the
> streams, albeit in comments.  It wouldn't make the source very
> 'consumable' to the Mrs. SE or ME developer.

Hmmm...  It's always possible to have a special output mode that puts
empty (or advertizing, hehe) comments, instead of other stream code
(thus, preserving line numbers).

To continue on my earlier example:

Java source => "j2se end-developer"
---


...

//  Download Harmony[tm] from
//
//  http://the.nice.harmony.url/download
//
//  :-)

  @Processor(Not in j2me!)
  int some_field =
some +
  initializing()
code;

...


Or, more likely:

Java source => "j2se end-developer"
---


...

//  Please  ignore this comment.  It has been
//  intentionally left here to preserve line numbers
//  for bug reporting purpose.
//
//  Please report bugs to http://bugs.of.harmony.url/...

  @Processor(Not in j2me!)
  int some_field =
some +
  initializing()
code;

...


So, J2ME & J2SE end-developers are kept happy.

As a bonu$, you can al$o $tart a nice busine$$ $elling advertizing $pace
in $ource code.  ;-P

Etienne

-- 
Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/
SableVM:   http://www.sablevm.org/
SableCC:   http://www.sablecc.org/


signature.asc
Description: OpenPGP digital signature

Re: [classlib] Preprocessor - CHECKPOINT

[Tim Ellison]

Etienne Gagnon wrote:
> Here's an idea.


IMO it's not ideal that the preprocessed source still contains all the
streams, albeit in comments.  It wouldn't make the source very
'consumable' to the Mrs. SE or ME developer.

The ability for classlib developers to do the switching would be good
though.  At least we could talk between ourselves in common terms.

Regards,
Tim

-- 

Tim Ellison ([EMAIL PROTECTED])

Re: [classlib] Preprocessor - CHECKPOINT

[Etienne Gagnon]

[WARNING:  Long!]

Here's an idea.

It would be quite easy (for me, at least ;-) ) to develop a
"line-preserving" AND "revertible" processing tool so that:

1- source + javac => j2se

2- source + processor(target) + javac => target,  where target is a
   parameter (j2me, etc.)

3- source + processor(target) => source for target, where line numbers
are identical to source

4- source for target + reverseprocessor(target) -> source :-)  That's
the thing lacking in most preprocessors, as often you would like to
modify the processed source and propagate these changes back to the
initial source; this is usually a pain as it must be done manually.  Not
here!


I am, usually, in favor of using branches, for development, yet, in the
current case, I think that branches have their drawbacks:

1- I have yet to see a branch/merge system that deals well with major
code reorganization in a single file.

2- There's always the infrequent, yet very annoying (if not dangerous
when it happens) semantically invalid merge result.

3- Developers of the main branch will remain mostly unaware of the
parallel development on other branches.  In this case, this can be a
problem, as having j2se developers aware that they are changing lines
for which other branches maintain distinct versions ones might be a good
"communication tool".


The idea, for line preservation, would be to either use "revertible"
comments (easiest) or maybe play with special Java annotations, if they
prove sufficient for the task.

The advantage of "intelligent" Java-syntax-aware processing (instead of
dumb preprocessing) is that the processor could work intelligently.

Here's a tentative example to illustrate my idea.  (This is not a fully
thought-up example/syntax; it's just a way of trying to explain the
concept):


Java source  (j2se)
---


...

//  @Processor(add to j2me
//
//  float some_field;
//
//  )

  @Processor(Not in j2me!)
  int some_field =
some +
  initializing()
code;

...


Processed source  (target => j2me)
--


...

  @BeginRevert(add to j2me)

  float some_field;

  @EndRevert()

//  @Revert(@Processor(Not in j2me!)
//  int some_field =
//some +
//  initializing()
//code; )

...


Modified processed source ;-)
-

...

  @BeginRevert(add to j2me)

  float some_field = missing initializer!;

  @EndRevert()

//  @Revert(@Processor(Not in j2me!)
//  int some_field =
//some +
//  initializing()
//code; )

...


Revert-processed source (=> ready for "svn ci")
---

...

//  @Processor(add to j2me)
//
//  float some_field = missing initializer!;
//
//  )

  @Processor(Not in j2me!)
  int some_field =
some +
  initializing()
code;

...



The "intelligent" part comes with the idea that the @Processor
directive, in the j2se uncommented source above, "knows" that it applies
to the whole field declaration, including any initializer, regardless of
the number of lines used by the field declaration.  Such directives
could be "limited" to work on:
1- classes/interface declarations
2- method/field declarations
3- static statement blocks
4- statement blocks in general
5- other things?

The idea is that it is not a "character/lined based" processor, but
really a syntax-driven one.

Using comments (as above) is optional, yet I recommend doing it so that
text editors don't syntax color unapplied code in any current situation
(source, processed-source).

Both j2se & j2me (as well as other target developers) will be able to
work on the "processed" and the original source code.  Typically, the
revert processor will be used by j2me developers before "svn ci" of
their modifications.


It would be quite easy for me to develop such a tool using SableCC.
FYI, the output of SableCC is not encumbered by any copyright other than
whatever the "grammar" developer wants.  Also, I will contribute the
developed tool to Harmony if this approach is chosen (or develop it as a
Harmony developer, if you prefer...).
[Geir: Yes, I know, I have to send the paperwork...]

Some advantages:

1- Even if you package "processed source" into a jar file (for j2me
users), line numbers match exactly.  Bug reports from users won't be
about Mars or Jupiter. :-)

2- Debugging using both initial and processed source works flawlessly.

3- Major code reorganization can be done cleanly without the need for
bug-prone, tricky, hand-made merges (as automated merge can be
misleading in such case)

4- We can make the tool and its syntax as "gentle" as you want; we have
total control on its syntax and semantics. :-)

5- We could add a "syntax checking" mode to the processor, so that in
that mode, *all* code would be checked for syntax correctness (even
commented processor directives)  instead of modified for a specific
target.  This could be added to the ant script for all developers, so
that at no point somebody "svn ci" synta

Re: [classlib] Preprocessor - CHECKPOINT

[Geir Magnusson Jr.]

So we all agree that it's not an ideal solution.

Can anyone think of anything else?  No one said this was going to be easy...

geir