Re: [PATCH 07 of 10 V5] rcutil: let environ override system configs (BC)

[Jun Wu]

Excerpts from Matt Harbison's message of 2017-04-07 23:14:20 -0400:
> That did the trick, thanks.  Out of curiosity, how did you figure you the  
> problem from that error message?  xargs would have been the last thing I  
> would have suspected.

I didn't figure it out from the error message - that's why I asked for your
environment to be able to reproduce.

The test is one big command piped together. I rewrote the pipe to use
temporary files. "hg locate" output differs - a couple of files were added.
Running check-code.py on added files showed no problem. Then I started to
suspect argv length limit.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 07 of 10 V5] rcutil: let environ override system configs (BC)

[Matt Harbison]

On Fri, 07 Apr 2017 01:13:20 -0400, Jun Wu  wrote:


Excerpts from Matt Harbison's message of 2017-04-07 00:35:42 -0400:

> Just to confirm: if you run "make local", it will say "cl.exe ..."
> instead
> of "gcc ..." (where gcc could come from MinGW).

Yep.

$ make local | grep cl.exe
c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\cl.exe  
/c

/nologo /Ox /MD /W3 /GS- /DNDEBUG -Ic:\Python27\include -Ic:\Python27\PC
/Tcmercurial/bdiff.c
/Fobuild\temp.win-amd64-2.7\Release\mercurial/bdiff.obj
...

> And you are using Python for
> Windows, not some cygwin or mingw Python.

Correct.  2.7.13, from the python.org Windows installer.

> And the shell is provided by MinGW.

I've never had cygwin or anything else installed.  The titlebar says
"MINGW32", and $SHELL is /bin/sh.  Not sure if there's something  
specific

I should check here.


Thanks. I have sent a fix.


That did the trick, thanks.  Out of curiosity, how did you figure you the  
problem from that error message?  xargs would have been the last thing I  
would have suspected.

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 07 of 10 V5] rcutil: let environ override system configs (BC)

[Jun Wu]

Excerpts from Matt Harbison's message of 2017-04-07 00:35:42 -0400:
> > Just to confirm: if you run "make local", it will say "cl.exe ..."  
> > instead
> > of "gcc ..." (where gcc could come from MinGW).
> 
> Yep.
> 
> $ make local | grep cl.exe
> c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\cl.exe /c  
> /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ic:\Python27\include -Ic:\Python27\PC  
> /Tcmercurial/bdiff.c  
> /Fobuild\temp.win-amd64-2.7\Release\mercurial/bdiff.obj
> ...
> 
> > And you are using Python for
> > Windows, not some cygwin or mingw Python.
> 
> Correct.  2.7.13, from the python.org Windows installer.
> 
> > And the shell is provided by MinGW.
> 
> I've never had cygwin or anything else installed.  The titlebar says  
> "MINGW32", and $SHELL is /bin/sh.  Not sure if there's something specific  
> I should check here.

Thanks. I have sent a fix.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 07 of 10 V5] rcutil: let environ override system configs (BC)

[Matt Harbison]

On Fri, 07 Apr 2017 00:25:03 -0400, Jun Wu  wrote:


Excerpts from Matt Harbison's message of 2017-04-07 00:09:54 -0400:

On Thu, 06 Apr 2017 23:50:31 -0400, Jun Wu  wrote:

> Excerpts from Matt Harbison's message of 2017-04-06 23:14:50 -0400:
>> On Mon, 27 Mar 2017 02:02:06 -0400, Jun Wu  wrote:
>>
>> > # HG changeset patch
>> > # User Jun Wu 
>> > # Date 1490589217 25200
>> > #  Sun Mar 26 21:33:37 2017 -0700
>> > # Node ID 38572bb2cffd815526a727bc6f3aacdca2902f4f
>> > # Parent  9b0aa30bf151b6c0e999b017fd328e29440bd447
>> > # Available At https://bitbucket.org/quark-zju/hg-draft
>> > #  hg pull https://bitbucket.org/quark-zju/hg-draft
-r

>> > 38572bb2cffd
>> > rcutil: let environ override system configs (BC)
>> >
>> > This is BC because system configs won't be able to override  
$EDITOR,

>> > $PAGER.
>> > The new behavior is arguably more rational.
>>
>> I'm not sure if this is a broader concern than just the test suite  
with

>> env-based config, but I bisected a new Windows failure back to this
>> change.
>
> How did you run tests under Windows? I couldn't reproduce this using
> cygwin.
> (and I have difficulty with "sh" when using the MSVC compiler)

I have MinGW installed, and from the tests directory just run  
"./run-tests
--local test-check-code.t".  I have MSVC 2008 installed, which I assume  
is

what is used.


Just to confirm: if you run "make local", it will say "cl.exe ..."  
instead

of "gcc ..." (where gcc could come from MinGW).


Yep.

$ make local | grep cl.exe
c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\cl.exe /c  
/nologo /Ox /MD /W3 /GS- /DNDEBUG -Ic:\Python27\include -Ic:\Python27\PC  
/Tcmercurial/bdiff.c  
/Fobuild\temp.win-amd64-2.7\Release\mercurial/bdiff.obj

...


And you are using Python for
Windows, not some cygwin or mingw Python.


Correct.  2.7.13, from the python.org Windows installer.


And the shell is provided by MinGW.


I've never had cygwin or anything else installed.  The titlebar says  
"MINGW32", and $SHELL is /bin/sh.  Not sure if there's something specific  
I should check here.




>>
>> --- c:/Users/Matt/Projects/hg/tests/test-check-code.t
>> +++ c:/Users/Matt/Projects/hg/tests/test-check-code.t.err
>> @@ -9,34 +9,7 @@
>>
>> $ hg locate -X contrib/python-zstandard -X
>> hgext/fsmonitor/pywatchman |
>> > sed 's-\\-/-g' | xargs "$check_code" --warnings --per-file=0 ||
>> false
>> -  Skipping i18n/polib.py it has no-che?k-code (glob)
>> -  mercurial/demandimport.py:312:
>> -   > if os.environ.get('HGDEMANDIMPORT') != 'disable':
>> -   use encoding.environ instead (py3)
>> -  mercurial/encoding.py:54:
>> -   > environ = os.environ
>> -   use encoding.environ instead (py3)
>> -  mercurial/encoding.py:56:
>> -   > environ = os.environb
>> -   use encoding.environ instead (py3)
>> -  mercurial/encoding.py:61:
>> -   >for k, v in os.environ.items())
>> -   use encoding.environ instead (py3)
>> -  mercurial/encoding.py:221:
>> -   >for k, v in os.environ.items())
>> -   use encoding.environ instead (py3)
>> -  Skipping mercurial/httpclient/__init__.py it has no-che?k-code  
(glob)
>> -  Skipping mercurial/httpclient/_readers.py it has no-che?k-code  
(glob)

>> -  mercurial/policy.py:46:
>> -   > if 'HGMODULEPOLICY' in os.environ:
>> -   use encoding.environ instead (py3)
>> -  mercurial/policy.py:47:
>> -   > policy = os.environ['HGMODULEPOLICY'].encode('utf-8')
>> -   use encoding.environ instead (py3)
>> -  mercurial/policy.py:49:
>> -   > policy = os.environ.get('HGMODULEPOLICY', policy)
>> -   use encoding.environ instead (py3)
>> -  Skipping mercurial/statprof.py it has no-che?k-code (glob)
>> +  /usr/bin/env: python: Bad file number
>> [1]
>>
>>   @commands in debugcommands.py should be in alphabetical order.
>>
>> > diff --git a/mercurial/commands.py b/mercurial/commands.py
>> > --- a/mercurial/commands.py
>> > +++ b/mercurial/commands.py
>> > @@ -1807,4 +1807,6 @@ def config(ui, repo, *values, **opts):
>> >  if t == 'path':
>> >  ui.debug('read config from: %s\n' % f)
>> > +elif t == 'items':
>> > +pass
>> >  else:
>> >  raise error.ProgrammingError('unknown rctype: %s' % t)
>> > diff --git a/mercurial/rcutil.py b/mercurial/rcutil.py
>> > --- a/mercurial/rcutil.py
>> > +++ b/mercurial/rcutil.py
>> > @@ -77,8 +77,12 @@ def rccomponents():
>> >  name, value, source) that should fill the config directly.
>> >  '''
>> > +envrc = ('items', envrcitems())
>> > +
>> >  global _rccomponents
>> >  if _rccomponents is None:
>> >  if 'HGRCPATH' in encoding.environ:
>> > -_rccomponents = []
>> > +# assume HGRCPATH is all about user configs so
>> environments
>> > can be
>> > +# overridden.
>> > +_rccomponents = [envrc]
>> >  for p in
>> > 

Re: [PATCH 07 of 10 V5] rcutil: let environ override system configs (BC)

[Jun Wu]

Excerpts from Matt Harbison's message of 2017-04-07 00:09:54 -0400:
> On Thu, 06 Apr 2017 23:50:31 -0400, Jun Wu  wrote:
> 
> > Excerpts from Matt Harbison's message of 2017-04-06 23:14:50 -0400:
> >> On Mon, 27 Mar 2017 02:02:06 -0400, Jun Wu  wrote:
> >>
> >> > # HG changeset patch
> >> > # User Jun Wu 
> >> > # Date 1490589217 25200
> >> > #  Sun Mar 26 21:33:37 2017 -0700
> >> > # Node ID 38572bb2cffd815526a727bc6f3aacdca2902f4f
> >> > # Parent  9b0aa30bf151b6c0e999b017fd328e29440bd447
> >> > # Available At https://bitbucket.org/quark-zju/hg-draft 
> >> > #  hg pull https://bitbucket.org/quark-zju/hg-draft   -r
> >> > 38572bb2cffd
> >> > rcutil: let environ override system configs (BC)
> >> >
> >> > This is BC because system configs won't be able to override $EDITOR,
> >> > $PAGER.
> >> > The new behavior is arguably more rational.
> >>
> >> I'm not sure if this is a broader concern than just the test suite with
> >> env-based config, but I bisected a new Windows failure back to this  
> >> change.
> >
> > How did you run tests under Windows? I couldn't reproduce this using  
> > cygwin.
> > (and I have difficulty with "sh" when using the MSVC compiler)
> 
> I have MinGW installed, and from the tests directory just run "./run-tests  
> --local test-check-code.t".  I have MSVC 2008 installed, which I assume is  
> what is used.

Just to confirm: if you run "make local", it will say "cl.exe ..." instead
of "gcc ..." (where gcc could come from MinGW). And you are using Python for
Windows, not some cygwin or mingw Python. And the shell is provided by
MinGW.

> 
> >>
> >> --- c:/Users/Matt/Projects/hg/tests/test-check-code.t
> >> +++ c:/Users/Matt/Projects/hg/tests/test-check-code.t.err
> >> @@ -9,34 +9,7 @@
> >>
> >> $ hg locate -X contrib/python-zstandard -X  
> >> hgext/fsmonitor/pywatchman |
> >> > sed 's-\\-/-g' | xargs "$check_code" --warnings --per-file=0 ||  
> >> false
> >> -  Skipping i18n/polib.py it has no-che?k-code (glob)
> >> -  mercurial/demandimport.py:312:
> >> -   > if os.environ.get('HGDEMANDIMPORT') != 'disable':
> >> -   use encoding.environ instead (py3)
> >> -  mercurial/encoding.py:54:
> >> -   > environ = os.environ
> >> -   use encoding.environ instead (py3)
> >> -  mercurial/encoding.py:56:
> >> -   > environ = os.environb
> >> -   use encoding.environ instead (py3)
> >> -  mercurial/encoding.py:61:
> >> -   >for k, v in os.environ.items())
> >> -   use encoding.environ instead (py3)
> >> -  mercurial/encoding.py:221:
> >> -   >for k, v in os.environ.items())
> >> -   use encoding.environ instead (py3)
> >> -  Skipping mercurial/httpclient/__init__.py it has no-che?k-code (glob)
> >> -  Skipping mercurial/httpclient/_readers.py it has no-che?k-code (glob)
> >> -  mercurial/policy.py:46:
> >> -   > if 'HGMODULEPOLICY' in os.environ:
> >> -   use encoding.environ instead (py3)
> >> -  mercurial/policy.py:47:
> >> -   > policy = os.environ['HGMODULEPOLICY'].encode('utf-8')
> >> -   use encoding.environ instead (py3)
> >> -  mercurial/policy.py:49:
> >> -   > policy = os.environ.get('HGMODULEPOLICY', policy)
> >> -   use encoding.environ instead (py3)
> >> -  Skipping mercurial/statprof.py it has no-che?k-code (glob)
> >> +  /usr/bin/env: python: Bad file number
> >> [1]
> >>
> >>   @commands in debugcommands.py should be in alphabetical order.
> >>
> >> > diff --git a/mercurial/commands.py b/mercurial/commands.py
> >> > --- a/mercurial/commands.py
> >> > +++ b/mercurial/commands.py
> >> > @@ -1807,4 +1807,6 @@ def config(ui, repo, *values, **opts):
> >> >  if t == 'path':
> >> >  ui.debug('read config from: %s\n' % f)
> >> > +elif t == 'items':
> >> > +pass
> >> >  else:
> >> >  raise error.ProgrammingError('unknown rctype: %s' % t)
> >> > diff --git a/mercurial/rcutil.py b/mercurial/rcutil.py
> >> > --- a/mercurial/rcutil.py
> >> > +++ b/mercurial/rcutil.py
> >> > @@ -77,8 +77,12 @@ def rccomponents():
> >> >  name, value, source) that should fill the config directly.
> >> >  '''
> >> > +envrc = ('items', envrcitems())
> >> > +
> >> >  global _rccomponents
> >> >  if _rccomponents is None:
> >> >  if 'HGRCPATH' in encoding.environ:
> >> > -_rccomponents = []
> >> > +# assume HGRCPATH is all about user configs so  
> >> environments
> >> > can be
> >> > +# overridden.
> >> > +_rccomponents = [envrc]
> >> >  for p in
> >> > encoding.environ['HGRCPATH'].split(pycompat.ospathsep):
> >> >  if not p:
> >> > @@ -86,5 +90,8 @@ def rccomponents():
> >> >  _rccomponents.extend(('path', p) for p in
> >> > _expandrcpath(p))
> >> >  else:
> >> > -paths = defaultrcpath() + systemrcpath() + userrcpath()
> >> > +paths = defaultrcpath() + systemrcpath()

Re: [PATCH 07 of 10 V5] rcutil: let environ override system configs (BC)

[Jun Wu]

Excerpts from Yuya Nishihara's message of 2017-03-28 21:53:31 +0900:
> And we'll need to avoid caching of envrcitems because of chg, could be fixed
> by followup.

Good catch. I think that also applies to config files - listing a directory
cannot be cached either. So it seems the bug has been there since the
beginning.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 07 of 10 V5] rcutil: let environ override system configs (BC)

[Jun Wu]

Excerpts from Ryan McElroy's message of 2017-03-28 09:47:18 +0100:
> On 3/27/17 7:02 AM, Jun Wu wrote:
> > # HG changeset patch
> > # User Jun Wu 
> > # Date 1490589217 25200
> > #  Sun Mar 26 21:33:37 2017 -0700
> > # Node ID 38572bb2cffd815526a727bc6f3aacdca2902f4f
> > # Parent  9b0aa30bf151b6c0e999b017fd328e29440bd447
> > rcutil: let environ override system configs (BC)
> 
> Small nits inline. Could easily be addressed by a follow-up.
> 
> >
> > This is BC because system configs won't be able to override $EDITOR, $PAGER.
> > The new behavior is arguably more rational.
> >
> > diff --git a/mercurial/commands.py b/mercurial/commands.py
> > --- a/mercurial/commands.py
> > +++ b/mercurial/commands.py
> > @@ -1807,4 +1807,6 @@ def config(ui, repo, *values, **opts):
> >   if t == 'path':
> >   ui.debug('read config from: %s\n' % f)
> > +elif t == 'items':
> > +pass
> >   else:
> >   raise error.ProgrammingError('unknown rctype: %s' % t)
> > diff --git a/mercurial/rcutil.py b/mercurial/rcutil.py
> > --- a/mercurial/rcutil.py
> > +++ b/mercurial/rcutil.py
> > @@ -77,8 +77,12 @@ def rccomponents():
> >   name, value, source) that should fill the config directly.
> >   '''
> > +envrc = ('items', envrcitems())
> > +
> >   global _rccomponents
> >   if _rccomponents is None:
> >   if 'HGRCPATH' in encoding.environ:
> > -_rccomponents = []
> > +# assume HGRCPATH is all about user configs so environments 
> > can be
> > +# overridden.
> > +_rccomponents = [envrc]
> >   for p in 
> > encoding.environ['HGRCPATH'].split(pycompat.ospathsep):
> >   if not p:
> > @@ -86,5 +90,8 @@ def rccomponents():
> >   _rccomponents.extend(('path', p) for p in 
> > _expandrcpath(p))
> >   else:
> > -paths = defaultrcpath() + systemrcpath() + userrcpath()
> > +paths = defaultrcpath() + systemrcpath()
> >   _rccomponents = [('path', os.path.normpath(p)) for p in paths]
> > +_rccomponents.append(envrc)
> > +paths = userrcpath()
> > +_rccomponents.extend(('path', os.path.normpath(p)) for p in 
> > paths)
> 
> This line is essentially repeated from above... I think it could be 
> factored out in a future clean-up.

I didn't do that because that makes the code at least 2 lines longer.

> 
> >   return _rccomponents
> > diff --git a/mercurial/ui.py b/mercurial/ui.py
> > --- a/mercurial/ui.py
> > +++ b/mercurial/ui.py
> > @@ -212,8 +212,18 @@ class ui(object):
> >   """Create a ui and load global and user configs"""
> >   u = cls()
> > -# we always trust global config files
> > +# we always trust global config files and environment variables
> >   for t, f in rcutil.rccomponents():
> >   if t == 'path':
> >   u.readconfig(f, trust=True)
> > +elif t == 'items':
> > +sections = set()
> > +for section, name, value, source in f:
> > +# do not set u._ocfg
> 
> This comment doesn't explain why, so it's not very helpful.

It will disappear with immutable config.

> 
> > +# XXX clean this up once immutable config object is a 
> > thing
> > +u._tcfg.set(section, name, value, source)
> > +u._ucfg.set(section, name, value, source)
> > +sections.add(section)
> > +for section in sections:
> > +u.fixconfig(section=section)
> >   else:
> >   raise error.ProgrammingError('unknown rctype: %s' % t)
> > diff --git a/tests/test-config-env.py b/tests/test-config-env.py
> > new file mode 100644
> > --- /dev/null
> > +++ b/tests/test-config-env.py
> > @@ -0,0 +1,48 @@
> > +# Test the config layer generated by environment variables
> > +
> > +from __future__ import absolute_import, print_function
> > +
> > +import os
> > +
> > +from mercurial import (
> > +encoding,
> > +rcutil,
> > +ui as uimod,
> > +)
> > +
> > +testtmp = encoding.environ['TESTTMP']
> > +
> > +# prepare hgrc files
> > +def join(name):
> > +return os.path.join(testtmp, name)
> > +
> > +with open(join('sysrc'), 'w') as f:
> > +f.write('[ui]\neditor=e0\n[pager]\npager=p0\n')
> > +
> > +with open(join('userrc'), 'w') as f:
> > +f.write('[ui]\neditor=e1')
> > +
> > +# replace rcpath functions so they point to the files above
> > +def systemrcpath():
> > +return [join('sysrc')]
> > +
> > +def userrcpath():
> > +return [join('userrc')]
> > +
> > +rcutil.systemrcpath = systemrcpath
> > +rcutil.userrcpath = userrcpath
> > +os.path.isdir = lambda x: False # hack: do not load default.d/*.rc
> > +
> > +# utility to print configs
> > +def printconfigs(env):
> > +encoding.environ = env
> > +rcutil._rccomponents = None #