Re: [OE-core] [PATCH 2/5] devtool: add configure-help subcommand

2016-01-25 Thread Paul Eggleton 
On Tue, 26 Jan 2016 17:59:53 Paul Eggleton wrote:
> On Mon, 25 Jan 2016 22:27:27 Khem Raj wrote:
> > On Mon, Jan 25, 2016 at 10:21 PM, Paul Eggleton
> > 
> >  wrote:
> > > On Tue, 26 Jan 2016 16:17:39 Paul Eggleton wrote:
> > >> Hi Khem,
> > >> 
> > >> On Mon, 25 Jan 2016 22:14:09 Khem Raj wrote:
> > >> > > On Jan 25, 2016, at 9:53 PM, Paul Eggleton
> > >> > > 
> > >> > > wrote:
> > >> > > 
> > >> > > When you need to set EXTRA_OECONF for a recipe, you need to know
> > >> > > what
> > >> > > options the configure script actually supports; the configure
> > >> > > script
> > >> > > however is only accessible from within a devshell and (at least in
> > >> > > the
> > >> > > case of autotooled software fetched from an SCM repository) may not
> > >> > > actually exist until do_configure has run. Thus, provide a "devtool
> > >> > > configure-help" subcommand that runs the configure script for a
> > >> > > recipe
> > >> > > with --help and shows you the output through a pager (e.g. less),
> > >> > > prefaced by a header describing the current options being
> > >> > > specified.
> > >> > > 
> > >> > > There is basic support for autotools, cmake and bare configure
> > >> > > scripts.
> > >> > > The cmake support is a little hacky since cmake doesn't really have
> > >> > > a
> > >> > > concise help option that lists user-defined knobs (without actually
> > >> > > running through the configure process), however that being a design
> > >> > > feature of cmake there's not much I can think of to do about that
> > >> > > at
> > >> > > the moment.
> > >> > 
> > >> > this option is autotools specific. We need to convey this precisely.
> > >> > may
> > >> > be
> > >> > involve autotools in the option name or something
> > >> 
> > >> Whilst not all non-autoconf configure scripts provide a --help option,
> > >> some
> > >> do. ffmpeg/libav is one example (and one where you really do need to
> > >> see
> > >> the output since there are a lot of options...)
> > > 
> > > I meant to add, even if configure --help doesn't work (or there is no
> > > configure script at all) the header this subcommand prints will work
> > > with
> > > any recipe provided that do_configure is actually implemented and
> > > contains some actual commands as opposed to calls to some other
> > > function.
> > 
> > my worry is it being associated to do_configure, and we know that only
> > a subset ( although a large one) is only using autotools but not all.
> > So think about avoiding that confusion.
> 
> I'm not terribly convinced there could be much confusion. FWIW, I just
> looked through the other recipes we have in OE-Core that have a call to a
> configure script  that don't use autoconf (man, zlib, x264, v86d, ed,
> libpostproc); the configure scripts for all of these at least produce some
> useful output when called with --help. I then took a look at meta-oe (the
> layer)

Actually this ended up being meta-oe, meta-networking, and meta-multimedia by 
the time I was finished.

> , and checked the following recipes:
> 
> * concurrencykit - works
> * ddrescue - works
> * fio - works
> * libvpx - works
> * lzip - works
> * nodejs - works
> * mg - works (no options, but message is sensible)
> * terminus-font - configure script is supplied without the execute bit set
> but does support --help
> * iscsi-initiator-utils - configure script is in a subdir so isn't found,
> but does support --help
> * netkit-rsh - works
> * netkit-rpc - works
> * netkit-telnet - works
> * netkit-tftp - works
> * netkit-rusers - works
> * netkit-rwho - works
> * netmap - configure script is in a subdir so isn't found, but does support
>   --help
> * ntimed - configure script is dumb and takes no options at all
> * tvheadend - works
> * mozjs - seems to break devtool modify, but configure script does support
>   --help
> 
> So there are some minor bugs to fix, but of those recipes I tested there is
> a grand total of 1 script that doesn't support --help. Honestly I think if
> a configure script exists it's reasonable to assume that
> --help is going to print something useful; even if it doesn't there
> shouldn't be any major harm and you're still going to get a printout of the
> do_configure value in which you'll be able to see what options are
> currently passed.
> 
> Cheers,
> Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 2/5] devtool: add configure-help subcommand

2016-01-25 Thread Paul Eggleton 
On Mon, 25 Jan 2016 22:27:27 Khem Raj wrote:
> On Mon, Jan 25, 2016 at 10:21 PM, Paul Eggleton
> 
>  wrote:
> > On Tue, 26 Jan 2016 16:17:39 Paul Eggleton wrote:
> >> Hi Khem,
> >> 
> >> On Mon, 25 Jan 2016 22:14:09 Khem Raj wrote:
> >> > > On Jan 25, 2016, at 9:53 PM, Paul Eggleton
> >> > > 
> >> > > wrote:
> >> > > 
> >> > > When you need to set EXTRA_OECONF for a recipe, you need to know what
> >> > > options the configure script actually supports; the configure script
> >> > > however is only accessible from within a devshell and (at least in
> >> > > the
> >> > > case of autotooled software fetched from an SCM repository) may not
> >> > > actually exist until do_configure has run. Thus, provide a "devtool
> >> > > configure-help" subcommand that runs the configure script for a
> >> > > recipe
> >> > > with --help and shows you the output through a pager (e.g. less),
> >> > > prefaced by a header describing the current options being specified.
> >> > > 
> >> > > There is basic support for autotools, cmake and bare configure
> >> > > scripts.
> >> > > The cmake support is a little hacky since cmake doesn't really have a
> >> > > concise help option that lists user-defined knobs (without actually
> >> > > running through the configure process), however that being a design
> >> > > feature of cmake there's not much I can think of to do about that at
> >> > > the moment.
> >> > 
> >> > this option is autotools specific. We need to convey this precisely.
> >> > may
> >> > be
> >> > involve autotools in the option name or something
> >> 
> >> Whilst not all non-autoconf configure scripts provide a --help option,
> >> some
> >> do. ffmpeg/libav is one example (and one where you really do need to see
> >> the output since there are a lot of options...)
> > 
> > I meant to add, even if configure --help doesn't work (or there is no
> > configure script at all) the header this subcommand prints will work with
> > any recipe provided that do_configure is actually implemented and
> > contains some actual commands as opposed to calls to some other function.
> 
> my worry is it being associated to do_configure, and we know that only
> a subset ( although a large one) is only using autotools but not all.
> So think about avoiding that confusion.

I'm not terribly convinced there could be much confusion. FWIW, I just looked 
through the other recipes we have in OE-Core that have a call to a configure 
script  that don't use autoconf (man, zlib, x264, v86d, ed, libpostproc); the 
configure scripts for all of these at least produce some useful output when 
called with --help. I then took a look at meta-oe (the layer), and checked the 
following recipes:

* concurrencykit - works
* ddrescue - works
* fio - works
* libvpx - works
* lzip - works
* nodejs - works
* mg - works (no options, but message is sensible)
* terminus-font - configure script is supplied without the execute bit set but
  does support --help
* iscsi-initiator-utils - configure script is in a subdir so isn't found, but
  does support --help
* netkit-rsh - works
* netkit-rpc - works
* netkit-telnet - works
* netkit-tftp - works
* netkit-rusers - works
* netkit-rwho - works
* netmap - configure script is in a subdir so isn't found, but does support 
  --help
* ntimed - configure script is dumb and takes no options at all
* tvheadend - works
* mozjs - seems to break devtool modify, but configure script does support 
  --help

So there are some minor bugs to fix, but of those recipes I tested there is a 
grand total of 1 script that doesn't support --help. Honestly I think if a 
configure script exists it's reasonable to assume that 
--help is going to print something useful; even if it doesn't there shouldn't 
be any major harm and you're still going to get a printout of the do_configure 
value in which you'll be able to see what options are currently passed.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/5] devtool: add configure-help subcommand

2016-01-25 Thread Paul Eggleton 
When you need to set EXTRA_OECONF for a recipe, you need to know what
options the configure script actually supports; the configure script
however is only accessible from within a devshell and (at least in the
case of autotooled software fetched from an SCM repository) may not
actually exist until do_configure has run. Thus, provide a "devtool
configure-help" subcommand that runs the configure script for a recipe
with --help and shows you the output through a pager (e.g. less),
prefaced by a header describing the current options being specified.

There is basic support for autotools, cmake and bare configure scripts.
The cmake support is a little hacky since cmake doesn't really have a
concise help option that lists user-defined knobs (without actually
running through the configure process), however that being a design
feature of cmake there's not much I can think of to do about that at
the moment.

Signed-off-by: Paul Eggleton 
---
 scripts/lib/devtool/__init__.py |   9 ++-
 scripts/lib/devtool/utilcmds.py | 170 
 2 files changed, 175 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index 0405d22..ff97dfc 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -129,7 +129,7 @@ def get_recipe_file(cooker, pn):
 logger.error("Unable to find any recipe file matching %s" % pn)
 return recipefile
 
-def parse_recipe(config, tinfoil, pn, appends):
+def parse_recipe(config, tinfoil, pn, appends, filter_workspace=True):
 """Parse recipe of a package"""
 import oe.recipeutils
 recipefile = get_recipe_file(tinfoil.cooker, pn)
@@ -138,9 +138,10 @@ def parse_recipe(config, tinfoil, pn, appends):
 return None
 if appends:
 append_files = tinfoil.cooker.collection.get_file_appends(recipefile)
-# Filter out appends from the workspace
-append_files = [path for path in append_files if
-not path.startswith(config.workspace_path)]
+if filter_workspace:
+# Filter out appends from the workspace
+append_files = [path for path in append_files if
+not path.startswith(config.workspace_path)]
 else:
 append_files = None
 return oe.recipeutils.parse_recipe(recipefile, append_files,
diff --git a/scripts/lib/devtool/utilcmds.py b/scripts/lib/devtool/utilcmds.py
index 375d7a3..a8f5e97 100644
--- a/scripts/lib/devtool/utilcmds.py
+++ b/scripts/lib/devtool/utilcmds.py
@@ -61,6 +61,165 @@ def edit_recipe(args, config, basepath, workspace):
 return 0
 
 
+def configure_help(args, config, basepath, workspace):
+"""Entry point for the devtool 'configure-help' subcommand"""
+import oe.utils
+
+check_workspace_recipe(workspace, args.recipename)
+tinfoil = setup_tinfoil(config_only=False, basepath=basepath)
+try:
+rd = parse_recipe(config, tinfoil, args.recipename, appends=True, 
filter_workspace=False)
+if not rd:
+return 1
+b = rd.getVar('B', True)
+s = rd.getVar('S', True)
+configurescript = os.path.join(s, 'configure')
+confdisabled = 'noexec' in rd.getVarFlags('do_configure') or 
'do_configure' not in (rd.getVar('__BBTASKS', False) or [])
+configureopts = oe.utils.squashspaces(rd.getVar('CONFIGUREOPTS', True) 
or '')
+extra_oeconf = oe.utils.squashspaces(rd.getVar('EXTRA_OECONF', True) 
or '')
+extra_oecmake = oe.utils.squashspaces(rd.getVar('EXTRA_OECMAKE', True) 
or '')
+do_configure = rd.getVar('do_configure', True) or ''
+do_configure_noexpand = rd.getVar('do_configure', False) or ''
+packageconfig = rd.getVarFlags('PACKAGECONFIG') or []
+autotools = bb.data.inherits_class('autotools', rd) and ('oe_runconf' 
in do_configure or 'autotools_do_configure' in do_configure)
+cmake = bb.data.inherits_class('cmake', rd) and ('cmake_do_configure' 
in do_configure)
+cmake_do_configure = rd.getVar('cmake_do_configure', True)
+pn = rd.getVar('PN', True)
+finally:
+tinfoil.shutdown()
+
+if 'doc' in packageconfig:
+del packageconfig['doc']
+
+if autotools and not os.path.exists(configurescript):
+logger.info('Running do_configure to generate configure script')
+try:
+stdout, _ = exec_build_env_command(config.init_path, basepath,
+   'bitbake -c configure %s' % 
msg, args.recipename,
+   stderr=subprocess.STDOUT)
+except bb.process.ExecutionError:
+pass
+
+if confdisabled or do_configure.strip() in ('', ':'):
+raise DevtoolError("do_configure task has been disabled for this 
recipe")
+elif args.no_pager and not os.path.exists(configurescript):
+raise DevtoolError("No configure script

Re: [OE-core] [PATCH 2/5] devtool: add configure-help subcommand

2016-01-25 Thread Khem Raj 

> On Jan 25, 2016, at 9:53 PM, Paul Eggleton  
> wrote:
> 
> When you need to set EXTRA_OECONF for a recipe, you need to know what
> options the configure script actually supports; the configure script
> however is only accessible from within a devshell and (at least in the
> case of autotooled software fetched from an SCM repository) may not
> actually exist until do_configure has run. Thus, provide a "devtool
> configure-help" subcommand that runs the configure script for a recipe
> with --help and shows you the output through a pager (e.g. less),
> prefaced by a header describing the current options being specified.
> 
> There is basic support for autotools, cmake and bare configure scripts.
> The cmake support is a little hacky since cmake doesn't really have a
> concise help option that lists user-defined knobs (without actually
> running through the configure process), however that being a design
> feature of cmake there's not much I can think of to do about that at
> the moment.
> 

this option is autotools specific. We need to convey this precisely. may be 
involve autotools
in the option name or something

> Signed-off-by: Paul Eggleton 
> ---
> scripts/lib/devtool/__init__.py |   9 ++-
> scripts/lib/devtool/utilcmds.py | 170 
> 2 files changed, 175 insertions(+), 4 deletions(-)
> 
> diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
> index 0405d22..ff97dfc 100644
> --- a/scripts/lib/devtool/__init__.py
> +++ b/scripts/lib/devtool/__init__.py
> @@ -129,7 +129,7 @@ def get_recipe_file(cooker, pn):
> logger.error("Unable to find any recipe file matching %s" % pn)
> return recipefile
> 
> -def parse_recipe(config, tinfoil, pn, appends):
> +def parse_recipe(config, tinfoil, pn, appends, filter_workspace=True):
> """Parse recipe of a package"""
> import oe.recipeutils
> recipefile = get_recipe_file(tinfoil.cooker, pn)
> @@ -138,9 +138,10 @@ def parse_recipe(config, tinfoil, pn, appends):
> return None
> if appends:
> append_files = tinfoil.cooker.collection.get_file_appends(recipefile)
> -# Filter out appends from the workspace
> -append_files = [path for path in append_files if
> -not path.startswith(config.workspace_path)]
> +if filter_workspace:
> +# Filter out appends from the workspace
> +append_files = [path for path in append_files if
> +not path.startswith(config.workspace_path)]
> else:
> append_files = None
> return oe.recipeutils.parse_recipe(recipefile, append_files,
> diff --git a/scripts/lib/devtool/utilcmds.py b/scripts/lib/devtool/utilcmds.py
> index 375d7a3..a8f5e97 100644
> --- a/scripts/lib/devtool/utilcmds.py
> +++ b/scripts/lib/devtool/utilcmds.py
> @@ -61,6 +61,165 @@ def edit_recipe(args, config, basepath, workspace):
> return 0
> 
> 
> +def configure_help(args, config, basepath, workspace):
> +"""Entry point for the devtool 'configure-help' subcommand"""
> +import oe.utils
> +
> +check_workspace_recipe(workspace, args.recipename)
> +tinfoil = setup_tinfoil(config_only=False, basepath=basepath)
> +try:
> +rd = parse_recipe(config, tinfoil, args.recipename, appends=True, 
> filter_workspace=False)
> +if not rd:
> +return 1
> +b = rd.getVar('B', True)
> +s = rd.getVar('S', True)
> +configurescript = os.path.join(s, 'configure')
> +confdisabled = 'noexec' in rd.getVarFlags('do_configure') or 
> 'do_configure' not in (rd.getVar('__BBTASKS', False) or [])
> +configureopts = oe.utils.squashspaces(rd.getVar('CONFIGUREOPTS', 
> True) or '')
> +extra_oeconf = oe.utils.squashspaces(rd.getVar('EXTRA_OECONF', True) 
> or '')
> +extra_oecmake = oe.utils.squashspaces(rd.getVar('EXTRA_OECMAKE', 
> True) or '')
> +do_configure = rd.getVar('do_configure', True) or ''
> +do_configure_noexpand = rd.getVar('do_configure', False) or ''
> +packageconfig = rd.getVarFlags('PACKAGECONFIG') or []
> +autotools = bb.data.inherits_class('autotools', rd) and 
> ('oe_runconf' in do_configure or 'autotools_do_configure' in do_configure)
> +cmake = bb.data.inherits_class('cmake', rd) and 
> ('cmake_do_configure' in do_configure)
> +cmake_do_configure = rd.getVar('cmake_do_configure', True)
> +pn = rd.getVar('PN', True)
> +finally:
> +tinfoil.shutdown()
> +
> +if 'doc' in packageconfig:
> +del packageconfig['doc']
> +
> +if autotools and not os.path.exists(configurescript):
> +logger.info('Running do_configure to generate configure script')
> +try:
> +stdout, _ = exec_build_env_command(config.init_path, basepath,
> +   'bitbake -c configure %s' % 
> msg,

Re: [OE-core] [PATCH 2/5] devtool: add configure-help subcommand

2016-01-25 Thread Paul Eggleton 
Hi Khem,

On Mon, 25 Jan 2016 22:14:09 Khem Raj wrote:
> > On Jan 25, 2016, at 9:53 PM, Paul Eggleton 
> > wrote:
> > 
> > When you need to set EXTRA_OECONF for a recipe, you need to know what
> > options the configure script actually supports; the configure script
> > however is only accessible from within a devshell and (at least in the
> > case of autotooled software fetched from an SCM repository) may not
> > actually exist until do_configure has run. Thus, provide a "devtool
> > configure-help" subcommand that runs the configure script for a recipe
> > with --help and shows you the output through a pager (e.g. less),
> > prefaced by a header describing the current options being specified.
> > 
> > There is basic support for autotools, cmake and bare configure scripts.
> > The cmake support is a little hacky since cmake doesn't really have a
> > concise help option that lists user-defined knobs (without actually
> > running through the configure process), however that being a design
> > feature of cmake there's not much I can think of to do about that at
> > the moment.
> 
> this option is autotools specific. We need to convey this precisely. may be
> involve autotools in the option name or something

Whilst not all non-autoconf configure scripts provide a --help option, some do. 
ffmpeg/libav is one example (and one where you really do need to see the output 
since there are a lot of options...)

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 2/5] devtool: add configure-help subcommand

2016-01-25 Thread Paul Eggleton 
On Tue, 26 Jan 2016 16:17:39 Paul Eggleton wrote:
> Hi Khem,
> 
> On Mon, 25 Jan 2016 22:14:09 Khem Raj wrote:
> > > On Jan 25, 2016, at 9:53 PM, Paul Eggleton
> > > 
> > > wrote:
> > > 
> > > When you need to set EXTRA_OECONF for a recipe, you need to know what
> > > options the configure script actually supports; the configure script
> > > however is only accessible from within a devshell and (at least in the
> > > case of autotooled software fetched from an SCM repository) may not
> > > actually exist until do_configure has run. Thus, provide a "devtool
> > > configure-help" subcommand that runs the configure script for a recipe
> > > with --help and shows you the output through a pager (e.g. less),
> > > prefaced by a header describing the current options being specified.
> > > 
> > > There is basic support for autotools, cmake and bare configure scripts.
> > > The cmake support is a little hacky since cmake doesn't really have a
> > > concise help option that lists user-defined knobs (without actually
> > > running through the configure process), however that being a design
> > > feature of cmake there's not much I can think of to do about that at
> > > the moment.
> > 
> > this option is autotools specific. We need to convey this precisely. may
> > be
> > involve autotools in the option name or something
> 
> Whilst not all non-autoconf configure scripts provide a --help option, some
> do. ffmpeg/libav is one example (and one where you really do need to see
> the output since there are a lot of options...)

I meant to add, even if configure --help doesn't work (or there is no configure 
script at all) the header this subcommand prints will work with any recipe 
provided that do_configure is actually implemented and contains some actual 
commands as opposed to calls to some other function.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 2/5] devtool: add configure-help subcommand

2016-01-25 Thread Khem Raj 
On Mon, Jan 25, 2016 at 10:21 PM, Paul Eggleton
 wrote:
> On Tue, 26 Jan 2016 16:17:39 Paul Eggleton wrote:
>> Hi Khem,
>>
>> On Mon, 25 Jan 2016 22:14:09 Khem Raj wrote:
>> > > On Jan 25, 2016, at 9:53 PM, Paul Eggleton
>> > > 
>> > > wrote:
>> > >
>> > > When you need to set EXTRA_OECONF for a recipe, you need to know what
>> > > options the configure script actually supports; the configure script
>> > > however is only accessible from within a devshell and (at least in the
>> > > case of autotooled software fetched from an SCM repository) may not
>> > > actually exist until do_configure has run. Thus, provide a "devtool
>> > > configure-help" subcommand that runs the configure script for a recipe
>> > > with --help and shows you the output through a pager (e.g. less),
>> > > prefaced by a header describing the current options being specified.
>> > >
>> > > There is basic support for autotools, cmake and bare configure scripts.
>> > > The cmake support is a little hacky since cmake doesn't really have a
>> > > concise help option that lists user-defined knobs (without actually
>> > > running through the configure process), however that being a design
>> > > feature of cmake there's not much I can think of to do about that at
>> > > the moment.
>> >
>> > this option is autotools specific. We need to convey this precisely. may
>> > be
>> > involve autotools in the option name or something
>>
>> Whilst not all non-autoconf configure scripts provide a --help option, some
>> do. ffmpeg/libav is one example (and one where you really do need to see
>> the output since there are a lot of options...)
>
> I meant to add, even if configure --help doesn't work (or there is no 
> configure
> script at all) the header this subcommand prints will work with any recipe
> provided that do_configure is actually implemented and contains some actual
> commands as opposed to calls to some other function.

my worry is it being associated to do_configure, and we know that only
a subset ( although a large one) is only using autotools but not all.
So think about avoiding that confusion.
>
> Cheers,
> Paul
>
> --
>
> Paul Eggleton
> Intel Open Source Technology Centre
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core