Re: [OpenWrt-Devel] Question regarding call python script

[Jason Wu]

On 14/03/2016 5:20 PM, John Crispin wrote:



On 14/03/2016 03:55, Jason Wu wrote:



On 13/03/2016 8:41 PM, Bastian Bittorf wrote:

Using n-dimensional arrays in build scripts is
a rare usecase. you can easily switch to AWK if this
fits more. But using Perl or Python should be avoided,
because it adds more complexity to the dependencies
and leads to more headache when maintaining:


Just to be sure, are you saying that we should also avoid indexed array
in bash? It is possible to create n-dimensional arrays with indexed
array. The main reason for this is to be sure that I am on the right track.

IMHO, every script language has it own pros and cons. But for a complex
build system (such as buildroot/openwrt), consistency, compatibility and
dependencies are difficult to maintain and also makes difficult for
testing (the number of possible test required to ensure the
compatibility increases exponential as we add more dependency). For
example, RHEL5 uses bash version 3.x which does not support associated
array. Sometimes some of the dependency can be avoid and best to avoid
from maintainers point of view.

Jason
_


please explain what problem you have that requires arrays to solve it.

John


Hi John

I am creating a Image Tree Source (ITS) generator script which allows 
user to passing to the image properties such as image type 
(kernel/ftd/rootfs) or hash algorithm, signature, compression, entry 
point and load addr, and which configuration properties. Since the ITS 
allows multiple images (multiple kernel/fdt/rootfs) and multiple 
configuration so two dimension array was used to store the data.


The main reason I asked is because the solution can be implemented as 
simple structure (like Bastian suggested) or used indexed arrays. I 
guess my question which is preferred in OpenWrt community.


Jason
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] Question regarding call python script

[John Crispin]

On 14/03/2016 03:55, Jason Wu wrote:
> 
> 
> On 13/03/2016 8:41 PM, Bastian Bittorf wrote:
>> Using n-dimensional arrays in build scripts is
>> a rare usecase. you can easily switch to AWK if this
>> fits more. But using Perl or Python should be avoided,
>> because it adds more complexity to the dependencies
>> and leads to more headache when maintaining:
> 
> Just to be sure, are you saying that we should also avoid indexed array
> in bash? It is possible to create n-dimensional arrays with indexed
> array. The main reason for this is to be sure that I am on the right track.
> 
> IMHO, every script language has it own pros and cons. But for a complex
> build system (such as buildroot/openwrt), consistency, compatibility and
> dependencies are difficult to maintain and also makes difficult for
> testing (the number of possible test required to ensure the
> compatibility increases exponential as we add more dependency). For
> example, RHEL5 uses bash version 3.x which does not support associated
> array. Sometimes some of the dependency can be avoid and best to avoid
> from maintainers point of view.
> 
> Jason
> _

please explain what problem you have that requires arrays to solve it.

John


__
> openwrt-devel mailing list
> openwrt-devel@lists.openwrt.org
> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] Question regarding call python script

[Bastian Bittorf]

* Jason Wu  [14.03.2016 07:39]:
> Just to be sure, are you saying that we should also avoid indexed
> array in bash? It is possible to create n-dimensional arrays with
> indexed array. The main reason for this is to be sure that I am on
> the right track.

"Complexity is a cost, spend it wisely."

ofcourse it depends on your usecase. I'am really not in
the position to decide anything, but if you ask me:
stick to portable constructs...

bye, bastian
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] Question regarding call python script

[Jason Wu]

On 13/03/2016 8:41 PM, Bastian Bittorf wrote:

Using n-dimensional arrays in build scripts is
a rare usecase. you can easily switch to AWK if this
fits more. But using Perl or Python should be avoided,
because it adds more complexity to the dependencies
and leads to more headache when maintaining:


Just to be sure, are you saying that we should also avoid indexed array 
in bash? It is possible to create n-dimensional arrays with indexed 
array. The main reason for this is to be sure that I am on the right track.


IMHO, every script language has it own pros and cons. But for a complex 
build system (such as buildroot/openwrt), consistency, compatibility and 
dependencies are difficult to maintain and also makes difficult for 
testing (the number of possible test required to ensure the 
compatibility increases exponential as we add more dependency). For 
example, RHEL5 uses bash version 3.x which does not support associated 
array. Sometimes some of the dependency can be avoid and best to avoid 
from maintainers point of view.


Jason
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] Question regarding call python script

[Bastian Bittorf]

* Karl Palsson  [13.03.2016 10:00]:
> > array_set "test" 7
> > array_get 7
> > 
> > or for 2-dimensional arrays:
> > array_set "2dim" 3 14
> > array_get 3 14
> 
> You know, this is _exactly_ why people want to use python or even
> perl instead of bash/awk/sed.
> 
> This is a great, wonderful solution if you _must_ use posix
> shell. But do we?

Yes, we do.
In terms of "this does not look nice and elegant" you are right:

Using n-dimensional arrays in build scripts is
a rare usecase. you can easily switch to AWK if this
fits more. But using Perl or Python should be avoided,
because it adds more complexity to the dependencies
and leads to more headache when maintaining:

Perl and Python are not well defined and all sorts
of things are breaking. (if not tomorrow, then next week...)

bye, bastian
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] Question regarding call python script

[John Crispin]

On 12/03/2016 23:43, Karl Palsson wrote:
> 
> Bastian Bittorf  wrote:
>> * Paul Fertser  [11.03.2016 11:15]:
>>> As a sidenote, POSIX shell doesn't support associative arrays (which
>>> were mentioned in this thread), so it's probably worth clarifying
>>> whether it's ok to require bash >= 4 for OpenWrt build scripts or not.
>>
>> yes, POSIX is "missing" some goodies of a full bash but:
>>
>> "Stick to portable constructs where possible, and
>>  you will make somebody's life easier in the future.
>>  Maybe your own."
>>
>> For all things you can use a workaround.
>> So, if you need an array e.g.
>>
>> foo[7]=bar
>>
>> for e.g. easy iterating over it? use e.g. this approach:
>>
>> array_put()
>> {
>>  local message="$1"
>>  local dim1="$2"
>>  local dim2="$3"
>>
>>  eval ${ARRAYNAME:-default}_${dim1}_${dim2}='$message'
>> }
>>
>> array_get()
>> {
>>  local dim1="$1"
>>  local dim2="$2"
>>  local var
>>
>>  eval var="\"\${${ARRAYNAME:-default}_${dim1}_${dim2}:-unset}\""
>>  echo "$var"
>> }
>>
>> array_set "test" 7
>> array_get 7
>>
>> or for 2-dimensional arrays:
>> array_set "2dim" 3 14
>> array_get 3 14
>>
>> you can simplify this by removing dim2 and ARRAYNAME if this is
>> too much.
> 
> You know, this is _exactly_ why people want to use python or even
> perl instead of bash/awk/sed.
> 
> This is a great, wonderful solution if you _must_ use posix
> shell. But do we?
> 
> Cheers,
> Karl P
> 

we are discussing solutions to a problem here that we dont even know.
personally i believe that if you need to use arrays in bash hen the
solution might not be the best. could whoever wants to use arrays please
explain what he wants to do in the target folder that requires arrays ?
once we understand the problem we can discuss if arrays are required.
discussing awk vs perl vs bash vs python without any context apart from
some snippets posted by basti seems suboptimal.

John
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] Question regarding call python script

[Bastian Bittorf]

* Paul Fertser  [11.03.2016 11:15]:
> As a sidenote, POSIX shell doesn't support associative arrays (which
> were mentioned in this thread), so it's probably worth clarifying
> whether it's ok to require bash >= 4 for OpenWrt build scripts or not.

yes, POSIX is "missing" some goodies of a full bash but:

"Stick to portable constructs where possible, and
 you will make somebody's life easier in the future.
 Maybe your own."

For all things you can use a workaround.
So, if you need an array e.g.

foo[7]=bar

for e.g. easy iterating over it? use e.g. this approach:

array_put()
{
local message="$1"
local dim1="$2"
local dim2="$3"

eval ${ARRAYNAME:-default}_${dim1}_${dim2}='$message'
}

array_get()
{
local dim1="$1"
local dim2="$2"
local var

eval var="\"\${${ARRAYNAME:-default}_${dim1}_${dim2}:-unset}\""
echo "$var"
}

array_set "test" 7
array_get 7

or for 2-dimensional arrays:
array_set "2dim" 3 14
array_get 3 14

you can simplify this by removing dim2 and ARRAYNAME if this is too much.

bye, bastian
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] Question regarding call python script

[Paul Fertser]

On Fri, Mar 11, 2016 at 08:37:31PM +1000, Jason Wu wrote:
> >>As a sidenote, POSIX shell doesn't support associative arrays (which
> >>were mentioned in this thread), so it's probably worth clarifying
> >>whether it's ok to require bash >= 4 for OpenWrt build scripts or not.
> >>
> >
> >what exactly do you need them for and can you solve it in a different
> >way ? ideally we run on as many distros as possible.
> 
> I can use indexed array instead of associative array.

Citing a SO answer: "POSIX shells have one array: the positional
parameters ($1, $2, etc., collectively refered to as "$@")".

-- 
Be free, use free (http://www.gnu.org/philosophy/free-sw.html) software!
mailto:fercer...@gmail.com
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] Question regarding call python script

[Jason Wu]

On 11/03/2016 8:12 PM, John Crispin wrote:



On 11/03/2016 11:11, Paul Fertser wrote:

Hey folks,

Bastian Bittorf  writes:

* John Crispin  [11.03.2016 06:27]:

depends on what you want to do. i dont really see anything that cannot
be handled using shell. what exactly do you want to write in python ?


also in general i think it's a good idea to have
less dependency than more. personally 8-) i want to
see perl and awk abandoned...


As a sidenote, POSIX shell doesn't support associative arrays (which
were mentioned in this thread), so it's probably worth clarifying
whether it's ok to require bash >= 4 for OpenWrt build scripts or not.



what exactly do you need them for and can you solve it in a different
way ? ideally we run on as many distros as possible.


I can use indexed array instead of associative array.

thanks,

Jason
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] Question regarding call python script

[John Crispin]

On 11/03/2016 11:11, Paul Fertser wrote:
> Hey folks,
> 
> Bastian Bittorf  writes:
>> * John Crispin  [11.03.2016 06:27]:
>>> depends on what you want to do. i dont really see anything that cannot
>>> be handled using shell. what exactly do you want to write in python ?
>>
>> also in general i think it's a good idea to have
>> less dependency than more. personally 8-) i want to
>> see perl and awk abandoned...
> 
> As a sidenote, POSIX shell doesn't support associative arrays (which
> were mentioned in this thread), so it's probably worth clarifying
> whether it's ok to require bash >= 4 for OpenWrt build scripts or not.
> 

what exactly do you need them for and can you solve it in a different
way ? ideally we run on as many distros as possible.
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] Question regarding call python script

[Paul Fertser]

Hey folks,

Bastian Bittorf  writes:
> * John Crispin  [11.03.2016 06:27]:
>> depends on what you want to do. i dont really see anything that cannot
>> be handled using shell. what exactly do you want to write in python ?
>
> also in general i think it's a good idea to have
> less dependency than more. personally 8-) i want to
> see perl and awk abandoned...

As a sidenote, POSIX shell doesn't support associative arrays (which
were mentioned in this thread), so it's probably worth clarifying
whether it's ok to require bash >= 4 for OpenWrt build scripts or not.

-- 
Be free, use free (http://www.gnu.org/philosophy/free-sw.html) software!
mailto:fercer...@gmail.com
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] Question regarding call python script

[Bastian Bittorf]

* John Crispin  [11.03.2016 06:27]:
> depends on what you want to do. i dont really see anything that cannot
> be handled using shell. what exactly do you want to write in python ?

also in general i think it's a good idea to have
less dependency than more. personally 8-) i want to
see perl and awk abandoned...

(perl is a deb for the kernel itself, but there are
patches to circumvent this. awk is mostly used for
e.g. command | awk $print_first_word - but 'awk' is
posix, so we have it anyway 8-))

bye, bastian
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] Question regarding call python script

[John Crispin]

On 11/03/2016 04:49, Jason Wu wrote:
> Hi John,
> 
> There are two scripts that i am thinking about.
> 
> 1. generate fitimage.its that is able to generate mufti-configuration
> its file with multiple kernel/dtb/rootfs. One of the benefit of using
> python is that it is easier to implement from my point of view. But have
> to said it can done with shell as well.
> 
> 2. dts/dtb modification existing dts file. e.g append/replace a the
> existing dts parameters, or add a subnode to a node and etc, this one
> will be harder to do it in shell but it can be done with shell+awk.
> 
> In both case, i am thinking to use nested
> array(shell)/dictionary(python) to create the database. each
> array/dictionary representing a node the key benefit of using python is
> that nested dictionary is much easier to handle than bash's associated
> array.
> 
> The other reason is that I got a existing python script does similar
> thing but more specific to my own purpose. Just though it will be easier
> to modify it to suit the need with OpenWrt.
> 
> There is no reason why the script can't be implemented in shell. That's
> why i ask.
> 
> Best regards,
> 
> jason
> 
> On 11/03/2016 7:12 AM, John Crispin wrote:
>>
>>
>> On 10/03/2016 22:11, Alexandru Ardelean wrote:
>>>
>>>
>>> On Thu, Mar 10, 2016 at 6:38 PM, Jo-Philipp Wich >> > wrote:
>>>
>>>  Hi Jason,
>>>
>>>  no reason against it per se but it would introduce a new build
>>> prereq I
>>>  suppose. If possible we'd like to stick to things we already
>>> require
>>>  (perl, awk, shell).
>>>
>>>
>>>  Regards, Jo
>>>  ___
>>>  openwrt-devel mailing list
>>>  openwrt-devel@lists.openwrt.org
>>> 
>>>  https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
>>>
>>>
>>> As far as I know, python is already required by the OpenWRT build
>>> system.
>>> The build stops if there is no /usr/bin/python (not 100% sure about this
>>> anymore)
>>>
>>> And in rules.mk  there's
>>> PYTHON:=python
>>> (and this is not related in any way to the Python package in the feeds)
>>>
>>> So, I guess Python is fine.
>>>
>>
>> depends on what you want to do. i dont really see anything that cannot
>> be handled using shell. what exactly do you want to write in python ?
>>
>>

please dont top post. i see your point but there is no technical reason
to use python beyond you being more skilled in python than the other
languages. i personally would prefer to stick to shell as that is what
the rest of the system uses. so i am basing my opinion more on terms of
consistency.

John



>>
>>>
>>>
>>> ___
>>> openwrt-devel mailing list
>>> openwrt-devel@lists.openwrt.org
>>> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
>>>
>> ___
>> openwrt-devel mailing list
>> openwrt-devel@lists.openwrt.org
>> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
>>
> ___
> openwrt-devel mailing list
> openwrt-devel@lists.openwrt.org
> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] Question regarding call python script

[Jason Wu]

HiJo and Alexandru,

Thanks for the input.

Cheers,

jason

On 11/03/2016 7:11 AM, Alexandru Ardelean wrote:



On Thu, Mar 10, 2016 at 6:38 PM, Jo-Philipp Wich > wrote:

Hi Jason,

no reason against it per se but it would introduce a new build prereq I
suppose. If possible we'd like to stick to things we already require
(perl, awk, shell).


Regards, Jo
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org 
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel


As far as I know, python is already required by the OpenWRT build system.
The build stops if there is no /usr/bin/python (not 100% sure about this
anymore)

And in rules.mk  there's
PYTHON:=python
(and this is not related in any way to the Python package in the feeds)

So, I guess Python is fine.



___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel


___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] Question regarding call python script

[Jason Wu]

Hi John,

There are two scripts that i am thinking about.

1. generate fitimage.its that is able to generate mufti-configuration 
its file with multiple kernel/dtb/rootfs. One of the benefit of using 
python is that it is easier to implement from my point of view. But have 
to said it can done with shell as well.


2. dts/dtb modification existing dts file. e.g append/replace a the 
existing dts parameters, or add a subnode to a node and etc, this one 
will be harder to do it in shell but it can be done with shell+awk.


In both case, i am thinking to use nested 
array(shell)/dictionary(python) to create the database. each 
array/dictionary representing a node the key benefit of using python is 
that nested dictionary is much easier to handle than bash's associated 
array.


The other reason is that I got a existing python script does similar 
thing but more specific to my own purpose. Just though it will be easier 
to modify it to suit the need with OpenWrt.


There is no reason why the script can't be implemented in shell. That's 
why i ask.


Best regards,

jason

On 11/03/2016 7:12 AM, John Crispin wrote:



On 10/03/2016 22:11, Alexandru Ardelean wrote:



On Thu, Mar 10, 2016 at 6:38 PM, Jo-Philipp Wich > wrote:

 Hi Jason,

 no reason against it per se but it would introduce a new build prereq I
 suppose. If possible we'd like to stick to things we already require
 (perl, awk, shell).


 Regards, Jo
 ___
 openwrt-devel mailing list
 openwrt-devel@lists.openwrt.org 
 https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel


As far as I know, python is already required by the OpenWRT build system.
The build stops if there is no /usr/bin/python (not 100% sure about this
anymore)

And in rules.mk  there's
PYTHON:=python
(and this is not related in any way to the Python package in the feeds)

So, I guess Python is fine.



depends on what you want to do. i dont really see anything that cannot
be handled using shell. what exactly do you want to write in python ?






___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel


___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel


___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] Question regarding call python script

[John Crispin]

On 10/03/2016 22:11, Alexandru Ardelean wrote:
> 
> 
> On Thu, Mar 10, 2016 at 6:38 PM, Jo-Philipp Wich  > wrote:
> 
> Hi Jason,
> 
> no reason against it per se but it would introduce a new build prereq I
> suppose. If possible we'd like to stick to things we already require
> (perl, awk, shell).
> 
> 
> Regards, Jo
> ___
> openwrt-devel mailing list
> openwrt-devel@lists.openwrt.org 
> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
> 
> 
> As far as I know, python is already required by the OpenWRT build system.
> The build stops if there is no /usr/bin/python (not 100% sure about this
> anymore)
> 
> And in rules.mk  there's
> PYTHON:=python
> (and this is not related in any way to the Python package in the feeds)
> 
> So, I guess Python is fine.
> 

depends on what you want to do. i dont really see anything that cannot
be handled using shell. what exactly do you want to write in python ?



> 
> 
> ___
> openwrt-devel mailing list
> openwrt-devel@lists.openwrt.org
> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
> 
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] Question regarding call python script

[Alexandru Ardelean]

On Thu, Mar 10, 2016 at 6:38 PM, Jo-Philipp Wich  wrote:

> Hi Jason,
>
> no reason against it per se but it would introduce a new build prereq I
> suppose. If possible we'd like to stick to things we already require
> (perl, awk, shell).
>
>
> Regards, Jo
> ___
> openwrt-devel mailing list
> openwrt-devel@lists.openwrt.org
> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
>

As far as I know, python is already required by the OpenWRT build system.
The build stops if there is no /usr/bin/python (not 100% sure about this
anymore)

And in rules.mk there's
PYTHON:=python
(and this is not related in any way to the Python package in the feeds)

So, I guess Python is fine.
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] Question regarding call python script

[Jo-Philipp Wich]

Hi Jason,

no reason against it per se but it would introduce a new build prereq I
suppose. If possible we'd like to stick to things we already require
(perl, awk, shell).


Regards, Jo
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel