On Sunday, March 1, 2020 at 8:54:21 AM UTC-5, Nadav Har'El wrote:
>
> On Sun, Mar 1, 2020 at 3:41 PM Waldek Kozaczuk <jwkoz...@gmail.com
> <javascript:>> wrote:
>
>> I like this approach. Shall we handle distribution upgrades (Fedora 30 to
>> 31 for example) and if so how ? If we do not user would need to know to
>> delete old downloaded_packages and re-run that new script. Ideally we could
>> have makefile somehow detect that the distro version is different than of
>> what is in downloaded_packages.
>>
>
> Don't you have the same problem with your current approach?
> One thing to consider is putting this stuff in build/, so that "rm -r
> build" (which you'd be wise to do after a distro upgrade, otherwise things
> break) will delete it to.
>
> Finally, there is something I don't understand / didn't follow:
>
> For me, I just installed the package gcc-aarch64-linux-gnu normally (with
> yum) and everything just worked. Why did you need to do something else? For
> non-Fedora?
> So for Fedora, you can still do "dnf install ...", and always get the
> correct version (and it gets updated on distro upgrades!). For non-Fedora,
> there is no issue of matching distro versions anyway - Ubuntu can't "match"
> with Fedora anyway.
>
>
>>
>> Also I am not sure you realized that the old aarch64 packages that are
>> part of externals are for Fedora which I think explains why we never (me
>> and Rick I think) were never successful to build runnable aarch64 image on
>> Ubuntu.
>>
>
> Hmm... No, I never noticed it. What was taken from there, libraries?
> Is there a way on Ubuntu to check the versions of the downloaded
> libraries, and if it doesn't match what we need (*what* do we need) ask to
> run the script again?
>
I should have been more clear. On Fedora setup.py installs the
gcc-c++-aarch64-linux-gnu, on Ubuntu it does not install anything
equivalent. So if on my Ubuntu machine I manually install that aarch64
package for c++ (cannot remember the name, but the Ubuntu package installs
the headers - besides compiler binaries - and puts them in some system
directories) and I build aarch64 OSv image it hangs on boot without
displaying anything. So I suspect that somehow the Ubuntu edition of
aarch64 gcc compiler does NOT work well with the aarch64 headers and
libraries in externals or (in future downloaded outside of externals). I
suspect there is something wrong in our Makefile that makes it mix aarch64
headers from system directories on Ubuntu with whatever Fedora files in
externals or downloaded. Does it make sense?
>
>
>> So should makefile any check like “flavor <> Fedora”?
>>
>> On Sun, Mar 1, 2020 at 02:16 Nadav Har'El <n...@scylladb.com
>> <javascript:>> wrote:
>>
>>>
>>> On Sun, Mar 1, 2020 at 6:49 AM Commit Bot <b...@cloudius-systems.com
>>> <javascript:>> wrote:
>>>
>>>> From: Waldemar Kozaczuk <jwkoz...@gmail.com <javascript:>>
>>>> Committer: Waldemar Kozaczuk <jwkoz...@gmail.com <javascript:>>
>>>> Branch: master
>>>>
>>>> externals: enhance setup.py to download aarch64 gcc and boost code and
>>>> libraries
>>>>
>>>> References #743
>>>>
>>>> Signed-off-by: Waldemar Kozaczuk <jwkoz...@gmail.com <javascript:>>
>>>>
>>>> ---
>>>> diff --git a/.gitignore b/.gitignore
>>>> --- a/.gitignore
>>>> +++ b/.gitignore
>>>> @@ -49,3 +49,4 @@ modules/libyaml/usr.manifest
>>>> modules/dl_tests/usr.manifest
>>>> .idea
>>>> compile_commands.json
>>>> +downloaded_packages
>>>> diff --git a/scripts/setup.py b/scripts/setup.py
>>>> --- a/scripts/setup.py
>>>> +++ b/scripts/setup.py
>>>> @@ -3,7 +3,7 @@
>>>> # set up a development environment for OSv. Run as root.
>>>>
>>>> import sys, argparse
>>>> -import subprocess
>>>> +import subprocess, os
>>>>
>>>> standard_ec2_packages = ['python-pip', 'wget']
>>>> standard_ec2_post_install = ['pip install awscli &&'
>>>> @@ -64,6 +64,40 @@ class Fedora(object):
>>>> test_packages = ['openssl-devel']
>>>> ec2_post_install = standard_ec2_post_install
>>>>
>>>> + def aarch64_download(self, version):
>>>> + gcc_packages = ['gcc',
>>>> + 'glibc',
>>>> + 'glibc-devel',
>>>> + 'libgcc',
>>>> + 'libstdc++',
>>>> + 'libstdc++-devel',
>>>> + 'libstdc++-static']
>>>> + boost_packages = ['boost-devel',
>>>> + 'boost-static',
>>>> + 'boost-system']
>>>> + osv_root =
>>>> os.path.join(os.path.dirname(os.path.abspath(__file__)), '..')
>>>> + script_path = '%s/scripts/download_rpm_package.sh' % osv_root
>>>> + destination = '%s/downloaded_packages/aarch64' % osv_root
>>>> + ##
>>>> + # The setup.py is typically run as root to allow yum properly
>>>> install packages
>>>> + # This however would cause all files downloaded to
>>>> downloaded_packages/aarch64 directory
>>>> + # get created and owned by the root user which in most cases
>>>> is not desirable
>>>> + # To prevent that let us compare current process user id with
>>>> the owner id of osv root
>>>> + # directory and if different run all download command with the
>>>> same user as the one owning
>>>> + # the root directory
>>>>
>>>
>>> I think this may be a good indication that this should not have been
>>> part of the "setup.py" script.
>>> Until today, the "setup.py" script was something you need to run - if at
>>> all - once. You don't need to
>>> run it every time you check out a clean OSv directory. You don't need to
>>> re-run it after a "make clean".
>>>
>>> Moreover, I think there are other reasons why this code should *not* be
>>> part of setup.py:
>>> First and foremost, the aarch stuff isn't needed by everyone - people
>>> who just compile OSv on x86 won't need it at all,
>>>
>>> I would prefer to see something like the following:
>>> 1. A separate script to download the aarch64 packages.
>>> 2. The makefile will verify, if arch=aarch64, that the aarch64 tools are
>>> installed (in the system downloaded), and if not, tell you to run the new
>>> script - and exit. (we already have a bunch of similar checks in the
>>> makefile).
>>>
>>>
>>> + current_user_id = os.getuid()
>>>> + osv_root_owner_id = os.stat(osv_root).st_uid
>>>> + if current_user_id != osv_root_owner_id and current_user_id ==
>>>> 0:
>>>> + command_prefix = "sudo -u '#%d'" % osv_root_owner_id #
>>>> Most likely setup.py is run by root so let us use sudo
>>>>
>>>
>>> If you suspect you're running as root, you can use "su" instead of
>>> "sudo".
>>> But as I said above, I would prefer not to need this at all, and have a
>>> separate script.
>>>
>>> + else:
>>>> + command_prefix = ''
>>>> +
>>>> + install_commands = ['%s %s %s %s %s/gcc' % (command_prefix,
>>>> script_path, package, version, destination) for package in gcc_packages]
>>>> + install_commands += ['%s %s %s %s %s/boost' % (command_prefix,
>>>> script_path, package, version, destination) for package in boost_packages]
>>>> + install_commands = ['%s rm -rf %s/gcc/install' %
>>>> (command_prefix, destination),
>>>> + '%s rm -rf %s/boost/install' %
>>>> (command_prefix, destination)] + install_commands
>>>> + return ' && '.join(install_commands)
>>>> +
>>>> class Fedora_25(object):
>>>> packages = ['java-1.8.0-openjdk', 'python2-requests',
>>>> 'openssl-devel', 'lua-5.3.*', 'lua-devel-5.3.*']
>>>> ec2_packages = []
>>>> @@ -361,6 +395,10 @@ def parse_file(f):
>>>> if cmdargs.test:
>>>> pkg += distro.test_packages + dver.test_packages
>>>> subprocess.check_call(distro.install + ' ' +
>>>> str.join(' ', pkg), shell=True)
>>>> + if 'aarch64_download' in dir(distro):
>>>> + print('Downloading aarch64 packages to
>>>> cross-compile ARM version ...')
>>>> +
>>>> subprocess.check_call(distro.aarch64_download(dver.version), shell=True)
>>>> + print('Downloaded all aarch64 packages!')
>>>> if cmdargs.ec2:
>>>> if distro.ec2_post_install:
>>>> subprocess.check_call(distro.ec2_post_install,
>>>> shell=True)
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "OSv Development" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to osv...@googlegroups.com <javascript:>.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/osv-dev/00000000000059fa59059fc3cb7d%40google.com
>>>> .
>>>>
>>>
--
You received this message because you are subscribed to the Google Groups "OSv
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/osv-dev/c72e965c-0281-472e-941b-bc1b6ab16ab6%40googlegroups.com.
