Re: Best Practices python virtualenv
On 2018-05-01, IL Ka wrote: >> while there are Python modules which rely on W|X. > Yes, but I do not use them. > > I only run Python and Django. > I am aware of the fact that my python is not compatible with some modules, > and I am ok with it since I do not need them. > >> use ports instead of pkg_add to get Python binary without W|X > That is exactly what I did: built python from official openbsd port with > USE_WXNEEDED=no. > I now have python that works from /home, but does not support some modules. Now you have a version of python that you need to rebuild whenever a library that it uses is bumped to a different major version number, otherwise you have conflicts in some cases, and if not an actual conflict, you're at least stuck with an old version of that library (which may mean you miss security fixes). This might not be too much of a problem if you stick to -stable but will get old fast on -current. > While I am still happy with my approach, I'll not suggest it anymore, > because everyone else says it is not correct, > and I do not want to give bad advice to people on this list:) > > Fixing every single port that needs WX seems like a huge amount of work. That's not necessary for your goal of python working without the flag. Only the libraries that might be used as part of python modules. The only ones we know about are the python webkit bindings and (at least last time I tested it) py-cryptography (something to do with a libssl callback IIRC). > People do not care about W|X security too much outside of openbsd > community:( grsecurity does block RWX mappings by default btw.
Re: Best Practices python virtualenv
Not exactly. For starters right now I am getting to know OpenBSD as a platform so at this point is more about evaluating if and how to alter my workflows to better fit with the platform. Next off though is in general with python code I do consider it important to think about what the deployment will look like on several target environments. Write once and run everywhere doesn't always work out exactly like that with python. It sounds nice but is rarely reality. In general though if I have a specific project to code, I am starting from several pre-canned requirements files or docker images/compose scripts that I have already set up for specific types of projects. So this is a case more of understanding how one of my use cases fits in the OpenBSD eco system more so than just trying to setup a dev environment to do XYZ. Ken On Tue, May 01, 2018 at 10:28:36PM +0200, Ve Telko wrote: > Hi, Ken. > > Did you noticed that you are spending much more time > with setting up your env every time you want to run your > python apps than with programming itself? > > No offense but I think this is not normal. > > Ve. >
Re: Best Practices python virtualenv
Hi, Ken. Did you noticed that you are spending much more time with setting up your env every time you want to run your python apps than with programming itself? No offense but I think this is not normal. Ve.
Re: Best Practices python virtualenv
On Mon, Apr 30, 2018 at 04:46:49PM +, Ken MacKenzie wrote: > Is there a recommended best practice when setting up an environment with > python > virtualenv with regards to wxallowed. > > My typical workflow is under my home directory I have a > dev/language/project/.venv type structure. I guess the simple solution is to > mount /home as wxallowed in /etc/fstab, but is that truly the preferred way. > Seems to make a new attack surface for anything in home. > > The other option I am thinking is to create a dev-username location in > /usr/local somewhere and then ln -s that into my home structure accordingly. > > Since I am new to OpenBSD I figured I would ask first and did not find much > on > this topic other than third parties that seem to want to casually just add > home > to wxallowed. > > Thanks in advance for any guidance, > > Ken Hi, Since moving to OpenBSD, I have moved all my venv to /usr/local (which is wxallowed by default). The folder is owned by my user. I source the proper venv when using each project. $ ls -l drwxr-xr-x7 raph raph 512 Apr 16 11:24 venv/ HTH r.
Re: Best Practices python virtualenv
On Tue, May 01, 2018 at 04:22:50PM +, Ken MacKenzie wrote: > I power my flying car with flying monkeys... > No need to, I am free as a bird, I can power my flying car with my flying wings. Oh, wait, I don't have wings... Anyway, nice joke, but I feel myself embarrasing.
Re: Best Practices python virtualenv
I power my flying car with flying monkeys... I think the best answer from all this comes back to, at least for python 3, switching over to the built in venv in python3. Other than that the symlink to a location in /usr/local for venv is another option for python 2 or packages that need to wxallowed. On the topic of pip install vs pkg_add for python pieces, this relates back to some thoughts I had about dealing with python development on FreeBSD. Let's face it things like docker exist in part because of the mess deploying python virtualenvs can become in production. I had the though of submitting an extension to iocage (my preferred jail manager, which is now written in python) so that you could basically translate a venv + pip workflow into a jail create and pkg add pyv-package type setup. However that also sounds like a big mess to maintain. Maybe something that would be more workable is something that quick translate a pip requirements.txt file into a pkg_add install script. That could work probably for any *BSD flavor. Another option is that in the scope of venv after the pkg is added through pkg_add symlinks are created in venv. I guess the only thing to figure out is what to do when a requirements file has version specifications that are incompatible with what is in the pkg system. Ken. On Tue, May 01, 2018 at 09:46:36AM -0600, Theo de Raadt wrote: > Leonid Bobrov wrote: > > I have a plan how to completely get rid of wxallowed mount option, > > but I am not yet skilled to fix W|X ports, especially the ones > > written in C++ (I've started learning C++ recently). > > Is that like > >"I have a plan to build a flying car, but I don't >yet have any metal work skills" > > Or, aspiring to commit your life to working on world peace, from > your basement? > > Not impressive. >
Re: Best Practices python virtualenv
Leonid Bobrov wrote: > I have a plan how to completely get rid of wxallowed mount option, > but I am not yet skilled to fix W|X ports, especially the ones > written in C++ (I've started learning C++ recently). Is that like "I have a plan to build a flying car, but I don't yet have any metal work skills" Or, aspiring to commit your life to working on world peace, from your basement? Not impressive.
Re: Best Practices python virtualenv
> while there are Python modules which rely on W|X. Yes, but I do not use them. I only run Python and Django. I am aware of the fact that my python is not compatible with some modules, and I am ok with it since I do not need them. > use ports instead of pkg_add to get Python binary without W|X That is exactly what I did: built python from official openbsd port with USE_WXNEEDED=no. I now have python that works from /home, but does not support some modules. While I am still happy with my approach, I'll not suggest it anymore, because everyone else says it is not correct, and I do not want to give bad advice to people on this list:) Fixing every single port that needs WX seems like a huge amount of work. People do not care about W|X security too much outside of openbsd community:(
Re: Best Practices python virtualenv
On Tue, May 01, 2018 at 06:09:59PM +0300, IL Ka wrote: > > > > So you're now on a custom built python and are unable to use standard > > pkg_add upgrades to new versions. I'd say that on balance, this is more > > likely to *reduce* your security. > > > I built it from ports, so I can update it using standard port update > procedure. > Since packages are built from ports, the only difference between default > package > and my package is wxneeded header. > > However, using ports instead of pkg_add is painful. > > To the topic starter: > It seems that I am the only person on this list who uses this approach. > Everyone else (including openbsd veterans) argue against it, > that means I am probably wrong.. It's wrong to get rid of USE_WXNEEDED in Python while there are Python modules which rely on W|X. What you suggest (and I suggested this winter) is not a solution, it only causes more harm. I have a plan how to completely get rid of wxallowed mount option, but I am not yet skilled to fix W|X ports, especially the ones written in C++ (I've started learning C++ recently). Meanwhile you may do the thing I did this winter: use ports instead of pkg_add to get Python binary without W|X
Re: Best Practices python virtualenv
> > So you're now on a custom built python and are unable to use standard pkg_add upgrades to new versions. I'd say that on balance, this is more > likely to *reduce* your security. I built it from ports, so I can update it using standard port update procedure. Since packages are built from ports, the only difference between default package and my package is wxneeded header. However, using ports instead of pkg_add is painful. To the topic starter: It seems that I am the only person on this list who uses this approach. Everyone else (including openbsd veterans) argue against it, that means I am probably wrong..
Re: Best Practices python virtualenv
- On Apr 30, 2018, at 5:46 PM, Ken MacKenzie k...@mack-z.com wrote:
> Is there a recommended best practice when setting up an environment with
> python
> virtualenv with regards to wxallowed.
I use a specific directory in /usr/local which is very close to your option 2:
# mkdir -p /usr/local/virtualenvs/{your users here}
$ virtualenv /usr/local/virtualenvs/my_user/new_project
$ source /usr/local/virtualenvs/my_user/new_project/bin/activate
virtualenvwwrapper also helps in this case
>
> My typical workflow is under my home directory I have a
> dev/language/project/.venv type structure. I guess the simple solution is to
> mount /home as wxallowed in /etc/fstab, but is that truly the preferred way.
> Seems to make a new attack surface for anything in home.
>
> The other option I am thinking is to create a dev-username location in
> /usr/local somewhere and then ln -s that into my home structure accordingly.
>
> Since I am new to OpenBSD I figured I would ask first and did not find much
> on
> this topic other than third parties that seem to want to casually just add
> home
> to wxallowed.
>
> Thanks in advance for any guidance,
>
> Ken
--
Marco Bonetti
Re: Best Practices python virtualenv
On 05/01/18 01:54, IL Ka wrote: Since /usr/local/ has wxallowed by default (see your /etc/fstab) it works. Does it affect security? In theory -- yes, because python can now create WX pages. True. But I use this solution because: - root controls what goes in /usr/local - I mount /home without wxallowed - I mount /usr/local read-only Also, I expect that in the future, less and less packages that I use require /usr/local to be mounted with wxallowed. I agree, this last point is not improving my security at all. -- Etienne
Re: Best Practices python virtualenv
On 2018-05-01, IL Ka wrote: > It is up to you, but I still belive that best solution is to rebuild python > without of wxneeded. > 1) It improves security So you're now on a custom built python and are unable to use standard pkg_add upgrades to new versions. I'd say that on balance, this is more likely to *reduce* your security.
Re: Best Practices python virtualenv
The only difference is venv creates link to python, it does not copy binary itself. You now have python3 -> /usr/local/bin/python3 in your venv. Since /usr/local/ has wxallowed by default (see your /etc/fstab) it works. Does it affect security? In theory -- yes, because python can now create WX pages. One may say that it sounds paranoic, and in fact it is, but OpenBSD is _paranoic_ about security. That is why it has W^X. They left wxallowed for /usr/local because there are a lot of software in ports, written by less "secure paranoics" than openbsd developers, and this software needs WX. Some python packages are good examples. Python port maintainer added wxneeded to python because of these packages. But if you do not use any of these packages, you can disable wxneeded removing any (theoretical) threat that may use WX. That will make python more secure and (as side-effect) fix virtualenv problems. It is less important for developer laptop, but if you can improve security, why not? You will not create "big security hole" with your current approach. You will not create it even by adding wxallowed to /home. So, it is not a critical issue. But I feel that decreasing security by running wxneeded app with out of good reason(if you do not need these packages of course) is not "OpenBSD way":)
Re: Best Practices python virtualenv
Not to disagree but if using python3 -m venv in home works and home is not mounted as wxallowed is there still a security issue with this workflow? Granted at this point talking about a development workstation and not a server. So while I am at it I guess I should ask is what you are saying more from a serving python web app perspective, security wise? Just trying to understand what I seem to be missing. Ken On Tue, May 01, 2018 at 03:18:00AM +0300, IL Ka wrote: > It is up to you, but I still belive that best solution is to rebuild python > without of wxneeded. > 1) It improves security > 2) It fixes your virtualenv issue. > > If you do not use packages that need WX, why do you need wxneed?
Re: Best Practices python virtualenv
It is up to you, but I still belive that best solution is to rebuild python without of wxneeded. 1) It improves security 2) It fixes your virtualenv issue. If you do not use packages that need WX, why do you need wxneed?
Re: Best Practices python virtualenv
I happen to like python and will be the first I reach for for many simple or even some bigger tasks. Nothing against those other languages. I actually have a special place in my heart for perl, but with the perl 5 vs 6 thing I wonder on the longer term future of the language. Honestly I need to get the rust off of my c skills. Ken PS - please don't let me perl comment start any flame wars... please please > My current position is to simply avoid python in favour of c,perl,sh > and even php. >
Re: Best Practices python virtualenv
Thanks for all the responses but it seems an alternate solution presented by
another user in a direct reply is to use python3 - m venv. Basically using the
venv built in to python3 as opposed to the legacy method of py-virtualenv that I
typically only have to use for older python 2 code bases.
Thanks all for the replies.
Ken
On Mon, Apr 30, 2018 at 06:23:32PM -0400, Dave Voutila wrote:
> Ken MacKenzie writes:
>
> > Is there a recommended best practice when setting up an environment with
> > python
> > virtualenv with regards to wxallowed.
>
> AFAIK nothing official.
>
> >
> > My typical workflow is under my home directory I have a
> > dev/language/project/.venv type structure. I guess the simple solution is to
> > mount /home as wxallowed in /etc/fstab, but is that truly the preferred way.
> > Seems to make a new attack surface for anything in home.
> >
>
> Turning off any default mitigations is probably the opposite of "preferred".
>
> > The other option I am thinking is to create a dev-username location in
> > /usr/local somewhere and then ln -s that into my home structure accordingly.
> >
> > Since I am new to OpenBSD I figured I would ask first and did not find much
> > on
> > this topic other than third parties that seem to want to casually just add
> > home
> > to wxallowed.
> >
>
> I've been too lazy to dig into and "fix" this in the py{2,3}-virtualenv
> port. The main issue is it copies the binary for the target python
> executable and doesn't symlink it. It's really not a bug and more an
> adaptation issue so I've not been inclined.
>
> However, symbolic links to /usr/local/bin/python work fine if they're
> located on partitions that aren't mounted wxallowed. I'd imagine if
> virtualenv created a symlink things would "just work."
>
> So what I do, instead of teaching virtualenv to symlink instead of copy,
> is just confine my virtualenvs to /usr/local/venv (owned by root:wsrc).
>
> I then just activate via the usual means of the activate script:
>
> kogelvis$ . /usr/local/venv/my-project/bin/activate
>
> Typically, on other systems, I'd locate it in ~/.venv but for my
> personal machine it's not an issue.
>
> I do set an environment variable:
>
> WORKON_HOME=/usr/local/venv
>
> in .profile so I can configure tools like emacs major modes to point to
> where I want them to create/find virtual environments.
>
> > Thanks in advance for any guidance,
> >
> > Ken
Re: Best Practices python virtualenv
Ken MacKenzie writes:
> Is there a recommended best practice when setting up an environment with
> python
> virtualenv with regards to wxallowed.
AFAIK nothing official.
>
> My typical workflow is under my home directory I have a
> dev/language/project/.venv type structure. I guess the simple solution is to
> mount /home as wxallowed in /etc/fstab, but is that truly the preferred way.
> Seems to make a new attack surface for anything in home.
>
Turning off any default mitigations is probably the opposite of "preferred".
> The other option I am thinking is to create a dev-username location in
> /usr/local somewhere and then ln -s that into my home structure accordingly.
>
> Since I am new to OpenBSD I figured I would ask first and did not find much
> on
> this topic other than third parties that seem to want to casually just add
> home
> to wxallowed.
>
I've been too lazy to dig into and "fix" this in the py{2,3}-virtualenv
port. The main issue is it copies the binary for the target python
executable and doesn't symlink it. It's really not a bug and more an
adaptation issue so I've not been inclined.
However, symbolic links to /usr/local/bin/python work fine if they're
located on partitions that aren't mounted wxallowed. I'd imagine if
virtualenv created a symlink things would "just work."
So what I do, instead of teaching virtualenv to symlink instead of copy,
is just confine my virtualenvs to /usr/local/venv (owned by root:wsrc).
I then just activate via the usual means of the activate script:
kogelvis$ . /usr/local/venv/my-project/bin/activate
Typically, on other systems, I'd locate it in ~/.venv but for my
personal machine it's not an issue.
I do set an environment variable:
WORKON_HOME=/usr/local/venv
in .profile so I can configure tools like emacs major modes to point to
where I want them to create/find virtual environments.
> Thanks in advance for any guidance,
>
> Ken
Re: Best Practices python virtualenv
Hello. Short answer: if you do not need py-cryptography and QtWebKit, just rebuild python from ports disabling USE_WXNEEDED. I run Django using virtualenv in my $HOME and it works. Long answer: To use mmap(2) with PROT_EXEC | PROT_WRITE you need to link binary with -z wxneeded (See ld(1)). It adds OPENBSD_WXNEEDED header to binary. You can check it with $ readelf -a /usr/local/bin/python2 | grep WX There should be "OPENBSD_WXNEED". With out of it, mmap(2) returns "Not supported" for such requests. When mounted with out of wxallowed, binaries with this header can't be executed. You will get "Permission denied". At some moment, people found that py-cryptography and QtWebKit need this header, so they added USE_WXNEEDED=yes to port. http://openbsd-archive.7691.n7.nabble.com/Patch-Python-for-non-wxallowed-usr-local-td335767.html See /usr/ports/lang/python/Makefile.inc See also /usr/ports/infrastructure/mk/bsd.port.mk for how this option is used and how Makefile.inc is included. /usr/local has wxallowed, so py-cryptography and QtWebKit works there. But they did not care about virtualenv. We now have python linked with wxneeded by default: $ readelf -a /usr/local/bin/python2 | grep WX OPENBSD_WXNEED 0x 0x 0x and its execution is not allowed on FS mounted with out of wxallowed. To fix it, simply rebuild python with this option disabled. Better solution is to create to flavors of python (with this option and with out), but that has not been done. Ilya.
Re: Best Practices python virtualenv
On Mon, 30 Apr 2018 16:46:49 + > Is there a recommended best practice when setting up an environment > with python > virtualenv with regards to wxallowed. sthen@ mentioned a compiler option and that only some modules actually require RWX memory. I don't know the details as to why python calls rwx memory in any case but perhaps there could be a python norwx package as currently with /usr/local without wxallowed then even simple tools like obmenu are killed. My current position is to simply avoid python in favour of c,perl,sh and even php.
Best Practices python virtualenv
Is there a recommended best practice when setting up an environment with python virtualenv with regards to wxallowed. My typical workflow is under my home directory I have a dev/language/project/.venv type structure. I guess the simple solution is to mount /home as wxallowed in /etc/fstab, but is that truly the preferred way. Seems to make a new attack surface for anything in home. The other option I am thinking is to create a dev-username location in /usr/local somewhere and then ln -s that into my home structure accordingly. Since I am new to OpenBSD I figured I would ask first and did not find much on this topic other than third parties that seem to want to casually just add home to wxallowed. Thanks in advance for any guidance, Ken
