Re: python
On Thu, 11 Apr 2019 09:29:06 +0200 Stefan Esser wrote: > Am 11.04.19 um 00:03 schrieb ajtiM via freebsd-ports: > > Hi! > > > > With portmaster I try to update todays ports and python default is > > version 3.7 which is okay for me but not for hplip: > > ===>>> All >> > > hplip-3.17.11_3 (5/43) > > > > ===>>> Returning to dependency check for print/hplip > > ===>>> Dependency check complete for print/hplip > > > > ===>>> All >> hplip-3.17.11_3 (5/43) > > > > ===> hplip-3.17.11_4 FLAVOR is defined (to py27) while this port > > does not have FLAVORS.. > > *** Error code 1 > > > > Stop. > > make: stopped in /usr/ports/print/hplip > > This is a bad effect caused by the removal of FLAVORs from an > installed port. > > Since the previous revision of this port required a flavor, it > will be updated with that same flavor passed as a parameter to > the build process. The build could just ignore the irrelevant > flavor, but that is not what the ports system does ... > > In this particular case the removal of QT4 is the cause. The > port used to support flavors "qt4" and "qt5" and to fix this > issue for portmaster, entries in MOVED should exist that make > upgrades use print/hplip without flavor, whether the qt4 or > qt5 version was installed. > > I plan to add code to portmaster to verify that a FLAVOR that > is to be used for a port still applies, before starting to > build it. > > But it is not obvious to me what to do for all of the cases > that exist: > > 1) FLAVORS removed from the port -> build without flavors > > 2) Specific FLAVOR removed from the port -> build with default >flavor (?) > > Anyway, for the time being you'll have to deinstall the port > (use "pkg delete -f hplip" to prevent recursive deletion of > dependent ports) and then re-install it with portmaster (which > will build it without flavor, then. > > Another possibility is to remove the "flavor" annotation > registered for this port in the PKGDB, then portmaster will > upgrade the port without providing a FLAVOR to the build ... > > I'll see that I fix this problem in portmaster, but it will take > some time ... > > Regards, STefan > ___ Thank you very much. -- by ajtiM -- FreeBSD 12.0-Release ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: python 3 subprocess performance
The reason is that that python does something stupid (tm). It tries to close all file descriptors (except a few whitelisted ones) up to the maximum file descriptor number. It does this by asking the kernel for the maximum possible number and closing everything it doesn't want to keep. Some time later someone came up with an optimization (read the open file descriptors from /dev/fd). All of this pain and suffering is caused by good old Ulrich Drepper braindamage: https://sourceware.org/bugzilla/show_bug.cgi?id=10353. Most Linux distros have lower default file descriptor limits than FreeBSD making this workaround less painful. The correct solution would be to teach python3 about closefrom(2). On 11.04.19 16:39, Konstantin Schukraft wrote: >> Please run python under truss -f. Does it try to close(2) all possible >> file descriptors? Does the runtime decrease if fdescfs is mounted at >> /dev/fd? > truss -f -p 17407 > > 17407: read(5,0x1265d2e9120,5) = 0 (0x0) > 17407: close(5) = 0 (0x0) > 17407: lseek(3,0x0,SEEK_CUR) ERR#29 'Illegal seek' > 17407: fstat(3,{ mode=p- ,inode=1610971,size=0,blksize=4096 }) = > 0 (0x0) > 17407: read(3,"FreeBSD midgard 12.0-STABLE-HBSD"...,8192) = 88 (0x58) > 17407: read(3,0x1265ceda078,8104) = 0 (0x0) > 17407: close(3) = 0 (0x0) > 17407: wait4(1948,{ EXITED,val=0 },0x0,0x0) = 1948 (0x79c) > 17407: write(1,"63\n",3) = 3 (0x3) > 17407: pipe2(0x63857dad22e8,O_CLOEXEC) = 0 (0x0) > 17407: fstat(3,{ mode=p- ,inode=1610975,size=0,blksize=4096 }) = > 0 (0x0) > 17407: ioctl(3,TIOCGETA,0x63857dad21a0) ERR#25 'Inappropriate > ioctl for device' > 17407: lseek(3,0x0,SEEK_CUR) ERR#29 'Illegal seek' > 17407: pipe2(0x63857dad21e8,O_CLOEXEC) = 0 (0x0) > 17407: sigprocmask(SIG_BLOCK,{ > SIGHUP|SIGINT|SIGQUIT|SIGILL|SIGTRAP|SIGABRT|SIGEMT|SIGFPE|SIGKILL|SIGBUS|SIGSEGV|SIGSYS|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF| > > SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2 },{ }) = 0 (0x0) > 97164: > 17407: fork() = 97164 (0x17b8c) > 97164: thr_self(0x1265c7e9000) = 0 (0x0) > 17407: sigprocmask(SIG_SETMASK,{ },0x0) = 0 (0x0) > 97164: sigprocmask(SIG_SETMASK,{ },0x0) = 0 (0x0) > 97164: close(3) = 0 (0x0) > 97164: close(5) = 0 (0x0) > 17407: close(6) = 0 (0x0) > 97164: dup2(4,1) = 1 (0x1) > 17407: close(4) = 0 (0x0) > 97164: dup2(4,2) = 2 (0x2) > 97164: sigprocmask(SIG_SETMASK,{ > SIGHUP|SIGINT|SIGQUIT|SIGILL|SIGTRAP|SIGABRT|SIGEMT|SIGFPE|SIGKILL|SIGBUS|SIGSEGV|SIGSYS|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPRO > > F|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2 },{ }) = 0 (0x0) > 97164: sigaction(SIGPIPE,{ SIG_DFL 0x0 ss_t },{ SIG_IGN 0x0 ss_t }) = 0 > (0x0) > 97164: sigprocmask(SIG_SETMASK,{ },0x0) = 0 (0x0) > 97164: sigprocmask(SIG_SETMASK,{ > SIGHUP|SIGINT|SIGQUIT|SIGILL|SIGTRAP|SIGABRT|SIGEMT|SIGFPE|SIGKILL|SIGBUS|SIGSEGV|SIGSYS|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPRO > > F|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2 },{ }) = 0 (0x0) > 97164: sigaction(SIGXFSZ,{ SIG_DFL 0x0 ss_t },{ SIG_IGN 0x0 ss_t }) = 0 > (0x0) > 97164: sigprocmask(SIG_SETMASK,{ },0x0) = 0 (0x0) > 97164: fstatat(AT_FDCWD,"/dev",{ mode=dr-xr-xr-x > ,inode=2,size=512,blksize=4096 },0x0) = 0 (0x0) > 97164: fstatat(AT_FDCWD,"/dev/fd",{ mode=dr-xr-xr-x > ,inode=135,size=512,blksize=4096 },0x0) = 0 (0x0) > 97164: getrlimit(RLIMIT_NOFILE,{ cur=940563,max=940563 }) = 0 (0x0) > 97164: close(3) ERR#9 'Bad file > descriptor' > 97164: close(4) = 0 (0x0) > 97164: close(5) ERR#9 'Bad file > descriptor' > 97164: close(7) ERR#9 'Bad file > descriptor' > 97164: close(8) ERR#9 'Bad file > descriptor' > 97164: close(9) ERR#9 'Bad file > descriptor' > > The last error repeats several 1 times. > > Mounting fdescfs does indeed speed python3 up to python2 levels. > >> On 11.04.19 15:16, Alexander Zagrebin wrote: >>> Hi! >>> >>> I've noticed the subprocess performance issue with python 3. >>> For example, this simple script takes just 0,15 second to complete >>> with python 2.7, but more than 5 sec with python 3.6: >>> >>> import subprocess >>> >>> for i in range(100): >>> p = subprocess.Popen(['uname', '-a'], stdout=subprocess.PIPE, >>> stderr=subprocess.STDOUT) >>>
Re: python 3 subprocess performance
Please run python under truss -f. Does it try to close(2) all possible file descriptors? Does the runtime decrease if fdescfs is mounted at /dev/fd? truss -f -p 17407 17407: read(5,0x1265d2e9120,5) = 0 (0x0) 17407: close(5) = 0 (0x0) 17407: lseek(3,0x0,SEEK_CUR) ERR#29 'Illegal seek' 17407: fstat(3,{ mode=p- ,inode=1610971,size=0,blksize=4096 }) = 0 (0x0) 17407: read(3,"FreeBSD midgard 12.0-STABLE-HBSD"...,8192) = 88 (0x58) 17407: read(3,0x1265ceda078,8104)= 0 (0x0) 17407: close(3) = 0 (0x0) 17407: wait4(1948,{ EXITED,val=0 },0x0,0x0) = 1948 (0x79c) 17407: write(1,"63\n",3) = 3 (0x3) 17407: pipe2(0x63857dad22e8,O_CLOEXEC) = 0 (0x0) 17407: fstat(3,{ mode=p- ,inode=1610975,size=0,blksize=4096 }) = 0 (0x0) 17407: ioctl(3,TIOCGETA,0x63857dad21a0) ERR#25 'Inappropriate ioctl for device' 17407: lseek(3,0x0,SEEK_CUR) ERR#29 'Illegal seek' 17407: pipe2(0x63857dad21e8,O_CLOEXEC) = 0 (0x0) 17407: sigprocmask(SIG_BLOCK,{ SIGHUP|SIGINT|SIGQUIT|SIGILL|SIGTRAP|SIGABRT|SIGEMT|SIGFPE|SIGKILL|SIGBUS|SIGSEGV|SIGSYS|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF| SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2 },{ }) = 0 (0x0) 97164: 17407: fork()= 97164 (0x17b8c) 97164: thr_self(0x1265c7e9000) = 0 (0x0) 17407: sigprocmask(SIG_SETMASK,{ },0x0) = 0 (0x0) 97164: sigprocmask(SIG_SETMASK,{ },0x0) = 0 (0x0) 97164: close(3) = 0 (0x0) 97164: close(5) = 0 (0x0) 17407: close(6) = 0 (0x0) 97164: dup2(4,1) = 1 (0x1) 17407: close(4) = 0 (0x0) 97164: dup2(4,2) = 2 (0x2) 97164: sigprocmask(SIG_SETMASK,{ SIGHUP|SIGINT|SIGQUIT|SIGILL|SIGTRAP|SIGABRT|SIGEMT|SIGFPE|SIGKILL|SIGBUS|SIGSEGV|SIGSYS|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPRO F|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2 },{ }) = 0 (0x0) 97164: sigaction(SIGPIPE,{ SIG_DFL 0x0 ss_t },{ SIG_IGN 0x0 ss_t }) = 0 (0x0) 97164: sigprocmask(SIG_SETMASK,{ },0x0) = 0 (0x0) 97164: sigprocmask(SIG_SETMASK,{ SIGHUP|SIGINT|SIGQUIT|SIGILL|SIGTRAP|SIGABRT|SIGEMT|SIGFPE|SIGKILL|SIGBUS|SIGSEGV|SIGSYS|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPRO F|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2 },{ }) = 0 (0x0) 97164: sigaction(SIGXFSZ,{ SIG_DFL 0x0 ss_t },{ SIG_IGN 0x0 ss_t }) = 0 (0x0) 97164: sigprocmask(SIG_SETMASK,{ },0x0) = 0 (0x0) 97164: fstatat(AT_FDCWD,"/dev",{ mode=dr-xr-xr-x ,inode=2,size=512,blksize=4096 },0x0) = 0 (0x0) 97164: fstatat(AT_FDCWD,"/dev/fd",{ mode=dr-xr-xr-x ,inode=135,size=512,blksize=4096 },0x0) = 0 (0x0) 97164: getrlimit(RLIMIT_NOFILE,{ cur=940563,max=940563 }) = 0 (0x0) 97164: close(3) ERR#9 'Bad file descriptor' 97164: close(4) = 0 (0x0) 97164: close(5) ERR#9 'Bad file descriptor' 97164: close(7) ERR#9 'Bad file descriptor' 97164: close(8) ERR#9 'Bad file descriptor' 97164: close(9) ERR#9 'Bad file descriptor' The last error repeats several 1 times. Mounting fdescfs does indeed speed python3 up to python2 levels. On 11.04.19 15:16, Alexander Zagrebin wrote: Hi! I've noticed the subprocess performance issue with python 3. For example, this simple script takes just 0,15 second to complete with python 2.7, but more than 5 sec with python 3.6: import subprocess for i in range(100): p = subprocess.Popen(['uname', '-a'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT) (stdoutdata, stderrdata) = p.communicate() Profiling with the cProfile shows, that this excessive 5 seconds was wasted in the "{built-in method posix.read}" Could anybody confirm this issue? ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org" signature.asc Description: PGP signature
Re: FreeBSD Port: rust-cbindgen-0.8.3_1
"Alex V. Petrov" writes: > error: internal compiler error: > src/librustc/middle/mem_categorization.rs:490: no type for node 190935: > expr err (id=190935) in mem_categorization Can you build any other rust-based port, including lang/rust itself? If not make sure you have COMPAT_FREEBSD11 in kernel config. This port builds fine on the package cluster: http://beefy9.nyi.freebsd.org/data/112amd64-default/498047/logs/rust-cbindgen-0.8.3_1.log http://beefy10.nyi.freebsd.org/data/112i386-default/498047/logs/rust-cbindgen-0.8.3_1.log http://beefy6.nyi.freebsd.org/data/120amd64-default/498047/logs/rust-cbindgen-0.8.3_1.log http://beefy5.nyi.freebsd.org/data/120i386-default/498047/logs/rust-cbindgen-0.8.3_1.log http://www.ipv6proxy.net/go.php?u=http://beefy12.nyi.freebsd.org/data/head-amd64-default/p498047_s345970/logs/rust-cbindgen-0.8.3_1.log http://www.ipv6proxy.net/go.php?u=http://beefy11.nyi.freebsd.org/data/head-i386-default/p498047_s345970/logs/rust-cbindgen-0.8.3_1.log http://thunderx1.nyi.freebsd.org/data/head-arm64-default/p497373_s345741/logs/rust-cbindgen-0.8.2.log http://pylon.nyi.freebsd.org/data/head-powerpc64-default/p497543_s345783/logs/rust-cbindgen-0.8.2.log ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: python 3 subprocess performance
Please run python under truss -f. Does it try to close(2) all possible file descriptors? Does the runtime decrease if fdescfs is mounted at /dev/fd? On 11.04.19 15:16, Alexander Zagrebin wrote: > Hi! > > I've noticed the subprocess performance issue with python 3. > For example, this simple script takes just 0,15 second to complete > with python 2.7, but more than 5 sec with python 3.6: > > import subprocess > > for i in range(100): > p = subprocess.Popen(['uname', '-a'], stdout=subprocess.PIPE, > stderr=subprocess.STDOUT) > (stdoutdata, stderrdata) = p.communicate() > > Profiling with the cProfile shows, that this excessive 5 seconds was > wasted in the "{built-in method posix.read}" > > Could anybody confirm this issue? > ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: python 3 subprocess performance
Hi, I've noticed the subprocess performance issue with python 3. For example, this simple script takes just 0,15 second to complete with python 2.7, but more than 5 sec with python 3.6: import subprocess for i in range(100): p = subprocess.Popen(['uname', '-a'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT) (stdoutdata, stderrdata) = p.communicate() Profiling with the cProfile shows, that this excessive 5 seconds was wasted in the "{built-in method posix.read}" Could anybody confirm this issue? I see this on HardenedBSD 12 with python2.7 (fast) and python3.6 and 3.7 (both slow) installed from packages. Interestingly, I don't see this on my FreeNAS 11.3, where python2.7 and 3.6 are both installed by default and run fast. Hope that helps, Konstantin signature.asc Description: PGP signature
python 3 subprocess performance
Hi! I've noticed the subprocess performance issue with python 3. For example, this simple script takes just 0,15 second to complete with python 2.7, but more than 5 sec with python 3.6: import subprocess for i in range(100): p = subprocess.Popen(['uname', '-a'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT) (stdoutdata, stderrdata) = p.communicate() Profiling with the cProfile shows, that this excessive 5 seconds was wasted in the "{built-in method posix.read}" Could anybody confirm this issue? -- Alexander Zagrebin ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
FreeBSD Port: rust-cbindgen-0.8.3_1
I can't update firefox, error in devel/rust-cbindgen: Running `/usr/local/bin/rustc --crate-name serde /usr/ports/devel/rust-cbindgen/work/cbindgen-0.8.3/cargo-crates/serde-1.0.90/src/lib.rs --color always --crate-type lib --emit=dep-info,link -C opt-level=2 --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="serde_derive"' --cfg 'feature="std"' -C metadata=da5cf54a7597bda2 -C extra-filename=-da5cf54a7597bda2 --out-dir /usr/ports/devel/rust-cbindgen/ work/target/release/deps -L dependency=/usr/ports/devel/rust-cbindgen/work/target/release/deps --extern serde_derive=/usr/ports/devel/rust-cbindgen/work/target/release/deps/libserde_derive-87a3892b64f18330.so --cap-lints allow -C linker=cc -C link-arg=-fstack-protector --cfg ops_bound --cfg core_reverse --cfg de_boxed_c_str --cfg de_rc_dst --cfg core_duration --cfg integer128 --cfg range_inclusive --cfg num_nonzero` error: internal compiler error: src/librustc/middle/mem_categorization.rs:490: no type for node 190935: expr err (id=190935) in mem_categorization thread 'rustc' panicked at 'Box', src/librustc_errors/lib.rs:588:9 note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace. error: aborting due to previous error note: the compiler unexpectedly panicked. this is a bug. note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports note: rustc 1.33.0 running on x86_64-unknown-freebsd note: compiler flags: -C opt-level=2 -C linker=cc -C link-arg=-fstack-protector --crate-type lib note: some of the compiler flags provided by cargo are hidden error: Could not compile `serde`. Caused by: process didn't exit successfully: `/usr/local/bin/rustc --crate-name serde /usr/ports/devel/rust-cbindgen/work/cbindgen-0.8.3/cargo-crates/serde-1.0.90/src/lib.rs --color always --crate-type lib --emit=dep-info,link -C opt-level=2 --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="serde_derive"' --cfg 'feature="std"' -C metadata=da5cf54a7597bda2 -C extra-filename=-da5cf54a7597bda2 --out-dir /usr/po$ts/devel/rust-cbindgen/work/target/release/deps -L dependency=/usr/ports/devel/rust-cbindgen/work/target/release/deps --extern serde_derive=/usr/ports/devel/rust-cbindgen/work/target/release/deps/libserde_derive-87a3892b64f18330.so --cap-lints allow -C linker=cc -C link-arg=-fstack-protector --cfg ops_bound --cfg core_reverse --cfg de_boxed_c_str --cfg de_rc_dst --cfg core_duration --cfg integer128 --cfg range_inclusive --cfg num_nonzero` (exit code: 101) *** Error code 101 Stop. make: stopped in /usr/ports/devel/rust-cbindgen ===>>> make build failed for devel/rust-cbindgen ===>>> Aborting update ===>>> Update for devel/rust-cbindgen failed ===>>> Aborting update ===>>> Update for www/firefox failed ===>>> Aborting update ===>>> The following actions were performed: Installation of lang/rust (rust-1.33.0_1) -- - Alex. ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: python
Am 11.04.19 um 00:03 schrieb ajtiM via freebsd-ports: > Hi! > > With portmaster I try to update todays ports and python default is > version 3.7 which is okay for me but not for hplip: > ===>>> All >> > hplip-3.17.11_3 (5/43) > > ===>>> Returning to dependency check for print/hplip > ===>>> Dependency check complete for print/hplip > > ===>>> All >> hplip-3.17.11_3 (5/43) > > ===> hplip-3.17.11_4 FLAVOR is defined (to py27) while this port does > not have FLAVORS.. > *** Error code 1 > > Stop. > make: stopped in /usr/ports/print/hplip This is a bad effect caused by the removal of FLAVORs from an installed port. Since the previous revision of this port required a flavor, it will be updated with that same flavor passed as a parameter to the build process. The build could just ignore the irrelevant flavor, but that is not what the ports system does ... In this particular case the removal of QT4 is the cause. The port used to support flavors "qt4" and "qt5" and to fix this issue for portmaster, entries in MOVED should exist that make upgrades use print/hplip without flavor, whether the qt4 or qt5 version was installed. I plan to add code to portmaster to verify that a FLAVOR that is to be used for a port still applies, before starting to build it. But it is not obvious to me what to do for all of the cases that exist: 1) FLAVORS removed from the port -> build without flavors 2) Specific FLAVOR removed from the port -> build with default flavor (?) Anyway, for the time being you'll have to deinstall the port (use "pkg delete -f hplip" to prevent recursive deletion of dependent ports) and then re-install it with portmaster (which will build it without flavor, then. Another possibility is to remove the "flavor" annotation registered for this port in the PKGDB, then portmaster will upgrade the port without providing a FLAVOR to the build ... I'll see that I fix this problem in portmaster, but it will take some time ... Regards, STefan ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"