[PATCH] Makefile: replace realpath with pwd

[Justin Cinkelj]

The realpath is not installed by default on Ubuntu.
Use pwd instead.

Fixes #946

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 modules/cli/rpmbuild/Makefile   | 2 +-
 modules/httpserver-api/Makefile | 2 +-
 modules/lua/Makefile| 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/modules/cli/rpmbuild/Makefile b/modules/cli/rpmbuild/Makefile
index ab52ea6..be9915d 100644
--- a/modules/cli/rpmbuild/Makefile
+++ b/modules/cli/rpmbuild/Makefile
@@ -1,4 +1,4 @@
-ROOT_DIR := $(shell dirname $(realpath $(lastword $(MAKEFILE_LIST
+ROOT_DIR := $(shell pwd)
 CLI_ROOT=$(ROOT_DIR)/..
 LUA_DIR=$(CLI_ROOT)/../lua
 VERSION=1.0
diff --git a/modules/httpserver-api/Makefile b/modules/httpserver-api/Makefile
index 3737b94..19cc526 100644
--- a/modules/httpserver-api/Makefile
+++ b/modules/httpserver-api/Makefile
@@ -54,7 +54,7 @@ all: lib$(TARGET).so api_api api_app api_env api_file api_fs 
api_hardware api_ne
$(call quiet, cat _usr_*.manifest | sort | uniq > usr.manifest, 
CREATE_MANIFEST)
 
 add_api_to_manifest = \
-   echo "/usr/mgmt/plugins/libhttpserver-$(1).so: `realpath 
libhttpserver-$(1).so`" > _usr_$(1).manifest
+   echo "/usr/mgmt/plugins/libhttpserver-$(1).so: 
`pwd`/libhttpserver-$(1).so" > _usr_$(1).manifest
 
 add_boost_dependencies_to_manifest = \
ldd libhttpserver-$(1).so | grep boost | sed 's/ *[^ ] *\(.*\) => 
\(.*\) .*/\/usr\/lib\/\1: \2/' > _usr_$(1)_boost.manifest
diff --git a/modules/lua/Makefile b/modules/lua/Makefile
index 9676f34..620a78f 100644
--- a/modules/lua/Makefile
+++ b/modules/lua/Makefile
@@ -71,7 +71,7 @@ $(BUILD)/$(LuaRocks_F)/src/luarocks: 
$(BUILD)/$(LuaRocks_F)/Makefile
make build
 
 $(BUILD)/$(LuaRocks_F)/Makefile: $(BUILD)/$(LuaRocks_F)/configure
-   $(eval PREFIX=$(shell dirname $(realpath $(lastword $(MAKEFILE_LIST)
+   $(eval PREFIX=$(shell pwd))
cd $(BUILD)/$(LuaRocks_F) && \
./configure --prefix=$(PREFIX)/$(OUT) --force-config \
--with-lua-include=$(PREFIX)/src --lua-version=$(LUA_V) \
-- 
1.9.1

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH v2] mkbootfs.py: reuse manifest_common - should fix #905

[Justin Cinkelj]

With v2 applied, I got:
scripts/build mode=debug image=cli,nginx -j8 fs=ramfs
...
  LINK build/debug.x64/tools/uush/uush.so
  LINK cpiod.so
  MKBOOTFS build/debug.x64/bootfs.bin
Traceback (most recent call last):
  File 
"/home/justin_cinkelj/devel/mikelangelo/osv-fc25/osv/scripts/mkbootfs.py", 
line 96, in 
main()
  File 
"/home/justin_cinkelj/devel/mikelangelo/osv-fc25/osv/scripts/mkbootfs.py", 
line 69, in main
size = os.stat(hostname).st_size
OSError: [Errno 2] No such file or directory: 
'%(gccbase)s/lib64/libgcc_s.so.1'
make: *** [build/debug.x64/bootfs.bin] Error 1
make: *** Deleting file `build/debug.x64/bootfs.bin'
make failed. Exiting from build script
#
xlab@osv-1:/home/justin_cinkelj/devel/mikelangelo/osv-fc25/osv$ find ./ 
-iname libgcc_s.so.1
./external/x64/gcc.bin/lib64/libgcc_s.so.1
./external/aarch64/gcc.bin/lib64/libgcc_s.so.1

Interesting, that :
scripts/build mode=debug image=cli,nginx -j8 fs=ramfs
scripts/build mode=debug image=cli,nginx -j8
  # doesn't finsh, make hangs at 100% CPU time 
scripts/build mode=debug image=cli,nginx -j1
  # image is built
scripts/build mode=debug image=cli,nginx -j8
  # now that one builts too
But that second detail is not related to that patch.


-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: osv-cassandra on openstack & cloudinit

[Justin Cinkelj]

Hi

Possible that --force flag is required to actually attempt to load config 
from openstack cloudinit (I'm not sure; maybe it was required for ISO image 
"nocloud" cloudinit last time I tried this).
CC miha for possible hint.

> From capstan repo
There are (at least) two capstan version now, the original one from 
https://github.com/cloudius-systems/capstan and fork at 
https://github.com/mikelangelo-project/capstan-packages.

> yet it gest its network config
I guess that's dhcp client.

On Tuesday, November 28, 2017 at 6:04:30 PM UTC+1, DEGAT Yann wrote:
>
> hi ppl
>   
> got stuck on osv on openstack   
>
> used the lastest osv-cassandra img available in the captsan repo
> index.yaml says it has been built with cloudinit & httpserver support
> yet 
> no way to pass userdata conf to my instances
>
> its seems to be completely ignored, yet it gets its network config
> dont know what to do
> whatever i put in the cloudinit file i give to nova boot, even broken 
> yaml, it doens't change anything to the boot of my instances
>
>  they boot as if i hadn't give any userdata
>
> any suggestion on this?
>
> thanks a lot
> -yann
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Enhance ramfs/zfs-less images

[Justin Cinkelj]

Symlinks were added to ramfs not so long ago in 
https://github.com/cloudius-systems/osv/commit/18a2e45c13d61832de4b2fc37131035cb36fa568.
 
Mentioning that just in case it helps - I noticed +#define mfs_symlink ((
vnop_symlink_t)vop_nullop), but didn't look deeper into code.

> with much printing to the console the timings gets affected
You could use timed tracepoints to measure time - 
https://github.com/cloudius-systems/osv/wiki/Trace-analysis-using-trace.py#timed-tracepoints.

And thank you for compile with zfs, then again with ramfs trick when 
changing lzkernel_base; I didn't notice that helps.

Justin

On Sunday, November 19, 2017 at 4:52:18 PM UTC+1, Waldek Kozaczuk wrote:
>
> For reference I am including the link to my mfs branch - 
> https://github.com/wkozaczuk/osv/tree/readonly_mfs - that includes all my 
> changes.
>
> Also I forgot to mention that I discover that mfs does not support links 
> to directories. I had to manually tweak java image.
>
> On Sunday, November 19, 2017 at 10:45:19 AM UTC-5, Waldek Kozaczuk wrote:
>>
>> I have applied the patch produced by James Root. I had to manually tweak 
>> Makefile and change gen-mfs-img.py by making it reuse manifest_common.py.
>>
>> Overall I think it works well and as designed. BTW I can send new MFS 
>> patch which would be original one plus my changes explained above.
>>
>> However I ran various tests to benchmark it and the results were not as 
>> great as I expected especially for bigger images like Java. On other hand 
>> it is much faster than ZFS for small images. Please see the numbers below.
>>
>> In essence I tested simple minimal Java "hello world" (./scripts/build 
>> image=openjdk9-java-base,java-example fs=) and native one 
>> (./scripts/build image=native-example fs=. I also added a stop watch 
>> around java app main (modules/java-base/java.cc). I assumed that loading 
>> and executing it (under 10K) would be negligible. Also I did not measure 
>> whole run by doing "time ./scripts/run ..." as this would include loading 
>> and starting QEMU which is besides what we need to measure.
>>
>> As you can see both native ramfs and mfs images boot in 100-110 ms where 
>> zfs adds ~ 1,400 ms (it seems my machine is slow as others report similar 
>> small images with ZFS to boot in 600-700ms). 
>>
>> Java on other hand was different. The fastest one was ramfs, 2nd zfs and 
>> slowest mfs. In both ramfs and zfs test Java app was able to run in ~ 200ms 
>> (which probably includes loading libjvm.so (17MB) and other files, 
>> initializing JVM, compiling byte code to native code and executing which 
>> makes 200ms pretty impressive). In mfs case as you see the boot time is 
>> very fast as fs is mounted rapidly but Java app executions takes over 3 
>> seconds :-(
>>
>> I added more stop watches in mfs code around mfsvnops.cc/mfs_read() to 
>> see where time gets spent. In case of Java example it loads 15 files/inodes 
>> with 2 bulky ones. Please see count of mfs_read calls:
>>
>>   5 of inode 110
>>  51 of inode 124
>>  28 of inode 125
>>  32 of inode 127
>> *630 of inode 128 // 17MB of /usr/lib/jvm/java/lib/server/libjvm.so*
>> *   2457 of inode 131 // 8MB of /usr/lib/jvm/java/lib/modules (Java 9 
>> replacement of rt.jar)*
>>  17 of inode 133
>>  31 of inode 134
>>  34 of inode 135
>>   1 of inode 32
>>  35 of inode 5
>>   1 of inode 55
>>  24 of inode 87
>>  19 of inode 88
>>  53 of inode 9
>>
>> Obviously with much printing to the console the timings gets affected and 
>> entire java run is 9,000ms where 5586.47 ms is spent on mfs_read.
>>
>> As I understand during the java mfs image execution dlopen loads 
>> libjvm.so file which involves calling mfs_read (4096 bytes) where each 
>> involves 8 calls to bread. So it looks the system ends up reading 8 times 
>> 2457 blocks of 512 bytes (the math does not seem to work as it means 
>> only 9828 KB of libvm.so gets loaded - shouldn't dlopen read all 17MBs of 
>> libjvm.so?). Please see attached java_mfs2.log that has all the stopwatches 
>> (node 131 - libjvm.so).
>>
>> Is there a way to speed it up. It loading 512 bytes (single block) at a 
>> time the only way to load data from disk? Why when ramfs image loads all 
>> files in real mode it takes under 200ms - see "disk read (real mode): 
>> 148.11ms, (+148.11ms)" line from bootchart of ramfs java benchmark and it 
>> takes over 3 seconds to load less data. Is it because it does it through 
>> all layers?
>>
>> Would it help if entire file was loaded at once on a first read into 
>> memory and then eventually released? Is it possible to load content in 
>> bigger chunks than 512 bytes from disk?
>>
>> I also wonder if it helped if the information about inodes in form of 
>> some kind of map (file path -> inode) was preloaded when mounting mfs?
>>
>> Any ideas?
>>
>> Waldek 
>>
>> PS1. Also BTW there is some problem with building ramfs images. Everytime 
>> one changes lzkernel_base 

[PATCH] cpiod: add verbose program option

[Justin Cinkelj]

cpiod started with "--verbose 0" will not print line for each uploaded
file. Per https://groups.google.com/d/msg/osv-dev/ZsD57qBaWM4/vNp7Fno8BAAJ
this output significantly slows down capstan image compose:
13 vs 5 secundes, and on VirtualBox even ~10 minutes to ~ 10 SECONDS.

By default verbose output is left enabled, so capstan is expected to start
cpiod with extra option.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 tools/cpiod/cpiod.cc | 29 -
 1 file changed, 24 insertions(+), 5 deletions(-)

diff --git a/tools/cpiod/cpiod.cc b/tools/cpiod/cpiod.cc
index d46ec6e6..5305b94c 100644
--- a/tools/cpiod/cpiod.cc
+++ b/tools/cpiod/cpiod.cc
@@ -52,9 +52,23 @@ static void change_mode(std::string path, mode_t mode) {
 
 class cpio_in_expand : public cpio_in {
 public:
-cpio_in_expand(std::string prefix): _prefix(prefix) {};
+class logger {
+public:
+logger(bool verbose): _verbose(verbose) {};
+template
+logger& operator << (const T& x) {
+if (_verbose) {
+std::cout << x;
+}
+return *this;
+}
+public:
+bool _verbose;
+};
+public:
+cpio_in_expand(std::string prefix, bool verbose): _prefix(prefix), 
_log(verbose) {};
 virtual void add_file(string name, istream& is, mode_t mode) override {
-cout << "Adding " << name << "...\n";
+_log << "Adding " << name << "...\n";
 name = add_prefix(name);
 make_directories(parent_path(name));
 ofstream os(name);
@@ -62,13 +76,13 @@ public:
 change_mode(name, mode);
 }
 virtual void add_dir(string name, mode_t mode) override {
-cout << "Adding " << name << "...\n";
+_log << "Adding " << name << "...\n";
 name = add_prefix(name);
 make_directories(name);
 change_mode(name, mode);
 }
 virtual void add_symlink(string oldpath, string newpath, mode_t mode) 
override {
-cout << "Link " << newpath << " to " << oldpath << " ...\n";
+_log << "Link " << newpath << " to " << oldpath << " ...\n";
 newpath = add_prefix(newpath);
 auto pos = newpath.rfind('/');
 if (pos != newpath.npos) {
@@ -82,6 +96,7 @@ public:
 
 private:
 std::string _prefix;
+logger _log;
 std::string add_prefix(std::string path) {
 if (_prefix.empty()) {
 return path;
@@ -116,11 +131,13 @@ int main(int ac, char** av)
 {
 int port;
 std::string prefix;
+bool verbose;
 // Declare the supported options.
 po::options_description desc("Allowed options");
 desc.add_options()
 ("help", "produce help message")
 ("port", po::value()->default_value(1), "set listening port")
+("verbose", po::value()->default_value(true), "disable verbose 
output")
 ("prefix", po::value()->default_value(std::string("/")), 
"set prefix");
 
 po::variables_map vm;
@@ -134,6 +151,7 @@ int main(int ac, char** av)
 
 port = vm["port"].as();
 prefix = vm["prefix"].as();
+verbose = vm["verbose"].as();
 
 boost::asio::io_service io_service;
 tcp::acceptor acceptor(io_service, tcp::endpoint(tcp::v4(), port));
@@ -141,7 +159,7 @@ int main(int ac, char** av)
 cout << "Waiting for connection from host...\n";
 boost::asio::ip::tcp::iostream socket;
 acceptor.accept(*socket.rdbuf());
-cpio_in_expand expand_files(prefix);
+cpio_in_expand expand_files(prefix, verbose);
 cpio_in::parse(socket, expand_files);
 sync();
 
@@ -158,4 +176,5 @@ int main(int ac, char** av)
 fprintf(stderr, "umount /zfs failed, error = %s\n", 
strerror(errno));
 }
 }
+cout << "cpiod finished" << endl;
 }
-- 
2.13.6

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH v4 0/2] namespace: enable per namespace alarm/signal

[Justin Cinkelj]

Compared to v2:
Patch 1 is simplified, as Nadav suggested.

Patch 2, result of symbol lookup for signal_actions[] is cached now.
The signal_actions[] is also moved out of osv:: namespace, so that
name mangling will not be a problem, as now we search for undecorated name.


Justin Cinkelj (2):
  namespace: enable per namespace alarm/signal
  namespace: fix signal handling called from unduplicated code

 Makefile   |  6 +-
 core/app.cc| 12 
 include/osv/app.hh | 14 ++
 libc/signal.cc | 17 -
 libc/signal.hh |  4 ++--
 usr.manifest.skel  |  1 +
 6 files changed, 46 insertions(+), 8 deletions(-)

-- 
2.13.6

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH v4 2/2] namespace: fix signal handling called from unduplicated code

[Justin Cinkelj]

The previous patch (namespace: enable per namespace alarm/signal)
implemented per elf namespace signals for common usecase of alarms.

But, as Nadav explained, it failed to allow application in a separate
ELF namespace to install its own handler for signals related to a
page fault, division by zero, and similar exceptions.
See also https://groups.google.com/forum/#!topic/osv-dev/AlYM5i4laPM

The apps/apache-spark is one such example. It installs its own handler
for SIGSEGV. Without this patch, the app started in initial/main ELF
namespace only. But it failed to start in a new ELF namespace.

This patch fixes this. Instead of always using signal_actions variable,
we ask current application/ELF namespace for its copy of signal_actions,
and use that one when generating signal.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 core/app.cc|  3 +++
 include/osv/app.hh | 12 
 libc/signal.cc | 17 -
 libc/signal.hh |  4 ++--
 4 files changed, 29 insertions(+), 7 deletions(-)

diff --git a/core/app.cc b/core/app.cc
index 7d9ba6c0..e7e3868b 100644
--- a/core/app.cc
+++ b/core/app.cc
@@ -18,6 +18,7 @@
 #include 
 #include 
 #include "libc/pthread.hh"
+#include "libc/signal.hh"
 
 using namespace boost::range;
 
@@ -151,6 +152,7 @@ application::application(const std::string& command,
 : _args(args)
 , _command(command)
 , _termination_requested(false)
+, _signal_actions(signal_actions) /* set to variable from initial ELF 
namespace */
 , _runtime(new application_runtime(*this))
 , _joiner(nullptr)
 , _terminated(false)
@@ -456,6 +458,7 @@ void application::clone_osv_signal()
 if (!_libsignal) {
 abort("could not load libsignal.so\n");
 }
+_signal_actions = _libsignal->lookup("signal_actions");
 }
 
 void application::clone_osv_environ()
diff --git a/include/osv/app.hh b/include/osv/app.hh
index d3015515..f4cafb48 100644
--- a/include/osv/app.hh
+++ b/include/osv/app.hh
@@ -19,6 +19,7 @@
 #include 
 #include 
 #include 
+#include 
 
 extern "C" void __libc_start_main(int(*)(int, char**), int, char**, void(*)(),
 void(*)(), void(*)(), void*);
@@ -183,6 +184,16 @@ public:
 std::shared_ptr lib() const { return _lib; }
 
 elf::program *program();
+
+/**
+  * Returns signal handlers of the current app (elf namespace).
+  *
+  * This is used to implement per-elf-namespace signal handling,
+  * when signal is called from the OSv kernel (for example, when
+  * application in a separate elf namespace generates SIGSEGV, and has
+  * installed a handler).
+  */
+struct sigaction* get_signal_actions() const { return _signal_actions; };
 private:
 void new_program();
 void clone_osv_signal();
@@ -215,6 +226,7 @@ private:
 std::shared_ptr _lib;
 std::shared_ptr _libenviron;
 std::shared_ptr _libsignal;
+struct sigaction* _signal_actions;
 main_func_t* _main;
 void (*_entry_point)();
 static app_registry apps;
diff --git a/libc/signal.cc b/libc/signal.cc
index 1992ff88..44836568 100644
--- a/libc/signal.cc
+++ b/libc/signal.cc
@@ -18,9 +18,12 @@
 #include 
 #include 
 #include 
+#include 
 
 using namespace osv::clock::literals;
 
+struct sigaction signal_actions[osv::nsignals];
+
 namespace osv {
 
 // we can't use have __thread sigset because of the constructor
@@ -33,8 +36,6 @@ __thread __attribute__((aligned(sizeof(sigset
 // pending signal changes: returning any one is fine
 __thread int thread_pending_signal;
 
-struct sigaction signal_actions[nsignals];
-
 sigset* from_libc(sigset_t* s)
 {
 return reinterpret_cast<sigset*>(s);
@@ -123,11 +124,17 @@ void generate_signal(siginfo_t , exception_frame* 
ef)
 // needs to be running to generate them. So definitely not waiting.
 abort();
 }
-if (is_sig_dfl(signal_actions[siginfo.si_signo])) {
+
+// Application in elf namespace have different signal handler than main
+// OSv kernel. Use it if it is installed.
+auto app_signal_actions = application::get_current()->get_signal_actions();
+assert(app_signal_actions);
+
+if (is_sig_dfl(app_signal_actions[siginfo.si_signo])) {
 // Our default is to abort the process
 abort();
-} else if(!is_sig_ign(signal_actions[siginfo.si_signo])) {
-arch::build_signal_frame(ef, siginfo, 
signal_actions[siginfo.si_signo]);
+} else if(!is_sig_ign(app_signal_actions[siginfo.si_signo])) {
+arch::build_signal_frame(ef, siginfo, 
app_signal_actions[siginfo.si_signo]);
 }
 }
 
diff --git a/libc/signal.hh b/libc/signal.hh
index c19246cf..8114a037 100644
--- a/libc/signal.hh
+++ b/libc/signal.hh
@@ -20,8 +20,6 @@ struct sigset {
 std::bitset mask;
 };
 
-extern struct sigaction signal_actions[nsignals];
-
 sigset* from_libc(sigset_t* s);
 const sigset* from_libc(const sigset_t* s);
 
@@ -31,6 +29,8 @@ void g

[PATCH v4 1/2] namespace: enable per namespace alarm/signal

[Justin Cinkelj]

From: Justin Cinkelj <justin@xlab.si>

Before patch, netperf netserver.so and netperf.so could not be run in
the same VM. The netserver reports "error starting alarm timer,
ret 3 errno 28".

The patch make alarm (and also signals) per namespace specific, in the
wat as environ was made per namespace specific in
e93af8d3622ba1cf491ba12c780f05bb447de8ac and
1d3645b6c1a119ea34e1a3dca6ae814d4902905f.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 Makefile   | 6 +-
 core/app.cc| 9 +
 include/osv/app.hh | 2 ++
 usr.manifest.skel  | 1 +
 4 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 0850f7c5..8317ba46 100644
--- a/Makefile
+++ b/Makefile
@@ -1868,10 +1868,14 @@ $(out)/bsd/%.o: COMMON += -DSMP 
-D'__FBSDID(__str__)=extern int __bogus__'
 
 environ_sources = $(addprefix libc/, $(environ_libc))
 environ_sources += $(addprefix musl/src/, $(environ_musl))
+signal_sources = libc/signal.cc
 
 $(out)/libenviron.so: $(environ_sources)
$(makedir)
 $(call quiet, $(CC) $(CFLAGS) -shared -o $(out)/libenviron.so 
$(environ_sources), CC libenviron.so)
+$(out)/libsignal.so: $(signal_sources)
+   $(makedir)
+$(call quiet, $(CXX) $(CXXFLAGS) -shared -o $(out)/libsignal.so 
$(signal_sources), CXX libsignal.so)
 
 bootfs_manifest ?= bootfs.manifest.skel
 
@@ -1885,7 +1889,7 @@ $(bootfs_manifest_dep): phony
fi
 
 $(out)/bootfs.bin: scripts/mkbootfs.py $(bootfs_manifest) 
$(bootfs_manifest_dep) $(tools:%=$(out)/%) \
-   $(out)/zpool.so $(out)/zfs.so $(out)/libenviron.so
+   $(out)/zpool.so $(out)/zfs.so $(out)/libenviron.so 
$(out)/libsignal.so
$(call quiet, olddir=`pwd`; cd $(out); $$olddir/scripts/mkbootfs.py -o 
bootfs.bin -d bootfs.bin.d -m $$olddir/$(bootfs_manifest) \
-D jdkbase=$(jdkbase) -D gccbase=$(gccbase) -D \
glibcbase=$(glibcbase) -D miscbase=$(miscbase), MKBOOTFS $@)
diff --git a/core/app.cc b/core/app.cc
index 527a5529..7d9ba6c0 100644
--- a/core/app.cc
+++ b/core/app.cc
@@ -161,6 +161,7 @@ application::application(const std::string& command,
 if (new_program) {
 this->new_program();
 clone_osv_environ();
+clone_osv_signal();
 current_program = _program.get();
 } else {
 // Do it in a separate branch because elf::get_program() would not
@@ -449,6 +450,14 @@ elf::program *application::program() {
 }
 
 
+void application::clone_osv_signal()
+{
+_libsignal = _program->get_library("libsignal.so");
+if (!_libsignal) {
+abort("could not load libsignal.so\n");
+}
+}
+
 void application::clone_osv_environ()
 {
 _libenviron = _program->get_library("libenviron.so");
diff --git a/include/osv/app.hh b/include/osv/app.hh
index 83e017ed..d3015515 100644
--- a/include/osv/app.hh
+++ b/include/osv/app.hh
@@ -185,6 +185,7 @@ public:
 elf::program *program();
 private:
 void new_program();
+void clone_osv_signal();
 void clone_osv_environ();
 void set_environ(const std::string , const std::string ,
  bool new_program);
@@ -213,6 +214,7 @@ private:
 mutex _termination_mutex;
 std::shared_ptr _lib;
 std::shared_ptr _libenviron;
+std::shared_ptr _libsignal;
 main_func_t* _main;
 void (*_entry_point)();
 static app_registry apps;
diff --git a/usr.manifest.skel b/usr.manifest.skel
index 583bdfe5..73f58c1f 100644
--- a/usr.manifest.skel
+++ b/usr.manifest.skel
@@ -1,5 +1,6 @@
 [manifest]
 /libenviron.so: libenviron.so
+/libsignal.so: libsignal.so
 /zpool.so: zpool.so
 /libzfs.so: libzfs.so
 /libuutil.so: libuutil.so
-- 
2.13.6

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH v2 2/2] namespace: fix signal handling called from unduplicated code

[Justin Cinkelj]

- Original Message -
> From: "Nadav Har'El" <n...@scylladb.com>
> To: "Justin Cinkelj" <justin.cink...@xlab.si>
> Cc: "Osv Dev" <osv-dev@googlegroups.com>
> Sent: Monday, October 16, 2017 7:27:17 AM
> Subject: Re: [PATCH v2 2/2] namespace: fix signal handling called from 
> unduplicated code
> 
> On Mon, Oct 16, 2017 at 4:30 AM, Justin Cinkelj <justin.cink...@xlab.si>
> wrote:
> 
> > The previous patch (namespace: enable per namespace alarm/signal)
> > implemented per elf namespace signals for common usecase of alarms.
> >
> > But, as Nadav explained, it failed to allow application in a separate
> > ELF namespace to install its own handler for signals related to a
> > page fault, division by zero, and similar exceptions.
> > See also https://groups.google.com/forum/#!topic/osv-dev/AlYM5i4laPM
> >
> > The apps/apache-spark is one such example. It installs its own handler
> > for SIGSEGV. Without this patch, the app started in initial/main ELF
> > namespace only. But it failed to start in a new ELF namespace.
> >
> > This patch fixes this. Instead of always using signal_actions variable,
> > we ask current application/ELF namespace for its copy of signal_actions,
> > and use that one when generating signal.
> >
> 
> Thanks. Your implementation is even nicer than I suggested (you wierdified
> a low-level
> function to get the array, keeping the more interesting functions sane).
> 
> But I have a few suggestions for further improvement below:
> 
> 
> >
> > Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
> > ---
> >  core/app.cc|  9 +
> >  include/osv/app.hh | 11 +++
> >  libc/signal.cc | 13 ++---
> >  3 files changed, 30 insertions(+), 3 deletions(-)
> >
> > diff --git a/core/app.cc b/core/app.cc
> > index 325ae820..acd9f611 100644
> > --- a/core/app.cc
> > +++ b/core/app.cc
> > @@ -18,6 +18,7 @@
> >  #include 
> >  #include 
> >  #include "libc/pthread.hh"
> > +#include "libc/signal.hh"
> >
> >  using namespace boost::range;
> >
> > @@ -480,6 +481,14 @@ void application::clone_osv_environ()
> >  }
> >  }
> >
> > +struct sigaction* application::get_signal_actions() {
> > +if (_libsignal == nullptr) {
> > +// running in initial ELF namespace
> > +return signal_actions;
> > +}
> > +return _libsignal->lookup("_ZN3osv14signal_
> > actionsE");
> >
> 
> 
> Looking at this implementation made me think: If we already have the "app"
> structure, and use its "_libsignal" field,
> why not add another app field, like signal_actions? I.e., when setting
> _libsignal you could also set _signal_actions - using
> the same lookup you did here - and then use it in the get_signal_actions
> method. The benefit would be to avoid
> the relatively complex and slow lookup code on every signal.
> This is not critical (I don't think there's a huge number of signals), but
> may make a difference when there are plenty of
> SIGSEGV signals (I was worried this is the case in Java, I don't remember
> if it's true) - what do you think?

Makes sense to cache that. 

> 
> Another thing is that I'm not happy about the need for the mangled name in
> this code - the mangling scheme may change
> in future versions of the compiler or different architectures or different
> compilers or whatever. One thing we could do is to move
> the signal_actions out of the osv namespace so it will have a normal
> non-mangled name. Another idea could be to have an
> "extern C" function in signal.cc, say, get_signal_actions() (this will be
> in signal.cc, not app.cc!) which returns osv::signal_actions
> (which could be made "static", I don't know why it isn't now...) - and then
> the code which sets app._signal_actions will lookup and
> call this function instead of looking up a variable directly..
> 

I will move signal_actions out of the osv namespace.

> 
> 
> 
> > +}
> > +
> >  void application::set_environ(const std::string , const std::string
> > ,
> >bool new_program)
> >  {
> > diff --git a/include/osv/app.hh b/include/osv/app.hh
> > index d3015515..0c3149fb 100644
> > --- a/include/osv/app.hh
> > +++ b/include/osv/app.hh
> > @@ -19,6 +19,7 @@
> >  #include 
> >  #include 
> >  #include 
> > +#include 
> >
> >  extern "C" void __libc_start_main(int(*)(int, char**), int, 

[PATCH v3 0/2] namespace: enable per namespace alarm/signal

[Justin Cinkelj]

Compared to v2:
Patch 1 is simplified, as Nadav suggested.
Pathc 2 is unmodifed.

Justin Cinkelj (2):
  namespace: enable per namespace alarm/signal
  namespace: fix signal handling called from unduplicated code

 Makefile   |  6 +-
 core/app.cc| 18 ++
 include/osv/app.hh | 13 +
 libc/signal.cc | 13 ++---
 usr.manifest.skel  |  1 +
 5 files changed, 47 insertions(+), 4 deletions(-)

-- 
2.13.6

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH v3 1/2] namespace: enable per namespace alarm/signal

[Justin Cinkelj]

From: Justin Cinkelj <justin@xlab.si>

Before patch, netperf netserver.so and netperf.so could not be run in
the same VM. The netserver reports "error starting alarm timer,
ret 3 errno 28".

The patch make alarm (and also signals) per namespace specific, in the
wat as environ was made per namespace specific in
e93af8d3622ba1cf491ba12c780f05bb447de8ac and
1d3645b6c1a119ea34e1a3dca6ae814d4902905f.

Signed-off-by: Justin Cinkelj <justin@xlab.si>
---
 Makefile   | 6 +-
 core/app.cc| 9 +
 include/osv/app.hh | 2 ++
 usr.manifest.skel  | 1 +
 4 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 0850f7c5..8317ba46 100644
--- a/Makefile
+++ b/Makefile
@@ -1868,10 +1868,14 @@ $(out)/bsd/%.o: COMMON += -DSMP 
-D'__FBSDID(__str__)=extern int __bogus__'
 
 environ_sources = $(addprefix libc/, $(environ_libc))
 environ_sources += $(addprefix musl/src/, $(environ_musl))
+signal_sources = libc/signal.cc
 
 $(out)/libenviron.so: $(environ_sources)
$(makedir)
 $(call quiet, $(CC) $(CFLAGS) -shared -o $(out)/libenviron.so 
$(environ_sources), CC libenviron.so)
+$(out)/libsignal.so: $(signal_sources)
+   $(makedir)
+$(call quiet, $(CXX) $(CXXFLAGS) -shared -o $(out)/libsignal.so 
$(signal_sources), CXX libsignal.so)
 
 bootfs_manifest ?= bootfs.manifest.skel
 
@@ -1885,7 +1889,7 @@ $(bootfs_manifest_dep): phony
fi
 
 $(out)/bootfs.bin: scripts/mkbootfs.py $(bootfs_manifest) 
$(bootfs_manifest_dep) $(tools:%=$(out)/%) \
-   $(out)/zpool.so $(out)/zfs.so $(out)/libenviron.so
+   $(out)/zpool.so $(out)/zfs.so $(out)/libenviron.so 
$(out)/libsignal.so
$(call quiet, olddir=`pwd`; cd $(out); $$olddir/scripts/mkbootfs.py -o 
bootfs.bin -d bootfs.bin.d -m $$olddir/$(bootfs_manifest) \
-D jdkbase=$(jdkbase) -D gccbase=$(gccbase) -D \
glibcbase=$(glibcbase) -D miscbase=$(miscbase), MKBOOTFS $@)
diff --git a/core/app.cc b/core/app.cc
index 527a5529..7d9ba6c0 100644
--- a/core/app.cc
+++ b/core/app.cc
@@ -161,6 +161,7 @@ application::application(const std::string& command,
 if (new_program) {
 this->new_program();
 clone_osv_environ();
+clone_osv_signal();
 current_program = _program.get();
 } else {
 // Do it in a separate branch because elf::get_program() would not
@@ -449,6 +450,14 @@ elf::program *application::program() {
 }
 
 
+void application::clone_osv_signal()
+{
+_libsignal = _program->get_library("libsignal.so");
+if (!_libsignal) {
+abort("could not load libsignal.so\n");
+}
+}
+
 void application::clone_osv_environ()
 {
 _libenviron = _program->get_library("libenviron.so");
diff --git a/include/osv/app.hh b/include/osv/app.hh
index 83e017ed..d3015515 100644
--- a/include/osv/app.hh
+++ b/include/osv/app.hh
@@ -185,6 +185,7 @@ public:
 elf::program *program();
 private:
 void new_program();
+void clone_osv_signal();
 void clone_osv_environ();
 void set_environ(const std::string , const std::string ,
  bool new_program);
@@ -213,6 +214,7 @@ private:
 mutex _termination_mutex;
 std::shared_ptr _lib;
 std::shared_ptr _libenviron;
+std::shared_ptr _libsignal;
 main_func_t* _main;
 void (*_entry_point)();
 static app_registry apps;
diff --git a/usr.manifest.skel b/usr.manifest.skel
index 583bdfe5..73f58c1f 100644
--- a/usr.manifest.skel
+++ b/usr.manifest.skel
@@ -1,5 +1,6 @@
 [manifest]
 /libenviron.so: libenviron.so
+/libsignal.so: libsignal.so
 /zpool.so: zpool.so
 /libzfs.so: libzfs.so
 /libuutil.so: libuutil.so
-- 
2.13.6

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH v3 2/2] namespace: fix signal handling called from unduplicated code

[Justin Cinkelj]

The previous patch (namespace: enable per namespace alarm/signal)
implemented per elf namespace signals for common usecase of alarms.

But, as Nadav explained, it failed to allow application in a separate
ELF namespace to install its own handler for signals related to a
page fault, division by zero, and similar exceptions.
See also https://groups.google.com/forum/#!topic/osv-dev/AlYM5i4laPM

The apps/apache-spark is one such example. It installs its own handler
for SIGSEGV. Without this patch, the app started in initial/main ELF
namespace only. But it failed to start in a new ELF namespace.

This patch fixes this. Instead of always using signal_actions variable,
we ask current application/ELF namespace for its copy of signal_actions,
and use that one when generating signal.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 core/app.cc|  9 +
 include/osv/app.hh | 11 +++
 libc/signal.cc | 13 ++---
 3 files changed, 30 insertions(+), 3 deletions(-)

diff --git a/core/app.cc b/core/app.cc
index 7d9ba6c0..79170680 100644
--- a/core/app.cc
+++ b/core/app.cc
@@ -18,6 +18,7 @@
 #include 
 #include 
 #include "libc/pthread.hh"
+#include "libc/signal.hh"
 
 using namespace boost::range;
 
@@ -479,6 +480,14 @@ void application::clone_osv_environ()
 }
 }
 
+struct sigaction* application::get_signal_actions() {
+if (_libsignal == nullptr) {
+// running in initial ELF namespace
+return signal_actions;
+}
+return _libsignal->lookup("_ZN3osv14signal_actionsE");
+}
+
 void application::set_environ(const std::string , const std::string ,
   bool new_program)
 {
diff --git a/include/osv/app.hh b/include/osv/app.hh
index d3015515..0c3149fb 100644
--- a/include/osv/app.hh
+++ b/include/osv/app.hh
@@ -19,6 +19,7 @@
 #include 
 #include 
 #include 
+#include 
 
 extern "C" void __libc_start_main(int(*)(int, char**), int, char**, void(*)(),
 void(*)(), void(*)(), void*);
@@ -183,6 +184,16 @@ public:
 std::shared_ptr lib() const { return _lib; }
 
 elf::program *program();
+
+/**
+  * Returns signal handlers of the current app (elf namespace).
+  *
+  * This is used to implement per-elf-namespace signal handling,
+  * when signal is called from the OSv kernel (for example, when
+  * application in a separate elf namespace generates SIGSEGV, and has
+  * installed a handler).
+  */
+struct sigaction* get_signal_actions();
 private:
 void new_program();
 void clone_osv_signal();
diff --git a/libc/signal.cc b/libc/signal.cc
index 1992ff88..b65bff2a 100644
--- a/libc/signal.cc
+++ b/libc/signal.cc
@@ -18,6 +18,7 @@
 #include 
 #include 
 #include 
+#include 
 
 using namespace osv::clock::literals;
 
@@ -123,11 +124,17 @@ void generate_signal(siginfo_t , exception_frame* 
ef)
 // needs to be running to generate them. So definitely not waiting.
 abort();
 }
-if (is_sig_dfl(signal_actions[siginfo.si_signo])) {
+
+// Application in elf namespace have different signal handler than main
+// OSv kernel. Use it if it is installed.
+auto app_signal_actions = 
application::get_current().get()->get_signal_actions();
+assert(app_signal_actions);
+
+if (is_sig_dfl(app_signal_actions[siginfo.si_signo])) {
 // Our default is to abort the process
 abort();
-} else if(!is_sig_ign(signal_actions[siginfo.si_signo])) {
-arch::build_signal_frame(ef, siginfo, 
signal_actions[siginfo.si_signo]);
+} else if(!is_sig_ign(app_signal_actions[siginfo.si_signo])) {
+arch::build_signal_frame(ef, siginfo, 
app_signal_actions[siginfo.si_signo]);
 }
 }
 
-- 
2.13.6

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH v2 2/2] namespace: fix signal handling called from unduplicated code

[Justin Cinkelj]

The previous patch (namespace: enable per namespace alarm/signal)
implemented per elf namespace signals for common usecase of alarms.

But, as Nadav explained, it failed to allow application in a separate
ELF namespace to install its own handler for signals related to a
page fault, division by zero, and similar exceptions.
See also https://groups.google.com/forum/#!topic/osv-dev/AlYM5i4laPM

The apps/apache-spark is one such example. It installs its own handler
for SIGSEGV. Without this patch, the app started in initial/main ELF
namespace only. But it failed to start in a new ELF namespace.

This patch fixes this. Instead of always using signal_actions variable,
we ask current application/ELF namespace for its copy of signal_actions,
and use that one when generating signal.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 core/app.cc|  9 +
 include/osv/app.hh | 11 +++
 libc/signal.cc | 13 ++---
 3 files changed, 30 insertions(+), 3 deletions(-)

diff --git a/core/app.cc b/core/app.cc
index 325ae820..acd9f611 100644
--- a/core/app.cc
+++ b/core/app.cc
@@ -18,6 +18,7 @@
 #include 
 #include 
 #include "libc/pthread.hh"
+#include "libc/signal.hh"
 
 using namespace boost::range;
 
@@ -480,6 +481,14 @@ void application::clone_osv_environ()
 }
 }
 
+struct sigaction* application::get_signal_actions() {
+if (_libsignal == nullptr) {
+// running in initial ELF namespace
+return signal_actions;
+}
+return _libsignal->lookup("_ZN3osv14signal_actionsE");
+}
+
 void application::set_environ(const std::string , const std::string ,
   bool new_program)
 {
diff --git a/include/osv/app.hh b/include/osv/app.hh
index d3015515..0c3149fb 100644
--- a/include/osv/app.hh
+++ b/include/osv/app.hh
@@ -19,6 +19,7 @@
 #include 
 #include 
 #include 
+#include 
 
 extern "C" void __libc_start_main(int(*)(int, char**), int, char**, void(*)(),
 void(*)(), void(*)(), void*);
@@ -183,6 +184,16 @@ public:
 std::shared_ptr lib() const { return _lib; }
 
 elf::program *program();
+
+/**
+  * Returns signal handlers of the current app (elf namespace).
+  *
+  * This is used to implement per-elf-namespace signal handling,
+  * when signal is called from the OSv kernel (for example, when
+  * application in a separate elf namespace generates SIGSEGV, and has
+  * installed a handler).
+  */
+struct sigaction* get_signal_actions();
 private:
 void new_program();
 void clone_osv_signal();
diff --git a/libc/signal.cc b/libc/signal.cc
index 1992ff88..b65bff2a 100644
--- a/libc/signal.cc
+++ b/libc/signal.cc
@@ -18,6 +18,7 @@
 #include 
 #include 
 #include 
+#include 
 
 using namespace osv::clock::literals;
 
@@ -123,11 +124,17 @@ void generate_signal(siginfo_t , exception_frame* 
ef)
 // needs to be running to generate them. So definitely not waiting.
 abort();
 }
-if (is_sig_dfl(signal_actions[siginfo.si_signo])) {
+
+// Application in elf namespace have different signal handler than main
+// OSv kernel. Use it if it is installed.
+auto app_signal_actions = 
application::get_current().get()->get_signal_actions();
+assert(app_signal_actions);
+
+if (is_sig_dfl(app_signal_actions[siginfo.si_signo])) {
 // Our default is to abort the process
 abort();
-} else if(!is_sig_ign(signal_actions[siginfo.si_signo])) {
-arch::build_signal_frame(ef, siginfo, 
signal_actions[siginfo.si_signo]);
+} else if(!is_sig_ign(app_signal_actions[siginfo.si_signo])) {
+arch::build_signal_frame(ef, siginfo, 
app_signal_actions[siginfo.si_signo]);
 }
 }
 
-- 
2.13.6

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH v2 1/2] namespace: enable per namespace alarm/signal

[Justin Cinkelj]

From: Justin Cinkelj <justin@xlab.si>

Before patch, netperf netserver.so and netperf.so could not be run in
the same VM. The netserver reports "error starting alarm timer,
ret 3 errno 28".

The patch make alarm (and also signals) per namespace specific, in the
wat as environ was made per namespace specific in
e93af8d3622ba1cf491ba12c780f05bb447de8ac and
1d3645b6c1a119ea34e1a3dca6ae814d4902905f.

Signed-off-by: Justin Cinkelj <justin@xlab.si>
---
 Makefile   |  9 -
 core/app.cc| 10 ++
 include/osv/app.hh |  2 ++
 usr.manifest.skel  |  1 +
 4 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 0850f7c5..ab4c9ee7 100644
--- a/Makefile
+++ b/Makefile
@@ -940,6 +940,7 @@ libc =
 musl =
 environ_libc =
 environ_musl =
+signal_libc =
 
 ifeq ($(arch),x64)
 musl_arch = x86_64
@@ -1009,6 +1010,8 @@ environ_musl += env/putenv.c
 environ_musl += env/setenv.c
 environ_musl += env/unsetenv.c
 
+signal_libc += signal.cc
+
 musl += ctype/__ctype_b_loc.o
 
 musl += errno/strerror.o
@@ -1868,10 +1871,14 @@ $(out)/bsd/%.o: COMMON += -DSMP 
-D'__FBSDID(__str__)=extern int __bogus__'
 
 environ_sources = $(addprefix libc/, $(environ_libc))
 environ_sources += $(addprefix musl/src/, $(environ_musl))
+signal_sources = $(addprefix libc/, $(signal_libc))
 
 $(out)/libenviron.so: $(environ_sources)
$(makedir)
 $(call quiet, $(CC) $(CFLAGS) -shared -o $(out)/libenviron.so 
$(environ_sources), CC libenviron.so)
+$(out)/libsignal.so: $(signal_sources)
+   $(makedir)
+$(call quiet, $(CXX) $(CXXFLAGS) -shared -o $(out)/libsignal.so 
$(signal_sources), CXX libsignal.so)
 
 bootfs_manifest ?= bootfs.manifest.skel
 
@@ -1885,7 +1892,7 @@ $(bootfs_manifest_dep): phony
fi
 
 $(out)/bootfs.bin: scripts/mkbootfs.py $(bootfs_manifest) 
$(bootfs_manifest_dep) $(tools:%=$(out)/%) \
-   $(out)/zpool.so $(out)/zfs.so $(out)/libenviron.so
+   $(out)/zpool.so $(out)/zfs.so $(out)/libenviron.so 
$(out)/libsignal.so
$(call quiet, olddir=`pwd`; cd $(out); $$olddir/scripts/mkbootfs.py -o 
bootfs.bin -d bootfs.bin.d -m $$olddir/$(bootfs_manifest) \
-D jdkbase=$(jdkbase) -D gccbase=$(gccbase) -D \
glibcbase=$(glibcbase) -D miscbase=$(miscbase), MKBOOTFS $@)
diff --git a/core/app.cc b/core/app.cc
index 527a5529..325ae820 100644
--- a/core/app.cc
+++ b/core/app.cc
@@ -161,6 +161,7 @@ application::application(const std::string& command,
 if (new_program) {
 this->new_program();
 clone_osv_environ();
+clone_osv_signal();
 current_program = _program.get();
 } else {
 // Do it in a separate branch because elf::get_program() would not
@@ -449,6 +450,15 @@ elf::program *application::program() {
 }
 
 
+void application::clone_osv_signal()
+{
+_libsignal = _program->get_library("libsignal.so");
+if (!_libsignal) {
+abort("could not load libsignal.so\n");
+return;
+}
+}
+
 void application::clone_osv_environ()
 {
 _libenviron = _program->get_library("libenviron.so");
diff --git a/include/osv/app.hh b/include/osv/app.hh
index 83e017ed..d3015515 100644
--- a/include/osv/app.hh
+++ b/include/osv/app.hh
@@ -185,6 +185,7 @@ public:
 elf::program *program();
 private:
 void new_program();
+void clone_osv_signal();
 void clone_osv_environ();
 void set_environ(const std::string , const std::string ,
  bool new_program);
@@ -213,6 +214,7 @@ private:
 mutex _termination_mutex;
 std::shared_ptr _lib;
 std::shared_ptr _libenviron;
+std::shared_ptr _libsignal;
 main_func_t* _main;
 void (*_entry_point)();
 static app_registry apps;
diff --git a/usr.manifest.skel b/usr.manifest.skel
index 583bdfe5..73f58c1f 100644
--- a/usr.manifest.skel
+++ b/usr.manifest.skel
@@ -1,5 +1,6 @@
 [manifest]
 /libenviron.so: libenviron.so
+/libsignal.so: libsignal.so
 /zpool.so: zpool.so
 /libzfs.so: libzfs.so
 /libuutil.so: libuutil.so
-- 
2.13.6

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH v2 0/2] namespace: enable per namespace alarm/signal

[Justin Cinkelj]

The first patch is unmodified from what I aready sent to mail list a while ago.

The second patch fixes problem Nadav noticed (and suggested a fix too) -
that "it will not really work in places where unduplicated OSv code calls
kill() or related functions". See
https://groups.google.com/d/msg/osv-dev/AlYM5i4laPM/bBryd5sgAgAJ

Indeed, osv-apps/apache-spark show that problem. Both patches are required to 
start it
in a new ELF namespace.

BR, Justin


Justin Cinkelj (2):
  namespace: enable per namespace alarm/signal
  namespace: fix signal handling called from unduplicated code

 Makefile   |  9 -
 core/app.cc| 19 +++
 include/osv/app.hh | 13 +
 libc/signal.cc | 13 ++---
 usr.manifest.skel  |  1 +
 5 files changed, 51 insertions(+), 4 deletions(-)

-- 
2.13.6

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [RFC] namespace: enable per namespace alarm/signal

[Justin Cinkelj]

On 10/11/2017 08:52 AM, Justin Cinkelj wrote:

> Did you test Java with this patch?
No. But yesterday Gasper tried it with apache spark, and java had 
problems. Problem was with SIGSEGV, as you predicted.


> code in exceptions.cc and core/mmu.cc calls the functions 
generate_signal() and handle_mmap_fault()

> Maybe there are other ways to solve this
I initially wanted to move just a few function into a separate .cc 
file, so that libsignal.so would be smaller. But one function required 
other, which required other, and than, I had about 90% of signal.cc in 
new file. So then I just compiled whole file into libsignal.so, to 
make patch smaller. Maybe I can move the handle_mmap_fault out of 
libsignal.so - so there will be only one copy of it?


> because the OSv kernel calls them (handle_mmap_fault() and 
generate_signal()), so it uses the version in the OSv kernel, not the 
one in the application.
I misunderstood that - the version from app should be called, while with 
current patch the version from kernel is called. Moving them out of 
libsignal.so (to a be only in kernel) doesn't make any sense.


Justin



Obviously, tests for both simple C apps and for java apps should be 
added too.


Justin

On 10/10/2017 11:29 PM, Nadav Har'El wrote:
This is a really cool idea. It will indeed work for alarms, because 
the application-specific alarm function will call the 
application-specific kill() function which will run the application's 
choice of handler.


However, I think it will not really work in places where unduplicated 
OSv code calls kill() or related functions. For example:


1. In drivers/line-discipline.cc, we call kill(0, SIGINT). This will 
call the main kill and the main handler - not the application's, so 
the application's attempt to catch SIGINT will simply not work, even 
if there's just one application (if started with the signal 
virtualization flag). Arguably this is not an important problem - an 
application cannot hope to catch SIGINT in this case, so it's ok that 
it doesn't.


2. When we have a page fault, division by zero, and similar 
exceptions, code in exceptions.cc and core/mmu.cc calls the functions 
generate_signal() and handle_mmap_fault() which are (to make a long 
story short) specialized versions of kill. Although both these 
functions are in signal.cc and are duplicated for each application, 
it doesn't help because the OSv kernel calls them, so it uses the 
version in the OSv kernel, not the one in the application. This might 
be a problem: If Java, for example, needs to capture SIGSEGV to do 
something (not just for error reporting but for actual correct 
operation), then its signal handler will be ignored. Did you test 
Java with this patch?


One way we can solve this problem is perhaps for the 
generate_signal() function to start by looking for a function by this 
name in the current application (the one the current thread belongs 
to) and if we find one which is not the one currently running, we run 
the one from the application instead. This will obviously slow down 
this function, but I hope it's not very frequently used (although 
it's worth checking how often some Java program runs it, we can 
easily add a tracepoint if we don't already have one, and count it 
with scripts/freq.py).


Maybe there are other ways to solve this. Or maybe we should just 
start by having this feature only "partially" work as it does now, 
for only some applications?


.


--
Nadav Har'El
n...@scylladb.com <mailto:n...@scylladb.com>

On Wed, Mar 8, 2017 at 9:22 AM, Justin Cinkelj 
<justin.cink...@xlab.si <mailto:justin.cink...@xlab.si>> wrote:


Before patch, netperf netserver.so and netperf.so could not be run in
the same VM. The netserver reports "error starting alarm timer,
ret 3 errno 28".

The patch make alarm (and also signals) per namespace specific,
in the
wat as environ was made per namespace specific in
e93af8d3622ba1cf491ba12c780f05bb447de8ac and
1d3645b6c1a119ea34e1a3dca6ae814d4902905f.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si
<mailto:justin.cink...@xlab.si>>
---
 Makefile           |  9 -
 core/app.cc        | 10 ++
 include/osv/app.hh |  2 ++
 usr.manifest.skel  |  1 +
 4 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 6846c5e..d982864 100644
--- a/Makefile
+++ b/Makefile
@@ -926,6 +926,7 @@ libc =
 musl =
 environ_libc =
 environ_musl =
+signal_libc =

 ifeq ($(arch),x64)
 musl_arch = x86_64
@@ -995,6 +996,8 @@ environ_musl += env/putenv.c
 environ_musl += env/setenv.c
 environ_musl += env/unsetenv.c

+signal_libc += signal.cc
+
 musl += ctype/__ctype_b_loc.o

 musl += errno/strerror.o
@@ -1830,10 +1833,14 @@ $(out)/bsd/%.o: COMMON += -DSMP
-D'__FBSDID(__str__)=extern int __bogu

Re: [RFC] namespace: enable per namespace alarm/signal

[Justin Cinkelj]

> Did you test Java with this patch?
No. But yesterday Gasper tried it with apache spark, and java had 
problems. Problem was with SIGSEGV, as you predicted.


> code in exceptions.cc and core/mmu.cc calls the functions 
generate_signal() and handle_mmap_fault()

> Maybe there are other ways to solve this
I initially wanted to move just a few function into a separate .cc file, 
so that libsignal.so would be smaller. But one function required other, 
which required other, and than, I had about 90% of signal.cc in new 
file. So then I just compiled whole file into libsignal.so, to make 
patch smaller. Maybe I can move the handle_mmap_fault out of 
libsignal.so - so there will be only one copy of it?


Obviously, tests for both simple C apps and for java apps should be 
added too.


Justin

On 10/10/2017 11:29 PM, Nadav Har'El wrote:
This is a really cool idea. It will indeed work for alarms, because 
the application-specific alarm function will call the 
application-specific kill() function which will run the application's 
choice of handler.


However, I think it will not really work in places where unduplicated 
OSv code calls kill() or related functions. For example:


1. In drivers/line-discipline.cc, we call kill(0, SIGINT). This will 
call the main kill and the main handler - not the application's, so 
the application's attempt to catch SIGINT will simply not work, even 
if there's just one application (if started with the signal 
virtualization flag). Arguably this is not an important problem - an 
application cannot hope to catch SIGINT in this case, so it's ok that 
it doesn't.


2. When we have a page fault, division by zero, and similar 
exceptions, code in exceptions.cc and core/mmu.cc calls the functions 
generate_signal() and handle_mmap_fault() which are (to make a long 
story short) specialized versions of kill. Although both these 
functions are in signal.cc and are duplicated for each application, it 
doesn't help because the OSv kernel calls them, so it uses the version 
in the OSv kernel, not the one in the application. This might be a 
problem: If Java, for example, needs to capture SIGSEGV to do 
something (not just for error reporting but for actual correct 
operation), then its signal handler will be ignored. Did you test Java 
with this patch?


One way we can solve this problem is perhaps for the generate_signal() 
function to start by looking for a function by this name in the 
current application (the one the current thread belongs to) and if we 
find one which is not the one currently running, we run the one from 
the application instead. This will obviously slow down this function, 
but I hope it's not very frequently used (although it's worth checking 
how often some Java program runs it, we can easily add a tracepoint if 
we don't already have one, and count it with scripts/freq.py).


Maybe there are other ways to solve this. Or maybe we should just 
start by having this feature only "partially" work as it does now, for 
only some applications?


.


--
Nadav Har'El
n...@scylladb.com <mailto:n...@scylladb.com>

On Wed, Mar 8, 2017 at 9:22 AM, Justin Cinkelj <justin.cink...@xlab.si 
<mailto:justin.cink...@xlab.si>> wrote:


Before patch, netperf netserver.so and netperf.so could not be run in
the same VM. The netserver reports "error starting alarm timer,
ret 3 errno 28".

The patch make alarm (and also signals) per namespace specific, in the
wat as environ was made per namespace specific in
e93af8d3622ba1cf491ba12c780f05bb447de8ac and
1d3645b6c1a119ea34e1a3dca6ae814d4902905f.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si
<mailto:justin.cink...@xlab.si>>
---
 Makefile           |  9 -
 core/app.cc        | 10 ++
 include/osv/app.hh |  2 ++
 usr.manifest.skel  |  1 +
 4 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 6846c5e..d982864 100644
--- a/Makefile
+++ b/Makefile
@@ -926,6 +926,7 @@ libc =
 musl =
 environ_libc =
 environ_musl =
+signal_libc =

 ifeq ($(arch),x64)
 musl_arch = x86_64
@@ -995,6 +996,8 @@ environ_musl += env/putenv.c
 environ_musl += env/setenv.c
 environ_musl += env/unsetenv.c

+signal_libc += signal.cc
+
 musl += ctype/__ctype_b_loc.o

 musl += errno/strerror.o
@@ -1830,10 +1833,14 @@ $(out)/bsd/%.o: COMMON += -DSMP
-D'__FBSDID(__str__)=extern int __bogus__'

 environ_sources = $(addprefix libc/, $(environ_libc))
 environ_sources += $(addprefix musl/src/, $(environ_musl))
+signal_sources = $(addprefix libc/, $(signal_libc))

 $(out)/libenviron.so: $(environ_sources)
        $(makedir)
         $(call quiet, $(CC) $(CFLAGS) -shared -o
$(out)/libenviron.so $(environ_sources), CC libenviron.so)
+$(out)/libsignal.so: $(signal_sources)
+       $(makedir)
+  

Re: [PATCH] update cloudinit to use the new libhttpserver-api

[Justin Cinkelj]

bump @nadav

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] update cloudinit to use the new libhttpserver-api

[Justin Cinkelj]

Test which failed to work before:

cat << EOF > cloud-init.yaml
run:
    - GET: /os/version
hostname: uber-vm
EOF

cloud-localds cloud-init.iso cloud-init.yaml
scripts/build -j8 mode=debug image=cli,cloud-init nfs=false
sudo ./scripts/run.py -V -d -e '/cli/cli.so' --cloud-init-image 
cloud-init.iso


Verbose output should include 'cloudinit hostname: uber-vm'.

BR Justin

On 09/19/2017 02:37 PM, Justin Cinkelj wrote:

The c446634acbb354497f21fba0149c5ca716b277f1 splited httpserver into
httpserver-api and httpserver-html5-gui. The new httpserver-api module
uploads binary libhttpserver-api.so to /libhttpserver.so, and old
httpserver module then runs the /libhttpserver.so.

This broke cloud-init module. At link time libhttpserver.so was not found.
Fix this by linking aginst -lhttpserver-api. Also, upload
libhttpserver-api.so with unmodified name, so that cloud-init is able
to find it during ELF loading.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
  modules/cloud-init/Makefile  | 2 +-
  modules/httpserver-api/module.py | 2 +-
  modules/httpserver/module.py | 4 ++--
  3 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/modules/cloud-init/Makefile b/modules/cloud-init/Makefile
index 62dc9b41..daee8c38 100644
--- a/modules/cloud-init/Makefile
+++ b/modules/cloud-init/Makefile
@@ -18,7 +18,7 @@ OBJ_FILES := $(addprefix obj/,$(CPP_FILES:.cc=.o))
  DEPS := $(OBJ_FILES:.o=.d)
  
  STUB_HTTPSERVER_LIBS = $(HTTPSERVER_API_DIR)/httpserver-stub.so

-LIBS = -lpthread $(boost-libs) $(DEPEDNDS_LIBS) -lyaml-cpp 
-L$(HTTPSERVER_API_DIR)/ -lhttpserver
+LIBS = -lpthread $(boost-libs) $(DEPEDNDS_LIBS) -lyaml-cpp 
-L$(HTTPSERVER_API_DIR)/ -lhttpserver-api
  
  quiet = $(if $V, $1, @echo " $2"; $1)

  very-quiet = $(if $V, $1, @$1)
diff --git a/modules/httpserver-api/module.py b/modules/httpserver-api/module.py
index 441c7226..25bc4948 100644
--- a/modules/httpserver-api/module.py
+++ b/modules/httpserver-api/module.py
@@ -5,7 +5,7 @@ from osv.modules import api
  
  _module = '${OSV_BASE}/modules/httpserver-api'
  
-_exe = '/libhttpserver.so'

+_exe = '/libhttpserver-api.so'
  
  usr_files = FileMap()

  usr_files.add(os.path.join(_module, 'libhttpserver-api.so')).to(_exe)
diff --git a/modules/httpserver/module.py b/modules/httpserver/module.py
index d782db30..195958eb 100644
--- a/modules/httpserver/module.py
+++ b/modules/httpserver/module.py
@@ -2,9 +2,9 @@ from osv.modules import api
  
  api.require('httpserver-html5-gui')
  
-fg = api.run('/libhttpserver.so')

+fg = api.run('/libhttpserver-api.so')
  
  # httpserver will run regardless of an explicit command line

  # passed with "run.py -e".
-daemon = api.run_on_init('/libhttpserver.so &!')
+daemon = api.run_on_init('/libhttpserver-api.so &!')
  default = daemon


--
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH] update cloudinit to use the new libhttpserver-api

[Justin Cinkelj]

The c446634acbb354497f21fba0149c5ca716b277f1 splited httpserver into
httpserver-api and httpserver-html5-gui. The new httpserver-api module
uploads binary libhttpserver-api.so to /libhttpserver.so, and old
httpserver module then runs the /libhttpserver.so.

This broke cloud-init module. At link time libhttpserver.so was not found.
Fix this by linking aginst -lhttpserver-api. Also, upload
libhttpserver-api.so with unmodified name, so that cloud-init is able
to find it during ELF loading.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 modules/cloud-init/Makefile  | 2 +-
 modules/httpserver-api/module.py | 2 +-
 modules/httpserver/module.py | 4 ++--
 3 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/modules/cloud-init/Makefile b/modules/cloud-init/Makefile
index 62dc9b41..daee8c38 100644
--- a/modules/cloud-init/Makefile
+++ b/modules/cloud-init/Makefile
@@ -18,7 +18,7 @@ OBJ_FILES := $(addprefix obj/,$(CPP_FILES:.cc=.o))
 DEPS := $(OBJ_FILES:.o=.d)
 
 STUB_HTTPSERVER_LIBS = $(HTTPSERVER_API_DIR)/httpserver-stub.so
-LIBS = -lpthread $(boost-libs) $(DEPEDNDS_LIBS) -lyaml-cpp 
-L$(HTTPSERVER_API_DIR)/ -lhttpserver
+LIBS = -lpthread $(boost-libs) $(DEPEDNDS_LIBS) -lyaml-cpp 
-L$(HTTPSERVER_API_DIR)/ -lhttpserver-api
 
 quiet = $(if $V, $1, @echo " $2"; $1)
 very-quiet = $(if $V, $1, @$1)
diff --git a/modules/httpserver-api/module.py b/modules/httpserver-api/module.py
index 441c7226..25bc4948 100644
--- a/modules/httpserver-api/module.py
+++ b/modules/httpserver-api/module.py
@@ -5,7 +5,7 @@ from osv.modules import api
 
 _module = '${OSV_BASE}/modules/httpserver-api'
 
-_exe = '/libhttpserver.so'
+_exe = '/libhttpserver-api.so'
 
 usr_files = FileMap()
 usr_files.add(os.path.join(_module, 'libhttpserver-api.so')).to(_exe)
diff --git a/modules/httpserver/module.py b/modules/httpserver/module.py
index d782db30..195958eb 100644
--- a/modules/httpserver/module.py
+++ b/modules/httpserver/module.py
@@ -2,9 +2,9 @@ from osv.modules import api
 
 api.require('httpserver-html5-gui')
 
-fg = api.run('/libhttpserver.so')
+fg = api.run('/libhttpserver-api.so')
 
 # httpserver will run regardless of an explicit command line
 # passed with "run.py -e".
-daemon = api.run_on_init('/libhttpserver.so &!')
+daemon = api.run_on_init('/libhttpserver-api.so &!')
 default = daemon
-- 
2.13.5

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH v3 0/6] Allow spaces in loader options

[Justin Cinkelj]

- Original Message -
> From: "Nadav Har'El" <n...@scylladb.com>
> To: "Justin Cinkelj" <justin.cink...@xlab.si>
> Cc: "Osv Dev" <osv-dev@googlegroups.com>
> Sent: Wednesday, September 13, 2017 2:30:38 PM
> Subject: Re: [PATCH v3 0/6] Allow spaces in loader options
> 
> On Wed, Sep 13, 2017 at 3:05 PM, Justin Cinkelj <justin.cink...@xlab.si>
> wrote:
> > Compared to v2 patch:
> > The stresep.c file is now first only added, and included into Makefile
> > after removing unneeded define and include namespace.h.
> > The memcpy bug from v2 patch was reported upstrem to netbsd, and current
> > file
> > now includes fix (a better fix from maintener, he also moved strlen out of
> > loop).
> >
> > I changed commit message for 3/6, as Nadav suggested.
> > Also added comment about loader_parse_cmdline() - its results argv and
> > app_cmdline point
> > to memory of str input param, meaning caller should not free/modify str
> > after calling
> > loader_parse_cmdline.
> >
> > The std::string is intentionally not used - there is comment about that a
> > bit higher
> > in file.
> >
> > I did want to do memmove(str-1, str, ), not memmove(str, str+1).
> >> +// Multiple consecutive delimiters found. Stresep will write
> >> multiple
> >> +// '\0' into str. Squash them into one, so that argv will be
> >> 'nice',
> >> +// in memory consecutive array of C strings.
> >> +if (str) {
> >> +my_debug("shift   str %p '%s' <- %p '%s'\n", str-1,
> >> str-1, str, str);
> >> +memmove(str-1, str, strlen(str) + 1);
> >> I'm confused... If I understand correctly "str" points to that second
> >> 0 you want to remove. So shouldn't you copy str+1 to str? why copy str
> >> to str-1?
> >
> > justin_cinkelj@jcpc:~/devel/mikelangelo/osv-fc25/osv$ ./scripts/run.py -V
> > -e ' --env=AA=aa'
> > OSv v0.24-442-g4bdaa11
> >   ap = 0x9151 --verbose, *ap=45
> >   ap = 0x9151000a , *ap=0
> > shift   str 0x9151000a '' <- 0x9151000b '--env=AA=aa'
> >   ap = 0x9151000a --env=AA=aa, *ap=45
> > make app_cmdline valid pointer to '\0' ap=0x9151000a
> > '--env=AA=aa', app_cmdline=0x146e180 ''
> >   ap0 = 0x9151 '--verbose', apE = 0x9151000a
> >   '--env=AA=aa', ntoken = 2, app_cmdline=0x91510015 ''
> >   argv[0] = 0x9151 --verbose
> >   argv[1] = 0x9151000a --env=AA=aa
> >   ntoken = 2, ii = 2
> > 4 CPUs detected
> >
> > Input str was "--verbose " + " --env=AA=aa"
> > After second stresp (e.g second iteration of outer while loop), both spaces
> > were replaced with '\0', ap=="\0", and str==ap+1 pointed to "--env=AA=aa".
> > I want str moved left by one, to overwrite '\0' at ap location.
> > I hope debug printf makes this clearer.
> > I will abuse the question to justify that debug code should be left :)
> 
> Really? The above output will appear for anyone who uses run.py -V?
> Doesn't look very useful... But we can always remove it later.

I used
#define MY_DEBUG(args...) if(0) printf(args)
So it is disabled by default, and any optimization level will (I believe; I 
didn't check compiled code)
just remove it.

Also, thank you for fixing the aarch64 code.

Justin

> 
> >
> >
> > Justin Cinkelj (6):
> >   libc: add stresep.c file
> >   include stresep function
> >   command line: add loader_parse_cmdline with tests
> >   command line: use loader_parse_cmdline
> >   command line: add tests for empty commandline string parsing
> >   command line: check for empty cmdline case after parsing cmdline
> >
> >  Makefile|   1 +
> >  arch/x64/boot.S |   4 +-
> >  core/commands.cc| 146 ++
> >  include/api/string.h|   1 +
> >  include/osv/commands.hh |   6 +-
> >  libc/string/stresep.c   |  87 
> >  loader.cc   |  55 --
> >  tests/tst-commands.cc   | 268
> >  
> >  8 files changed, 510 insertions(+), 58 deletions(-)
> >  create mode 100644 libc/string/stresep.c
> >
> > --
> > 2.9.5
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "OSv Development" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to osv-dev+unsubscr...@googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
> 

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH v3 5/6] command line: add tests for empty commandline string parsing

[Justin Cinkelj]

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 tests/tst-commands.cc | 57 +++
 1 file changed, 57 insertions(+)

diff --git a/tests/tst-commands.cc b/tests/tst-commands.cc
index 78f4155..cc7ffdc 100644
--- a/tests/tst-commands.cc
+++ b/tests/tst-commands.cc
@@ -22,6 +22,60 @@ static void report(bool ok, const char* msg)
 printf("%s: %s\n", (ok ? "PASS" : "FAIL"), msg);
 }
 
+static bool test_parse_empty()
+{
+std::vector<std::vector > result;
+bool ok;
+
+result = osv::parse_command_line(std::string(""), ok);
+
+if (!ok) {
+return false;
+}
+
+if (result.size() != 0) {
+return false;
+}
+
+return true;
+}
+
+static bool test_parse_space()
+{
+std::vector<std::vector > result;
+bool ok;
+
+result = osv::parse_command_line(std::string(" "), ok);
+
+if (!ok) {
+return false;
+}
+
+if (result.size() != 0) {
+return false;
+}
+
+return true;
+}
+
+static bool test_parse_spaces()
+{
+std::vector<std::vector > result;
+bool ok;
+
+result = osv::parse_command_line(std::string(" \t\n;"), ok);
+
+if (!ok) {
+return false;
+}
+
+if (result.size() != 0) {
+return false;
+}
+
+return true;
+}
+
 static bool test_parse_simplest()
 {
 std::vector<std::vector > result;
@@ -1088,6 +1142,9 @@ int main(int argc, char *argv[])
 {
 all_test_loader_parse_cmdline();
 
+report(test_parse_empty(), "empty string");
+report(test_parse_space(), "single space");
+report(test_parse_spaces(), "multiple diffrent whitespaces");
 report(test_parse_simplest(), "simplest command line");
 report(test_parse_simplest_with_args(), "simplest command line with args");
 report(test_parse_simplest_with_quotes(),
-- 
2.9.5

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH v3 3/6] command line: add loader_parse_cmdline with tests

[Justin Cinkelj]

loader_parse_cmdline accepts input str - OSv commandline, say:
--env=AA=aa --env=BB=bb1\ bb2 app.so arg1 arg2

The loader options are parsed and saved into argv, up to first not-loader-option
token. argc is set to number of loader options.
app_cmdline is set to unconsumed part of input str.

Note: quoting loader options containing space with actual double-quote or
single-quote character is not supported. Instead, backlash character is
used to allow spaces in loader options.
Missing support for quoting with single/double-quote is reason to left
commandline beyond loader options unmodified, so it can be parsed later, by
code with support for quoting.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 core/commands.cc| 120 +++
 include/osv/commands.hh |   1 +
 tests/tst-commands.cc   | 211 
 3 files changed, 332 insertions(+)

diff --git a/core/commands.cc b/core/commands.cc
index 6287d76..03777f5 100644
--- a/core/commands.cc
+++ b/core/commands.cc
@@ -318,6 +318,126 @@ std::string getcmdline()
 return std::string(osv_cmdline);
 }
 
+#define MY_DEBUG(args...) if(0) printf(args)
+/*
+loader_parse_cmdline accepts input str - OSv commandline, say:
+--env=AA=aa --env=BB=bb1\ bb2 app.so arg1 arg2
+
+The loader options are parsed and saved into argv, up to first 
not-loader-option
+token. argc is set to number of loader options.
+app_cmdline is set to unconsumed part of input str.
+Example output:
+argc = 2
+argv[0] = "--env=AA=aa"
+argv[1] = "--env=BB=bb1 bb2"
+argv[2] = NULL
+app_cmdline = "app.so arg1 arg2"
+
+The whitespaces can be escaped with '\' to allow options with spaces.
+Notes:
+ - _quoting_ loader options with space is not supported.
+ - input str is modified.
+ - output argv has to be free-d by caller.
+ - the strings pointed to by output argv and app_cmdline are in same memory as
+   original input str. The caller is not permited to modify or free data at str
+   after the call to loader_parse_cmdline, as that would corrupt returned
+   results in argv and app_cmdline.
+
+Note that std::string is intentionly not used, as it is not fully functional 
when
+called early during boot.
+*/
+void loader_parse_cmdline(char* str, int *pargc, char*** pargv, char** 
app_cmdline) {
+*pargv = nullptr;
+*pargc = 0;
+*app_cmdline = nullptr;
+
+const char *delim = " \t\n";
+char esc = '\\';
+
+// parse string
+char *ap;
+char *ap0=nullptr, *apE=nullptr; // first and last token.
+int ntoken = 0;
+ap0 = nullptr;
+while(1) {
+// Did we already consume all loader options?
+// Look at first non-space char - if =='-', than this is loader option.
+// Otherwise, it is application command.
+char *ch = str;
+while (ch && *ch != '\0') {
+if (strchr(delim, *ch)) {
+ch++;
+continue;
+}
+else if (*ch == '-') {
+// this is a loader option, continue with loader parsing
+break;
+}
+else {
+// ch is not space or '-', it is start of application command
+// Save current position and stop loader parsing.
+*app_cmdline = str;
+break;
+}
+}
+if (*ch == '\0') {
+// empty str, contains only spaces
+*app_cmdline = str;
+}
+if (*app_cmdline) {
+break;
+}
+// there are loader options, continue with parsing
+
+ap = stresep(, delim, esc);
+assert(ap);
+
+MY_DEBUG("  ap = %p %s, *ap=%d\n", ap, ap, *ap);
+if (*ap != '\0') {
+// valid token found
+ntoken++;
+if (ap0 == nullptr) {
+ap0 = ap;
+}
+apE = ap;
+}
+else {
+// Multiple consecutive delimiters found. Stresep will write 
multiple
+// '\0' into str. Squash them into one, so that argv will be 
'nice',
+// in memory consecutive array of C strings.
+if (str) {
+MY_DEBUG("shift   str %p '%s' <- %p '%s'\n", str-1, str-1, 
str, str);
+memmove(str-1, str, strlen(str) + 1);
+str--;
+}
+}
+if (str == nullptr) {
+// end of string, last char was delimiter
+*app_cmdline = ap + strlen(ap); // make app_cmdline valid pointer 
to '\0'.
+MY_DEBUG("make app_cmdline valid pointer to '\\0' ap=%p '%s', 
app_cmdline=%p '%s'\n",
+ap, ap, app_cmdline, app_cmdline);
+break;
+}
+
+}
+MY_DEBUG("  ap0 = %p '%s', apE = %p '%s', ntoken = %d, app_cmdline=%p 
'%s'\n",
+ap0, ap0, apE, apE, ntoken, *app_cmdline, *app_cmdline);
+*

[PATCH v3 1/6] libc: add stresep.c file

[Justin Cinkelj]

File is literal copy from
ftp.tku.edu.tw/NetBSD/NetBSD-current/src/lib/libc/string/stresep.c

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 libc/string/stresep.c | 98 +++
 1 file changed, 98 insertions(+)
 create mode 100644 libc/string/stresep.c

diff --git a/libc/string/stresep.c b/libc/string/stresep.c
new file mode 100644
index 000..3898305
--- /dev/null
+++ b/libc/string/stresep.c
@@ -0,0 +1,98 @@
+/* $NetBSD: stresep.c,v 1.4 2017/08/23 10:27:41 christos Exp $ */
+
+/*-
+ * Copyright (c) 1990, 1993
+ * The Regents of the University of California.  All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in the
+ *documentation and/or other materials provided with the distribution.
+ * 3. Neither the name of the University nor the names of its contributors
+ *may be used to endorse or promote products derived from this software
+ *without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include 
+#if defined(LIBC_SCCS) && !defined(lint)
+#if 0
+static char sccsid[] = "@(#)strsep.c   8.1 (Berkeley) 6/4/93";
+#else
+__RCSID("$NetBSD: stresep.c,v 1.4 2017/08/23 10:27:41 christos Exp $");
+#endif
+#endif /* LIBC_SCCS and not lint */
+
+#include "namespace.h"
+
+#include 
+#include 
+
+#ifdef __weak_alias
+__weak_alias(stresep,_stresep)
+#endif
+
+/*
+ * Get next token from string *stringp, where tokens are possibly-empty
+ * strings separated by characters from delim. If esc is not NUL, then
+ * the characters followed by esc are ignored and are not taken into account
+ * when splitting the string.
+ *
+ * Writes NULs into the string at *stringp to end tokens.
+ * delim need not remain constant from call to call.
+ * On return, *stringp points past the last NUL written (if there might
+ * be further tokens), or is NULL (if there are definitely no more tokens).
+ *
+ * If *stringp is NULL, stresep returns NULL.
+ */
+char *
+stresep(char **stringp, const char *delim, int esc)
+{
+   char *s;
+   const char *spanp;
+   int c, sc;
+   size_t l;
+   char *tok;
+
+   _DIAGASSERT(stringp != NULL);
+   _DIAGASSERT(delim != NULL);
+
+   if ((s = *stringp) == NULL)
+   return NULL;
+   l = strlen(s) + 1;
+   for (tok = s;;) {
+   c = *s++;
+   l--;
+   while (esc != '\0' && c == esc) {
+   memmove(s - 1, s, l);
+   c = *s++;
+   l--;
+   }
+   spanp = delim;
+   do {
+   if ((sc = *spanp++) == c) {
+   if (c == '\0')
+   s = NULL;
+   else
+   s[-1] = '\0';
+   *stringp = s;
+   return tok;
+   }
+   } while (sc != '\0');
+   }
+}
-- 
2.9.5

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH v3 0/6] Allow spaces in loader options

[Justin Cinkelj]

Compared to v2 patch:
The stresep.c file is now first only added, and included into Makefile
after removing unneeded define and include namespace.h.
The memcpy bug from v2 patch was reported upstrem to netbsd, and current file
now includes fix (a better fix from maintener, he also moved strlen out of 
loop).

I changed commit message for 3/6, as Nadav suggested.
Also added comment about loader_parse_cmdline() - its results argv and 
app_cmdline point
to memory of str input param, meaning caller should not free/modify str after 
calling
loader_parse_cmdline.

The std::string is intentionally not used - there is comment about that a bit 
higher
in file.

I did want to do memmove(str-1, str, ), not memmove(str, str+1).
> +// Multiple consecutive delimiters found. Stresep will write 
> multiple
> +// '\0' into str. Squash them into one, so that argv will be 
> 'nice',
> +// in memory consecutive array of C strings.
> +if (str) {
> +my_debug("shift   str %p '%s' <- %p '%s'\n", str-1, 
> str-1, str, str);
> +memmove(str-1, str, strlen(str) + 1);
> I'm confused... If I understand correctly "str" points to that second
> 0 you want to remove. So shouldn't you copy str+1 to str? why copy str
> to str-1?

justin_cinkelj@jcpc:~/devel/mikelangelo/osv-fc25/osv$ ./scripts/run.py -V -e ' 
--env=AA=aa'
OSv v0.24-442-g4bdaa11
  ap = 0x9151 --verbose, *ap=45
  ap = 0x9151000a , *ap=0
shift   str 0x9151000a '' <- 0x9151000b '--env=AA=aa'
  ap = 0x9151000a --env=AA=aa, *ap=45
make app_cmdline valid pointer to '\0' ap=0x9151000a '--env=AA=aa', 
app_cmdline=0x146e180 ''
  ap0 = 0x9151 '--verbose', apE = 0x9151000a '--env=AA=aa', 
ntoken = 2, app_cmdline=0x91510015 ''
  argv[0] = 0x9151 --verbose
  argv[1] = 0x9151000a --env=AA=aa
  ntoken = 2, ii = 2
4 CPUs detected

Input str was "--verbose " + " --env=AA=aa"
After second stresp (e.g second iteration of outer while loop), both spaces 
were replaced with '\0', ap=="\0", and str==ap+1 pointed to "--env=AA=aa".
I want str moved left by one, to overwrite '\0' at ap location.
I hope debug printf makes this clearer.
I will abuse the question to justify that debug code should be left :)


Justin Cinkelj (6):
  libc: add stresep.c file
  include stresep function
  command line: add loader_parse_cmdline with tests
  command line: use loader_parse_cmdline
  command line: add tests for empty commandline string parsing
  command line: check for empty cmdline case after parsing cmdline

 Makefile|   1 +
 arch/x64/boot.S |   4 +-
 core/commands.cc| 146 ++
 include/api/string.h|   1 +
 include/osv/commands.hh |   6 +-
 libc/string/stresep.c   |  87 
 loader.cc   |  55 --
 tests/tst-commands.cc   | 268 
 8 files changed, 510 insertions(+), 58 deletions(-)
 create mode 100644 libc/string/stresep.c

-- 
2.9.5

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH v3 2/6] include stresep function

[Justin Cinkelj]

Remove unneded #define and reference to namespace.h in stresep.c.
Include file into Makefile.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 Makefile  |  1 +
 include/api/string.h  |  1 +
 libc/string/stresep.c | 13 +
 3 files changed, 3 insertions(+), 12 deletions(-)

diff --git a/Makefile b/Makefile
index d09b3ed..0850f7c 100644
--- a/Makefile
+++ b/Makefile
@@ -1601,6 +1601,7 @@ musl += string/strnlen.o
 libc += string/strpbrk.o
 musl += string/strrchr.o
 libc += string/strsep.o
+libc += string/stresep.o
 libc += string/strsignal.o
 libc += string/strspn.o
 musl += string/strstr.o
diff --git a/include/api/string.h b/include/api/string.h
index 09201e3..6f45c3d 100644
--- a/include/api/string.h
+++ b/include/api/string.h
@@ -86,6 +86,7 @@ void *memccpy (void *__restrict, const void *__restrict, int, 
size_t);
 
 #if defined(_GNU_SOURCE) || defined(_BSD_SOURCE)
 char *strsep(char **, const char *);
+char *stresep(char **, const char *, int);
 size_t strlcat (char *, const char *, size_t);
 size_t strlcpy (char *, const char *, size_t);
 #endif
diff --git a/libc/string/stresep.c b/libc/string/stresep.c
index 3898305..5983330 100644
--- a/libc/string/stresep.c
+++ b/libc/string/stresep.c
@@ -30,22 +30,11 @@
  */
 
 #include 
-#if defined(LIBC_SCCS) && !defined(lint)
-#if 0
-static char sccsid[] = "@(#)strsep.c   8.1 (Berkeley) 6/4/93";
-#else
-__RCSID("$NetBSD: stresep.c,v 1.4 2017/08/23 10:27:41 christos Exp $");
-#endif
-#endif /* LIBC_SCCS and not lint */
-
-#include "namespace.h"
 
 #include 
 #include 
 
-#ifdef __weak_alias
-__weak_alias(stresep,_stresep)
-#endif
+#define _DIAGASSERT assert
 
 /*
  * Get next token from string *stringp, where tokens are possibly-empty
-- 
2.9.5

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] iperf: fix compile on gcc 6.3

[Justin Cinkelj]

Actually, you saw that patch, but you also found better solution, and 
committed that one.

Thank you :)

On 09/12/2017 10:32 AM, Nadav Har'El wrote:

Hi, sorry, I didn't see this patch, and committed a different patch
(c8b1906cf59d73a7cb39926c53dc4e136611d4f5) at about the same time,
doing the same thing in a different way.

--
Nadav Har'El
n...@scylladb.com


On Tue, Aug 29, 2017 at 11:37 AM, Justin Cinkelj <justin.cink...@xlab.si> wrote:

iperf failed to compile on Fedora 25, gcc 6.3, with:
In file included from /usr/include/c++/6.3.1/cmath:42:0,
  from /usr/include/c++/6.3.1/math.h:36,
  from ../include/headers.h:85,
  from ../include/Timestamp.hpp:63,
  from delay.cpp:53:
/usr/include/c++/6.3.1/bits/cpp_type_traits.h:205:12: error: redefinition of ‘struct 
std::__is_integer’
  struct __is_integer 
/*---*/
 ^
/usr/include/c++/6.3.1/bits/cpp_type_traits.h:138:12: error: previous definition of 
‘struct std::__is_integer’
  struct __is_integer 
/*--*/
 ^~

Problem is due to checking for sizeof(bool) with gcc (not g++).
Workaround is to comment out '#define bool int' in generated config.h.

Also add .gitignore

Fixes #55

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
  iperf/.gitignore | 3 +++
  iperf/Makefile   | 1 +
  2 files changed, 4 insertions(+)
  create mode 100644 iperf/.gitignore

diff --git a/iperf/.gitignore b/iperf/.gitignore
new file mode 100644
index 000..4939ce0
--- /dev/null
+++ b/iperf/.gitignore
@@ -0,0 +1,3 @@
+iperf-[0-9\.]*/
+iperf-[0-9\.]*.tar.gz
+iperf
diff --git a/iperf/Makefile b/iperf/Makefile
index 6a79af1..d956050 100644
--- a/iperf/Makefile
+++ b/iperf/Makefile
@@ -18,6 +18,7 @@ $(iperfv)/src/iperf: $(tarball)
 tar xzf $^
 patch -p0 < so.patch
 cd $(iperfv) && CFLAGS='-fPIC -g' CXXFLAGS='-fPIC -g' ./configure
+   cd $(iperfv) && sed -i 's|^#define bool int|// #define bool int|' 
config.h
 +$(MAKE) -C $(iperfv)

  $(tarball):
--
2.9.4

--
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH 2/2] wrk: allow larger file descriptor value

[Justin Cinkelj]

Commit c8b1906cf59d73a7cb39926c53dc4e136611d4f5 was broken, as it
allowed use of allocated array beyond its end. File descriptor value is
used as index into array. Array is allocated to be setsize large, and
check for "fd > setsize" prevents out-of-bound usage. Removing the check
was rather dumb :/

With this patch, the check remains in place. Instead, some extra memory
is allocated, so wrk is less likely to fail.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 ...3-wrk-allow-larger-file-descriptor-values.patch | 53 ++
 wrk/Makefile   |  1 +
 2 files changed, 54 insertions(+)
 create mode 100644 wrk/0003-wrk-allow-larger-file-descriptor-values.patch

diff --git a/wrk/0003-wrk-allow-larger-file-descriptor-values.patch 
b/wrk/0003-wrk-allow-larger-file-descriptor-values.patch
new file mode 100644
index 000..bf83a17
--- /dev/null
+++ b/wrk/0003-wrk-allow-larger-file-descriptor-values.patch
@@ -0,0 +1,53 @@
+From 40a8b2fac59623a7833d04bd9fc094f1a34f0310 Mon Sep 17 00:00:00 2001
+From: Justin Cinkelj <justin.cink...@xlab.si>
+Date: Mon, 4 Sep 2017 21:39:00 +0200
+Subject: [PATCH] wrk: allow larger file descriptor values
+
+Wrk uses socket FD number as index into dynamicaly allocated arrays.
+With say 1 conection, max expected FD was 10+3*1.
+
+On OSv, once other apps consume some FDs, arbitrary large FDs can be seen.
+Wrk than fails to start. Avoid this by allowing significantly higher
+FD values.
+
+Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
+---
+ src/ae.c  | 5 +
+ src/wrk.c | 4 +++-
+ 2 files changed, 8 insertions(+), 1 deletion(-)
+
+diff --git a/src/ae.c b/src/ae.c
+index 90be4e2..f473626 100644
+--- a/src/ae.c
 b/src/ae.c
+@@ -105,7 +105,12 @@ void aeStop(aeEventLoop *eventLoop) {
+ int aeCreateFileEvent(aeEventLoop *eventLoop, int fd, int mask,
+ aeFileProc *proc, void *clientData)
+ {
++// The eventLoop->setsize is set by:
++//  ./src/wrk.c:145:t->loop= aeCreateEventLoop(10 + 
cfg.connections * 3);
++// This is problematic on OSv. FD numbers are shared by all apps,
++// and can get arbitrary large.
+ if (fd >= eventLoop->setsize) {
++fprintf(stderr, "ERROR: wrk fd=%d > %d, increase setsize in wrk.c, 
aeCreateFileEvent()\n", fd, eventLoop->setsize);
+ errno = ERANGE;
+ return AE_ERR;
+ }
+diff --git a/src/wrk.c b/src/wrk.c
+index e64acae..14bd5e0 100644
+--- a/src/wrk.c
 b/src/wrk.c
+@@ -137,7 +137,9 @@ int main(int argc, char **argv) {
+ 
+ for (uint64_t i = 0; i < cfg.threads; i++) {
+ thread *t = [i];
+-t->loop= aeCreateEventLoop(10 + cfg.connections * 3);
++// "10 + cfg.connections * 3" is OK for Linux, and
++// "+ 200" is required for OSv to allocate some extra space.
++t->loop= aeCreateEventLoop(10 + cfg.connections * 3 + 200);
+ t->connections = connections;
+ t->stop_at = stop_at;
+ 
+-- 
+2.9.5
+
diff --git a/wrk/Makefile b/wrk/Makefile
index cce8f17..3263376 100644
--- a/wrk/Makefile
+++ b/wrk/Makefile
@@ -13,6 +13,7 @@ wrk: $(tarball)
mv wrk-$(version) wrk
cd wrk && patch -p1 < ../0001-make-build-as-DSO.patch
cd wrk && patch -p1 < ../wrk-fix-uninit-connection.patch
+   cd wrk && patch -p1 < 
../0003-wrk-allow-larger-file-descriptor-values.patch
 
 clean:
rm -rf wrk $(tarball)
-- 
2.9.5

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH 0/2] wrk: allow larger file descriptor value

[Justin Cinkelj]

The c8b1906cf59d73a7cb39926c53dc4e136611d4f5 commit allowed
array access beyond its end. I guess I didn't read what the code does :/
Hence reverting it now.

Instead, larger array is allocated, and rest of wrk code can continue
to assume that fd numbers are relatively small. The additional room
of 200 will be hopefully large enough.

Justin Cinkelj (2):
  Revert "wrk: remove check of file descriptor value"
  wrk: allow larger file descriptor value

 ...3-wrk-allow-larger-file-descriptor-values.patch | 53 ++
 ...wrk-remove-check-of-file-descriptor-value.patch | 37 ---
 wrk/Makefile   |  2 +-
 3 files changed, 54 insertions(+), 38 deletions(-)
 create mode 100644 wrk/0003-wrk-allow-larger-file-descriptor-values.patch
 delete mode 100644 wrk/0003-wrk-remove-check-of-file-descriptor-value.patch

-- 
2.9.5

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH 1/2] Revert "wrk: remove check of file descriptor value"

[Justin Cinkelj]

This reverts commit c8b1906cf59d73a7cb39926c53dc4e136611d4f5.
---
 ...wrk-remove-check-of-file-descriptor-value.patch | 37 --
 wrk/Makefile   |  1 -
 2 files changed, 38 deletions(-)
 delete mode 100644 wrk/0003-wrk-remove-check-of-file-descriptor-value.patch

diff --git a/wrk/0003-wrk-remove-check-of-file-descriptor-value.patch 
b/wrk/0003-wrk-remove-check-of-file-descriptor-value.patch
deleted file mode 100644
index 459fe07..000
--- a/wrk/0003-wrk-remove-check-of-file-descriptor-value.patch
+++ /dev/null
@@ -1,37 +0,0 @@
-From 96322a4915582f69c5d7ef1816c215885b681ba7 Mon Sep 17 00:00:00 2001
-From: Justin Cinkelj <justin.cink...@xlab.si>
-Date: Wed, 30 Aug 2017 16:37:06 +0200
-Subject: [PATCH] wrk: remove check of file descriptor value
-
-The check incorrectly fails on OSv. With say 1 conection, max expected
-fd was 10+3*1. Once other apps consume some fds, the check fails.
-
-Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>

- src/ae.c | 9 +++--
- 1 file changed, 7 insertions(+), 2 deletions(-)
-
-diff --git a/src/ae.c b/src/ae.c
-index 90be4e2..ef8ca01 100644
 a/src/ae.c
-+++ b/src/ae.c
-@@ -105,9 +105,14 @@ void aeStop(aeEventLoop *eventLoop) {
- int aeCreateFileEvent(aeEventLoop *eventLoop, int fd, int mask,
- aeFileProc *proc, void *clientData)
- {
-+// The eventLoop->setsize is set by:
-+//  ./src/wrk.c:145:t->loop= aeCreateEventLoop(10 + 
cfg.connections * 3);
-+// This does not work on OSv. FD numbers are shared by all apps,
-+// and can get arbitrary large.
- if (fd >= eventLoop->setsize) {
--errno = ERANGE;
--return AE_ERR;
-+//fprintf(stderr, "INFO: wrk, allow large fd=%d > %d in 
aeCreateFileEvent on OSv\n", fd, eventLoop->setsize);
-+//errno = ERANGE;
-+//return AE_ERR;
- }
- aeFileEvent *fe = >events[fd];
- 
--- 
-2.9.4
-
diff --git a/wrk/Makefile b/wrk/Makefile
index 0cb9165..cce8f17 100644
--- a/wrk/Makefile
+++ b/wrk/Makefile
@@ -13,7 +13,6 @@ wrk: $(tarball)
mv wrk-$(version) wrk
cd wrk && patch -p1 < ../0001-make-build-as-DSO.patch
cd wrk && patch -p1 < ../wrk-fix-uninit-connection.patch
-   cd wrk && patch -p1 < 
../0003-wrk-remove-check-of-file-descriptor-value.patch
 
 clean:
rm -rf wrk $(tarball)
-- 
2.9.5

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH] wrk: remove check of file descriptor value

[Justin Cinkelj]

wrk expects to be the only user of file descriptors, and checks their
integer value for unexpectedly high values. On OSv are file descriptors
shared among all apps/threads, and fd value does get high if other apps
started before wrk. Fix this by commenting out the check.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 ...wrk-remove-check-of-file-descriptor-value.patch | 37 ++
 wrk/Makefile   |  1 +
 2 files changed, 38 insertions(+)
 create mode 100644 wrk/0003-wrk-remove-check-of-file-descriptor-value.patch

diff --git a/wrk/0003-wrk-remove-check-of-file-descriptor-value.patch 
b/wrk/0003-wrk-remove-check-of-file-descriptor-value.patch
new file mode 100644
index 000..459fe07
--- /dev/null
+++ b/wrk/0003-wrk-remove-check-of-file-descriptor-value.patch
@@ -0,0 +1,37 @@
+From 96322a4915582f69c5d7ef1816c215885b681ba7 Mon Sep 17 00:00:00 2001
+From: Justin Cinkelj <justin.cink...@xlab.si>
+Date: Wed, 30 Aug 2017 16:37:06 +0200
+Subject: [PATCH] wrk: remove check of file descriptor value
+
+The check incorrectly fails on OSv. With say 1 conection, max expected
+fd was 10+3*1. Once other apps consume some fds, the check fails.
+
+Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
+---
+ src/ae.c | 9 +++--
+ 1 file changed, 7 insertions(+), 2 deletions(-)
+
+diff --git a/src/ae.c b/src/ae.c
+index 90be4e2..ef8ca01 100644
+--- a/src/ae.c
 b/src/ae.c
+@@ -105,9 +105,14 @@ void aeStop(aeEventLoop *eventLoop) {
+ int aeCreateFileEvent(aeEventLoop *eventLoop, int fd, int mask,
+ aeFileProc *proc, void *clientData)
+ {
++// The eventLoop->setsize is set by:
++//  ./src/wrk.c:145:t->loop= aeCreateEventLoop(10 + 
cfg.connections * 3);
++// This does not work on OSv. FD numbers are shared by all apps,
++// and can get arbitrary large.
+ if (fd >= eventLoop->setsize) {
+-errno = ERANGE;
+-return AE_ERR;
++//fprintf(stderr, "INFO: wrk, allow large fd=%d > %d in 
aeCreateFileEvent on OSv\n", fd, eventLoop->setsize);
++//errno = ERANGE;
++//return AE_ERR;
+ }
+ aeFileEvent *fe = >events[fd];
+ 
+-- 
+2.9.4
+
diff --git a/wrk/Makefile b/wrk/Makefile
index cce8f17..0cb9165 100644
--- a/wrk/Makefile
+++ b/wrk/Makefile
@@ -13,6 +13,7 @@ wrk: $(tarball)
mv wrk-$(version) wrk
cd wrk && patch -p1 < ../0001-make-build-as-DSO.patch
cd wrk && patch -p1 < ../wrk-fix-uninit-connection.patch
+   cd wrk && patch -p1 < 
../0003-wrk-remove-check-of-file-descriptor-value.patch
 
 clean:
rm -rf wrk $(tarball)
-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH] iperf: fix compile on gcc 6.3

[Justin Cinkelj]

iperf failed to compile on Fedora 25, gcc 6.3, with:
In file included from /usr/include/c++/6.3.1/cmath:42:0,
 from /usr/include/c++/6.3.1/math.h:36,
 from ../include/headers.h:85,
 from ../include/Timestamp.hpp:63,
 from delay.cpp:53:
/usr/include/c++/6.3.1/bits/cpp_type_traits.h:205:12: error: redefinition of 
‘struct std::__is_integer’
 struct __is_integer 
/*---*/
^
/usr/include/c++/6.3.1/bits/cpp_type_traits.h:138:12: error: previous 
definition of ‘struct std::__is_integer’
 struct __is_integer 
/*--*/
^~

Problem is due to checking for sizeof(bool) with gcc (not g++).
Workaround is to comment out '#define bool int' in generated config.h.

Also add .gitignore

Fixes #55

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 iperf/.gitignore | 3 +++
 iperf/Makefile   | 1 +
 2 files changed, 4 insertions(+)
 create mode 100644 iperf/.gitignore

diff --git a/iperf/.gitignore b/iperf/.gitignore
new file mode 100644
index 000..4939ce0
--- /dev/null
+++ b/iperf/.gitignore
@@ -0,0 +1,3 @@
+iperf-[0-9\.]*/
+iperf-[0-9\.]*.tar.gz
+iperf
diff --git a/iperf/Makefile b/iperf/Makefile
index 6a79af1..d956050 100644
--- a/iperf/Makefile
+++ b/iperf/Makefile
@@ -18,6 +18,7 @@ $(iperfv)/src/iperf: $(tarball)
tar xzf $^
patch -p0 < so.patch
cd $(iperfv) && CFLAGS='-fPIC -g' CXXFLAGS='-fPIC -g' ./configure
+   cd $(iperfv) && sed -i 's|^#define bool int|// #define bool int|' 
config.h
+$(MAKE) -C $(iperfv)
 
 $(tarball):
-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH v2 0/6] Allow spaces in loader options

[Justin Cinkelj]

v1 patch failed on simple
./scripts/run.py  -e "/asdf"

Problem was in:
[PATCH 4/6] command line: use loader_parse_cmdline
core/commands:cc
assert(cmdline == __loader_argv[0]);

v2 updates only this. The string returned by strdup is now stored into
static variable, so it can be easly free-d later.

Justin

Justin Cinkelj (6):
  libc: add stresep
  libc: fix of-by-one error in stresep
  command line: add loader_parse_cmdline with tests
  command line: use loader_parse_cmdline
  command line: add tests for empty commandline string parsing
  command line: check for empty cmdline case after parsing cmdline

 Makefile|   1 +
 arch/x64/boot.S |   4 +-
 core/commands.cc| 139 +
 include/api/string.h|   1 +
 include/osv/commands.hh |   6 +-
 libc/string/stresep.c   |  82 +++
 loader.cc   |  55 --
 tests/tst-commands.cc   | 268 
 8 files changed, 498 insertions(+), 58 deletions(-)
 create mode 100644 libc/string/stresep.c

-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH v2 6/6] command line: check for empty cmdline case after parsing cmdline

[Justin Cinkelj]

Always parse app_cmdline. It there is nothing to run, commands array
will be empty.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 loader.cc | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/loader.cc b/loader.cc
index 504e230..c3a7256 100644
--- a/loader.cc
+++ b/loader.cc
@@ -295,10 +295,6 @@ void parse_options(int loader_argc, char** loader_argv)
 // return the std::string and the commands_args poiting to them as a move
 std::vector<std::vector > prepare_commands(char* app_cmdline)
 {
-if (strlen(app_cmdline) == 0) {
-puts("This image has an empty command line. Nothing to run.");
-osv::poweroff();
-}
 std::vector<std::vector > commands;
 bool ok;
 
@@ -308,6 +304,10 @@ std::vector<std::vector > 
prepare_commands(char* app_cmdline)
 puts("Failed to parse command line.");
 osv::poweroff();
 }
+if (commands.size() == 0) {
+puts("This image has an empty command line. Nothing to run.");
+osv::poweroff();
+}
 
 return commands;
 }
-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH v2 2/6] libc: fix of-by-one error in stresep

[Justin Cinkelj]

memcpy had of-by-one error.
Also, remove unneeded defines.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 libc/string/stresep.c | 16 ++--
 1 file changed, 2 insertions(+), 14 deletions(-)

diff --git a/libc/string/stresep.c b/libc/string/stresep.c
index 4d72391..5ac4b85 100644
--- a/libc/string/stresep.c
+++ b/libc/string/stresep.c
@@ -30,22 +30,10 @@
  */
 
 #include 
-#if defined(LIBC_SCCS) && !defined(lint)
-#if 0
-static char sccsid[] = "@(#)strsep.c   8.1 (Berkeley) 6/4/93";
-#else
-__RCSID("$NetBSD: stresep.c,v 1.3 2017/02/12 17:19:00 maya Exp $");
-#endif
-#endif /* LIBC_SCCS and not lint */
-
-#include "namespace.h"
 
 #include 
 #include 
-
-#ifdef __weak_alias
-__weak_alias(stresep,_stresep)
-#endif
+#define _DIAGASSERT assert
 
 /*
  * Get next token from string *stringp, where tokens are possibly-empty
@@ -76,7 +64,7 @@ stresep(char **stringp, const char *delim, int esc)
for (tok = s;;) {
c = *s++;
while (esc != '\0' && c == esc) {
-   memmove(s - 1, s, strlen(s));
+   memmove(s - 1, s, strlen(s) + 1);
c = *s++;
}
spanp = delim;
-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH v2 3/6] command line: add loader_parse_cmdline with tests

[Justin Cinkelj]

loader_parse_cmdline accepts input str - OSv commandline, say:
--env=AA=aa --env=BB=bb1\ bb2 app.so arg1 arg2

The loader options are parsed and saved into argv, up to first not-loader-option
token. argc is set to number of loader options.
app_cmdline is set to unconsumed part of input str.

Note: quoting loader options with space is not supported.
This is reason to left commandline beyond loader options unmodified,
so it can be parsed later, by code with support for quoting.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 core/commands.cc| 113 ++
 include/osv/commands.hh |   1 +
 tests/tst-commands.cc   | 211 
 3 files changed, 325 insertions(+)

diff --git a/core/commands.cc b/core/commands.cc
index 6287d76..22024fa 100644
--- a/core/commands.cc
+++ b/core/commands.cc
@@ -318,6 +318,119 @@ std::string getcmdline()
 return std::string(osv_cmdline);
 }
 
+#define my_debug if(0) printf
+/*
+loader_parse_cmdline accepts input str - OSv commandline, say:
+--env=AA=aa --env=BB=bb1\ bb2 app.so arg1 arg2
+
+The loader options are parsed and saved into argv, up to first 
not-loader-option
+token. argc is set to number of loader options.
+app_cmdline is set to unconsumed part of input str.
+Example output:
+argc = 2
+argv[0] = "--env=AA=aa"
+argv[1] = "--env=BB=bb1 bb2"
+argv[2] = NULL
+app_cmdline = "app.so arg1 arg2"
+
+The whitespaces can be escaped with '\' to allow options with spaces.
+Notes:
+ - _quoting_ loader options with space is not supported.
+ - input str is modified.
+ - output argv has to be free-d by caller.
+*/
+void loader_parse_cmdline(char* str, int *pargc, char*** pargv, char** 
app_cmdline) {
+*pargv = nullptr;
+*pargc = 0;
+*app_cmdline = nullptr;
+
+const char *delim = " \t\n";
+char esc = '\\';
+
+// parse string
+char *ap;
+char *ap0=nullptr, *apE=nullptr; // first and last token.
+int ntoken = 0;
+ap0 = nullptr;
+while(1) {
+// Did we already consume all loader options?
+// Look at first non-space char - if =='-', than this is loader option.
+// Otherwise, it is application command.
+char *ch = str;
+while (ch && *ch != '\0') {
+if (strchr(delim, *ch)) {
+ch++;
+continue;
+}
+else if (*ch == '-') {
+// this is a loader option, continue with loader parsing
+break;
+}
+else {
+// ch is not space or '-', it is start of application command
+// Save current position and stop loader parsing.
+*app_cmdline = str;
+break;
+}
+}
+if (*ch == '\0') {
+// empty str, contains only spaces
+*app_cmdline = str;
+}
+if (*app_cmdline) {
+break;
+}
+// there are loader options, continue with parsing
+
+ap = stresep(, delim, esc);
+assert(ap);
+
+my_debug("  ap = %p %s, *ap=%d\n", ap, ap, *ap);
+if (*ap != '\0') {
+// valid token found
+ntoken++;
+if (ap0 == nullptr) {
+ap0 = ap;
+}
+apE = ap;
+}
+else {
+// Multiple consecutive delimiters found. Stresep will write 
multiple
+// '\0' into str. Squash them into one, so that argv will be 
'nice',
+// in memory consecutive array of C strings.
+if (str) {
+my_debug("shift   str %p '%s' <- %p '%s'\n", str-1, str-1, 
str, str);
+memmove(str-1, str, strlen(str) + 1);
+str--;
+}
+}
+if (str == nullptr) {
+// end of string, last char was delimiter
+*app_cmdline = ap + strlen(ap); // make app_cmdline valid pointer 
to '\0'.
+my_debug("make app_cmdline valid pointer to '\\0' ap=%p '%s', 
app_cmdline=%p '%s'\n",
+ap, ap, app_cmdline, app_cmdline);
+break;
+}
+
+}
+my_debug("  ap0 = %p '%s', apE = %p '%s', ntoken = %d, app_cmdline=%p 
'%s'\n",
+ap0, ap0, apE, apE, ntoken, *app_cmdline, *app_cmdline);
+*pargv = (char**)malloc(sizeof(char*) * (ntoken+1));
+// str was modified, tokes are separated by exactly one '\0'
+int ii;
+for(ap = ap0, ii = 0; ii < ntoken; ap += strlen(ap)+1, ii++) {
+assert(ap != nullptr);
+assert(*ap != '\0');
+my_debug("  argv[%d] = %p %s\n", ii, ap, ap);
+(*pargv)[ii] = ap;
+}
+my_debug("  ntoken = %d, ii = %d\n", ntoken, ii);
+assert(ii == ntoken);
+(*pargv)[ii] = nullptr;
+*pargc = ntoken;
+}
+#undef my_debug
+
 int parse_cmdline(const char *p)
 {
 ch

[PATCH v2 1/6] libc: add stresep

[Justin Cinkelj]

File is literal copy from
ftp.tku.edu.tw/NetBSD/NetBSD-current/src/lib/libc/string/stresep.c

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 Makefile  |  1 +
 include/api/string.h  |  1 +
 libc/string/stresep.c | 94 +++
 3 files changed, 96 insertions(+)
 create mode 100644 libc/string/stresep.c

diff --git a/Makefile b/Makefile
index d09b3ed..0850f7c 100644
--- a/Makefile
+++ b/Makefile
@@ -1601,6 +1601,7 @@ musl += string/strnlen.o
 libc += string/strpbrk.o
 musl += string/strrchr.o
 libc += string/strsep.o
+libc += string/stresep.o
 libc += string/strsignal.o
 libc += string/strspn.o
 musl += string/strstr.o
diff --git a/include/api/string.h b/include/api/string.h
index 09201e3..6f45c3d 100644
--- a/include/api/string.h
+++ b/include/api/string.h
@@ -86,6 +86,7 @@ void *memccpy (void *__restrict, const void *__restrict, int, 
size_t);
 
 #if defined(_GNU_SOURCE) || defined(_BSD_SOURCE)
 char *strsep(char **, const char *);
+char *stresep(char **, const char *, int);
 size_t strlcat (char *, const char *, size_t);
 size_t strlcpy (char *, const char *, size_t);
 #endif
diff --git a/libc/string/stresep.c b/libc/string/stresep.c
new file mode 100644
index 000..4d72391
--- /dev/null
+++ b/libc/string/stresep.c
@@ -0,0 +1,94 @@
+/* $NetBSD: stresep.c,v 1.3 2017/02/12 17:19:00 maya Exp $ */
+
+/*-
+ * Copyright (c) 1990, 1993
+ * The Regents of the University of California.  All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in the
+ *documentation and/or other materials provided with the distribution.
+ * 3. Neither the name of the University nor the names of its contributors
+ *may be used to endorse or promote products derived from this software
+ *without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include 
+#if defined(LIBC_SCCS) && !defined(lint)
+#if 0
+static char sccsid[] = "@(#)strsep.c   8.1 (Berkeley) 6/4/93";
+#else
+__RCSID("$NetBSD: stresep.c,v 1.3 2017/02/12 17:19:00 maya Exp $");
+#endif
+#endif /* LIBC_SCCS and not lint */
+
+#include "namespace.h"
+
+#include 
+#include 
+
+#ifdef __weak_alias
+__weak_alias(stresep,_stresep)
+#endif
+
+/*
+ * Get next token from string *stringp, where tokens are possibly-empty
+ * strings separated by characters from delim. If esc is not NUL, then
+ * the characters followed by esc are ignored and are not taken into account
+ * when splitting the string.
+ *
+ * Writes NULs into the string at *stringp to end tokens.
+ * delim need not remain constant from call to call.
+ * On return, *stringp points past the last NUL written (if there might
+ * be further tokens), or is NULL (if there are definitely no more tokens).
+ *
+ * If *stringp is NULL, stresep returns NULL.
+ */
+char *
+stresep(char **stringp, const char *delim, int esc)
+{
+   char *s;
+   const char *spanp;
+   int c, sc;
+   char *tok;
+
+   _DIAGASSERT(stringp != NULL);
+   _DIAGASSERT(delim != NULL);
+
+   if ((s = *stringp) == NULL)
+   return NULL;
+   for (tok = s;;) {
+   c = *s++;
+   while (esc != '\0' && c == esc) {
+   memmove(s - 1, s, strlen(s));
+   c = *s++;
+   }
+   spanp = delim;
+   do {
+   if ((sc = *spanp++) == c) {
+   if (c == 0)
+   s = NULL;
+   else
+   s[-1] = 0;
+   *stringp = s;
+   return tok;
+   }
+   } while (sc != 0);
+   }
+}
-- 
2.9.4

-- 
You received this mes

[PATCH v2 5/6] command line: add tests for empty commandline string parsing

[Justin Cinkelj]

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 tests/tst-commands.cc | 57 +++
 1 file changed, 57 insertions(+)

diff --git a/tests/tst-commands.cc b/tests/tst-commands.cc
index 78f4155..cc7ffdc 100644
--- a/tests/tst-commands.cc
+++ b/tests/tst-commands.cc
@@ -22,6 +22,60 @@ static void report(bool ok, const char* msg)
 printf("%s: %s\n", (ok ? "PASS" : "FAIL"), msg);
 }
 
+static bool test_parse_empty()
+{
+std::vector<std::vector > result;
+bool ok;
+
+result = osv::parse_command_line(std::string(""), ok);
+
+if (!ok) {
+return false;
+}
+
+if (result.size() != 0) {
+return false;
+}
+
+return true;
+}
+
+static bool test_parse_space()
+{
+std::vector<std::vector > result;
+bool ok;
+
+result = osv::parse_command_line(std::string(" "), ok);
+
+if (!ok) {
+return false;
+}
+
+if (result.size() != 0) {
+return false;
+}
+
+return true;
+}
+
+static bool test_parse_spaces()
+{
+std::vector<std::vector > result;
+bool ok;
+
+result = osv::parse_command_line(std::string(" \t\n;"), ok);
+
+if (!ok) {
+return false;
+}
+
+if (result.size() != 0) {
+return false;
+}
+
+return true;
+}
+
 static bool test_parse_simplest()
 {
 std::vector<std::vector > result;
@@ -1088,6 +1142,9 @@ int main(int argc, char *argv[])
 {
 all_test_loader_parse_cmdline();
 
+report(test_parse_empty(), "empty string");
+report(test_parse_space(), "single space");
+report(test_parse_spaces(), "multiple diffrent whitespaces");
 report(test_parse_simplest(), "simplest command line");
 report(test_parse_simplest_with_args(), "simplest command line with args");
 report(test_parse_simplest_with_quotes(),
-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH 6/6] command line: check for empty cmdline case after parsing cmdline

[Justin Cinkelj]

Always parse app_cmdline. It there is nothing to run, commands array
will be empty.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 loader.cc | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/loader.cc b/loader.cc
index 504e230..c3a7256 100644
--- a/loader.cc
+++ b/loader.cc
@@ -295,10 +295,6 @@ void parse_options(int loader_argc, char** loader_argv)
 // return the std::string and the commands_args poiting to them as a move
 std::vector<std::vector > prepare_commands(char* app_cmdline)
 {
-if (strlen(app_cmdline) == 0) {
-puts("This image has an empty command line. Nothing to run.");
-osv::poweroff();
-}
 std::vector<std::vector > commands;
 bool ok;
 
@@ -308,6 +304,10 @@ std::vector<std::vector > 
prepare_commands(char* app_cmdline)
 puts("Failed to parse command line.");
 osv::poweroff();
 }
+if (commands.size() == 0) {
+puts("This image has an empty command line. Nothing to run.");
+osv::poweroff();
+}
 
 return commands;
 }
-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH 3/6] command line: add loader_parse_cmdline with tests

[Justin Cinkelj]

loader_parse_cmdline accepts input str - OSv commandline, say:
--env=AA=aa --env=BB=bb1\ bb2 app.so arg1 arg2

The loader options are parsed and saved into argv, up to first not-loader-option
token. argc is set to number of loader options.
app_cmdline is set to unconsumed part of input str.

Note: quoting loader options with space is not supported.
This is reason to left commandline beyond loader options unmodified,
so it can be parsed later, by code with support for quoting.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 core/commands.cc| 113 ++
 include/osv/commands.hh |   1 +
 tests/tst-commands.cc   | 211 
 3 files changed, 325 insertions(+)

diff --git a/core/commands.cc b/core/commands.cc
index 6287d76..22024fa 100644
--- a/core/commands.cc
+++ b/core/commands.cc
@@ -318,6 +318,119 @@ std::string getcmdline()
 return std::string(osv_cmdline);
 }
 
+#define my_debug if(0) printf
+/*
+loader_parse_cmdline accepts input str - OSv commandline, say:
+--env=AA=aa --env=BB=bb1\ bb2 app.so arg1 arg2
+
+The loader options are parsed and saved into argv, up to first 
not-loader-option
+token. argc is set to number of loader options.
+app_cmdline is set to unconsumed part of input str.
+Example output:
+argc = 2
+argv[0] = "--env=AA=aa"
+argv[1] = "--env=BB=bb1 bb2"
+argv[2] = NULL
+app_cmdline = "app.so arg1 arg2"
+
+The whitespaces can be escaped with '\' to allow options with spaces.
+Notes:
+ - _quoting_ loader options with space is not supported.
+ - input str is modified.
+ - output argv has to be free-d by caller.
+*/
+void loader_parse_cmdline(char* str, int *pargc, char*** pargv, char** 
app_cmdline) {
+*pargv = nullptr;
+*pargc = 0;
+*app_cmdline = nullptr;
+
+const char *delim = " \t\n";
+char esc = '\\';
+
+// parse string
+char *ap;
+char *ap0=nullptr, *apE=nullptr; // first and last token.
+int ntoken = 0;
+ap0 = nullptr;
+while(1) {
+// Did we already consume all loader options?
+// Look at first non-space char - if =='-', than this is loader option.
+// Otherwise, it is application command.
+char *ch = str;
+while (ch && *ch != '\0') {
+if (strchr(delim, *ch)) {
+ch++;
+continue;
+}
+else if (*ch == '-') {
+// this is a loader option, continue with loader parsing
+break;
+}
+else {
+// ch is not space or '-', it is start of application command
+// Save current position and stop loader parsing.
+*app_cmdline = str;
+break;
+}
+}
+if (*ch == '\0') {
+// empty str, contains only spaces
+*app_cmdline = str;
+}
+if (*app_cmdline) {
+break;
+}
+// there are loader options, continue with parsing
+
+ap = stresep(, delim, esc);
+assert(ap);
+
+my_debug("  ap = %p %s, *ap=%d\n", ap, ap, *ap);
+if (*ap != '\0') {
+// valid token found
+ntoken++;
+if (ap0 == nullptr) {
+ap0 = ap;
+}
+apE = ap;
+}
+else {
+// Multiple consecutive delimiters found. Stresep will write 
multiple
+// '\0' into str. Squash them into one, so that argv will be 
'nice',
+// in memory consecutive array of C strings.
+if (str) {
+my_debug("shift   str %p '%s' <- %p '%s'\n", str-1, str-1, 
str, str);
+memmove(str-1, str, strlen(str) + 1);
+str--;
+}
+}
+if (str == nullptr) {
+// end of string, last char was delimiter
+*app_cmdline = ap + strlen(ap); // make app_cmdline valid pointer 
to '\0'.
+my_debug("make app_cmdline valid pointer to '\\0' ap=%p '%s', 
app_cmdline=%p '%s'\n",
+ap, ap, app_cmdline, app_cmdline);
+break;
+}
+
+}
+my_debug("  ap0 = %p '%s', apE = %p '%s', ntoken = %d, app_cmdline=%p 
'%s'\n",
+ap0, ap0, apE, apE, ntoken, *app_cmdline, *app_cmdline);
+*pargv = (char**)malloc(sizeof(char*) * (ntoken+1));
+// str was modified, tokes are separated by exactly one '\0'
+int ii;
+for(ap = ap0, ii = 0; ii < ntoken; ap += strlen(ap)+1, ii++) {
+assert(ap != nullptr);
+assert(*ap != '\0');
+my_debug("  argv[%d] = %p %s\n", ii, ap, ap);
+(*pargv)[ii] = ap;
+}
+my_debug("  ntoken = %d, ii = %d\n", ntoken, ii);
+assert(ii == ntoken);
+(*pargv)[ii] = nullptr;
+*pargc = ntoken;
+}
+#undef my_debug
+
 int parse_cmdline(const char *p)
 {
 ch

[PATCH 5/6] command line: add tests for empty commandline string parsing

[Justin Cinkelj]

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 tests/tst-commands.cc | 57 +++
 1 file changed, 57 insertions(+)

diff --git a/tests/tst-commands.cc b/tests/tst-commands.cc
index 78f4155..cc7ffdc 100644
--- a/tests/tst-commands.cc
+++ b/tests/tst-commands.cc
@@ -22,6 +22,60 @@ static void report(bool ok, const char* msg)
 printf("%s: %s\n", (ok ? "PASS" : "FAIL"), msg);
 }
 
+static bool test_parse_empty()
+{
+std::vector<std::vector > result;
+bool ok;
+
+result = osv::parse_command_line(std::string(""), ok);
+
+if (!ok) {
+return false;
+}
+
+if (result.size() != 0) {
+return false;
+}
+
+return true;
+}
+
+static bool test_parse_space()
+{
+std::vector<std::vector > result;
+bool ok;
+
+result = osv::parse_command_line(std::string(" "), ok);
+
+if (!ok) {
+return false;
+}
+
+if (result.size() != 0) {
+return false;
+}
+
+return true;
+}
+
+static bool test_parse_spaces()
+{
+std::vector<std::vector > result;
+bool ok;
+
+result = osv::parse_command_line(std::string(" \t\n;"), ok);
+
+if (!ok) {
+return false;
+}
+
+if (result.size() != 0) {
+return false;
+}
+
+return true;
+}
+
 static bool test_parse_simplest()
 {
 std::vector<std::vector > result;
@@ -1088,6 +1142,9 @@ int main(int argc, char *argv[])
 {
 all_test_loader_parse_cmdline();
 
+report(test_parse_empty(), "empty string");
+report(test_parse_space(), "single space");
+report(test_parse_spaces(), "multiple diffrent whitespaces");
 report(test_parse_simplest(), "simplest command line");
 report(test_parse_simplest_with_args(), "simplest command line with args");
 report(test_parse_simplest_with_quotes(),
-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH 4/6] command line: use loader_parse_cmdline

[Justin Cinkelj]

Parse loader options with loader_parse_cmdline instead of strtok_r.
The cmdline passed in prepare_commands to osv::parse_command_line
is not reassebled from already parsed argv anymore, as app_cmdline
contains unmodifed commandline string (only loader options are removed).

Minor bug: commandline with only whitespaces results in VM to nicely
shutdown instead of reporting:
"This image has an empty command line. Nothing to run."
E.g.
./scripts/run.py -e ' --env=AA=aa ' # OK
./scripts/run.py -e ' --env=AA=aa  ' # bug
Will be fixed in a follow up commit.

Cosmetic change: many argc/argv were renamed to loader_argc/loader_argv.

Fixes #892

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 arch/x64/boot.S |  4 ++--
 core/commands.cc| 31 ---
 include/osv/commands.hh |  5 +++--
 loader.cc   | 49 -
 4 files changed, 33 insertions(+), 56 deletions(-)

diff --git a/arch/x64/boot.S b/arch/x64/boot.S
index 1782a4c..c4b97e2 100644
--- a/arch/x64/boot.S
+++ b/arch/x64/boot.S
@@ -102,8 +102,8 @@ start64:
 mov %rbx, osv_multiboot_info
 lea init_stack_top, %rsp
 call premain
-mov __argc, %edi
-mov __argv, %rsi
+mov __loader_argc, %edi
+mov __loader_argv, %rsi
 call main
 .cfi_endproc
 
diff --git a/core/commands.cc b/core/commands.cc
index 22024fa..8607ab6 100644
--- a/core/commands.cc
+++ b/core/commands.cc
@@ -311,7 +311,6 @@ parse_command_line(const std::string line,  bool )
 // time.  So let's go for a more traditional memory management to avoid testing
 // early / not early, etc
 char *osv_cmdline = nullptr;
-static std::vector<char*> args;
 
 std::string getcmdline()
 {
@@ -433,31 +432,25 @@ void loader_parse_cmdline(char* str, int *pargc, char*** 
pargv, char** app_cmdli
 
 int parse_cmdline(const char *p)
 {
-char* save;
-
-if (args.size()) {
-// From the strtok manpage, we see that: "The first call to strtok()
-// sets this pointer to point to the first byte of the string." It
-// follows from this that the first argument contains the address we
-// should use to free the memory allocated for the string
-free(args[0]);
+if (__loader_argv) {
+if (__loader_argv[0]) {
+// __loader_argv[0] was allocated by us
+free(__loader_argv[0]);
+}
+// __loader_argv was allocated by loader_parse_cmdline
+free(__loader_argv);
 }
 
-args.resize(0);
 if (osv_cmdline) {
 free(osv_cmdline);
 }
 osv_cmdline = strdup(p);
 
-char* cmdline = strdup(p);
-
-while ((p = strtok_r(cmdline, " \t\n", )) != nullptr) {
-args.push_back(const_cast(p));
-cmdline = nullptr;
-}
-args.push_back(nullptr);
-__argv = args.data();
-__argc = args.size() - 1;
+char* cmdline = strdup(p); // will be freed on next call, as 
__loader_argv[0].
+loader_parse_cmdline(cmdline, &__loader_argc, &__loader_argv, 
&__app_cmdline);
+// Ensure that even with space at start of string p, __loader_argv[0] == 
cmdline is still true.
+assert(__loader_argv);
+assert(cmdline == __loader_argv[0]);
 
 return 0;
 }
diff --git a/include/osv/commands.hh b/include/osv/commands.hh
index 0693d28..e25ab51 100644
--- a/include/osv/commands.hh
+++ b/include/osv/commands.hh
@@ -13,8 +13,9 @@
 #include 
 #include 
 
-extern int __argc;
-extern char** __argv;
+extern int __loader_argc;
+extern char** __loader_argv;
+extern char* __app_cmdline;
 static constexpr size_t max_cmdline = 1024;
 
 namespace osv {
diff --git a/loader.cc b/loader.cc
index 0f99ba5..504e230 100644
--- a/loader.cc
+++ b/loader.cc
@@ -113,11 +113,11 @@ void premain()
 boot_time.event(".init functions");
 }
 
-int main(int ac, char **av)
+int main(int loader_argc, char **loader_argv)
 {
 smp_initial_find_current_cpu()->init_on_cpu();
-void main_cont(int ac, char** av);
-sched::init([=] { main_cont(ac, av); });
+void main_cont(int loader_argc, char** loader_argv);
+sched::init([=] { main_cont(loader_argc, loader_argv); });
 }
 
 static bool opt_leak = false;
@@ -144,20 +144,13 @@ int maxnic;
 static int sampler_frequency;
 static bool opt_enable_sampler = false;
 
-std::tuple<int, char**> parse_options(int ac, char** av)
+void parse_options(int loader_argc, char** loader_argv)
 {
 namespace bpo = boost::program_options;
 namespace bpos = boost::program_options::command_line_style;
 
 std::vector args = { "osv" };
-
-// due to https://svn.boost.org/trac/boost/ticket/6991, we can't terminate
-// command line parsing on the executable name, so we need to look for it
-// ourselves
-
-auto nr_options = std::find_if(av, av + ac,
-   [](const char* arg) { return arg[0] != '-'; 
}) - av;
-std::copy(av, av + nr_option

[PATCH 2/6] libc: fix of-by-one error in stresep

[Justin Cinkelj]

memcpy had of-by-one error.
Also, remove unneeded defines.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 libc/string/stresep.c | 16 ++--
 1 file changed, 2 insertions(+), 14 deletions(-)

diff --git a/libc/string/stresep.c b/libc/string/stresep.c
index 4d72391..5ac4b85 100644
--- a/libc/string/stresep.c
+++ b/libc/string/stresep.c
@@ -30,22 +30,10 @@
  */
 
 #include 
-#if defined(LIBC_SCCS) && !defined(lint)
-#if 0
-static char sccsid[] = "@(#)strsep.c   8.1 (Berkeley) 6/4/93";
-#else
-__RCSID("$NetBSD: stresep.c,v 1.3 2017/02/12 17:19:00 maya Exp $");
-#endif
-#endif /* LIBC_SCCS and not lint */
-
-#include "namespace.h"
 
 #include 
 #include 
-
-#ifdef __weak_alias
-__weak_alias(stresep,_stresep)
-#endif
+#define _DIAGASSERT assert
 
 /*
  * Get next token from string *stringp, where tokens are possibly-empty
@@ -76,7 +64,7 @@ stresep(char **stringp, const char *delim, int esc)
for (tok = s;;) {
c = *s++;
while (esc != '\0' && c == esc) {
-   memmove(s - 1, s, strlen(s));
+   memmove(s - 1, s, strlen(s) + 1);
c = *s++;
}
spanp = delim;
-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH 1/6] libc: add stresep

[Justin Cinkelj]

File is literal copy from
ftp.tku.edu.tw/NetBSD/NetBSD-current/src/lib/libc/string/stresep.c

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 Makefile  |  1 +
 include/api/string.h  |  1 +
 libc/string/stresep.c | 94 +++
 3 files changed, 96 insertions(+)
 create mode 100644 libc/string/stresep.c

diff --git a/Makefile b/Makefile
index d09b3ed..0850f7c 100644
--- a/Makefile
+++ b/Makefile
@@ -1601,6 +1601,7 @@ musl += string/strnlen.o
 libc += string/strpbrk.o
 musl += string/strrchr.o
 libc += string/strsep.o
+libc += string/stresep.o
 libc += string/strsignal.o
 libc += string/strspn.o
 musl += string/strstr.o
diff --git a/include/api/string.h b/include/api/string.h
index 09201e3..6f45c3d 100644
--- a/include/api/string.h
+++ b/include/api/string.h
@@ -86,6 +86,7 @@ void *memccpy (void *__restrict, const void *__restrict, int, 
size_t);
 
 #if defined(_GNU_SOURCE) || defined(_BSD_SOURCE)
 char *strsep(char **, const char *);
+char *stresep(char **, const char *, int);
 size_t strlcat (char *, const char *, size_t);
 size_t strlcpy (char *, const char *, size_t);
 #endif
diff --git a/libc/string/stresep.c b/libc/string/stresep.c
new file mode 100644
index 000..4d72391
--- /dev/null
+++ b/libc/string/stresep.c
@@ -0,0 +1,94 @@
+/* $NetBSD: stresep.c,v 1.3 2017/02/12 17:19:00 maya Exp $ */
+
+/*-
+ * Copyright (c) 1990, 1993
+ * The Regents of the University of California.  All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in the
+ *documentation and/or other materials provided with the distribution.
+ * 3. Neither the name of the University nor the names of its contributors
+ *may be used to endorse or promote products derived from this software
+ *without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include 
+#if defined(LIBC_SCCS) && !defined(lint)
+#if 0
+static char sccsid[] = "@(#)strsep.c   8.1 (Berkeley) 6/4/93";
+#else
+__RCSID("$NetBSD: stresep.c,v 1.3 2017/02/12 17:19:00 maya Exp $");
+#endif
+#endif /* LIBC_SCCS and not lint */
+
+#include "namespace.h"
+
+#include 
+#include 
+
+#ifdef __weak_alias
+__weak_alias(stresep,_stresep)
+#endif
+
+/*
+ * Get next token from string *stringp, where tokens are possibly-empty
+ * strings separated by characters from delim. If esc is not NUL, then
+ * the characters followed by esc are ignored and are not taken into account
+ * when splitting the string.
+ *
+ * Writes NULs into the string at *stringp to end tokens.
+ * delim need not remain constant from call to call.
+ * On return, *stringp points past the last NUL written (if there might
+ * be further tokens), or is NULL (if there are definitely no more tokens).
+ *
+ * If *stringp is NULL, stresep returns NULL.
+ */
+char *
+stresep(char **stringp, const char *delim, int esc)
+{
+   char *s;
+   const char *spanp;
+   int c, sc;
+   char *tok;
+
+   _DIAGASSERT(stringp != NULL);
+   _DIAGASSERT(delim != NULL);
+
+   if ((s = *stringp) == NULL)
+   return NULL;
+   for (tok = s;;) {
+   c = *s++;
+   while (esc != '\0' && c == esc) {
+   memmove(s - 1, s, strlen(s));
+   c = *s++;
+   }
+   spanp = delim;
+   do {
+   if ((sc = *spanp++) == c) {
+   if (c == 0)
+   s = NULL;
+   else
+   s[-1] = 0;
+   *stringp = s;
+   return tok;
+   }
+   } while (sc != 0);
+   }
+}
-- 
2.9.4

-- 
You received this mes

Re: [PATCH] cmdline: allow parameters with escaped spaces

[Justin Cinkelj]

I will assume that std:: and boost:: cannot be used in early in loader.cc.

with stresep, arguments with spaces cannot be quoted, but the spaces can 
be escaped. So

--env=AAA="aa bb"
cannot be used, but instead we can use
--env=AAA=aa\ bb

The commands.cc can than directly receive unparsed/original cmdline, 
just after the loader options.
So in that part of cmdline, escaping should work, I think (it is 
boost::spirit::qi based).


It is sort of ugly, if second part of cmdline will support escaping and 
quoting, but first part will support only escaping.

But in order to be lazy, I would like to go via that path.

Nadav, Miha, Gregor, rest - you opinion please. Any better ideas?

Justin

--
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] cmdline: allow parameters with escaped spaces

[Justin Cinkelj]

On 08/22/2017 02:28 PM, Nadav Har'El wrote:


On Tue, Aug 22, 2017 at 2:48 PM, Justin Cinkelj 
<justin.cink...@xlab.si <mailto:justin.cink...@xlab.si>> wrote:


> What does it not fix?
The $892 mentions environ varibales with spaces or empty one. And
command line parsing is done in loader.cc to get loader options,
and in commands.cc, to get argv/argc for app, and to implement
runscript. This one only touched loader.cc options, and ensured
that commands.cc can get argv[i] with spaces. It didn't fix
runcript. Also, when commands.cc re-assembles and re-parses
cmdline, the initial "aa bb" argv[i] get incorrectly split on space.

> everything else in the command line is glued together again into
one string
In
https://github.com/cloudius-systems/osv/blob/master/loader.cc#L319
<https://github.com/cloudius-systems/osv/blob/master/loader.cc#L319>
// concatenate everything
for (auto i = 0; i < ac; i++) {
line += std::string(av[i]) + " ";
   }
It doesn't expect av[i] to contain space. Also " or ' or \ inside
would not work, I guess.

> above parse_cmdline() there is an old (3 year old) comment saying it 
cannot use std::string.
> I guess you checked and this is incorrect?
I have not checked. I just blindly tried to use some existing
parser, and not invent/write my own.
So this code might be broken and problematic.
PS:
I also used not only debug_early, but also printf, and even
std::cout, and it worked - for me.
On the other side, Gregor complained in
https://github.com/cloudius-systems/osv/issues/892#issuecomment-323910628
<https://github.com/cloudius-systems/osv/issues/892#issuecomment-323910628>
about
std::cout not working (inside parse_cmdline, I think). So it might
depend on gcc/boost/something version.

> Why do we even need this argc/argv? We don't, really
And neither we don't really need this patch ? ;) I vote to abandon it.

I was looking if there is way to get from boost::tokenizer both
token and its start/end position in original string.
Then, loader would consume first few tokes (the loader options,
they all start with '-') from argv array.
The remaining part would be consumed by commands.cc - end position
of last loader option would be passed as commandline to it.

--env=AAA="aa bb"
is problematic case here, as it contains spaces.
So we need to support quoting and spaces and escaping.
Nothing from C++/std/boost can be used.



Why, because it is done early?
Or because you think nothing from C++/std/boost fits?

Things from std/boost would fit.
My problem/worry is it is done early, and I'm afraid that Gregor could 
not use std:cout because of that.

@Gergor
On what platform did you try to use std:cout in parse_cmdline (loader.cc)?
Was enough to add a single "cout << "blah" << endl;" to get well 
repeatable problems?
I would like to be able to check for that problem, but fedora 25 as 
build system seems to produce immune image.




strtok from C doesn't support quoting/escaping.
Are you aware of any C alternative to strtok with quoting/escaping
support? Just asking before trying to do it myself.

stresep would fit (I think), it has escape character. It is not in nm 
loader.elf output, but if I can find BSD licensed file (and it seems to 
be part of netbsd), then I will give it a try.


--
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] cmdline: allow parameters with escaped spaces

[Justin Cinkelj]

justin_cinkelj@jcpc:~/devel/mikelangelo/osv-fc25/osv$ ./scripts/run.py 
-e '/tests/misc-execve.so 1 /tests/misc-execve-payload.so aa bb \"cc 
dd\" \"ee \\\"ff gg\\\" hh\" '

OSv v0.24-432-g8388445
DBG av[0] = '/tests/misc-execve.so'
DBG av[1] = '1'
DBG av[2] = '/tests/misc-execve-payload.so'
DBG av[3] = 'aa'
DBG av[4] = 'bb'
DBG av[5] = '"cc'
DBG av[6] = 'dd"'
DBG av[7] = '"ee'
DBG av[8] = '\"ff'
DBG av[9] = 'gg\"'
DBG av[10] = 'hh"'
DBG av[11] = ''
eth0: 192.168.122.15
TT In tst-execve.c main.
TT argc 7
TT argv[0] = /tests/misc-execve.so
TT argv[1] = 1
TT argv[2] = /tests/misc-execve-payload.so
TT argv[3] = aa
TT argv[4] = bb
TT argv[5] = cc dd
TT argv[6] = ee "ff gg" hh
TT id=0 before osv_execve
PP tst-execve-payload.c main.
PP argc 6
PP argv[0] = /tests/misc-execve-payload.so
PP argv[1] = 0
PP argv[2] = aa
PP argv[3] = bb
PP argv[4] = cc dd
PP argv[5] = ee "ff gg" hh
PP delay = 0 us
PP global_var=0x100400601038 10

While it is possible to push 'ee "ff gg" hh' as a single argument, it 
requires quite a lot of escaping.

Maybe there is some additional code in commands.cc I can simplify/remove.

Justin

On 08/21/2017 06:04 PM, Justin Cinkelj wrote:

Example debug output:
./scripts/run.py -e "blah aa bb \"asdf tt\" 'a \\\"b c\\\" d' "
OSv v0.24-431-g67ecd81
DBG av[0] = 'blah'
DBG av[1] = 'aa'
DBG av[2] = 'bb'
DBG av[3] = 'asdf tt'
DBG av[4] = 'a "b c" d'
DBG av[5] = ''
eth0: 192.168.122.15
Failed to load object: blah. Powering off.

I forgot to change "#if 1" to "#if 0" in debug output.

Justin

On 08/21/2017 06:01 PM, Justin Cinkelj wrote:

The strtok_r doesn't allow escaping the input parameters, which means
each space is considered a separator. The boost::tokenizer supports
escaping. This commit replaces strtok_r with boost::tokenizer. The
final __argv is still continus in memory, as expected by some programs.

Spaces can be quoted with " or with ', and " or ' can be escaped with \.

(partially) fixes #892

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
  core/commands.cc | 52 
++--

  1 file changed, 42 insertions(+), 10 deletions(-)

diff --git a/core/commands.cc b/core/commands.cc
index 6287d76..86b25c2 100644
--- a/core/commands.cc
+++ b/core/commands.cc
@@ -19,6 +19,8 @@
  #include 
  #include 
  #include 
+#include 
+#include 
namespace qi = boost::spirit::qi;
  namespace ascii = boost::spirit::ascii;
@@ -320,13 +322,8 @@ std::string getcmdline()
int parse_cmdline(const char *p)
  {
-char* save;
-
  if (args.size()) {
-// From the strtok manpage, we see that: "The first call to 
strtok()
-// sets this pointer to point to the first byte of the 
string." It
-// follows from this that the first argument contains the 
address we

-// should use to free the memory allocated for the string
+// free date returend by malloc, and stored into args_data
  free(args[0]);
  }
  @@ -336,16 +333,51 @@ int parse_cmdline(const char *p)
  }
  osv_cmdline = strdup(p);
  -char* cmdline = strdup(p);
+std::string escSep("\\"); // escape character
+std::string delim(" \t\n"); // split on spaces, tab, newline
+std::string quote("\"'"); // allow quoted arguments
+boost::escaped_list_separator esc(escSep, delim, quote);
+typedef boost::tokenizer<boost::escaped_list_separator> 
tokenizer;

+
+// Split cmdline into tokens.
+std::string cmdline = p;
+tokenizer token{cmdline, esc};
+// args2 has same content as __argv, but strings in the later 
are also

+// consecutive in memory.
+std::vector<char*> args2;
+for (const auto  : token) {
+// strdup will create copy of temporal variable tt.
+args2.push_back(strdup(tt.c_str()));
+}
  -while ((p = strtok_r(cmdline, " \t\n", )) != nullptr) {
-args.push_back(const_cast(p));
-cmdline = nullptr;
+// Copy content from args2 to one large array args_data - string 
with multiple '\0' inside.

+// args than contains pointers into args_data.
+size_t sz = 0;
+for (const auto arg: args2) {
+sz += strlen(arg) + 1;
+}
+char* args_data = (char*)malloc(sz);
+size_t pos = 0;
+for (const auto arg: args2) {
+memcpy(args_data + pos, arg, strlen(arg) + 1);
+args.push_back(args_data + pos);
+pos += strlen(arg) + 1;
+free(arg);
  }
+assert(pos == sz);
  args.push_back(nullptr);
+
  __argv = args.data();
  __argc = args.size() - 1;
  +#if 1
+int ii = 0;
+char** ch;
+for (ch = __argv; *ch != nullptr; ch++, ii++) {
+printf("DBG av[%d] = '%s'\n", ii, *ch);
+}
+#endif
+
  return 0;
  }




--
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] cmdline: allow parameters with escaped spaces

[Justin Cinkelj]

Example debug output:
./scripts/run.py -e "blah aa bb \"asdf tt\" 'a \\\"b c\\\" d' "
OSv v0.24-431-g67ecd81
DBG av[0] = 'blah'
DBG av[1] = 'aa'
DBG av[2] = 'bb'
DBG av[3] = 'asdf tt'
DBG av[4] = 'a "b c" d'
DBG av[5] = ''
eth0: 192.168.122.15
Failed to load object: blah. Powering off.

I forgot to change "#if 1" to "#if 0" in debug output.

Justin

On 08/21/2017 06:01 PM, Justin Cinkelj wrote:

The strtok_r doesn't allow escaping the input parameters, which means
each space is considered a separator. The boost::tokenizer supports
escaping. This commit replaces strtok_r with boost::tokenizer. The
final __argv is still continus in memory, as expected by some programs.

Spaces can be quoted with " or with ', and " or ' can be escaped with \.

(partially) fixes #892

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
  core/commands.cc | 52 ++--
  1 file changed, 42 insertions(+), 10 deletions(-)

diff --git a/core/commands.cc b/core/commands.cc
index 6287d76..86b25c2 100644
--- a/core/commands.cc
+++ b/core/commands.cc
@@ -19,6 +19,8 @@
  #include 
  #include 
  #include 
+#include 
+#include 
  
  namespace qi = boost::spirit::qi;

  namespace ascii = boost::spirit::ascii;
@@ -320,13 +322,8 @@ std::string getcmdline()
  
  int parse_cmdline(const char *p)

  {
-char* save;
-
  if (args.size()) {
-// From the strtok manpage, we see that: "The first call to strtok()
-// sets this pointer to point to the first byte of the string." It
-// follows from this that the first argument contains the address we
-// should use to free the memory allocated for the string
+// free date returend by malloc, and stored into args_data
  free(args[0]);
  }
  
@@ -336,16 +333,51 @@ int parse_cmdline(const char *p)

  }
  osv_cmdline = strdup(p);
  
-char* cmdline = strdup(p);

+std::string escSep("\\"); // escape character
+std::string delim(" \t\n"); // split on spaces, tab, newline
+std::string quote("\"'"); // allow quoted arguments
+boost::escaped_list_separator esc(escSep, delim, quote);
+typedef boost::tokenizer<boost::escaped_list_separator> tokenizer;
+
+// Split cmdline into tokens.
+std::string cmdline = p;
+tokenizer token{cmdline, esc};
+// args2 has same content as __argv, but strings in the later are also
+// consecutive in memory.
+std::vector<char*> args2;
+for (const auto  : token) {
+// strdup will create copy of temporal variable tt.
+args2.push_back(strdup(tt.c_str()));
+}
  
-while ((p = strtok_r(cmdline, " \t\n", )) != nullptr) {

-args.push_back(const_cast(p));
-cmdline = nullptr;
+// Copy content from args2 to one large array args_data - string with 
multiple '\0' inside.
+// args than contains pointers into args_data.
+size_t sz = 0;
+for (const auto arg: args2) {
+sz += strlen(arg) + 1;
+}
+char* args_data = (char*)malloc(sz);
+size_t pos = 0;
+for (const auto arg: args2) {
+memcpy(args_data + pos, arg, strlen(arg) + 1);
+args.push_back(args_data + pos);
+pos += strlen(arg) + 1;
+free(arg);
  }
+assert(pos == sz);
  args.push_back(nullptr);
+
  __argv = args.data();
  __argc = args.size() - 1;
  
+#if 1

+int ii = 0;
+char** ch;
+for (ch = __argv; *ch != nullptr; ch++, ii++) {
+printf("DBG av[%d] = '%s'\n", ii, *ch);
+}
+#endif
+
  return 0;
  }
  


--
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH] cmdline: allow parameters with escaped spaces

[Justin Cinkelj]

The strtok_r doesn't allow escaping the input parameters, which means
each space is considered a separator. The boost::tokenizer supports
escaping. This commit replaces strtok_r with boost::tokenizer. The
final __argv is still continus in memory, as expected by some programs.

Spaces can be quoted with " or with ', and " or ' can be escaped with \.

(partially) fixes #892

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 core/commands.cc | 52 ++--
 1 file changed, 42 insertions(+), 10 deletions(-)

diff --git a/core/commands.cc b/core/commands.cc
index 6287d76..86b25c2 100644
--- a/core/commands.cc
+++ b/core/commands.cc
@@ -19,6 +19,8 @@
 #include 
 #include 
 #include 
+#include 
+#include 
 
 namespace qi = boost::spirit::qi;
 namespace ascii = boost::spirit::ascii;
@@ -320,13 +322,8 @@ std::string getcmdline()
 
 int parse_cmdline(const char *p)
 {
-char* save;
-
 if (args.size()) {
-// From the strtok manpage, we see that: "The first call to strtok()
-// sets this pointer to point to the first byte of the string." It
-// follows from this that the first argument contains the address we
-// should use to free the memory allocated for the string
+// free date returend by malloc, and stored into args_data
 free(args[0]);
 }
 
@@ -336,16 +333,51 @@ int parse_cmdline(const char *p)
 }
 osv_cmdline = strdup(p);
 
-char* cmdline = strdup(p);
+std::string escSep("\\"); // escape character
+std::string delim(" \t\n"); // split on spaces, tab, newline
+std::string quote("\"'"); // allow quoted arguments
+boost::escaped_list_separator esc(escSep, delim, quote);
+typedef boost::tokenizer<boost::escaped_list_separator> tokenizer;
+
+// Split cmdline into tokens.
+std::string cmdline = p;
+tokenizer token{cmdline, esc};
+// args2 has same content as __argv, but strings in the later are also
+// consecutive in memory.
+std::vector<char*> args2;
+for (const auto  : token) {
+// strdup will create copy of temporal variable tt.
+args2.push_back(strdup(tt.c_str()));
+}
 
-while ((p = strtok_r(cmdline, " \t\n", )) != nullptr) {
-args.push_back(const_cast(p));
-cmdline = nullptr;
+// Copy content from args2 to one large array args_data - string with 
multiple '\0' inside.
+// args than contains pointers into args_data.
+size_t sz = 0;
+for (const auto arg: args2) {
+sz += strlen(arg) + 1;
+}
+char* args_data = (char*)malloc(sz);
+size_t pos = 0;
+for (const auto arg: args2) {
+memcpy(args_data + pos, arg, strlen(arg) + 1);
+args.push_back(args_data + pos);
+pos += strlen(arg) + 1;
+free(arg);
 }
+assert(pos == sz);
 args.push_back(nullptr);
+
 __argv = args.data();
 __argc = args.size() - 1;
 
+#if 1
+int ii = 0;
+char** ch;
+for (ch = __argv; *ch != nullptr; ch++, ii++) {
+printf("DBG av[%d] = '%s'\n", ii, *ch);
+}
+#endif
+
 return 0;
 }
 
-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH] busybox: avoid X_Y_stable branch name

[Justin Cinkelj]

After 1_27_stable changed from 1.27.1 to 1.27.2, the GET script didn't
work anymore. We should use fixed version number.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 busybox/GET | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/busybox/GET b/busybox/GET
index cf7f916..f822965 100755
--- a/busybox/GET
+++ b/busybox/GET
@@ -3,8 +3,8 @@
 set -e
 
 URL=https://github.com/mirror/busybox
-VERSION_GIT=1_27_stable
-VERSION_LIB=1.27.1
+VERSION_GIT=1_27_2
+VERSION_LIB=`echo $VERSION_GIT | sed 's/_/./g'`
 
 [ ! -d build ] && mkdir build
 cd build
-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] busybox ash: do not exit if command doesn't exist

[Justin Cinkelj]

My bad, I assumed 1_27_stable is stable. But the busybox source was 
updated, and 1_27_stable now points to 1_27_2 instead of 1_27_2. I 
switched version in GET to avoid X_Y_stable branch name. Patch sent.


Justin

On 08/20/2017 06:45 PM, Nadav Har'El wrote:
Hi Justin, I committed this patch (to master branch of osv.git, you'll 
need to go to it in apps/ to get it), but something doesn't work now - 
when I try to build the busybox image, I get this:


libbusybox: 0_lib/libbusybox.so.1.27.2
busybox linked against libbusybox: 0_lib/busybox
make[1]: Leaving directory 
'/home/nyh/osv.tmp2/osv/apps/busybox/build/busybox'

/usr/bin/ld: cannot find -lbusybox
collect2: error: ld returned 1 exit status

Any idea what might be wrong?


--
Nadav Har'El
n...@scylladb.com <mailto:n...@scylladb.com>

On Fri, Aug 18, 2017 at 3:02 PM, Justin Cinkelj 
<justin.cink...@xlab.si <mailto:justin.cink...@xlab.si>> wrote:


0004 patch prevents poping up spurious error message 'command not
found'
when running in ash say 'ls /'. It requires patch to osv_execve to be
applied - osv_execve should clear errno on success, and set errno on
failure.

0005 patch prevent ash exit when non-existent command is tried.

As @nyh suggested, symlink /usr/bin -> /usr/lib is added, so that
ash works without adding /usr/lib to PATH environ.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si
<mailto:justin.cink...@xlab.si>>
---
 busybox/GET|   5 +-
 ...-command-not-found-error-message-when-com.patch | 39 ++
 ...not-exit-if-program-to-be-run-fails-to-st.patch | 140
+
 busybox/test-script/cmd0.sh|   0
 4 files changed, 183 insertions(+), 1 deletion(-)
 create mode 100644

busybox/patches/0004-OSv-ash-fix-command-not-found-error-message-when-com.patch
 create mode 100644

busybox/patches/0005-OSv-ash-do-not-exit-if-program-to-be-run-fails-to-st.patch
 mode change 100644 => 100755 busybox/test-script/cmd0.sh

diff --git a/busybox/GET b/busybox/GET
index 90f2ab1..cf7f916 100755
--- a/busybox/GET
+++ b/busybox/GET
@@ -34,7 +34,10 @@ cat < usr.manifest
 /usr/lib/echo: ->/usr/lib/busybox
 /usr/lib/ping: ->/usr/lib/busybox
 #
+# Link /usr/bin (in ash's default PATH) to /usr/lib where we put
the executables
+/usr/bin: ->/usr/lib
+#
 # test scripts
 ## /**: \${MODULE_DIR}/test-script/**

-EOF
\ No newline at end of file
+EOF
diff --git

a/busybox/patches/0004-OSv-ash-fix-command-not-found-error-message-when-com.patch

b/busybox/patches/0004-OSv-ash-fix-command-not-found-error-message-when-com.patch
new file mode 100644
index 000..2d5043f
--- /dev/null
+++

b/busybox/patches/0004-OSv-ash-fix-command-not-found-error-message-when-com.patch
@@ -0,0 +1,39 @@
+From 1512e2b29f3d5c19d0735470d74c546bd15ed15a Mon Sep 17 00:00:00
2001
+From: Justin Cinkelj <justin.cink...@xlab.si
<mailto:justin.cink...@xlab.si>>
+Date: Fri, 18 Aug 2017 07:01:58 +0200
+Subject: [PATCH 4/5] OSv ash: fix command not found error message
when command
+ was found
+
+Running simple 'ls' in ash showed directory content. But at the end,
+spurious error message 'command not found' was shown. This was due to
+errno not being cleared by osv_execve. This was fixed in
osv_execve, now
+it clears errno on success, or sets it to meanigful value (currently
    +always ENOENT).
+
+Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si
<mailto:justin.cink...@xlab.si>>
+---
+ shell/ash.c | 8 
+ 1 file changed, 4 insertions(+), 4 deletions(-)
+
+diff --git a/shell/ash.c b/shell/ash.c
+index bc16394..dc88bd8 100644
+--- a/shell/ash.c
 b/shell/ash.c
+@@ -7731,10 +7731,10 @@ tryexec(IF_FEATURE_SH_STANDALONE(int
applet_no,) char *cmd, char **argv, char **
+   int ret = 0;
+   pid_t ch_pid = -1;
+   ret = osv_execve(cmd, argv, envp, (long*)_pid, -1);
+-  //errno = 0; // osv_execve should clear errno on
success
+-  if(ret)
+-  errno = ENOSYS;
+-  osv_waittid(ch_pid, NULL, 0);
++  // osv_execve now clears errno on success, as
expected by caller.
++  if(ret == 0) {
++  osv_waittid(ch_pid, NULL, 0);
++  }
+   return;
+   }
+ #endif
+--
+2.9.4
+
diff --git

a/busybox/patches/0005-OSv-ash-do-not-exit-if-program-to-be-run-fails-to-st.patch

b/busybox/patches/0005-OSv-ash-do-not-exit-if-program-to-be-run-fails-to-st.patch
new file mode 100644
index 000..3206cd2
--- /dev/null
+++
  

Re: [PATCH 2/2] osv_execve: on failure set errno and return -1

[Justin Cinkelj]

On 08/20/2017 06:33 PM, Nadav Har'El wrote:
I'll commit this patch, but I have some minor questions below. You can 
send a new incremental patch if you want to change something.



--
Nadav Har'El
n...@scylladb.com <mailto:n...@scylladb.com>

On Fri, Aug 18, 2017 at 3:01 PM, Justin Cinkelj 
<justin.cink...@xlab.si <mailto:justin.cink...@xlab.si>> wrote:


On failure execve() returns -1 and sets errno. This patch implements
same behaviour for osv_execve. In addition, it clears errno on
success.


What does it even mean to clear errno on execve() success? If execve() 
succeeds, the original program is no longer running :-)
You're right that the new program should start with a zero errno, so I 
guess if that's what you mean, it's true.

You are right it could (should?) be done in different way.
In Linux, after fork, execve succeeds and original program is no longer 
running, or it fails, original program get returned -1 and can check errno.
In OSv, here is no fork, and osv_execve starts new thread. Then 
osv_execve succeeds, but the original program is still running, and can 
check errno. Ash does such check.


It is similar with nginx. There, line with "pid = fork();" was replaced 
with "pid = 0;", and that is enough to make same thread to run both 
parent and child code.


Justin


thread_run_app_in_namespace and osv_execve run in different
threads. To
send errno value from first to the second, and additional value has to
be saved into exited_threads map. So instead of saving a single int
(thread/app exit code) into exited_threads, now we save an object
containing exit and errno code.

osv_execve then has to wait and check if errno_code is set. This
requires
osv_execve to always ask for thread_id of newly started app, so it can
later search for exit/errno code of the app. Actualy this
was already required, as thread_run_app_in_namespace always
dereferences
thread_id pointer. Second it also requires osv_execve to always pass a
valid parent_waiter, as it will always have to check for possible
failure.
Additional assert()s were added, "just in case".

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si
<mailto:justin.cink...@xlab.si>>
---
 core/osv_execve.cc | 70
+++---
 1 file changed, 46 insertions(+), 24 deletions(-)

diff --git a/core/osv_execve.cc b/core/osv_execve.cc
index 862dac8..f4505cf 100644
--- a/core/osv_execve.cc
+++ b/core/osv_execve.cc
@@ -8,7 +8,17 @@
 /* Record thread state changes (termination) by storing exit
status into a map.
  * It is used to implement waitpid like functionality for threads
(osv_waittid).
  **/
-static std::unordered_map<long, int> exited_threads;
+class application_status {
+public:
+application_status() {
+exit_code = 0;
+errno_code = 0;
+};
+public:
+int exit_code;
+int errno_code;

+};
+static std::unordered_map<long, application_status> exited_threads;
 static mutex exec_mutex;
 static condvar cond;
 extern "C" long gettid(); // implemented in linux.cc
@@ -24,32 +34,32 @@ static int
thread_run_app_in_namespace(std::string filename,
 {
 const bool new_program = true; // run in new ELF namespace
 long tid = gettid(); // sched::thread::current()->id();
-int ret = -1;
-bool exception_raised = false;
+application_status app_status;

 debugf_execve("thread_run_app_in_namespace... tid=%ld\n", tid);
+assert(thread_id != nullptr);
+assert(parent_waiter != nullptr);
 *thread_id = tid;

 try {
 auto app = osv::application::run_and_join(filename, args,
new_program, , parent_waiter);
-ret = app->get_return_code();
-debugf_execve("thread_run_app_in_namespace ret = %d
tid=%ld\n", ret, tid);
+app_status.exit_code = app->get_return_code();

+debugf_execve("thread_run_app_in_namespace ret = %d
tid=%ld\n", app_status.exit_code, tid);
 }
 catch (osv::launch_error ) {
-exception_raised = true;
+app_status.errno_code = ENOENT; // Assume the only
possible problem is "no such file"
+app_status.exit_code = -1;

 }


 WITH_LOCK(exec_mutex) {
-exited_threads[tid] = ret;
+exited_threads[tid] = app_status;
 cond.wake_all();
 }

-if (exception_raised) {
+if (app_status.errno_code != 0) {
 // osv::application::run_and_join failed, and likely
didn't wake up parent_waiter.
-// Wake parent now.
-if (parent_waiter) {
-parent_waiter-

Re: [PATCH] Add busybox

[Justin Cinkelj]

On 08/20/2017 05:15 PM, Nadav Har'El wrote:




On Fri, Aug 18, 2017 at 3:17 PM, Justin Cinkelj 
<justin.cink...@xlab.si <mailto:justin.cink...@xlab.si>> wrote:


Thank you for commit, and for suggested improvements.

3 inter-dependent patches were sent, 2 for osv, 1 for osv-apps
(that one includes 2 patches for busybox source).

I wasn't sure if osv_execve should (is allowed to) clear errno on
success. This doesn't make any sense in linux - new program will
start executing, and nobody can check the errno in old program.
But for busybox/ash, someone (ash or OSv) has to clear it, so I
moved that to OSv.

If I understand correctly, syscalls are permitted to clear errno
on success (but most of them don't), and errno value should be
retrieved immediately after failed call. So setting errno to 0 is
not illegal.

Even with these patches, ash is still not 'fully functional'. I
see (only now) that script can be run as
run.py -e 'ash -c /cmd0.sh'
but not as:
run.py -e 'ash'
/ # /cmd0.sh
ash: /cmd0.sh: not found

Second case fails, as osv::app tries to execute cmd0.sh, which is
not a shared object file. I guess '!#/usr/bin/ash' should be added
to head of cmd0.sh, and ash should check for it.


In Linux, and Unix starting the mid 1980s, execve() itself checks the 
beginning of the file for the "shebang" characters: "#!", and if it 
exists, it runs that command specified there instead of the file the 
user asked to run.


If the Shebang is *missing* in the file, then Linux assumes it is a 
native Linux executable. But if it turns out not to be, execve() 
returns ENOEXEC. When ash or other shells get ENOEXEC from their 
execve() call, they assume (or should assume, I didn't check the ash 
source code) that this is a shell script, and run it themselves.

Thank you for explanation; I'm glad I mentioned the rather obvious problem.


So I think there are two things to fix - the first one probably more 
interesting and the second can be postponed indefinitely:


1. osv_execve() should recognize the case where the files it is trying 
to load doesn't have the executable magic header, and fail with 
ENOEXEC. This will hopefully allow ash to run scripts without a 
shebang. I don't remember what the code in core/elf.cc does in this 
case, if it doesn't fail gracefully when running a non-executable, it 
should be fixed. Please let me know if you need help with that. I'll 
open an issue.
I looked at ash, and there is if(errno==ENOEXEC) in ash.c code (tryexec 
function), and - 20 line long comment there also explains what you 
already said. So this approach should work.


2. osv_execve() should recognize the shebang in a file and behave 
Linux. This will allow running scripts with shebangs as expected.


For the same reason, cmd0.sh cannot call a second script.

I'm also suspicious about what would happen when called script
does exit or return. Maybe also a parent shell(s) will exit/return.


Normally, a script is run in a separate process - in Unix, when 
shebang is not used, this is a "subshell" (a *forked* copy of the 
original shell), when shebang is used this is a completely new shell 
(fork and exec). Since it is a separate process, it can exit normally.


Since you modified ash to replace fork by a new thread, and "exit" 
with exiting a thread, maybe all of this would just work normally.
However, what really worries me is that unlike Unix processes, when an 
OSv thread exits, nothing is cleaned behind it: memory allocated 
remain allocated, open files remain open, etc. All these subshells may 
be leaking resources. It might be good enough for one-of things, but 
may cause problems if you intend to run a lot of these shells on the 
same VM - although I doubt anyone would.
Yes, I realized that. Hopefully busybox valgrind/memory tests helped to 
remove many of the leaks. One hard limit is the max 32 ELF namespaces 
(which are never reclaimed); so we can run ls only 32-times. I guess 
interactive use will hit that pretty fast :/


What I hope for is that Miha will use environment variables expansion, 
if/while sentences and other 'light' features of ash. Light features - I 
think some of busybox modules are implemented in a way that permit to 
just call their main, without doing execve/osv_execve; but maybe I 
couldn't use that, because I needed to build busybox as a shared lib, 
and both options might be mutually exclusive.


I noticed ENABLE_FEATURE_CLEAN_UP somewhere in busybox code. I'm not 
sure what it is, but it might be something we want to have enabled.


Justin



Anyway, ash will be now a bit more useful, and Miha can start
writing ash scripts. But only simple scripts - calling an
additional script is considered an advanced topic, and he should
not try that ;)

Justin




--
You received this message because you are subscribed to the Google 

[PATCH 0/2] osv_execve fix behaviour in case of failure

[Justin Cinkelj]

While testing ash from busybox, Nadav suggested that osv_execve should:
1) Catch exception when command to be run cannot be run. This will prevent
VM shutdown when we attempt to run non-existen script in ash.
This is implemented in first patch.

2) osv_execve should behave more linux-like when it fails. It should
return -1 and set errno. This is in the second patch.

Additional patch for osv-apps/busybox will be sent.

Justin Cinkelj (2):
  osv_execve: fail gracefully if program file does not exist
  osv_execve: on failure set errno and return -1

 core/osv_execve.cc | 71 +-
 1 file changed, 54 insertions(+), 17 deletions(-)

-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH 1/2] osv_execve: fail gracefully if program file does not exist

[Justin Cinkelj]

If non-existent path is passed to osv_execve, osv::launch_error is raised.
This terminates VM if nobody catches the exception.

This patch implements exception catching, and silently ignores it -
osv_execve still returns 0 and doesn't set errno if exception occurs.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 core/osv_execve.cc | 21 ++---
 1 file changed, 18 insertions(+), 3 deletions(-)

diff --git a/core/osv_execve.cc b/core/osv_execve.cc
index 218cc78..862dac8 100644
--- a/core/osv_execve.cc
+++ b/core/osv_execve.cc
@@ -24,19 +24,34 @@ static int thread_run_app_in_namespace(std::string filename,
 {
 const bool new_program = true; // run in new ELF namespace
 long tid = gettid(); // sched::thread::current()->id();
+int ret = -1;
+bool exception_raised = false;
 
 debugf_execve("thread_run_app_in_namespace... tid=%ld\n", tid);
 *thread_id = tid;
 
-auto app = osv::application::run_and_join(filename, args, new_program, 
, parent_waiter);
-auto ret = app->get_return_code();
-debugf_execve("thread_run_app_in_namespace ret = %d tid=%ld\n", ret, tid);
+try {
+auto app = osv::application::run_and_join(filename, args, new_program, 
, parent_waiter);
+ret = app->get_return_code();
+debugf_execve("thread_run_app_in_namespace ret = %d tid=%ld\n", ret, 
tid);
+}
+catch (osv::launch_error ) {
+exception_raised = true;
+}
 
 WITH_LOCK(exec_mutex) {
 exited_threads[tid] = ret;
 cond.wake_all();
 }
 
+if (exception_raised) {
+// osv::application::run_and_join failed, and likely didn't wake up 
parent_waiter.
+// Wake parent now.
+if (parent_waiter) {
+parent_waiter->wake();
+}
+}
+
 // Trigger event notification via file descriptor (fd created with 
eventfd).
 if (notification_fd > 0) {
 uint64_t notif = 1;
-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH] busybox ash: do not exit if command doesn't exist

[Justin Cinkelj]

0004 patch prevents poping up spurious error message 'command not found'
when running in ash say 'ls /'. It requires patch to osv_execve to be
applied - osv_execve should clear errno on success, and set errno on
failure.

0005 patch prevent ash exit when non-existent command is tried.

As @nyh suggested, symlink /usr/bin -> /usr/lib is added, so that
ash works without adding /usr/lib to PATH environ.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 busybox/GET|   5 +-
 ...-command-not-found-error-message-when-com.patch |  39 ++
 ...not-exit-if-program-to-be-run-fails-to-st.patch | 140 +
 busybox/test-script/cmd0.sh|   0
 4 files changed, 183 insertions(+), 1 deletion(-)
 create mode 100644 
busybox/patches/0004-OSv-ash-fix-command-not-found-error-message-when-com.patch
 create mode 100644 
busybox/patches/0005-OSv-ash-do-not-exit-if-program-to-be-run-fails-to-st.patch
 mode change 100644 => 100755 busybox/test-script/cmd0.sh

diff --git a/busybox/GET b/busybox/GET
index 90f2ab1..cf7f916 100755
--- a/busybox/GET
+++ b/busybox/GET
@@ -34,7 +34,10 @@ cat < usr.manifest
 /usr/lib/echo: ->/usr/lib/busybox
 /usr/lib/ping: ->/usr/lib/busybox
 #
+# Link /usr/bin (in ash's default PATH) to /usr/lib where we put the 
executables
+/usr/bin: ->/usr/lib
+#
 # test scripts
 ## /**: \${MODULE_DIR}/test-script/**
 
-EOF
\ No newline at end of file
+EOF
diff --git 
a/busybox/patches/0004-OSv-ash-fix-command-not-found-error-message-when-com.patch
 
b/busybox/patches/0004-OSv-ash-fix-command-not-found-error-message-when-com.patch
new file mode 100644
index 000..2d5043f
--- /dev/null
+++ 
b/busybox/patches/0004-OSv-ash-fix-command-not-found-error-message-when-com.patch
@@ -0,0 +1,39 @@
+From 1512e2b29f3d5c19d0735470d74c546bd15ed15a Mon Sep 17 00:00:00 2001
+From: Justin Cinkelj <justin.cink...@xlab.si>
+Date: Fri, 18 Aug 2017 07:01:58 +0200
+Subject: [PATCH 4/5] OSv ash: fix command not found error message when command
+ was found
+
+Running simple 'ls' in ash showed directory content. But at the end,
+spurious error message 'command not found' was shown. This was due to
+errno not being cleared by osv_execve. This was fixed in osv_execve, now
+it clears errno on success, or sets it to meanigful value (currently
+always ENOENT).
+
+Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
+---
+ shell/ash.c | 8 
+ 1 file changed, 4 insertions(+), 4 deletions(-)
+
+diff --git a/shell/ash.c b/shell/ash.c
+index bc16394..dc88bd8 100644
+--- a/shell/ash.c
 b/shell/ash.c
+@@ -7731,10 +7731,10 @@ tryexec(IF_FEATURE_SH_STANDALONE(int applet_no,) char 
*cmd, char **argv, char **
+   int ret = 0;
+   pid_t ch_pid = -1;
+   ret = osv_execve(cmd, argv, envp, (long*)_pid, -1);
+-  //errno = 0; // osv_execve should clear errno on success
+-  if(ret)
+-  errno = ENOSYS;
+-  osv_waittid(ch_pid, NULL, 0);
++  // osv_execve now clears errno on success, as expected by 
caller.
++  if(ret == 0) {
++  osv_waittid(ch_pid, NULL, 0);
++  }
+   return;
+   }
+ #endif
+-- 
+2.9.4
+
diff --git 
a/busybox/patches/0005-OSv-ash-do-not-exit-if-program-to-be-run-fails-to-st.patch
 
b/busybox/patches/0005-OSv-ash-do-not-exit-if-program-to-be-run-fails-to-st.patch
new file mode 100644
index 000..3206cd2
--- /dev/null
+++ 
b/busybox/patches/0005-OSv-ash-do-not-exit-if-program-to-be-run-fails-to-st.patch
@@ -0,0 +1,140 @@
+From 1009e202f7c1214e7ab6185782f1033506a634a8 Mon Sep 17 00:00:00 2001
+From: Justin Cinkelj <justin.cink...@xlab.si>
+Date: Fri, 18 Aug 2017 07:26:12 +0200
+Subject: [PATCH 5/5] OSv ash: do not exit if program to be run fails to start
+
+If program to be started fails to start (executable file not found), ash
+should not return from its main loop, as this terminates shell.
+
+ash_main interceps exception from raise_exception, and exits because
+exception_type == EXEXIT. The exit is avoided if instead of returning from
+ash_main, code just continues with cmdloop(). Important detail - we have
+to first call popstackmark, and there re-execute setjmp, so that longjmp
+will work.
+
+As EXEXIT is used for both abnormal exit (exception handling, like
+file not found) and normal exit (entering 'exit' cmd into ash), this
+breaks 'exit' cmd. Thus we have to separete the two cases.
+This is done by replacing EXEXIT with:
+ - EXEXIT_CMD, when 'exit' was passed to ash interpreter.
+ - EXEXIT_VAR, for all other reasons.
+
+Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
+---
+ shell/ash.c | 44 +---
+ 1 file changed, 37 insertions(+), 7 deletions(-)
+
+diff --git a/shell/ash.c b/shell/ash.c
+index dc88bd8..d8111c0 100644
+--- a/shell/ash.c
 b/shell/ash.c
+@@ -324,7 +324,

[PATCH 2/2] osv_execve: on failure set errno and return -1

[Justin Cinkelj]

On failure execve() returns -1 and sets errno. This patch implements
same behaviour for osv_execve. In addition, it clears errno on success.

thread_run_app_in_namespace and osv_execve run in different threads. To
send errno value from first to the second, and additional value has to
be saved into exited_threads map. So instead of saving a single int
(thread/app exit code) into exited_threads, now we save an object
containing exit and errno code.

osv_execve then has to wait and check if errno_code is set. This requires
osv_execve to always ask for thread_id of newly started app, so it can
later search for exit/errno code of the app. Actualy this
was already required, as thread_run_app_in_namespace always dereferences
thread_id pointer. Second it also requires osv_execve to always pass a
valid parent_waiter, as it will always have to check for possible failure.
Additional assert()s were added, "just in case".

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 core/osv_execve.cc | 70 +++---
 1 file changed, 46 insertions(+), 24 deletions(-)

diff --git a/core/osv_execve.cc b/core/osv_execve.cc
index 862dac8..f4505cf 100644
--- a/core/osv_execve.cc
+++ b/core/osv_execve.cc
@@ -8,7 +8,17 @@
 /* Record thread state changes (termination) by storing exit status into a map.
  * It is used to implement waitpid like functionality for threads 
(osv_waittid).
  **/
-static std::unordered_map<long, int> exited_threads;
+class application_status {
+public:
+application_status() {
+exit_code = 0;
+errno_code = 0;
+};
+public:
+int exit_code;
+int errno_code;
+};
+static std::unordered_map<long, application_status> exited_threads;
 static mutex exec_mutex;
 static condvar cond;
 extern "C" long gettid(); // implemented in linux.cc
@@ -24,32 +34,32 @@ static int thread_run_app_in_namespace(std::string filename,
 {
 const bool new_program = true; // run in new ELF namespace
 long tid = gettid(); // sched::thread::current()->id();
-int ret = -1;
-bool exception_raised = false;
+application_status app_status;
 
 debugf_execve("thread_run_app_in_namespace... tid=%ld\n", tid);
+assert(thread_id != nullptr);
+assert(parent_waiter != nullptr);
 *thread_id = tid;
 
 try {
 auto app = osv::application::run_and_join(filename, args, new_program, 
, parent_waiter);
-ret = app->get_return_code();
-debugf_execve("thread_run_app_in_namespace ret = %d tid=%ld\n", ret, 
tid);
+app_status.exit_code = app->get_return_code();
+debugf_execve("thread_run_app_in_namespace ret = %d tid=%ld\n", 
app_status.exit_code, tid);
 }
 catch (osv::launch_error ) {
-exception_raised = true;
+app_status.errno_code = ENOENT; // Assume the only possible problem is 
"no such file"
+app_status.exit_code = -1;
 }
 
 WITH_LOCK(exec_mutex) {
-exited_threads[tid] = ret;
+exited_threads[tid] = app_status;
 cond.wake_all();
 }
 
-if (exception_raised) {
+if (app_status.errno_code != 0) {
 // osv::application::run_and_join failed, and likely didn't wake up 
parent_waiter.
-// Wake parent now.
-if (parent_waiter) {
-parent_waiter->wake();
-}
+// Wake parent now, after errno is stored into exited_threads
+parent_waiter->wake();
 }
 
 // Trigger event notification via file descriptor (fd created with 
eventfd).
@@ -57,7 +67,7 @@ static int thread_run_app_in_namespace(std::string filename,
 uint64_t notif = 1;
 write(notification_fd, , sizeof(notif));
 }
-return ret;
+return app_status.exit_code;
 }
 
 static std::vector argv_to_array(const char *const argv[])
@@ -99,6 +109,9 @@ extern "C" {
  * terminates.
  * If notification_fd is used, osv_waittid can wait on arbitrary thread and
  * return thread_id of (first) finished thread.
+ *
+ * On sucess, 0 is returned. In addition, errno is cleared to 0.
+ * On failure, -1 is returned and errno is set.
  **/
 int osv_execve(const char *path, char *const argv[], char *const envp[],
 long *thread_id, int notification_fd)
@@ -109,9 +122,10 @@ int osv_execve(const char *path, char *const argv[], char 
*const envp[],
 debugf_execve("OSv osv_execve:%d   argv[0]=%p %s\n", __LINE__, argv, argv? 
argv[0]:NULL);
 debugf_execve("OSv osv_execve:%d   envp[0]=%p %s\n", __LINE__, envp, envp? 
envp[0]:NULL);
 
-if (thread_id) {
-*thread_id = 0;
-}
+// input thread_id might be NULL
+long tid, *ptid;
+ptid = thread_id? thread_id: 
+*ptid = 0;
 
 /*
  * We have to start new program in new thread, otherwise current thread
@@ -129,16 +143,24 @@ int osv_execve(const char *path, char *const argv[], char 
*const envp[],
 waiter w(sch

Re: [PATCH] Enhanced scripts/build and related Python scripts to support export files that make up an image versus uploading them to an image

[Justin Cinkelj]

On 08/15/2017 03:38 PM, Nadav Har'El wrote:


On Wed, Aug 9, 2017 at 2:47 PM, Waldemar Kozaczuk 
> wrote:


This patch enhances OSv build scripts to allow exporting files in
addition
of uploading them to an image. It addresses 3 usage scenarios
described in
 #900 and demostrated by examples below:

- ./scripts/build image=empty export=all - export files that are
specified is usr.manifest.skel in order to be able to produce
capstan osv.bootstrap package

- ./scripts/build image=java-isolated export=all usrskel=none -
export files that are part of OSv modules (./modules directory)
along with all other dependent modules minus what is part of
usr.manifest.skel

- ./scripts/build image=openjdk8-zulu-compact1 export=selected
usrskel=none - export files that are part of an app (most
applicable to apps/openjdk8-) without any dependent modules

The export logic is enabled by passing new parameter export
[=none|all|selected] to scripts/build.
Exported files are placed under build/export directory.

Please note that the changes are backwards compatible. However
changes to scripts/upload_manifest.py
 will break patches that are part of

https://github.com/mikelangelo-project/capstan-packages/tree/master/docker_files/common

.
Hopefully this patch will make some of these patches obsolete.


Justin, can you please take a look what kind of grief (or on the 
contrary?) this patch will cause you if committed?
Miha, I guess you are familiar (author?) of building capstan image in 
docker feature? Could you review the code?

Justin



Fixed #900

Signed-off-by: Waldemar Kozaczuk >
---
 modules/empty/module.py|  0
 scripts/build  | 16 +++-
 scripts/export_manifest.py | 91
++
 scripts/manifest_common.py | 78
+++
 scripts/module.py  |  8 +++-
 scripts/upload_manifest.py | 79
+---
 6 files changed, 192 insertions(+), 80 deletions(-)
 create mode 100644 modules/empty/module.py
 create mode 100755 scripts/export_manifest.py
 create mode 100644 scripts/manifest_common.py

diff --git a/modules/empty/module.py b/modules/empty/module.py
new file mode 100644
index 000..e69de29
diff --git a/scripts/build b/scripts/build
index 7cee19c..14b4fe1 100755
--- a/scripts/build
+++ b/scripts/build
@@ -62,6 +62,7 @@ then
exit 2
 fi
 OUT=build/`readlink build/last`
+EXPORT_DIR=`realpath build/export`


 # Look for arguments that look like "name=value", and set them in an
 # assoative array "vars". Also look for the "-j ..." parallelism
option,
@@ -69,6 +70,7 @@ OUT=build/`readlink build/last`
 declare -A vars
 j_arg=
 checknumber=
+vars[export]=none
 for i
 do
if test -n "$checknumber"
@@ -147,7 +149,12 @@ then
usrskel_arg="--usrskel ${vars[usrskel]}"
 fi

-jdkbase=$jdkbase ARCH=$arch mode=$mode OSV_BASE=$SRC
OSV_BUILD_PATH=$OSV_BUILD_PATH scripts/module.py $j_arg build -c
$modules $usrskel_arg

+export=${vars[export]}


Instead of setting the default "none" above, you could do this in this 
line:


export=${vars[export]-none}

I think it will be clearer (and more similar to the way we set the 
defaults for the other variables).


+if [ "$export" != "none" ]
+then
+rm -rf $EXPORT_DIR


This is really scary. What if we have a bug and $EXPORT_DIR is . or / 
or some other mistake? What if there's a space in EXPORT_DIR?


For example, consider that the current directory is "/home/nyh/osv 2" 
(with a space).

This code will run
 rm -rf /home/nyh/osv 2

So will remove the unrelated directory (!) /home/nyh/osv and also ./2, 
who knows what those might contain :-(


I suggest to just be explicit and write

rm -rf build/export

Unless I'm missing something, you don't even need the full path.

+fi
+jdkbase=$jdkbase ARCH=$arch mode=$mode OSV_BASE=$SRC
OSV_BUILD_PATH=$OSV_BUILD_PATH scripts/module.py $j_arg build -c
$modules $usrskel_arg --export $export


 bootfs_manifest=$manifest make "${args[@]}" | tee -a build.out
 # check exit status of make
@@ -180,7 +187,12 @@ zfs)
qemu-img convert -f raw -O qcow2 bare.raw usr.img
qemu-img resize usr.img ${fs_size}b >/dev/null 2>&1

-   $SRC/scripts/upload_manifest.py -o usr.img -m usr.manifest
-D jdkbase=$jdkbase -D gccbase=$gccbase -D glibcbase=$glibcbase -D
miscbase=$miscbase
+   if [ "$export" == "none" ]
+   then
+   $SRC/scripts/upload_manifest.py 

Re: [PATCH] Add busybox

[Justin Cinkelj]

I mentioned BSD licence in Makefile only. I can remove it if that makes 
patch busybox/GPL compliant, and is also acceptable for OSv.


--
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [COMMIT osv-apps master] Add python2.7 from the build system

[Justin Cinkelj]

Nadav, you intentionally committed this one instead of the "Add 
python2/3 from the build system"?
The second one is more generic, as python3 can be compiled (albeit 
cannot be run "yet").


Justin

On 08/03/2017 02:26 PM, Commit Bot wrote:

From: Justin Cinkelj <justin.cink...@xlab.si>
Committer: Nadav Har'El <n...@scylladb.com>
Branch: master

Add python2.7 from the build system

Instead of compiling python from source, we reuse libpython2.7.so
from the build system. Only the python executable is recompiled as
shared object. The python.c file is almost literal copy from the git
source (only 'include "Python.h"' is replaced with declaration of
Py_Main function). The gcc command to compile python.so includes
-lpython2.7. This likely requires python-devel to be installed on
build system.

The GET file includes all required shared libraries. The install_shlibs()
is called 4 times, until no new libs are required (actually, it only
checks that no new libs were added - should be good enough).

usr.manifest adds symlink /lib64 to /lib. This is required to run
python simply as:
./scripts/run.py -e "/python"
./scripts/run.py -e "/python -c \"aa={1:22,3:44}; print aa; print 
'asdf'\""

Otherwise, it was required to set PYTHONPATH, (like in
./scripts/run.py -e "--env=PYTHONPATH=/lib/python2.7/ /python").

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
Message-Id: <20170801114115.2452-3-justin.cink...@xlab.si>

---
diff --git a/python27/GET b/python27/GET
--- a/python27/GET
+++ b/python27/GET
@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+set -e
+
+VERSION=2.7
+BASEDIR=$PWD
+ROOTFS=$BASEDIR/ROOTFS
+
+install_shlibs() {
+SHLIBS=""
+SHLIBS+=" $ROOTFS/python.so "
+SHLIBS+=" `find $ROOTFS -iname '*\.so' | grep -v '/site-packages/'` "
+set +e
+SHLIBS+=" `find $ROOTFS -iname '*\.so[\.0-9]*' | grep -v 
'/site-packages/'` "

+set -e
+SHLIBS_COUNT="`echo \"$SHLIBS\" | wc -l`"
+
+ldd $SHLIBS | grep -Po '(?<=> )/[^ ]+' | sort | uniq | grep -Pv 
'lib(c|gcc|dl|m|util|rt|pthread|stdc\+\+|selinux|krb5|gssapi_krb5)\.so' 
| xargs -I {} install  {} $ROOTFS/usr/lib
+# ROOTFS/lib/python2.7/config/libpython2.7.so is a symlink to 
../../libpython2.7.so,
+# so create a valid destination to avoid ldd error due to dangling 
symlink.
+(cd $ROOTFS/lib && ln -sf ../usr/lib/libpython$VERSION.so.1.0 
libpython$VERSION.so)

+echo "$SHLIBS_COUNT"
+}
+
+main() {
+mkdir -p build/
+gcc -o build/python.so python.c -fPIC -shared -lpython${VERSION}
+
+rm -rf "$ROOTFS"
+mkdir -p "$ROOTFS/usr/lib"
+mkdir -p "$ROOTFS/lib/python$VERSION"
+
+cp build/python.so "$ROOTFS"
+install_shlibs
+# TODO /lib64/python2.7/ should not be hardcoded?
+PY_LIB1=/lib64/python$VERSION/
+rsync -a $PY_LIB1 $ROOTFS/lib/python$VERSION/ --exclude test 
--exclude unittest \

+--exclude '*.pyc' --exclude '*.pyo' --exclude '*.egg-info'
+
+SHLIBS_COUNT4=`install_shlibs`
+echo "Python SHLIBS_COUNT4=$SHLIBS_COUNT4"
+SHLIBS_COUNT3=`install_shlibs`
+echo "Python SHLIBS_COUNT3=$SHLIBS_COUNT3"
+SHLIBS_COUNT2=`install_shlibs`
+echo "Python SHLIBS_COUNT2=$SHLIBS_COUNT2"
+SHLIBS_COUNT1=`install_shlibs`
+echo "Python SHLIBS_COUNT1=$SHLIBS_COUNT1"
+if [ $SHLIBS_COUNT1 -ne $SHLIBS_COUNT2 ]
+then
+# if this happens, just add additional calls to install_shlibs()
+echo "ERROR some libraries required by python might be missing"
+exit 1
+fi
+}
+
+main
diff --git a/python27/Makefile b/python27/Makefile
--- a/python27/Makefile
+++ b/python27/Makefile
@@ -0,0 +1,8 @@
+
+.PHONY: module clean
+
+module:
+./GET
+
+clean:
+rm -fr ROOTFS build
diff --git a/python27/python.c b/python27/python.c
--- a/python27/python.c
+++ b/python27/python.c
@@ -0,0 +1,22 @@
+/* Minimal main program -- everything is loaded from the library */
+
+//#include "Python.h"
+extern int Py_Main(int argc, char **argv);
+
+#ifdef __FreeBSD__
+#include 
+#endif
+
+int
+main(int argc, char **argv)
+{
+/* 754 requires that FP exceptions run in "no stop" mode by default,
+ * and until C vendors implement C99's ways to control FP 
exceptions,

+ * Python requires non-stop mode.  Alas, some platforms enable FP
+ * exceptions by default.  Here we disable them.
+ */
+#ifdef __FreeBSD__
+fedisableexcept(FE_OVERFLOW);
+#endif
+return Py_Main(argc, argv);
+}
diff --git a/python27/usr.manifest b/python27/usr.manifest
--- a/python27/usr.manifest
+++ b/python27/usr.manifest
@@ -0,0 +1,5 @@
+/python: ->/python.so
+# to remove need for --env=PYTHONPATH=/lib/python2.7/, as _my_ build 
system has python .py files in /lib64/python2.7/

+/lib64: ->/lib
+
+/**: ${MODULE_DIR}/ROOTFS/**


--
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH v2 2/2] Add python2/3 from the build system

[Justin Cinkelj]

Instead of compiling python from source, we reuse libpythonX.Y.so
from the build system. Only the python executable is recompiled as
shared object.

While python 2.7, 3.3, 3.4 and 3.5 can be compiled, only the 2.7
actually works. The 3.Y executables require additional symbols/functions
from OSv. A list for 3.5:
./scripts/check-libcfunc-avail.sh debug 
apps/python/ROOTFS/usr/lib/libpython3.5m.so.1.0
mkfifoat not found
sched_getscheduler not found
flistxattr not found
unlinkat not found
__wcscat_chk not found
fsetxattr not found
llistxattr not found
fremovexattr not found
lgetxattr not found
listxattr not found
sched_getparam not found
sched_setscheduler not found
sigtimedwait not found
readlinkat not found
fexecve not found
mkdirat not found
__wcsncpy_chk not found
__wcscpy_chk not found
lsetxattr not found
clock_settime not found
fgetxattr not found
sched_rr_get_interval not found
__xmknodat not found
linkat not found
removexattr not found
getgrouplist not found
sigpending not found
sched_setparam not found
fchownat not found
getxattr not found
renameat not found
setxattr not found
symlinkat not found
waitid not found
lremovexattr not found

The GET file includes all required shared libraries. The install_shlibs()
is called 4 times, until no new libs are required (actually, it only
checks that no new libs were added - should be good enough).

usr.manifest adds symlink /lib64 to /lib. This is required to run
python simply as:
./scripts/run.py -e "/python"
./scripts/run.py -e "/python -c \"aa={1:22,3:44}; print aa; print 'asdf'\""
Otherwise, it was required to set PYTHONPATH, (like in
./scripts/run.py -e "--env=PYTHONPATH=/lib/python2.7/ /python").

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 python/.gitignore   |   2 ++
 python/GET  | 100 
 python/Makefile |  10 ++
 python/Python-2.7.h |   9 +
 python/Python-3.3.h |  16 +
 python/Python-3.4.h |  15 
 python/Python-3.5.h |  16 +
 7 files changed, 168 insertions(+)
 create mode 100644 python/.gitignore
 create mode 100755 python/GET
 create mode 100644 python/Makefile
 create mode 100644 python/Python-2.7.h
 create mode 100644 python/Python-3.3.h
 create mode 100644 python/Python-3.4.h
 create mode 100644 python/Python-3.5.h

diff --git a/python/.gitignore b/python/.gitignore
new file mode 100644
index 000..b807de2
--- /dev/null
+++ b/python/.gitignore
@@ -0,0 +1,2 @@
+build
+usr.manifest
diff --git a/python/GET b/python/GET
new file mode 100755
index 000..4371560
--- /dev/null
+++ b/python/GET
@@ -0,0 +1,100 @@
+#!/usr/bin/env bash
+set -e
+
+#VERSION=2.7
+BASEDIR=$PWD
+ROOTFS=$BASEDIR/ROOTFS
+
+case "$VERSION" in
+2.7)
+   
PYTHON_C_URL="https://raw.githubusercontent.com/python/cpython/$VERSION/Modules/python.c;
+   PYLIB_VERSION="2.7"
+;;
+3.3)
+   
PYTHON_C_URL="https://raw.githubusercontent.com/python/cpython/$VERSION/Modules/python.c;
+   PYLIB_VERSION="3.3m"
+;;
+3.4)
+   
PYTHON_C_URL="https://raw.githubusercontent.com/python/cpython/$VERSION/Modules/python.c;
+   PYLIB_VERSION="3.4m"
+;;
+3.5)
+   
PYTHON_C_URL="https://raw.githubusercontent.com/python/cpython/$VERSION/Programs/python.c;
+   # we have to link agains libpython3.5m.so.1.0
+   PYLIB_VERSION="3.5m"
+;;
+*)
+echo "Unknown python version"
+exit 1
+esac
+
+install_shlibs() {
+SHLIBS=""
+SHLIBS+=" $ROOTFS/python$VERSION.so "
+SHLIBS+=" `find $ROOTFS -iname '*\.so' | grep -v '/site-packages/'` "
+set +e
+SHLIBS+=" `find $ROOTFS -iname '*\.so[\.0-9]*' | grep -v '/site-packages/'` "
+set -e
+SHLIBS_COUNT="`echo \"$SHLIBS\" | wc -l`"
+
+ldd $SHLIBS | grep -Po '(?<=> )/[^ ]+' | sort | uniq | grep -Pv 
'lib(c|gcc|dl|m|util|rt|pthread|stdc\+\+|selinux|krb5|gssapi_krb5)\.so' | xargs 
-I {} install  {} $ROOTFS/usr/lib
+# ROOTFS/lib/python2.7/config/libpython2.7.so is a symlink to 
../../libpython2.7.so,
+# so create a valid destination to avoid ldd error due to dangling symlink.
+(cd $ROOTFS/lib && ln -sf ../usr/lib/libpython$PYLIB_VERSION.so.1.0 
libpython$PYLIB_VERSION.so)
+echo "$SHLIBS_COUNT"
+}
+
+main() {
+mkdir -p build/
+
+# compile python executable
+wget $PYTHON_C_URL -O build/python-$VERSION.c
+sed -i "s|Python.h|../Python-$VERSION.h|" build/python-$VERSION.c
+gcc -o build/python$VERSION.so build/python-$VERSION.c -fPIC -shared 
-lpython${PYLIB_VERSION}
+
+# copy python executable, required .so libs and .py files to ROOTFS
+rm -rf "$ROOTFS"
+mkdir -p "$ROOTFS/usr/lib"
+mkdir -p "$ROOTFS/lib/python$VERSION"
+
+cp build/python$VERSION.so "$ROOTFS"
+install_shlibs
+# TODO /lib64/python2.7/ should not be hardcoded?
+# FIXME /lib64/python$VERSION/ can contai

[PATCH v2 1/2] Revert "Add python2.7"

[Justin Cinkelj]

This reverts commit 9cdcf94d7d14d117bcbc790ff52287f3fb5ecf0d.
---
 python27/Makefile  | 52 --
 python27/copy-required-system-libraries.sh | 27 
 2 files changed, 79 deletions(-)
 delete mode 100644 python27/Makefile
 delete mode 100755 python27/copy-required-system-libraries.sh

diff --git a/python27/Makefile b/python27/Makefile
deleted file mode 100644
index 7e1f2da..000
--- a/python27/Makefile
+++ /dev/null
@@ -1,52 +0,0 @@
-VERSION=2.7
-SOURCE=https://github.com/python/cpython.git
-
-PYTHON_PREFIX=$(PWD)/src/python-install
-
-
-.PHONY: module clean ROOTFS
-
-SRC=src/cpython
-
-module: usr.manifest
-
-usr.manifest: ROOTFS
-   echo '/**: $${MODULE_DIR}/ROOTFS/**' > usr.manifest
-
-clean:
-   rm -fr usr.manifest ROOTFS src
-
-# Note: the touch commands below are needed after commands which create files
-# "in the past", like wget and tar, and can confuse Make to rebuild a target
-# which is new, just pretends to be old.
-$(SRC)/.git/HEAD:
-   mkdir src
-   cd src; git clone $(SOURCE)
-   cd $(SRC); git checkout -b $(VERSION) origin/$(VERSION)
-   #touch $@
-
-$(SRC)/Makefile: $(SRC)/.git/HEAD
-   cd $(SRC); ./configure --enable-shared --prefix=$(PYTHON_PREFIX)
-   cd $(SRC); cp Makefile Makefile.orig;
-   cd $(SRC); sed -i 's/^LDFLAGS=.*/LDFLAGS=-shared/g' Makefile
-
-$(SRC)/python: $(SRC)/Makefile
-   cd $(SRC); $(MAKE) python
-   cd $(SRC); mv python python.so
-   # Re-link python, this time as ELF executable.
-   # This is required for make install to succeed.
-   cd $(SRC); $(MAKE) -f Makefile.orig install
-
-ROOTFS: $(SRC)/python
-   rm ROOTFS -rf
-   mkdir -p ROOTFS/usr/lib
-
-   cp $(SRC)/python.so ROOTFS/python
-   cp $(PYTHON_PREFIX)/lib/libpython$(VERSION).so.1.0 ROOTFS/usr/lib
-   # Copy the Python environment. Since we'd like to exclude some dirs, 
rsync is used
-   rsync -a $(PYTHON_PREFIX)/lib/python$(VERSION) ROOTFS/lib --exclude 
test --exclude unittest \
-   --exclude '*.pyc' --exclude '*.pyo' --exclude '*.egg-info' 
--exclude '*.o' --exclude 'libpython'$(VERSION)'.a'
-   # TODO maybe .so files in ROOTFS/lib/python2.7/lib-dynload should be 
stripped.
-   ./copy-required-system-libraries.sh
-
-.DELETE_ON_ERROR:
diff --git a/python27/copy-required-system-libraries.sh 
b/python27/copy-required-system-libraries.sh
deleted file mode 100755
index 81a585b..000
--- a/python27/copy-required-system-libraries.sh
+++ /dev/null
@@ -1,27 +0,0 @@
-#!/bin/bash
-
-# Copy required libraries from the build system.
-# Likely, we don't want to include all libs (and their dependencies) from
-# ldd ROOTFS/lib/python2.7/lib-dynload/*so | grep -v -e '^ROOTFS/' | sed 
's/(0x.*)//' | sort | uniq | awk '{print $3}'
-# So for now, only two (most obviously required) dependency libs will be 
included.
-
-set -e
-
-PYLIBS=""
-PYLIBS+=" ROOTFS/lib/python2.7/lib-dynload/readline.so "
-PYLIBS+=" ROOTFS/lib/python2.7/lib-dynload/_curses.so "
-
-DEPLIBS=""
-for pylib in $PYLIBS
-do
-   DEPLIBS+=`ldd $pylib | egrep -v -e linux-vdso.so -e ld-linux-x86-64.so 
-e 'lib(python[23]\.[0-9]|pthread|c|dl|m|util|).so.*'`
-   DEPLIBS+="\n"
-done
-#echo -e "DEPLIBS1:\n$DEPLIBS"
-DEPLIBS=`echo -e "$DEPLIBS" | sed '/^$/d' | awk '{print $3}' | sort | uniq`
-#echo -e "DEPLIBS2:\n$DEPLIBS"
-
-for deplib in $DEPLIBS
-do
-   /bin/cp $deplib ROOTFS/usr/lib
-done
-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH v2 0/2] Add python2/3

[Justin Cinkelj]

Per Nadav comments, I extended v1 patch to also compile python3.Y
binaries. The problem with 3.Y is that additional symbols are
required, so binaries cannot be run yet :)

I also don't like that copying core python .py files from
/lib64/pythonX.Y/ includes some python files, for which I
belive are not part of minimal python installation.
For example, /lib64/python2.7/{idlelib,distutils,email,bsddb}.
Minimal image size can still be achieved by building image on a
clean machine, and that should be good enough.

Justin Cinkelj (2):
  Revert "Add python2.7"
  Add python2/3 from the build system

 python/.gitignore  |   2 +
 python/GET | 100 +
 python/Makefile|  10 +++
 python/Python-2.7.h|   9 +++
 python/Python-3.3.h|  16 +
 python/Python-3.4.h|  15 +
 python/Python-3.5.h|  16 +
 python27/Makefile  |  52 ---
 python27/copy-required-system-libraries.sh |  27 
 9 files changed, 168 insertions(+), 79 deletions(-)
 create mode 100644 python/.gitignore
 create mode 100755 python/GET
 create mode 100644 python/Makefile
 create mode 100644 python/Python-2.7.h
 create mode 100644 python/Python-3.3.h
 create mode 100644 python/Python-3.4.h
 create mode 100644 python/Python-3.5.h
 delete mode 100644 python27/Makefile
 delete mode 100755 python27/copy-required-system-libraries.sh

-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [RFC PATCH 2/2] Add python2.7 from the build system

[Justin Cinkelj]

I tried to compile python3.5. At least it compiles, and then comes 
missing functions part.
justin_cinkelj@jcpc:~/devel/mikelangelo/osv-fc25/osv$ 
./scripts/check-libcfunc-avail.sh debug 
apps/python27/ROOTFS/usr/lib/libpython3.5m.so.1.0

mkfifoat not found
sched_getscheduler not found
flistxattr not found
unlinkat not found
__wcscat_chk not found
fsetxattr not found
llistxattr not found
fremovexattr not found
lgetxattr not found
listxattr not found
sched_getparam not found
sched_setscheduler not found
sigtimedwait not found
readlinkat not found
fexecve not found
mkdirat not found
__wcsncpy_chk not found
__wcscpy_chk not found
lsetxattr not found
clock_settime not found
fgetxattr not found
sched_rr_get_interval not found
__xmknodat not found
linkat not found
removexattr not found
getgrouplist not found
sigpending not found
sched_setparam not found
fchownat not found
getxattr not found
renameat not found
setxattr not found
symlinkat not found
waitid not found
lremovexattr not found

I don't plan to implement (or to just _blindly_ reuse musl 
implementations, as I did with python2.7, and then Nadav had to check 
the musl code) all of this.

At least for now, nobody need python 3.x, so this can wait.
But I will include update to patch v2.

As this approach (reuse libs from build system) requires - well, libs on 
the build system, I can build only python2.7, 3.3 and 3.5. I don't think 
there is python 3.6 in (stock) fedora 25.
And with other/older distros, it will also be a bit problematic to build 
newer pythons.


Justin

On 08/02/2017 08:08 AM, Justin Cinkelj wrote:



On 08/01/2017 11:38 PM, Nadav Har'El wrote:


On Tue, Aug 1, 2017 at 2:41 PM, Justin Cinkelj 
<justin.cink...@xlab.si <mailto:justin.cink...@xlab.si>> wrote:


Instead of compiling python from source, we reuse libpython2.7.so
<http://libpython2.7.so>
from the build system. Only the python executable is recompiled as
shared object. The python.c file is almost literal copy from the git
source (only 'include "Python.h"' is replaced with declaration of
Py_Main function). The gcc command to compile python.so includes
-lpython2.7. This likely requires python-devel to be installed on
build system.


No, libpython2.7.so <http://libpython2.7.so> comes (in Fedora) from 
the python-libs package, not
python-devel. It is necessary to run (not just develop) python, 
because the

main python executable (a tiny executable, as you saw) just links to it.
I thought that "gcc -lpython2.7" searches for libpython2.7.a, and that 
that one comes from -devel package.

Thank you for the clarification.
I'm curious, is there any reason why you are doing this for python 2, 
not 3?
Will exactly the same tricks also work for python 3, or will that 
have new

problems (and potentially more missing functions, etc.)
In the old patch, I tried switching VERSION from 2.7 to 3.6, and it 
didn't work.

https://github.com/python/cpython/blob/master/Programs/python.c
https://github.com/python/cpython/blob/2.7/Modules/python.c
main() in new versions calls more Py_* functions, so I guess 
additional "extern " should be added.

I will try this, it seems it will work.

Is there a way to build OSv image like ./scripts/build 
image=python.v27, and have multiple targets in Makefile?
What I want to achieve is one app/python/ dir, which can build more 
that one python version.
I know python.v27 already has different meaning (it would require 
attribute named v27 in apps/python/module.py,

and it would do a totally different thing).



The GET file includes all required shared libraries. The
install_shlibs()
is called 4 times, until no new libs are required (actually, it only
checks that no new libs were added - should be good enough).

usr.manifest adds symlink /lib64 to /lib. This is required to run
python simply as:
./scripts/run.py -e "/python"
./scripts/run.py -e "/python -c \"aa={1:22,3:44}; print aa; print
'asdf'\""
Otherwise, it was required to set PYTHONPATH, (like in
./scripts/run.py -e "--env=PYTHONPATH=/lib/python2.7/ /python").

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si
<mailto:justin.cink...@xlab.si>>
---
 python27/GET  | 55
+++
 python27/Makefile |  8 
 python27/python.c | 22 +
 python27/usr.manifest |  5 +
 4 files changed, 90 insertions(+)
 create mode 100755 python27/GET
 create mode 100644 python27/Makefile
 create mode 100644 python27/python.c
 create mode 100644 python27/usr.manifest

diff --git a/python27/GET b/python27/GET
new file mode 100755
index 000..05774df
--- /dev/null
+++ b/python27/GET
@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+set -e
+
+VERSION=2.7
+BASEDIR=$PWD
+ROOTFS=$

Re: [RFC PATCH 2/2] Add python2.7 from the build system

[Justin Cinkelj]

On 08/01/2017 11:38 PM, Nadav Har'El wrote:


On Tue, Aug 1, 2017 at 2:41 PM, Justin Cinkelj <justin.cink...@xlab.si 
<mailto:justin.cink...@xlab.si>> wrote:


Instead of compiling python from source, we reuse libpython2.7.so
<http://libpython2.7.so>
from the build system. Only the python executable is recompiled as
shared object. The python.c file is almost literal copy from the git
source (only 'include "Python.h"' is replaced with declaration of
Py_Main function). The gcc command to compile python.so includes
-lpython2.7. This likely requires python-devel to be installed on
build system.


No, libpython2.7.so <http://libpython2.7.so> comes (in Fedora) from 
the python-libs package, not
python-devel. It is necessary to run (not just develop) python, 
because the

main python executable (a tiny executable, as you saw) just links to it.
I thought that "gcc -lpython2.7" searches for libpython2.7.a, and that 
that one comes from -devel package.

Thank you for the clarification.
I'm curious, is there any reason why you are doing this for python 2, 
not 3?

Will exactly the same tricks also work for python 3, or will that have new
problems (and potentially more missing functions, etc.)
In the old patch, I tried switching VERSION from 2.7 to 3.6, and it 
didn't work.

https://github.com/python/cpython/blob/master/Programs/python.c
https://github.com/python/cpython/blob/2.7/Modules/python.c
main() in new versions calls more Py_* functions, so I guess additional 
"extern " should be added.

I will try this, it seems it will work.

Is there a way to build OSv image like ./scripts/build image=python.v27, 
and have multiple targets in Makefile?
What I want to achieve is one app/python/ dir, which can build more that 
one python version.
I know python.v27 already has different meaning (it would require 
attribute named v27 in apps/python/module.py,

and it would do a totally different thing).



The GET file includes all required shared libraries. The
install_shlibs()
is called 4 times, until no new libs are required (actually, it only
checks that no new libs were added - should be good enough).

usr.manifest adds symlink /lib64 to /lib. This is required to run
python simply as:
./scripts/run.py -e "/python"
./scripts/run.py -e "/python -c \"aa={1:22,3:44}; print aa; print
'asdf'\""
Otherwise, it was required to set PYTHONPATH, (like in
./scripts/run.py -e "--env=PYTHONPATH=/lib/python2.7/ /python").

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si
<mailto:justin.cink...@xlab.si>>
---
 python27/GET  | 55
+++
 python27/Makefile |  8 
 python27/python.c | 22 +
 python27/usr.manifest |  5 +
 4 files changed, 90 insertions(+)
 create mode 100755 python27/GET
 create mode 100644 python27/Makefile
 create mode 100644 python27/python.c
 create mode 100644 python27/usr.manifest

diff --git a/python27/GET b/python27/GET
new file mode 100755
index 000..05774df
--- /dev/null
+++ b/python27/GET
@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+set -e
+
+VERSION=2.7
+BASEDIR=$PWD
+ROOTFS=$BASEDIR/ROOTFS
+
+install_shlibs() {
+SHLIBS=""
+SHLIBS+=" $ROOTFS/python.so "
+SHLIBS+=" `find $ROOTFS -iname '*\.so' | grep -v '/site-packages/'` "
+set +e
+SHLIBS+=" `find $ROOTFS -iname '*\.so[\.0-9]*' | grep -v
'/site-packages/'` "
+set -e
+SHLIBS_COUNT="`echo \"$SHLIBS\" | wc -l`"
+
+ldd $SHLIBS | grep -Po '(?<=> )/[^ ]+' | sort | uniq | grep -Pv
'lib(c|gcc|dl|m|util|rt|pthread|stdc\+\+|selinux|krb5|gssapi_krb5)\.so'
| xargs -I {} install  {} $ROOTFS/usr/lib


I'm not sure why Kerberos cannot be used on OSv, but never mind, 
everything should also work without it.

Maybe it conflicts with krb-something libs from external/ ?


+# ROOTFS/lib/python2.7/config/libpython2.7.so
<http://libpython2.7.so> is a symlink to ../../libpython2.7.so
<http://libpython2.7.so>,
+# so create a valid destination to avoid ldd error due to
dangling symlink.
+(cd $ROOTFS/lib && ln -sf ../usr/lib/libpython$VERSION.so.1.0
libpython$VERSION.so)
+echo "$SHLIBS_COUNT"
+}
+
+main() {
+mkdir -p build/
+gcc -o build/python.so python.c -fPIC -shared -lpython${VERSION}
+
+rm -rf "$ROOTFS"
+mkdir -p "$ROOTFS/usr/lib"
+mkdir -p "$ROOTFS/lib/python$VERSION"
+
+cp build/python.so "$ROOTFS"
+install_shlibs
+# TODO /lib64/python2.7/ should not be hardcoded?
+PY_LIB1=/lib64/python$VERSION/
+rsync -a $PY_LIB1 $ROOTFS/lib/python

Re: [PATCH] libc: add functions required for libpython2.7.so

[Justin Cinkelj]

On 08/01/2017 02:23 PM, Nadav Har'El wrote:


On Tue, Aug 1, 2017 at 2:16 PM, Justin Cinkelj <justin.cink...@xlab.si 
<mailto:justin.cink...@xlab.si>> wrote:


When we try to reuse python from the build system (instead of
compiling it from source), libpython2.7.so
<http://libpython2.7.so> requires additional
libc/glibc symbols. The patch was prepared on Fedora 25.

One problem with musl/src/linux/wait4.c remains. It should include
#include 
just before
#include "syscall.h"
to actually compile.


So your patch should instead copy wait4.c to libc/, and use that 
instead of directly musl?

But did you take a look at the content of that wait4.c?
All it does is to call a system call, which we never implemented.
Did you see a message about it not being implemented?
No, I only saw message from ELF dynamic linker about "failed looking up 
symbol ".


For waitpid() we did something different - see libc/process/waitpid.cc 
- we just returned ECHILD. Maybe we should do the same here?

That way, waitpid returns error, and nothing is shown on console.
With musl wait3 and wait4 implementation, syscall() from linux.cc would 
show "unimplemented system call" message, which makes debugging easier 
in case it ever gets called.

I added implementation to runtime.cc with WARN_STUBBED() and ECHILD.




Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si
<mailto:justin.cink...@xlab.si>>
---
 Makefile   | 15 +++
 runtime.cc |  6 ++
 2 files changed, 21 insertions(+)

diff --git a/Makefile b/Makefile
index 329b749..637281f 100644
--- a/Makefile
+++ b/Makefile
@@ -1327,6 +1327,15 @@ libc += misc/lockf.o
 libc += misc/mntent.o
 musl += misc/nftw.o
 libc += misc/__longjmp_chk.o
+musl += misc/getresuid.o
+musl += misc/getresgid.o


These two also call a system call, which will fail. Are they used too, 
and you see the errors?
Again, I saw only "failed looking up symbol" message. Replaced that with 
WARN_STUBBED and ENOSYS in runtime.cc
https://github.com/termux/termux-packages/issues/901 - it mentions that 
ENOSYS is set, so I used that code.


+musl += misc/openpty.o

This implementation will never actually work on OSv... It will try 
some open, and fail... So shouldn't we just have a stub function that 
fails immediately?
Again, I'm surprised Python actually uses that when you run a simple 
application


+musl += misc/forkpty.o


Since this calls fork(), it is really silly to include this 
implementation... Again, we could stub it, but I am guessing it is 
*NOT* called in Python, so I wonder why you needed it.

Replaced openpty and forkpty with WARN_STUBBED and ENOENT in runtime.cc.


+
+musl += linux/wait3.o
+musl += linux/wait4.o


See comment on these above.

+musl += signal/killpg.o


This also makes no sense on OSv (no processes, so no process 
groups...) but I guess the implmenetation is trivial enough to include 
it anyway?

OK, I leave it as is.


+musl += signal/siginterrupt.o


This one is fine (although has less of an influence in OSv)


 musl += multibyte/btowc.o
 musl += multibyte/internal.o
@@ -1396,6 +1405,7 @@ libc += process/execle.o
 musl += process/execv.o
 musl += process/execl.o
 libc += process/waitpid.o
+musl += process/wait.o


Ok.


 libc += arch/$(arch)/setjmp/setjmp.o
 libc += arch/$(arch)/setjmp/longjmp.o
@@ -1677,6 +1687,11 @@ libc += unistd/getsid.o
 libc += unistd/setsid.o
 libc += unistd/ttyname_r.o
 libc += unistd/ttyname.o
+musl += unistd/tcgetpgrp.o


This one is fine.

+musl += unistd/nice.o


I don't like the #ifdef SYS_nice case.

Can you move this code, without the SYS_nice case, into runtime.cc, 
where setpriority() is already defined?

OK.


+musl += unistd/tcsetpgrp.o


Fine.

+musl += unistd/setpgrp.o


Fine.

+musl += unistd/ctermid.o


I think this won't work as-is because we don't have a file "/dev/tty". 
If it's important, we can add it.

The underlying ttyname_r does work.
I guess current implementation (which will not really work) is good 
enough, for now.

I moved code to runtime.cc and add WARN_STUBBED, in case it ever get called.
Returned string will be always "", which is same as musl code would do.



 musl += regex/fnmatch.o
 musl += regex/glob.o
diff --git a/runtime.cc b/runtime.cc
index c81adea..c60a729 100644
--- a/runtime.cc
+++ b/runtime.cc
@@ -554,3 +554,9 @@ int sysctl(int *, int, void *, size_t *, void
*, size_t)
 errno = ENOTDIR;
 return -1;
 }
+
+extern "C"
+char *tmpnam_r(char *s)
+{
+return tmpnam(s);


The canonical implementation (see the manual page) is

 return s ? tmpnam(s) : NULL;

(your implementation is also fine in the "normal" case but doesn'

[PATCH v2] libc: add functions required for libpython2.7.so

[Justin Cinkelj]

When we try to reuse python from the build system (instead of
compiling it from source), libpython2.7.so requires additional
libc/glibc symbols. It seems that (at least for most trivial python
programs) those functions are not really called and can be implemented
as stubs. The patch was prepared on Fedora 25.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 Makefile   |  7 +++
 runtime.cc | 70 ++
 2 files changed, 77 insertions(+)

diff --git a/Makefile b/Makefile
index 329b749..d09b3ed 100644
--- a/Makefile
+++ b/Makefile
@@ -1328,6 +1328,9 @@ libc += misc/mntent.o
 musl += misc/nftw.o
 libc += misc/__longjmp_chk.o
 
+musl += signal/killpg.o
+musl += signal/siginterrupt.o
+
 musl += multibyte/btowc.o
 musl += multibyte/internal.o
 musl += multibyte/mblen.o
@@ -1396,6 +1399,7 @@ libc += process/execle.o
 musl += process/execv.o
 musl += process/execl.o
 libc += process/waitpid.o
+musl += process/wait.o
 
 libc += arch/$(arch)/setjmp/setjmp.o
 libc += arch/$(arch)/setjmp/longjmp.o
@@ -1677,6 +1681,9 @@ libc += unistd/getsid.o
 libc += unistd/setsid.o
 libc += unistd/ttyname_r.o
 libc += unistd/ttyname.o
+musl += unistd/tcgetpgrp.o
+musl += unistd/tcsetpgrp.o
+musl += unistd/setpgrp.o
 
 musl += regex/fnmatch.o
 musl += regex/glob.o
diff --git a/runtime.cc b/runtime.cc
index c81adea..04d38e7 100644
--- a/runtime.cc
+++ b/runtime.cc
@@ -554,3 +554,73 @@ int sysctl(int *, int, void *, size_t *, void *, size_t)
 errno = ENOTDIR;
 return -1;
 }
+
+extern "C"
+char *tmpnam_r(char *s)
+{
+return s ? tmpnam(s) : NULL;
+}
+
+extern "C"
+pid_t wait3(int *status, int options, struct rusage *usage)
+{
+WARN_STUBBED();
+errno = ECHILD;
+return -1;
+}
+
+extern "C"
+pid_t wait4(pid_t pid, int *status, int options, struct rusage *usage)
+{
+WARN_STUBBED();
+errno = ECHILD;
+return -1;
+}
+
+int getresuid(uid_t *ruid, uid_t *euid, uid_t *suid)
+{
+WARN_STUBBED();
+errno = ENOSYS;
+return -1;
+}
+
+int getresgid(gid_t *rgid, gid_t *egid, gid_t *sgid)
+{
+WARN_STUBBED();
+errno = ENOSYS;
+return -1;
+}
+
+extern "C"
+int openpty(int *amaster, int *aslave, char *name,
+   const struct termios *termp,
+   const struct winsize *winp)
+{
+WARN_STUBBED();
+errno = ENOENT;
+return -1;
+}
+
+extern "C"
+pid_t forkpty(int *amaster, char *name,
+ const struct termios *termp,
+ const struct winsize *winp)
+{
+WARN_STUBBED();
+errno = ENOENT;
+return -1;
+}
+
+int nice(int inc)
+{
+return setpriority(PRIO_PROCESS, 0, getpriority(PRIO_PROCESS, 0)+inc);
+}
+
+char *ctermid(char *s)
+{
+static char s2[L_ctermid];
+WARN_STUBBED();
+if (!s) s = s2;
+*s = 0;
+return s;
+}
-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] libc: add functions required for libpython2.7.so

[Justin Cinkelj]

I should said first that I didn't really try to verify if 
implementations are correct, and that I guess most of those functions 
are not called.
More or less, I just wanted to avoid "failed looking up symbol xyz" 
error. I didn't even open any of the .c files (except wait4.c, because 
it didn't compile).

Will send v2 patch based on you comments.
Thank you, Justin

On 08/01/2017 02:23 PM, Nadav Har'El wrote:


On Tue, Aug 1, 2017 at 2:16 PM, Justin Cinkelj <justin.cink...@xlab.si 
<mailto:justin.cink...@xlab.si>> wrote:


When we try to reuse python from the build system (instead of
compiling it from source), libpython2.7.so
<http://libpython2.7.so> requires additional
libc/glibc symbols. The patch was prepared on Fedora 25.

One problem with musl/src/linux/wait4.c remains. It should include
#include 
just before
#include "syscall.h"
to actually compile.


So your patch should instead copy wait4.c to libc/, and use that 
instead of directly musl?

But did you take a look at the content of that wait4.c?
All it does is to call a system call, which we never implemented.
Did you see a message about it not being implemented?

For waitpid() we did something different - see libc/process/waitpid.cc 
- we just returned ECHILD. Maybe we should do the same here?



Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si
<mailto:justin.cink...@xlab.si>>
---
 Makefile   | 15 +++
 runtime.cc |  6 ++
 2 files changed, 21 insertions(+)

diff --git a/Makefile b/Makefile
index 329b749..637281f 100644
--- a/Makefile
+++ b/Makefile
@@ -1327,6 +1327,15 @@ libc += misc/lockf.o
 libc += misc/mntent.o
 musl += misc/nftw.o
 libc += misc/__longjmp_chk.o
+musl += misc/getresuid.o
+musl += misc/getresgid.o


These two also call a system call, which will fail. Are they used too, 
and you see the errors?


+musl += misc/openpty.o

This implementation will never actually work on OSv... It will try 
some open, and fail... So shouldn't we just have a stub function that 
fails immediately?
Again, I'm surprised Python actually uses that when you run a simple 
application


+musl += misc/forkpty.o


Since this calls fork(), it is really silly to include this 
implementation... Again, we could stub it, but I am guessing it is 
*NOT* called in Python, so I wonder why you needed it.


+
+musl += linux/wait3.o
+musl += linux/wait4.o


See comment on these above.

+musl += signal/killpg.o


This also makes no sense on OSv (no processes, so no process 
groups...) but I guess the implmenetation is trivial enough to include 
it anyway?


+musl += signal/siginterrupt.o


This one is fine (although has less of an influence in OSv)


 musl += multibyte/btowc.o
 musl += multibyte/internal.o
@@ -1396,6 +1405,7 @@ libc += process/execle.o
 musl += process/execv.o
 musl += process/execl.o
 libc += process/waitpid.o
+musl += process/wait.o


Ok.


 libc += arch/$(arch)/setjmp/setjmp.o
 libc += arch/$(arch)/setjmp/longjmp.o
@@ -1677,6 +1687,11 @@ libc += unistd/getsid.o
 libc += unistd/setsid.o
 libc += unistd/ttyname_r.o
 libc += unistd/ttyname.o
+musl += unistd/tcgetpgrp.o


This one is fine.

+musl += unistd/nice.o


I don't like the #ifdef SYS_nice case.

Can you move this code, without the SYS_nice case, into runtime.cc, 
where setpriority() is already defined?


+musl += unistd/tcsetpgrp.o


Fine.

+musl += unistd/setpgrp.o


Fine.

+musl += unistd/ctermid.o


I think this won't work as-is because we don't have a file "/dev/tty". 
If it's important, we can add it.

The underlying ttyname_r does work.


 musl += regex/fnmatch.o
 musl += regex/glob.o
diff --git a/runtime.cc b/runtime.cc
index c81adea..c60a729 100644
--- a/runtime.cc
+++ b/runtime.cc
@@ -554,3 +554,9 @@ int sysctl(int *, int, void *, size_t *, void
*, size_t)
 errno = ENOTDIR;
 return -1;
 }
+
+extern "C"
+char *tmpnam_r(char *s)
+{
+return tmpnam(s);


The canonical implementation (see the manual page) is

 return s ? tmpnam(s) : NULL;

(your implementation is also fine in the "normal" case but doesn't 
"prevent" people from passing null as s and becoming non-reentrant).



+}
--
2.9.4

--
You received this message because you are subscribed to the Google
Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to osv-dev+unsubscr...@googlegroups.com
<mailto:osv-dev%2bunsubscr...@googlegroups.com>.
For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.




--
You received this message because you are subscribed to the Google Groups "

[RFC PATCH 1/2] Revert "Add python2.7"

[Justin Cinkelj]

This reverts commit 9cdcf94d7d14d117bcbc790ff52287f3fb5ecf0d.
---
 python27/Makefile  | 52 --
 python27/copy-required-system-libraries.sh | 27 
 2 files changed, 79 deletions(-)
 delete mode 100644 python27/Makefile
 delete mode 100755 python27/copy-required-system-libraries.sh

diff --git a/python27/Makefile b/python27/Makefile
deleted file mode 100644
index 7e1f2da..000
--- a/python27/Makefile
+++ /dev/null
@@ -1,52 +0,0 @@
-VERSION=2.7
-SOURCE=https://github.com/python/cpython.git
-
-PYTHON_PREFIX=$(PWD)/src/python-install
-
-
-.PHONY: module clean ROOTFS
-
-SRC=src/cpython
-
-module: usr.manifest
-
-usr.manifest: ROOTFS
-   echo '/**: $${MODULE_DIR}/ROOTFS/**' > usr.manifest
-
-clean:
-   rm -fr usr.manifest ROOTFS src
-
-# Note: the touch commands below are needed after commands which create files
-# "in the past", like wget and tar, and can confuse Make to rebuild a target
-# which is new, just pretends to be old.
-$(SRC)/.git/HEAD:
-   mkdir src
-   cd src; git clone $(SOURCE)
-   cd $(SRC); git checkout -b $(VERSION) origin/$(VERSION)
-   #touch $@
-
-$(SRC)/Makefile: $(SRC)/.git/HEAD
-   cd $(SRC); ./configure --enable-shared --prefix=$(PYTHON_PREFIX)
-   cd $(SRC); cp Makefile Makefile.orig;
-   cd $(SRC); sed -i 's/^LDFLAGS=.*/LDFLAGS=-shared/g' Makefile
-
-$(SRC)/python: $(SRC)/Makefile
-   cd $(SRC); $(MAKE) python
-   cd $(SRC); mv python python.so
-   # Re-link python, this time as ELF executable.
-   # This is required for make install to succeed.
-   cd $(SRC); $(MAKE) -f Makefile.orig install
-
-ROOTFS: $(SRC)/python
-   rm ROOTFS -rf
-   mkdir -p ROOTFS/usr/lib
-
-   cp $(SRC)/python.so ROOTFS/python
-   cp $(PYTHON_PREFIX)/lib/libpython$(VERSION).so.1.0 ROOTFS/usr/lib
-   # Copy the Python environment. Since we'd like to exclude some dirs, 
rsync is used
-   rsync -a $(PYTHON_PREFIX)/lib/python$(VERSION) ROOTFS/lib --exclude 
test --exclude unittest \
-   --exclude '*.pyc' --exclude '*.pyo' --exclude '*.egg-info' 
--exclude '*.o' --exclude 'libpython'$(VERSION)'.a'
-   # TODO maybe .so files in ROOTFS/lib/python2.7/lib-dynload should be 
stripped.
-   ./copy-required-system-libraries.sh
-
-.DELETE_ON_ERROR:
diff --git a/python27/copy-required-system-libraries.sh 
b/python27/copy-required-system-libraries.sh
deleted file mode 100755
index 81a585b..000
--- a/python27/copy-required-system-libraries.sh
+++ /dev/null
@@ -1,27 +0,0 @@
-#!/bin/bash
-
-# Copy required libraries from the build system.
-# Likely, we don't want to include all libs (and their dependencies) from
-# ldd ROOTFS/lib/python2.7/lib-dynload/*so | grep -v -e '^ROOTFS/' | sed 
's/(0x.*)//' | sort | uniq | awk '{print $3}'
-# So for now, only two (most obviously required) dependency libs will be 
included.
-
-set -e
-
-PYLIBS=""
-PYLIBS+=" ROOTFS/lib/python2.7/lib-dynload/readline.so "
-PYLIBS+=" ROOTFS/lib/python2.7/lib-dynload/_curses.so "
-
-DEPLIBS=""
-for pylib in $PYLIBS
-do
-   DEPLIBS+=`ldd $pylib | egrep -v -e linux-vdso.so -e ld-linux-x86-64.so 
-e 'lib(python[23]\.[0-9]|pthread|c|dl|m|util|).so.*'`
-   DEPLIBS+="\n"
-done
-#echo -e "DEPLIBS1:\n$DEPLIBS"
-DEPLIBS=`echo -e "$DEPLIBS" | sed '/^$/d' | awk '{print $3}' | sort | uniq`
-#echo -e "DEPLIBS2:\n$DEPLIBS"
-
-for deplib in $DEPLIBS
-do
-   /bin/cp $deplib ROOTFS/usr/lib
-done
-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] Add python2.7

[Justin Cinkelj]

I didn't want to include every it-might-be-actually-required library from 
the build system. Complete list from my system is:
justin_cinkelj@jcpc:~/devel/mikelangelo/osv-fc25/osv/apps/python27$ ldd 
ROOTFS/lib/python2.7/lib-dynload/*so | grep -v -e '^ROOTFS/' | sed 
's/(0x.*)//' | sort | uniq | awk '{print $3}' 

/lib64/libbz2.so.1
/lib64/libcom_err.so.2
/lib64/libcrypto.so.10
/lib64/libcrypt.so.1
/lib64/libc.so.6
/usr/lib64/libc.so.6
/usr/lib64/libdb-5.3.so
/lib64/libdl.so.2
/usr/lib64/libdl.so.2
/lib64/libfreebl3.so
/lib64/libgdbm_compat.so.4
/lib64/libgdbm.so.4
/lib64/libgssapi_krb5.so.2
/lib64/libk5crypto.so.3
/lib64/libkeyutils.so.1
/lib64/libkrb5.so.3
/lib64/libkrb5support.so.0
/lib64/libm.so.6
/usr/lib64/libm.so.6
/lib64/libncursesw.so.6
/lib64/libnsl.so.1
/lib64/libpanelw.so.6
/lib64/libpcre.so.1
/lib64/libpthread.so.0
/usr/lib64/libpthread.so.0
/lib64/libpython2.7.so.1.0
/usr/lib64/libpython2.7.so.1.0
/lib64/libreadline.so.6
/lib64/libresolv.so.2
/lib64/libselinux.so.1
/lib64/libssl.so.10
/lib64/libtinfo.so.6
/lib64/libutil.so.1
/usr/lib64/libutil.so.1
/lib64/libz.so.1

I'm not sure if there is anything absolutely required.

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Porting Aerospike server to OSv: /asd.so: failed looking up symbol je_chunksize_mask

[Justin Cinkelj]

Also, I see there is /libluajit.so in usr.manifest - you might have problem 
if cli.so is also running. In 
https://github.com/cloudius-systems/osv/issues/882 it helped running the 
application (wrk benchmark) in a new ELF namespace.

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH v2 1/2] libc: add tcflow

[Justin Cinkelj]

At least a symbol needs to be defined for python2.7.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 Makefile | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/Makefile b/Makefile
index 850e1a9..c376b74 100644
--- a/Makefile
+++ b/Makefile
@@ -1663,6 +1663,8 @@ libc += time/wcsftime.o
 libc += time/ftime.o # verbatim copy of the file as in 4b15d9f46a2b@musl
 $(out)/libc/time/ftime.o: CFLAGS += -Ilibc/include
 
+musl += termios/tcflow.o
+
 musl += unistd/sleep.o
 musl += unistd/gethostname.o
 libc += unistd/sethostname.o
-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH v2 2/2] libc: add __mbrlen

[Justin Cinkelj]

At least a symbol needs to be defined for python2.7.
File mbrlen.c is moved from musl/ to libc/, as the weak alias declaration
must be in same file as function definition.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 Makefile|  2 +-
 libc/multibyte/mbrlen.c | 20 
 2 files changed, 21 insertions(+), 1 deletion(-)
 create mode 100644 libc/multibyte/mbrlen.c

diff --git a/Makefile b/Makefile
index c376b74..5acdc7d 100644
--- a/Makefile
+++ b/Makefile
@@ -1331,7 +1331,7 @@ libc += misc/__longjmp_chk.o
 musl += multibyte/btowc.o
 musl += multibyte/internal.o
 musl += multibyte/mblen.o
-musl += multibyte/mbrlen.o
+libc += multibyte/mbrlen.o
 musl += multibyte/mbrtowc.o
 musl += multibyte/mbsinit.o
 musl += multibyte/mbsnrtowcs.o
diff --git a/libc/multibyte/mbrlen.c b/libc/multibyte/mbrlen.c
new file mode 100644
index 000..827f1da
--- /dev/null
+++ b/libc/multibyte/mbrlen.c
@@ -0,0 +1,20 @@
+/* 
+ * This code was written by Rich Felker in 2010; no copyright is claimed.
+ * This code is in the public domain. Attribution is appreciated but
+ * unnecessary.
+ */
+
+#include 
+#include 
+#include 
+#include 
+
+//#include "internal.h"
+
+size_t mbrlen(const char *restrict s, size_t n, mbstate_t *restrict st)
+{
+   static unsigned internal;
+   return mbrtowc(0, s, n, st ? st : (mbstate_t *));
+}
+
+size_t __mbrlen (const char *, size_t, mbstate_t *) __attribute__ ((weak, 
alias ("mbrlen")));
-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH v2 0/2] python2.7 support

[Justin Cinkelj]

Gregor Berginc started working on python2.7 for OSv.
I only continue with his work. He occasinally gets
"failed looking up symbol __mbrlen", but never got missing tcflow error.
He compiled code on Ubuntu 16.04.

I compiled python2.7 on FC25, and cannot even start interpreter as:
./scripts/run.py -e "--env=PYTHONHOME=/pyenv /usr/lib/python"
without both tcflow and __mbrlen. Hence the two tiny patches.

The __mbrlen is not used if command to execute is given on cmdline:
./scripts/run.py -e "--env=PYTHONHOME=/pyenv /usr/lib/python -c 
\"aa={1:22,3:44}; print aa; print 'asdf'\""

Justin Cinkelj (2):
  libc: add tcflow
  libc: add __mbrlen

 Makefile|  4 +++-
 libc/multibyte/mbrlen.c | 20 
 2 files changed, 23 insertions(+), 1 deletion(-)
 create mode 100644 libc/multibyte/mbrlen.c

-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH 2/2] libc: add __mbrlen

[Justin Cinkelj]

Ups ups ups, my bad.

I was moving this from file in ./osv/musl/ (submodule) into main ./osv, 
and tested with:
./scripts/run.py -e "--env=PYTHONHOME=/pyenv /usr/lib/python -c 
\"aa={1:22,3:44}; print aa; print 'asdf'\""

And that one works! Only interactive interpreter example fails:
./scripts/run.py -e "--env=PYTHONHOME=/pyenv /usr/lib/python"

My problem was (still is), that line
weak_alias(mbrlen, __mbrlen);
in musl/src/multibyte/mbrlen.c works, but in libc/libc.cc I get error
libc/libc.cc:253:20: error: ‘size_t __mbrlen(const char*, size_t, 
mbstate_t*)’ aliased to undefined symbol ‘mbrlen’

Annoying.


On 07/20/2017 04:34 PM, Nadav Har'El wrote:
But the patch leaves the line commented out - I don't suppose this can 
help :-)


On Jul 20, 2017 15:08, "Justin Cinkelj" <justin.cink...@xlab.si 
<mailto:justin.cink...@xlab.si>> wrote:


At least a symbol needs to be defined for python2.7.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si
<mailto:justin.cink...@xlab.si>>
---
 libc/libc.cc | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/libc/libc.cc b/libc/libc.cc
index a9d4c48..a166559 100644
--- a/libc/libc.cc
+++ b/libc/libc.cc
@@ -248,3 +248,5 @@ extern "C" {
 return "OSv";
 }
 }
+
+// weak_alias(mbrlen, __mbrlen);
--
2.9.4

--
You received this message because you are subscribed to the Google
Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to osv-dev+unsubscr...@googlegroups.com
<mailto:osv-dev%2bunsubscr...@googlegroups.com>.
For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.




--
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH 0/2] python2.7 support

[Justin Cinkelj]

Gregor Berginc started working on python2.7 for OSv.
I only continue with his work. He occasinally gets
"failed looking up symbol __mbrlen", but never got missing tcflow error.
He compiled code on Ubuntu 16.04.

I compiled python2.7 on FC25, and cannot even start interpreter
without both tcflow and __mbrlen. Hence the two tiny patches.

Justin Cinkelj (2):
  libc: add tcflow
  libc: add __mbrlen

 Makefile | 2 ++
 libc/libc.cc | 2 ++
 2 files changed, 4 insertions(+)

-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH 1/2] libc: add tcflow

[Justin Cinkelj]

At least a symbol needs to be defined for python2.7.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 Makefile | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/Makefile b/Makefile
index 850e1a9..c376b74 100644
--- a/Makefile
+++ b/Makefile
@@ -1663,6 +1663,8 @@ libc += time/wcsftime.o
 libc += time/ftime.o # verbatim copy of the file as in 4b15d9f46a2b@musl
 $(out)/libc/time/ftime.o: CFLAGS += -Ilibc/include
 
+musl += termios/tcflow.o
+
 musl += unistd/sleep.o
 musl += unistd/gethostname.o
 libc += unistd/sethostname.o
-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH 2/2] libc: add __mbrlen

[Justin Cinkelj]

At least a symbol needs to be defined for python2.7.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 libc/libc.cc | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/libc/libc.cc b/libc/libc.cc
index a9d4c48..a166559 100644
--- a/libc/libc.cc
+++ b/libc/libc.cc
@@ -248,3 +248,5 @@ extern "C" {
 return "OSv";
 }
 }
+
+// weak_alias(mbrlen, __mbrlen);
-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [COMMIT osv master] af_local: stub ioctl FIOASYNC

[Justin Cinkelj]

I will try to use that one. If multithreaded nginx requires it, than 
this will be also an additional test for it :)


On 07/19/2017 03:08 PM, Nadav Har'El wrote:
By the way, I just noticed that we have this full implementation of 
FIOASYNC from myechuri:


https://github.com/cloudius-systems/osv/issues/753
https://github.com/myechuri/osv/commit/df0c07a842a55baa1f6de1af155c7b69873c9eb6

It would probably be a good idea to take that instead of the stub 
(even though it's not really needed for nginx. but maybe for the 
multi-threaded one it is needed?)




--
Nadav Har'El
n...@scylladb.com <mailto:n...@scylladb.com>

On Tue, Jul 18, 2017 at 5:36 PM, Commit Bot <b...@cloudius-systems.com 
<mailto:b...@cloudius-systems.com>> wrote:


From: Justin Cinkelj <justin.cink...@xlab.si
<mailto:justin.cink...@xlab.si>>
Committer: Nadav Har'El <n...@scylladb.com <mailto:n...@scylladb.com>>
Branch: master

af_local: stub ioctl FIOASYNC

nginx doesn't start if ioctl with cmd=FIOASYNC fails. Stub it, so that
nginx can start unpatched.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si
<mailto:justin.cink...@xlab.si>>
Message-Id: <20170718091623.23284-2-justin.cink...@xlab.si
<mailto:20170718091623.23284-2-justin.cink...@xlab.si>>

---
diff --git a/libc/af_local.cc b/libc/af_local.cc

--- a/libc/af_local.cc
+++ b/libc/af_local.cc
@@ -19,6 +19,8 @@
 #include 
 #include 

+#include 
+
 using namespace std;

 struct af_local final : public special_file {
@@ -42,11 +44,17 @@ int af_local::ioctl(u_long cmd, void *data)
 int error = ENOTTY;
 switch (cmd) {
 case FIONBIO:
-SCOPE_LOCK(f_lock);
-if (*(int *)data)
-f_flags |= FNONBLOCK;
-else
-f_flags &= ~FNONBLOCK;
+{
+SCOPE_LOCK(f_lock);
+if (*(int *)data)
+f_flags |= FNONBLOCK;
+else
+f_flags &= ~FNONBLOCK;
+}
+error = 0;
+break;
+case FIOASYNC:
+WARN_ONCE("af_local::ioctl(FIOASYNC) stubbed\n");
 error = 0;
 break;
 }

-- 
You received this message because you are subscribed to the Google

Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to osv-dev+unsubscr...@googlegroups.com
<mailto:osv-dev%2bunsubscr...@googlegroups.com>.
For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.




--
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH v2] Add nginx

[Justin Cinkelj]

I also tried with/without cli, debug/release build, and did clean it before 
rebuild
./scripts/build -j4 mode=release clean
./scripts/build -j4 mode=release image=nginx

And it still works 
justin_cinkelj@jcpc:~/devel/mikelangelo/osv-fc25/osv$ scripts/run.py 
--forward tcp:::80
qemu-system-x86_64: -redir tcp:::80: The -redir option is deprecated. 
Please use '-netdev user,hostfwd=...' instead.
OSv v0.24-403-gb88aa48
eth0: 192.168.122.15
WARNING: af_local::ioctl(FIOASYNC) stubbed
WARNING: fcntl(F_SETOWN) stubbed
initgroups() stubbed

justin_cinkelj@jcpc:~/devel/mikelangelo/osv-fc25/osv/apps$ curl 
http://localhost:/index.html



Welcome to nginx!

Nadav, how do you build image? I guess on FC26, and you somehow tell 
./scripts/build to use specific gcc version? I doubt that is relevant, but 
I just cant reproduce.

Also, can you try to send nginx logs to stderr - the ngnix.conf already has 
the required lines, just for that purpose :). But, they are commented out. 
I guess something in worker code fails, then worker doesn't really start 
up, its code returns, and master code gets executed.

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH v2] Add nginx

[Justin Cinkelj]

But... I tried this before, and it did work. Scratching my head now.

What works for me. Building on FC25
justin_cinkelj@jcpc:~/devel/mikelangelo/osv-fc25/osv$ git log 
--pretty=oneline
b88aa485e8beac18cb51f28dcbb710ed5bdcdab8 af_local: stub ioctl FIOASYNC
62839c888be763a3afb1051234ae9a6fef04e654 signals: provide sigsuspend symbol
  so , current master, git diff empty, except for nginx patch in apps/

justin_cinkelj@jcpc:~/devel/mikelangelo/osv-fc25/osv/apps$ git diff
justin_cinkelj@jcpc:~/devel/mikelangelo/osv-fc25/osv/apps$ git log 
--pretty=oneline 
c6b119009555346cc8bcdaeb3c3539e530b3d74d Add nginx
945ebfd812bf0d2d430f2e2ab606ae1bc1e7bfdc netperf: update download link
b36e0d61f16a1b434ab02555443abc746ac92998 Add lighttpd application

justin_cinkelj@jcpc:~/devel/mikelangelo/osv-fc25/osv$ (cd apps/nginx; make 
clean)
justin_cinkelj@jcpc:~/devel/mikelangelo/osv-fc25/osv$ ./scripts/build -j4 
mode=debug image=cli,nginx
justin_cinkelj@jcpc:~/devel/mikelangelo/osv-fc25/osv$ sudo ./scripts/run.py 
--novnc -nv -c4 -Vd
...
[I/244 dhcp]: Configuring eth0: ip 192.168.122.90 subnet mask 255.255.255.0 
gateway 192.168.122.1 MTU 1500
[I/244 dhcp]: Set hostname to: osv
Running from /init/30-auto-00: /libhttpserver.so &!
could not load libicudata.so.57
could not load libicui18n.so.57
could not load libicuuc.so.57
could not load libcrypt.so.1
WARNING: af_local::ioctl(FIOASYNC) stubbed
WARNING: fcntl(F_SETOWN) stubbed
initgroups() stubbed
/# 


justin_cinkelj@jcpc:~/devel/mikelangelo/osv-fc25/osv/apps$ curl 
http://192.168.122.90:80/index.html



Welcome to nginx!
...

Only what is different on your machine? No cli in image, compiler, 
libraries? I didn't try clean OSv checkout yet.

Justin

On Tuesday, July 18, 2017 at 2:53:47 PM UTC+2, Nadav Har'El wrote:
>
> Hi,
>
> I tried this, and unfortunately it seems like is *is* calling sigsuspend, 
> so the implemenation using IMPL() (and aborting) is not good enough:
>
> /home/nyh/osv.tmp2/osv$ scripts/run.py --forward tcp:::80
> 
> WARNING: unimplemented int sigsuspend(const sigset_t *mask)
> Aborted
>
> Backtrace in gdb:
>
> #12 0x002258c0 in abort () at runtime.cc:96
> #13 0x0044921c in sigsuspend (mask=mask@entry=0x201ffa00)
> at libc/signal.cc:216
> #14 0x10c430ec in ngx_master_process_cycle (
> cycle=cycle@entry=0x83152090)
> at src/os/unix/ngx_process_cycle.c:163
> #15 0x10c18733 in main (argc=, argv=)
> at src/core/nginx.c:375
>
>
>
> --
> Nadav Har'El
> n...@scylladb.com 
>
> On Tue, Jul 18, 2017 at 1:16 PM, Justin Cinkelj <justin@xlab.si 
> > wrote:
>
>> Based on @myechuri work,
>> https://github.com/myechuri/osv-apps/commits/nginx-osv
>>
>> A single nginx worker thread is run, so only one CPU core is utilized.
>>
>> Example configuration includes very long keepalive_timeout/requests
>> setting to avoid possible problems with OSV and TCP client port reuse.
>> With example configuration will server offer two URLs:
>>  http://$IP/index.html and
>>  http://$IP/basic_status
>>
>> Signed-off-by: Justin Cinkelj <justin@xlab.si >
>> ---
>>  nginx/.gitignore   |   2 +
>>  nginx/Makefile |  45 +++
>>  nginx/module.py|   3 +
>>  .../0001-nginx-OSv-fix-process-spawning.patch  |  33 +
>>  nginx/patches/nginx.conf   | 141 
>> +
>>  5 files changed, 224 insertions(+)
>>  create mode 100644 nginx/.gitignore
>>  create mode 100644 nginx/Makefile
>>  create mode 100644 nginx/module.py
>>  create mode 100644 
>> nginx/patches/0001-nginx-OSv-fix-process-spawning.patch
>>  create mode 100644 nginx/patches/nginx.conf
>>
>> diff --git a/nginx/.gitignore b/nginx/.gitignore
>> new file mode 100644
>> index 000..cb8005e
>> --- /dev/null
>> +++ b/nginx/.gitignore
>> @@ -0,0 +1,2 @@
>> +usr.manifest
>> +upstream.tar
>> diff --git a/nginx/Makefile b/nginx/Makefile
>> new file mode 100644
>> index 000..b826c0d
>> --- /dev/null
>> +++ b/nginx/Makefile
>> @@ -0,0 +1,45 @@
>> +VERSION=1.12.1
>> +SOURCE=http://nginx.org/download/nginx-${VERSION}.tar.gz
>> +CONFIGURE_MODULES=--prefix=/nginx/ --with-debug 
>> --without-http_rewrite_module --with-threads --with-http_stub_status_module
>> +
>> +.PHONY: module clean
>> +
>> +SRC=upstream/nginx
>> +
>> +module: usr.manifest
>> +
>> +usr.manifest: $(SRC)/nginx.so
>> +   echo '[manifest]' > usr.manifest
>> +   echo '/nginx.so: $${MODULE_DIR}/$(SRC)/nginx.so' &g

[PATCH v2] signals: provide sigsuspend symbol

[Justin Cinkelj]

sigsuspend is used in nginx. With current (not yet commited) patch, it
is never really called. Nginx master process code is used to execute
worker code, and this prevents master code to actually call sigsuspend.
So UNIMPL implementation is sufficient for now.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 libc/signal.cc | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/libc/signal.cc b/libc/signal.cc
index 832002c..1992ff8 100644
--- a/libc/signal.cc
+++ b/libc/signal.cc
@@ -213,6 +213,8 @@ int sigprocmask(int how, const sigset_t* _set, sigset_t* 
_oldset)
 return 0;
 }
 
+UNIMPL(int sigsuspend(const sigset_t *mask));
+
 int sigaction(int signum, const struct sigaction* act, struct sigaction* 
oldact)
 {
 // FIXME: We do not support any sa_flags besides SA_SIGINFO.
-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH 1/2] af_local: stub ioctl FIOASYNC

[Justin Cinkelj]

On 07/18/2017 01:18 PM, Nadav Har'El wrote:

Thanks. One question below:

On Tue, Jul 18, 2017 at 12:16 PM, Justin Cinkelj 
<justin.cink...@xlab.si <mailto:justin.cink...@xlab.si>> wrote:


nginx doesn't start if ioctl with cmd=FIOASYNC fails. Stub it, so that
nginx can start unpatched.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si
<mailto:justin.cink...@xlab.si>>
---
 libc/af_local.cc | 18 +-
 1 file changed, 13 insertions(+), 5 deletions(-)

diff --git a/libc/af_local.cc b/libc/af_local.cc
index b68c2b0..e65e2b1 100644
--- a/libc/af_local.cc
+++ b/libc/af_local.cc
@@ -19,6 +19,8 @@
 #include 
 #include 

+#include 
+
 using namespace std;

 struct af_local final : public special_file {
@@ -42,11 +44,17 @@ int af_local::ioctl(u_long cmd, void *data)
 int error = ENOTTY;
 switch (cmd) {
 case FIONBIO:
-SCOPE_LOCK(f_lock);
-if (*(int *)data)
-f_flags |= FNONBLOCK;
-else
-f_flags &= ~FNONBLOCK;
+{
+SCOPE_LOCK(f_lock);
+if (*(int *)data)
+f_flags |= FNONBLOCK;
+else
+f_flags &= ~FNONBLOCK;
+}



Why did you need to change this? did SCOPE_LOCK not work inside a 
"case", so you needed to add this extra scope?
Compiler complained about breaking/jumping out of case. Adding extra {} 
was the easiest solution.


libc/af_local.cc: In member function ‘virtual int 
af_local::ioctl(u_long, void*)’:

libc/af_local.cc:56:10: error: jump to case label [-fpermissive]
 case FIOASYNC:
  ^
In file included from libc/pipe_buffer.hh:15:0,
 from libc/af_local.cc:9:
include/osv/mutex.h:109:49: note:   crosses initialization of 
‘std::lock_guard _SCOPE_LOCK0’
 std::lock_guard<decltype(lock)> CONCATENATE(_SCOPE_LOCK, 
__COUNTER__) (lock)

 ^
include/osv/mutex.h:106:28: note: in definition of macro ‘CONCATENATE2’
 #define CONCATENATE2(x, y) x##y
^
include/osv/mutex.h:109:37: note: in expansion of macro ‘CONCATENATE’
 std::lock_guard<decltype(lock)> CONCATENATE(_SCOPE_LOCK, 
__COUNTER__) (lock)

 ^~~
libc/af_local.cc:48:13: note: in expansion of macro ‘SCOPE_LOCK’
 SCOPE_LOCK(f_lock);
 ^~
make: *** [build/debug.x64/libc/af_local.o] Error 1

I could also move SCOP_LOCK to function level. But that is not needed 
for the additional FIOASYNC stub. And if later additional cases are 
added, then those might not need that lock, so I didn't want to do that.



+error = 0;
+break;
+case FIOASYNC:
+WARN_ONCE("af_local::ioctl(FIOASYNC) stubbed\n");


I'm surprised nginx needed this in AF_LOCAL (i.e., unix domain 
sockets), not on normal TCP/IP sockets. All the more reason to think 
it is't important :-)

Socket is created by
if (socketpair(AF_UNIX, SOCK_STREAM, 0, ngx_processes[s].channel) == -1)
I guess it is used for interprocess communication. Maybe only to 
coordinate at app shutdown, maybe for more things.



 error = 0;
 break;
 }
--
2.9.4

--
You received this message because you are subscribed to the Google
Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to osv-dev+unsubscr...@googlegroups.com
<mailto:osv-dev%2bunsubscr...@googlegroups.com>.
For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.




--
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] Add nginx

[Justin Cinkelj]

The v2 patch:
The real sigsuspend patch (4 of them) are not required.
But, the stubs for sigsuspend, ioctl(FIOASYNC) and fcntl(F_SETOWN) are now 
required :)

In nginx.conf, I removed keepalive_timeout, as it is not really required.
And in makefile $(MAKE) is used instead of 'make -j4', so that parallel 
flag is inherited.

I still don't fully understang what problems will arise once we try to get 
more than one nginx "process". I guess there might be problems with workers 
being asked to terminate, etc. I guess threads in separate ELF namespaces 
will be required.

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH] signals: stub sigsuspend

[Justin Cinkelj]

sigsuspend is used in nginx. With current (not yet commited) patch, it
is never really called. Nginx master process code is used to execute
worker code, and this prevents master code to actually call sigsuspend.
So stub implementation is sufficient for now.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 libc/signal.cc | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/libc/signal.cc b/libc/signal.cc
index 832002c..74917c6 100644
--- a/libc/signal.cc
+++ b/libc/signal.cc
@@ -213,6 +213,12 @@ int sigprocmask(int how, const sigset_t* _set, sigset_t* 
_oldset)
 return 0;
 }
 
+int sigsuspend(const sigset_t *mask)
+{
+WARN_STUBBED();
+return -1;
+}
+
 int sigaction(int signum, const struct sigaction* act, struct sigaction* 
oldact)
 {
 // FIXME: We do not support any sa_flags besides SA_SIGINFO.
-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH 0/2] stub ioctl(FIOASYNC) and fcntl(F_SETOWN)

[Justin Cinkelj]

Both ioctl(FIOASYNC) and fcntl(F_SETOWN) are used in nginx.
We stub them, so than nginx doesn't need to be patched.
Or at least, only fork+execve will require a pathc.

Justin Cinkelj (2):
  af_local: stub ioctl FIOASYNC
  fcntl: stub F_SETOWN

 fs/vfs/main.cc   |  3 +++
 libc/af_local.cc | 18 +-
 2 files changed, 16 insertions(+), 5 deletions(-)

-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] Add nginx

[Justin Cinkelj]

On 07/16/2017 03:14 PM, Nadav Har'El wrote:


On Fri, Jul 14, 2017 at 12:02 PM, Justin Cinkelj 
<justin.cink...@xlab.si <mailto:justin.cink...@xlab.si>> wrote:


Based on @myechuri work,
https://github.com/myechuri/osv-apps/commits/nginx-osv
<https://github.com/myechuri/osv-apps/commits/nginx-osv>


Thanks (to both of you)!

I want to commit this patch (we can always improve it later), but have 
one question first - does this patch depend on any of your recent (yet 
uncomitted) patches, like the sigsuspend stuff?

Yes, the sigsuspend patch is required.



I also have some more minor comments and questions below:

A single nginx worker thread is run, so only one CPU core is utilized.


What would it involve to have multiple worker threads?
The linux fork is commented out, and then the main thread is used to run 
the only one worker thread. I think I can replace execve with osv_execve 
(or something like that), but I haven't tried yet.



Example configuration includes very long keepalive_timeout/requests


I think you don't actually need a very long keepalive *timeout* - even 
1 second should be more than enough (and anything more will obviously 
be fine) - if you only start a new connection once ever 1 second, in 2 
minutes (the default segment life, if I remember correctly) you only 
pass through 120 connections.


What you did need is a long keepalive *request limit* - say, 10,000. I 
think that in any case, regardless of any problem in OSv, it is a good 
idea to have a very large limit there.  I still remember why this 
limit was added in Apache HTTPd: The thinking is that the server 
surely has bugs, and after you process a large number of requests in 
the same server process (it was processes, not threads, originally) is 
likely to have accrued bugs - especially memory leaks - so it's a good 
idea to stop the process and start a new one, once in a while, e.g., 
every 1000 requests. This rationale is no longer relevant in nginx 
(where we have just one thread anyway, which never stops even if we 
break the client's connection), and there is no rationale, as far as I 
can see, why it ever makes sense to break a connection with a client 
who is still constantly sending requests. And there's especially no 
reason to break the connection after as little as 100 requests. Is 
this "100" an nginx default or some random limit which myechuri used 
in his original config file?
100 is nginx default. I used very high limit for both keepalive_timeout 
(to avoid problem with test longer than 75 sec - the default value is 
75s) and keepalive_requests (as some test are also longer that 100 req 
:)). I agree that restart process due to bugs cannot be applied on OSv 
(even after nginx stops, leaked memory is not freed - we need to restart 
whole VM to achieve that).



setting to avoid possible problems with OSV and TCP client port reuse.
With example configuration will server offer two URLs:
 http://$IP/index.html and
 http://$IP/basic_status

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si
<mailto:justin.cink...@xlab.si>>
---
 nginx/.gitignore   |   2 +
 nginx/Makefile |  45 +++
 nginx/module.py|   3 +
 .../0001-nginx-OSv-fix-process-spawning.patch |  59 +
 nginx/patches/nginx.conf   | 143
+
 5 files changed, 252 insertions(+)
 create mode 100644 nginx/.gitignore
 create mode 100644 nginx/Makefile
 create mode 100644 nginx/module.py
 create mode 100644
nginx/patches/0001-nginx-OSv-fix-process-spawning.patch
 create mode 100644 nginx/patches/nginx.conf

diff --git a/nginx/.gitignore b/nginx/.gitignore
new file mode 100644
index 000..cb8005e
--- /dev/null
+++ b/nginx/.gitignore
@@ -0,0 +1,2 @@
+usr.manifest
+upstream.tar
diff --git a/nginx/Makefile b/nginx/Makefile
new file mode 100644
index 000..be5fa9a
--- /dev/null
+++ b/nginx/Makefile
@@ -0,0 +1,45 @@
+VERSION=1.12.1
+SOURCE=http://nginx.org/download/nginx-${VERSION}.tar.gz
<http://nginx.org/download/nginx-$%7BVERSION%7D.tar.gz>
+CONFIGURE_MODULES=--prefix=/nginx/ --with-debug
--without-http_rewrite_module --with-threads
--with-http_stub_status_module
+
+.PHONY: module clean
+
+SRC=upstream/nginx
+
+module: usr.manifest
+
+usr.manifest: $(SRC)/nginx.so
+   echo '[manifest]' > usr.manifest
+   echo '/nginx.so: $${MODULE_DIR}/$(SRC)/nginx.so' >>
usr.manifest
+   echo '/nginx/html/**:
$${MODULE_DIR}/upstream/nginx/html/**' >> usr.manifest
+   echo '/nginx/logs/**:
$${MODULE_DIR}/upstream/nginx/logs/**' >> usr.manifest
+   echo '/nginx/conf/**:
$${MODULE_DIR}/up

[RFC 3/4] test blocked signal is delivered after unmask

[Justin Cinkelj]

From: Madhuri Yechuri 

---
 tests/tst-sigsuspend.cc | 35 +--
 1 file changed, 25 insertions(+), 10 deletions(-)

diff --git a/tests/tst-sigsuspend.cc b/tests/tst-sigsuspend.cc
index 603e817..ec95f48 100644
--- a/tests/tst-sigsuspend.cc
+++ b/tests/tst-sigsuspend.cc
@@ -23,32 +23,47 @@
 #include 
 #include 
 
-std::atomic siguser_received { 0 };
-void handler(int sig) { siguser_received = 1;}
+std::atomic siguser1_received { 0 };
+std::atomic siguser2_received { 0 };
+void handler1(int sig) {
+printf("Siguser1 handler received signal %d\n", sig);
+siguser1_received += 1;
+}
+void handler2(int sig) {
+printf("Siguser2 handler received signal %d\n", sig);
+siguser2_received += 1;
+}
 
 int main(int ac, char** av)
 {
 int sig;
-auto sr = signal(SIGUSR1, handler);
+auto sr = signal(SIGUSR1, handler1);
+assert(sr != SIG_ERR);
+sr = signal(SIGUSR2, handler2);
 assert(sr != SIG_ERR);
 
 sigset_t set, old_set, full_set;
-sigfillset();
 sigfillset(_set);
-sigdelset(, SIGUSR1);
+
 sigprocmask(SIG_SETMASK, _set, _set);
 printf("Masked all signals.\n");
 
-// Start a thread that sends SIGUSR1 to proc 0.
-std::thread thread1([&] { kill(0, SIGUSR1); });
-printf("Supending after unblocking SIGUSR1..\n");
+// Start a thread that sends SIGUSR1 and SIGUSR2 to proc 0.
+sigprocmask(SIG_SETMASK, _set, nullptr);
+std::thread thread1([&] { kill(0, SIGUSR1); kill(0, SIGUSR2);});
+printf("Test 1: Supend after unblocking SIGUSR1 and SIGUSR2..\n");
+sigfillset();
+sigdelset(, SIGUSR2);
 sigsuspend();
 thread1.join();
-assert(siguser_received == 1);
-printf("Verified handler for SIGUSR1 is invoked.\n");
+assert(siguser1_received == 0 && siguser2_received == 1);
+printf("Verified handler for SIGUSR2 is invoked.\n");
 
 // Reset proc mask.
+printf("Test 2: Reset proc mask to process SIGUSR1 generated from Test 
1\n");
 sigprocmask(SIG_SETMASK, _set, nullptr);
+assert(siguser1_received == 1);
+printf("Verified handler for SIGUSR1 is invoked after restoring mask.\n");
 printf ("Done.\n");
 return 0;
 }
-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[RFC 2/4] ignore blocked signals

[Justin Cinkelj]

From: Madhuri Yechuri 

---
 libc/signal.cc | 13 +++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/libc/signal.cc b/libc/signal.cc
index abbcedc..1c5c923 100644
--- a/libc/signal.cc
+++ b/libc/signal.cc
@@ -228,8 +228,17 @@ int sigsuspend(const sigset_t *mask) {
 }
 }
 
-// Wait until we receive a signal, then call its handler.
-sched::thread::wait_until([] { return sig = thread_pending_signal; });
+// Wait until we receive unblocked signal, then call its handler.
+sched::thread::wait_until([new_mask] {
+if (new_mask.test(thread_pending_signal)) {
+// Ignore masked signals.
+thread_pending_signal = 0;
+return false;
+} else {
+return true;
+}
+});
+sig = thread_pending_signal;
 thread_pending_signal = 0;
 const auto sa = signal_actions[sig];
 sa.sa_sigaction(sig, nullptr, nullptr);
-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[RFC 1/4] first draft of sigsuspend and unit test

[Justin Cinkelj]

From: Madhuri Yechuri 

---
 libc/signal.cc  | 33 ++
 modules/tests/Makefile  |  2 +-
 tests/tst-sigsuspend.cc | 54 +
 3 files changed, 88 insertions(+), 1 deletion(-)
 create mode 100644 tests/tst-sigsuspend.cc

diff --git a/libc/signal.cc b/libc/signal.cc
index 832002c..abbcedc 100644
--- a/libc/signal.cc
+++ b/libc/signal.cc
@@ -213,6 +213,39 @@ int sigprocmask(int how, const sigset_t* _set, sigset_t* 
_oldset)
 return 0;
 }
 
+int sigsuspend(const sigset_t *mask) {
+auto new_mask = from_libc(mask)->mask;
+auto old_mask = thread_signals()->mask;
+int sig;
+
+// Temporarily replace thread's mask set with input set.
+thread_signals()->mask = new_mask;
+for (unsigned i = 0; i < nsignals; ++i) {
+if (new_mask.test(i) and !old_mask.test(i)) {
+unwait_for_signal(i);
+} else if (old_mask.test(i) and !new_mask.test(i)) {
+wait_for_signal(i);
+}
+}
+
+// Wait until we receive a signal, then call its handler.
+sched::thread::wait_until([] { return sig = thread_pending_signal; });
+thread_pending_signal = 0;
+const auto sa = signal_actions[sig];
+sa.sa_sigaction(sig, nullptr, nullptr);
+
+// Restore original mask.
+thread_signals()->mask = old_mask;
+for (unsigned i = 0; i < nsignals; ++i) {
+if (old_mask.test(i) and !new_mask.test(i)) {
+unwait_for_signal(i);
+} else if (new_mask.test(i) and !old_mask.test(i)) {
+wait_for_signal(i);
+}
+}
+return -1;
+}
+
 int sigaction(int signum, const struct sigaction* act, struct sigaction* 
oldact)
 {
 // FIXME: We do not support any sa_flags besides SA_SIGINFO.
diff --git a/modules/tests/Makefile b/modules/tests/Makefile
index a65e5eb..236bb6d 100644
--- a/modules/tests/Makefile
+++ b/modules/tests/Makefile
@@ -87,7 +87,7 @@ tests := tst-pthread.so misc-ramdisk.so tst-vblk.so 
tst-bsd-evh.so \
tst-ifaddrs.so tst-pthread-affinity-inherit.so tst-sem-timed-wait.so \
tst-ttyname.so tst-pthread-barrier.so tst-feexcept.so tst-math.so \
tst-sigaltstack.so tst-fread.so tst-tcp-cork.so tst-tcp-v6.so \
-   tst-calloc.so
+   tst-calloc.so tst-sigsuspend.so
 
 #  libstatic-thread-variable.so tst-static-thread-variable.so \
 
diff --git a/tests/tst-sigsuspend.cc b/tests/tst-sigsuspend.cc
new file mode 100644
index 000..603e817
--- /dev/null
+++ b/tests/tst-sigsuspend.cc
@@ -0,0 +1,54 @@
+/*
+ * Copyright (C) 2014 Cloudius Systems, Ltd.
+ *
+ * This work is open source software, licensed under the terms of the
+ * BSD license as described in the LICENSE file in the top-level directory.
+ *
+ * Testing on Linux:
+ * Step 1: g++ -g -pthread -std=c++11 tests/tst-sigsuspend.cc
+ * Step 2: a.out
+ *
+ * Testing on OSv:
+ * Step 1: scripts/build mode=debug image=tests
+ * Step 2: scripts/run.py -e /tests/tst-sigsuspend.so
+ */
+
+
+#include 
+#include 
+#include 
+
+#include 
+#include 
+#include 
+#include 
+
+std::atomic siguser_received { 0 };
+void handler(int sig) { siguser_received = 1;}
+
+int main(int ac, char** av)
+{
+int sig;
+auto sr = signal(SIGUSR1, handler);
+assert(sr != SIG_ERR);
+
+sigset_t set, old_set, full_set;
+sigfillset();
+sigfillset(_set);
+sigdelset(, SIGUSR1);
+sigprocmask(SIG_SETMASK, _set, _set);
+printf("Masked all signals.\n");
+
+// Start a thread that sends SIGUSR1 to proc 0.
+std::thread thread1([&] { kill(0, SIGUSR1); });
+printf("Supending after unblocking SIGUSR1..\n");
+sigsuspend();
+thread1.join();
+assert(siguser_received == 1);
+printf("Verified handler for SIGUSR1 is invoked.\n");
+
+// Reset proc mask.
+sigprocmask(SIG_SETMASK, _set, nullptr);
+printf ("Done.\n");
+return 0;
+}
-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH v2] cloud-init: support standard ISO images

[Justin Cinkelj]

In 56c6195b252b127c7fe90cb9fa46c370b384b51a was added option for
reading configuration file from a secondary disk. This is so-called
NoCloud configuration method, see also
http://cloudinit.readthedocs.io/en/0.7.9/topics/datasources/nocloud.html.
Due to missing ISO filesystem support, a custom disk format was used.
This patch replaces custom disk format with standard ISO images.
ISO FS support is provided by libcdio.

After preparing user-data file (meta-data file is currently ignored),
the image can be build with cloud-localds program (provided by cloud-utils
package on fedora).

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 modules/cloud-init/Makefile  |  3 +-
 modules/cloud-init/main.cc   | 80 +++-
 modules/cloud-init/module.py |  1 +
 scripts/file2img | 15 -
 4 files changed, 45 insertions(+), 54 deletions(-)
 delete mode 100755 scripts/file2img

diff --git a/modules/cloud-init/Makefile b/modules/cloud-init/Makefile
index 2ebd479..8b2b03b 100644
--- a/modules/cloud-init/Makefile
+++ b/modules/cloud-init/Makefile
@@ -1,4 +1,5 @@
-INCLUDES += -I../../include -I.
+INCLUDES = -I. -I../../include -I../../arch/$(ARCH) -I../.. \
+-I../../build/$(mode)/gen/include
 
 autodepend = -MD -MT $@ -MP
 CXXFLAGS  = -g -rdynamic -Wall -std=c++11 -fPIC $(INCLUDES) $(autodepend)
diff --git a/modules/cloud-init/main.cc b/modules/cloud-init/main.cc
index a64d342..1148006 100644
--- a/modules/cloud-init/main.cc
+++ b/modules/cloud-init/main.cc
@@ -5,6 +5,9 @@
  * BSD license as described in the LICENSE file in the top-level directory.
  */
 
+#include 
+#include 
+#include 
 #include "cloud-init.hh"
 #include "files-module.hh"
 #include "server-module.hh"
@@ -15,59 +18,60 @@
 #include 
 #include 
 #include 
+#include 
 
 using namespace std;
 using namespace init;
 namespace po = boost::program_options;
 
-// config_disk() checks whether we have a second disk (/dev/vblk1) which
-// holds nothing but a configuration file, and if there is, it copies the
-// configuration to the given file.
-// Currently, the configuration disk must be in a trivial format generated
-// by scripts/file2img: A magic header, then the length of the file (in
-// decimal), followed by the content.
+// config_disk() allows to use NoCloud VM configuration method - see
+// http://cloudinit.readthedocs.io/en/0.7.9/topics/datasources/nocloud.html.
+// NoCloud method provides two files with cnfiguration data (/user-data and
+// /meta-data) on a disk. The disk is required to have label "cidata".
+// It can contain ISO9660 or FAT filesystem.
+//
+// config_disk() checks whether we have a second disk (/dev/vblk1) with
+// ISO image, and if there is, it copies the configuration file from
+// /user-data to the given file.
 // config_disk() returns true if it has successfully read the configuration
 // into the requested file.
+//
+// OSv implementation limitations:
+// The /meta-data file is currently ignored.
+// Only ISO9660 filesystem is supported.
+// The mandatory "cidata" volume label is not checked.
+//
+// Example ISO image can be created by running
+// cloud-localds cloud-init.img cloud-init.yaml
+// The cloud-localds command is provided by cloud-utils package (fedora).
 static bool config_disk(const char* outfile) {
-int fd = open("/dev/vblk1", O_RDONLY);
-if (fd < 0) {
+char disk[] = "/dev/vblk1";
+char srcfile[] = "/user-data";
+struct stat sb;
+int ret;
+int app_ret = -1;
+
+ret = stat(disk, );
+if (ret != 0) {
 return false;
 }
-char data[512];
-ssize_t r = read(fd, data, sizeof(data));
-static const char* magic = "!file_in_image\n";
-ssize_t magic_len = strlen(magic);
-if (r < magic_len || strncmp(data, magic, magic_len)) {
-close(fd);
+
+std::vector cmd = {"/usr/bin/iso-read.so", "-e", srcfile, 
"-o", outfile, disk};
+osv::run(cmd[0], cmd, _ret);
+if (app_ret != 0) {
+debug("cloud-init: warning, %s exited with code %d (%s is not ISO 
image?)\n", cmd[0], app_ret, disk);
 return false;
 }
-debug("cloud-init: found configuration in /dev/vblk1\n");
-int out = open(outfile, O_WRONLY|O_CREAT, 0777);
-if (out < 0) {
-debug("cloud-init: failed to copy configuration to %s\n", outfile);
-close(fd);
+ret = stat(outfile, );
+if (ret != 0) {
+debug("cloud-init: stat(%s) failed, errno=%d\n", outfile, errno);
 return false;
 }
-unsigned offset = magic_len;
-while (offset < r && data[offset] != '\n')
-offset++;
-offset++; // skip the \n too
-ssize_t filelen = atoi(data + magic_len);
-while (filelen) {
-int len = std::min(filelen, r - offset);
-if (len <= 0) {
-debug("

Re: [PATCH] Support mmap() flag MAP_32BIT

[Justin Cinkelj]

The patch works for me. With it I'm able to run wrk.
Thank you.

Justin

On Tuesday, July 11, 2017 at 4:51:42 PM UTC+2, Nadav Har'El wrote:
>
> Support the mmap() flag MAP_32BIT which asks for the allocated address 
> to be in the lower 32 bits. Some applications (like wrk and luajit) 
> use this flag and need it to work as expected. 
>
> By default, OSv mappings look for a hole starting at 0x2000. 
> With this patch and MAP_32BIT, we change that to 0x200. 
>
> Signed-off-by: Nadav Har'El  
> --- 
>  libc/mman.cc | 7 +++ 
>  1 file changed, 7 insertions(+) 
>
> diff --git a/libc/mman.cc b/libc/mman.cc 
> index 0441265..73a3f41 100644 
> --- a/libc/mman.cc 
> +++ b/libc/mman.cc 
> @@ -130,6 +130,13 @@ void *mmap(void *addr, size_t length, int prot, int 
> flags, 
>  auto mmap_flags = libc_flags_to_mmap(flags); 
>  auto mmap_perm  = libc_prot_to_perm(prot); 
>   
> +if ((flags & MAP_32BIT) && !(flags & MAP_FIXED) && !addr) { 
> +// If addr is not specified, OSv by default starts mappings at 
> address 
> +// 0x2000ul (see mmu::allocate()).  MAP_32BIT asks for a 
> lower 
> +// default. If MAP_FIXED or addr were specified, the default does 
> not 
> +// matter anyway. 
> +addr = (void*)0x200ul; 
> +} 
>  if (flags & MAP_ANONYMOUS) { 
>  // We have already determined (see below) the region where the 
> heap must be located. Now the JVM will request 
>  // fixed mappings inside that region 
> -- 
> 2.9.4 
>
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] command line: expand env vars before using them to expand cmdline

[Justin Cinkelj]

Miha, please confirm this fixes problem you have.

Justin

On 07/11/2017 04:31 PM, Justin Cinkelj wrote:

Previously, runscript with content
"--env=PORT?= /usr/lib/mpi_hello.so aaa $PORT ccc",
run as
/scripts/run.py -Vd -e '--env=PORT=newport runscript /script/my-script'
started mpi_hello.so with
argv[2] == "newport" and environ["PORT"] == "".
This was wrong and confusing.

Fix it by first evaluating env var, and only then insert env var values
into final command line.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
  core/commands.cc | 68 +++-
  1 file changed, 47 insertions(+), 21 deletions(-)

diff --git a/core/commands.cc b/core/commands.cc
index 2947e92..6287d76 100644
--- a/core/commands.cc
+++ b/core/commands.cc
@@ -96,23 +96,31 @@ Everything after first $ is assumed to be env var.
  So with AA=aa, "$AA" -> "aa", and "X$AA" => "Xaa"
  More than one $ is not supported, and "${AA}" is also not.
  */
+void expand_environ_vars(std::string& word)
+{
+size_t pos;
+if ((pos = word.find_first_of('$')) != std::string::npos) {
+std::string new_word = word.substr(0, pos);
+std::string key = word.substr(pos+1);
+auto tmp = getenv(key.c_str());
+//debug("new_word=%s, key=%s, tmp=%s\n", new_word.c_str(), 
key.c_str(), tmp);
+if (tmp) {
+new_word += tmp;
+}
+word = new_word;
+}
+}
+
+/*
+Expand environ vars in each word.
+*/
  void expand_environ_vars(std::vector<std::vector>& result)
  {
  std::vector<std::vector>::iterator cmd_iter;
  std::vector::iterator line_iter;
  for (cmd_iter=result.begin(); cmd_iter!=result.end(); cmd_iter++) {
  for (line_iter=cmd_iter->begin(); line_iter!=cmd_iter->end(); 
line_iter++) {
-size_t pos;
-if ((pos = line_iter->find_first_of('$')) != std::string::npos) {
-std::string new_word = line_iter->substr(0, pos);
-std::string key = line_iter->substr(pos+1);
-auto tmp = getenv(key.c_str());
-//debug("new_word=%s, key=%s, tmp=%s\n", new_word.c_str(), 
key.c_str(), tmp);
-if (tmp) {
-new_word += tmp;
-}
-*line_iter = new_word;
-}
+expand_environ_vars(*line_iter);
  }
  }
  }
@@ -167,21 +175,36 @@ static void 
runscript_process_options(std::vector<std::vector >& re
  if (vars.count("env")) {
  for (auto t : vars["env"].as<std::vector>()) {
  size_t pos = t.find("?=");
+std::string key, value;
  if (std::string::npos == pos) {
  // the basic "KEY=value" syntax
-debug("Setting in environment: %s\n", t);
-putenv(strdup(t.c_str()));
+size_t pos2 = t.find("=");
+assert(std::string::npos != pos2);
+key = t.substr(0, pos2);
+value = t.substr(pos2+1);
+//debug("Setting in environment (def): %s=%s\n", key, 
value);
  }
  else {
  // "KEY?=value", makefile-like syntax, set variable only 
if not yet set
-auto key = t.substr(0, pos);
-auto value = t.substr(pos+2);
-if (nullptr == getenv(key.c_str())) {
-debug("Setting in environment: %s=%s\n", key, value);
-setenv(key.c_str(), value.c_str(), 1);
+key = t.substr(0, pos);
+value = t.substr(pos+2);
+if (nullptr != getenv(key.c_str())) {
+// key already used, do not overwrite it
+//debug("NOT setting in environment (makefile): 
%s=%s\n", key, value);
+key = "";
+}
+else {
+//debug("Setting in environment (makefile): %s=%s\n", 
key, value);
  }
  }
  
+if (key.length() > 0) {

+// we have something to set
+expand_environ_vars(value);
+debug("Setting in environment: %s=%s\n", key, value);
+setenv(key.c_str(), value.c_str(), 1);
+}
+
  }
  }
  
@@ -191,7 +214,7 @@ static void runscript_process_options(std::vector<std::vector >& re

  }
  
  /*

-If cmd starts with "runcript file", read content of file and
+If cmd starts w

[PATCH] command line: expand env vars before using them to expand cmdline

[Justin Cinkelj]

Previously, runscript with content
"--env=PORT?= /usr/lib/mpi_hello.so aaa $PORT ccc",
run as
/scripts/run.py -Vd -e '--env=PORT=newport runscript /script/my-script'
started mpi_hello.so with
argv[2] == "newport" and environ["PORT"] == "".
This was wrong and confusing.

Fix it by first evaluating env var, and only then insert env var values
into final command line.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 core/commands.cc | 68 +++-
 1 file changed, 47 insertions(+), 21 deletions(-)

diff --git a/core/commands.cc b/core/commands.cc
index 2947e92..6287d76 100644
--- a/core/commands.cc
+++ b/core/commands.cc
@@ -96,23 +96,31 @@ Everything after first $ is assumed to be env var.
 So with AA=aa, "$AA" -> "aa", and "X$AA" => "Xaa"
 More than one $ is not supported, and "${AA}" is also not.
 */
+void expand_environ_vars(std::string& word)
+{
+size_t pos;
+if ((pos = word.find_first_of('$')) != std::string::npos) {
+std::string new_word = word.substr(0, pos);
+std::string key = word.substr(pos+1);
+auto tmp = getenv(key.c_str());
+//debug("new_word=%s, key=%s, tmp=%s\n", new_word.c_str(), 
key.c_str(), tmp);
+if (tmp) {
+new_word += tmp;
+}
+word = new_word;
+}
+}
+
+/*
+Expand environ vars in each word.
+*/
 void expand_environ_vars(std::vector<std::vector>& result)
 {
 std::vector<std::vector>::iterator cmd_iter;
 std::vector::iterator line_iter;
 for (cmd_iter=result.begin(); cmd_iter!=result.end(); cmd_iter++) {
 for (line_iter=cmd_iter->begin(); line_iter!=cmd_iter->end(); 
line_iter++) {
-size_t pos;
-if ((pos = line_iter->find_first_of('$')) != std::string::npos) {
-std::string new_word = line_iter->substr(0, pos);
-std::string key = line_iter->substr(pos+1);
-auto tmp = getenv(key.c_str());
-//debug("new_word=%s, key=%s, tmp=%s\n", new_word.c_str(), 
key.c_str(), tmp);
-if (tmp) {
-new_word += tmp;
-}
-*line_iter = new_word;
-}
+expand_environ_vars(*line_iter);
 }
 }
 }
@@ -167,21 +175,36 @@ static void 
runscript_process_options(std::vector<std::vector >& re
 if (vars.count("env")) {
 for (auto t : vars["env"].as<std::vector>()) {
 size_t pos = t.find("?=");
+std::string key, value;
 if (std::string::npos == pos) {
 // the basic "KEY=value" syntax
-debug("Setting in environment: %s\n", t);
-putenv(strdup(t.c_str()));
+size_t pos2 = t.find("=");
+assert(std::string::npos != pos2);
+key = t.substr(0, pos2);
+value = t.substr(pos2+1);
+//debug("Setting in environment (def): %s=%s\n", key, 
value);
 }
 else {
 // "KEY?=value", makefile-like syntax, set variable only 
if not yet set
-auto key = t.substr(0, pos);
-auto value = t.substr(pos+2);
-if (nullptr == getenv(key.c_str())) {
-debug("Setting in environment: %s=%s\n", key, value);
-setenv(key.c_str(), value.c_str(), 1);
+key = t.substr(0, pos);
+value = t.substr(pos+2);
+if (nullptr != getenv(key.c_str())) {
+// key already used, do not overwrite it
+//debug("NOT setting in environment (makefile): 
%s=%s\n", key, value);
+key = "";
+}
+else {
+//debug("Setting in environment (makefile): %s=%s\n", 
key, value);
 }
 }
 
+if (key.length() > 0) {
+// we have something to set
+expand_environ_vars(value);
+debug("Setting in environment: %s=%s\n", key, value);
+setenv(key.c_str(), value.c_str(), 1);
+}
+
 }
 }
 
@@ -191,7 +214,7 @@ static void 
runscript_process_options(std::vector<std::vector >& re
 }
 
 /*
-If cmd starts with "runcript file", read content of file and
+If cmd starts with "runscript file", read content of file and
 return vector of all programs to be run.
 File can contain multiple commands per line.
 ok

[PATCH] netperf: update download link

[Justin Cinkelj]

The netperf.org site was migrated to github.
The ftp.netperf.org is not even in DNS anymore.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 netperf/GET | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/netperf/GET b/netperf/GET
index 1eb060f..3cdb12d 100755
--- a/netperf/GET
+++ b/netperf/GET
@@ -7,8 +7,9 @@ rm -rf upstream
 
 mkdir upstream
 cd upstream
-wget ftp://ftp.netperf.org/netperf/netperf-$VERSION.tar.bz2
-tar jxf netperf-$VERSION.tar.bz2
+wget https://codeload.github.com/HewlettPackard/netperf/zip/netperf-$VERSION 
-O netperf-$VERSION.zip
+unzip netperf-$VERSION.zip
+mv netperf-netperf-$VERSION netperf-$VERSION
 cd netperf-$VERSION
 
 ./configure
-- 
2.9.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH 1/2] modules: add libcdio

[Justin Cinkelj]

The iso-read from libcdio will be used to read ISO image
(contextualization CD, cloud metadata image). This will allow use of
standard ISO images in OSv cloud-init implementation, instead of using a
custom format (see 56c6195b252b127c7fe90cb9fa46c370b384b51a).

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 modules/libcdio/.gitignore   |  2 ++
 modules/libcdio/Makefile | 52 
 modules/libcdio/exit.patch   | 14 +++
 modules/libcdio/module.py| 13 ++
 modules/libcdio/relink-binary.sh | 14 +++
 5 files changed, 95 insertions(+)
 create mode 100644 modules/libcdio/.gitignore
 create mode 100644 modules/libcdio/Makefile
 create mode 100644 modules/libcdio/exit.patch
 create mode 100644 modules/libcdio/module.py
 create mode 100755 modules/libcdio/relink-binary.sh

diff --git a/modules/libcdio/.gitignore b/modules/libcdio/.gitignore
new file mode 100644
index 000..5fc3ba0
--- /dev/null
+++ b/modules/libcdio/.gitignore
@@ -0,0 +1,2 @@
+build
+download
diff --git a/modules/libcdio/Makefile b/modules/libcdio/Makefile
new file mode 100644
index 000..c94ead2
--- /dev/null
+++ b/modules/libcdio/Makefile
@@ -0,0 +1,52 @@
+# A small Makefile "download and build" for libcdio
+
+# Directories for downloading and building
+DOWNLOAD=download
+BUILD=build
+
+# libcdio
+LIBCDIO_VERSION=0.94
+LIBCDIO_FOLDER=libcdio-$(LIBCDIO_VERSION)
+LIBCDIO_DOWNLOAD=https://ftp.gnu.org/gnu/libcdio/libcdio-$(LIBCDIO_VERSION).tar.gz
+LIBCDIO_ARCHIVE=download/$(LIBCDIO_FOLDER).tar.gz
+LIBCDIO_BUILD=$(BUILD)/$(LIBCDIO_FOLDER)
+
+CFLAGS:=-fPIC
+# libtool would ignore -shared anyway
+# We should set LDFLAGS=-shared. But then configure stage fails with 
+# "configure: error: C compiler cannot create executables".
+# If we later edit Makefile to set LDFLAGS=-shared, then libtool removes it.
+# So, we will capture required cmd from make V=1, and relink the binary.
+LDFLAGS:=
+
+MAIN=$(LIBCDIO_BUILD)/src/.libs/iso-read
+
+module: $(MAIN) $(MAIN).so
+
+$(MAIN): $(LIBCDIO_BUILD)/Makefile
+   cd $(LIBCDIO_BUILD) && make
+
+$(MAIN).so: $(MAIN)
+   ./relink-binary.sh $@
+
+$(LIBCDIO_BUILD)/Makefile: $(LIBCDIO_BUILD)/configure
+   cd $(LIBCDIO_BUILD) && CFLAGS=$(CFLAGS) LDFLAGS=$(LDFLAGS) ./configure
+
+$(LIBCDIO_BUILD)/configure: $(LIBCDIO_ARCHIVE) | $(BUILD)
+   cd $(BUILD) && tar xzf ../$(LIBCDIO_ARCHIVE)
+   cd $(LIBCDIO_BUILD) && patch -p1 < ../../exit.patch
+   touch $(LIBCDIO_BUILD)/configure
+
+$(LIBCDIO_ARCHIVE): | $(DOWNLOAD)
+   cd $(DOWNLOAD) && \
+   curl --remote-name --remote-time $(LIBCDIO_DOWNLOAD)
+
+$(DOWNLOAD) $(BUILD):
+   @mkdir -p $@
+
+clean:
+   rm -rf $(BUILD) $(DOWNLOAD)
+
+
+
+.PHONY: module clean
diff --git a/modules/libcdio/exit.patch b/modules/libcdio/exit.patch
new file mode 100644
index 000..403736c
--- /dev/null
+++ b/modules/libcdio/exit.patch
@@ -0,0 +1,14 @@
+diff --git a/lib/driver/logging.c b/lib/driver/logging.c
+index b89ab37..202256c 100644
+--- a/lib/driver/logging.c
 b/lib/driver/logging.c
+@@ -49,7 +49,8 @@ cdio_default_log_handler(cdio_log_level_t level, const char 
message[])
+ fprintf (stderr, "**ERROR: %s\n", message);
+ fflush (stderr);
+   }
+-  exit (EXIT_FAILURE);
++  // Is triggered by invalid ISO image - input file smaller that about 32 
kB
++  // exit (EXIT_FAILURE);
+   break;
+ case CDIO_LOG_DEBUG:
+   if (level >= cdio_loglevel_default) {
diff --git a/modules/libcdio/module.py b/modules/libcdio/module.py
new file mode 100644
index 000..f7b98d9
--- /dev/null
+++ b/modules/libcdio/module.py
@@ -0,0 +1,13 @@
+from osv.modules.filemap import FileMap
+
+VERSION=0.94
+
+usr_files = FileMap()
+usr_files.add('${OSV_BASE}/modules/libcdio/build/libcdio-%s/src/.libs' % 
VERSION).to('/usr/bin') \
+   .include('iso-read.so')
+usr_files.add('${OSV_BASE}/modules/libcdio/build/libcdio-%s/lib/udf/.libs' % 
VERSION).to('/usr/lib') \
+   .include('libudf.so.0')
+usr_files.add('${OSV_BASE}/modules/libcdio/build/libcdio-%s/lib/iso9660/.libs' 
% VERSION).to('/usr/lib') \
+   .include('libiso9660.so.10')
+usr_files.add('${OSV_BASE}/modules/libcdio/build/libcdio-%s/lib/driver/.libs' 
% VERSION).to('/usr/lib') \
+   .include('libcdio.so.16')
diff --git a/modules/libcdio/relink-binary.sh b/modules/libcdio/relink-binary.sh
new file mode 100755
index 000..7a89f54
--- /dev/null
+++ b/modules/libcdio/relink-binary.sh
@@ -0,0 +1,14 @@
+#!/bin/bash
+
+set -ev
+
+BIN=$1
+SRC_DIR=$(dirname "$BIN")/..
+SRC_FILE_BASE=$(basename "$BIN" | sed 's/\.so$//')
+SRC_FILE=$SRC_FILE_BASE.c
+
+cd $SRC_DIR
+touch $SRC_FILE
+CMD1=$(make V=1 | grep "\-o .libs/$SRC_FILE_BASE")
+CMD2=$(echo $CMD1 | sed -e "s|^libtool: link: ccache ||" -e "s| -o 
.libs/$SRC_FILE_BASE | -shared -o .libs/$SRC_FIL

[PATCH 0/2] cloud-init: support standard ISO images

[Justin Cinkelj]

First patch includes libcdio for reading files from ISO images.
Second patch updates cloud-init to allow use of ISO images for
"no-cloud" cloud configuration.

The libcdio patch requires calloc() fix to be applied (a v2 patch
was sent today).

Justin Cinkelj (2):
  modules: add libcdio
  cloud-init: support standard ISO images

 modules/cloud-init/Makefile  |  3 +-
 modules/cloud-init/main.cc   | 68 ++--
 modules/cloud-init/module.py |  1 +
 modules/libcdio/.gitignore   |  2 ++
 modules/libcdio/Makefile | 52 ++
 modules/libcdio/exit.patch   | 14 +
 modules/libcdio/module.py| 13 
 modules/libcdio/relink-binary.sh | 14 +
 scripts/file2img | 15 -
 9 files changed, 128 insertions(+), 54 deletions(-)
 create mode 100644 modules/libcdio/.gitignore
 create mode 100644 modules/libcdio/Makefile
 create mode 100644 modules/libcdio/exit.patch
 create mode 100644 modules/libcdio/module.py
 create mode 100755 modules/libcdio/relink-binary.sh
 delete mode 100755 scripts/file2img

-- 
2.9.3

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH 2/2] cloud-init: support standard ISO images

[Justin Cinkelj]

In 56c6195b252b127c7fe90cb9fa46c370b384b51a was added option for
reading configuration file from a secondary disk. Due to missing ISO
filesystem support, a custom disk format was used. This patch replaces
custom disk format with standard ISO images. ISO FS support is provided
by libcdio.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 modules/cloud-init/Makefile  |  3 +-
 modules/cloud-init/main.cc   | 68 +++-
 modules/cloud-init/module.py |  1 +
 scripts/file2img | 15 --
 4 files changed, 33 insertions(+), 54 deletions(-)
 delete mode 100755 scripts/file2img

diff --git a/modules/cloud-init/Makefile b/modules/cloud-init/Makefile
index 2ebd479..8b2b03b 100644
--- a/modules/cloud-init/Makefile
+++ b/modules/cloud-init/Makefile
@@ -1,4 +1,5 @@
-INCLUDES += -I../../include -I.
+INCLUDES = -I. -I../../include -I../../arch/$(ARCH) -I../.. \
+-I../../build/$(mode)/gen/include
 
 autodepend = -MD -MT $@ -MP
 CXXFLAGS  = -g -rdynamic -Wall -std=c++11 -fPIC $(INCLUDES) $(autodepend)
diff --git a/modules/cloud-init/main.cc b/modules/cloud-init/main.cc
index a64d342..284570f 100644
--- a/modules/cloud-init/main.cc
+++ b/modules/cloud-init/main.cc
@@ -5,6 +5,9 @@
  * BSD license as described in the LICENSE file in the top-level directory.
  */
 
+#include 
+#include 
+#include 
 #include "cloud-init.hh"
 #include "files-module.hh"
 #include "server-module.hh"
@@ -15,59 +18,48 @@
 #include 
 #include 
 #include 
+#include 
 
 using namespace std;
 using namespace init;
 namespace po = boost::program_options;
 
-// config_disk() checks whether we have a second disk (/dev/vblk1) which
-// holds nothing but a configuration file, and if there is, it copies the
-// configuration to the given file.
-// Currently, the configuration disk must be in a trivial format generated
-// by scripts/file2img: A magic header, then the length of the file (in
-// decimal), followed by the content.
+// config_disk() checks whether we have a second disk (/dev/vblk1) with
+// ISO image, and if there is, it copies the configuration file from
+// /user-data to the given file.
 // config_disk() returns true if it has successfully read the configuration
 // into the requested file.
+//
+// Example ISO image can be created by running
+// cloud-localds cloud-init.img cloud-init.yaml
 static bool config_disk(const char* outfile) {
-int fd = open("/dev/vblk1", O_RDONLY);
-if (fd < 0) {
+char disk[] = "/dev/vblk1";
+char srcfile[] = "/user-data";
+struct stat sb;
+int ret;
+int app_ret = -1;
+
+ret = stat(disk, );
+if (ret != 0) {
 return false;
 }
-char data[512];
-ssize_t r = read(fd, data, sizeof(data));
-static const char* magic = "!file_in_image\n";
-ssize_t magic_len = strlen(magic);
-if (r < magic_len || strncmp(data, magic, magic_len)) {
-close(fd);
+
+std::vector cmd = {"/usr/bin/iso-read.so", "-e", srcfile, 
"-o", outfile, disk};
+osv::run(cmd[0], cmd, _ret);
+if (app_ret != 0) {
+debug("cloud-init: %s exited with code %d\n", cmd[0], app_ret);
 return false;
 }
-debug("cloud-init: found configuration in /dev/vblk1\n");
-int out = open(outfile, O_WRONLY|O_CREAT, 0777);
-if (out < 0) {
-debug("cloud-init: failed to copy configuration to %s\n", outfile);
-close(fd);
+ret = stat(outfile, );
+if (ret != 0) {
+debug("cloud-init: stat(%s) failed, errno=%d\n", outfile, errno);
 return false;
 }
-unsigned offset = magic_len;
-while (offset < r && data[offset] != '\n')
-offset++;
-offset++; // skip the \n too
-ssize_t filelen = atoi(data + magic_len);
-while (filelen) {
-int len = std::min(filelen, r - offset);
-if (len <= 0) {
-debug("cloud-init: unexpected end of image\n", outfile);
-close(fd);
-close(out);
-return false;
-}
-write(out, data + offset, len);
-filelen -= len;
-offset = 0;
-r = read(fd, data, sizeof(data));
+if ((sb.st_mode & S_IFMT) != S_IFREG) {
+debug("cloud-init: %s is not a file\n", outfile);
+return false;
 }
-close(fd);
-close(out);
+debug("cloud-init: copied file %s -> %s from ISO image %s\n", srcfile, 
outfile, disk);
 return true;
 }
 
diff --git a/modules/cloud-init/module.py b/modules/cloud-init/module.py
index 7586eb6..c544742 100644
--- a/modules/cloud-init/module.py
+++ b/modules/cloud-init/module.py
@@ -12,6 +12,7 @@ usr_files.add(os.path.join(_module, 
'cmdline')).to('/init/00-cmdline')
 
 api.require('httpserver')
 api.require('libyaml')
+api.require('libcdio')
 
 #default = api.run('/usr/mgmt/clou

Re: [PATCH] calloc: allow 0-sized allocation

[Justin Cinkelj]

On 06/14/2017 05:23 PM, Nadav Har'El wrote:


On Wed, Jun 14, 2017 at 4:56 PM, Justin Cinkelj 
<justin.cink...@xlab.si <mailto:justin.cink...@xlab.si>> wrote:


The calloc(nmemb, size=0) failed due to division by 0 in
"std::numeric_limits::max() / size".
Fix this by checking for 0 value before division.

Included is also a small test case for calloc and memset.

    Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si
<mailto:justin.cink...@xlab.si>>
---
 core/mempool.cc|  2 +
 modules/tests/Makefile |  2 +-
 tests/tst-calloc.cc| 99
++
 3 files changed, 102 insertions(+), 1 deletion(-)
 create mode 100644 tests/tst-calloc.cc

diff --git a/core/mempool.cc b/core/mempool.cc
index 0b1e69e..575e496 100644
--- a/core/mempool.cc
+++ b/core/mempool.cc
@@ -1564,6 +1564,8 @@ static inline void* std_malloc(size_t size,
size_t alignment)

 void* calloc(size_t nmemb, size_t size)
 {
+if (nmemb == 0 || size == 0)
+return nullptr;


You could have also done return malloc(0) here, in case you don't want 
to return a nullptr in this case (I think out malloc(0) doesn't return 
nullptr)?
But according to the calloc() what you did is fine: calloc with 0 
members or member size 0 can return nullptr.

I will do that, because that is what Linux does.


 if (nmemb > std::numeric_limits::max() / size)
 return nullptr;
 auto n = nmemb * size;
diff --git a/modules/tests/Makefile b/modules/tests/Makefile
index 0ac3d63..9bb0180 100644
--- a/modules/tests/Makefile
+++ b/modules/tests/Makefile
@@ -86,7 +86,7 @@ tests := tst-pthread.so misc-ramdisk.so
tst-vblk.so tst-bsd-evh.so \
tst-pthread-setcancelstate.so tst-syscall.so tst-pin.so
tst-run.so \
tst-ifaddrs.so tst-pthread-affinity-inherit.so
tst-sem-timed-wait.so \
tst-ttyname.so tst-pthread-barrier.so tst-feexcept.so
tst-math.so \
-   tst-sigaltstack.so tst-fread.so
+   tst-sigaltstack.so tst-fread.so tst-calloc.so

 #  libstatic-thread-variable.so tst-static-thread-variable.so \

diff --git a/tests/tst-calloc.cc b/tests/tst-calloc.cc
new file mode 100644
index 000..a66bb51
--- /dev/null
+++ b/tests/tst-calloc.cc
@@ -0,0 +1,99 @@
+/*
+* Copyright (C) 2017 XLAB d.o.o.
+*
+* This work is open source software, licensed under the terms of the
+* BSD license as described in the LICENSE file in the top-level
directory.
+*/
+// To compile on Linux, use: g++ -g -pthread -std=c++11
tests/tst-calloc.cc
+
+#include 
+#include 
+
+// This test assures that the calloc and memset functions works
as expected in the various
+// situations it can be used at.
+void pass_if(const char *got, const char *expected, int size,
const char* msg)
+{
+if (memcmp(expected, got, size)) {
+std::cerr << "FAIL: " << msg << "\n";
+std::cerr << "ERROR: got ";
+std::cerr << got;
+std::cerr << " but expected ";
+std::cerr << expected;
+std::cerr << "\n";
+exit(1);
+}
+else {
+std::cerr << "PASS: " << msg << "\n";
+}
+}
+
+int main()
+{
+char *buf1;
+const char ref0[11] = "\0";
+const char refX[11] = "XX";
+int len1, len2;
+
+// calloc
+len1 = 1;
+len2 = 10;
+buf1 = (char*)calloc(len1, len2);
+pass_if(buf1, ref0, len1 * len2, "calloc(1, 10)");
+memset(buf1, 'X', len1 * len2);


I didn't understand why you bother with the memset() in all these 
tests, right before you free the buffer.

How is the memset stuff even related to this calloc() test?
I included memset into test 'by accident'. After crash, I got useless 
backtrace in gdb, so I really didn't know what was wrong. After pinning 
it to calloc, I started looking into functions calloc calls - malloc and 
memset. Only later I notice x / size, with size==0. You are right that 
memset testing doesn't belong to this patch. I will remove it.


Thank you for comments
Justin



+free(buf1);
+
+len1 = 2;
+len2 = 5;
+buf1 = (char*)calloc(len1, len2);
+pass_if(buf1, ref0, len1 * len2, "calloc(2, 5)");
+memset(buf1, 'X', len1 * len2);
+free(buf1);
+
+len1 = 1;
+len2 = 0;
+buf1 = (char*)calloc(len1, len2);
+pass_if(buf1, ref0, len1 * len2, "calloc(1, 0)");
+//memset(buf1, 'X', len1 * len2);
+free(buf1);
+
+len1 = 0

[PATCH v2] calloc: allow 0-sized allocation

[Justin Cinkelj]

The calloc(nmemb, size=0) failed due to division by 0 in
"std::numeric_limits::max() / size".
Fix this by checking for 0 value before division.

Included is also a small test case for calloc.

Signed-off-by: Justin Cinkelj <justin.cink...@xlab.si>
---
 core/mempool.cc|  2 ++
 modules/tests/Makefile |  2 +-
 tests/tst-calloc.cc| 67 ++
 3 files changed, 70 insertions(+), 1 deletion(-)
 create mode 100644 tests/tst-calloc.cc

diff --git a/core/mempool.cc b/core/mempool.cc
index 0b1e69e..569a0ac 100644
--- a/core/mempool.cc
+++ b/core/mempool.cc
@@ -1564,6 +1564,8 @@ static inline void* std_malloc(size_t size, size_t 
alignment)
 
 void* calloc(size_t nmemb, size_t size)
 {
+if (nmemb == 0 || size == 0)
+return malloc(0);
 if (nmemb > std::numeric_limits::max() / size)
 return nullptr;
 auto n = nmemb * size;
diff --git a/modules/tests/Makefile b/modules/tests/Makefile
index 0ac3d63..9bb0180 100644
--- a/modules/tests/Makefile
+++ b/modules/tests/Makefile
@@ -86,7 +86,7 @@ tests := tst-pthread.so misc-ramdisk.so tst-vblk.so 
tst-bsd-evh.so \
tst-pthread-setcancelstate.so tst-syscall.so tst-pin.so tst-run.so \
tst-ifaddrs.so tst-pthread-affinity-inherit.so tst-sem-timed-wait.so \
tst-ttyname.so tst-pthread-barrier.so tst-feexcept.so tst-math.so \
-   tst-sigaltstack.so tst-fread.so
+   tst-sigaltstack.so tst-fread.so tst-calloc.so
 
 #  libstatic-thread-variable.so tst-static-thread-variable.so \
 
diff --git a/tests/tst-calloc.cc b/tests/tst-calloc.cc
new file mode 100644
index 000..a0cb554
--- /dev/null
+++ b/tests/tst-calloc.cc
@@ -0,0 +1,67 @@
+/*
+* Copyright (C) 2017 XLAB d.o.o.
+*
+* This work is open source software, licensed under the terms of the
+* BSD license as described in the LICENSE file in the top-level directory.
+*/
+// To compile on Linux, use: g++ -g -pthread -std=c++11 tests/tst-calloc.cc
+
+#include 
+#include 
+
+// This test assures that the calloc function works as expected in the various
+// situations it can be used at.
+void pass_if(const char *got, const char *expected, int size, const char* msg)
+{
+if (memcmp(expected, got, size)) {
+std::cerr << "FAIL: " << msg << "\n";
+std::cerr << "ERROR: got ";
+std::cerr << got;
+std::cerr << " but expected ";
+std::cerr << expected;
+std::cerr << "\n";
+exit(1);
+}
+else {
+std::cerr << "PASS: " << msg << "\n";
+}
+}
+
+int main()
+{
+char *buf1;
+const char ref0[11] = "\0";
+const char refX[11] = "XX";
+int len1, len2;
+
+len1 = 1;
+len2 = 10;
+buf1 = (char*)calloc(len1, len2);
+pass_if(buf1, ref0, len1 * len2, "calloc(1, 10)");
+memset(buf1, 'X', len1 * len2);
+free(buf1);
+
+len1 = 2;
+len2 = 5;
+buf1 = (char*)calloc(len1, len2);
+pass_if(buf1, ref0, len1 * len2, "calloc(2, 5)");
+memset(buf1, 'X', len1 * len2);
+free(buf1);
+
+len1 = 1;
+len2 = 0;
+buf1 = (char*)calloc(len1, len2);
+pass_if(buf1, ref0, len1 * len2, "calloc(1, 0)");
+//memset(buf1, 'X', len1 * len2);
+free(buf1);
+
+len1 = 0;
+len2 = 1;
+buf1 = (char*)calloc(len1, len2);
+pass_if(buf1, ref0, len1 * len2, "calloc(0, 1)");
+memset(buf1, 'X', len1 * len2);
+free(buf1);
+
+std::cerr << "PASSED\n";
+return 0;
+}
-- 
2.9.3

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.