date:20180813

getrandom/sys_getrandom

2018-08-13 Thread Waldek Kozaczuk

I managed to get Python 3.5 working on OSv but reporting missing 318 
syscall. With version 3.6 and above Python is trying to invoke getrandom 
which obviously fails due to missing symbol.

How difficult would it be implement this function?

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH] Improved python27

2018-08-13 Thread Waldemar Kozaczuk

This patch renames python27 to python2x and improves it
by discovering Python home directory by reading sys.* variables.

Signed-off-by: Waldemar Kozaczuk 
---
 python27/README |  8 
 {python27 => python2x}/GET  | 17 ++---
 {python27 => python2x}/Makefile |  0
 python2x/README | 13 +
 {python27 => python2x}/python.c |  0
 {python27 => python2x}/usr.manifest |  0
 6 files changed, 23 insertions(+), 15 deletions(-)
 delete mode 100644 python27/README
 rename {python27 => python2x}/GET (67%)
 rename {python27 => python2x}/Makefile (100%)
 create mode 100644 python2x/README
 rename {python27 => python2x}/python.c (100%)
 rename {python27 => python2x}/usr.manifest (100%)

diff --git a/python27/README b/python27/README
deleted file mode 100644
index 3569e47..000
--- a/python27/README
+++ /dev/null
@@ -1,8 +0,0 @@
-This app builds an image containing the already-compiled Python shared
-object (libpython2.7.so) installed on the build macine, and some of
-the required libraries from the build machine.
-
-Example usage:
-
-./scripts/run.py -e "/python"
-./scripts/run.py -e "/python -c \"aa={1:22,3:44}; print aa; print 'asdf'\""
diff --git a/python27/GET b/python2x/GET
similarity index 67%
rename from python27/GET
rename to python2x/GET
index 05774df..8942577 100755
--- a/python27/GET
+++ b/python2x/GET
@@ -1,10 +1,14 @@
 #!/usr/bin/env bash
 set -e
 
-VERSION=2.7
 BASEDIR=$PWD
 ROOTFS=$BASEDIR/ROOTFS
 
+PYTHON_PREFIX_DIR=`python -c 'import sys; print(sys.prefix)'`
+PYTHON_MAJOR_VERSION=`python -c 'import sys; print(sys.version_info.major)'`
+PYTHON_MINOR_VERSION=`python -c 'import sys; print(sys.version_info.minor)'`
+PYTHON_VERSION="${PYTHON_MAJOR_VERSION}.${PYTHON_MINOR_VERSION}"
+
 install_shlibs() {
 SHLIBS=""
 SHLIBS+=" $ROOTFS/python.so "
@@ -17,23 +21,22 @@ SHLIBS_COUNT="`echo \"$SHLIBS\" | wc -l`"
 ldd $SHLIBS | grep -Po '(?<=> )/[^ ]+' | sort | uniq | grep -Pv 
'lib(c|gcc|dl|m|util|rt|pthread|stdc\+\+|selinux|krb5|gssapi_krb5)\.so' | xargs 
-I {} install  {} $ROOTFS/usr/lib
 # ROOTFS/lib/python2.7/config/libpython2.7.so is a symlink to 
../../libpython2.7.so,
 # so create a valid destination to avoid ldd error due to dangling symlink.
-(cd $ROOTFS/lib && ln -sf ../usr/lib/libpython$VERSION.so.1.0 
libpython$VERSION.so)
+(cd $ROOTFS/lib && ln -sf ../usr/lib/libpython${PYTHON_VERSION}.so.1.0 
libpython${PYTHON_VERSION}.so)
 echo "$SHLIBS_COUNT"
 }
 
 main() {
 mkdir -p build/
-gcc -o build/python.so python.c -fPIC -shared -lpython${VERSION}
+gcc -o build/python.so python.c -fPIC -shared -lpython${PYTHON_VERSION}
 
 rm -rf "$ROOTFS"
 mkdir -p "$ROOTFS/usr/lib"
-mkdir -p "$ROOTFS/lib/python$VERSION"
+mkdir -p "$ROOTFS/lib/python${PYTHON_VERSION}"
 
 cp build/python.so "$ROOTFS"
 install_shlibs
-# TODO /lib64/python2.7/ should not be hardcoded?
-PY_LIB1=/lib64/python$VERSION/
-rsync -a $PY_LIB1 $ROOTFS/lib/python$VERSION/ --exclude test --exclude 
unittest \
+PY_LIB1="${PYTHON_PREFIX_DIR}/lib/python${PYTHON_VERSION}/"
+rsync -a $PY_LIB1 $ROOTFS/lib/python${PYTHON_VERSION}/ --safe-links --exclude 
test --exclude unittest \
 --exclude '*.pyc' --exclude '*.pyo' --exclude '*.egg-info'
 
 SHLIBS_COUNT4=`install_shlibs`
diff --git a/python27/Makefile b/python2x/Makefile
similarity index 100%
rename from python27/Makefile
rename to python2x/Makefile
diff --git a/python2x/README b/python2x/README
new file mode 100644
index 000..217a00b
--- /dev/null
+++ b/python2x/README
@@ -0,0 +1,13 @@
+This app builds an image containing the already-compiled Python shared
+object for Python 2.x (libpython2.7.so for example) installed on the build 
macine,
+and some of the required libraries from the build machine.
+
+The python home directory is assumed to be installed under
+/lib/python./
+where sys is Python package this script reads the sys.* values from.
+
+Example usage:
+
+./scripts/run.py -e "/python"
+./scripts/run.py -e "/python -c \"aa={1:22,3:44}; print aa; print 'asdf'\""
+./scripts/run.py --api -e "/python -m SimpleHTTPServer 8000"
diff --git a/python27/python.c b/python2x/python.c
similarity index 100%
rename from python27/python.c
rename to python2x/python.c
diff --git a/python27/usr.manifest b/python2x/usr.manifest
similarity index 100%
rename from python27/usr.manifest
rename to python2x/usr.manifest
-- 
2.17.1

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

RE: [PATCH 08/16] bsd: OSv API support for IPv6

2018-08-13 Thread Myers, Charles

It was left as a FIXME because I wasn’t sure how to best address this issue.

AF_INET is just lucky because it is the same for Linux and FreeBSD (2).
AF_INET6 is 28 in FreeBSD and 10 on Linux.

AF_INET6 could be changed to 10 in FreeBSD code to avoid issues with IPv6 too.
but if you want to keep FreeBSD IDs unmodified then what you suggested makes 
sense.

Currently when I include linux.h required for linux_socket.h it defines 
ifr_name as

#define ifr_nameifr_ifrn.ifrn_name  /* Interface name */

which causes issues because ifr_name is used to access the freebsd struct.

$ make
Building into build/release.x64
  GEN gen/include/osv/version.h
  CXX bsd/porting/networking.cc
In file included from bsd/porting/networking.cc:30:0:
bsd/porting/networking.cc: In function ‘int 
osv::if_set_mtu(std::__cxx11::string, u16)’:
./bsd/sys/compat/linux/linux.h:147:18: error: ‘struct bsd_ifreq’ has no member 
named ‘ifr_ifrn’
#define ifr_name ifr_ifrn.ifrn_name /* Interface name */
  ^
bsd/porting/networking.cc:63:19: note: in expansion of macro ‘ifr_name’
 strlcpy(ifreq.ifr_name, if_name.c_str(), IFNAMSIZ);


I will need to remove the ifr_name macro in linux.h to fix this.

diff --git a/bsd/sys/compat/linux/linux.h b/bsd/sys/compat/linux/linux.h
index b693d81..1300f15 100644
--- a/bsd/sys/compat/linux/linux.h
+++ b/bsd/sys/compat/linux/linux.h
@@ -124,9 +124,7 @@ struct l_ifmap {
#defineLINUX_IFNAMSIZ  16

struct l_ifreq {
-   union {
-   charifrn_name[LINUX_IFNAMSIZ];
-   } ifr_ifrn;
+   charifr_name[LINUX_IFNAMSIZ];

union {
struct l_sockaddr   ifru_addr;
@@ -144,7 +142,6 @@ struct l_ifreq {
} ifr_ifru;
} __packed;

-#defineifr_nameifr_ifrn.ifrn_name  /* Interface name */
#defineifr_hwaddr  ifr_ifru.ifru_hwaddr/* MAC address */

struct l_ifconf {

Thanks,
-Charles

From: Nadav Har'El 
Sent: Sunday, August 12, 2018 11:40 AM
To: Myers, Charles 
Cc: Osv Dev 
Subject: Re: [PATCH 08/16] bsd: OSv API support for IPv6


On Tue, Aug 7, 2018 at 5:49 AM, Charles Myers 
mailto:charles.my...@spirent.com>> wrote:
Signed-off-by: Charles Myers 
mailto:charles.my...@spirent.com>>
---
 bsd/porting/networking.cc | 178 +-
 bsd/porting/networking.hh |   8 +++
 bsd/porting/route.cc  |  86 ++
 bsd/sys/netinet/in.cc |  10 +++
 bsd/sys/netinet/in.h  |   1 +
 loader.cc |  76 +++-
 6 files changed, 327 insertions(+), 32 deletions(-)

diff --git a/bsd/porting/networking.cc b/bsd/porting/networking.cc
index 016106c..e2767a7 100644
--- a/bsd/porting/networking.cc
+++ b/bsd/porting/networking.cc
@@ -17,6 +17,15 @@
 #include 
 #include 
 #include 
+#ifdef INET6
+#include 
+#include 
+#include 
+#include 
+
+// FIXME: inet_pton() is from musl which uses different AF_INET6
+#define LINUX_AF_INET6 10

We already have this exact #define in bsd/sys/compat/linux/linux_socket.h, can 
we include that instead?

I think you should remove the "FIXME" - this is the correct behavior of 
inet_pton and other User-facing
functions, which take the Linux versions of the ABI - and unfortunately while 
in Linux and BSD AF_INET
is the same, AF_INET6 is not :-(

Instead of a "FIXME" here you can explain at the point you use LINUX_AF_INET6 
below, why you use it.

+#endif // INET6
 #include 
 #include 

@@ -61,6 +70,18 @@ int if_set_mtu(std::string if_name, u16 mtu)

 int start_if(std::string if_name, std::string ip_addr, std::string mask_addr)
 {
+return if_add_addr(if_name, ip_addr, mask_addr);
+}
+
+int stop_if(std::string if_name, std::string ip_addr)
+{
+std::string mask_addr;

If you intend to use an empty string, you can (I think) also just use "" below, 
and don't need to name it (but you can, if you want...).
+
+return if_del_addr(if_name, ip_addr, mask_addr);
+}
+
+int if_add_ipv4_addr(std::string if_name, std::string ip_addr, std::string 
mask_addr)
+{
 int error, success;
 struct bsd_ifreq oldaddr;
 struct in_aliasreq ifra;
@@ -99,7 +120,9 @@ int start_if(std::string if_name, std::string ip_addr, 
std::string mask_addr)
 error = EINVAL;
 goto out;
 }
+mask->sin_family = AF_INET;
 mask->sin_len = sizeof(struct bsd_sockaddr_in);
+
 broadcast->sin_family  = AF_INET;
 broadcast->sin_len = sizeof(struct bsd_sockaddr_in);
 broadcast->sin_addr.s_addr = (addr->sin_addr.s_addr &
@@ -117,7 +140,7 @@ out:
 return (error);
 }

-int stop_if(std::string if_name, std::string ip_addr)
+int if_del_ipv4_addr(std::string if_name, std::string ip_addr)
 {
 int error, success;
 struct in_aliasreq ifra;
@@ -155,6 +178,157 @@ out:
 return (error);
 }

+#ifdef INET6
+
+int if_add_ipv6_addr(std::string if_name, std::string ip_addr, std::string 
netmask)
+{
+int error, success;
+struct in6_ifreq oldaddr;
+struct in6_aliasreq ifra;

[PATCH] Added libc __wcscpy_chk function

2018-08-13 Thread Waldemar Kozaczuk

This patch adds libc __wcscpy_chk needed by Python 3. Please see
http://refspecs.linux-foundation.org/LSB_4.1.0/LSB-Core-generic/LSB-Core-generic/libc---wcscpy-chk-1.html
for details.

Signed-off-by: Waldemar Kozaczuk 
---
 Makefile   |  1 +
 libc/string/__wcscpy_chk.c | 15 +++
 2 files changed, 16 insertions(+)
 create mode 100644 libc/string/__wcscpy_chk.c

diff --git a/Makefile b/Makefile
index add72f9d..5a24ca67 100644
--- a/Makefile
+++ b/Makefile
@@ -1625,6 +1625,7 @@ libc += string/wcscat.o
 musl += string/wcschr.o
 musl += string/wcscmp.o
 libc += string/wcscpy.o
+libc += string/__wcscpy_chk.o
 musl += string/wcscspn.o
 musl += string/wcsdup.o
 musl += string/wcslen.o
diff --git a/libc/string/__wcscpy_chk.c b/libc/string/__wcscpy_chk.c
new file mode 100644
index ..64d0144f
--- /dev/null
+++ b/libc/string/__wcscpy_chk.c
@@ -0,0 +1,15 @@
+/*
+ * Copyright (C) 2018 Waldemar Kozaczuk
+ *
+ * This work is open source software, licensed under the terms of the
+ * BSD license as described in the LICENSE file in the top-level directory.
+ */
+
+#include 
+#include 
+
+wchar_t *__wcscpy_chk(wchar_t *__restrict dest, const wchar_t *__restrict src, 
size_t destlen)
+{
+assert(wcslen(src) + sizeof(L'\0') <= destlen);
+return wcscpy(dest, src);
+}
-- 
2.17.1

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

RE: [PATCH 07/16] bsd: Modifications to shared IPv4/IPv6 code (eth,tcp,udp) for IPv6

2018-08-13 Thread Myers, Charles

The lltable stuff could probably have been split into another commit since it 
isn’t really related to anything else
other than it is in the same file, but it was small/simple enough that I 
thought that wasn’t necessary.

The rest of the change changes are enabling IPv6 stuff in the IPv4 code which 
was disabled/removed when the IPv4 code was ported to OSv.

I just grouped these changes together to save myself some time and reduce 
number of parts in the patch set.

-Charles


From: Nadav Har'El 
Sent: Sunday, August 12, 2018 11:12 AM
To: Myers, Charles 
Cc: Osv Dev 
Subject: Re: [PATCH 07/16] bsd: Modifications to shared IPv4/IPv6 code 
(eth,tcp,udp) for IPv6

The patch looks reasonable to me, but there are some things that I didn't 
understand how they fit in.
For example, how do the changes bsd/sys/netinet/tcp_reass.cc related to some 
function for iterating link-level sockets?


--
Nadav Har'El
n...@scylladb.com

On Tue, Aug 7, 2018 at 5:49 AM, Charles Myers 
mailto:charles.my...@spirent.com>> wrote:
lltable_foreach(), lltable_foreach_lle() added to support NETLINK sockets

Signed-off-by: Charles Myers 
mailto:charles.my...@spirent.com>>
---
 bsd/sys/net/if_ethersubr.cc |  5 ++---
 bsd/sys/net/if_llatbl.cc| 47 ++---
 bsd/sys/net/if_llatbl.h | 13 
 bsd/sys/netinet/tcp_lro.cc  |  4 ++--
 bsd/sys/netinet/tcp_lro.h   | 15 +++--
 bsd/sys/netinet/tcp_reass.cc|  4 ++--
 bsd/sys/netinet/tcp_subr.cc |  6 --
 bsd/sys/netinet/tcp_syncache.cc |  3 +--
 bsd/sys/netinet/udp_usrreq.cc   |  2 +-
 9 files changed, 78 insertions(+), 21 deletions(-)

diff --git a/bsd/sys/net/if_ethersubr.cc b/bsd/sys/net/if_ethersubr.cc
index 0443c83..d2c4fe9 100644
--- a/bsd/sys/net/if_ethersubr.cc
+++ b/bsd/sys/net/if_ethersubr.cc
@@ -1140,8 +1140,7 @@ ether_resolvemulti(struct ifnet *ifp, struct bsd_sockaddr 
**llsa,
}
if (!IN6_IS_ADDR_MULTICAST(&sin6->sin6_addr))
return EADDRNOTAVAIL;
-   sdl = malloc(sizeof *sdl, M_IFMADDR,
-  M_NOWAIT|M_ZERO);
+   sdl = (struct bsd_sockaddr_dl *) calloc(1, sizeof *sdl);
if (sdl == NULL)
return (ENOMEM);
sdl->sdl_len = sizeof *sdl;
@@ -1149,7 +1148,7 @@ ether_resolvemulti(struct ifnet *ifp, struct bsd_sockaddr 
**llsa,
sdl->sdl_index = ifp->if_index;
sdl->sdl_type = IFT_ETHER;
sdl->sdl_alen = ETHER_ADDR_LEN;
-   e_addr = LLADDR(sdl);
+   e_addr = (u_char*)LLADDR(sdl);
ETHER_MAP_IPV6_MULTICAST(&sin6->sin6_addr, e_addr);
*llsa = (struct bsd_sockaddr *)sdl;
return 0;
diff --git a/bsd/sys/net/if_llatbl.cc b/bsd/sys/net/if_llatbl.cc
index 869c552..1bc7ca2 100644
--- a/bsd/sys/net/if_llatbl.cc
+++ b/bsd/sys/net/if_llatbl.cc
@@ -40,9 +40,9 @@
 #include 
 #include 
 #include 
-#if 0
-#include 
-#include 
+#ifdef INET6
+#include 
+#include 
 #endif

 MALLOC_DEFINE(M_LLTABLE, "lltable", "link level address tables");
@@ -497,3 +497,44 @@ DB_SHOW_ALL_COMMAND(lltables, db_show_all_lltables)
}
 }
 #endif
+
+/*
+ * Iterate over all lltables
+ */
+int lltable_foreach(int (*func)(struct lltable *llt, void *cbdata), void 
*cbdata)
+{
+   struct lltable *llt;
+   int error = 0;
+
+   LLTABLE_RLOCK();
+   SLIST_FOREACH(llt, &V_lltables, llt_link) {
+   if ((error = func(llt, cbdata)) != 0)
+   break;
+   }
+   LLTABLE_RUNLOCK();
+
+   return error;
+}
+
+/*
+ * Iterate over all llentries in the lltable
+ */
+int lltable_foreach_lle(struct lltable *llt, int (*func)(struct lltable *llt, 
struct llentry *lle, void *cbdata), void *cbdata)
+{
+   struct llentry *lle;
+   int i;
+   int error = 0;
+
+   for (i = 0; i < LLTBL_HASHTBL_SIZE; i++) {
+   LIST_FOREACH(lle, &llt->lle_head[i], lle_next) {
+   /* skip deleted entries */
+   if ((lle->la_flags & LLE_DELETED) == LLE_DELETED)
+   continue;
+   if ((error = func(llt, lle, cbdata)) != 0)
+   break;
+   }
+   }
+
+   return error;
+}
+
diff --git a/bsd/sys/net/if_llatbl.h b/bsd/sys/net/if_llatbl.h
index 1cac880..6e0985e 100644
--- a/bsd/sys/net/if_llatbl.h
+++ b/bsd/sys/net/if_llatbl.h
@@ -197,6 +197,17 @@ intlltable_sysctl_dumparp(int, struct 
sysctl_req *);
 size_t llentry_free(struct llentry *);
 struct llentry  *llentry_alloc(struct ifnet *, struct lltable *,
struct bsd_sockaddr_storage *);
+
+/*
+ * Iterate over all lltables
+ */
+int lltable_foreach(int (*func)(struct lltable *llt, void *cbdata), void 
*cbdata);
+
+/*
+ * Iterate over all llentries in the lltable
+ */
+int lltable_foreach_lle(struct lltab

Re: __wcscpy_chk and Python 3

2018-08-13 Thread Nadav Har'El

On Tue, Aug 14, 2018 at 12:34 AM, Waldek Kozaczuk 
wrote:

> I managed to get Python 3.5 working on OSv (I only tested simple 'Hello
> world!' and httpserver example). However to get it working I had to add
> missing __wcscpy_chk to libc.
>
> According to http://refspecs.linux-foundation.org/LSB_4.1.0/LSB-
> Core-generic/LSB-Core-generic/libc---wcscpy-chk-1.html it should check
> for buffer overflow condition and I am not sure this implementation would
> suffice:
>
> libc/string/__wcscpy_chk.c:
>
> #include 
> #include 
>
> extern wchar_t *wcscpy(wchar_t *__restrict d, const wchar_t *__restrict s);
> extern size_t wcslen(const wchar_t *s);
>

These should come , you shouldn't need to have to specify them
here...


>
> wchar_t *__wcscpy_chk(wchar_t *__restrict dest, const wchar_t *__restrict
> src, size_t destlen)
> {
> assert(wcslen(src) + sizeof(L'\0') <= destlen);
> return wcscpy(dest, src);
> }
>

Looks correct to me. But I never even heard of this function until a minute
ago ;-)


> What do you think?
>
> --
> You received this message because you are subscribed to the Google Groups
> "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to osv-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: __wcscpy_chk and Python 3

2018-08-13 Thread Geraldo Netto

Hello Waldek/Friends! :)

On Mon, 13 Aug 2018 at 18:34, Waldek Kozaczuk  wrote:

> I managed to get Python 3.5 working on OSv (I only tested simple 'Hello
> world!' and httpserver example). However to get it working I had to add
> missing __wcscpy_chk to libc.
>
> According to
> http://refspecs.linux-foundation.org/LSB_4.1.0/LSB-Core-generic/LSB-Core-generic/libc---wcscpy-chk-1.html
> it should check for buffer overflow condition and I am not sure this
> implementation would suffice:
>
> libc/string/__wcscpy_chk.c:
>
> #include 
> #include 
>
> extern wchar_t *wcscpy(wchar_t *__restrict d, const wchar_t *__restrict s);
> extern size_t wcslen(const wchar_t *s);
>
> wchar_t *__wcscpy_chk(wchar_t *__restrict dest, const wchar_t *__restrict
> src, size_t destlen)
> {
> assert(wcslen(src) + sizeof(L'\0') <= destlen);
> return wcscpy(dest, src);
> }
>
> What do you think?
>

If you allow me to suggest, just add a TODO/XXX mark and push it upstream
I'd rather prefer an ugly working car than a super cool futuristic car that
is on paper :)
In any case, time will tell us if it's enough or not
Ghost readers are welcome'd to contribute too!!! :P


Kind Regards,

Geraldo Netto
Sapere Aude => Non dvcor, dvco
http://exdev.sf.net/

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

__wcscpy_chk and Python 3

2018-08-13 Thread Waldek Kozaczuk

I managed to get Python 3.5 working on OSv (I only tested simple 'Hello 
world!' and httpserver example). However to get it working I had to add 
missing __wcscpy_chk to libc.

According 
to 
http://refspecs.linux-foundation.org/LSB_4.1.0/LSB-Core-generic/LSB-Core-generic/libc---wcscpy-chk-1.html
 
it should check for buffer overflow condition and I am not sure this 
implementation would suffice:

libc/string/__wcscpy_chk.c:

#include 
#include 

extern wchar_t *wcscpy(wchar_t *__restrict d, const wchar_t *__restrict s);
extern size_t wcslen(const wchar_t *s);

wchar_t *__wcscpy_chk(wchar_t *__restrict dest, const wchar_t *__restrict 
src, size_t destlen)
{
assert(wcslen(src) + sizeof(L'\0') <= destlen);
return wcscpy(dest, src);
}

What do you think?

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: change redis appliance to cluster mode

2018-08-13 Thread Waldek Kozaczuk

Yeah in essence capstan will work with anything that allows downloading 
files over HTTP/S (here is mikelangelo capstan repo - 
https://mikelangelo-capstan.s3.amazonaws.com/). The only caveat is that 
capstan expects a listing of files in S3 format which is xml. It should not 
be to difficult to change capstan to support other formats. Another 
workaround for non-S3 repo is to handcraft a XML listing and make 
/index.html return it. 

Currently the newest artifacts are published as OSv release artifacts (see 
- https://github.com/cloudius-systems/osv/releases/tag/v0.51.0 - for *.mpm 
and *.yaml files) and for each release I am planning to upload new kernel 
and bootstrap package. One can use following type of script 
-https://raw.githubusercontent.com/wkozaczuk/docker_osv_runner/master/scripts/download_packages.sh
 
- to download all packages into local capstan repo under ~/.capstan. But 
yeah it would be nice to have proper official repo for the app packages 
(java, node, python, redis, etc).

As far as Kubernetes is concerned there are at least two projects that 
support deploying and managing VM instances 
- https://github.com/Mirantis/virtlet 
and https://github.com/kubevirt/kubevirt. Mikelangelo team actually managed 
to run OSv on the first one 
-https://www.mikelangelo-project.eu/2017/05/the-microservice-demo-application-running-inside-osv-unikernels-on-kubernetes/

Waldek

On Friday, August 10, 2018 at 2:02:31 PM UTC-4, Russ Tremain wrote:
>
> re: capstan hosting infrastructure: 
>
> my knowledge is a bit rusty, but I believe that Casptan has the ability to 
> create a repository, so it would be a matter of "renting" space on a public 
> cloud and providing a method of access for posting, plus a way to browse 
> the repository to find images of interest. 
>
> that way we could build up a supply of community images and get things 
> rolling... 
>
> as it is now (or used to be), only the base osv images are posted, and you 
> have to build everything starting from those. 
>
> As an example, I built a osv+perl image.  Then I added cado (my 
> codegen/templating language).  so now we have three images that need 
> posting: osv_base, osv_base+perl, osv_base+perl+cado. 
>
> if I wanted to build a cado service, then I could then start with the 
> third image. 
>
> similar for any other language env...  or web-services platform.  think of 
> what it would take to build a complex web-service api composed of many 
> unikernels, for example. 
>
> then of course you need an orchestration tool to manage your growing 
> stable of unikernel services... like a rewrite of kubernetes designed to 
> run unikernels instead of (or in addition to) docker images. 
>
> I think there is a demand for unikernels in the microservices space, but 
> I'm no market expert.. 
>
> cheers. 
> /r 
>
> P.S.  many apologies for hijacking your thread!  we had a thread about 
> this a couple of years back which didn't go anywhere - search for "Casptan" 
> in the list. 
>
> P.P.S: there is an interesting research project funded by VMware that 
> provides the ability to manage vm's like containers: 
> https://github.com/vmware/vic 
>
>
> At 6:57 AM -0700 8/10/18, adamr...@gmail.com  wrote: 
> >Russ - I'd be interested in something like that. What kind of 
> infrastructure is needed to do the hosting like you suggest? 
> > 
> >nzois - Yes, I only plan to cluster for high availability of a mem cache 
> layer. I don't need replication as the app would just fetch updated 
> database data for the cache layer if it was missing. My concern is more 
> about just only having 1 active redis instance (currently) and not some 
> sort of active backup in case that server goes down. When I was reading 
> about the clustering on redis.io I was more interested in the Redis 
> Sentinel. 
> > 
> >Are you saying I can change my Redis OSv ESXi deployment to support the 
> Sentinel HA cluster without rebuilding the OSv image?  I was under the 
> impression from Waldek that any config change to OSv requires rebuilding 
> the image. 
> > 
> >-- 
> >You received this message because you are subscribed to the Google Groups 
> "OSv Development" group. 
> >To unsubscribe from this group and stop receiving emails from it, send an 
> email to osv-dev+u...@googlegroups.com . 
> >For more options, visit https://groups.google.com/d/optout. 
>
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH 00/16] OSv IPv6 support

2018-08-13 Thread Nadav Har'El

On Tue, Aug 7, 2018 at 5:49 AM, Charles Myers 
wrote:

> This patch series adds IPv6 support to OSv.
>
> The first 5 parts of the series should have no impact on existing
> OSv code.  The rest of the patches starting from the net channel
> changes do impact existing code.
>

Thanks. Very impressive work!

I tried to review these patches, and everything looks good - I just wrote
you a few fairly
minor comments on the individual patches.

However, I have to admit that due to the sheer breadth of these patches,
and my
lack of familiarity with some of the code involved, I can't really be sure
that no bugs
will creap in. So I'm happy that you added more tests, and I understand
that you
guys are testing this code - in both IPv4 (right?) and IPv6 - in
production, so if bugs crept
in, we'll probably find them soon, so I'm fine with committing these
patches now.

So please look at my comments, fix whatever you decide to fixed based on
these
comments, and provide a new version for me to commit. You can either post
the
series to the mailing list, or just provide me a link to a github branch to
merge.


>
> All parts should compile and test without issues, but IPv6 is only
> enabled with the last patch in the series.
>
> The IPv6 support is optional and can be enabled/disabled using the
> conf-INET6 option in the conf/base.mk file.
>
> Currently there are 2 known remaining issues left which are due to
> limitations of the FreeBSD 9.0 IPv6 code which this code is base off of.
>
> 1. The NDP code is not thread safe
>The FreeBSD 9.0 IPv6 code is not MPSAFE and expects callouts to run
>with the giant lock and uses splnet() to ensure thread safety.
>Unfortunatley OSv does not support these mechanisms and it is pretty
>tricky to modify the FreeBSD 9.0 IPv6 code to make it thread safe
>without them due to multiple locks required in the NDP code.
>
> 2. IPv6 MAC cache entries can expire causing extra neighbor solicits
>This does not occur when using TCP sockets which keep the cache
>entry active using ND6_HINT() but can occur for UDP or raw IP.
>

Can you please open issues in the OSv bug tracker with these explanations
of these remaining problems?


>
> These issues appear to be fixed in FreeBSD 11, however back porting these
> fixes is pretty involved and impacts a lot of the existing FreeBSD
> code not only IPv6.
>
> IPv6 Stateless Autoconfiguration is enabled, but has not been tested yet.
>
> DHCPv6 is not supported yet.
>
> Static IPv6 addresses may be configured using command line arguments:
>
> ./scripts/run.py  --execute \
> "--ip=eth0,2001:1:1::501,64 --defaultgw=2001:1:1::1 \
>  /tools/iperf -s -V -B ::"
>
> or using cloud-init network version 1 yaml which is also added with
> this patch series:
>
> #cloud-config
> network:
> version: 1
> config:
> - type: physical
>   name: eth0
>   subnets:
>   - type: static
> address: 2001:1:1::501/64
>
> Charles Myers (16):
>   bsd: Added unmodified IPv6 .h files from FreeBSD 9.0
>   bsd: Added unmodified IPv6 .c files from FreeBSD 9.0
>   bsd: Renamed IPv6 files from *.c to *.cc
>   bsd: Modify FreeBSD IPv6 .h files for OSv
>   bsd: Modify FreeBSD IPv6 .cc files for OSv
>   bsd: Added IPv6 net channel support
>   bsd: Modifications to shared IPv4/IPv6 code (eth,tcp,udp) for IPv6
>   bsd: OSv API support for IPv6
>   bsd: Added partial Linux NETLINK socket support
>   libc: Fix if_indextoname(), if_nametoindex()
>   libc: Add IPv6 support to getifaddrs(), if_nameindex() using NETLINK
> socket
>   bsd: Fix SIOCSIFNAME when using linux compatiblity socket
>   bsd: linux socket support for IPv6, IP_PKTINFO, IPV6_PKTINFO,
> SCM_TIMESTAMP
>   cloud-init: Added support for Network v1 and ConfigDrive data source
>   bsd: Added unit tests for IPv6 TCP, IP_PKTINFO, IPV6_PKTINFO,
> SCM_TIMESTAMP
>   bsd: Added conf-INET6 option to enable IPv6 support
>
>  Makefile  |   29 +-
>  bsd/net.cc|   28 +-
>  bsd/porting/netport.h |4 +
>  bsd/porting/networking.cc |  178 +-
>  bsd/porting/networking.hh |8 +
>  bsd/porting/route.cc  |   86 +-
>  bsd/sys/compat/linux/linux.h  |   12 +-
>  bsd/sys/compat/linux/linux_ioctl.cc   |   37 +-
>  bsd/sys/compat/linux/linux_netlink.cc |  904 +
>  bsd/sys/compat/linux/linux_netlink.h  |  175 ++
>  bsd/sys/compat/linux/linux_socket.cc  |  731 ++--
>  bsd/sys/compat/linux/linux_socket.h   |  102 +-
>  bsd/sys/dev/xen/netfront/netfront.cc  |4 +
>  bsd/sys/kern/sys_socket.cc|1 +
>  bsd/sys/kern/uipc_sockbuf.cc  |1 +
>  bsd/sys/kern/uipc_socket.cc   |1 +
>  bsd/sys/kern/uipc_syscalls.cc |   45 +-
>  bsd/sys/kern/uipc_syscalls_wrap.cc|6 +-
>  bsd/sys/net/if.cc |   15 +
>  bsd/sys/net/if_ethersubr.cc   |5 +-
>  bsd/sys/net/if_llatbl.cc  |

Re: [PATCH 16/16] bsd: Added conf-INET6 option to enable IPv6 support

2018-08-13 Thread Nadav Har'El

On Tue, Aug 7, 2018 at 5:49 AM, Charles Myers 
wrote:

> Signed-off-by: Charles Myers 
> ---
>  Makefile | 27 ++-
>  bsd/net.cc   | 23 +++-
>  bsd/porting/netport.h|  4 +++
>  bsd/sys/dev/xen/netfront/netfront.cc |  4 +++
>  conf/base.mk |  5 
>  include/api/netinet/in.h | 52 +++---
> --
>  6 files changed, 64 insertions(+), 51 deletions(-)
>
> diff --git a/Makefile b/Makefile
> index d5b6fda..317e727 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -347,7 +347,7 @@ $(out)/bsd/%.o: INCLUDES += -isystem bsd/
>  # for machine/
>  $(out)/bsd/%.o: INCLUDES += -isystem bsd/$(arch)
>
> -configuration-defines = conf-preempt conf-debug_memory conf-logger_debug
> +configuration-defines = conf-preempt conf-debug_memory conf-logger_debug
> conf-INET6
>
>  configuration = $(foreach cf,$(configuration-defines), \
>-D$(cf:conf-%=CONF_%)=$($(cf)))
> @@ -611,6 +611,31 @@ bsd += bsd/sys/netinet/cc/cc_cubic.o
>  bsd += bsd/sys/netinet/cc/cc_htcp.o
>  bsd += bsd/sys/netinet/cc/cc_newreno.o
>  bsd += bsd/sys/netinet/arpcache.o
> +ifeq ($(conf-INET6), 1)
> +bsd += bsd/sys/netinet6/dest6.o
> +bsd += bsd/sys/netinet6/frag6.o
> +bsd += bsd/sys/netinet6/icmp6.o
> +bsd += bsd/sys/netinet6/in6.o
> +bsd += bsd/sys/netinet6/in6_cksum.o
> +bsd += bsd/sys/netinet6/in6_ifattach.o
> +bsd += bsd/sys/netinet6/in6_mcast.o
> +bsd += bsd/sys/netinet6/in6_pcb.o
> +bsd += bsd/sys/netinet6/in6_proto.o
> +bsd += bsd/sys/netinet6/in6_rmx.o
> +bsd += bsd/sys/netinet6/in6_src.o
> +bsd += bsd/sys/netinet6/ip6_forward.o
> +bsd += bsd/sys/netinet6/ip6_id.o
> +bsd += bsd/sys/netinet6/ip6_input.o
> +bsd += bsd/sys/netinet6/ip6_output.o
> +bsd += bsd/sys/netinet6/mld6.o
> +bsd += bsd/sys/netinet6/nd6.o
> +bsd += bsd/sys/netinet6/nd6_nbr.o
> +bsd += bsd/sys/netinet6/nd6_rtr.o
> +bsd += bsd/sys/netinet6/raw_ip6.o
> +bsd += bsd/sys/netinet6/route6.o
> +bsd += bsd/sys/netinet6/scope6.o
> +bsd += bsd/sys/netinet6/udp6_usrreq.o
> +endif
>  bsd += bsd/sys/xdr/xdr.o
>  bsd += bsd/sys/xdr/xdr_array.o
>  bsd += bsd/sys/xdr/xdr_mem.o
> diff --git a/bsd/net.cc b/bsd/net.cc
> index f548e09..3fe334a 100644
> --- a/bsd/net.cc
> +++ b/bsd/net.cc
> @@ -24,6 +24,11 @@
>  #include 
>  #include 
>  #include 
> +#ifdef INET6
> +#include 
> +#include 
> +#include 
> +#endif
>
>  /* Generation of ip ids */
>  void ip_initid(void);
> @@ -31,6 +36,10 @@ void ip_initid(void);
>  extern "C" {
>  /* AF_INET */
>  extern  struct domain inetdomain;
> +#ifdef INET6
> +/* AF_INET6 */
> +extern  struct domain inet6domain;
> +#endif
>  /* AF_ROUTE */
>  extern  struct domain routedomain;
>  /* AF_NETLINK */
> @@ -55,12 +64,20 @@ void net_init(void)
>  vnet_pfil_init();
>  domaininit(NULL);
>  OSV_DOMAIN_SET(inet);
> +#ifdef INET6
> +OSV_DOMAIN_SET(inet6);
> +#endif
>  OSV_DOMAIN_SET(route);
>  OSV_DOMAIN_SET(netlink);
>  rts_init();
>  route_init();
>  vnet_route_init();
>  netlink_init();
> +#ifdef INET6
> +ip6_init2(NULL);
> +mld_init(NULL);
> +vnet_mld_init(NULL);
> +#endif
>  ipport_tick_init(NULL);
>  arp_init();
>  domainfinalize(NULL);
> @@ -68,9 +85,13 @@ void net_init(void)
>  if_attachdomain(NULL);
>  vnet_loif_init();
>
> +/* Adding IPv4 address before starting the loopback interface
> + * cause the interface to be brought up without IPv6 support.
> + */
> +
>  /* Start the loopback device */
> -osv::start_if("lo0", "127.0.0.1", "255.0.0.0");
>  osv::ifup("lo0");
> +osv::start_if("lo0", "127.0.0.1", "255.0.0.0");
>
>  debug(" - done\n");
>  }
> diff --git a/bsd/porting/netport.h b/bsd/porting/netport.h
> index a433370..c797f25 100644
> --- a/bsd/porting/netport.h
> +++ b/bsd/porting/netport.h
> @@ -163,6 +163,10 @@ extern int tick;
>  #define INET (1)
>  #endif
>
> +#if defined(CONF_INET6) && (CONF_INET6 != 0)
> +#define INET6 (1)
> +#endif
> +
>  #ifdef _KERNEL
>
>  #define panic(...) do { tprintf_e("bsd-panic", __VA_ARGS__); \
> diff --git a/bsd/sys/dev/xen/netfront/netfront.cc
> b/bsd/sys/dev/xen/netfront/netfront.cc
> index c2b40ac..c4b40ba 100644
> --- a/bsd/sys/dev/xen/netfront/netfront.cc
> +++ b/bsd/sys/dev/xen/netfront/netfront.cc
> @@ -61,6 +61,10 @@ __FBSDID("$FreeBSD$");
>  #include 
>  #include 
>  #include 
> +#ifdef INET6
> +#include 
> +#include 
> +#endif
>  #include 
>  #if __FreeBSD_version >= 70
>  #include 
> diff --git a/conf/base.mk b/conf/base.mk
> index dcf98cd..54cd163 100644
> --- a/conf/base.mk
> +++ b/conf/base.mk
> @@ -9,3 +9,8 @@ conf-logger_debug=0
>  # This macro controls the NDEBUG macro that is used to identify the debug
>  # build variant in the code.
>  conf-DEBUG_BUILD=0
> +
> +# Set to 1 to enable IPV6
> +# This macro controls the FreeBSD INET6 macro defined in
> bsd/porting/netport.h
> +conf-INET6=1
> +
> diff --git

Re: [PATCH 15/16] bsd: Added unit tests for IPv6 TCP, IP_PKTINFO, IPV6_PKTINFO, SCM_TIMESTAMP

2018-08-13 Thread Nadav Har'El

Thanks. I am very happy you added these tests :-)

Only some minor comments below.


On Tue, Aug 7, 2018 at 5:49 AM, Charles Myers 
wrote:

> Signed-off-by: Charles Myers 
> ---
>  modules/tests/Makefile|  16 ++-
>  tests/tst-pktinfo.cc  | 245 ++
> 
>  tests/tst-socket-timestamp.cc | 162 
>  tests/tst-tcp-v6.cc   | 240 ++
> +++
>  4 files changed, 641 insertions(+), 22 deletions(-)
>  create mode 100644 tests/tst-pktinfo.cc
>  create mode 100644 tests/tst-socket-timestamp.cc
>
> diff --git a/modules/tests/Makefile b/modules/tests/Makefile
> index cec4feb..60fc1eb 100644
> --- a/modules/tests/Makefile
> +++ b/modules/tests/Makefile
> @@ -22,6 +22,13 @@ makedir = $(call very-quiet, mkdir -p $(dir $@))
>
>  autodepend = -MD -MT $@ -MP
>
> +include $(OSV_BASE)/conf/base.mk
> +
> +configuration-defines = conf-preempt conf-debug_memory conf-logger_debug
> conf-INET6
> +configuration = $(foreach cf,$(configuration-defines), \
> +  -D$(cf:conf-%=CONF_%)=$($(cf)))
> +
> +
>  INCLUDES = -I$(src)/arch/$(ARCH) -I$(src) -I$(src)/include \
> -I$(src)/arch/common -isystem $(src)/include/glibc-compat \
> $(shell $(CXX) -E -xc++ - -v &1 | awk '/^End/ {exit}
> /^ .*c\+\+/ {print "-isystem" $$0}') \
> @@ -29,7 +36,7 @@ INCLUDES = -I$(src)/arch/$(ARCH) -I$(src)
> -I$(src)/include \
> -isystem $(out)/gen/include
>
>  COMMON = $(autodepend) $(INCLUDES) -g -O2 -fPIC -DBOOST_TEST_DYN_LINK \
> -   -U _FORTIFY_SOURCE -D_KERNEL -D__OSV__ -DCONF_debug_memory=0 \
> +   -U _FORTIFY_SOURCE -D_KERNEL -D__OSV__ $(configuration) \
> -Wall -Wno-pointer-arith -Wformat=0 -Wno-format-security
>
>  LIBS =
> @@ -113,8 +120,9 @@ tests := tst-pthread.so misc-ramdisk.so tst-vblk.so
> tst-bsd-evh.so \
> tst-pthread-setcancelstate.so tst-syscall.so tst-pin.so tst-run.so
> \
> tst-ifaddrs.so tst-pthread-affinity-inherit.so
> tst-sem-timed-wait.so \
> tst-ttyname.so tst-pthread-barrier.so tst-feexcept.so tst-math.so \
> -   tst-sigaltstack.so tst-fread.so tst-tcp-cork.so tst-tcp-v6.so \
> -   tst-calloc.so tst-crypt.so
> +   tst-sigaltstack.so tst-fread.so tst-tcp-cork.so \
> +   tst-calloc.so tst-crypt.so \
> +   tst-socket-timestamp.so tst-pktinfo.so
>
>  #  libstatic-thread-variable.so tst-static-thread-variable.so \
>
> @@ -145,7 +153,7 @@ boost-tests := tst-vfs.so tst-libc-locking.so
> misc-fs-stress.so \
> tst-bsd-tcp1-zsndrcv.so tst-async.so tst-rcu-list.so
> tst-tcp-listen.so \
> tst-poll.so tst-bitset-iter.so tst-timer-set.so tst-clock.so \
> tst-rcu-hashtable.so tst-unordered-ring-mpsc.so \
> -   tst-seek.so
> +   tst-seek.so tst-tcp-v6.so
>
>  rofs-only-boost-tests :=
>
> diff --git a/tests/tst-pktinfo.cc b/tests/tst-pktinfo.cc
> new file mode 100644
> index 000..41022ac
> --- /dev/null
> +++ b/tests/tst-pktinfo.cc
> @@ -0,0 +1,245 @@
> +/*
> + * Copyright (C) 2014 Cloudius Systems, Ltd.
>

You can use your own copyright statement (and current year) instead.


> + *
> + * This work is open source software, licensed under the terms of the
> + * BSD license as described in the LICENSE file in the top-level
> directory.
> + */
> +// To compile on Linux, use: g++ -g -pthread -std=c++11 tests/tst-uio.cc
>

Change tst-uio.cc on this command line to tst-pktinfo.cc


> +
> +// This test tests the SO_TIMESTMAP socket option.
>

Is this really what this test tests? Looks more like testing IP_PKTINFO
(I'm not familiar with this option, so I don't know if it tests anything
else).


> +
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +
> +#include 
> +#include 
> +#include 
> +#include 
> +
> +#ifdef __OSV__
> +#include  // Get INET6
> +#else
> +#define INET6
> +#endif
> +
> +
> +// Multiple threads can call expect functions at the same time
> +// so need to protect against concurrent writes to cout.
> +std::mutex test_mutex;
> +
> +static int tests = 0, fails = 0;
> +
> +template
> +bool do_expect(T actual, T expected, const char *actuals, const char
> *expecteds, const char *file, int line)
> +{
> +std::lock_guard lock(test_mutex);
> +
> +++tests;
> +if (actual != expected) {
> +fails++;
> +std::cout << "FAIL: " << file << ":" << line << ": For " <<
> actuals
> +<< " expected " << expecteds << "(" << expected << "),
> saw "
> +<< actual << ".\n";
> +return false;
> +}
> +std::cout << "OK: " << file << ":" << line << ".\n";
> +return true;
> +}
> +template
> +bool do_expectge(T actual, T expected, const char *actuals, const char
> *expecteds, const char *file, int line)
> +{
> +std::lock_guard lock(test_mutex);
> +
> +++tests;
> +if (actual < expected) {
> +fails++;
> +std

Re: [PATCH 14/16] cloud-init: Added support for Network v1 and ConfigDrive data source

2018-08-13 Thread Nadav Har'El

All looks good. Thanks.


--
Nadav Har'El
n...@scylladb.com

On Tue, Aug 7, 2018 at 5:49 AM, Charles Myers 
wrote:

> https://cloudinit.readthedocs.io/en/latest/topics/network-
> config-format-v1.html
>
> Currently only interface naming, static IP, routes and DNS are supported.
>
> Signed-off-by: Charles Myers 
> ---
>  modules/cloud-init/Makefile  |   2 +-
>  modules/cloud-init/main.cc   |  77 +---
>  modules/cloud-init/network-module.cc | 369 ++
> +
>  modules/cloud-init/network-module.hh |  43 
>  4 files changed, 465 insertions(+), 26 deletions(-)
>  create mode 100644 modules/cloud-init/network-module.cc
>  create mode 100644 modules/cloud-init/network-module.hh
>
> diff --git a/modules/cloud-init/Makefile b/modules/cloud-init/Makefile
> index daee8c3..ae45b24 100644
> --- a/modules/cloud-init/Makefile
> +++ b/modules/cloud-init/Makefile
> @@ -13,7 +13,7 @@ INCLUDES += -I$(HTTPSERVER_API_DIR)
>
>  # the build target executable:
>  TARGET = cloud-init
> -CPP_FILES := client.cc cloud-init.cc data-source.cc main.cc template.cc
> cassandra-module.cc json.cc
> +CPP_FILES := client.cc cloud-init.cc data-source.cc main.cc template.cc
> network-module.cc cassandra-module.cc json.cc
>  OBJ_FILES := $(addprefix obj/,$(CPP_FILES:.cc=.o))
>  DEPS := $(OBJ_FILES:.o=.d)
>
> diff --git a/modules/cloud-init/main.cc b/modules/cloud-init/main.cc
> index 5af23f0..a6674ef 100644
> --- a/modules/cloud-init/main.cc
> +++ b/modules/cloud-init/main.cc
> @@ -9,6 +9,7 @@
>  #include 
>  #include 
>  #include "cloud-init.hh"
> +#include "network-module.hh"
>  #include "files-module.hh"
>  #include "server-module.hh"
>  #include "cassandra-module.hh"
> @@ -24,59 +25,84 @@ using namespace std;
>  using namespace init;
>  namespace po = boost::program_options;
>
> -// config_disk() allows to use NoCloud VM configuration method - see
> +// config_disk() allows to use NoCloud and ConfigDrive VM configuration
> method - see
>  // http://cloudinit.readthedocs.io/en/0.7.9/topics/
> datasources/nocloud.html.
> +// http://cloudinit.readthedocs.io/en/0.7.9/topics/
> datasources/configdrive.html
> +//
>  // NoCloud method provides two files with cnfiguration data (/user-data
> and
>  // /meta-data) on a disk. The disk is required to have label "cidata".
>  // It can contain ISO9660 or FAT filesystem.
>  //
> +// ConfigDrive (version 2) method uses an unpartitioned VFAT or ISO9660
> disk
> +// with files.
> +// openstack/
> +//  - 2012-08-10/ or latest/
> +//- meta_data.json
> +//- user_data (not mandatory)
> +//  - content/
> +//-  (referenced content files)
> +//- 0001
> +//- 
> +// ec2
> +//  - latest/
> +//- meta-data.json (not mandatory)
> +//- user-data
> +//
>  // config_disk() checks whether we have a second disk (/dev/vblkX) with
>  // ISO image, and if there is, it copies the configuration file from
> -// /user-data to the given file.
> +// the user user-data file to the given file.
>  // config_disk() returns true if it has successfully read the
> configuration
> -// into the requested file. It triest to get configuratioe from first few
> +// into the requested file. It tries to get configuration from first few
>  // vblk devices, namely vblk1 to vblk10.
>  //
>  // OSv implementation limitations:
>  // The /meta-data file is currently ignored.
>  // Only ISO9660 filesystem is supported.
> -// The mandatory "cidata" volume label is not checked.
> +// The mandatory "cidata" (NoCloud) and "config-2" (ConfigDrive) volume
> labels are not checked.
>  //
>  // Example ISO image can be created by running
>  // cloud-localds cloud-init.img cloud-init.yaml
>  // The cloud-localds command is provided by cloud-utils package (fedora).
>  static bool config_disk(const char* outfile) {
> +const char * userdata_file_paths[] {
> +"/user-data",  // NoCloud
> +"/openstack/latest/user_data", // ConfigDrive OpenStack
> +"/ec2/latest/user-data",   // ConfigDrive EC2
> +};
> +
>  for (int ii=1; ii<=10; ii++) {
>  char disk[20];
> -char srcfile[] = "/user-data";
>  struct stat sb;
> -int ret;
> -int app_ret = -1;
>
>  snprintf(disk, sizeof(disk), "/dev/vblk%d", ii);
> -ret = stat(disk, &sb);
> -if (ret != 0) {
> -continue;
> -}
>
> -std::vector cmd = {"/usr/bin/iso-read.so", "-e",
> srcfile, "-o", outfile, disk};
> -osv::run(cmd[0], cmd, &app_ret);
> -if (app_ret != 0) {
> -debug("cloud-init: warning, %s exited with code %d (%s is not
> ISO image?)\n", cmd[0], app_ret, disk);
> +if (stat(disk, &sb) != 0) {
>  continue;
>  }
> -ret = stat(outfile, &sb);
> -if (ret != 0) {
> -debug("cloud-init: disk %s, stat(%s) failed, errno=%d\n",
> disk, outfile, errno);
> -continue;
> -}
> -if ((sb.st_mode & S_IFMT) != S

Re: [PATCH 12/16] bsd: Fix SIOCSIFNAME when using linux compatiblity socket

2018-08-13 Thread Nadav Har'El

On Tue, Aug 7, 2018 at 5:49 AM, Charles Myers 
wrote:

> Signed-off-by: Charles Myers 
> ---
>  bsd/sys/compat/linux/linux_ioctl.cc | 37 ++
> +++
>  1 file changed, 33 insertions(+), 4 deletions(-)
>
> diff --git a/bsd/sys/compat/linux/linux_ioctl.cc
> b/bsd/sys/compat/linux/linux_ioctl.cc
> index 43a50bb..d49386d 100644
> --- a/bsd/sys/compat/linux/linux_ioctl.cc
> +++ b/bsd/sys/compat/linux/linux_ioctl.cc
> @@ -43,8 +43,8 @@
>  #include 
>  #include 
>
> -#include 
>  #include 
> +#include 
>
>  #include 
>  #include 
> @@ -199,6 +199,17 @@ linux_gifhwaddr(struct ifnet *ifp, struct l_ifreq
> *ifr)
>  return (ENOENT);
>  }
>
> +static int
> +linux_sifname(socket_file *fp, struct l_ifreq *data)
> +{
> +struct bsd_ifreq ifreq;
> +
> +// FreeBSD ifreq uses a pointer to the new name string instead of
> including it in the struct
> +memcpy((char *)&ifreq, data->ifr_name, IFNAMSIZ);
>

Wouldn't it be better to use strncpy instead of memcpy? So that if the
string is shorter than IFNAMSIZ we don't try to read beyond its end?


> +ifreq.ifr_ifru.ifru_data = (caddr_t)(data->ifr_ifru.ifru_newname);
>

Is this the only field in ifreq we need to fill?

+
> +return fp->bsd_ioctl(SIOCSIFNAME, &ifreq);
> +}
>
>  /*
>   * Fix the interface address field in bsd_ifreq. The bsd stack expects a
> @@ -223,6 +234,16 @@ linux_to_bsd_ifreq(struct bsd_ifreq *ifr_p)
>  }
>
>  /*
> + * FreeBSD ifru_index is short but Linux is an int so need to clear extra
> bits.
>

Doesn't bsd_to_linux_ifreq do among other things, also this?
Maybe not, I'm not familiar with the details.

+ */
> +static inline void
> +bsd_to_linux_ifreq_ifindex(struct bsd_ifreq *ifr_p)
> +{
> +void *ptr = &ifr_p->ifr_index;
> +*(int *)(ptr) = ifr_p->ifr_index;
> +}
> +
> +/*
>   * Socket related ioctls
>   */
>
> @@ -241,8 +262,8 @@ linux_ioctl_socket(socket_file *fp, u_long cmd, void
> *data)
>  switch (cmd) {
>  case SIOCSIFADDR:
>  case SIOCSIFNETMASK:
> -case SIOCSIFDSTADDR:
> -case SIOCSIFBRDADDR:
> +case SIOCSIFDSTADDR:
> +case SIOCSIFBRDADDR:
>  if ((ifp = ifunit_ref((char *)data)) == NULL)
>  return (EINVAL);
>  linux_to_bsd_ifreq((struct bsd_ifreq *)data) ;
> @@ -251,10 +272,16 @@ linux_ioctl_socket(socket_file *fp, u_long cmd, void
> *data)
>
>  case SIOCGIFMTU:
>  case SIOCSIFMTU:
> +if ((ifp = ifunit_ref((char *)data)) == NULL)
> +return (EINVAL);
> +error = fp->bsd_ioctl(cmd, data);
> +break;
> +
>  case SIOCGIFINDEX:
>  if ((ifp = ifunit_ref((char *)data)) == NULL)
>  return (EINVAL);
>  error = fp->bsd_ioctl(cmd, data);
> +   bsd_to_linux_ifreq_ifindex((struct bsd_ifreq *)data);
>  break;
>
>  case SIOCGIFADDR:
> @@ -278,7 +305,9 @@ linux_ioctl_socket(socket_file *fp, u_long cmd, void
> *data)
>  break;
>
>  case SIOCSIFNAME:
> -error = ENOIOCTL;
> +if ((ifp = ifunit_ref((char *)data)) == NULL)
> +return (EINVAL);
> +error = linux_sifname(fp, (struct l_ifreq *)data);
>  break;
>
>  case SIOCGIFHWADDR:
> --
> 2.7.4
>
> --
> You received this message because you are subscribed to the Google Groups
> "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to osv-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH 11/16] libc: Add IPv6 support to getifaddrs(), if_nameindex() using NETLINK socket

2018-08-13 Thread Nadav Har'El

On Tue, Aug 7, 2018 at 5:49 AM, Charles Myers 
wrote:

> libc code is from musl library


Which version? It's not the version we have in musl/, right? So which
version?


> with minor change to prevent partial reads if faster than sender.
>

Is this fixing a bug in musl (in which case you should probably report it
to them), or in OSv?


>
> Signed-off-by: Charles Myers 
> ---
>  Makefile|   1 +
>  libc/network/getifaddrs.c   | 392 --
> --
>  libc/network/if_nameindex.c | 136 ++-
>  libc/network/netlink.c  |  66 
>  libc/network/netlink.h  |  94 +++
>  5 files changed, 435 insertions(+), 254 deletions(-)
>  create mode 100644 libc/network/netlink.c
>  create mode 100644 libc/network/netlink.h
>
> diff --git a/Makefile b/Makefile
> index 49e61b8..d5b6fda 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -1387,6 +1387,7 @@ musl += network/getservbyname.o
>  musl += network/getservbyport_r.o
>  musl += network/getservbyport.o
>  libc += network/getifaddrs.o
> +libc += network/netlink.o
>  libc += network/if_nameindex.o
>  musl += network/if_freenameindex.o
>  libc += network/res_init.o
> diff --git a/libc/network/getifaddrs.c b/libc/network/getifaddrs.c
> index 4cdf4d6..3b92599 100644
> --- a/libc/network/getifaddrs.c
> +++ b/libc/network/getifaddrs.c
> @@ -1,252 +1,216 @@
> -/* (C) 2013 John Spencer. released under musl's standard MIT license. */
> -#undef _GNU_SOURCE
> -#define _GNU_SOURCE
> -#include 
> -#include 
> -#include  /* IFNAMSIZ, ifreq, ifconf */
> -#include 
> -#include 
> -#include 
> +//#define _GNU_SOURCE
>  #include 
> -#include  /* inet_pton */
> +#include 
> +#include 
>  #include 
> -#include 
> -#include 
> -#include 
> -
> -typedef union {
> -   struct sockaddr_in6 v6;
> +#include 
> +#include 
> +#include 
> +#include 
> +#include "netlink.h"
> +
> +#define IFADDRS_HASH_SIZE 64
> +
> +/* getifaddrs() reports hardware addresses with PF_PACKET that implies
> + * struct sockaddr_ll.  But e.g. Infiniband socket address length is
> + * longer than sockaddr_ll.ssl_addr[8] can hold. Use this hack struct
> + * to extend ssl_addr - callers should be able to still use it. */
> +struct sockaddr_ll_hack {
> +   unsigned short sll_family, sll_protocol;
> +   int sll_ifindex;
> +   unsigned short sll_hatype;
> +   unsigned char sll_pkttype, sll_halen;
> +   unsigned char sll_addr[24];
> +};
> +
> +union sockany {
> +   struct sockaddr sa;
> +   struct sockaddr_ll_hack ll;
> struct sockaddr_in v4;
> -   struct sockaddr_ll hw;
> -} soa;
> +   struct sockaddr_in6 v6;
> +};
>
> -typedef struct ifaddrs_storage {
> +struct ifaddrs_storage {
> struct ifaddrs ifa;
> -   soa addr;
> -   soa netmask;
> -   soa dst;
> +   struct ifaddrs_storage *hash_next;
> +   union sockany addr, netmask, ifu;
> +   unsigned int index;
> char name[IFNAMSIZ+1];
> -} stor;
> -#define next ifa.ifa_next
> +};
>
> -static stor* list_add(stor** list, stor** head, char* ifname)
> -{
> -   stor* curr = calloc(1, sizeof(stor));
> -   if(curr) {
> -   strcpy(curr->name, ifname);
> -   curr->ifa.ifa_name = curr->name;
> -   if(*head) (*head)->next = (struct ifaddrs*) curr;
> -   *head = curr;
> -   if(!*list) *list = curr;
> -   }
> -   return curr;
> -}
> +struct ifaddrs_ctx {
> +   struct ifaddrs_storage *first;
> +   struct ifaddrs_storage *last;
> +   struct ifaddrs_storage *hash[IFADDRS_HASH_SIZE];
> +};
>
>  void freeifaddrs(struct ifaddrs *ifp)
>  {
> -   stor *head = (stor *) ifp;
> -   while(head) {
> -   void *p = head;
> -   head = (stor *) head->next;
> -   free(p);
> +   struct ifaddrs *n;
> +   while (ifp) {
> +   n = ifp->ifa_next;
> +   free(ifp);
> +   ifp = n;
> }
>  }
>
> -static void ipv6netmask(unsigned prefix_length, struct sockaddr_in6 *sa)
> +static void copy_addr(struct sockaddr **r, int af, union sockany *sa,
> void *addr, size_t addrlen, int ifindex)
>  {
> -   unsigned char* hb = sa->sin6_addr.s6_addr;
> -   unsigned onebytes = prefix_length / 8;
> -   unsigned bits = prefix_length % 8;
> -   unsigned nullbytes = 16 - onebytes;
> -   memset(hb, -1, onebytes);
> -   memset(hb+onebytes, 0, nullbytes);
> -   if(bits) {
> -   unsigned char x = -1;
> -   x <<= 8 - bits;
> -   hb[onebytes] = x;
> +   uint8_t *dst;
> +   int len;
> +
> +   switch (af) {
> +   case AF_INET:
> +   dst = (uint8_t*) &sa->v4.sin_addr;
> +   len = 4;
> +   break;
> +   case AF_INET6:
> +   dst = (uint8_t*) &sa->v6.sin6_addr;
> +   len = 16;
> +   if (IN6_IS_ADDR_LINKLOCAL(addr) ||
> IN6_IS_ADDR_MC_LINKLOCAL(addr))
> +

Re: [PATCH 09/16] bsd: Added partial Linux NETLINK socket support

2018-08-13 Thread Nadav Har'El

Looks good.

You say that "NETLINK is used to support IPv4/IPv6 LIBC getifaddrs(),
if_nameindex().". But our implementations of these functions
don't use netlink. Do you plan to change them (I guess I'll see it in a
following patch)? Or just thinking of other software which uses
these netlink sockets directly?

Some more minor questions inline below.


--
Nadav Har'El
n...@scylladb.com

On Tue, Aug 7, 2018 at 5:49 AM, Charles Myers 
wrote:

> Signed-off-by: Charles Myers 
> ---
>  Makefile  |   1 +
>  bsd/net.cc|   5 +
>  bsd/sys/compat/linux/linux_netlink.cc | 904
> ++
>  bsd/sys/compat/linux/linux_netlink.h  | 175 +++
>  bsd/sys/compat/linux/linux_socket.cc  |   4 +
>  bsd/sys/compat/linux/linux_socket.h   |   1 +
>  bsd/sys/net/netisr.h  |   1 +
>  7 files changed, 1091 insertions(+)
>  create mode 100644 bsd/sys/compat/linux/linux_netlink.cc
>  create mode 100644 bsd/sys/compat/linux/linux_netlink.h
>
> diff --git a/Makefile b/Makefile
> index add72f9..49e61b8 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -564,6 +564,7 @@ bsd += bsd/porting/kobj.o
>  bsd += bsd/sys/netinet/if_ether.o
>  bsd += bsd/sys/compat/linux/linux_socket.o
>  bsd += bsd/sys/compat/linux/linux_ioctl.o
> +bsd += bsd/sys/compat/linux/linux_netlink.o
>  bsd += bsd/sys/net/if_ethersubr.o
>  bsd += bsd/sys/net/if_llatbl.o
>  bsd += bsd/sys/net/radix.o
> diff --git a/bsd/net.cc b/bsd/net.cc
> index 3e42757..f548e09 100644
> --- a/bsd/net.cc
> +++ b/bsd/net.cc
> @@ -23,6 +23,7 @@
>  #include 
>  #include 
>  #include 
> +#include 
>
>  /* Generation of ip ids */
>  void ip_initid(void);
> @@ -32,6 +33,8 @@ extern "C" {
>  extern  struct domain inetdomain;
>  /* AF_ROUTE */
>  extern  struct domain routedomain;
> +/* AF_NETLINK */
> +extern  struct domain netlinkdomain;
>  }
>
>  void net_init(void)
> @@ -53,9 +56,11 @@ void net_init(void)
>  domaininit(NULL);
>  OSV_DOMAIN_SET(inet);
>  OSV_DOMAIN_SET(route);
> +OSV_DOMAIN_SET(netlink);
>  rts_init();
>  route_init();
>  vnet_route_init();
> +netlink_init();
>  ipport_tick_init(NULL);
>  arp_init();
>  domainfinalize(NULL);
> diff --git a/bsd/sys/compat/linux/linux_netlink.cc
> b/bsd/sys/compat/linux/linux_netlink.cc
> new file mode 100644
> index 000..ecfa33d
> --- /dev/null
> +++ b/bsd/sys/compat/linux/linux_netlink.cc
> @@ -0,0 +1,904 @@
> +/*
> + * Linux NETLINK socket implementation.
> + *
> + * NETLINK is used to support IPv4/IPv6 LIBC getifaddrs(), if_nameindex().
> + *
> + * Warning: Tx/Rx messages are compatible with Linux not FreeBSD.
>

I don't see a copyright statement here. Is this derived from FreeBSD code
or any other existing code?
Or if it is your new code, do you want to add your own copyright statement
(and BSD license blurb)?


> + */
> +
> +#include 
> +#include 
> +
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +
> +#include 
> +#include 
> +#include 
> +
> +#ifdef INET6
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#endif
> +
> +#include 
> +#include 
> +#include 
> +
> +#if !defined(offsetof)
> +#define offsetof(TYPE, MEMBER) __builtin_offsetof (TYPE, MEMBER)
> +#endif
> +
> +mutex netlink_mtx;
> +
> +#define NETLINK_LOCK()  mutex_lock(&netlink_mtx)
> +#define NETLINK_UNLOCK() mutex_unlock(&netlink_mtx)
> +#define NETLINK_LOCK_ASSERT()   assert(netlink_mtx.owned())
> +
> +struct bsd_sockaddr_nl {
> +   uint8_t nl_len;   /* length of this struct */
> +   bsd_sa_family_t nl_family;/* AF_NETLINK */
> +   unsigned short  nl_pad;   /* Zero */
> +   pid_t   nl_pid;   /* Port ID */
> +   uint32_tnl_groups;/* Multicast groups mask */
> +};
> +
> +MALLOC_DEFINE(M_NETLINK, "netlink", "netlink socket");
> +
> +static struct  bsd_sockaddr netlink_src = { 2, PF_NETLINK, };
> +
> +
> +
> +static size_t mask_to_prefix_len(const uint8_t *bytes, size_t n_bytes)
> +{
> +   for (size_t i=0; i  +   uint8_t val = bytes[n_bytes - i - 1];
> +   if (val == 0)
> +   continue;
> +   /* Find first bit in byte which is set */
> +   int bit_pos = __builtin_ffs((long)val) - 1;
> +   size_t pos = 8 * (n_bytes - i) - bit_pos;
> +   return pos;
> +   }
> +   return 0;
> +}
> +
> +static int get_sockaddr_mask_prefix_len(struct bsd_sockaddr *sa)
> +{
> +   void *data;
> +   int   data_len;
> +
> +   if (!sa)
> +   return 0;
> +
> +   switch (sa->sa_family) {
> +#ifdef INET
> +   case AF_INET:
> +   data = &((struct bsd_sockaddr_in *)sa)->sin_addr;
> +   data_len = sizeof(((struct bsd_sockaddr_in
> *)sa)->sin_addr);

getrandom/sys_getrandom

[PATCH] Improved python27

RE: [PATCH 08/16] bsd: OSv API support for IPv6

[PATCH] Added libc __wcscpy_chk function

RE: [PATCH 07/16] bsd: Modifications to shared IPv4/IPv6 code (eth,tcp,udp) for IPv6

Re: __wcscpy_chk and Python 3

Re: __wcscpy_chk and Python 3

__wcscpy_chk and Python 3

Re: change redis appliance to cluster mode

Re: [PATCH 00/16] OSv IPv6 support

Re: [PATCH 16/16] bsd: Added conf-INET6 option to enable IPv6 support

Re: [PATCH 15/16] bsd: Added unit tests for IPv6 TCP, IP_PKTINFO, IPV6_PKTINFO, SCM_TIMESTAMP

Re: [PATCH 14/16] cloud-init: Added support for Network v1 and ConfigDrive data source

Re: [PATCH 12/16] bsd: Fix SIOCSIFNAME when using linux compatiblity socket

Re: [PATCH 11/16] libc: Add IPv6 support to getifaddrs(), if_nameindex() using NETLINK socket

Re: [PATCH 09/16] bsd: Added partial Linux NETLINK socket support

16 matches

Site Navigation

Mail list logo

Footer information