Re: [Maria-developers] Help needed TokuDB build errors in Debian
Otto, Can you please provide the diagnostic information I asked for? I do not have the time to set up my own identical environment. -- Cheers, Leif On Fri, Jul 11, 2014 at 4:28 PM, Otto Kekäläinen o...@seravo.fi wrote: Today the virtual Debian unstable machine that runs the buildbot builds for Debian unstable was upgraded and a manual re-build triggered with the result that it now failed: http://buildbot.askmonty.org/buildbot/builders/debpkg-sid/builds/27/steps/compile/logs/stdio This pretty much confirms that some newer version of some cmake or TokuDB or whatever dependency caused the issue. 2014-07-09 17:47 GMT+03:00 Leif Walsh leif.wa...@gmail.com: This error rings a faint bell, but I believe it was a problem with my toolchain, not a bug in the code or build system. There are also sometimes dependency problems with cmake that building with -j1 can fix. Can you reproduce it multiple times in a row with the same configured build directory? Can you reproduce it without using parallel make? If both of those are true, can you build again with VERBOSE=1 and post the log, as well as objdump the symbols defined in libtokuportability_static.a? I am a bit short on time here. Do you Leif have access to a Debian unstable instance or could you install one on your laptop as a virtual machine so you could do the above? Also when you have isolated the issue, can you write a small patch I can apply on top of 5.5.38 so that I can upload it without disabling TokuDB? The Debian build is extensively documented at https://github.com/ottok/mariadb-5.5/blob/master/debian/README.Maintainer and alternatively you can just launch the debian/autobuild.sh script. Also the build bot configs include step-by-step build commands.___ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
Re: [Maria-developers] Help needed TokuDB build errors in Debian
This error rings a faint bell, but I believe it was a problem with my toolchain, not a bug in the code or build system. There are also sometimes dependency problems with cmake that building with -j1 can fix. Can you reproduce it multiple times in a row with the same configured build directory? Can you reproduce it without using parallel make? If both of those are true, can you build again with VERBOSE=1 and post the log, as well as objdump the symbols defined in libtokuportability_static.a? On Wed, Jul 9, 2014 at 7:57 AM, Otto Kekäläinen o...@seravo.fi wrote: Hello Rich! For some strange reason, re-building 5.5.37 in Debian unstable stopped working. Any ideas what might be going on here?: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=753222 Recently a build of 5.5.38 generated a similar error: http://paste.ubuntu.com/7747594/ Are these maybe related? I have no idea how to fix these apart from disabling TokuDB in Debian when uploading 5.5.38. -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen ___ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp -- Cheers, Leif ___ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
Re: [Maria-developers] TokuDB disabled in Debian and Ubuntu - help me get it working?
Hi Otto, The answer to your first question is, that's how CMake works. CMake's Cross Compiling guide says that it can't guess the target processor details, and you're supposed to provide that information either by explicitly setting the variables, or by providing a toolchain file: http://www.cmake.org/Wiki/CMake_Cross_Compiling I would be surprised if launchpad.net's infrastructure did not include suitable toolchain files, but this really isn't my area of expertise. If you can find suitable ones to use, then you should use them, otherwise I think you should probably just add something to the rules file to set those variables explicitly. Regarding your second problem, it sounds like your packaging scripts aren't properly linking with jemalloc as the first library, with --whole-archive. I say that because we get a failure (from Elena's stacktrace) inside jemalloc code when calling free() inside the library constructor: #2 signal handler called #3 extent_ad_comp (a=0x7fff22e3f930, b=0x0) at extra/jemalloc/src/extent.c:32 #4 jemalloc_internal_extent_tree_ad_search (rbtree=rbtree@entry=0x7f5ceadfb0c0 huge, key=key@entry=0x7fff22e3f930) at extra/jemalloc/src/extent.c:38 #5 0x7f5ceab7fab8 in jemalloc_internal_huge_salloc (ptr=0x7f5cedf7ce00) at extra/jemalloc/src/huge.c:229 #6 0x7f5ceab6e335 in jemalloc_internal_isalloc (demote=false, ptr=0x7f5cedf7ce00) at include/jemalloc/internal/jemalloc_internal.h:863 #7 free (ptr=0x7f5cedf7ce00) at extra/jemalloc/src/jemalloc.c:1267 #8 0x7f5ceaac78de in toku_get_processor_frequency_cpuinfo (hzret=0x7fff22e3fa78) at storage/tokudb/ft-index/portability/portability.cc:371 #9 toku_os_get_processor_frequency (hzret=0x7fff22e3fa78) at storage/tokudb/ft-index/portability/portability.cc:409 #10 0x7f5ceaac7a5d in toku_portability_init () at storage/tokudb/ft-index/portability/portability.cc:139 #11 0x7f5ceaaf72bc in toku_ft_layer_init () at storage/tokudb/ft-index/ft/ft-ops.cc:6275 #12 0x7f5ceaa80f55 in _GLOBAL__I_65535_0_libtokufractaltree_static.a_0x235798 () at storage/tokudb/ft-index/src/ydb_lib.cc:103 I've seen this happen before when a buffer is allocated with the system allocator's malloc() (as likely happens in this call to getline(3) https://github.com/Tokutek/ft-index/blob/releases/tokudb-7.1/portability/portability.cc#L360), and then the fractal tree tries to free it with jemalloc ( https://github.com/Tokutek/ft-index/blob/releases/tokudb-7.1/portability/portability.cc#L371 ). Please make sure that jemalloc is being linked properly (as the first library, and with --whole-archive) *into mysqld*. It is not sufficient to only link it to ha_tokudb.so, because in that case the process (mysqld) will be using the system allocator, and there will be some possibly inlined calls to jemalloc's interface inside ha_tokudb.so. If you need help with this, please show me how your linking is being done and I'll try to give the right advice. If it is against policy to ship with the allocator statically linked in a binary, then you should make sure jemalloc isn't linked in ha_tokudb.so anywhere, but I strongly recommend against that. On Mon, Apr 14, 2014 at 5:01 PM, Rich Prohaska proha...@tokutek.com wrote: Hello Otto, Have not investigated these problems yet. Created a tokudb issue to track: https://github.com/Tokutek/mariadb-5.5/issues/53 On Mon, Apr 14, 2014 at 5:03 AM, Otto Kekäläinen o...@seravo.fi wrote: Hello Richard, Any chance of getting your comments on this..? Thanks! 2014-04-01 12:25 GMT+03:00 Otto Kekäläinen o...@seravo.fi: Hello Rick, Last year I spent a lot of time packaging MariaDB 5.5 for Debian and finally this year it has landed in Ubuntu 14.04 and Debian testing. Unfortunately the Debian/Ubuntu version does not include TokuDB and I need your help to get it there. In 5.5.35 (I think) the TokuDB plugn was added to MariaDB but I had issues getting it build 100% correctly and I eventually dropped it (added build parameter -DWITHOUT_TOKUDB=true), as getting MariaDB in Debian at all was a bigger priority than getting it there with every possible plugin. The root cause seems to be that when Debian and Ubuntu packages are built in chroot environments (the build systems of Debian and Ubuntu use pbuilder/sbuilder systems, see https://en.wikipedia.org/wiki/Debian_build_toolchain#Isolated_build_environments ) the code that builds the plugin does not seem to correctly detect the CPU features. It seems to read the values from the build machine and not the inputted target values (in a cross-compile situation). There are two related issues that needs a solution: 1) Currenlty the code that checks what the architecture is (32-bit/64-bit) is the first lines of https://bazaar.launchpad.net/~maria-captains/maria/10.0/view/head:/storage/tokudb/CMakeLists.txt . This works well for real and virtual machhines, but it does not seem to work in the pbuilder/sbuilder
[Maria-developers] packaging
Hi all, How do you make debian and rpm packages for mariadb? Which are the blessed config files and how is the process automated (and how might I run it manually)? -- Cheers, Leif ___ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
Re: [Maria-developers] packaging
Thanks, Sergei! On Tue, Jan 21, 2014 at 12:27 PM, Sergei Golubchik s...@mariadb.org wrote: Hi, Leif! On Jan 21, Leif Walsh wrote: Hi all, How do you make debian and rpm packages for mariadb? Which are the blessed config files and how is the process automated (and how might I run it manually)? This is for RPM: https://mariadb.com/kb/en/source-building-mariadb-on-centos/ This is for DEB: https://mariadb.com/kb/en/building-mariadb-on-ubuntu/ Note the up to 5.3 and starting from 5.5 sections there, make sure you read the correct one. Alternatively, you can see build logs directly, they show how exactly our release packages are built and unlike the knowledge base or any other description elsewhere they are always correct. For example, this is for fedora18: http://buildbot.askmonty.org/buildbot/builders/kvm-rpm-fedora18-amd64/builds/1388/steps/compile/logs/stdio And this is for Debian Wheezy: http://buildbot.askmonty.org/buildbot/builders/kvm-deb-wheezy-amd64/builds/1862/steps/compile/logs/stdio When I need to build deb packages manually (to fix some specific packaging bug) I always copy paste commands from the buildbot log :) Regards, Sergei -- Cheers, Leif ___ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
Re: [Maria-developers] Build fail of MariaDB10 with TokuDB enabled
That's a good solution, but I'd still be wary of messing around too much with
cmake's RPATH handling. I'll do this in ft-index soon though.
Sent from my iPhone
On Dec 3, 2013, at 4:09, Sergei Golubchik s...@mariadb.org wrote:
Hi, Leif!
On Dec 02, Leif Walsh wrote:
Yep, CMAKE_SKIP_BUILD_RPATH=0 would cause this exact problem. We
should add something to our cmake to warn if that is set to 0.
I'd suggest this change:
--- storage/tokudb/ft-index/ft/CMakeLists.txt 2013-11-19 14:35:31 +
+++ storage/tokudb/ft-index/ft/CMakeLists.txt 2013-12-03 09:04:29 +
@@ -11 +11 @@
-target_link_libraries(logformat ${LIBTOKUPORTABILITY})
+target_link_libraries(logformat ${LIBTOKUPORTABILITY}_static)
logformat is used internally during the build, it's not going to be
installed. Thus it can use static libraries and avoid the whole rpath
issue altogether.
This should probably be done for all binaries that are only used during
the build.
Regards,
Sergei
___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help : https://help.launchpad.net/ListHelp
Re: [Maria-developers] Build fail of MariaDB10 with TokuDB enabled
The problem is that we're trying to run logformat and it can't find libtokuportability.so, even though a few lines up, it says we just build libtokuportability.so. Maybe the loader path is different on suse, can you try looking in the build directory for libtokuportability.so (it's supposed to land in portability/), and can you try running ldd on ft/logformat in the build directory? On Mon, Dec 2, 2013 at 4:00 PM, Sergei Golubchik s...@mariadb.org wrote: Hi, darx! On Dec 02, d...@sent.com wrote: On the (eventual) road to successfully building MariaDB10 + TokuDB, applying a needed src patch cd /usr/local/src/mariadb # BUG @ http://www.linuxfromscratch.org/blfs/view/svn/server/mariadb.html sed -i s/srv_buf_size/srv_sort_buf_size/ storage/innobase/row/row0log.cc I do't know what it is (and couldn't find it on that page), but I suppose it's innodb only, so shouldn't matter here. repeating, but with TokuDB build enabled, cd bld cmake .. \ ... \ - -DWITHOUT_TOKUDB=1 + -DWITHOUT_TOKUDB=0 build now fails, make ... [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/toku_time.cc.o Linking CXX shared library libtokuportability.so [ 31%] Built target tokuportability Scanning dependencies of target logformat [ 31%] Building CXX object storage/tokudb/ft-index/ft/CMakeFiles/logformat.dir/logformat.cc.o Linking CXX executable logformat [ 31%] Built target logformat Scanning dependencies of target generate_log_code [ 31%] Generating log_code.cc, log_print.cc, log_header.h ./logformat: error while loading shared libraries: libtokuportability.so: cannot open shared object file: No such file or directory make[2]: *** [storage/tokudb/ft-index/ft/log_code.cc] Error 127 make[1]: *** [storage/tokudb/ft-index/ft/CMakeFiles/generate_log_code.dir/all] Error 2 make: *** [all] Error 2 yes, I'm aware that opensuse is not a currently supported platform for TokuDB. If you mean this page: https://mariadb.com/kb/en/how-to-enable-tokudb-in-mariadb/ It only lists platforms where we build tokudb binaries. While we have an opensuse builder, it's openSUSE 11.1, with gcc 4.3.2 and cmake 2.6, so it cannot build TokuDB. That's why openSUSE is not in the list. I don't know of any reason why TokuDB cannot be built on openSUSE, presuming all build requirements are met. And in your case they obviously are. If it's not, I'd like to keep trying to get it working, here. Ah, wait a sec. SuSE. Cannot find .so library. This does ring a bell - may be it's not exactly your problem, but take a look at this: https://github.com/Tokutek/ft-index/issues/73 We have it fixed. But perhaps what you see is somehow related? Try to run 'make VERBOSE=1' instead of 'make', there can be a clue there. Or email the output and let me look at it. Regards, Sergei ___ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp -- Cheers, Leif ___ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
Re: [Maria-developers] Build fail of MariaDB10 with TokuDB enabled
Can you run objdump -x /path/to/logformat | grep RPATH?
For example, I get this:
% ldd logformat
linux-vdso.so.1 (0x7fffe03cd000)
libtokuportability.so =
/home/leif/git/ft-index/opt/portability/libtokuportability.so
(0x7fba00ec1000)
libpthread.so.0 = /usr/lib/libpthread.so.0 (0x7fba00ca3000)
libdl.so.2 = /usr/lib/libdl.so.2 (0x7fba00a9f000)
librt.so.1 = /usr/lib/librt.so.1 (0x7fba00897000)
libstdc++.so.6 = /usr/lib/libstdc++.so.6 (0x7fba00593000)
libm.so.6 = /usr/lib/libm.so.6 (0x7fba0029)
libgcc_s.so.1 = /usr/lib/libgcc_s.so.1 (0x7fba0007a000)
libc.so.6 = /usr/lib/libc.so.6 (0x7fb9ffccf000)
/lib64/ld-linux-x86-64.so.2 (0x7fba010cd000)
% objdump -x logformat | grep RPATH
RPATH/home/leif/git/ft-index/opt/portability
Maybe RPATH is set differently on suse and our cmake isn't doing it right.
On Mon, Dec 2, 2013 at 4:47 PM, d...@sent.com wrote:
Hi
On Mon, Dec 2, 2013, at 01:00 PM, Sergei Golubchik wrote:
I do't know what it is (and couldn't find it on that page), but I
suppose it's innodb only, so shouldn't matter here.
Just following instructions. It's @
Install MariaDB by running the following commands:
sed -i s@data/test@\${INSTALL_MYSQLTESTDIR}@g
sql/CMakeLists.txt
== sed -i s/srv_buf_size/srv_sort_buf_size/
storage/innobase/row/row0log.cc
yes, I'm aware that opensuse is not a currently supported platform
for
TokuDB.
If you mean this page:
https://mariadb.com/kb/en/how-to-enable-tokudb-in-mariadb/
It only lists platforms where we build tokudb binaries. While we have an
opensuse builder, it's openSUSE 11.1, with gcc 4.3.2 and cmake 2.6, so
it cannot build TokuDB. That's why openSUSE is not in the list.
thanks
I don't know of any reason why TokuDB cannot be built on openSUSE,
presuming all build requirements are met. And in your case they
obviously are.
ok
If it's not, I'd like to keep trying to get it working, here.
Ah, wait a sec. SuSE. Cannot find .so library. This does ring a bell -
may be it's not exactly your problem, but take a look at this:
https://github.com/Tokutek/ft-index/issues/73
We have it fixed. But perhaps what you see is somehow related?
/lib vs /lib64 still crops up every once in a rare while
I'm not doing a ./configure ..., obviously, but my @MariaDB 'cmake'
already _does_ include
cmake .. \
...
-DINSTALL_LIBDIR=/lib64 \
...
perhaps it's not getting properly captured/passed?
Try to run 'make VERBOSE=1' instead of 'make', there can be a clue
there. Or email the output and let me look at it.
make VERBOSE=1
...
http://pastebin.com/Gqj5FP8N
The problem is that we're trying to run logformat and it can't find
libtokuportability.so, even though a few lines up, it says we just build
libtokuportability.so. Maybe the loader path is different on suse, can
you
try looking in the build directory for libtokuportability.so (it's
supposed
to land in portability/), and can you try running ldd on ft/logformat in
the build directory?
find . | grep libtokuportability
./storage/tokudb/ft-index/portability/libtokuportability_static.a
./storage/tokudb/ft-index/portability/libtokuportability_static_conv.a
./storage/tokudb/ft-index/portability/libtokuportability.so
find . | grep ft/logformat
./storage/tokudb/ft-index/ft/logformat
ldd ./storage/tokudb/ft-index/ft/logformat
linux-vdso.so.1 (0x7fffb2bdf000)
libpthread.so.0 = /lib64/libpthread.so.0 (0x7f506fb65000)
libtokuportability.so = not found
libdl.so.2 = /lib64/libdl.so.2 (0x7f506f961000)
librt.so.1 = /lib64/librt.so.1 (0x7f506f759000)
libstdc++.so.6 = /usr/lib64/libstdc++.so.6 (0x7f506f451000)
libm.so.6 = /lib64/libm.so.6 (0x7f506f153000)
libgcc_s.so.1 = /lib64/libgcc_s.so.1 (0x7f506ef3c000)
libc.so.6 = /lib64/libc.so.6 (0x7f506eb8e000)
/lib64/ld-linux-x86-64.so.2 (0x7f506fd81000)
darx
--
Cheers,
Leif
___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help : https://help.launchpad.net/ListHelp
Re: [Maria-developers] Build fail of MariaDB10 with TokuDB enabled
Yep, CMAKE_SKIP_BUILD_RPATH=0 would cause this exact problem. We should add something to our cmake to warn if that is set to 0. Sent from my iPhone On Dec 2, 2013, at 20:36, d...@sent.com wrote: On Mon, Dec 2, 2013, at 05:21 PM, Leif Walsh wrote: look online for reports of problems with cmake and suse. cmake is used quite reglarly with no issues, here anyway. it is, after all, used for the MariaDB build successfully, when TokuDB is disabled. re: rpath, if it's being used for the build phase to get at locally installed libs, then I should note that I cmake with, cmake ..\ ... -DCMAKE_SKIP_BUILD_RPATH=0 \ -DCMAKE_BUILD_WITH_INSTALL_RPATH=1 \ -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=1 \ ... these were (still are?) necessary to properly use run-time link ssl libs external to the default sys paths -- e.g., ssl installed in /usr/local/ssl, so, together, cmake .. -LAH | egrep -i crypto|ssl|rpath|suffix -- suffixes .a;.so -- OPENSSL_INCLUDE_DIR = /usr/local/ssl/include -- OPENSSL_LIBRARIES = /usr/local/ssl/lib64/libssl.so -- CRYPTO_LIBRARY = /usr/local/ssl/lib64/libcrypto.so -- OPENSSL_MAJOR_VERSION = 1 -- SSL_LIBRARIES = /usr/local/ssl/lib64/libssl.so;/usr/local/ssl/lib64/libcrypto.so;dl CMAKE_SKIP_INSTALL_RPATH:BOOL=NO CMAKE_SKIP_RPATH:BOOL=NO CRYPTO_LIBRARY:FILEPATH=/usr/local/ssl/lib64/libcrypto.so OPENSSL_INCLUDE_DIR:PATH=/usr/local/ssl/include OPENSSL_LIBRARIES:FILEPATH=/usr/local/ssl/lib64/libssl.so OPENSSL_ROOT_DIR:PATH=/usr/local/ssl // path to custom SSL installation WITH_SSL_PATH:PATH=/usr/local/ssl results in mysqld --version mysqld Ver 10.0.7-MariaDB-log for Linux on x86_64 (Source distribution) ldd `which mysqld` `which mysql` | egrep ssl|crypto|bin/ /usr/local/mariadb/bin/mysqld: libssl.so.1.0.0 = /usr/local/ssl/lib64/libssl.so.1.0.0 (0x7fd80fea9000) libcrypto.so.1.0.0 = /usr/local/ssl/lib64/libcrypto.so.1.0.0 (0x7fd80fabd000) /usr/local/mariadb/bin/mysql: libssl.so.1.0.0 = /usr/local/ssl/lib64/libssl.so.1.0.0 (0x7f2d5e65d000) libcrypto.so.1.0.0 = /usr/local/ssl/lib64/libcrypto.so.1.0.0 (0x7f2d5e271000) atm, I'm staring suspiciously at that -DCMAKE_SKIP_BUILD_RPATH=0 ... darx ___ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
Re: [Maria-developers] Build fail of MariaDB10 with TokuDB enabled
Wait, skip build RPATH off is a double-negative that I missed earlier. I think if you fool around with cmake's RPATH-related settings you'll be able to find the right combination, I'm not at a computer right now. At the very least, you should be able to leave them all as their defaults and the build should succeed. Sent from my iPhone On Dec 2, 2013, at 20:56, d...@sent.com wrote: testing that premise, rm -rf bld mkdir bld cd bld cmake .. \ ... - -DCMAKE_SKIP_BUILD_RPATH=0 \ + -DCMAKE_SKIP_BUILD_RPATH=1 \ ... -DWITHOUT_TOKUDB=0 make VERBOSE=1 unfortunately still fails, ... [ 31%] Built target tokuportability make -f storage/tokudb/ft-index/ft/CMakeFiles/logformat.dir/build.make storage/tokudb/ft-index/ft/CMakeFiles/logformat.dir/depend make[2]: Entering directory `/usr/local/src/mariadb/bld' cd /usr/local/src/mariadb/bld /usr/bin/cmake -E cmake_depends Unix Makefiles /usr/local/src/mariadb /usr/local/src/mariadb/storage/tokudb/ft-index/ft /usr/local/src/mariadb/bld /usr/local/src/mariadb/bld/storage/tokudb/ft-index/ft /usr/local/src/mariadb/bld/storage/tokudb/ft-index/ft/CMakeFiles/logformat.dir/DependInfo.cmake --color= Dependee /usr/local/src/mariadb/bld/storage/tokudb/ft-index/ft/CMakeFiles/logformat.dir/DependInfo.cmake is newer than depender /usr/local/src/mariadb/bld/storage/tokudb/ft-index/ft/CMakeFiles/logformat.dir/depend.internal. Dependee /usr/local/src/mariadb/bld/storage/tokudb/ft-index/ft/CMakeFiles/CMakeDirectoryInformation.cmake is newer than depender /usr/local/src/mariadb/bld/storage/tokudb/ft-index/ft/CMakeFiles/logformat.dir/depend.internal. Scanning dependencies of target logformat make[2]: Leaving directory `/usr/local/src/mariadb/bld' make -f storage/tokudb/ft-index/ft/CMakeFiles/logformat.dir/build.make storage/tokudb/ft-index/ft/CMakeFiles/logformat.dir/build make[2]: Entering directory `/usr/local/src/mariadb/bld' /usr/bin/cmake -E cmake_progress_report /usr/local/src/mariadb/bld/CMakeFiles [ 31%] Building CXX object storage/tokudb/ft-index/ft/CMakeFiles/logformat.dir/logformat.cc.o cd /usr/local/src/mariadb/bld/storage/tokudb/ft-index/ft /usr/bin/g++-4.8 -DHAVE_CONFIG_H -DTOKUDB_VERSION=\7.0.4\ -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -D_SVID_SOURCE -D_XOPEN_SOURCE=600 -D__LONG_LONG_SUPPORTED -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -std=c++11 -Wall -Werror -Wcast-align -Wshadow -Wmissing-format-attribute -Wpointer-arith -Wmissing-declarations -Wno-missing-noreturn -Wextra -Wno-error=strict-overflow -Wpacked -fno-exceptions -fno-rtti -Wno-error=missing-format-attribute -Wmissing-format-attribute -Wlogical-op -Wtrampolines -Wswitch -Winit-self -Wstrict-null-sentinel -Wno-missing-field-initializers -Wall -felide-constructors -fno-exceptions -fno-rtti -Wall -Wno-unused-parameter -fno-exceptions -fno-rtti -flto -fuse-linker-plugin -O2 -g -DNDEBUG -DDBUG_OFF -DMY_PTHREAD_FASTMUTEX=1 -g -O3 -UNDEBUG -I/usr/local/src/mariadb/bld/storage/tokudb/ft-index/buildheader -I/usr/local/src/mariadb/bld/include -I/usr/local/src/mariadb/bld/storage/tokudb/ft-index/./xz/include -I/usr/local/src/mariadb/storage/tokudb/ft-index/include -I/usr/local/src/mariadb/storage/tokudb/ft-index/toku_include -I/usr/local/src/mariadb/storage/tokudb/ft-index/portability -I/usr/local/src/mariadb/storage/tokudb/ft-index -I/usr/local/src/mariadb/bld/storage/tokudb/ft-index -I/usr/local/src/mariadb/bld/storage/tokudb/ft-index/toku_include -isystem /usr/local/include -I/usr/local/src/mariadb/storage/tokudb/ft-index/ft-o CMakeFiles/logformat.dir/logformat.cc.o -c /usr/local/src/mariadb/storage/tokudb/ft-index/ft/logformat.cc Linking CXX executable logformat cd /usr/local/src/mariadb/bld/storage/tokudb/ft-index/ft /usr/bin/cmake -E cmake_link_script CMakeFiles/logformat.dir/link.txt --verbose=1 /usr/bin/g++-4.8 -std=c++11 -Wall -Werror -Wcast-align -Wshadow -Wmissing-format-attribute -Wpointer-arith -Wmissing-declarations -Wno-missing-noreturn -Wextra -Wno-error=strict-overflow -Wpacked -fno-exceptions -fno-rtti -Wno-error=missing-format-attribute -Wmissing-format-attribute -Wlogical-op -Wtrampolines -Wswitch -Winit-self -Wstrict-null-sentinel -Wno-missing-field-initializers -Wall -felide-constructors -fno-exceptions -fno-rtti -Wall -Wno-unused-parameter -fno-exceptions -fno-rtti -flto -fuse-linker-plugin -O2 -g -DNDEBUG -DDBUG_OFF -DMY_PTHREAD_FASTMUTEX=1 -g -O3 -UNDEBUG -g -fuse-linker-plugin -Wno-error=strict-overflow CMakeFiles/logformat.dir/logformat.cc.o -o logformat -lpthread ../portability/libtokuportability.so -lpthread -ldl -lrt make[2]: Leaving directory `/usr/local/src/mariadb/bld' /usr/bin/cmake
Re: [Maria-developers] Can there be a better storage engine API?
Hi Mark, TokuMX is a quite different beast than TokuDB. First of all, we already had the experience of integrating our engine into one database product before we started. So many kinks in the TokuKV layer had already been worked out. But more importantly, TokuMX/MongoDB doesn't have a storage engine API. I think some people thought we were going to add a storage engine API to MongoDB and then plug ourselves into it. That wasn't the goal of TokuMX, the goal was simply to get our engine inside MongoDB as fast as possible, and the way to do that was to avoid thinking about what would be a good interface and instead to just do it. As everyone here I'm sure knows, making a good storage engine API is /really/ hard. Probably the hardest things in the TokuMX integration were learning how to deal with DDL (everything in MongoDB seems to use lazy initialization---for DDL operations at least), finding the right model within the MongoDB code to represent transactions, and reorganizing the locking. All these things were tightly coupled with the way the MongoDB storage system works (except transactions, well, because they didn't exist), but now in TokuMX they're pretty tightly coupled with the way TokuKV does things. In a way, we've created a storage API, but the API is defined by our version of db.h and nothing else implements that with the same assumptions we have, so it's probably not useful to compare the TokuMX storage engine API with the one in MySQL. In short, I'd say yes it was easier, but not because MongoDB has a better API (it doesn't have one), but because we had a bit of experience and because we didn't try to create or conform to a generic API. On Mon, Aug 19, 2013 at 11:36 AM, MARK CALLAGHAN mdcal...@gmail.com wrote: Thanks for your response. On Fri, Aug 16, 2013 at 11:23 AM, Zardosht Kasheff zardo...@gmail.comwrote: I've worked on the TokuDB storage engine for quite a while now. I have had many experiences over the years, so I guess it's hard to know where to begin. I guess I will start small, and if the conversation evolves, I can contribute more thoughts. I think the current API is really good, as evidenced by the fact that many storage engines have used it to plug into MySQL. The two areas that I see we can really benefit from are the following: Many were written in the long-ago past. Besides TokuDB how many new storage engines have reached GA in the past decade? I worked on a custom storage engine and I am sure others have done the same, but there hasn't been much innovation in the public. Aria is also GA, but that was written by people who know and wrote parts of the API, so it isn't a sign that the API is something people want to use. Was TokuMX easier to implement than TokuDB? -- Mark Callaghan mdcal...@gmail.com ___ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp -- Cheers, Leif ___ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
[Maria-developers] Plugin security
Hi, I'm working on adding a plugin API (not exactly a storage engine API, rather an extension one) to TokuMX[1] and I'm looking for advice on security. The basic idea is fairly simple: a plugin is a shared library (we're only targeting Linux right now, by the way) that defines a symbol to bootstrap the loading of additional functionality. We have an admin-only command that calls dlopen and dlsym to load the plugin. I haven't read any of the MariaDB plugin code yet, so pointers to where to start reading would be appreciated. Apart from that, I'm just looking for any advice from you all about what to watch out for. My concerns break down into two main areas: 1. When loading a single plugin, what should I watch out for? The command itself is admin-only so calling it is protected, but what about an attacker that drops a malicious plugin in a directory earlier on the load path? What should I check about the permissions/owner of the library file and the directory in which it lives? Is erroring out on world-writable libraries enough? If my process is setuid, should I be more careful? What does Maria do here? 2. About plugin persistence/autoloading: I understand MariaDB has a system table that tracks which plugins are installed and automatically loads those on startup. I believe I can do the same thing in TokuMX and I can restrict access to that collection to the admin user. What does MariaDB store in that table and how does it use it? I could imagine storing the checksum of the installed plugin in the system table and verifying it before loading the plugin, but this seems rather draconian and it might make plugin upgrades too difficult. Does MariaDB verify the permissions of the data files that store this system table? I could imagine an attack where you would replace this system table with one that includes instructions to load your malicious plugin, while the server is offline. Again, how does MariaDB solve these problems? Another option is to autoload all libraries in some directory, and to just be very careful about the permissions of that directory, but this seems very hard to get totally right so I'm leaning away from it. [1]: http://www.tokutek.com/products/tokumx-for-mongodb -- Cheers, Leif___ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
Re: [Maria-developers] Plugin security
TokuMX is MongoDB but with the storage code completely replaced by the same fractal tree library that we use in TokuDB. The storage code integration is already done. What I'm asking about here is not storage related. I'm trying to add the ability to create plugins to load into the existing server. You could do the same thing with vanilla MongoDB. I just want to be able to load shared libs that add arbitrary functionality. Since MariaDB has done this, I'm looking for advice on how to load plugins securely. -- Cheers, Leif On Mon, Jul 29, 2013 at 11:44 AM, Roberto Spadim robe...@spadim.com.br wrote: hi Leif! about your plugin... you will use mongodb (like a federate engine), or you will port the tokudb to mariadb (like a innodb engine)? i think the tokudb is being implemented today there's a MDEV in jira, for it at least (https://mariadb.atlassian.net/browse/MDEV-4507) i don't know about tokumx it's a engine to mongodb, or a query language (like a full text index)? about others doubts i don't know, must check with others users or run a test database___ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
Re: [Maria-developers] Installing jemalloc on build VMs
Hi, In previous versions, we shipped jemalloc as a shared library and patched mysqld_safe to inject it into LD_PRELOAD. Some users ran mysqld directly and therefore didn't have jemalloc. To protect users from this, we opted to statically link jemalloc into mysqld instead. To my knowledge that's the only reason it's a build dependency. -- Cheers, Leif On Fri, Apr 26, 2013 at 3:26 AM, Vladislav Vaintroub w...@montyprogram.com wrote: Mark, your links do not tell me why build dependency is good. Yes, yes, malloc() replacement du jour is nice, is fast, will bring peace to Middle East.. but it is extremely easy to setup on Linux. So, my question depends why would anyone want to have *build* dependency, remains unanswered. From: MARK CALLAGHAN [mailto:mdcal...@gmail.com] Sent: Freitag, 26. April 2013 08:16 To: Vladislav Vaintroub Cc: Kristian Nielsen; Daniel Bartholomew; maria-developers@lists.launchpad.net Subject: Re: [Maria-developers] Installing jemalloc on build VMs Educated snark is much better than uneducated snark. Why don't you take the time to understand why TokuDB benefits from jemalloc before suggesting they fix their code. mysqld has always been faster for me with tcmalloc jemalloc versus glibc malloc. I don't recall much support from everyone at official MySQL when I was reporting those problem. Why didn't you fix mysqld back then so I didn't have to waste time on that? http://mysqlha.blogspot.com/2009/01/double-sysbench-throughput-with_18.html http://mysqlha.blogspot.com/2008/12/make-mysql-faster-in-one-hour_14.html http://mysqlha.blogspot.com/2009/01/innodb-is-faster-tcmalloc-is-nice.html On Thu, Apr 25, 2013 at 9:47 AM, Vladislav Vaintroub w...@montyprogram.com wrote: -Original Message- From: Maria-developers [mailto:maria-developers- bounces+wlad=montyprogram@lists.launchpad.net] On Behalf Of Kristian Nielsen Sent: Donnerstag, 25. April 2013 18:30 To: Daniel Bartholomew Cc: maria-developers@lists.launchpad.net Subject: Re: [Maria-developers] Installing jemalloc on build VMs It is said that tokutek requires jemalloc - maybe it would be better to fix tokutek to work with standard libraries? I wanted to say the same. I do not understand quite how it is possible to rely on malloc replacement library - it is a replacement, it is typically handled at runtime (using LD_PRELOAD environment variable). Perhaps someone familiar with the subject can enlighten us on it. ___ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp -- Mark Callaghan mdcal...@gmail.com ___ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
Re: [Maria-developers] Installing jemalloc on build VMs
As far as I know there are no other technical reasons. I don't know how else to be certain that jemalloc is loaded if the user isn't using mysqld_safe though. -- Cheers, Leif On Fri, Apr 26, 2013 at 7:30 AM, Vladislav Vaintroub w...@montyprogram.com wrote: From: Leif Walsh [mailto:leif.wa...@gmail.com] Sent: Freitag, 26. April 2013 13:18 To: Vladislav Vaintroub Cc: MARK CALLAGHAN; Kristian Nielsen; maria-developers@lists.launchpad.net Subject: Re: [Maria-developers] Installing jemalloc on build VMs Hi, In previous versions, we shipped jemalloc as a shared library and patched mysqld_safe to inject it into LD_PRELOAD. Some users ran mysqld directly and therefore didn't have jemalloc. To protect users from this, we opted to statically link jemalloc into mysqld instead. To my knowledge that's the only reason it's a build dependency. Thanks. So, is this correct, that unlike what XL said previously, there is no technical reason to have this build dependency, you just want to be nice and protect people from libc. Every Linux user can run server with jemalloc if they wished to, e.g via mysqld_safe –malloc-lib=jemalloc, right? -- Cheers, Leif On Fri, Apr 26, 2013 at 3:26 AM, Vladislav Vaintroub w...@montyprogram.com wrote: Mark, your links do not tell me why build dependency is good. Yes, yes, malloc() replacement du jour is nice, is fast, will bring peace to Middle East.. but it is extremely easy to setup on Linux. So, my question depends why would anyone want to have *build* dependency, remains unanswered. From: MARK CALLAGHAN [mailto:mdcal...@gmail.com] Sent: Freitag, 26. April 2013 08:16 To: Vladislav Vaintroub Cc: Kristian Nielsen; Daniel Bartholomew; maria-developers@lists.launchpad.net Subject: Re: [Maria-developers] Installing jemalloc on build VMs Educated snark is much better than uneducated snark. Why don't you take the time to understand why TokuDB benefits from jemalloc before suggesting they fix their code. mysqld has always been faster for me with tcmalloc jemalloc versus glibc malloc. I don't recall much support from everyone at official MySQL when I was reporting those problem. Why didn't you fix mysqld back then so I didn't have to waste time on that? http://mysqlha.blogspot.com/2009/01/double-sysbench-throughput-with_18.html http://mysqlha.blogspot.com/2008/12/make-mysql-faster-in-one-hour_14.html http://mysqlha.blogspot.com/2009/01/innodb-is-faster-tcmalloc-is-nice.html On Thu, Apr 25, 2013 at 9:47 AM, Vladislav Vaintroub w...@montyprogram.com wrote: -Original Message- From: Maria-developers [mailto:maria-developers- bounces+wlad=montyprogram@lists.launchpad.net] On Behalf Of Kristian Nielsen Sent: Donnerstag, 25. April 2013 18:30 To: Daniel Bartholomew Cc: maria-developers@lists.launchpad.net Subject: Re: [Maria-developers] Installing jemalloc on build VMs It is said that tokutek requires jemalloc - maybe it would be better to fix tokutek to work with standard libraries? I wanted to say the same. I do not understand quite how it is possible to rely on malloc replacement library - it is a replacement, it is typically handled at runtime (using LD_PRELOAD environment variable). Perhaps someone familiar with the subject can enlighten us on it. ___ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp -- Mark Callaghan mdcal...@gmail.com ___ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
