Re: Merging hammer in master: conflict on gmock
Am 10.03.2015 um 09:02 schrieb Loic Dachary: Hi Danny, On 10/03/2015 06:59, Danny Al-Gaaf wrote: Am 10.03.2015 um 00:06 schrieb Loic Dachary: [...] Do you know what changes have been done in hammer in src/gmock ? It seems as if there was no change to src/gmock in hammer that is not already in master. After the set of commands you suggest, how could we make sure nothing was trashed or unintentionally included ? I can't think of a sure method from the top of my head. Since these commands only trash/ignore the changes to src/gmock from the hammer branch. And I guess there will be nothing for this directory we need to merge back to master. In fact the merge conflict is not about commits AFAICS but about the change in master from inline to submodule code. One possibility would always be to extract the patches from hammer (git checkout hammer git format-patch master), and check after the merge if one is missing (e.g. run 'git am' until you have a patch that can be applied). I know this is not ideal ... IMO git is missing an automated way to handle these kind of merge conflict. Danny -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: conditions for removing experimental feature marks
Dear Loic, I really apologize that our reply is too late. Since removing experimental flag from SHEC at v0.94 is our immediate hope, we should have started any actions for that earlier. By the way, you mean we must add just the same workaround as 10887 for SHEC if our target is Hammer. Is my understanding correct ? Running teuthology locally is non trivial, maybe we can provide you with access to the community lab so that you can run teuthology suites against the existing teuthology cluster (i.e. http://pulpito.ceph.com/). Would you like me to ask ? Thank you for your proposal. It would be greatly helpful if we were allowed to use it ! Best regards, Takeshi Miyamae -Original Message- From: Loic Dachary [mailto:l...@dachary.org] Sent: Monday, February 16, 2015 6:17 PM To: Miyamae, Takeshi/宮前 剛; ceph-devel@vger.kernel.org Cc: Shiozawa, Kensuke/塩沢 賢輔; Nakao, Takanori/中尾 鷹詔 Subject: Re: conditions for removing experimental feature marks Hi, On 16/02/2015 08:37, Miyamae, Takeshi wrote: Dear Loic, Thank you for your help on the pull request of SHEC last week. We believe that marking experimental feature on SHEC was inevitable and the way to restrict the feature is proper. By the way, could you let us know what are the conditions for removing experimental feature marks in the future? Are additional thorough tests required? The next step is to run integration tests with the shec plugin. The integration tests have shown that the shec plugin does not disrupt anything. Now we should check if it works properly under stress and upgrades. I created two tickets for that purpose: http://tracker.ceph.com/issues/10886 : integration / theuthology integration / theuthology thrasher tests for the shec erasure code plugin http://tracker.ceph.com/issues/10887 : erasure-code: allow upgrades for shec plugins It would be fantastic if you could work on http://tracker.ceph.com/issues/7291 : it would make the integration of new erasure code plugins easier. I realize that it is infrastructure work not directly of interest to the shec plugin implementation. Running teuthology locally is non trivial, maybe we can provide you with access to the community lab so that you can run teuthology suites against the existing teuthology cluster (i.e. http://pulpito.ceph.com/). Would you like me to ask ? Cheers Best regards, Takeshi Miyamae -- Loïc Dachary, Artisan Logiciel Libre
Re: conditions for removing experimental feature marks
Hi !
On 10/03/2015 11:35, Miyamae, Takeshi wrote:
Dear Loic,
I really apologize that our reply is too late.
Since removing experimental flag from SHEC at v0.94 is our immediate hope, we
should
have started any actions for that earlier.
No worries, it's never too late to do good :-)
By the way, you mean we must add just the same workaround as 10887 for SHEC
if our target is Hammer.
Is my understanding correct ?
What is needed is integration tests running during a few weeks and showing the
plugin behaves well.
Running teuthology locally is non trivial, maybe we can provide you with
access to the
community lab so that you can run teuthology suites against the existing
teuthology cluster
(i.e. http://pulpito.ceph.com/). Would you like me to ask ?
Thank you for your proposal.
It would be greatly helpful if we were allowed to use it !
Could you please connect to irc.oftc.net#sepia ? This is the IRC channel where
lab related matters are discussed. We can figure out the details there.
In the meantime you can try teuthology and write a single test quite easily, on
your local machine. The tricky part comes where we want to run suites with
large number of jobs. You can adapt the instructions from
http://dachary.org/?p=2204 to your local environment. All you need really is
the ability to run two virtual machines.
Cheers
Best regards,
Takeshi Miyamae
-Original Message-
From: Loic Dachary [mailto:l...@dachary.org]
Sent: Monday, February 16, 2015 6:17 PM
To: Miyamae, Takeshi/宮前 剛; ceph-devel@vger.kernel.org
Cc: Shiozawa, Kensuke/塩沢 賢輔; Nakao, Takanori/中尾 鷹詔
Subject: Re: conditions for removing experimental feature marks
Hi,
On 16/02/2015 08:37, Miyamae, Takeshi wrote:
Dear Loic,
Thank you for your help on the pull request of SHEC last week.
We believe that marking experimental feature on SHEC was inevitable
and the way to restrict the feature is proper.
By the way, could you let us know what are the conditions for removing
experimental feature marks in the future?
Are additional thorough tests required?
The next step is to run integration tests with the shec plugin. The
integration tests have shown that the shec plugin does not disrupt anything.
Now we should check if it works properly under stress and upgrades. I created
two tickets for that purpose:
http://tracker.ceph.com/issues/10886 : integration / theuthology integration
/ theuthology thrasher tests for the shec erasure code plugin
http://tracker.ceph.com/issues/10887 : erasure-code: allow upgrades for shec
plugins
It would be fantastic if you could work on
http://tracker.ceph.com/issues/7291 : it would make the integration of new
erasure code plugins easier. I realize that it is infrastructure work not
directly of interest to the shec plugin implementation.
Running teuthology locally is non trivial, maybe we can provide you with
access to the community lab so that you can run teuthology suites against the
existing teuthology cluster (i.e. http://pulpito.ceph.com/). Would you like
me to ask ?
Cheers
Best regards,
Takeshi Miyamae
--
Loïc Dachary, Artisan Logiciel Libre
N�r��y���b�X��ǧv�^�){.n�+���z�]z���{ay� ʇڙ�,j��f���h���z� �w���
���j:+v���w�j�mzZ+�ݢj��!tml=
--
Loïc Dachary, Artisan Logiciel Libre
signature.asc
Description: OpenPGP digital signature
Re: FileStore performance: coalescing operations
Can we also consider to coalesce two OP_SETATTR transaction to a
single OP_SETATTRS transaction?
Regards
Ning Yao
2015-03-05 15:04 GMT+08:00 Haomai Wang haomaiw...@gmail.com:
I think the performance improvement can be refer to
https://github.com/ceph/ceph/pull/2972 which I did a simple benchmark
comparison.
This coalescing should get a lighter improvement but I think obviously
it should be better.
On Thu, Mar 5, 2015 at 8:10 AM, Sage Weil s...@newdream.net wrote:
On Tue, 3 Mar 2015, Sage Weil wrote:
I think we should try to avoid the dups in the first place in the upper
layers before resorting to deduping in the FileStore.
I took a stab at this:
https://github.com/ceph/ceph/pull/3878
Here's what the txns look like now:
{
ops: [
{
op_num: 0,
op_name: omap_setkeys,
collection: 0.2_head,
oid: 2\/\/head\/\/0,
attr_lens: {
05.0002: 178,
_epoch: 4,
_info: 729,
can_rollback_to: 12,
rollback_info_trimmed_to: 12
}
},
{
op_num: 1,
op_name: write,
collection: 0.2_head,
oid: 8b08fa52\/benchmark_data_maetl_15609_object5\/head\/\/0,
length: 123,
offset: 0,
bufferlist length: 123
},
{
op_num: 2,
op_name: setattr,
collection: 0.2_head,
oid: 8b08fa52\/benchmark_data_maetl_15609_object5\/head\/\/0,
name: _,
length: 269
},
{
op_num: 3,
op_name: setattr,
collection: 0.2_head,
oid: 8b08fa52\/benchmark_data_maetl_15609_object5\/head\/\/0,
name: snapset,
length: 31
}
]
}
Care to give that a spin and see if it give syou a similar (or better)
improvement?
sage
--
To unsubscribe from this list: send the line unsubscribe ceph-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Best Regards,
Wheat
--
To unsubscribe from this list: send the line unsubscribe ceph-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line unsubscribe ceph-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
ceph v0.80.9 debian source packages???
Hi guys, The last trusty version 0.80.9 have been pushed in the deb http://ceph.com/debian-firefly/ trusty main repository yesterday. The last packages have the version 0.80.9-1trusty, but I can not find the corresponding source packages in http://gitbuilder.ceph.com/ceph-deb-trusty-x86_64-basic/ref/v0.80.9 Where can I find the correct ceph_0.80.9-1trusty.dsc and ceph_0.80.9-1trusty.tar.gz source packages? Regards, Valery -- SWITCH -- Valery Tschopp, Software Engineer, Peta Solutions Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland email: valery.tsch...@switch.ch phone: +41 44 268 1544 smime.p7s Description: S/MIME Cryptographic Signature
[PATCH V6 1/5] Build support for BlkKin (LTTng + Zipkin) tracing
* Adds --with-blkin to autoconf (default without)
* Adds BLKIN_LIBS to necessary automake files
* Adds documentation for testing BlkKin tracing
* Adds ztrace initialization
Signed-off-by: Andrew Shewmaker ags...@gmail.com
Signed-off-by: Marios-Evaggelos Kogias marioskog...@gmail.com
Signed-off-by: Chendi.Xue chendi@intel.com
---
configure.ac | 11 +++
do_autogen.sh|5 +
doc/dev/blkin.rst| 168 +++
src/Makefile-env.am |6 +
src/Makefile.am | 14 +++
src/ceph_mds.cc |2
src/ceph_mon.cc |2
src/ceph_osd.cc |2
src/librados/Makefile.am |9 ++
src/librbd/Makefile.am | 10 ++
src/mon/Makefile.am | 11 ++-
src/msg/Makefile.am |8 ++
src/os/Makefile.am |8 ++
src/osd/Makefile.am |9 ++
src/osdc/Makefile.am |9 ++
15 files changed, 271 insertions(+), 3 deletions(-)
create mode 100644 doc/dev/blkin.rst
diff --git a/configure.ac b/configure.ac
index 05f0cf9..78d431d 100644
--- a/configure.ac
+++ b/configure.ac
@@ -333,6 +333,17 @@ if test x$enable_coverage != xno; then
fi
AC_SUBST(GCOV_PREFIX_STRIP, `echo $(pwd)/src | tr -dc / | wc -c`)
+# blkin (lttng+zipkin) tracing?
+AC_ARG_WITH([blkin],
+ [AS_HELP_STRING([--with-blkin], [blkin (lttng + zipkin) tracing])],
+ [],
+ [with_blkin=no])
+have_blkin=no
+AS_IF([test x$with_blkin == xyes],
+[PKG_CHECK_MODULES([BLKIN], [blkin], [have_blkin=yes])])
+AM_CONDITIONAL(WITH_BLKIN, test x$have_blkin == xyes)
+AM_COND_IF([WITH_BLKIN], [AC_DEFINE([WITH_BLKIN], [1], [Defined if using
BlkKin])])
+
# radosgw?
AC_ARG_WITH([radosgw],
[AS_HELP_STRING([--with-radosgw], [build RADOS gateway])],
diff --git a/do_autogen.sh b/do_autogen.sh
index 7df1c93..0e50654 100755
--- a/do_autogen.sh
+++ b/do_autogen.sh
@@ -5,6 +5,7 @@ usage() {
do_autogen.sh: make a ceph build by running autogen, etc.
-h: this help message
+-b blkin tracing
-d level debug build
level 0: no debug
level 1: -g
@@ -32,9 +33,11 @@ debug_level=0
verbose=0
profile=0
CONFIGURE_FLAGS=--disable-static --with-lttng
-while getopts d:e:hHrTPJLjpcvO: flag
+while getopts bd:e:hHrTPJLjpcvO: flag
do
case $flag in
+b) CONFIGURE_FLAGS=$CONFIGURE_FLAGS --with-blkin;;
+
d) debug_level=$OPTARG;;
O) CFLAGS=${CFLAGS} -O$OPTARG;;
diff --git a/doc/dev/blkin.rst b/doc/dev/blkin.rst
new file mode 100644
index 000..bf14142
--- /dev/null
+++ b/doc/dev/blkin.rst
@@ -0,0 +1,168 @@
+=
+ Tracing Ceph With BlkKin
+=
+
+Ceph can use Blkin, a library created by Marios Kogias and others,
+which enables tracking a specific request from the time it enters
+the system at higher levels till it is finally served by RADOS.
+
+In general, Blkin implements the Dapper_ tracing semantics
+in order to show the causal relationships between the different
+processing phases that an IO request may trigger. The goal is an
+end-to-end visualisation of the request's route in the system,
+accompanied by information concerning latencies in each processing
+phase. Thanks to LTTng this can happen with a minimal overhead and
+in realtime. The LTTng traces can then be visualized with Twitter's
+Zipkin_.
+
+.. _Dapper:
http://static.googleusercontent.com/media/research.google.com/el//pubs/archive/36356.pdf
+.. _Zipkin: http://twitter.github.io/zipkin/
+
+
+Installing Blkin
+
+
+You can install Markos Kogias' upstream Blkin_ by hand.::
+
+ cd blkin/
+ make make install
+
+ or build
+distribution packages using DistroReadyBlkin_, which also comes
+with pkgconfig support. If you choose the latter, then you must
+generate the configure and make files first.::
+
+ cd blkin
+ autoreconf -i
+
+.. _Blkin: https://github.com/marioskogias/blkin
+.. _DistroReadyBlkin: https://github.com/agshew/blkin
+
+
+Configuring Ceph with Blkin
+===
+
+If you built and installed Blkin by hand, rather than building and
+installing packages, then set these variables before configuring
+Ceph.::
+
+ export BLKIN_CFLAGS=-Iblkin/
+ export BLKIN_LIBS=-lzipkin-cpp
+
+Since there are separate lttng and blkin changes to Ceph, you may
+want to configure with something like::
+
+ ./configure --with-blkin --without-lttng --with-debug
+
+
+Testing Blkin
+=
+
+It's easy to test Ceph's Blkin tracing. Let's assume you don't have
+Ceph already running, and you compiled Ceph with Blkin support but
+you did't install it. Then launch Ceph with the ``vstart.sh`` script
+in Ceph's src directgory so you can see the possible tracepoints.::
+
+ cd src
+ OSD=3 MON=3 RGW=1 ./vstart.sh -n
+ lttng list --userspace
+
+You'll see something like the following:::
+
+ UST events:
+
[no subject]
The following patches are based on the work of Marios Kogias, first posted in August. http://www.spinics.net/lists/ceph-devel/msg19890.html This patch is against HEAD as of March 10th, commit 5d5b510810e96503b9323b010149f7bd5b45db7c. It can also be found at https://github.com/agshew/ceph/tree/wip-blkin-v6 Thanks to Josh Durgin again for comments on the V5 patchset. I think the blkin patchset is looking pretty good at this point. Outstanding issues: 1) librados will need more general blkin tracing currently it only has aio_read_traced() and aio_write_traced() calls 2) some work will need to be done filtering blkin events/keyvalues To Do: 1. push a wip-blkin branch to github.com/ceph and take advantage of gitbuilder test/qa 2. submit a pull request 3. add Andreas' tracepoints https://github.com/ceph/ceph/pull/2877 using Blkin and investigate how easy it is to select the level of tracing detail Changes since V5: * put MOSDOp encode(features) statement back that was accidentally left out in V5 * moved OSD daemonize call back to original spot * initialized blkin in ceph-mds (and moved all initializations to first patch) * updated aio_read_traced() and aio_write_traced() to match non-traced versions * improved blkin wrapper readability by removing unnecessary stringification Changes since V4: * removed messenger_end trace event In Pipe::reader(), when message is enqueued, it will be destroyed. Naive pointer checks don't work here. You can't depend on pointers being set to null on destruction. It may be possible to wrap trace event with m-get() and m-put() to keep it around, or put this trace event in dispatch paths, but just removing trace event for now in order to move forward. * removed mutex in aio_*_traced() methods A mutex was carried forward from Marios' original patch while rebasing when it should have been removed. * removed Message::trace_end_after_span Message::trace_end_after_span did not ever appear to be true, so it has been removed * added asserts for master trace and endpoint checks Tried to use asserts in more places, but they prevented execution. Tried to use douts and ldouts instead, but they didn't work. * added parens around macro pointer args parens make it safer to use pointers passed as arguments in a macro Changes since V2: * WITH_BLKIN added to makefile vars when necessary * added Blkin build instructions * added Zipkin build instructions * Blkin wrapper macros do not stringify args any longer. The macro wrappers will be more flexible/robust if they don't turn arguments into strings. * added missing blkin_trace_info struct prototype to librados.h * TrackedOp trace creation methods are virtual, implemented in OpRequest * avoid faults due to non-existent traces Check if osd_trace exists when creating a pg_trace, etc. Return true only if trace creation was successful. Use dout() if trace_osd, trace_pg, etc. fail, in order to ease debugging. * create trace_osd in ms_fast_dispatch Changes since V1: * split build changes into separate patch * conditional build support for blkin (default off) * blkin is not a Ceph repo submodule build and install packages from https://github.com/agshew/blkin.git Note: rpms don't support babeltrace plugins for use with Zipkin * removal of debugging in Message::init_trace_info() With this patchset Ceph can use Blkin, a library created by Marios Kogias and others, which enables tracking a specific request from the time it enters the system at higher levels till it is finally served by RADOS. In general, Blkin implements the tracing semantics described in the Dapper paper http://static.googleusercontent.com/media/research.google.com/el/pubs/archive/36356.pdf in order to trace the causal relationships between the different processing phases that an IO request may trigger. The goal is an end-to-end visualisation of the request's route in the system, accompanied by information concerning latencies in each processing phase. Thanks to LTTng this can happen with a minimal overhead and in realtime. In order to visualize the results Blkin was integrated with Twitter's Zipkin http://twitter.github.io/zipkin/ (which is a tracing system entirely based on Dapper). A short document describing how to test Blkin tracing in Ceph with Zipkin is in doc/dev/trace.rst -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V6 3/5] OSD support for blkin (LTTng + Zipkin) tracing
* Adds tracing for OSDs and Placement Groups, specifically:
messages for OSD operations, FileJournal, FileStore,
ECBackend, replication, and traced versions of Objecter
read/write (only basic read/write functions).
* Moves global_init_daemonize() earlier in ceph_osd.c:main
Signed-off-by: Marios-Evaggelos Kogias marioskog...@gmail.com
Signed-off-by: Filippos Giannakos philipg...@grnet.gr
Signed-off-by: Andrew Shewmaker ags...@gmail.com
Signed-off-by: Chendi.Xue chendi@intel.com
---
20 files changed, 323 insertions(+), 14 deletions(-)
messages/MOSDOp.h | 13
messages/MOSDOpReply.h| 13
messages/MOSDSubOp.h | 11 ++
messages/MOSDSubOpReply.h | 12 +++
os/FileJournal.cc | 14 ++--
os/FileJournal.h |6 +++
os/FileStore.cc | 20
os/FileStore.h|1
osd/ECBackend.cc |8 +++-
osd/OSD.cc| 21 +
osd/OSD.h |1
osd/OpRequest.cc | 74 ++
osd/OpRequest.h |9 +
osd/PG.cc |2 +
osd/PG.h |1
osd/ReplicatedBackend.cc | 19 +--
osd/ReplicatedPG.cc | 35 +
osdc/Objecter.cc |7
osdc/Objecter.h | 63 ++-
19 files changed, 319 insertions(+), 11 deletions(-)
diff --git a/src/messages/MOSDOp.h b/src/messages/MOSDOp.h
index 5b88f31..388c9b9 100644
--- a/src/messages/MOSDOp.h
+++ b/src/messages/MOSDOp.h
@@ -32,7 +32,11 @@ class OSD;
class MOSDOp : public Message {
+#ifdef WITH_BLKIN
+ static const int HEAD_VERSION = 6;
+#else
static const int HEAD_VERSION = 5;
+#endif
static const int COMPAT_VERSION = 3;
private:
@@ -185,6 +189,7 @@ public:
// marshalling
virtual void encode_payload(uint64_t features) {
+BLKIN_GET_MASTER(mt);
OSDOp::merge_osd_op_vector_in_data(ops, data);
@@ -261,11 +266,14 @@ struct ceph_osd_request_head {
::encode(retry_attempt, payload);
::encode(features, payload);
+
+ BLKIN_MSG_ENCODE_TRACE();
}
}
virtual void decode_payload() {
bufferlist::iterator p = payload.begin();
+BLKIN_MSG_DO_INIT_TRACE();
if (header.version 2) {
// old decode
@@ -348,6 +356,8 @@ struct ceph_osd_request_head {
::decode(features, p);
else
features = 0;
+
+ BLKIN_MSG_DECODE_TRACE(6);
}
OSDOp::split_osd_op_vector_in_data(ops, data);
@@ -390,6 +400,9 @@ struct ceph_osd_request_head {
out e osdmap_epoch;
out );
}
+
+ BLKIN_MSG_INFO_DECL(MOSDOp)
+ BLKIN_MSG_END_DECL(MOSDOp)
};
diff --git a/src/messages/MOSDOpReply.h b/src/messages/MOSDOpReply.h
index b2d5155..64f01c1 100644
--- a/src/messages/MOSDOpReply.h
+++ b/src/messages/MOSDOpReply.h
@@ -32,7 +32,11 @@
class MOSDOpReply : public Message {
+#ifdef WITH_BLKIN
+ static const int HEAD_VERSION = 7;
+#else
static const int HEAD_VERSION = 6;
+#endif
static const int COMPAT_VERSION = 2;
object_t oid;
@@ -143,6 +147,7 @@ public:
if (ignore_out_data)
ops[i].outdata.clear();
}
+BLKIN_MSG_CHECK_SPAN();
}
private:
~MOSDOpReply() {}
@@ -150,6 +155,8 @@ private:
public:
virtual void encode_payload(uint64_t features) {
+BLKIN_GET_MASTER(mt);
+
OSDOp::merge_osd_op_vector_out_data(ops, data);
if ((features CEPH_FEATURE_PGID64) == 0) {
@@ -189,10 +196,13 @@ public:
::encode(replay_version, payload);
::encode(user_version, payload);
::encode(redirect, payload);
+
+ BLKIN_MSG_ENCODE_TRACE();
}
}
virtual void decode_payload() {
bufferlist::iterator p = payload.begin();
+BLKIN_MSG_DO_INIT_TRACE();
if (header.version 2) {
ceph_osd_reply_head head;
::decode(head, p);
@@ -244,6 +254,8 @@ public:
if (header.version = 6)
::decode(redirect, p);
+
+ BLKIN_MSG_DECODE_TRACE(7);
}
}
@@ -270,6 +282,7 @@ public:
out );
}
+ BLKIN_MSG_END_DECL(MOSDOpReply)
};
diff --git a/src/messages/MOSDSubOp.h b/src/messages/MOSDSubOp.h
index 544dfcf..96f48a9 100644
--- a/src/messages/MOSDSubOp.h
+++ b/src/messages/MOSDSubOp.h
@@ -25,8 +25,13 @@
class MOSDSubOp : public Message {
+#ifdef WITH_BLKIN
+ static const int HEAD_VERSION = 12;
+#else
static const int HEAD_VERSION = 11;
static const int COMPAT_VERSION = 7;
+#endif
+ static const int COMPAT_VERSION = 1;
public:
epoch_t map_epoch;
@@ -103,6 +108,7 @@ public:
//version =7
assert (header.version = 7);
bufferlist::iterator p = payload.begin();
+BLKIN_MSG_DO_INIT_TRACE();
::decode(map_epoch, p);
::decode(reqid, p);
::decode(pgid.pgid, p);
@@ -170,6 +176,7 @@ public:
} else {
pg_trim_rollback_to = pg_trim_to;
}
+BLKIN_MSG_DECODE_TRACE(12);
}
virtual void
[PATCH V6 2/5] Initial support for blkin (LTTng + Zipkin) tracing
* Adds conditional build support.
* Adds BLKIN_* macro wrappers to reduce need for ifdef WITH_BLKIN.
* Adds blkin header include.
* Adds message infrastructure for blkin tracing.
* Adds blkin trace events and keyvals to OpTracker.
* Adds osd, pg, journal, and filestore blkin traces to OpTracker/OpRequest.
These changes are Marios', with the following exceptions:
* split out initial support for Marios' additional tracing changes
* conditional build and BLKIN_* macro wrappers
* blkin include path is blkin/ztracer.hpp
* only Message.h includes blkin, as others include Message.h
* commented code has been removed
* unused lname var/code in SimpleMessenger has been removed
* excluded messenger_trace because it appears not to be used
* braces added to conditionals to match recommended coding style
note: did not change member variable names to use m_ prefix
since member vars in same class did not
* Ref suffix added to TrackedOp shared_ptr vars to be consistent
note: did not add -Ref suffix to ZTrace*Ref vars in Message
* TrackedOp trace creation methods are virtual, implemented in OpRequest
* avoid faults due to non-existent traces
Check if osd_trace exists when creating a pg_trace, etc.
Return true only if trace creation was successful.
Use dout() if trace_osd, trace_pg, etc. fail, in order to ease debugging.
Signed-off-by: Marios-Evaggelos Kogias marioskog...@gmail.com
Signed-off-by: Filippos Giannakos philipg...@grnet.gr
Signed-off-by: Andrew Shewmaker ags...@gmail.com
Signed-off-by: Chendi.Xue chendi@intel.com
---
src/common/TrackedOp.cc | 82 ++
src/common/TrackedOp.h| 26 ++
src/common/blkin_wrapper.h| 171 ++
src/msg/Message.cc| 76 +
src/msg/Message.h | 25 ++
src/msg/msg_types.cc | 23 -
src/msg/msg_types.h | 1 +
src/msg/simple/Pipe.cc| 31 +++
src/msg/simple/Pipe.h | 4 +
src/msg/simple/SimpleMessenger.cc | 1 +
10 files changed, 437 insertions(+), 3 deletions(-)
create mode 100644 src/common/blkin_wrapper.h
diff --git a/src/common/TrackedOp.cc b/src/common/TrackedOp.cc
index 32dbc53..611911f 100644
--- a/src/common/TrackedOp.cc
+++ b/src/common/TrackedOp.cc
@@ -293,8 +293,22 @@ void OpTracker::_mark_event(TrackedOp *op, const string
evt,
}
+#ifdef WITH_BLKIN
+void OpTracker::trace_event(TrackedOp *op, TrackedOpTraceRef t, const string
evt, TrackedOpEndpointRef ep)
+{
+ t-event(evt, ep);
+}
+
+void OpTracker::trace_keyval(TrackedOp *op, TrackedOpTraceRef t, const string
key,
+const string val, TrackedOpEndpointRef ep)
+{
+ t-keyval(key, val, ep);
+}
+#endif
+
void OpTracker::RemoveOnDelete::operator()(TrackedOp *op) {
op-mark_event(done);
+ BLKIN_OP_TRACE_EVENT(op, osd, span_ended);
if (!tracker-tracking_enabled) {
op-_unregistered();
delete op;
@@ -332,3 +346,71 @@ void TrackedOp::dump(utime_t now, Formatter *f) const
f-close_section();
}
}
+
+#ifdef WITH_BLKIN
+void TrackedOp::trace_osd(string event)
+{
+ if (!osd_trace) {
+dout(5) trace_osd failed, osd_trace doesn't exist, event: event
dendl;
+return;
+ }
+
+ tracker-trace_event(this, osd_trace, event, osd_trace-get_endpoint());
+}
+
+void TrackedOp::trace_osd(string key, string val)
+{
+ if (!osd_trace) {
+dout(5) trace_osd failed, osd_trace doesn't exist, key: key
val: val dendl;
+return;
+ }
+
+ tracker-trace_keyval(this, osd_trace, key, val, osd_trace-get_endpoint());
+}
+
+void TrackedOp::trace_pg(string event)
+{
+ if (!pg_trace) {
+dout(5) trace_pg failed, pg_trace doesn't exist, event: event
dendl;
+return;
+ } else if (!osd_trace) {
+dout(5) trace_pg failed, osd_trace doesn't exist, event: event
dendl;
+return;
+ }
+
+ tracker-trace_event(this, osd_trace, event, pg_trace-get_endpoint());
+}
+
+void TrackedOp::get_pg_trace_info(struct blkin_trace_info *info)
+{
+ if (!pg_trace) {
+dout(5) get_pg_trace failed, pg_trace doesn't exist dendl;
+return;
+ } else if (!osd_trace) {
+dout(5) get_pg_trace failed, osd_trace doesn't exist dendl;
+return;
+ }
+
+ osd_trace-get_trace_info(info);
+}
+
+void TrackedOp::trace_journal(string event)
+{
+ if (!journal_trace) {
+dout(5) trace_journal failed, journal_trace doesn't exist, event:
event dendl;
+return;
+ }
+
+ tracker-trace_event(this, journal_trace, event,
journal_trace-get_endpoint());
+}
+
+void TrackedOp::trace_filestore(string event)
+{
+ if (!filestore_trace) {
+dout(5) trace_filestore failed, filestore_trace doesn't exist, event:
event dendl;
+return;
+ }
+
+ tracker-trace_event(this, filestore_trace, event,
filestore_trace-get_endpoint());
+}
+#endif // WITH_BLKIN
diff --git a/src/common/TrackedOp.h b/src/common/TrackedOp.h
index 42d1eaf..fc861f7
[PATCH V6 4/5] Rados support for blkin (LTTng + Zipkin) tracing
* Adds traced versions of Rados aio_read/write
Signed-off-by: Marios-Evaggelos Kogias marioskog...@gmail.com
Signed-off-by: Filippos Giannakos philipg...@grnet.gr
Signed-off-by: Andrew Shewmaker ags...@gmail.com
Signed-off-by: Chendi.Xue chendi@intel.com
---
include/rados/librados.h | 13 ++
librados/IoCtxImpl.cc| 91 +++
librados/IoCtxImpl.h | 12 ++
librados/RadosClient.cc |1
librados/librados.cc | 30 +++
5 files changed, 147 insertions(+)
diff --git a/src/include/rados/librados.h b/src/include/rados/librados.h
index 15a225b..aa0416e 100644
--- a/src/include/rados/librados.h
+++ b/src/include/rados/librados.h
@@ -1736,6 +1736,13 @@ CEPH_RADOS_API int rados_aio_write(rados_ioctx_t io,
const char *oid,
rados_completion_t completion,
const char *buf, size_t len, uint64_t off);
+#ifdef WITH_BLKIN
+struct blkin_trace_info;
+int rados_aio_write_traced(rados_ioctx_t io, const char *oid,
+rados_completion_t completion,
+const char *buf, size_t len, uint64_t off,
+struct blkin_trace_info *info);
+#endif
/**
* Asychronously append data to an object
*
@@ -1818,6 +1825,12 @@ CEPH_RADOS_API int rados_aio_read(rados_ioctx_t io,
const char *oid,
rados_completion_t completion,
char *buf, size_t len, uint64_t off);
+#ifdef WTIH_BLKIN
+int rados_aio_read_traced(rados_ioctx_t io, const char *oid,
+rados_completion_t completion,
+char *buf, size_t len, uint64_t off,
+struct blkin_trace_info *info);
+#endif
/**
* Block until all pending writes in an io context are safe
*
diff --git a/src/librados/IoCtxImpl.cc b/src/librados/IoCtxImpl.cc
index 5ef56c0..cfff8a6 100644
--- a/src/librados/IoCtxImpl.cc
+++ b/src/librados/IoCtxImpl.cc
@@ -31,6 +31,7 @@ librados::IoCtxImpl::IoCtxImpl() :
aio_write_seq(0),
cached_pool_names_lock(librados::IoCtxImpl::cached_pool_names_lock),
objecter(NULL)
{
+BLKIN_MSG_END(ioctx_endpoint, 0.0.0.0, 0, ioctx);
}
librados::IoCtxImpl::IoCtxImpl(RadosClient *c, Objecter *objecter,
@@ -42,6 +43,7 @@ librados::IoCtxImpl::IoCtxImpl(RadosClient *c, Objecter
*objecter,
aio_write_seq(0),
cached_pool_names_lock(librados::IoCtxImpl::cached_pool_names_lock),
objecter(objecter)
{
+BLKIN_MSG_END(ioctx_endpoint, 0.0.0.0, 0, ioctx);
}
void librados::IoCtxImpl::set_snap_read(snapid_t s)
@@ -516,6 +518,7 @@ int librados::IoCtxImpl::operate(const object_t oid,
::ObjectOperation *o,
Objecter::Op *objecter_op = objecter-prepare_mutate_op(oid, oloc,
*o, snapc, ut, flags,
NULL, oncommit, ver);
+ BLKIN_OP_SET_TRACE(objecter_op, o-trace);
objecter-op_submit(objecter_op);
mylock.Lock();
@@ -645,6 +648,48 @@ int librados::IoCtxImpl::aio_read(const object_t oid,
AioCompletionImpl *c,
return 0;
}
+#ifdef WITH_BLKIN
+int librados::IoCtxImpl::aio_read_traced(const object_t oid, AioCompletionImpl
*c,
+char *buf, size_t len, uint64_t off,
+uint64_t snapid,
+struct blkin_trace_info *info)
+{
+ if (len (size_t) INT_MAX)
+return -EDOM;
+
+ /*handle trace*/
+ ZTracer::ZTraceRef trace;
+ trace = ZTracer::create_ZTrace(librados, ioctx_endpoint, info, true);
+ if (!trace) {
+ldout(client-cct, 5) librados read trace could not be created
dendl;
+return -1;
+ }
+
+ trace-event(librados accept);
+ Context *onack = new C_aio_Ack(c);
+
+ c-is_read = true;
+ c-io = this;
+ c-bl.clear();
+ c-bl.push_back(buffer::create_static(len, buf));
+ c-blp = c-bl;
+
+ c-tid = objecter-read(oid, oloc,
+off, len, snapid, c-bl, 0,
+onack, c-objver);
+
+ trace-event(send to objecter);
+ struct blkin_trace_info *child_info = (struct blkin_trace_info *)
+ malloc(sizeof(struct blkin_trace_info));
+ trace-get_trace_info(child_info);
+ objecter-read_traced(oid, oloc,
+off, len, snapid, c-bl, 0,
+onack, child_info, c-objver);
+
+ return 0;
+}
+#endif // WITH_BLKIN
+
class C_ObjectOperation : public Context {
public:
::ObjectOperation m_ops;
@@ -705,6 +750,52 @@ int librados::IoCtxImpl::aio_write(const object_t oid,
AioCompletionImpl *c,
return 0;
}
+#ifdef WITH_BLKIN
+int librados::IoCtxImpl::aio_write_traced(const object_t oid,
AioCompletionImpl *c,
+ const bufferlist bl, size_t len,
+ uint64_t off, struct blkin_trace_info
*info)
+{
+ utime_t ut = ceph_clock_now(client-cct);
+ ldout(client-cct, 20) aio_write_traced oid off ~
len snapc= snapc snap_seq= snap_seq dendl;
+
+
[PATCH V6 5/5] Rados example for blkin (LTTng + Zipkin) tracing
This code is Marios', with the following exceptions:
* split out example from other tracing changes
* changed name of example
* whitespace issues have been fixed
* Makefile cleans up the new example
Signed-off-by: Marios-Evaggelos Kogias marioskog...@gmail.com
Signed-off-by: Filippos Giannakos philipg...@grnet.gr
Signed-off-by: Andrew Shewmaker ags...@gmail.com
---
Makefile |9
hello_blkin.c | 107 ++
2 files changed, 116 insertions(+)
create mode 100644 examples/librados/hello_blkin.c
diff --git a/examples/librados/Makefile b/examples/librados/Makefile
index 190cca2..f8d8e1d 100644
--- a/examples/librados/Makefile
+++ b/examples/librados/Makefile
@@ -1,4 +1,8 @@
+ifdef WITH_BLKIN
+all: librados_hello_world librados_hello_world_c librados_hello_blkin
+else
all: librados_hello_world librados_hello_world_c
+endif
librados_hello_world: hello_world.cc
g++ -g -c hello_world.cc -o hello_world.o $(CFLAGS)
@@ -8,6 +12,11 @@ librados_hello_world_c: hello_world_c.c
cc -g -c hello_world_c.c -o hello_world_c.o $(CFLAGS)
cc -g hello_world_c.o -lrados -o librados_hello_world_c $(LDFLAGS)
+librados_hello_blkin: hello_blkin.c
+ cc -g -c hello_blkin.c -o hello_blkin.o $(CFLAGS)
+ cc -g hello_blkin.o -lrados -lblkin-front -o librados_hello_blkin
$(LDFLAGS)
+
clean:
rm hello_world.o librados_hello_world
rm hello_world_c.o librados_hello_world_c
+ rm hello_blkin.o librados_hello_blkin
diff --git a/examples/librados/hello_blkin.c b/examples/librados/hello_blkin.c
new file mode 100644
index 000..1bc1c50
--- /dev/null
+++ b/examples/librados/hello_blkin.c
@@ -0,0 +1,107 @@
+// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*-
+// vim: ts=8 sw=2 smarttab
+
+#include stdio.h
+#include rados/librados.h
+#include stdlib.h
+#include blkin-front.h
+
+#define POOL data
+
+const char *object_name = hello_world_object;
+const char *object_val = hello world!;
+
+rados_ioctx_t *connect_to_rados()
+{
+ rados_t rados;
+ rados_ioctx_t *ioctx = malloc(sizeof(rados_ioctx_t));
+
+ if (rados_create(rados, NULL) 0) {
+return NULL;
+ }
+ if (rados_conf_read_file(rados, NULL) 0){
+return NULL;
+ }
+ if (rados_connect(rados) 0) {
+printf(Could not connect\n);
+rados_shutdown(rados);
+return NULL;
+ }
+ if (rados_pool_lookup(rados, POOL) 0) {
+printf(Could not find pool\n);
+rados_shutdown(rados);
+return NULL;
+ }
+ if (rados_ioctx_create(rados, POOL, ioctx) 0){
+rados_shutdown(rados);
+return NULL;
+ }
+ return ioctx;
+}
+
+void end_read(rados_completion_t c, void *arg)
+{
+ fprintf(stderr, just read\n);
+}
+
+void end_write(rados_completion_t c, void *arg)
+{
+ fprintf(stderr, just wrote\n);
+}
+
+int main()
+{
+ int r, i;
+ char temp[12];
+ rados_completion_t compl;
+ rados_ioctx_t *rados_ctx;
+ /*initialize lib*/
+ r = blkin_init();
+ if (r 0) {
+fprintf(stderr, Could not initialize blkin\n);
+exit(1);
+ }
+ /*initialize endpoint*/
+ struct blkin_endpoint endp;
+ blkin_init_endpoint(endp, 10.0.0.1, 5000, rados client);
+ struct blkin_trace trace;
+ blkin_init_new_trace(trace, client, endp);
+ BLKIN_TIMESTAMP(trace, endp, start);
+
+ //connect
+ rados_ctx = connect_to_rados();
+ if (rados_ctx == NULL){
+printf(Connect return NULL\n);
+exit(1);
+ }
+ printf(connected\n);
+
+ //write an object
+ r = rados_aio_create_completion(NULL, end_write, end_write, compl);
+ if (r0) {
+printf(could not create completion\n);
+exit(1);
+ }
+ printf(created completion\n);
+ r = rados_aio_write_traced(*rados_ctx, object_name, compl, object_val, 12,
+ 0, trace.info);
+
+ rados_aio_wait_for_complete(compl);
+ printf(completed\n);
+
+ //read an object
+ r = rados_aio_create_completion(NULL, end_read, end_read, compl);
+ if (r0) {
+printf(could not create completion\n);
+exit(1);
+ }
+ printf(created completion\n);
+ r = rados_aio_read_traced(*rados_ctx, object_name, compl, temp, 12, 0,
+ trace.info);
+ rados_aio_wait_for_complete(compl);
+ printf(I read:\n);
+ for (i=0;i12;i++)
+printf(%c,temp[i]);
+ printf(completed\n);
+ BLKIN_TIMESTAMP(trace, endp, Span ended);
+}
--
To unsubscribe from this list: send the line unsubscribe ceph-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: wip-auth
Hi -- was running some baselines on 0.93 and wanted to check, will wip-auth changes go into Hamer or in a later release? (Sorry if I missed the discussion in one of the perf meetings last month) Thanks, Stephen -Original Message- From: ceph-devel-ow...@vger.kernel.org [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Mark Nelson Sent: Wednesday, February 4, 2015 4:25 PM To: Sage Weil; Andreas Bluemle Cc: Blinick, Stephen L; ceph-devel@vger.kernel.org Subject: Re: wip-auth Hi All, I completed some tests with wip-auth to the memstore on ubuntu earlier today. Basic gist of it is that the improvements in wip-auth help but don't quite get us to what can be achieved with auth disabled. RHEL7 (without auth) continues to do very well in latency bound situations. Next up will be to see if how much this matters when testing against on SSDs. Here are the results: sync 4k object writes = CephOS AuthAvg Lat (ms)Avg IOPS master Ubuntu 14.04Yes 0.991007 Wip-authUbuntu 14.04Yes 0.811237 master Ubuntu 14.04No 0.641549 Wip-authUbuntu 14.04No 0.651563 master RHEL7 No 0.323158 sync 4k object reads CephOS AuthAvg Lat (ms)Avg IOPS master Ubuntu 14.04Yes 0.591695 Wip-authUbuntu 14.04Yes 0.412409 master Ubuntu 14.04No 0.293425 Wip-authUbuntu 14.04No 0.293474 master RHEL7 No 0.175853 256 concurrent 4k object writes === CephOS AuthAvg Lat (ms)Avg IOPS master Ubuntu 14.04Yes 40.39 6339 Wip-authUbuntu 14.04Yes 26.22 9763 master Ubuntu 14.04No 17.46 14662 Wip-authUbuntu 14.04No 17.34 14759 master RHEL7 No 14.93 17139 256 concurrent 4k object reads == CephOS AuthAvg Lat (ms)Avg IOPS master Ubuntu 14.04Yes 31.47 8134 Wip-authUbuntu 14.04Yes 19.81 12922 master Ubuntu 14.04No 12.82 19968 Wip-authUbuntu 14.04No 12.75 20080 master RHEL7 No 12.04 21257 Mark On 01/30/2015 03:08 PM, Sage Weil wrote: Hi Andreas, It looks like that was a stale sha1, but the newer one was also broken. I've retested and it's working for me now. See latest wip-auth, sha1 0c21a7875059bef80842756dfb003f47cc2d66a6. Thanks! sage On Fri, 30 Jan 2015, Andreas Bluemle wrote: Hi Sage, I tried to integrate wip-auth into my 0.91 build environment. I had not been able to start the cluster successfully: ceph-mon crashes with a segmentation fault in CryptoKey::encrypt() (see attachment). This happens when linking with libnss or libcryptopp (version 5.6.2). I created the patch to add wip-auth based on github pull request 3523 and was able to use this patch directly with 0.91 with only a minor adaptation for common/ceph_context.h: the 0.91 version of ceph_context.h did not know anything about the experimental class CephContextObs. wip-auth commit ID is 1a0507a2940f6edcc2bf9533cfa6c210b0b41933. As my build environment is rpm, I had to modify the invocation of the configure script in the spec file instead of the do_autogen.sh script. Best Regards Andreas Bluemle On Tue, 27 Jan 2015 09:18:45 -0800 (PST) Sage Weil sw...@redhat.com wrote: I spent some time focusing on just CryptoKey::encrypt(). I benchmarked 100,000 encrypts of 128 bytes and got, at baseline, cryptopp: 10 encoded in 0.655651 libnss : 10 encoded in 1.288786 Ouch! With a (fixed) version of my earlier patch that avoids uselessly copying the input buffer: 10 encoded in 1.231977 With a patch that puts the key structures in CryptoKey instead of recreating them each time: 10 encoded in 0.396208 -- ~70% improvement over original This is pushed to wip-auth. There's also a patch that caches key structs for crypopp.. it now takes 10 encoded in 0.440758 -- ~33% improvement over original (Not that almost anybody will ever care; we use libnss by default for both rpm and deb distros.) So, yay, nss is now a bit faster. What I'm not completely certain about is whether the structures I've preserved are truly stateless
RE: wip-auth
On Tue, 10 Mar 2015, Blinick, Stephen L wrote: Hi -- was running some baselines on 0.93 and wanted to check, will wip-auth changes go into Hamer or in a later release? (Sorry if I missed the discussion in one of the perf meetings last month) It will go in after hammer. We can backport as needed. sage Thanks, Stephen -Original Message- From: ceph-devel-ow...@vger.kernel.org [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Mark Nelson Sent: Wednesday, February 4, 2015 4:25 PM To: Sage Weil; Andreas Bluemle Cc: Blinick, Stephen L; ceph-devel@vger.kernel.org Subject: Re: wip-auth Hi All, I completed some tests with wip-auth to the memstore on ubuntu earlier today. Basic gist of it is that the improvements in wip-auth help but don't quite get us to what can be achieved with auth disabled. RHEL7 (without auth) continues to do very well in latency bound situations. Next up will be to see if how much this matters when testing against on SSDs. Here are the results: sync 4k object writes = Ceph OS AuthAvg Lat (ms)Avg IOPS masterUbuntu 14.04Yes 0.991007 Wip-auth Ubuntu 14.04Yes 0.811237 masterUbuntu 14.04No 0.641549 Wip-auth Ubuntu 14.04No 0.651563 masterRHEL7 No 0.323158 sync 4k object reads Ceph OS AuthAvg Lat (ms)Avg IOPS masterUbuntu 14.04Yes 0.591695 Wip-auth Ubuntu 14.04Yes 0.412409 masterUbuntu 14.04No 0.293425 Wip-auth Ubuntu 14.04No 0.293474 masterRHEL7 No 0.175853 256 concurrent 4k object writes === Ceph OS AuthAvg Lat (ms)Avg IOPS masterUbuntu 14.04Yes 40.39 6339 Wip-auth Ubuntu 14.04Yes 26.22 9763 masterUbuntu 14.04No 17.46 14662 Wip-auth Ubuntu 14.04No 17.34 14759 masterRHEL7 No 14.93 17139 256 concurrent 4k object reads == Ceph OS AuthAvg Lat (ms)Avg IOPS masterUbuntu 14.04Yes 31.47 8134 Wip-auth Ubuntu 14.04Yes 19.81 12922 masterUbuntu 14.04No 12.82 19968 Wip-auth Ubuntu 14.04No 12.75 20080 masterRHEL7 No 12.04 21257 Mark On 01/30/2015 03:08 PM, Sage Weil wrote: Hi Andreas, It looks like that was a stale sha1, but the newer one was also broken. I've retested and it's working for me now. See latest wip-auth, sha1 0c21a7875059bef80842756dfb003f47cc2d66a6. Thanks! sage On Fri, 30 Jan 2015, Andreas Bluemle wrote: Hi Sage, I tried to integrate wip-auth into my 0.91 build environment. I had not been able to start the cluster successfully: ceph-mon crashes with a segmentation fault in CryptoKey::encrypt() (see attachment). This happens when linking with libnss or libcryptopp (version 5.6.2). I created the patch to add wip-auth based on github pull request 3523 and was able to use this patch directly with 0.91 with only a minor adaptation for common/ceph_context.h: the 0.91 version of ceph_context.h did not know anything about the experimental class CephContextObs. wip-auth commit ID is 1a0507a2940f6edcc2bf9533cfa6c210b0b41933. As my build environment is rpm, I had to modify the invocation of the configure script in the spec file instead of the do_autogen.sh script. Best Regards Andreas Bluemle On Tue, 27 Jan 2015 09:18:45 -0800 (PST) Sage Weil sw...@redhat.com wrote: I spent some time focusing on just CryptoKey::encrypt(). I benchmarked 100,000 encrypts of 128 bytes and got, at baseline, cryptopp: 10 encoded in 0.655651 libnss : 10 encoded in 1.288786 Ouch! With a (fixed) version of my earlier patch that avoids uselessly copying the input buffer: 10 encoded in 1.231977 With a patch that puts the key structures in CryptoKey instead of recreating them each time: 10 encoded in 0.396208 -- ~70% improvement over original This is pushed to wip-auth. There's also a patch that caches key structs for crypopp.. it now takes 10
Re: teuthology-lock and choosing a kernel
On Tue, Mar 10, 2015 at 1:32 AM, Loic Dachary l...@dachary.org wrote: Hi Andrew, I successfully installed a 3.19 kernel (details at http://dachary.org/?p=3594). It turns out that the loop module is compiled in and defaults to having zero partitions allowed by default. Since I was looking for a solution to have /dev/loop useable for tests, I rebooted with /boot/grub/grub.conf as [ubuntu@vpm083 src]$ cat /boot/grub/grub.conf default=0 timeout=5 splashimage=(hd0,0)/boot/grub/splash.xpm.gz hiddenmenu title rhel-6.5-cloudinit (3.19.0-ceph-00029-gaf5b96e) root (hd0,0) kernel /boot/vmlinuz-3.19.0-ceph-00029-gaf5b96e ro root=LABEL=79d3d2d4 loop.max_part=16 initrd /boot/initramfs-3.19.0-ceph-00029-gaf5b96e.img and that works fine. Do you know if I could do that from the yaml file directly ? Alternatively I could use a kernel that does not have the loop module compiled in and modprobe it with loop.max_part=16, but it's unclear to me what kernels are available and what their names are. I think you can run a set of commands from yaml (see exec stanza), so you can sort of script it (GRUB_CMDLINE_LINUX=, etc), but that's never going to be reliable. Basically the only kernel available right now is the testing flavor. debug flavor is some random config and has been unused for a while. I can try changing testing config to do CONFIG_BLK_DEV_LOOP=m if you don't find a good way to workaround it. https://lists.ubuntu.com/archives/kernel-team/2009-September/007364.html is apparently the reason it's compiled in (our config is a somewhat old and slightly stripped down Ubuntu config). Thanks, Ilya -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: teuthology-lock and choosing a kernel
Hi Ilya, On 10/03/2015 08:19, Ilya Dryomov wrote: On Tue, Mar 10, 2015 at 1:32 AM, Loic Dachary l...@dachary.org wrote: Hi Andrew, I successfully installed a 3.19 kernel (details at http://dachary.org/?p=3594). It turns out that the loop module is compiled in and defaults to having zero partitions allowed by default. Since I was looking for a solution to have /dev/loop useable for tests, I rebooted with /boot/grub/grub.conf as [ubuntu@vpm083 src]$ cat /boot/grub/grub.conf default=0 timeout=5 splashimage=(hd0,0)/boot/grub/splash.xpm.gz hiddenmenu title rhel-6.5-cloudinit (3.19.0-ceph-00029-gaf5b96e) root (hd0,0) kernel /boot/vmlinuz-3.19.0-ceph-00029-gaf5b96e ro root=LABEL=79d3d2d4 loop.max_part=16 initrd /boot/initramfs-3.19.0-ceph-00029-gaf5b96e.img and that works fine. Do you know if I could do that from the yaml file directly ? Alternatively I could use a kernel that does not have the loop module compiled in and modprobe it with loop.max_part=16, but it's unclear to me what kernels are available and what their names are. I think you can run a set of commands from yaml (see exec stanza), so you can sort of script it (GRUB_CMDLINE_LINUX=, etc), but that's never going to be reliable. Basically the only kernel available right now is the testing flavor. debug flavor is some random config and has been unused for a while. I can try changing testing config to do CONFIG_BLK_DEV_LOOP=m if you don't find a good way to workaround it. https://lists.ubuntu.com/archives/kernel-team/2009-September/007364.html is apparently the reason it's compiled in (our config is a somewhat old and slightly stripped down Ubuntu config). Thanks for exploring the history so far back ;-) I hope to find an easier path but its comforting to know where we stand. Cheers -- Loïc Dachary, Artisan Logiciel Libre signature.asc Description: OpenPGP digital signature
Re: giant integration branch for v0.87.1 ready for QE / fix for 10433/8011 ? osd/ReplicatedPG.cc: 5545: FAILED assert(soid scrubber.start || soid = scrubber.end)
Hi, Any reason why the complete fix for 8011/10433 was not included in giant 0.87.1? Missing fix: http://workbench.dachary.org/ceph/ceph/commit/9b26de3f3653d38dcdfc5b97874089f19d2a59d7 Issues: http://tracker.ceph.com/issues/8011 http://tracker.ceph.com/issues/10433#note-7 Thanks in advance, Sincerely, Laurent On Fri, 2015-02-20 at 16:49 -0500, Yuri Weinstein wrote: Team leads, Please review QE validation results summary in http://tracker.ceph.com/issues/10501 Loic - this RC looks ready for release (in my opinion) ! Thx YuriW - Original Message - From: Yuri Weinstein ywein...@redhat.com To: Loic Dachary l...@dachary.org Cc: Ceph Development ceph-devel@vger.kernel.org Sent: Monday, February 16, 2015 9:33:44 AM Subject: Re: giant integration branch for v0.87.1 ready for QE I have completed suites execution on giant branch (v0.87.1 RC) All results are summarized in http://tracker.ceph.com/issues/10501 under QE VALIDATION section. Some suites had to be run more then ones due to environment noise. Two suites are being re-run now - upgrade:firefly-x and powecycle. Next steps: - the team leads to review/confirm results - Loic - can you review and triage issues as needed. - two suites require results analysis: multimds rados (two known tickets, but need more checking) ## 10209, 9891 krbd (two new tickets, but need more checking) ## 10889, 10890 Thx YuriW - Original Message - From: Loic Dachary l...@dachary.org To: Yuri Weinstein ywein...@redhat.com Cc: Ceph Development ceph-devel@vger.kernel.org Sent: Wednesday, February 11, 2015 7:30:06 AM Subject: Re: giant integration branch for v0.87.1 ready for QE Hi Yuri, The giant-backports pull requests were merged into https://github.com/ceph/ceph/tree/giant which is not ready for testing. For the record, the head is https://github.com/ceph/ceph/commit/78c71b9200da5e7d832ec58765478404d31ae6b5 Cheers On 10/02/2015 18:20, Loic Dachary wrote: Hi Yuri, The giant integration branch for v0.87.1 as found at https://github.com/ceph/ceph/commits/giant-backports has been approved by Yehuda, Josh and Sam and is ready for QE. For the record, the head is https://github.com/ceph/ceph/commit/6b08a729540c61f3c8b15c5a3ce9382634bf800c Cheers -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Merging hammer in master: conflict on gmock
Hi Danny, On 10/03/2015 06:59, Danny Al-Gaaf wrote: Am 10.03.2015 um 00:06 schrieb Loic Dachary: On 09/03/2015 18:31, Danny Al-Gaaf wrote: Hi Loic, this one is a tricky one. I didn't find a smooth/automatic solution yet. This seems to work: git checkout master git merge --no-ff origin/hammer git mergetool - for the conflict: select 'local' for src/gmock (this should keep the changes im master and ignore changes from hammer) Do you know what changes have been done in hammer in src/gmock ? It seems as if there was no change to src/gmock in hammer that is not already in master. After the set of commands you suggest, how could we make sure nothing was trashed or unintentionally included ? I can't think of a sure method from the top of my head. Cheers Danny -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- Loïc Dachary, Artisan Logiciel Libre signature.asc Description: OpenPGP digital signature
Re: giant integration branch for v0.87.1 ready for QE / fix for 10433/8011 ? osd/ReplicatedPG.cc: 5545: FAILED assert(soid scrubber.start || soid = scrubber.end)
Hi Laurent, http://tracker.ceph.com/issues/8011 is now marked with a higher priority for backporting (I'm using the Severity field for that purpose). Please let me know if other backports need more attention. Cheers On 10/03/2015 07:56, Laurent GUERBY wrote: Hi, Any reason why the complete fix for 8011/10433 was not included in giant 0.87.1? Missing fix: http://workbench.dachary.org/ceph/ceph/commit/9b26de3f3653d38dcdfc5b97874089f19d2a59d7 Issues: http://tracker.ceph.com/issues/8011 http://tracker.ceph.com/issues/10433#note-7 Thanks in advance, Sincerely, Laurent On Fri, 2015-02-20 at 16:49 -0500, Yuri Weinstein wrote: Team leads, Please review QE validation results summary in http://tracker.ceph.com/issues/10501 Loic - this RC looks ready for release (in my opinion) ! Thx YuriW - Original Message - From: Yuri Weinstein ywein...@redhat.com To: Loic Dachary l...@dachary.org Cc: Ceph Development ceph-devel@vger.kernel.org Sent: Monday, February 16, 2015 9:33:44 AM Subject: Re: giant integration branch for v0.87.1 ready for QE I have completed suites execution on giant branch (v0.87.1 RC) All results are summarized in http://tracker.ceph.com/issues/10501 under QE VALIDATION section. Some suites had to be run more then ones due to environment noise. Two suites are being re-run now - upgrade:firefly-x and powecycle. Next steps: - the team leads to review/confirm results - Loic - can you review and triage issues as needed. - two suites require results analysis: multimds rados (two known tickets, but need more checking) ## 10209, 9891 krbd (two new tickets, but need more checking) ## 10889, 10890 Thx YuriW - Original Message - From: Loic Dachary l...@dachary.org To: Yuri Weinstein ywein...@redhat.com Cc: Ceph Development ceph-devel@vger.kernel.org Sent: Wednesday, February 11, 2015 7:30:06 AM Subject: Re: giant integration branch for v0.87.1 ready for QE Hi Yuri, The giant-backports pull requests were merged into https://github.com/ceph/ceph/tree/giant which is not ready for testing. For the record, the head is https://github.com/ceph/ceph/commit/78c71b9200da5e7d832ec58765478404d31ae6b5 Cheers On 10/02/2015 18:20, Loic Dachary wrote: Hi Yuri, The giant integration branch for v0.87.1 as found at https://github.com/ceph/ceph/commits/giant-backports has been approved by Yehuda, Josh and Sam and is ready for QE. For the record, the head is https://github.com/ceph/ceph/commit/6b08a729540c61f3c8b15c5a3ce9382634bf800c Cheers -- Loïc Dachary, Artisan Logiciel Libre signature.asc Description: OpenPGP digital signature
