Probably a case of trying to read too fast. Sorry about that. As far as your theory on the cache pool, I haven't tried that, but my gut feeling is that it won't help as much as having the journal on the SSD. The Cache tier isn't trying to collate writes, not like the journal is doing. Then on the spindle you are having to write to two very different parts of the drive for every piece of data, although this is somewhat reduced by the journal, I feel it will still be significant. When I see writes coming off my SSD journals to the spindles, I'm still getting a lot of merged IO (at least during a backfill/recovery). I'm interested in your results.
As far as the foreign journal, I would run dd over the journal partition and try it again. It sounds like something didn't get cleaned up from a previous run. On Wed, Mar 25, 2015 at 11:14 AM, Antonio Messina <antonio.s.mess...@gmail.com> wrote: > On Wed, Mar 25, 2015 at 6:06 PM, Robert LeBlanc <rob...@leblancnet.us> wrote: >> I don't know much about ceph-deploy, but I know that ceph-disk has >> problems "automatically" adding an SSD OSD when there are journals of >> other disks already on it. I've had to partition the disk ahead of >> time and pass in the partitions to make ceph-disk work. > > This is not my case: the journal is created automatically by > ceph-deploy on the same disk, so that for each disk, /dev/sdX1 is the > data partition and /dev/sdX2 is the journal partition. This is also > what I want: I know there is a performance drop, but I expect it to be > mitigated by the cache tier. (and I plan to test both configuration > anyway) > >> Also, unless you are sure that the dev devices will be deterministicly >> named the same each time, I'd recommend you not use /dev/sd* for >> pointing to your journals. Instead use something that will always be >> the same, since Ceph with partition the disks with GPT, you can use >> the partuuid to point to the journal partition and it will always be >> right. A while back I used this to "fix" my journal links when I did >> it wrong. You will want to double check that it will work right for >> you. no warranty and all that jazz... > > Thank you for pointing this out, it's an important point. However, the > links are actually created using the partuuid. The command I posted in > my previous email included the output of a pair of nested "readlink" > in order to get the /dev/sd* names, because in this way it's easier to > see if there are duplicates and where :) > > The output of "ls -l /var/lib/ceph/osd/ceph-*/journal" is actually: > > lrwxrwxrwx 1 root root 58 Mar 25 11:38 > /var/lib/ceph/osd/ceph-0/journal -> > /dev/disk/by-partuuid/18305316-96b0-4654-aaad-7aeb891429f6 > lrwxrwxrwx 1 root root 58 Mar 25 11:49 > /var/lib/ceph/osd/ceph-7/journal -> > /dev/disk/by-partuuid/a263b19a-cb0d-4b4c-bd81-314619d5755d > lrwxrwxrwx 1 root root 58 Mar 25 12:21 > /var/lib/ceph/osd/ceph-14/journal -> > /dev/disk/by-partuuid/79734e0e-87dd-40c7-ba83-0d49695a75fb > lrwxrwxrwx 1 root root 58 Mar 25 12:31 > /var/lib/ceph/osd/ceph-21/journal -> > /dev/disk/by-partuuid/73a504bc-3179-43fd-942c-13c6bd8633c5 > lrwxrwxrwx 1 root root 58 Mar 25 12:42 > /var/lib/ceph/osd/ceph-28/journal -> > /dev/disk/by-partuuid/ecff10df-d757-4b1f-bef4-88dd84d84ef1 > lrwxrwxrwx 1 root root 58 Mar 25 12:52 > /var/lib/ceph/osd/ceph-35/journal -> > /dev/disk/by-partuuid/5be30238-3f07-4950-b39f-f5e4c7305e4c > lrwxrwxrwx 1 root root 58 Mar 25 13:02 > /var/lib/ceph/osd/ceph-42/journal -> > /dev/disk/by-partuuid/3cdb65f2-474c-47fb-8d07-83e7518418ff > lrwxrwxrwx 1 root root 58 Mar 25 13:12 > /var/lib/ceph/osd/ceph-49/journal -> > /dev/disk/by-partuuid/a47fe2b7-e375-4eea-b7a9-0354a24548dc > lrwxrwxrwx 1 root root 58 Mar 25 13:22 > /var/lib/ceph/osd/ceph-56/journal -> > /dev/disk/by-partuuid/fb42b7d6-bc6c-4063-8b73-29beb1f65107 > lrwxrwxrwx 1 root root 58 Mar 25 13:33 > /var/lib/ceph/osd/ceph-63/journal -> > /dev/disk/by-partuuid/72aff32b-ca56-4c25-b8ea-ff3aba8db507 > lrwxrwxrwx 1 root root 58 Mar 25 13:43 > /var/lib/ceph/osd/ceph-70/journal -> > /dev/disk/by-partuuid/b7c17a75-47cd-401e-b963-afe910612bd6 > lrwxrwxrwx 1 root root 58 Mar 25 13:53 > /var/lib/ceph/osd/ceph-77/journal -> > /dev/disk/by-partuuid/2c1c2501-fa82-4fc9-a586-03cc4d68faef > lrwxrwxrwx 1 root root 58 Mar 25 14:03 > /var/lib/ceph/osd/ceph-84/journal -> > /dev/disk/by-partuuid/46f619a5-3edf-44e9-99a6-24d98bcd174a > lrwxrwxrwx 1 root root 58 Mar 25 14:13 > /var/lib/ceph/osd/ceph-91/journal -> > /dev/disk/by-partuuid/5feef832-dd82-4aa0-9264-dc9496a3f93a > lrwxrwxrwx 1 root root 58 Mar 25 14:24 > /var/lib/ceph/osd/ceph-98/journal -> > /dev/disk/by-partuuid/055793a0-99d4-49c4-9698-bd8880c21d9c > lrwxrwxrwx 1 root root 58 Mar 25 14:34 > /var/lib/ceph/osd/ceph-105/journal -> > /dev/disk/by-partuuid/20547f26-6ef3-422b-9732-ad8b0b5b5379 > lrwxrwxrwx 1 root root 58 Mar 25 14:44 > /var/lib/ceph/osd/ceph-112/journal -> > /dev/disk/by-partuuid/2abea809-59c4-41da-bb52-28ef1911ec43 > lrwxrwxrwx 1 root root 58 Mar 25 14:54 > /var/lib/ceph/osd/ceph-119/journal -> > /dev/disk/by-partuuid/d8d15bb8-4b3d-4375-b6e1-62794971df7e > lrwxrwxrwx 1 root root 58 Mar 25 15:05 > /var/lib/ceph/osd/ceph-126/journal -> > /dev/disk/by-partuuid/ff6ee2b2-9c33-4902-a5e3-f6e9db5714e9 > lrwxrwxrwx 1 root root 58 Mar 25 15:15 > /var/lib/ceph/osd/ceph-133/journal -> > /dev/disk/by-partuuid/9faccb6e-ada9-4742-aa31-eb1308769205 > lrwxrwxrwx 1 root root 58 Mar 25 15:25 > /var/lib/ceph/osd/ceph-140/journal -> > /dev/disk/by-partuuid/2df13c88-ee58-4881-a373-a36a09fb6366 > lrwxrwxrwx 1 root root 58 Mar 25 15:36 > /var/lib/ceph/osd/ceph-147/journal -> > /dev/disk/by-partuuid/13cda9d1-0fec-40cc-a6fc-7cc56f7ffb78 > lrwxrwxrwx 1 root root 58 Mar 25 15:46 > /var/lib/ceph/osd/ceph-154/journal -> > /dev/disk/by-partuuid/5d37bfe9-c0f9-49e0-a951-b0ed04c5de51 > lrwxrwxrwx 1 root root 58 Mar 25 15:57 > /var/lib/ceph/osd/ceph-161/journal -> > /dev/disk/by-partuuid/d34f3abb-3fb7-4875-90d3-d2d3836f6e4d > lrwxrwxrwx 1 root root 58 Mar 25 16:07 > /var/lib/ceph/osd/ceph-168/journal -> > /dev/disk/by-partuuid/02c3db3e-159c-47d9-8a63-0389ea89fad1 > lrwxrwxrwx 1 root root 58 Mar 25 16:16 > /var/lib/ceph/osd/ceph-175/journal -> > /dev/disk/by-partuuid/8f98cee1-c275-41fc-8053-918242ed3526 > lrwxrwxrwx 1 root root 58 Mar 25 16:26 > /var/lib/ceph/osd/ceph-182/journal -> > /dev/disk/by-partuuid/468338ca-7a93-4f63-895b-73852fb78df6 > lrwxrwxrwx 1 root root 58 Mar 25 16:36 > /var/lib/ceph/osd/ceph-189/journal -> > /dev/disk/by-partuuid/e0df3d71-10e3-48d4-a877-2d0900187f30 > lrwxrwxrwx 1 root root 58 Mar 25 16:45 > /var/lib/ceph/osd/ceph-196/journal -> > /dev/disk/by-partuuid/fa830818-d4f7-46f8-8745-64f799f1a1c0 > lrwxrwxrwx 1 root root 58 Mar 25 16:55 > /var/lib/ceph/osd/ceph-203/journal -> > /dev/disk/by-partuuid/6c68f1bf-d96c-4598-82a5-07746ec55190 > > > Antonio > > -- > antonio.s.mess...@gmail.com > antonio.mess...@uzh.ch +41 (0)44 635 42 22 > S3IT: Service and Support for Science IT http://www.s3it.uzh.ch/ > University of Zurich > Winterthurerstrasse 190 > CH-8057 Zurich Switzerland _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
