from:"Michael Kolomiets"

[ceph-users] Cephfs: sporadic damages uploaded files

2014-08-27 Thread Michael Kolomiets

Hi!
I use ceph pool mounted via cephfs for cloudstack secondary storage
and have problem with consistency of files stored on it.
I have uploaded file for three time and checked it, but at each time i
have got different checksum (at second time it was a valid checksum).
Each try of upload gave permanent result (I checked twice each time),
but each next try did different one.
Please help me with finding of PoF.

root@lw01p01-mgmt01:/export/secondary# uname -a
Linux lw01p01-mgmt01 3.14.1-031401-generic #201404141220 SMP Mon Apr
14 16:21:48 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux


root@lw01p01-mgmt01:/export/secondary# ceph status
cluster e405d974-3fb6-42c8-b34a-a0ac5a1fef3a
 health HEALTH_OK
 monmap e1: 3 mons at
{lw01p01-node01=10.0.15.1:6789/0,lw01p01-node02=10.0.15.2:6789/0,lw01p01-node03=10.0.15.3:6789/0},
election epoch 52, quorum 0,1,2
lw01p01-node01,lw01p01-node02,lw01p01-node03
 mdsmap e17: 1/1/1 up {0=lw01p01-node01=up:active}
 osdmap e338: 20 osds: 20 up, 20 in
  pgmap v160161: 656 pgs, 6 pools, 30505 MB data, 8159 objects
61377 MB used, 25512 GB / 25572 GB avail
 656 active+clean
  client io 0 B/s rd, 1418 B/s wr, 1 op/s


root@lw01p01-mgmt01:/export/secondary# mount | grep ceph
10.0.15.1:/ on /export/secondary type ceph
(name=cloudstack,key=client.cloudstack)


root@lw01p01-mgmt01:/export/secondary# cephfs /export/secondary show_layout
layout.data_pool: 3
layout.object_size:   4194304
layout.stripe_unit:   4194304
layout.stripe_count:  1


root@lw01p01-mgmt01:/export/secondary# wget
http://lw01p01-templates01.example.com/ISO/XXX.iso
--2014-08-27 10:12:39--  http://lw01p01-templates01.example.com/ISO/XXX.iso
Resolving lw01p01-templates01.example.com
(lw01p01-templates01.example.com)... 10.0.15.1
Connecting to lw01p01-templates01.example.com
(lw01p01-templates01.example.com)|10.0.15.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3249803264 (3.0G) [application/x-iso9660-image]
Saving to: ‘XXX.iso’

100%[=]
3,249,803,264  179MB/s   in 18s

2014-08-27 10:12:57 (173 MB/s) - ‘XXX.iso’ saved [3249803264/3249803264]

root@lw01p01-mgmt01:/export/secondary# md5sum XXX.iso
45b940c6cb76ed0e76c9fac4cba01c3c  XXX.iso
root@lw01p01-mgmt01:/export/secondary# md5sum XXX.iso
45b940c6cb76ed0e76c9fac4cba01c3c  XXX.iso

root@lw01p01-mgmt01:/export/secondary# wget
http://lw01p01-templates01.example.com/ISO/XXX.iso
--2014-08-27 10:14:11--  http://lw01p01-templates01.example.com/ISO/XXX.iso
Resolving lw01p01-templates01.example.com
(lw01p01-templates01.example.com)... 10.0.15.1
Connecting to lw01p01-templates01.example.com
(lw01p01-templates01.example.com)|10.0.15.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3249803264 (3.0G) [application/x-iso9660-image]
Saving to: ‘XXX.iso.1’

100%[=]
3,249,803,264  154MB/s   in 19s

2014-08-27 10:14:30 (161 MB/s) - ‘XXX.iso.1’ saved [3249803264/3249803264]

root@lw01p01-mgmt01:/export/secondary# md5sum XXX.iso.1
5488d85797cd53d1d1562e73122522c1  XXX.iso.1
root@lw01p01-mgmt01:/export/secondary# md5sum XXX.iso.1
5488d85797cd53d1d1562e73122522c1  XXX.iso.1

root@lw01p01-mgmt01:/export/secondary# wget
http://lw01p01-templates01.example.com/ISO/XXX.iso
--2014-08-27 10:15:23--  http://lw01p01-templates01.example.com/ISO/XXX.iso
Resolving lw01p01-templates01.example.com
(lw01p01-templates01.example.com)... 10.0.15.1
Connecting to lw01p01-templates01.example.com
(lw01p01-templates01.example.com)|10.0.15.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3249803264 (3.0G) [application/x-iso9660-image]
Saving to: ‘XXX.iso.2’

100%[=]
3,249,803,264  160MB/s   in 20s

2014-08-27 10:15:44 (152 MB/s) - ‘XXX.iso.2’ saved [3249803264/3249803264]

root@lw01p01-mgmt01:/export/secondary# md5sum XXX.iso.2
5e28d425f828440b025d769609c5bb41  XXX.iso.2
root@lw01p01-mgmt01:/export/secondary# md5sum XXX.iso.2
5e28d425f828440b025d769609c5bb41  XXX.iso.2

-- 
Michael Kolomiets
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cephfs: sporadic damages uploaded files

2014-08-27 Thread Michael Kolomiets

Hi!
I checked if XXX.iso.2 contains zeros, it isn't. Could be cause is
caching and/or buffering on the client?

root@lw01p01-mgmt01:/export/secondary# dd if=XXX.iso.2 bs=1M count=1000 | md5sum
2245e239a9e8f3387adafc7319191015  -
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 7.81345 s, 134 MB/s

root@lw01p01-mgmt01:/export/secondary# dd if=/dev/zero bs=1M count=1000 | md5sum
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 2.16045 s, 485 MB/s
e5c834fbdaa6bfd8eac5eb9404eefdd4  -

2014-08-27 12:24 GMT+03:00 Yan, Zheng uker...@gmail.com:
 I suspect the client does not have permission to write to pool 3.
 could you check if the contents of XXX.iso.2 are all zeros.

 Yan, Zheng

 On Wed, Aug 27, 2014 at 5:05 PM, Michael Kolomiets
 michael.kolomi...@gmail.com wrote:
 Hi!
 I use ceph pool mounted via cephfs for cloudstack secondary storage
 and have problem with consistency of files stored on it.
 I have uploaded file for three time and checked it, but at each time i
 have got different checksum (at second time it was a valid checksum).
 Each try of upload gave permanent result (I checked twice each time),
 but each next try did different one.
 Please help me with finding of PoF.

 root@lw01p01-mgmt01:/export/secondary# uname -a
 Linux lw01p01-mgmt01 3.14.1-031401-generic #201404141220 SMP Mon Apr
 14 16:21:48 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux


 root@lw01p01-mgmt01:/export/secondary# ceph status
 cluster e405d974-3fb6-42c8-b34a-a0ac5a1fef3a
  health HEALTH_OK
  monmap e1: 3 mons at
 {lw01p01-node01=10.0.15.1:6789/0,lw01p01-node02=10.0.15.2:6789/0,lw01p01-node03=10.0.15.3:6789/0},
 election epoch 52, quorum 0,1,2
 lw01p01-node01,lw01p01-node02,lw01p01-node03
  mdsmap e17: 1/1/1 up {0=lw01p01-node01=up:active}
  osdmap e338: 20 osds: 20 up, 20 in
   pgmap v160161: 656 pgs, 6 pools, 30505 MB data, 8159 objects
 61377 MB used, 25512 GB / 25572 GB avail
  656 active+clean
   client io 0 B/s rd, 1418 B/s wr, 1 op/s


 root@lw01p01-mgmt01:/export/secondary# mount | grep ceph
 10.0.15.1:/ on /export/secondary type ceph
 (name=cloudstack,key=client.cloudstack)


 root@lw01p01-mgmt01:/export/secondary# cephfs /export/secondary show_layout
 layout.data_pool: 3
 layout.object_size:   4194304
 layout.stripe_unit:   4194304
 layout.stripe_count:  1


 root@lw01p01-mgmt01:/export/secondary# wget
 http://lw01p01-templates01.example.com/ISO/XXX.iso
 --2014-08-27 10:12:39--  http://lw01p01-templates01.example.com/ISO/XXX.iso
 Resolving lw01p01-templates01.example.com
 (lw01p01-templates01.example.com)... 10.0.15.1
 Connecting to lw01p01-templates01.example.com
 (lw01p01-templates01.example.com)|10.0.15.1|:80... connected.
 HTTP request sent, awaiting response... 200 OK
 Length: 3249803264 (3.0G) [application/x-iso9660-image]
 Saving to: 'XXX.iso'

 100%[=]
 3,249,803,264  179MB/s   in 18s

 2014-08-27 10:12:57 (173 MB/s) - 'XXX.iso' saved [3249803264/3249803264]

 root@lw01p01-mgmt01:/export/secondary# md5sum XXX.iso
 45b940c6cb76ed0e76c9fac4cba01c3c  XXX.iso
 root@lw01p01-mgmt01:/export/secondary# md5sum XXX.iso
 45b940c6cb76ed0e76c9fac4cba01c3c  XXX.iso

 root@lw01p01-mgmt01:/export/secondary# wget
 http://lw01p01-templates01.example.com/ISO/XXX.iso
 --2014-08-27 10:14:11--  http://lw01p01-templates01.example.com/ISO/XXX.iso
 Resolving lw01p01-templates01.example.com
 (lw01p01-templates01.example.com)... 10.0.15.1
 Connecting to lw01p01-templates01.example.com
 (lw01p01-templates01.example.com)|10.0.15.1|:80... connected.
 HTTP request sent, awaiting response... 200 OK
 Length: 3249803264 (3.0G) [application/x-iso9660-image]
 Saving to: 'XXX.iso.1'

 100%[=]
 3,249,803,264  154MB/s   in 19s

 2014-08-27 10:14:30 (161 MB/s) - 'XXX.iso.1' saved [3249803264/3249803264]

 root@lw01p01-mgmt01:/export/secondary# md5sum XXX.iso.1
 5488d85797cd53d1d1562e73122522c1  XXX.iso.1
 root@lw01p01-mgmt01:/export/secondary# md5sum XXX.iso.1
 5488d85797cd53d1d1562e73122522c1  XXX.iso.1

 root@lw01p01-mgmt01:/export/secondary# wget
 http://lw01p01-templates01.example.com/ISO/XXX.iso
 --2014-08-27 10:15:23--  http://lw01p01-templates01.example.com/ISO/XXX.iso
 Resolving lw01p01-templates01.example.com
 (lw01p01-templates01.example.com)... 10.0.15.1
 Connecting to lw01p01-templates01.example.com
 (lw01p01-templates01.example.com)|10.0.15.1|:80... connected.
 HTTP request sent, awaiting response... 200 OK
 Length: 3249803264 (3.0G) [application/x-iso9660-image]
 Saving to: 'XXX.iso.2'

 100%[=]
 3,249,803,264  160MB/s   in 20s

 2014-08-27 10:15:44 (152 MB/s) - 'XXX.iso.2' saved [3249803264/3249803264]

 root@lw01p01-mgmt01:/export/secondary# md5sum XXX.iso.2
 5e28d425f828440b025d769609c5bb41  XXX.iso.2
 root@lw01p01-mgmt01:/export/secondary# md5sum XXX.iso.2

[ceph-users] Cache tiering and CRUSH map

2014-08-18 Thread Michael Kolomiets

Hi
I am trying to use cache tiering and read the topic about mapping OSD
with pools 
(http://ceph.com/docs/master/rados/operations/crush-map/#placing-different-pools-on-different-osds).
I can't realize why OSDs were splitted on spinner and SSD type on root
level of CRUSH map?

Is it possible to to use some location type under host level to group
OSDs by type and use then it in mapping rules?

-- 
Michael Kolomiets
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Cephfs: sporadic damages uploaded files

Re: [ceph-users] Cephfs: sporadic damages uploaded files

[ceph-users] Cache tiering and CRUSH map

3 matches

Site Navigation

Mail list logo

Footer information