from:"José M . Martín"

Re: [ceph-users] Ceph health warn MDS failing to respond to cache pressure

2017-05-12 Thread José M . Martín

Hi,

I'm having the same issues running MDS version 11.2.0 and kernel clients
4.10.

Regards
Jose

El 10/05/17 a las 09:11, gjprabu escribió:
> HI John,
>   
> Thanks for you reply , we are using below version for client and
> MDS (ceph version 10.2.2)
>
> Regards
> Prabu GJ
>
>
>  On Wed, 10 May 2017 12:29:06 +0530 *John Spray
> * wrote 
>
> On Thu, May 4, 2017 at 7:28 AM, gjprabu  > wrote:
> > Hi Team,
> >
> > We are running cephfs with 5 OSD and 3 Mon and 1 MDS. There is
> > Heath Warn "failing to respond to cache pressure" . Kindly
> advise to fix
> > this issue.
>
> This is usually due to buggy old clients, and occasionally due to a
> buggy old MDS. What client and MDS versions are you using?
>
> John
>
> >
> >
> > cluster b466e09c-f7ae-4e89-99a7-99d30eba0a13
> > health HEALTH_WARN
> > mds0: Client integ-hm8-1.csez.zohocorpin.com failing to respond
> > to cache pressure
> > mds0: Client integ-hm5 failing to respond to cache pressure
> > mds0: Client integ-hm9 failing to respond to cache pressure
> > mds0: Client integ-hm2 failing to respond to cache pressure
> > monmap e2: 3 mons at
> >
> 
> {intcfs-mon1=192.168.113.113:6789/0,intcfs-mon2=192.168.113.114:6789/0,intcfs-mon3=192.168.113.72:6789/0}
>
> > election epoch 16, quorum 0,1,2
> > intcfs-mon3,intcfs-mon1,intcfs-mon2
> > fsmap e79409: 1/1/1 up {0=intcfs-osd1=up:active}, 1 up:standby
> > osdmap e3343: 5 osds: 5 up, 5 in
> > flags sortbitwise
> > pgmap v13065759: 564 pgs, 3 pools, 5691 GB data, 12134 kobjects
> > 11567 GB used, 5145 GB / 16713 GB avail
> > 562 active+clean
> > 2 active+clean+scrubbing+deep
> > client io 8090 kB/s rd, 29032 kB/s wr, 25 op/s rd, 129 op/s wr
> >
> >
> > Regards
> > Prabu GJ
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com 
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Minimize data lost with PG incomplete

2017-02-01 Thread José M . Martín

It doesn't matter anymore. MDS has crushed and it is stuck in rejoin
state in rejoin state.
Now, thinking in delete the pool and start again. Is it safe or
advisable use an erasured code pool for CephFS?

Thank you very much for your time. I like very much this software.

Cheers,
José

El 01/02/17 a las 14:29, José M. Martín escribió:
> Hi Maxime
>
> I have 3 of the original disks but I don't know which OSD correspond
> each one. Besides, I don't think I have enough technical skills to do
> that and I don't want to go worse...
> I'm trying to write a script that copy files from the damaged CephFS to
> a new location.
> Any help will be very gratefull
>
> José
>
>
> El 01/02/17 a las 07:56, Maxime Guyot escribió:
>> Hi José
>>
>> If you have some of the original OSDs (not zapped or erased) then you might 
>> be able to just re-add them to your cluster and have a happy cluster.
>> If you attempt the ceph_objectstore_tool –op export & import make sure to do 
>> it on a temporary OSD of weight 0 as recommended in the link provided.
>>
>> Either way and from what I can see inthe pg dump you provided, if you 
>> restore osd.0, osd.3, osd.20, osd.21 and osd.22 it should be enough to bring 
>> back the pg that are down.
>>
>> Cheers,
>>  
>> On 31/01/17 11:48, "ceph-users on behalf of José M. Martín" 
>>  
>> wrote:
>>
>> Any idea of how could I recover files from the filesystem mount?
>> Doing a cp, it hungs when find a damaged file/folder. I would be happy
>> getting no damaged files
>> 
>> Thanks
>> 
>> El 31/01/17 a las 11:19, José M. Martín escribió:
>> > Thanks.
>> > I just realized I keep some of the original OSD. If it contains some of
>> > the incomplete PGs , would be possible to add then into the new disks?
>> > Maybe following this steps? 
>> http://ceph.com/community/incomplete-pgs-oh-my/
>> >
>> > El 31/01/17 a las 10:44, Maxime Guyot escribió:
>> >> Hi José,
>> >>
>> >> Too late, but you could have updated the CRUSHmap *before* moving the 
>> disks. Something like: “ceph osd crush set osd.0 0.90329 root=default 
>> rack=sala2.2  host=loki05” would move the osd.0 to loki05 and would trigger 
>> the appropriate PG movements before any physical move. Then the physical 
>> move is done as usual: set noout, stop osd, physically move, active osd, 
>> unnset noout.
>> >>
>> >> It’s a way to trigger the data movement overnight (maybe with a cron) 
>> and do the physical move at your own convenience in the morning.
>> >>
>> >> Cheers, 
>> >> Maxime 
>> >>
>> >> On 31/01/17 10:35, "ceph-users on behalf of José M. Martín" 
>>  
>> wrote:
>>     >>
>> >> Already min_size = 1
>> >> 
>> >> Thanks,
>> >> Jose M. Martín
>> >> 
>> >> El 31/01/17 a las 09:44, Henrik Korkuc escribió:
>> >> > I am not sure about "incomplete" part out of my head, but you 
>> can try
>> >> > setting min_size to 1 for pools toreactivate some PG, if they 
>> are
>> >> > down/inactive due to missing replicas.
>> >> >
>> >> > On 17-01-31 10:24, José M. Martín wrote:
>> >> >> # ceph -s
>> >> >>  cluster 29a91870-2ed2-40dc-969e-07b22f37928b
>> >> >>   health HEALTH_ERR
>> >> >>  clock skew detected on mon.loki04
>> >> >>  155 pgs are stuck inactive for more than 300 
>> seconds
>> >> >>  7 pgs backfill_toofull
>> >> >>  1028 pgs backfill_wait
>> >> >>  48 pgs backfilling
>> >> >>  892 pgs degraded
>> >> >>  20 pgs down
>> >> >>  153 pgs incomplete
>> >> >>  2 pgs peering
>> >> >>  155 pgs stuck inactive
>> >> >>  1077 pgs stuck unclean
>> >> >>  892 pgs undersized
>> >> >>  1471 requests are blocked > 32 sec
>> >> >>  rec

Re: [ceph-users] Minimize data lost with PG incomplete

2017-02-01 Thread José M . Martín

Hi Maxime

I have 3 of the original disks but I don't know which OSD correspond
each one. Besides, I don't think I have enough technical skills to do
that and I don't want to go worse...
I'm trying to write a script that copy files from the damaged CephFS to
a new location.
Any help will be very gratefull

José


El 01/02/17 a las 07:56, Maxime Guyot escribió:
> Hi José
>
> If you have some of the original OSDs (not zapped or erased) then you might 
> be able to just re-add them to your cluster and have a happy cluster.
> If you attempt the ceph_objectstore_tool –op export & import make sure to do 
> it on a temporary OSD of weight 0 as recommended in the link provided.
>
> Either way and from what I can see inthe pg dump you provided, if you restore 
> osd.0, osd.3, osd.20, osd.21 and osd.22 it should be enough to bring back the 
> pg that are down.
>
> Cheers,
>  
> On 31/01/17 11:48, "ceph-users on behalf of José M. Martín" 
>  wrote:
>
> Any idea of how could I recover files from the filesystem mount?
> Doing a cp, it hungs when find a damaged file/folder. I would be happy
>     getting no damaged files
> 
> Thanks
> 
> El 31/01/17 a las 11:19, José M. Martín escribió:
> > Thanks.
> > I just realized I keep some of the original OSD. If it contains some of
> > the incomplete PGs , would be possible to add then into the new disks?
> > Maybe following this steps? 
> http://ceph.com/community/incomplete-pgs-oh-my/
> >
> > El 31/01/17 a las 10:44, Maxime Guyot escribió:
> >> Hi José,
> >>
> >> Too late, but you could have updated the CRUSHmap *before* moving the 
> disks. Something like: “ceph osd crush set osd.0 0.90329 root=default 
> rack=sala2.2  host=loki05” would move the osd.0 to loki05 and would trigger 
> the appropriate PG movements before any physical move. Then the physical move 
> is done as usual: set noout, stop osd, physically move, active osd, unnset 
> noout.
> >>
> >> It’s a way to trigger the data movement overnight (maybe with a cron) 
> and do the physical move at your own convenience in the morning.
> >>
> >> Cheers, 
> >> Maxime 
> >>
> >> On 31/01/17 10:35, "ceph-users on behalf of José M. Martín" 
>  wrote:
> >>
> >> Already min_size = 1
> >> 
> >> Thanks,
> >> Jose M. Martín
> >> 
> >> El 31/01/17 a las 09:44, Henrik Korkuc escribió:
> >> > I am not sure about "incomplete" part out of my head, but you 
> can try
> >> > setting min_size to 1 for pools toreactivate some PG, if they are
> >> > down/inactive due to missing replicas.
> >> >
> >> > On 17-01-31 10:24, José M. Martín wrote:
> >> >> # ceph -s
> >> >>  cluster 29a91870-2ed2-40dc-969e-07b22f37928b
> >> >>   health HEALTH_ERR
> >> >>  clock skew detected on mon.loki04
> >> >>  155 pgs are stuck inactive for more than 300 
> seconds
> >> >>  7 pgs backfill_toofull
> >> >>  1028 pgs backfill_wait
> >> >>  48 pgs backfilling
> >> >>  892 pgs degraded
> >> >>  20 pgs down
> >> >>  153 pgs incomplete
> >> >>  2 pgs peering
> >> >>  155 pgs stuck inactive
> >> >>  1077 pgs stuck unclean
> >> >>  892 pgs undersized
> >> >>  1471 requests are blocked > 32 sec
> >> >>  recovery 3195781/36460868 objects degraded (8.765%)
> >> >>  recovery 5079026/36460868 objects misplaced 
> (13.930%)
> >> >>  mds0: Behind on trimming (175/30)
> >> >>  noscrub,nodeep-scrub flag(s) set
> >> >>  Monitor clock skew detected
> >> >>   monmap e5: 5 mons at
> >> >> 
> {loki01=192.168.3.151:6789/0,loki02=192.168.3.152:6789/0,loki03=192.168.3.153:6789/0,loki04=192.168.3.154:6789/0,loki05=192.168.3.155:6789/0}
> >> >>
> >> >>  election epoch 4028, quorum 0,1,2,3

Re: [ceph-users] Minimize data lost with PG incomplete

2017-01-31 Thread José M . Martín

Any idea of how could I recover files from the filesystem mount?
Doing a cp, it hungs when find a damaged file/folder. I would be happy
getting no damaged files

Thanks

El 31/01/17 a las 11:19, José M. Martín escribió:
> Thanks.
> I just realized I keep some of the original OSD. If it contains some of
> the incomplete PGs , would be possible to add then into the new disks?
> Maybe following this steps? http://ceph.com/community/incomplete-pgs-oh-my/
>
> El 31/01/17 a las 10:44, Maxime Guyot escribió:
>> Hi José,
>>
>> Too late, but you could have updated the CRUSHmap *before* moving the disks. 
>> Something like: “ceph osd crush set osd.0 0.90329 root=default rack=sala2.2  
>> host=loki05” would move the osd.0 to loki05 and would trigger the 
>> appropriate PG movements before any physical move. Then the physical move is 
>> done as usual: set noout, stop osd, physically move, active osd, unnset 
>> noout.
>>
>> It’s a way to trigger the data movement overnight (maybe with a cron) and do 
>> the physical move at your own convenience in the morning.
>>
>> Cheers, 
>> Maxime 
>>
>> On 31/01/17 10:35, "ceph-users on behalf of José M. Martín" 
>>  
>> wrote:
>>
>> Already min_size = 1
>> 
>> Thanks,
>> Jose M. Martín
>> 
>> El 31/01/17 a las 09:44, Henrik Korkuc escribió:
>> > I am not sure about "incomplete" part out of my head, but you can try
>> > setting min_size to 1 for pools toreactivate some PG, if they are
>> > down/inactive due to missing replicas.
>> >
>> > On 17-01-31 10:24, José M. Martín wrote:
>> >> # ceph -s
>> >>  cluster 29a91870-2ed2-40dc-969e-07b22f37928b
>> >>   health HEALTH_ERR
>> >>  clock skew detected on mon.loki04
>> >>  155 pgs are stuck inactive for more than 300 seconds
>> >>  7 pgs backfill_toofull
>> >>  1028 pgs backfill_wait
>> >>  48 pgs backfilling
>> >>  892 pgs degraded
>> >>  20 pgs down
>> >>  153 pgs incomplete
>> >>  2 pgs peering
>> >>  155 pgs stuck inactive
>> >>  1077 pgs stuck unclean
>> >>  892 pgs undersized
>> >>  1471 requests are blocked > 32 sec
>> >>  recovery 3195781/36460868 objects degraded (8.765%)
>> >>  recovery 5079026/36460868 objects misplaced (13.930%)
>> >>  mds0: Behind on trimming (175/30)
>> >>  noscrub,nodeep-scrub flag(s) set
>> >>  Monitor clock skew detected
>> >>   monmap e5: 5 mons at
>> >> 
>> {loki01=192.168.3.151:6789/0,loki02=192.168.3.152:6789/0,loki03=192.168.3.153:6789/0,loki04=192.168.3.154:6789/0,loki05=192.168.3.155:6789/0}
>> >>
>> >>  election epoch 4028, quorum 0,1,2,3,4
>> >> loki01,loki02,loki03,loki04,loki05
>> >>fsmap e95494: 1/1/1 up {0=zeus2=up:active}, 1 up:standby
>> >>   osdmap e275373: 42 osds: 42 up, 42 in; 1077 remapped pgs
>> >>  flags noscrub,nodeep-scrub
>> >>pgmap v36642778: 4872 pgs, 4 pools, 24801 GB data, 17087 
>> kobjects
>> >>  45892 GB used, 34024 GB / 79916 GB avail
>> >>  3195781/36460868 objects degraded (8.765%)
>> >>  5079026/36460868 objects misplaced (13.930%)
>> >>  3640 active+clean
>> >>   838 
>> active+undersized+degraded+remapped+wait_backfill
>> >>   184 active+remapped+wait_backfill
>> >>   134 incomplete
>> >>48 active+undersized+degraded+remapped+backfilling
>> >>19 down+incomplete
>> >> 6
>> >> active+undersized+degraded+remapped+wait_backfill+backfill_toofull
>> >> 1 active+remapped+backfill_toofull
>> >> 1 peering
>> >> 1 down+peering
>> >> recovery io 93909 kB/s, 10 keys/s, 67 objects/s
>> >>
>> >>
>> >>
>> >> # ceph osd tree
>&

Re: [ceph-users] Minimize data lost with PG incomplete

2017-01-31 Thread José M . Martín

Thanks.
I just realized I keep some of the original OSD. If it contains some of
the incomplete PGs , would be possible to add then into the new disks?
Maybe following this steps? http://ceph.com/community/incomplete-pgs-oh-my/

El 31/01/17 a las 10:44, Maxime Guyot escribió:
> Hi José,
>
> Too late, but you could have updated the CRUSHmap *before* moving the disks. 
> Something like: “ceph osd crush set osd.0 0.90329 root=default rack=sala2.2  
> host=loki05” would move the osd.0 to loki05 and would trigger the appropriate 
> PG movements before any physical move. Then the physical move is done as 
> usual: set noout, stop osd, physically move, active osd, unnset noout.
>
> It’s a way to trigger the data movement overnight (maybe with a cron) and do 
> the physical move at your own convenience in the morning.
>
> Cheers, 
> Maxime 
>
> On 31/01/17 10:35, "ceph-users on behalf of José M. Martín" 
>  wrote:
>
> Already min_size = 1
> 
> Thanks,
> Jose M. Martín
> 
> El 31/01/17 a las 09:44, Henrik Korkuc escribió:
> > I am not sure about "incomplete" part out of my head, but you can try
> > setting min_size to 1 for pools toreactivate some PG, if they are
> > down/inactive due to missing replicas.
> >
> > On 17-01-31 10:24, José M. Martín wrote:
> >> # ceph -s
> >>  cluster 29a91870-2ed2-40dc-969e-07b22f37928b
> >>   health HEALTH_ERR
> >>  clock skew detected on mon.loki04
> >>  155 pgs are stuck inactive for more than 300 seconds
> >>  7 pgs backfill_toofull
> >>  1028 pgs backfill_wait
> >>  48 pgs backfilling
> >>  892 pgs degraded
> >>  20 pgs down
> >>  153 pgs incomplete
> >>  2 pgs peering
> >>  155 pgs stuck inactive
> >>  1077 pgs stuck unclean
> >>  892 pgs undersized
> >>  1471 requests are blocked > 32 sec
> >>  recovery 3195781/36460868 objects degraded (8.765%)
> >>  recovery 5079026/36460868 objects misplaced (13.930%)
> >>  mds0: Behind on trimming (175/30)
> >>  noscrub,nodeep-scrub flag(s) set
> >>  Monitor clock skew detected
> >>   monmap e5: 5 mons at
> >> 
> {loki01=192.168.3.151:6789/0,loki02=192.168.3.152:6789/0,loki03=192.168.3.153:6789/0,loki04=192.168.3.154:6789/0,loki05=192.168.3.155:6789/0}
> >>
> >>  election epoch 4028, quorum 0,1,2,3,4
> >> loki01,loki02,loki03,loki04,loki05
> >>fsmap e95494: 1/1/1 up {0=zeus2=up:active}, 1 up:standby
> >>   osdmap e275373: 42 osds: 42 up, 42 in; 1077 remapped pgs
> >>  flags noscrub,nodeep-scrub
> >>pgmap v36642778: 4872 pgs, 4 pools, 24801 GB data, 17087 
> kobjects
> >>  45892 GB used, 34024 GB / 79916 GB avail
> >>  3195781/36460868 objects degraded (8.765%)
> >>  5079026/36460868 objects misplaced (13.930%)
> >>  3640 active+clean
> >>   838 active+undersized+degraded+remapped+wait_backfill
> >>   184 active+remapped+wait_backfill
> >>   134 incomplete
> >>48 active+undersized+degraded+remapped+backfilling
> >>19 down+incomplete
> >> 6
> >> active+undersized+degraded+remapped+wait_backfill+backfill_toofull
> >> 1 active+remapped+backfill_toofull
> >> 1 peering
> >> 1 down+peering
> >> recovery io 93909 kB/s, 10 keys/s, 67 objects/s
> >>
> >>
> >>
> >> # ceph osd tree
> >> ID  WEIGHT   TYPE NAME   UP/DOWN REWEIGHT PRIMARY-AFFINITY
> >>   -1 77.22777 root default
> >>   -9 27.14778 rack sala1
> >>   -2  5.41974 host loki01
> >>   14  0.90329 osd.14   up  1.0  1.0
> >>   15  0.90329 osd.15   up  1.0  1.0
> >>   16  0.90329 osd.16   up  1.0  1.0
> >>   17  0.90329 osd.17   up  1.0  1.0
> >>   18  0

Re: [ceph-users] Minimize data lost with PG incomplete

2017-01-31 Thread José M . Martín

Already min_size = 1

Thanks,
Jose M. Martín

El 31/01/17 a las 09:44, Henrik Korkuc escribió:
> I am not sure about "incomplete" part out of my head, but you can try
> setting min_size to 1 for pools toreactivate some PG, if they are
> down/inactive due to missing replicas.
>
> On 17-01-31 10:24, José M. Martín wrote:
>> # ceph -s
>>  cluster 29a91870-2ed2-40dc-969e-07b22f37928b
>>   health HEALTH_ERR
>>  clock skew detected on mon.loki04
>>  155 pgs are stuck inactive for more than 300 seconds
>>  7 pgs backfill_toofull
>>  1028 pgs backfill_wait
>>  48 pgs backfilling
>>  892 pgs degraded
>>  20 pgs down
>>  153 pgs incomplete
>>  2 pgs peering
>>  155 pgs stuck inactive
>>  1077 pgs stuck unclean
>>  892 pgs undersized
>>  1471 requests are blocked > 32 sec
>>  recovery 3195781/36460868 objects degraded (8.765%)
>>  recovery 5079026/36460868 objects misplaced (13.930%)
>>  mds0: Behind on trimming (175/30)
>>  noscrub,nodeep-scrub flag(s) set
>>  Monitor clock skew detected
>>   monmap e5: 5 mons at
>> {loki01=192.168.3.151:6789/0,loki02=192.168.3.152:6789/0,loki03=192.168.3.153:6789/0,loki04=192.168.3.154:6789/0,loki05=192.168.3.155:6789/0}
>>
>>  election epoch 4028, quorum 0,1,2,3,4
>> loki01,loki02,loki03,loki04,loki05
>>fsmap e95494: 1/1/1 up {0=zeus2=up:active}, 1 up:standby
>>   osdmap e275373: 42 osds: 42 up, 42 in; 1077 remapped pgs
>>  flags noscrub,nodeep-scrub
>>pgmap v36642778: 4872 pgs, 4 pools, 24801 GB data, 17087 kobjects
>>  45892 GB used, 34024 GB / 79916 GB avail
>>  3195781/36460868 objects degraded (8.765%)
>>  5079026/36460868 objects misplaced (13.930%)
>>  3640 active+clean
>>   838 active+undersized+degraded+remapped+wait_backfill
>>   184 active+remapped+wait_backfill
>>   134 incomplete
>>48 active+undersized+degraded+remapped+backfilling
>>19 down+incomplete
>> 6
>> active+undersized+degraded+remapped+wait_backfill+backfill_toofull
>> 1 active+remapped+backfill_toofull
>> 1 peering
>> 1 down+peering
>> recovery io 93909 kB/s, 10 keys/s, 67 objects/s
>>
>>
>>
>> # ceph osd tree
>> ID  WEIGHT   TYPE NAME   UP/DOWN REWEIGHT PRIMARY-AFFINITY
>>   -1 77.22777 root default
>>   -9 27.14778 rack sala1
>>   -2  5.41974 host loki01
>>   14  0.90329 osd.14   up  1.0  1.0
>>   15  0.90329 osd.15   up  1.0  1.0
>>   16  0.90329 osd.16   up  1.0  1.0
>>   17  0.90329 osd.17   up  1.0  1.0
>>   18  0.90329 osd.18   up  1.0  1.0
>>   25  0.90329 osd.25   up  1.0  1.0
>>   -4  3.61316 host loki03
>>0  0.90329 osd.0up  1.0  1.0
>>2  0.90329 osd.2up  1.0  1.0
>>   20  0.90329 osd.20   up  1.0  1.0
>>   24  0.90329 osd.24   up  1.0  1.0
>>   -3  9.05714 host loki02
>>1  0.90300 osd.1up  0.90002  1.0
>>   31  2.72198 osd.31   up  1.0  1.0
>>   29  0.90329 osd.29   up  1.0  1.0
>>   30  0.90329 osd.30   up  1.0  1.0
>>   33  0.90329 osd.33   up  1.0  1.0
>>   32  2.72229 osd.32   up  1.0  1.0
>>   -5  9.05774 host loki04
>>3  0.90329 osd.3up  1.0  1.0
>>   19  0.90329 osd.19   up  1.0  1.0
>>   21  0.90329 osd.21   up  1.0  1.0
>>   22  0.90329 osd.22   up  1.0  1.0
>>   23  2.72229 osd.23   up  1.0  1.0
>>   28  2.72229 osd.28   up  1.0  1.0
>> -10 24.61000 rack sala2.2
>>   -6 24.61000 host loki05
>>5  2.73000

Re: [ceph-users] Minimize data lost with PG incomplete

2017-01-31 Thread José M . Martín

000 osd.35   up  1.0  1.0
 36  2.73000 osd.36   up  1.0  1.0
 37  2.73000 osd.37   up  1.0  1.0
 38  2.73000 osd.38   up  1.0  1.0
 39  2.73000 osd.39   up  1.0  1.0
 40  2.73000 osd.40   up  1.0  1.0
 43  2.73000 osd.43   up  1.0  1.0
 42  0.90999 osd.42   up  1.0  1.0
 41  2.71999 osd.41   up  1.0  1.0

# ceph pg dump
You can find it in this link:
http://ergodic.ugr.es/pgdumpoutput.txt

What I did:
My cluster is  heterogeneous, having old oss nodes with 1TB disks and
new ones with 3TB. I was having problems with balance, some 1TB osd got
nearly full meanwhile there was plenty of space in others. My plan was
changing some disks to another one biggers. I started the process with
no problems, changing one disk. Reweight to 0.0, wait for rebalance, and
removed.
After that, searching for my problem, I read about straw2. Then, I
changed the algorithm editing the crush map and some data movement did.
My setup was not optimal, I had the journal in the xfs filesystem, so I
decided to change it also. First, I did it slowly, disk by disk, but as
rebalance take much time and my group was pushing me to finish quickly,
I did
ceph osd out osd.id
ceph osd crush remove osd.id
ceph auth del osd.id
ceph osd rm id

Then umount the disks, and using ceph-deploy add then again
ceph-deploy disk zap loki01:/dev/sda
ceph-deploy osd create loki01:/dev/sda

For every disk in rack "sala1". First, I finished loki02. Then, I did
this steps en loki04, loki01 and loki03 at the same time.

Thanks,
--
José M. Martín

El 31/01/17 a las 00:43, Shinobu Kinjo escribió:
> First off, the followings, please.
>
>  * ceph -s
>  * ceph osd tree
>  * ceph pg dump
>
> and
>
>  * what you actually did with exact commands.
>
> Regards,
>
> On Tue, Jan 31, 2017 at 6:10 AM, José M. Martín  
> wrote:
>> Dear list,
>>
>> I'm having some big problems with my setup.
>>
>> I was trying to increase the global capacity by changing some osds by
>> bigger ones. I changed them without wait the rebalance process finished,
>> thinking the replicas were saved in other buckets, but I found a lot of
>> PGs incomplete, so replicas of a PG were placed in a same bucket. I have
>> assumed I have lost data because I zapped the disks and used in other tasks.
>>
>> My question is: what should I do to recover as much data as possible?
>> I'm using the filesystem and RBD.
>>
>> Thank you so much,
>>
>> --
>>
>> Jose M. Martín
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Minimize data lost with PG incomplete

2017-01-30 Thread José M . Martín

Dear list,

I'm having some big problems with my setup.

I was trying to increase the global capacity by changing some osds by
bigger ones. I changed them without wait the rebalance process finished,
thinking the replicas were saved in other buckets, but I found a lot of
PGs incomplete, so replicas of a PG were placed in a same bucket. I have
assumed I have lost data because I zapped the disks and used in other tasks.

My question is: what should I do to recover as much data as possible?
I'm using the filesystem and RBD.

Thank you so much,

--

Jose M. Martín


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph health warn MDS failing to respond to cache pressure

Re: [ceph-users] Minimize data lost with PG incomplete

Re: [ceph-users] Minimize data lost with PG incomplete

Re: [ceph-users] Minimize data lost with PG incomplete

Re: [ceph-users] Minimize data lost with PG incomplete

Re: [ceph-users] Minimize data lost with PG incomplete

Re: [ceph-users] Minimize data lost with PG incomplete

[ceph-users] Minimize data lost with PG incomplete

8 matches

Site Navigation

Mail list logo

Footer information