On Tue, Mar 6, 2018 at 5:26 PM, Marco Baldini - H.S. Amiata < mbald...@hsamiata.it> wrote:
> Hi > > I monitor dmesg in each of the 3 nodes, no hardware issue reported. And > the problem happens with various different OSDs in different nodes, for me > it is clear it's not an hardware problem. > If you have osd_debug set to 25 or greater when you run the deep scrub you should get more information about the nature of the read error in the ReplicatedBackend::be_deep_scrub() function (assuming this is a replicated pool). This may create large logs so watch they don't exhaust storage. > Thanks for reply > > > > Il 05/03/2018 21:45, Vladimir Prokofev ha scritto: > > > always solved by ceph pg repair <PG> > That doesn't necessarily means that there's no hardware issue. In my case > repair also worked fine and returned cluster to OK state every time, but in > time faulty disk fail another scrub operation, and this repeated multiple > times before we replaced that disk. > One last thing to look into is dmesg at your OSD nodes. If there's a > hardware read error it will be logged in dmesg. > > 2018-03-05 18:26 GMT+03:00 Marco Baldini - H.S. Amiata < > mbald...@hsamiata.it>: > >> Hi and thanks for reply >> >> The OSDs are all healthy, in fact after a ceph pg repair <PG> the ceph >> health is back to OK and in the OSD log I see <PG> repair ok, 0 fixed >> >> The SMART data of the 3 OSDs seems fine >> >> *OSD.5* >> >> # ceph-disk list | grep osd.5 >> /dev/sdd1 ceph data, active, cluster ceph, osd.5, block /dev/sdd2 >> >> # smartctl -a /dev/sdd >> smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.13.13-6-pve] (local build) >> Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org >> >> === START OF INFORMATION SECTION === >> Model Family: Seagate Barracuda 7200.14 (AF) >> Device Model: ST1000DM003-1SB10C >> Serial Number: Z9A1MA1V >> LU WWN Device Id: 5 000c50 090c7028b >> Firmware Version: CC43 >> User Capacity: 1,000,204,886,016 bytes [1.00 TB] >> Sector Sizes: 512 bytes logical, 4096 bytes physical >> Rotation Rate: 7200 rpm >> Form Factor: 3.5 inches >> Device is: In smartctl database [for details use: -P show] >> ATA Version is: ATA8-ACS T13/1699-D revision 4 >> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) >> Local Time is: Mon Mar 5 16:17:22 2018 CET >> SMART support is: Available - device has SMART capability. >> SMART support is: Enabled >> >> === START OF READ SMART DATA SECTION === >> SMART overall-health self-assessment test result: PASSED >> >> General SMART Values: >> Offline data collection status: (0x82) Offline data collection activity >> was completed without error. >> Auto Offline Data Collection: Enabled. >> Self-test execution status: ( 0) The previous self-test routine >> completed >> without error or no self-test has ever >> been run. >> Total time to complete Offline >> data collection: ( 0) seconds. >> Offline data collection >> capabilities: (0x7b) SMART execute Offline immediate. >> Auto Offline data collection on/off >> support. >> Suspend Offline collection upon new >> command. >> Offline surface scan supported. >> Self-test supported. >> Conveyance Self-test supported. >> Selective Self-test supported. >> SMART capabilities: (0x0003) Saves SMART data before entering >> power-saving mode. >> Supports SMART auto save timer. >> Error logging capability: (0x01) Error logging supported. >> General Purpose Logging supported. >> Short self-test routine >> recommended polling time: ( 1) minutes. >> Extended self-test routine >> recommended polling time: ( 109) minutes. >> Conveyance self-test routine >> recommended polling time: ( 2) minutes. >> SCT capabilities: (0x1085) SCT Status supported. >> >> SMART Attributes Data Structure revision number: 10 >> Vendor Specific SMART Attributes with Thresholds: >> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED >> WHEN_FAILED RAW_VALUE >> 1 Raw_Read_Error_Rate 0x000f 082 063 006 Pre-fail Always >> - 193297722 >> 3 Spin_Up_Time 0x0003 097 097 000 Pre-fail Always >> - 0 >> 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always >> - 60 >> 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always >> - 0 >> 7 Seek_Error_Rate 0x000f 091 060 045 Pre-fail Always >> - 1451132477 >> 9 Power_On_Hours 0x0032 085 085 000 Old_age Always >> - 13283 >> 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always >> - 0 >> 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always >> - 61 >> 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always >> - 0 >> 184 End-to-End_Error 0x0032 100 100 099 Old_age Always >> - 0 >> 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always >> - 0 >> 188 Command_Timeout 0x0032 100 100 000 Old_age Always >> - 0 0 0 >> 189 High_Fly_Writes 0x003a 086 086 000 Old_age Always >> - 14 >> 190 Airflow_Temperature_Cel 0x0022 071 055 040 Old_age Always >> - 29 (Min/Max 23/32) >> 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always >> - 607 >> 194 Temperature_Celsius 0x0022 029 014 000 Old_age Always >> - 29 (0 14 0 0 0) >> 195 Hardware_ECC_Recovered 0x001a 004 001 000 Old_age Always >> - 193297722 >> 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always >> - 0 >> 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline >> - 0 >> 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always >> - 0 >> 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline >> - 13211h+23m+08.363s >> 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline >> - 53042120064 >> 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline >> - 170788993187 >> >> *OSD.4* >> >> # ceph-disk list | grep osd.4 >> /dev/sdc1 ceph data, active, cluster ceph, osd.4, block /dev/sdc2 >> >> # smartctl -a /dev/sdc >> smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.13.13-6-pve] (local build) >> Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org >> >> === START OF INFORMATION SECTION === >> Model Family: Seagate Barracuda 7200.14 (AF) >> Device Model: ST1000DM003-1SB10C >> Serial Number: Z9A1M1BW >> LU WWN Device Id: 5 000c50 090c78d27 >> Firmware Version: CC43 >> User Capacity: 1,000,204,886,016 bytes [1.00 TB] >> Sector Sizes: 512 bytes logical, 4096 bytes physical >> Rotation Rate: 7200 rpm >> Form Factor: 3.5 inches >> Device is: In smartctl database [for details use: -P show] >> ATA Version is: ATA8-ACS T13/1699-D revision 4 >> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) >> Local Time is: Mon Mar 5 16:20:46 2018 CET >> SMART support is: Available - device has SMART capability. >> SMART support is: Enabled >> >> === START OF READ SMART DATA SECTION === >> SMART overall-health self-assessment test result: PASSED >> >> General SMART Values: >> Offline data collection status: (0x82) Offline data collection activity >> was completed without error. >> Auto Offline Data Collection: Enabled. >> Self-test execution status: ( 0) The previous self-test routine >> completed >> without error or no self-test has ever >> been run. >> Total time to complete Offline >> data collection: ( 0) seconds. >> Offline data collection >> capabilities: (0x7b) SMART execute Offline immediate. >> Auto Offline data collection on/off >> support. >> Suspend Offline collection upon new >> command. >> Offline surface scan supported. >> Self-test supported. >> Conveyance Self-test supported. >> Selective Self-test supported. >> SMART capabilities: (0x0003) Saves SMART data before entering >> power-saving mode. >> Supports SMART auto save timer. >> Error logging capability: (0x01) Error logging supported. >> General Purpose Logging supported. >> Short self-test routine >> recommended polling time: ( 1) minutes. >> Extended self-test routine >> recommended polling time: ( 109) minutes. >> Conveyance self-test routine >> recommended polling time: ( 2) minutes. >> SCT capabilities: (0x1085) SCT Status supported. >> >> SMART Attributes Data Structure revision number: 10 >> Vendor Specific SMART Attributes with Thresholds: >> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED >> WHEN_FAILED RAW_VALUE >> 1 Raw_Read_Error_Rate 0x000f 082 063 006 Pre-fail Always >> - 194906537 >> 3 Spin_Up_Time 0x0003 097 097 000 Pre-fail Always >> - 0 >> 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always >> - 64 >> 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always >> - 0 >> 7 Seek_Error_Rate 0x000f 091 060 045 Pre-fail Always >> - 1485899434 >> 9 Power_On_Hours 0x0032 085 085 000 Old_age Always >> - 13390 >> 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always >> - 0 >> 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always >> - 65 >> 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always >> - 0 >> 184 End-to-End_Error 0x0032 100 100 099 Old_age Always >> - 0 >> 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always >> - 0 >> 188 Command_Timeout 0x0032 100 100 000 Old_age Always >> - 0 0 0 >> 189 High_Fly_Writes 0x003a 095 095 000 Old_age Always >> - 5 >> 190 Airflow_Temperature_Cel 0x0022 074 051 040 Old_age Always >> - 26 (Min/Max 19/29) >> 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always >> - 616 >> 194 Temperature_Celsius 0x0022 026 014 000 Old_age Always >> - 26 (0 14 0 0 0) >> 195 Hardware_ECC_Recovered 0x001a 004 001 000 Old_age Always >> - 194906537 >> 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always >> - 0 >> 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline >> - 0 >> 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always >> - 0 >> 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline >> - 13315h+20m+30.974s >> 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline >> - 52137467719 >> 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline >> - 177227508503 >> >> >> *OSD.8* >> >> # ceph-disk list | grep osd.8 >> /dev/sda1 ceph data, active, cluster ceph, osd.8, block /dev/sda2 >> >> # smartctl -a /dev/sda >> smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.13.13-6-pve] (local build) >> Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org >> >> === START OF INFORMATION SECTION === >> Model Family: Seagate Barracuda 7200.14 (AF) >> Device Model: ST1000DM003-1SB10C >> Serial Number: Z9A2BEF2 >> LU WWN Device Id: 5 000c50 0910f5427 >> Firmware Version: CC43 >> User Capacity: 1,000,203,804,160 bytes [1.00 TB] >> Sector Sizes: 512 bytes logical, 4096 bytes physical >> Rotation Rate: 7200 rpm >> Form Factor: 3.5 inches >> Device is: In smartctl database [for details use: -P show] >> ATA Version is: ATA8-ACS T13/1699-D revision 4 >> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) >> Local Time is: Mon Mar 5 16:22:47 2018 CET >> SMART support is: Available - device has SMART capability. >> SMART support is: Enabled >> >> === START OF READ SMART DATA SECTION === >> SMART overall-health self-assessment test result: PASSED >> >> General SMART Values: >> Offline data collection status: (0x82) Offline data collection activity >> was completed without error. >> Auto Offline Data Collection: Enabled. >> Self-test execution status: ( 0) The previous self-test routine >> completed >> without error or no self-test has ever >> been run. >> Total time to complete Offline >> data collection: ( 0) seconds. >> Offline data collection >> capabilities: (0x7b) SMART execute Offline immediate. >> Auto Offline data collection on/off >> support. >> Suspend Offline collection upon new >> command. >> Offline surface scan supported. >> Self-test supported. >> Conveyance Self-test supported. >> Selective Self-test supported. >> SMART capabilities: (0x0003) Saves SMART data before entering >> power-saving mode. >> Supports SMART auto save timer. >> Error logging capability: (0x01) Error logging supported. >> General Purpose Logging supported. >> Short self-test routine >> recommended polling time: ( 1) minutes. >> Extended self-test routine >> recommended polling time: ( 110) minutes. >> Conveyance self-test routine >> recommended polling time: ( 2) minutes. >> SCT capabilities: (0x1085) SCT Status supported. >> >> SMART Attributes Data Structure revision number: 10 >> Vendor Specific SMART Attributes with Thresholds: >> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED >> WHEN_FAILED RAW_VALUE >> 1 Raw_Read_Error_Rate 0x000f 083 063 006 Pre-fail Always >> - 224621855 >> 3 Spin_Up_Time 0x0003 097 097 000 Pre-fail Always >> - 0 >> 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always >> - 275 >> 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always >> - 0 >> 7 Seek_Error_Rate 0x000f 081 060 045 Pre-fail Always >> - 149383284 >> 9 Power_On_Hours 0x0032 093 093 000 Old_age Always >> - 6210 >> 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always >> - 0 >> 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always >> - 265 >> 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always >> - 0 >> 184 End-to-End_Error 0x0032 100 100 099 Old_age Always >> - 0 >> 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always >> - 0 >> 188 Command_Timeout 0x0032 100 100 000 Old_age Always >> - 0 0 0 >> 189 High_Fly_Writes 0x003a 098 098 000 Old_age Always >> - 2 >> 190 Airflow_Temperature_Cel 0x0022 069 058 040 Old_age Always >> - 31 (Min/Max 21/35) >> 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always >> - 516 >> 194 Temperature_Celsius 0x0022 031 017 000 Old_age Always >> - 31 (0 17 0 0 0) >> 195 Hardware_ECC_Recovered 0x001a 005 001 000 Old_age Always >> - 224621855 >> 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always >> - 0 >> 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline >> - 0 >> 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always >> - 0 >> 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline >> - 6154h+03m+35.126s >> 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline >> - 24333847321 >> 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline >> - 50261005553 >> >> >> >> However it's not only these 3 OSD to have PG with errors, these are onlyl >> the most recent, in the last 3 months I had often OSD_SCRUB_ERRORS in >> various OSDs, always solved by ceph pg repair <PG>, I don't think it's an >> hardware issue. >> >> >> >> >> >> Il 05/03/2018 13:40, Vladimir Prokofev ha scritto: >> >> > candidate had a read error >> speaks for itself - while scrubbing it coudn't read data. >> I had similar issue, and it was just OSD dying - errors and relocated >> sectors in SMART, just replaced the disk. But in your case it seems that >> errors are on different OSDs? Are your OSDs all healthy? >> You can use this command to see some details. >> rados list-inconsistent-obj <pg.id> --format=json-pretty >> pg.id is the PG that's reporting as inconsistent. My guess is that >> you'll see read errors in this output, with OSD number that encountered >> error. After that you have to check that OSD health - SMART details, etc. >> Not always it's the disk itself that causing problems - for example we >> had read errors because of a faulty backplane interface in a server; >> changing the chassis resolved this issue. >> >> >> 2018-03-05 14:21 GMT+03:00 Marco Baldini - H.S. Amiata < >> mbald...@hsamiata.it>: >> >>> Hi >>> >>> After some days with debug_osd 5/5 I found [ERR] in different days, >>> different PGs, different OSDs, different hosts. This is what I get in the >>> OSD logs: >>> >>> *OSD.5 (host 3)* >>> 2018-03-01 20:30:02.702269 7fdf4d515700 2 osd.5 pg_epoch: 16486 pg[9.1c( v >>> 16486'51798 (16431'50251,16486'51798] local-lis/les=16474/16475 n=3629 >>> ec=1477/1477 lis/c 16474/16474 les/c/f 16475/16477/0 16474/16474/16474) >>> [5,6] r=0 lpr=16474 crt=16486'51798 lcod 16486'51797 mlcod 16486'51797 >>> active+clean+scrubbing+deep] 9.1c shard 6: soid >>> 9:3b157c56:::rbd_data.1526386b8b4567.0000000000001761:head candidate had a >>> read error >>> 2018-03-01 20:30:02.702278 7fdf4d515700 -1 log_channel(cluster) log [ERR] : >>> 9.1c shard 6: soid >>> 9:3b157c56:::rbd_data.1526386b8b4567.0000000000001761:head candidate had a >>> read error >>> >>> * >>> OSD.4 (host 3)* >>> 2018-02-28 00:03:33.458558 7f112cf76700 -1 log_channel(cluster) log [ERR] : >>> 13.65 shard 2: soid >>> 13:a719ecdf:::rbd_data.5f65056b8b4567.000000000000f8eb:head candidate had a >>> read error >>> >>> *OSD.8 (host 2)* >>> 2018-02-27 23:55:15.100084 7f4dd0816700 -1 log_channel(cluster) log [ERR] : >>> 14.31 shard 1: soid >>> 14:8cc6cd37:::rbd_data.30b15b6b8b4567.00000000000081a1:head candidate had a >>> read error >>> >>> I don't know what this error is meaning, and as always a ceph pg repair >>> fixes it. I don't think this is normal. >>> >>> Ideas? >>> >>> Thanks >>> >>> Il 28/02/2018 14:48, Marco Baldini - H.S. Amiata ha scritto: >>> >>> Hi >>> >>> I read the bugtracker issue and it seems a lot like my problem, even if >>> I can't check the reported checksum because I don't have it in my logs, >>> perhaps it's because of debug osd = 0/0 in ceph.conf >>> >>> I just raised the OSD log level >>> >>> ceph tell osd.* injectargs --debug-osd 5/5 >>> >>> I'll check OSD logs in the next days... >>> >>> Thanks >>> >>> >>> >>> Il 28/02/2018 11:59, Paul Emmerich ha scritto: >>> >>> Hi, >>> >>> might be http://tracker.ceph.com/issues/22464 >>> >>> Can you check the OSD log file to see if the reported checksum >>> is 0x6706be76? >>> >>> >>> Paul >>> >>> Am 28.02.2018 um 11:43 schrieb Marco Baldini - H.S. Amiata < >>> mbald...@hsamiata.it>: >>> >>> Hello >>> >>> I have a little ceph cluster with 3 nodes, each with 3x1TB HDD and >>> 1x240GB SSD. I created this cluster after Luminous release, so all OSDs are >>> Bluestore. In my crush map I have two rules, one targeting the SSDs and one >>> targeting the HDDs. I have 4 pools, one using the SSD rule and the others >>> using the HDD rule, three pools are size=3 min_size=2, one is size=2 >>> min_size=1 (this one have content that it's ok to lose) >>> >>> In the last 3 month I'm having a strange random problem. I planned my >>> osd scrubs during the night (osd scrub begin hour = 20, osd scrub end hour >>> = 7) when office is closed so there is low impact on the users. Some >>> mornings, when I ceph the cluster health, I find: >>> >>> HEALTH_ERR X scrub errors; Possible data damage: Y pgs inconsistent >>> OSD_SCRUB_ERRORS X scrub errors >>> PG_DAMAGED Possible data damage: Y pg inconsistent >>> >>> X and Y sometimes are 1, sometimes 2. >>> >>> I issue a ceph health detail, check the damaged PGs, and run a ceph pg >>> repair for the damaged PGs, I get >>> >>> instructing pg PG on osd.N to repair >>> >>> PG are different, OSD that have to repair PG is different, even the node >>> hosting the OSD is different, I made a list of all PGs and OSDs. This >>> morning is the most recent case: >>> >>> > ceph health detail >>> HEALTH_ERR 2 scrub errors; Possible data damage: 2 pgs inconsistent >>> OSD_SCRUB_ERRORS 2 scrub errors >>> PG_DAMAGED Possible data damage: 2 pgs inconsistent >>> pg 13.65 is active+clean+inconsistent, acting [4,2,6] >>> pg 14.31 is active+clean+inconsistent, acting [8,3,1] >>> >>> > ceph pg repair 13.65 >>> instructing pg 13.65 on osd.4 to repair >>> >>> (node-2)> tail /var/log/ceph/ceph-osd.4.log >>> 2018-02-28 08:38:47.593447 7f112cf76700 0 log_channel(cluster) log [DBG] : >>> 13.65 repair starts >>> 2018-02-28 08:39:37.573342 7f112cf76700 0 log_channel(cluster) log [DBG] : >>> 13.65 repair ok, 0 fixed >>> >>> > ceph pg repair 14.31 >>> instructing pg 14.31 on osd.8 to repair >>> >>> (node-3)> tail /var/log/ceph/ceph-osd.8.log >>> 2018-02-28 08:52:37.297490 7f4dd0816700 0 log_channel(cluster) log [DBG] : >>> 14.31 repair starts >>> 2018-02-28 08:53:00.704020 7f4dd0816700 0 log_channel(cluster) log [DBG] : >>> 14.31 repair ok, 0 fixed >>> >>> >>> >>> I made a list of when I got OSD_SCRUB_ERRORS, what PG and what OSD had >>> to repair PG. Date is dd/mm/yyyy >>> >>> 21/12/2017 -- pg 14.29 is active+clean+inconsistent, acting [6,2,4] >>> >>> 18/01/2018 -- pg 14.5a is active+clean+inconsistent, acting [6,4,1] >>> >>> 22/01/2018 -- pg 9.3a is active+clean+inconsistent, acting [2,7] >>> >>> 29/01/2018 -- pg 13.3e is active+clean+inconsistent, acting [4,6,1] >>> instructing pg 13.3e on osd.4 to repair >>> >>> 07/02/2018 -- pg 13.7e is active+clean+inconsistent, acting [8,2,5] >>> instructing pg 13.7e on osd.8 to repair >>> >>> 09/02/2018 -- pg 13.30 is active+clean+inconsistent, acting [7,3,2] >>> instructing pg 13.30 on osd.7 to repair >>> >>> 15/02/2018 -- pg 9.35 is active+clean+inconsistent, acting [1,8] >>> instructing pg 9.35 on osd.1 to repair >>> >>> pg 13.3e is active+clean+inconsistent, acting [4,6,1] >>> instructing pg 13.3e on osd.4 to repair >>> >>> 17/02/2018 -- pg 9.2d is active+clean+inconsistent, acting [7,5] >>> instructing pg 9.2d on osd.7 to repair >>> >>> 22/02/2018 -- pg 9.24 is active+clean+inconsistent, acting [5,8] >>> instructing pg 9.24 on osd.5 to repair >>> >>> 28/02/2018 -- pg 13.65 is active+clean+inconsistent, acting [4,2,6] >>> instructing pg 13.65 on osd.4 to repair >>> >>> pg 14.31 is active+clean+inconsistent, acting [8,3,1] >>> instructing pg 14.31 on osd.8 to repair >>> >>> >>> >>> >>> If can be useful, my ceph.conf is here: >>> >>> [global] >>> auth client required = none >>> auth cluster required = none >>> auth service required = none >>> fsid = 24d5d6bc-0943-4345-b44e-46c19099004b >>> cluster network = 10.10.10.0/24 >>> public network = 10.10.10.0/24 >>> keyring = /etc/pve/priv/$cluster.$name.keyring >>> mon allow pool delete = true >>> osd journal size = 5120 >>> osd pool default min size = 2 >>> osd pool default size = 3 >>> bluestore_block_db_size = 64424509440 >>> >>> debug asok = 0/0 >>> debug auth = 0/0 >>> debug buffer = 0/0 >>> debug client = 0/0 >>> debug context = 0/0 >>> debug crush = 0/0 >>> debug filer = 0/0 >>> debug filestore = 0/0 >>> debug finisher = 0/0 >>> debug heartbeatmap = 0/0 >>> debug journal = 0/0 >>> debug journaler = 0/0 >>> debug lockdep = 0/0 >>> debug mds = 0/0 >>> debug mds balancer = 0/0 >>> debug mds locker = 0/0 >>> debug mds log = 0/0 >>> debug mds log expire = 0/0 >>> debug mds migrator = 0/0 >>> debug mon = 0/0 >>> debug monc = 0/0 >>> debug ms = 0/0 >>> debug objclass = 0/0 >>> debug objectcacher = 0/0 >>> debug objecter = 0/0 >>> debug optracker = 0/0 >>> debug osd = 0/0 >>> debug paxos = 0/0 >>> debug perfcounter = 0/0 >>> debug rados = 0/0 >>> debug rbd = 0/0 >>> debug rgw = 0/0 >>> debug throttle = 0/0 >>> debug timer = 0/0 >>> debug tp = 0/0 >>> >>> >>> [osd] >>> keyring = /var/lib/ceph/osd/ceph-$id/keyring >>> osd max backfills = 1 >>> osd recovery max active = 1 >>> >>> osd scrub begin hour = 20 >>> osd scrub end hour = 7 >>> osd scrub during recovery = false >>> osd scrub load threshold = 0.3 >>> >>> [client] >>> rbd cache = true >>> rbd cache size = 268435456 # 256MB >>> rbd cache max dirty = 201326592 # 192MB >>> rbd cache max dirty age = 2 >>> rbd cache target dirty = 33554432 # 32MB >>> rbd cache writethrough until flush = true >>> >>> >>> #[mgr] >>> #debug_mgr = 20 >>> >>> >>> [mon.pve-hs-main] >>> host = pve-hs-main >>> mon addr = 10.10.10.251:6789 >>> >>> [mon.pve-hs-2] >>> host = pve-hs-2 >>> mon addr = 10.10.10.252:6789 >>> >>> [mon.pve-hs-3] >>> host = pve-hs-3 >>> mon addr = 10.10.10.253:6789 >>> >>> My ceph versions: >>> >>> { >>> "mon": { >>> "ceph version 12.2.2 (215dd7151453fae88e6f968c975b6ce309d42dcf) >>> luminous (stable)": 3 >>> }, >>> "mgr": { >>> "ceph version 12.2.2 (215dd7151453fae88e6f968c975b6ce309d42dcf) >>> luminous (stable)": 3 >>> }, >>> "osd": { >>> "ceph version 12.2.2 (215dd7151453fae88e6f968c975b6ce309d42dcf) >>> luminous (stable)": 12 >>> }, >>> "mds": {}, >>> "overall": { >>> "ceph version 12.2.2 (215dd7151453fae88e6f968c975b6ce309d42dcf) >>> luminous (stable)": 18 >>> } >>> } >>> >>> >>> >>> >>> My ceph osd tree: >>> >>> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF >>> -1 8.93686 root default >>> -6 2.94696 host pve-hs-2 >>> 3 hdd 0.90959 osd.3 up 1.00000 1.00000 >>> 4 hdd 0.90959 osd.4 up 1.00000 1.00000 >>> 5 hdd 0.90959 osd.5 up 1.00000 1.00000 >>> 10 ssd 0.21819 osd.10 up 1.00000 1.00000 >>> -3 2.86716 host pve-hs-3 >>> 6 hdd 0.85599 osd.6 up 1.00000 1.00000 >>> 7 hdd 0.85599 osd.7 up 1.00000 1.00000 >>> 8 hdd 0.93700 osd.8 up 1.00000 1.00000 >>> 11 ssd 0.21819 osd.11 up 1.00000 1.00000 >>> -7 3.12274 host pve-hs-main >>> 0 hdd 0.96819 osd.0 up 1.00000 1.00000 >>> 1 hdd 0.96819 osd.1 up 1.00000 1.00000 >>> 2 hdd 0.96819 osd.2 up 1.00000 1.00000 >>> 9 ssd 0.21819 osd.9 up 1.00000 1.00000 >>> >>> >>> My pools: >>> >>> pool 9 'cephbackup' replicated size 2 min_size 1 crush_rule 1 object_hash >>> rjenkins pg_num 64 pgp_num 64 last_change 5665 flags hashpspool >>> stripe_width 0 application rbd >>> removed_snaps [1~3] >>> pool 13 'cephwin' replicated size 3 min_size 2 crush_rule 1 object_hash >>> rjenkins pg_num 128 pgp_num 128 last_change 16454 flags hashpspool >>> stripe_width 0 application rbd >>> removed_snaps [1~5] >>> pool 14 'cephnix' replicated size 3 min_size 2 crush_rule 1 object_hash >>> rjenkins pg_num 128 pgp_num 128 last_change 16482 flags hashpspool >>> stripe_width 0 application rbd >>> removed_snaps [1~227] >>> pool 17 'cephssd' replicated size 3 min_size 2 crush_rule 2 object_hash >>> rjenkins pg_num 64 pgp_num 64 last_change 8601 flags hashpspool >>> stripe_width 0 application rbd >>> removed_snaps [1~3] >>> >>> >>> >>> I can't understand where the problem comes from, I don't think it's >>> hardware, if I had a failed disk, then I should have problems always on the >>> same OSD. Any ideas >>> >>> Thanks >>> >>> >>> >>> -- >>> *Marco Baldini* >>> *H.S. Amiata Srl* >>> Ufficio: 0577-779396 >>> Cellulare: 335-8765169 >>> WEB: www.hsamiata.it >>> EMAIL: mbald...@hsamiata.it >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >>> >>> -- >>> Mit freundlichen Grüßen / Best Regards >>> Paul Emmerich >>> >>> croit GmbH >>> Freseniusstr. 31h >>> 81247 München >>> www.croit.io >>> Tel: +49 89 1896585 90 <+49%2089%20189658590> >>> >>> Geschäftsführer: Martin Verges >>> Handelsregister: Amtsgericht München >>> USt-IdNr: DE310638492 >>> >>> >>> -- >>> *Marco Baldini* >>> *H.S. Amiata Srl* >>> Ufficio: 0577-779396 >>> Cellulare: 335-8765169 >>> WEB: www.hsamiata.it >>> EMAIL: mbald...@hsamiata.it >>> >>> >>> _______________________________________________ >>> ceph-users mailing >>> listceph-us...@lists.ceph.comhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >>> >>> -- >>> *Marco Baldini* >>> *H.S. Amiata Srl* >>> Ufficio: 0577-779396 >>> Cellulare: 335-8765169 >>> WEB: www.hsamiata.it >>> EMAIL: mbald...@hsamiata.it >>> >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >>> >> >> >> _______________________________________________ >> ceph-users mailing >> listceph-us...@lists.ceph.comhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> >> -- >> *Marco Baldini* >> *H.S. Amiata Srl* >> Ufficio: 0577-779396 >> Cellulare: 335-8765169 >> WEB: www.hsamiata.it >> EMAIL: mbald...@hsamiata.it >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> > > > _______________________________________________ > ceph-users mailing > listceph-us...@lists.ceph.comhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > -- > *Marco Baldini* > *H.S. Amiata Srl* > Ufficio: 0577-779396 > Cellulare: 335-8765169 > WEB: www.hsamiata.it > EMAIL: mbald...@hsamiata.it > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > -- Cheers, Brad
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
