2009/7/15 Doug Judd <[email protected]>: > I doubt that this else clause in the AccessGroup::run_compaction has > anything to do with it. The cell store that is referenced in that else > clause is never listed in the METADATA table. Unless you're doing directory > scanning, it should be invisible to the GC logic. Plus, this zero sized > cell store scenario should be a fairly rare event. We've already seen 91841 > instances of this error on the Production Zvents cluster this month.
I'm not doing a directory scan yet, though probably should once in a while. Upon closer inspection, it looks like it really shouldn't cause the problem. So it remain a mystery why stale file list are still appearing in the METADATA. > > - Doug > > 2009/7/15 Luke <[email protected]> >> >> Did you delete a lot cells recently (like delete * from sometable)? >> >> The algorithm is quite simple: it tries look at history of the >> Files:<access_group> column in the METADATA table and push all the >> latest file list into a hash map with reference count 1, and the file >> list of older cells. reference count of 0. At the end the scan, we >> delete anything with reference count of 0. The older cells are deleted >> as well, so they should not show up on the delete list in the next >> scan. I think the reason you see these warning is that >> AccessGroup.cc:505 (Global::dfs->remove(fname);) deletes cellstores >> that are empty as a result of merge compaction of deleted cells. The >> GC algorithm then tried to remove these files that were already >> deleted, hence the warnings. >> >> I think the whole else clause starting on AccessGroup.cc:501 should be >> removed and let the GC algorithm do the job. Do you want to give it >> try to see if it works for you? >> >> __Luke >> >> 2009/7/15 kuer <[email protected]>: >> > >> > Thanks, Doug, >> > >> > I would try it. >> > Of course, I have to figure out which files fall into the filelist, >> > firstly. >> > >> > thanks >> > >> > -- kuer >> > >> > On 7月16日, 上午1时02分, Doug Judd <[email protected]> wrote: >> >> Hi kuer, >> >> >> >> Thanks for the report. According to Luke (who wrote the code), these >> >> messages are innocuous. I agree that the code should be fixed so that >> >> it >> >> doesn't try to delete these files that don't exist. However, I think >> >> the >> >> code does not ever delete files that are in use. This code has been in >> >> place for about a year and we've never experienced that kind of >> >> problem. >> >> >> >> If you want to take a shot at fixing the non-existent file delete >> >> problem, >> >> go for it. We'll pull it in if it looks good. >> >> >> >> - Doug >> >> >> >> On Wed, Jul 15, 2009 at 2:15 AM, kuer <[email protected]> wrote: >> >> >> >> > Hi, all. >> >> >> >> > After restarting hypertable + kfs in my work-boxes, there are some >> >> > endless messages logging into Hypertable.Master.log : >> >> >> >> > 2009-07-15 16:45:42,431 1350199616 Hypertable.Master [WARN] (Master/ >> >> > MasterGc.cc:197) Error removing DFS file: /hypertable/tables/ >> >> > storage_se/agPORT/4F852B7EB462D2CC311753C0/cs12 >> >> > 2009-07-15 16:45:42,432 1350199616 Hypertable.Master [WARN] (Master/ >> >> > MasterGc.cc:197) Error removing DFS file: /hypertable/tables/ >> >> > storage_se/agPORT/4F852B7EB462D2CC311753C0/cs14 >> >> > 2009-07-15 16:45:42,432 1350199616 Hypertable.Master [WARN] (Master/ >> >> > MasterGc.cc:197) Error removing DFS file: /hypertable/tables/ >> >> > storage_se/default/590FCA21F8FB70BB2A1F56A2/cs3 >> >> > 2009-07-15 16:45:42,433 1350199616 Hypertable.Master [WARN] (Master/ >> >> > MasterGc.cc:197) Error removing DFS file: /hypertable/tables/ >> >> > storage_se/default/590FCA21F8FB70BB2A1F56A2/cs4 >> >> > ... >> >> > ... >> >> > ... >> >> > 2009-07-15 16:45:42,433 1350199616 Hypertable.Master [WARN] (Master/ >> >> > MasterGc.cc:197) Error removing DFS file: /hypertable/tables/ >> >> > storage_se/default/590FCA21F8FB70BB2A1F56A2/cs8 >> >> > 2009-07-15 16:45:42,434 1350199616 Hypertable.Master [WARN] (Master/ >> >> > MasterGc.cc:197) Error removing DFS file: /hypertable/tables/ >> >> > storage_se/default/13C3704B81B2919AE7F6A936/cs12 >> >> >> >> > Then, I check the dfs.log : >> >> >> >> > 2009-07-15 16:55:44,060 1319373120 kosmosBroker [ERROR] (kosmos/ >> >> > KosmosBroker.cc:273) unlink failed: file='/hypertable/tables/ >> >> > storage_se/agPORT/4F852B7EB462D2CC311753C0/cs10' - No such file or >> >> > directory >> >> > 2009-07-15 16:55:44,061 1308883264 kosmosBroker [ERROR] (kosmos/ >> >> > KosmosBroker.cc:273) unlink failed: file='/hypertable/tables/ >> >> > storage_se/agPORT/4F852B7EB462D2CC311753C0/cs12' - No such file or >> >> > directory >> >> > 2009-07-15 16:55:44,061 1130555712 kosmosBroker [ERROR] (kosmos/ >> >> > KosmosBroker.cc:273) unlink failed: file='/hypertable/tables/ >> >> > storage_se/agPORT/4F852B7EB462D2CC311753C0/cs14' - No such file or >> >> > directory >> >> > 2009-07-15 16:55:44,062 1151535424 kosmosBroker [ERROR] (kosmos/ >> >> > KosmosBroker.cc:273) unlink failed: file='/hypertable/tables/ >> >> > storage_se/default/590FCA21F8FB70BB2A1F56A2/cs3' - No such file or >> >> > directory >> >> > 2009-07-15 16:55:44,062 1141045568 kosmosBroker [ERROR] (kosmos/ >> >> > KosmosBroker.cc:273) unlink failed: file='/hypertable/tables/ >> >> > storage_se/default/590FCA21F8FB70BB2A1F56A2/cs4' - No such file or >> >> > directory >> >> > ... >> >> > ... >> >> > ... >> >> > 2009-07-15 16:55:44,063 1162025280 kosmosBroker [ERROR] (kosmos/ >> >> > KosmosBroker.cc:273) unlink failed: file='/hypertable/tables/ >> >> > storage_se/default/590FCA21F8FB70BB2A1F56A2/cs8' - No such file or >> >> > directory >> >> > 2009-07-15 16:55:44,064 1172515136 kosmosBroker [ERROR] (kosmos/ >> >> > KosmosBroker.cc:273) unlink failed: file='/hypertable/tables/ >> >> > storage_se/default/13C3704B81B2919AE7F6A936/cs12' - No such file or >> >> > directory >> >> >> >> > When reading source code src/cc/Hypertable/Master/MasterGc.cc : >> >> > 182 void >> >> > 183 reap(CountMap &files_map) { >> >> > 184 size_t nf = 0, nf_done = 0, nd = 0, nd_done = 0; >> >> > 185 CountMap dirs_map; // reap empty range directories as well >> >> > 186 >> >> > 187 foreach (const CountMap::value_type &v, files_map) { >> >> > 188 if (!v.second) { >> >> > 189 HT_DEBUGF("MasterGc: removing file %s", v.first); >> >> > 190 >> >> > 191 if (!m_dryrun) { >> >> > 192 try { >> >> > 193 m_fs->remove(v.first); >> >> > 194 ++nf_done; >> >> > 195 } >> >> > 196 catch (Exception &e) { >> >> > 197 HT_WARNF("%s", e.what()); >> >> > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ THIS LINE IS Logging-MESSAGE >> >> > COME FROM >> >> > 198 } >> >> > 199 } >> >> > 200 ++nf; >> >> > 201 } >> >> > 202 char *p = strrchr(v.first, '/'); >> >> > 203 >> >> > 204 if (p) { >> >> > 205 string dir_name(v.first, p - v.first); >> >> > 206 insert_file(dirs_map, dir_name.c_str(), v.second); >> >> > 207 } >> >> > 208 } >> >> > 209 foreach (const CountMap::value_type &v, dirs_map) { >> >> > 210 if (!v.second) { >> >> > 211 HT_DEBUGF("MasterGc: removing directory %s", v.first); >> >> > 212 >> >> >> >> > I found that reap(file_map) just remove the specified files, but do >> >> > NOT notify/mark the status of deletion to anyone else. Recursively, >> >> > the `file_map` was built in scan_metadata() function. Apparently, >> >> > scan_metadata() does NOT check the existence of the file to delete >> >> > when building deletion-file-list. >> >> >> >> > When Master.GC-ing, this reap operations would repeat. At present, I >> >> > donot read the source code totally, I donot understand the rational >> >> > of >> >> > scan_metadata(). But the reap() do *delete* the exist file; So there >> >> > is risk that reap() will delete some WORKING ( IN-USE) file by name >> >> > which should be kept alive!!! >> >> >> >> > Any one can help me out ? >> >> >> >> > Thanks a lot. >> >> >> >> > -- kuer >> > > >> > >> >> > > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en -~----------~----~----~----~------~----~------~--~---
