On 7月16日, 上午9时17分, Luke <[email protected]> wrote:
> Did you delete a lot cells recently (like delete * from sometable)?
>
^^^^^^^^^^^^^^^^^^^^^^^
No any DELETE operations were performed. Just stop HT and start HT.
Stopping servers in sequence :
- stop thriftbroker
- stop rangeserver on all boxes
- stop Hypertable.Master
- stop hyperspace.Master
- stop kosmosBroker on all boxes
- stop kfs-chunkserver on all boxes
- stop kfs-metaserver
Did incorrect stopping operations leave those junk-files/records ?
-- kuer
> The algorithm is quite simple: it tries look at history of the
> Files:<access_group> column in the METADATA table and push all the
> latest file list into a hash map with reference count 1, and the file
> list of older cells. reference count of 0. At the end the scan, we
> delete anything with reference count of 0. The older cells are deleted
> as well, so they should not show up on the delete list in the next
> scan. I think the reason you see these warning is that
> AccessGroup.cc:505 (Global::dfs->remove(fname);) deletes cellstores
> that are empty as a result of merge compaction of deleted cells. The
> GC algorithm then tried to remove these files that were already
> deleted, hence the warnings.
>
> I think the whole else clause starting on AccessGroup.cc:501 should be
> removed and let the GC algorithm do the job. Do you want to give it
> try to see if it works for you?
>
> __Luke
>
> 2009/7/15 kuer <[email protected]>:
>
>
>
> > Thanks, Doug,
>
> > I would try it.
> > Of course, I have to figure out which files fall into the filelist,
> > firstly.
>
> > thanks
>
> > -- kuer
>
> > On 7月16日, 上午1时02分, Doug Judd <[email protected]> wrote:
> >> Hi kuer,
>
> >> Thanks for the report. According to Luke (who wrote the code), these
> >> messages are innocuous. I agree that the code should be fixed so that it
> >> doesn't try to delete these files that don't exist. However, I think the
> >> code does not ever delete files that are in use. This code has been in
> >> place for about a year and we've never experienced that kind of problem.
>
> >> If you want to take a shot at fixing the non-existent file delete problem,
> >> go for it. We'll pull it in if it looks good.
>
> >> - Doug
>
> >> On Wed, Jul 15, 2009 at 2:15 AM, kuer <[email protected]> wrote:
>
> >> > Hi, all.
>
> >> > After restarting hypertable + kfs in my work-boxes, there are some
> >> > endless messages logging into Hypertable.Master.log :
>
> >> > 2009-07-15 16:45:42,431 1350199616 Hypertable.Master [WARN] (Master/
> >> > MasterGc.cc:197) Error removing DFS file: /hypertable/tables/
> >> > storage_se/agPORT/4F852B7EB462D2CC311753C0/cs12
> >> > 2009-07-15 16:45:42,432 1350199616 Hypertable.Master [WARN] (Master/
> >> > MasterGc.cc:197) Error removing DFS file: /hypertable/tables/
> >> > storage_se/agPORT/4F852B7EB462D2CC311753C0/cs14
> >> > 2009-07-15 16:45:42,432 1350199616 Hypertable.Master [WARN] (Master/
> >> > MasterGc.cc:197) Error removing DFS file: /hypertable/tables/
> >> > storage_se/default/590FCA21F8FB70BB2A1F56A2/cs3
> >> > 2009-07-15 16:45:42,433 1350199616 Hypertable.Master [WARN] (Master/
> >> > MasterGc.cc:197) Error removing DFS file: /hypertable/tables/
> >> > storage_se/default/590FCA21F8FB70BB2A1F56A2/cs4
> >> > ...
> >> > ...
> >> > ...
> >> > 2009-07-15 16:45:42,433 1350199616 Hypertable.Master [WARN] (Master/
> >> > MasterGc.cc:197) Error removing DFS file: /hypertable/tables/
> >> > storage_se/default/590FCA21F8FB70BB2A1F56A2/cs8
> >> > 2009-07-15 16:45:42,434 1350199616 Hypertable.Master [WARN] (Master/
> >> > MasterGc.cc:197) Error removing DFS file: /hypertable/tables/
> >> > storage_se/default/13C3704B81B2919AE7F6A936/cs12
>
> >> > Then, I check the dfs.log :
>
> >> > 2009-07-15 16:55:44,060 1319373120 kosmosBroker [ERROR] (kosmos/
> >> > KosmosBroker.cc:273) unlink failed: file='/hypertable/tables/
> >> > storage_se/agPORT/4F852B7EB462D2CC311753C0/cs10' - No such file or
> >> > directory
> >> > 2009-07-15 16:55:44,061 1308883264 kosmosBroker [ERROR] (kosmos/
> >> > KosmosBroker.cc:273) unlink failed: file='/hypertable/tables/
> >> > storage_se/agPORT/4F852B7EB462D2CC311753C0/cs12' - No such file or
> >> > directory
> >> > 2009-07-15 16:55:44,061 1130555712 kosmosBroker [ERROR] (kosmos/
> >> > KosmosBroker.cc:273) unlink failed: file='/hypertable/tables/
> >> > storage_se/agPORT/4F852B7EB462D2CC311753C0/cs14' - No such file or
> >> > directory
> >> > 2009-07-15 16:55:44,062 1151535424 kosmosBroker [ERROR] (kosmos/
> >> > KosmosBroker.cc:273) unlink failed: file='/hypertable/tables/
> >> > storage_se/default/590FCA21F8FB70BB2A1F56A2/cs3' - No such file or
> >> > directory
> >> > 2009-07-15 16:55:44,062 1141045568 kosmosBroker [ERROR] (kosmos/
> >> > KosmosBroker.cc:273) unlink failed: file='/hypertable/tables/
> >> > storage_se/default/590FCA21F8FB70BB2A1F56A2/cs4' - No such file or
> >> > directory
> >> > ...
> >> > ...
> >> > ...
> >> > 2009-07-15 16:55:44,063 1162025280 kosmosBroker [ERROR] (kosmos/
> >> > KosmosBroker.cc:273) unlink failed: file='/hypertable/tables/
> >> > storage_se/default/590FCA21F8FB70BB2A1F56A2/cs8' - No such file or
> >> > directory
> >> > 2009-07-15 16:55:44,064 1172515136 kosmosBroker [ERROR] (kosmos/
> >> > KosmosBroker.cc:273) unlink failed: file='/hypertable/tables/
> >> > storage_se/default/13C3704B81B2919AE7F6A936/cs12' - No such file or
> >> > directory
>
> >> > When reading source code src/cc/Hypertable/Master/MasterGc.cc :
> >> > 182 void
> >> > 183 reap(CountMap &files_map) {
> >> > 184 size_t nf = 0, nf_done = 0, nd = 0, nd_done = 0;
> >> > 185 CountMap dirs_map; // reap empty range directories as well
> >> > 186
> >> > 187 foreach (const CountMap::value_type &v, files_map) {
> >> > 188 if (!v.second) {
> >> > 189 HT_DEBUGF("MasterGc: removing file %s", v.first);
> >> > 190
> >> > 191 if (!m_dryrun) {
> >> > 192 try {
> >> > 193 m_fs->remove(v.first);
> >> > 194 ++nf_done;
> >> > 195 }
> >> > 196 catch (Exception &e) {
> >> > 197 HT_WARNF("%s", e.what());
> >> > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ THIS LINE IS Logging-MESSAGE
> >> > COME FROM
> >> > 198 }
> >> > 199 }
> >> > 200 ++nf;
> >> > 201 }
> >> > 202 char *p = strrchr(v.first, '/');
> >> > 203
> >> > 204 if (p) {
> >> > 205 string dir_name(v.first, p - v.first);
> >> > 206 insert_file(dirs_map, dir_name.c_str(), v.second);
> >> > 207 }
> >> > 208 }
> >> > 209 foreach (const CountMap::value_type &v, dirs_map) {
> >> > 210 if (!v.second) {
> >> > 211 HT_DEBUGF("MasterGc: removing directory %s", v.first);
> >> > 212
>
> >> > I found that reap(file_map) just remove the specified files, but do
> >> > NOT notify/mark the status of deletion to anyone else. Recursively,
> >> > the `file_map` was built in scan_metadata() function. Apparently,
> >> > scan_metadata() does NOT check the existence of the file to delete
> >> > when building deletion-file-list.
>
> >> > When Master.GC-ing, this reap operations would repeat. At present, I
> >> > donot read the source code totally, I donot understand the rational of
> >> > scan_metadata(). But the reap() do *delete* the exist file; So there
> >> > is risk that reap() will delete some WORKING ( IN-USE) file by name
> >> > which should be kept alive!!!
>
> >> > Any one can help me out ?
>
> >> > Thanks a lot.
>
> >> > -- kuer
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Hypertable Development" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/hypertable-dev?hl=en
-~----------~----~----~----~------~----~------~--~---
