[jira] [Updated] (HDFS-3307) when save FSImage ，HDFS（ or SecondaryNameNode or FSImage）can't handle some file whose file name has some special messy code(乱码)
[
https://issues.apache.org/jira/browse/HDFS-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anonymous updated HDFS-3307:
Hadoop Flags: Incompatible change,Reviewed
Status: Patch Available (was: Reopened)
when save FSImage ，HDFS（ or SecondaryNameNode or FSImage）can't handle some
file whose file name has some special messy code(乱码)
-
Key: HDFS-3307
URL: https://issues.apache.org/jira/browse/HDFS-3307
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 0.20.1
Environment: SUSE LINUX
Reporter: yixiaohua
Attachments: FSImage.java, ProblemString.txt,
TestUTF8AndStringGetBytes.java, TestUTF8AndStringGetBytes.java
Original Estimate: 12h
Remaining Estimate: 12h
this the log information of the exception from the SecondaryNameNode:
2012-03-28 00:48:42,553 ERROR
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
java.io.IOException: Found lease for
non-existent file
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/@???
??tor.qzone.qq.com/keypart-00174
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFilesUnderConstruction(FSImage.java:1211)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:959)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.doMerge(SecondaryNameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.access$000(SecondaryNameNode.java:473)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doMerge(SecondaryNameNode.java:350)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:314)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:225)
at java.lang.Thread.run(Thread.java:619)
this is the log information about the file from namenode:
2012-03-28 00:32:26,528 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
ip=/10.131.16.34cmd=create
src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174 dst=null
perm=boss:boss:rw-r--r--
2012-03-28 00:37:42,387 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.allocateBlock:
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174.
blk_2751836614265659170_184668759
2012-03-28 00:37:42,696 INFO org.apache.hadoop.hdfs.StateChange: DIR*
NameSystem.completeFile: file
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174 is closed by
DFSClient_attempt_201203271849_0016_r_000174_0
2012-03-28 00:37:50,315 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
ip=/10.131.16.34cmd=rename
src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174
dst=/user/boss/pgv/fission/task16/split/ @?
tor.qzone.qq.com/keypart-00174 perm=boss:boss:rw-r--r--
after check the code that save FSImage,I found there are a problem that maybe
a bug of HDFS Code,I past below:
-this is the saveFSImage method in FSImage.java, I make some
mark at the problem code
/**
* Save the contents of the FS image to the file.
*/
void saveFSImage(File newFile) throws IOException {
FSNamesystem fsNamesys = FSNamesystem.getFSNamesystem();
FSDirectory fsDir = fsNamesys.dir;
long startTime = FSNamesystem.now();
//
// Write out data
//
DataOutputStream out = new DataOutputStream(
new BufferedOutputStream(
new
FileOutputStream(newFile)));
try {
.
// save the rest of the nodes
saveImage(strbuf, 0, fsDir.rootDir, out);--problem
fsNamesys.saveFilesUnderConstruction(out);--problem
detail is below
strbuf = null;
} finally {
out.close();
}
LOG.info(Image file of size + newFile.length() + saved in
+ (FSNamesystem.now() - startTime)/1000 + seconds.);
}
/**
* Save file tree image starting from the given root.
* This is a recursive procedure, which first saves all children of
* a current directory and then moves inside the sub-directories.
*/
[jira] [Updated] (HDFS-3307) when save FSImage ，HDFS（ or SecondaryNameNode or FSImage）can't handle some file whose file name has some special messy code(乱码)
[
https://issues.apache.org/jira/browse/HDFS-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yixiaohua updated HDFS-3307:
Attachment: ProblemString.txt
TestUTF8AndStringGetBytes.java
when save FSImage ，HDFS（ or SecondaryNameNode or FSImage）can't handle some
file whose file name has some special messy code(乱码)
-
Key: HDFS-3307
URL: https://issues.apache.org/jira/browse/HDFS-3307
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 0.20.1
Environment: SUSE LINUX
Reporter: yixiaohua
Attachments: FSImage.java, ProblemString.txt,
TestUTF8AndStringGetBytes.java
Original Estimate: 12h
Remaining Estimate: 12h
this the log information of the exception from the SecondaryNameNode:
2012-03-28 00:48:42,553 ERROR
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
java.io.IOException: Found lease for
non-existent file
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/@???
??tor.qzone.qq.com/keypart-00174
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFilesUnderConstruction(FSImage.java:1211)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:959)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.doMerge(SecondaryNameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.access$000(SecondaryNameNode.java:473)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doMerge(SecondaryNameNode.java:350)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:314)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:225)
at java.lang.Thread.run(Thread.java:619)
this is the log information about the file from namenode:
2012-03-28 00:32:26,528 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
ip=/10.131.16.34cmd=create
src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174 dst=null
perm=boss:boss:rw-r--r--
2012-03-28 00:37:42,387 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.allocateBlock:
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174.
blk_2751836614265659170_184668759
2012-03-28 00:37:42,696 INFO org.apache.hadoop.hdfs.StateChange: DIR*
NameSystem.completeFile: file
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174 is closed by
DFSClient_attempt_201203271849_0016_r_000174_0
2012-03-28 00:37:50,315 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
ip=/10.131.16.34cmd=rename
src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174
dst=/user/boss/pgv/fission/task16/split/ @?
tor.qzone.qq.com/keypart-00174 perm=boss:boss:rw-r--r--
after check the code that save FSImage,I found there are a problem that maybe
a bug of HDFS Code,I past below:
-this is the saveFSImage method in FSImage.java, I make some
mark at the problem code
/**
* Save the contents of the FS image to the file.
*/
void saveFSImage(File newFile) throws IOException {
FSNamesystem fsNamesys = FSNamesystem.getFSNamesystem();
FSDirectory fsDir = fsNamesys.dir;
long startTime = FSNamesystem.now();
//
// Write out data
//
DataOutputStream out = new DataOutputStream(
new BufferedOutputStream(
new
FileOutputStream(newFile)));
try {
.
// save the rest of the nodes
saveImage(strbuf, 0, fsDir.rootDir, out);--problem
fsNamesys.saveFilesUnderConstruction(out);--problem
detail is below
strbuf = null;
} finally {
out.close();
}
LOG.info(Image file of size + newFile.length() + saved in
+ (FSNamesystem.now() - startTime)/1000 + seconds.);
}
/**
* Save file tree image starting from the given root.
* This is a recursive procedure, which first saves all children of
* a current directory and then moves inside the sub-directories.
*/
private static void saveImage(ByteBuffer
[jira] [Updated] (HDFS-3307) when save FSImage ，HDFS（ or SecondaryNameNode or FSImage）can't handle some file whose file name has some special messy code(乱码)
[
https://issues.apache.org/jira/browse/HDFS-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yixiaohua updated HDFS-3307:
Attachment: TestUTF8AndStringGetBytes.java
when save FSImage ，HDFS（ or SecondaryNameNode or FSImage）can't handle some
file whose file name has some special messy code(乱码)
-
Key: HDFS-3307
URL: https://issues.apache.org/jira/browse/HDFS-3307
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 0.20.1
Environment: SUSE LINUX
Reporter: yixiaohua
Attachments: FSImage.java, ProblemString.txt,
TestUTF8AndStringGetBytes.java, TestUTF8AndStringGetBytes.java
Original Estimate: 12h
Remaining Estimate: 12h
this the log information of the exception from the SecondaryNameNode:
2012-03-28 00:48:42,553 ERROR
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
java.io.IOException: Found lease for
non-existent file
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/@???
??tor.qzone.qq.com/keypart-00174
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFilesUnderConstruction(FSImage.java:1211)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:959)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.doMerge(SecondaryNameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.access$000(SecondaryNameNode.java:473)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doMerge(SecondaryNameNode.java:350)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:314)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:225)
at java.lang.Thread.run(Thread.java:619)
this is the log information about the file from namenode:
2012-03-28 00:32:26,528 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
ip=/10.131.16.34cmd=create
src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174 dst=null
perm=boss:boss:rw-r--r--
2012-03-28 00:37:42,387 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.allocateBlock:
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174.
blk_2751836614265659170_184668759
2012-03-28 00:37:42,696 INFO org.apache.hadoop.hdfs.StateChange: DIR*
NameSystem.completeFile: file
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174 is closed by
DFSClient_attempt_201203271849_0016_r_000174_0
2012-03-28 00:37:50,315 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
ip=/10.131.16.34cmd=rename
src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174
dst=/user/boss/pgv/fission/task16/split/ @?
tor.qzone.qq.com/keypart-00174 perm=boss:boss:rw-r--r--
after check the code that save FSImage,I found there are a problem that maybe
a bug of HDFS Code,I past below:
-this is the saveFSImage method in FSImage.java, I make some
mark at the problem code
/**
* Save the contents of the FS image to the file.
*/
void saveFSImage(File newFile) throws IOException {
FSNamesystem fsNamesys = FSNamesystem.getFSNamesystem();
FSDirectory fsDir = fsNamesys.dir;
long startTime = FSNamesystem.now();
//
// Write out data
//
DataOutputStream out = new DataOutputStream(
new BufferedOutputStream(
new
FileOutputStream(newFile)));
try {
.
// save the rest of the nodes
saveImage(strbuf, 0, fsDir.rootDir, out);--problem
fsNamesys.saveFilesUnderConstruction(out);--problem
detail is below
strbuf = null;
} finally {
out.close();
}
LOG.info(Image file of size + newFile.length() + saved in
+ (FSNamesystem.now() - startTime)/1000 + seconds.);
}
/**
* Save file tree image starting from the given root.
* This is a recursive procedure, which first saves all children of
* a current directory and then moves inside the sub-directories.
*/
private static void saveImage(ByteBuffer
[jira] [Updated] (HDFS-3307) when save FSImage ，HDFS（ or SecondaryNameNode or FSImage）can't handle some file whose file name has some special messy code(乱码)
[
https://issues.apache.org/jira/browse/HDFS-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yixiaohua updated HDFS-3307:
Component/s: name-node
Description:
this the log information of the exception from the SecondaryNameNode:
2012-03-28 00:48:42,553 ERROR
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.io.IOException:
Found lease for
non-existent file
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/@???
??tor.qzone.qq.com/keypart-00174
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFilesUnderConstruction(FSImage.java:1211)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:959)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.doMerge(SecondaryNameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.access$000(SecondaryNameNode.java:473)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doMerge(SecondaryNameNode.java:350)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:314)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:225)
at java.lang.Thread.run(Thread.java:619)
this is the log information about the file from namenode:
2012-03-28 00:32:26,528 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
ip=/10.131.16.34cmd=create
src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174 dst=null
perm=boss:boss:rw-r--r--
2012-03-28 00:37:42,387 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.allocateBlock:
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174.
blk_2751836614265659170_184668759
2012-03-28 00:37:42,696 INFO org.apache.hadoop.hdfs.StateChange: DIR*
NameSystem.completeFile: file
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174 is closed by
DFSClient_attempt_201203271849_0016_r_000174_0
2012-03-28 00:37:50,315 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
ip=/10.131.16.34cmd=rename
src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174
dst=/user/boss/pgv/fission/task16/split/ @?
tor.qzone.qq.com/keypart-00174 perm=boss:boss:rw-r--r--
after check the code that save FSImage,I found there are a problem that maybe a
bug of HDFS Code,I past below:
-this is the saveFSImage method in FSImage.java, I make some mark
at the problem code
/**
* Save the contents of the FS image to the file.
*/
void saveFSImage(File newFile) throws IOException {
FSNamesystem fsNamesys = FSNamesystem.getFSNamesystem();
FSDirectory fsDir = fsNamesys.dir;
long startTime = FSNamesystem.now();
//
// Write out data
//
DataOutputStream out = new DataOutputStream(
new BufferedOutputStream(
new
FileOutputStream(newFile)));
try {
.
// save the rest of the nodes
saveImage(strbuf, 0, fsDir.rootDir, out);--problem
fsNamesys.saveFilesUnderConstruction(out);--problem
detail is below
strbuf = null;
} finally {
out.close();
}
LOG.info(Image file of size + newFile.length() + saved in
+ (FSNamesystem.now() - startTime)/1000 + seconds.);
}
/**
* Save file tree image starting from the given root.
* This is a recursive procedure, which first saves all children of
* a current directory and then moves inside the sub-directories.
*/
private static void saveImage(ByteBuffer parentPrefix,
int prefixLength,
INodeDirectory current,
DataOutputStream out) throws IOException {
int newPrefixLength = prefixLength;
if (current.getChildrenRaw() == null)
return;
for(INode child : current.getChildren()) {
// print all children first
parentPrefix.position(prefixLength);
parentPrefix.put(PATH_SEPARATOR).put(child.getLocalNameBytes());--problem
saveINode2Image(parentPrefix, child, out);
}
..
}
// Helper function that writes an INodeUnderConstruction
// into the input stream
//
static void writeINodeUnderConstruction(DataOutputStream out,
[jira] [Updated] (HDFS-3307) when save FSImage ，HDFS（ or SecondaryNameNode or FSImage）can't handle some file whose file name has some special messy code(乱码)
[
https://issues.apache.org/jira/browse/HDFS-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yixiaohua updated HDFS-3307:
Attachment: FSImage.java
I only change the writeString method to make it the same with INode
stringToBytes method,and then the problem disappears.
when save FSImage ，HDFS（ or SecondaryNameNode or FSImage）can't handle some
file whose file name has some special messy code(乱码)
-
Key: HDFS-3307
URL: https://issues.apache.org/jira/browse/HDFS-3307
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 0.20.1
Environment: SUSE LINUX
Reporter: yixiaohua
Attachments: FSImage.java
Original Estimate: 12h
Remaining Estimate: 12h
this the log information of the exception from the SecondaryNameNode:
2012-03-28 00:48:42,553 ERROR
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
java.io.IOException: Found lease for
non-existent file
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/@???
??tor.qzone.qq.com/keypart-00174
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFilesUnderConstruction(FSImage.java:1211)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:959)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.doMerge(SecondaryNameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.access$000(SecondaryNameNode.java:473)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doMerge(SecondaryNameNode.java:350)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:314)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:225)
at java.lang.Thread.run(Thread.java:619)
this is the log information about the file from namenode:
2012-03-28 00:32:26,528 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
ip=/10.131.16.34cmd=create
src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174 dst=null
perm=boss:boss:rw-r--r--
2012-03-28 00:37:42,387 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.allocateBlock:
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174.
blk_2751836614265659170_184668759
2012-03-28 00:37:42,696 INFO org.apache.hadoop.hdfs.StateChange: DIR*
NameSystem.completeFile: file
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174 is closed by
DFSClient_attempt_201203271849_0016_r_000174_0
2012-03-28 00:37:50,315 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
ip=/10.131.16.34cmd=rename
src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174
dst=/user/boss/pgv/fission/task16/split/ @?
tor.qzone.qq.com/keypart-00174 perm=boss:boss:rw-r--r--
after check the code that save FSImage,I found there are a problem that maybe
a bug of HDFS Code,I past below:
-this is the saveFSImage method in FSImage.java, I make some
mark at the problem code
/**
* Save the contents of the FS image to the file.
*/
void saveFSImage(File newFile) throws IOException {
FSNamesystem fsNamesys = FSNamesystem.getFSNamesystem();
FSDirectory fsDir = fsNamesys.dir;
long startTime = FSNamesystem.now();
//
// Write out data
//
DataOutputStream out = new DataOutputStream(
new BufferedOutputStream(
new
FileOutputStream(newFile)));
try {
.
// save the rest of the nodes
saveImage(strbuf, 0, fsDir.rootDir, out);--problem
fsNamesys.saveFilesUnderConstruction(out);--problem
detail is below
strbuf = null;
} finally {
out.close();
}
LOG.info(Image file of size + newFile.length() + saved in
+ (FSNamesystem.now() - startTime)/1000 + seconds.);
}
/**
* Save file tree image starting from the given root.
* This is a recursive procedure, which first saves all children of
* a current directory and then moves inside the sub-directories.
*/
private
[jira] [Updated] (HDFS-3307) when save FSImage ，HDFS（ or SecondaryNameNode or FSImage）can't handle some file whose file name has some special messy code(乱码)
[
https://issues.apache.org/jira/browse/HDFS-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yixiaohua updated HDFS-3307:
Target Version/s: (was: 0.20.1)
when save FSImage ，HDFS（ or SecondaryNameNode or FSImage）can't handle some
file whose file name has some special messy code(乱码)
-
Key: HDFS-3307
URL: https://issues.apache.org/jira/browse/HDFS-3307
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 0.20.1
Environment: SUSE LINUX
Reporter: yixiaohua
Attachments: FSImage.java
Original Estimate: 12h
Remaining Estimate: 12h
this the log information of the exception from the SecondaryNameNode:
2012-03-28 00:48:42,553 ERROR
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
java.io.IOException: Found lease for
non-existent file
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/@???
??tor.qzone.qq.com/keypart-00174
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFilesUnderConstruction(FSImage.java:1211)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:959)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.doMerge(SecondaryNameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.access$000(SecondaryNameNode.java:473)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doMerge(SecondaryNameNode.java:350)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:314)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:225)
at java.lang.Thread.run(Thread.java:619)
this is the log information about the file from namenode:
2012-03-28 00:32:26,528 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
ip=/10.131.16.34cmd=create
src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174 dst=null
perm=boss:boss:rw-r--r--
2012-03-28 00:37:42,387 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.allocateBlock:
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174.
blk_2751836614265659170_184668759
2012-03-28 00:37:42,696 INFO org.apache.hadoop.hdfs.StateChange: DIR*
NameSystem.completeFile: file
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174 is closed by
DFSClient_attempt_201203271849_0016_r_000174_0
2012-03-28 00:37:50,315 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
ip=/10.131.16.34cmd=rename
src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174
dst=/user/boss/pgv/fission/task16/split/ @?
tor.qzone.qq.com/keypart-00174 perm=boss:boss:rw-r--r--
after check the code that save FSImage,I found there are a problem that maybe
a bug of HDFS Code,I past below:
-this is the saveFSImage method in FSImage.java, I make some
mark at the problem code
/**
* Save the contents of the FS image to the file.
*/
void saveFSImage(File newFile) throws IOException {
FSNamesystem fsNamesys = FSNamesystem.getFSNamesystem();
FSDirectory fsDir = fsNamesys.dir;
long startTime = FSNamesystem.now();
//
// Write out data
//
DataOutputStream out = new DataOutputStream(
new BufferedOutputStream(
new
FileOutputStream(newFile)));
try {
.
// save the rest of the nodes
saveImage(strbuf, 0, fsDir.rootDir, out);--problem
fsNamesys.saveFilesUnderConstruction(out);--problem
detail is below
strbuf = null;
} finally {
out.close();
}
LOG.info(Image file of size + newFile.length() + saved in
+ (FSNamesystem.now() - startTime)/1000 + seconds.);
}
/**
* Save file tree image starting from the given root.
* This is a recursive procedure, which first saves all children of
* a current directory and then moves inside the sub-directories.
*/
private static void saveImage(ByteBuffer parentPrefix,
int prefixLength,
