[jira] [Updated] (HADOOP-9103) UTF8 class does not properly decode Unicode characters outside the basic multilingual plane
[
https://issues.apache.org/jira/browse/HADOOP-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HADOOP-9103:
Attachment: hadoop-9103.txt
Attached patch should fix this issue. I also amended the javadoc for UTF8 to
indicate that it encodes CESU-8
> UTF8 class does not properly decode Unicode characters outside the basic
> multilingual plane
> ---
>
> Key: HADOOP-9103
> URL: https://issues.apache.org/jira/browse/HADOOP-9103
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
>Affects Versions: 0.20.1
> Environment: SUSE LINUX
>Reporter: yixiaohua
>Assignee: Todd Lipcon
> Attachments: FSImage.java, hadoop-9103.txt, ProblemString.txt,
> TestUTF8AndStringGetBytes.java, TestUTF8AndStringGetBytes.java
>
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> this the log information of the exception from the SecondaryNameNode:
> 2012-03-28 00:48:42,553 ERROR
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
> java.io.IOException: Found lease for
> non-existent file
> /user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/@???
> ??tor.qzone.qq.com/keypart-00174
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFilesUnderConstruction(FSImage.java:1211)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:959)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.doMerge(SecondaryNameNode.java:589)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.access$000(SecondaryNameNode.java:473)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doMerge(SecondaryNameNode.java:350)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:314)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:225)
> at java.lang.Thread.run(Thread.java:619)
> this is the log information about the file from namenode:
> 2012-03-28 00:32:26,528 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
> ip=/10.131.16.34cmd=create
> src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
> @?tor.qzone.qq.com/keypart-00174 dst=null
> perm=boss:boss:rw-r--r--
> 2012-03-28 00:37:42,387 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.allocateBlock:
> /user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
> @?tor.qzone.qq.com/keypart-00174.
> blk_2751836614265659170_184668759
> 2012-03-28 00:37:42,696 INFO org.apache.hadoop.hdfs.StateChange: DIR*
> NameSystem.completeFile: file
> /user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
> @?tor.qzone.qq.com/keypart-00174 is closed by
> DFSClient_attempt_201203271849_0016_r_000174_0
> 2012-03-28 00:37:50,315 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
> ip=/10.131.16.34cmd=rename
> src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
> @?tor.qzone.qq.com/keypart-00174
> dst=/user/boss/pgv/fission/task16/split/ @?
> tor.qzone.qq.com/keypart-00174 perm=boss:boss:rw-r--r--
> after check the code that save FSImage,I found there are a problem that maybe
> a bug of HDFS Code,I past below:
> -this is the saveFSImage method in FSImage.java, I make some
> mark at the problem code
> /**
>* Save the contents of the FS image to the file.
>*/
> void saveFSImage(File newFile) throws IOException {
> FSNamesystem fsNamesys = FSNamesystem.getFSNamesystem();
> FSDirectory fsDir = fsNamesys.dir;
> long startTime = FSNamesystem.now();
> //
> // Write out data
> //
> DataOutputStream out = new DataOutputStream(
> new BufferedOutputStream(
> new
> FileOutputStream(newFile)));
> try {
> .
>
> // save the rest of the nodes
> saveImage(strbuf, 0, fsDir.rootDir, out);--problem
> fsNamesys.saveFilesUnderConstruction(out);--problem
> detail is below
> strbuf = null;
> } finally {
> out.close();
> }
> LOG.info("Image file of size " + newFile.length() + " saved in "
> + (FSNamesystem.now() - startTime)/1000 + " seconds.");
> }
> /**
>* Save file tree image starting from the given root.
>
[jira] [Updated] (HADOOP-9103) UTF8 class does not properly decode Unicode characters outside the basic multilingual plane
[
https://issues.apache.org/jira/browse/HADOOP-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HADOOP-9103:
Summary: UTF8 class does not properly decode Unicode characters outside the
basic multilingual plane (was: when save FSImage ，HDFS（ or SecondaryNameNode
or FSImage）can't handle some file whose file name has some special messy
code(乱码))
> UTF8 class does not properly decode Unicode characters outside the basic
> multilingual plane
> ---
>
> Key: HADOOP-9103
> URL: https://issues.apache.org/jira/browse/HADOOP-9103
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
>Affects Versions: 0.20.1
> Environment: SUSE LINUX
>Reporter: yixiaohua
>Assignee: Todd Lipcon
> Attachments: FSImage.java, hadoop-9103.txt, ProblemString.txt,
> TestUTF8AndStringGetBytes.java, TestUTF8AndStringGetBytes.java
>
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> this the log information of the exception from the SecondaryNameNode:
> 2012-03-28 00:48:42,553 ERROR
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
> java.io.IOException: Found lease for
> non-existent file
> /user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/@???
> ??tor.qzone.qq.com/keypart-00174
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFilesUnderConstruction(FSImage.java:1211)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:959)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.doMerge(SecondaryNameNode.java:589)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.access$000(SecondaryNameNode.java:473)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doMerge(SecondaryNameNode.java:350)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:314)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:225)
> at java.lang.Thread.run(Thread.java:619)
> this is the log information about the file from namenode:
> 2012-03-28 00:32:26,528 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
> ip=/10.131.16.34cmd=create
> src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
> @?tor.qzone.qq.com/keypart-00174 dst=null
> perm=boss:boss:rw-r--r--
> 2012-03-28 00:37:42,387 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.allocateBlock:
> /user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
> @?tor.qzone.qq.com/keypart-00174.
> blk_2751836614265659170_184668759
> 2012-03-28 00:37:42,696 INFO org.apache.hadoop.hdfs.StateChange: DIR*
> NameSystem.completeFile: file
> /user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
> @?tor.qzone.qq.com/keypart-00174 is closed by
> DFSClient_attempt_201203271849_0016_r_000174_0
> 2012-03-28 00:37:50,315 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
> ip=/10.131.16.34cmd=rename
> src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
> @?tor.qzone.qq.com/keypart-00174
> dst=/user/boss/pgv/fission/task16/split/ @?
> tor.qzone.qq.com/keypart-00174 perm=boss:boss:rw-r--r--
> after check the code that save FSImage,I found there are a problem that maybe
> a bug of HDFS Code,I past below:
> -this is the saveFSImage method in FSImage.java, I make some
> mark at the problem code
> /**
>* Save the contents of the FS image to the file.
>*/
> void saveFSImage(File newFile) throws IOException {
> FSNamesystem fsNamesys = FSNamesystem.getFSNamesystem();
> FSDirectory fsDir = fsNamesys.dir;
> long startTime = FSNamesystem.now();
> //
> // Write out data
> //
> DataOutputStream out = new DataOutputStream(
> new BufferedOutputStream(
> new
> FileOutputStream(newFile)));
> try {
> .
>
> // save the rest of the nodes
> saveImage(strbuf, 0, fsDir.rootDir, out);--problem
> fsNamesys.saveFilesUnderConstruction(out);--problem
> detail is below
> strbuf = null;
> } finally {
> out.close();
> }
> LOG.info("Image file of size " + newFile.length() + " saved in "
> + (FSNamesystem.now() - startTime)/10
[jira] [Moved] (HADOOP-9103) when save FSImage ，HDFS（ or SecondaryNameNode or FSImage）can't handle some file whose file name has some special messy code(乱码)
[
https://issues.apache.org/jira/browse/HADOOP-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon moved HDFS-3307 to HADOOP-9103:
---
Component/s: (was: namenode)
io
Affects Version/s: (was: 0.20.1)
0.20.1
Key: HADOOP-9103 (was: HDFS-3307)
Project: Hadoop Common (was: Hadoop HDFS)
> when save FSImage ，HDFS（ or SecondaryNameNode or FSImage）can't handle some
> file whose file name has some special messy code(乱码)
> -
>
> Key: HADOOP-9103
> URL: https://issues.apache.org/jira/browse/HADOOP-9103
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
>Affects Versions: 0.20.1
> Environment: SUSE LINUX
>Reporter: yixiaohua
>Assignee: Todd Lipcon
> Attachments: FSImage.java, hadoop-9103.txt, ProblemString.txt,
> TestUTF8AndStringGetBytes.java, TestUTF8AndStringGetBytes.java
>
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> this the log information of the exception from the SecondaryNameNode:
> 2012-03-28 00:48:42,553 ERROR
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
> java.io.IOException: Found lease for
> non-existent file
> /user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/@???
> ??tor.qzone.qq.com/keypart-00174
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFilesUnderConstruction(FSImage.java:1211)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:959)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.doMerge(SecondaryNameNode.java:589)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.access$000(SecondaryNameNode.java:473)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doMerge(SecondaryNameNode.java:350)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:314)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:225)
> at java.lang.Thread.run(Thread.java:619)
> this is the log information about the file from namenode:
> 2012-03-28 00:32:26,528 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
> ip=/10.131.16.34cmd=create
> src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
> @?tor.qzone.qq.com/keypart-00174 dst=null
> perm=boss:boss:rw-r--r--
> 2012-03-28 00:37:42,387 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.allocateBlock:
> /user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
> @?tor.qzone.qq.com/keypart-00174.
> blk_2751836614265659170_184668759
> 2012-03-28 00:37:42,696 INFO org.apache.hadoop.hdfs.StateChange: DIR*
> NameSystem.completeFile: file
> /user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
> @?tor.qzone.qq.com/keypart-00174 is closed by
> DFSClient_attempt_201203271849_0016_r_000174_0
> 2012-03-28 00:37:50,315 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
> ip=/10.131.16.34cmd=rename
> src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
> @?tor.qzone.qq.com/keypart-00174
> dst=/user/boss/pgv/fission/task16/split/ @?
> tor.qzone.qq.com/keypart-00174 perm=boss:boss:rw-r--r--
> after check the code that save FSImage,I found there are a problem that maybe
> a bug of HDFS Code,I past below:
> -this is the saveFSImage method in FSImage.java, I make some
> mark at the problem code
> /**
>* Save the contents of the FS image to the file.
>*/
> void saveFSImage(File newFile) throws IOException {
> FSNamesystem fsNamesys = FSNamesystem.getFSNamesystem();
> FSDirectory fsDir = fsNamesys.dir;
> long startTime = FSNamesystem.now();
> //
> // Write out data
> //
> DataOutputStream out = new DataOutputStream(
> new BufferedOutputStream(
> new
> FileOutputStream(newFile)));
> try {
> .
>
> // save the rest of the nodes
> saveImage(strbuf, 0, fsDir.rootDir, out);--problem
> fsNamesys.saveFilesUnderConstruction(out);--problem
> detail is below
> strbuf = null;
> } finally {
> out.close();
> }
> LOG.info("I
[jira] [Updated] (HADOOP-8981) TestMetricsSystemImpl fails on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated HADOOP-8981: -- Attachment: HADOOP-8981-branch-trunk-win.2.patch Format the code > TestMetricsSystemImpl fails on Windows > -- > > Key: HADOOP-8981 > URL: https://issues.apache.org/jira/browse/HADOOP-8981 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: trunk-win >Reporter: Chris Nauroth >Assignee: Arpit Agarwal > Attachments: HADOOP-8981-branch-trunk-win.1.patch, > HADOOP-8981-branch-trunk-win.2.patch > > > The test is failing on an expected mock interaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9090) Refactor MetricsSystemImpl to allow for an on-demand publish system
[ https://issues.apache.org/jira/browse/HADOOP-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mostafa Elhemali updated HADOOP-9090: - Attachment: HADOOP-9090.justEnhanceDefaultImpl.4.patch (I highly appreciate the responsive feedback I'm getting from you Luke.) Thanks - you raise good points. My fear was that do we want the immediate sinks to be queued up or not, and do we want to leave the queue thread hanging if the sink times out or not. I think though I was swayed to your way of thinking by two things: a) the flush() problem you point out and b) that my previous patch would've left sinks possible called by multiple threads instead of just the queue thread which can be bad. I've implemented the spirit of what you said. I opted though to have the putMetricsImmediate() wait for its specific MetricsBuffer since I figured the way you had it may end up in race conditions if multiple threads are calling publishMetricsNow() at the same time. Let me know what you think - thanks! > Refactor MetricsSystemImpl to allow for an on-demand publish system > --- > > Key: HADOOP-9090 > URL: https://issues.apache.org/jira/browse/HADOOP-9090 > Project: Hadoop Common > Issue Type: New Feature > Components: metrics >Reporter: Mostafa Elhemali >Priority: Minor > Attachments: HADOOP-9090.2.patch, > HADOOP-9090.justEnhanceDefaultImpl.2.patch, > HADOOP-9090.justEnhanceDefaultImpl.3.patch, > HADOOP-9090.justEnhanceDefaultImpl.4.patch, > HADOOP-9090.justEnhanceDefaultImpl.patch, HADOOP-9090.patch > > > We have a need to publish metrics out of some short-living processes, which > is not really well-suited to the current metrics system implementation which > periodically publishes metrics asynchronously (a behavior that works great > for long-living processes). Of course I could write my own metrics system, > but it seems like such a waste to rewrite all the awesome code currently in > the MetricsSystemImpl and supporting classes. > The way I'm proposing to solve this is to: > 1. Refactor the MetricsSystemImpl class into an abstract base > MetricsSystemImpl class (common configuration and other code) and a concrete > PeriodicPublishMetricsSystemImpl class (timer thread). > 2. Refactor the MetricsSinkAdapter class into an abstract base > MetricsSinkAdapter class (common configuration and other code) and a concrete > AsyncMetricsSinkAdapter class (asynchronous publishing using the SinkQueue). > 3. Derive a new simple class OnDemandPublishMetricsSystemImpl from > MetricsSystemImpl, that just exposes a synchronous publish() method to do all > the work. > 4. Derive a SyncMetricsSinkAdapter class from MetricsSinkAdapter to just > synchronously push metrics to the underlying sink. > Does that sound reasonable? I'll attach the patch with all this coded up and > simple tests (could use some polish I guess, but wanted to get everyone's > opinion first). Notice that this is somewhat of a breaking change since > MetricsSystemImpl is public (although it's marked with > InterfaceAudience.Private); if the breaking change is a problem I could just > rename the refactored classes so that PeriodicPublishMetricsSystemImpl is > still called MetricsSystemImpl (and MetricsSystemImpl -> > BaseMetricsSystemImpl). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9057) TestMetricsSystemImpl.testInitFirst fails intermittently
[ https://issues.apache.org/jira/browse/HADOOP-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506103#comment-13506103 ] Luke Lu commented on HADOOP-9057: - Mostafa: the timeout in mockito verify is a good idea. If you can provide trunk patch and branch-1 patches, I'll commit them. > TestMetricsSystemImpl.testInitFirst fails intermittently > > > Key: HADOOP-9057 > URL: https://issues.apache.org/jira/browse/HADOOP-9057 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1-win >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Attachments: HADOOP-9057.branch-1-win.metricsrace.patch > > > Error Message > Wanted but not invoked: metricsSink.putMetrics(); -> at > org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl.testInitFirst(TestMetricsSystemImpl.java:80) > Actually, there were zero interactions with this mock. > Stacktrace > Wanted but not invoked: > metricsSink.putMetrics(); > -> at > org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl.testInitFirst(TestMetricsSystemImpl.java:80) > Actually, there were zero interactions with this mock. > at > org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl.testInitFirst(TestMetricsSystemImpl.java:80) > at > org.mockito.internal.runners.JUnit45AndHigherRunnerImpl.run(JUnit45AndHigherRunnerImpl.java:37) > at > org.mockito.runners.MockitoJUnitRunner.run(MockitoJUnitRunner.java:62) > Standard Output > 2012-10-04 11:43:55,641 INFO impl.MetricsConfig > (MetricsConfig.java:loadFirst(99)) - loaded properties from > hadoop-metrics2-test.properties -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9102) winutils task isAlive does not return a non-zero exit code if the requested task is not alive
Chris Nauroth created HADOOP-9102: - Summary: winutils task isAlive does not return a non-zero exit code if the requested task is not alive Key: HADOOP-9102 URL: https://issues.apache.org/jira/browse/HADOOP-9102 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 1-win Reporter: Chris Nauroth Assignee: Chris Nauroth Work on YARN-233 noted that winutils task isAlive does not return a non-zero exit code if the requested task is not alive. This is inconsistent with the equivalent check on Unix, kill -0, which uses a non-zero exit code to communicate status. By changing winutils task isAlive to return a non-zero exit code, we can make the error handling code on the Java side consistent, avoiding the need for additional if (WINDOWS) checks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8981) TestMetricsSystemImpl fails on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506049#comment-13506049 ] Arpit Agarwal commented on HADOOP-8981: --- Yes, it is the same. Your fix is very similar to what Chuan has submitted i.e. adding a short sleep before the verification. > TestMetricsSystemImpl fails on Windows > -- > > Key: HADOOP-8981 > URL: https://issues.apache.org/jira/browse/HADOOP-8981 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: trunk-win >Reporter: Chris Nauroth >Assignee: Arpit Agarwal > Attachments: HADOOP-8981-branch-trunk-win.1.patch > > > The test is failing on an expected mock interaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9090) Refactor MetricsSystemImpl to allow for an on-demand publish system
[
https://issues.apache.org/jira/browse/HADOOP-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506045#comment-13506045
]
Luke Lu commented on HADOOP-9090:
-
I don't think you can control what happens to external systems, which _should_
handle arbitrary connection errors etc by unexpected client (even
router/firewall) shutdown. Another problem of the new patch is that you're
creating a latch and a semaphore per record (vs per MetricsBuffer) and there
can be hundreds (up to a few thousands) of records per put. If the sink hangs,
you'll be recreating new thread/latch/semaphore per record and the user
perceived timeout would be configured timeout * number of records. Another
issue is that hanging can happen in sink.flush as well.
Why not do the simple notification in the existing code like the following
(untested sketch)?:
{code}
boolean oobPut;
// illustration only, should be in the ctor after retry* variables are defined
final long OOB_PUT_TIMEOUT = retryDelay * Math.pow(retryBackoff, retryCount) *
1000;
synchronized void putMetricsImmediate(MetricsBuffer mb) {
if (!oobPut) {
oobPut = true;
if (queue.enqueue(buffer)) {
wait(OOB_PUT_TIMEOUT);
oobPut = false;
} // otherwise queue is full due to sink issues anyway.
} else { // another oobPut in progress
wait(OOB_PUT_TIMEOUT);
oobPut = false; // just in case
}
}
// after queue.consumeAll(this); in publishMetricsFromQueue (needs to be
synchronized now)
if (oobPut) {
notifyAll();
}
{code}
Now you get all the retry/timeout logic for free :)
> Refactor MetricsSystemImpl to allow for an on-demand publish system
> ---
>
> Key: HADOOP-9090
> URL: https://issues.apache.org/jira/browse/HADOOP-9090
> Project: Hadoop Common
> Issue Type: New Feature
> Components: metrics
>Reporter: Mostafa Elhemali
>Priority: Minor
> Attachments: HADOOP-9090.2.patch,
> HADOOP-9090.justEnhanceDefaultImpl.2.patch,
> HADOOP-9090.justEnhanceDefaultImpl.3.patch,
> HADOOP-9090.justEnhanceDefaultImpl.patch, HADOOP-9090.patch
>
>
> We have a need to publish metrics out of some short-living processes, which
> is not really well-suited to the current metrics system implementation which
> periodically publishes metrics asynchronously (a behavior that works great
> for long-living processes). Of course I could write my own metrics system,
> but it seems like such a waste to rewrite all the awesome code currently in
> the MetricsSystemImpl and supporting classes.
> The way I'm proposing to solve this is to:
> 1. Refactor the MetricsSystemImpl class into an abstract base
> MetricsSystemImpl class (common configuration and other code) and a concrete
> PeriodicPublishMetricsSystemImpl class (timer thread).
> 2. Refactor the MetricsSinkAdapter class into an abstract base
> MetricsSinkAdapter class (common configuration and other code) and a concrete
> AsyncMetricsSinkAdapter class (asynchronous publishing using the SinkQueue).
> 3. Derive a new simple class OnDemandPublishMetricsSystemImpl from
> MetricsSystemImpl, that just exposes a synchronous publish() method to do all
> the work.
> 4. Derive a SyncMetricsSinkAdapter class from MetricsSinkAdapter to just
> synchronously push metrics to the underlying sink.
> Does that sound reasonable? I'll attach the patch with all this coded up and
> simple tests (could use some polish I guess, but wanted to get everyone's
> opinion first). Notice that this is somewhat of a breaking change since
> MetricsSystemImpl is public (although it's marked with
> InterfaceAudience.Private); if the breaking change is a problem I could just
> rename the refactored classes so that PeriodicPublishMetricsSystemImpl is
> still called MetricsSystemImpl (and MetricsSystemImpl ->
> BaseMetricsSystemImpl).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8981) TestMetricsSystemImpl fails on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506040#comment-13506040 ] Ivan Mitic commented on HADOOP-8981: Is this the same issue as reported by HADOOP-9057? > TestMetricsSystemImpl fails on Windows > -- > > Key: HADOOP-8981 > URL: https://issues.apache.org/jira/browse/HADOOP-8981 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: trunk-win >Reporter: Chris Nauroth >Assignee: Arpit Agarwal > Attachments: HADOOP-8981-branch-trunk-win.1.patch > > > The test is failing on an expected mock interaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8981) TestMetricsSystemImpl fails on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506031#comment-13506031 ] Arpit Agarwal commented on HADOOP-8981: --- Tabs should be replaced with two spaces. Looks fine otherwise. > TestMetricsSystemImpl fails on Windows > -- > > Key: HADOOP-8981 > URL: https://issues.apache.org/jira/browse/HADOOP-8981 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: trunk-win >Reporter: Chris Nauroth >Assignee: Arpit Agarwal > Attachments: HADOOP-8981-branch-trunk-win.1.patch > > > The test is failing on an expected mock interaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[ https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506017#comment-13506017 ] Jonathan Allen commented on HADOOP-8989: Allen - I haven't done any specific studies of the NN impact but I would expect it to be the same as any of the existing recursive command (e.g. ls -R). The code uses the existing recursion code and applies the expressions to the returned FileStatus on the client side. The -exec option obviously impacts the NN but no more than if you ran the command directly (though you don't have the JVM start-up overhead so the rate of NN messages will be higher). Wolfgang - I'll look at adding those option expressions in (I got to the point where it looked like I'd be adding expressions forever so I decided to stop and see what people asked for). Daryn - if you do get a chance to review this then the main code is pretty much finished (though I need to sort out the formatting before submitting properly) but I've reworked the unit tests since uploading the last patch (I'll upload a new at the weekend). I still need to update the documentation web page and add some stuff in the CLI test suites. > hadoop dfs -find feature > > > Key: HADOOP-8989 > URL: https://issues.apache.org/jira/browse/HADOOP-8989 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Marco Nicosia >Assignee: Jonathan Allen > Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch > > > Both sysadmins and users make frequent use of the unix 'find' command, but > Hadoop has no correlate. Without this, users are writing scripts which make > heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs > -lsr is somewhat taxing on the NameNode, and a really slow experience on the > client side. Possibly an in-NameNode find operation would be only a bit more > taxing on the NameNode, but significantly faster from the client's point of > view? > The minimum set of options I can think of which would make a Hadoop find > command generally useful is (in priority order): > * -type (file or directory, for now) > * -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments) > * -print0 (for piping to xargs -0) > * -depth > * -owner/-group (and -nouser/-nogroup) > * -name (allowing for shell pattern, or even regex?) > * -perm > * -size > One possible special case, but could possibly be really cool if it ran from > within the NameNode: > * -delete > The "hadoop dfs -lsr | hadoop dfs -rm" cycle is really, really slow. > Lower priority, some people do use operators, mostly to execute -or searches > such as: > * find / \(-nouser -or -nogroup\) > Finally, I thought I'd include a link to the [Posix spec for > find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9090) Refactor MetricsSystemImpl to allow for an on-demand publish system
[ https://issues.apache.org/jira/browse/HADOOP-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mostafa Elhemali updated HADOOP-9090: - Attachment: HADOOP-9090.justEnhanceDefaultImpl.3.patch Thanks Luke. It's a fundamentally hard problem: if we assume that putMetrics() from arbitrary sinks can hang, then we'll have to interrupt the thread that doing the work and hope that unsticks it (that's how thread pools cancel work for example). Having said that, I got uneasy about using stop(), and I agree with your comment that I was creating too many threads willy-nilly. The new patch thus does two things: 1. Creates a single thread (if needed) for on-demand publishing per sink, and reuses that for subsequent publishing. This should cut down considerably on how many threads are created. 2. If it sees a sink timeout, it interrupts the worker thread but doesn't stop() it (joins it and hangs with it if needed). I personally think that's the most reliable thing we can do. One alternative as you say is to mark the worker thread as a daemon, so we don't have to join it and hang with it if it hangs, but just abandon it. The problem there is that sinks may be doing things to external systems that will leave them in inconsistent states if the process just shuts down while they're still working with no warning. Let me know what you think. > Refactor MetricsSystemImpl to allow for an on-demand publish system > --- > > Key: HADOOP-9090 > URL: https://issues.apache.org/jira/browse/HADOOP-9090 > Project: Hadoop Common > Issue Type: New Feature > Components: metrics >Reporter: Mostafa Elhemali >Priority: Minor > Attachments: HADOOP-9090.2.patch, > HADOOP-9090.justEnhanceDefaultImpl.2.patch, > HADOOP-9090.justEnhanceDefaultImpl.3.patch, > HADOOP-9090.justEnhanceDefaultImpl.patch, HADOOP-9090.patch > > > We have a need to publish metrics out of some short-living processes, which > is not really well-suited to the current metrics system implementation which > periodically publishes metrics asynchronously (a behavior that works great > for long-living processes). Of course I could write my own metrics system, > but it seems like such a waste to rewrite all the awesome code currently in > the MetricsSystemImpl and supporting classes. > The way I'm proposing to solve this is to: > 1. Refactor the MetricsSystemImpl class into an abstract base > MetricsSystemImpl class (common configuration and other code) and a concrete > PeriodicPublishMetricsSystemImpl class (timer thread). > 2. Refactor the MetricsSinkAdapter class into an abstract base > MetricsSinkAdapter class (common configuration and other code) and a concrete > AsyncMetricsSinkAdapter class (asynchronous publishing using the SinkQueue). > 3. Derive a new simple class OnDemandPublishMetricsSystemImpl from > MetricsSystemImpl, that just exposes a synchronous publish() method to do all > the work. > 4. Derive a SyncMetricsSinkAdapter class from MetricsSinkAdapter to just > synchronously push metrics to the underlying sink. > Does that sound reasonable? I'll attach the patch with all this coded up and > simple tests (could use some polish I guess, but wanted to get everyone's > opinion first). Notice that this is somewhat of a breaking change since > MetricsSystemImpl is public (although it's marked with > InterfaceAudience.Private); if the breaking change is a problem I could just > rename the refactored classes so that PeriodicPublishMetricsSystemImpl is > still called MetricsSystemImpl (and MetricsSystemImpl -> > BaseMetricsSystemImpl). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9100) Fix TODO items in FsShell
[ https://issues.apache.org/jira/browse/HADOOP-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505846#comment-13505846 ] Eli Collins commented on HADOOP-9100: - See HADOOP-8690 for one of them. > Fix TODO items in FsShell > - > > Key: HADOOP-9100 > URL: https://issues.apache.org/jira/browse/HADOOP-9100 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Minor > > Bunch of TODO items in FsShell needs to be fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8981) TestMetricsSystemImpl fails on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated HADOOP-8981: -- Attachment: HADOOP-8981-branch-trunk-win.1.patch Attached the patch that addresses the test fails on windows. Add a test case to verify the callback correctly, abd test case to verify the stop call immediately > TestMetricsSystemImpl fails on Windows > -- > > Key: HADOOP-8981 > URL: https://issues.apache.org/jira/browse/HADOOP-8981 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: trunk-win >Reporter: Chris Nauroth >Assignee: Arpit Agarwal > Attachments: HADOOP-8981-branch-trunk-win.1.patch > > > The test is failing on an expected mock interaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9078) enhance unit-test coverage of class org.apache.hadoop.fs.FileContext
[ https://issues.apache.org/jira/browse/HADOOP-9078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HADOOP-9078: --- Affects Version/s: 0.23.6 2.0.3-alpha 3.0.0 Status: Patch Available (was: Open) > enhance unit-test coverage of class org.apache.hadoop.fs.FileContext > > > Key: HADOOP-9078 > URL: https://issues.apache.org/jira/browse/HADOOP-9078 > Project: Hadoop Common > Issue Type: Test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 >Reporter: Ivan A. Veselovsky >Assignee: Ivan A. Veselovsky > Attachments: HADOOP-9078-branch-0.23.patch, > HADOOP-9078-branch-2.patch, HADOOP-9078.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9078) enhance unit-test coverage of class org.apache.hadoop.fs.FileContext
[ https://issues.apache.org/jira/browse/HADOOP-9078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HADOOP-9078: --- Attachment: HADOOP-9078.patch "HADOOP-9078" is for trunk, other patches are for the branches respectively. > enhance unit-test coverage of class org.apache.hadoop.fs.FileContext > > > Key: HADOOP-9078 > URL: https://issues.apache.org/jira/browse/HADOOP-9078 > Project: Hadoop Common > Issue Type: Test >Reporter: Ivan A. Veselovsky >Assignee: Ivan A. Veselovsky > Attachments: HADOOP-9078-branch-0.23.patch, > HADOOP-9078-branch-2.patch, HADOOP-9078.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9078) enhance unit-test coverage of class org.apache.hadoop.fs.FileContext
[ https://issues.apache.org/jira/browse/HADOOP-9078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HADOOP-9078: --- Attachment: HADOOP-9078-branch-2.patch HADOOP-9078-branch-0.23.patch > enhance unit-test coverage of class org.apache.hadoop.fs.FileContext > > > Key: HADOOP-9078 > URL: https://issues.apache.org/jira/browse/HADOOP-9078 > Project: Hadoop Common > Issue Type: Test >Reporter: Ivan A. Veselovsky >Assignee: Ivan A. Veselovsky > Attachments: HADOOP-9078-branch-0.23.patch, > HADOOP-9078-branch-2.patch, HADOOP-9078.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9101) make s3n NativeFileSystemStore interface public instead of package-private
Steve Loughran created HADOOP-9101:
--
Summary: make s3n NativeFileSystemStore interface public instead
of package-private
Key: HADOOP-9101
URL: https://issues.apache.org/jira/browse/HADOOP-9101
Project: Hadoop Common
Issue Type: Improvement
Components: fs/s3
Affects Versions: 3.0.0
Reporter: Steve Loughran
Priority: Trivial
It would be easier to implement new blockstore filesystems if the the
{{NativeFileSystemStore} and dependent classes in the
{{org.apache.hadoop.fs.s3native}} package were public -currently you need to
put them into the s3 directory.
They could be made public with the appropriate scope attribute. Internal?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9046) provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewAction
[
https://issues.apache.org/jira/browse/HADOOP-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505661#comment-13505661
]
Robert Joseph Evans commented on HADOOP-9046:
-
It is looking good, and I like how much faster the tests run.
I have a few more comments.
# {code}TODO: should we reject the addition if an action for this 'fs' is
already present in the queue?{code} If this is an issue then we should file a
JIRA to fix it, otherwise please remove the TODO. I am leaning more towards
filing a separate JIRA.
# I don't really like the name lock0 and available0. I would rather have them
be named something more descriptive. This is very minor
# in removeRenewAction we are no longer canceling the token when it is removed.
I am not sure if that is intentional or not, but it definitely changes the
functionality of the method.
# I am not sure that renewerThreadStarted.compairAndSet is that much better
then isAlive(). isAlive has the problem that if the thread stopped, calling
add again would result in an IllegalThreadStateException, but compairAndSet has
the problem that if a new thread cannot be created (Which I have seen happen)
then only the first call to add will get an exception all other calls to add
will look like they succeeded but no renewal will ever happen. I would rather
see errors happen on other calls to add then to have them silently fail.
> provide unit-test coverage of class
> org.apache.hadoop.fs.DelegationTokenRenewer.RenewAction
> --
>
> Key: HADOOP-9046
> URL: https://issues.apache.org/jira/browse/HADOOP-9046
> Project: Hadoop Common
> Issue Type: Test
>Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6
>Reporter: Ivan A. Veselovsky
>Assignee: Ivan A. Veselovsky
>Priority: Minor
> Attachments: HADOOP-9046-branch-0.23--c.patch,
> HADOOP-9046-branch-0.23-over-9049.patch, HADOOP-9046-branch-0.23.patch,
> HADOOP-9046--c.patch, HADOOP-9046-over-9049.patch, HADOOP-9046.patch
>
>
> The class org.apache.hadoop.fs.DelegationTokenRenewer.RenewAction has zero
> coverage in entire cumulative test run. Provide test(s) to cover this class.
> Note: the request submitted to HDFS project because the class likely to be
> tested by tests in that project.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9046) provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewAction
[ https://issues.apache.org/jira/browse/HADOOP-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HADOOP-9046: --- Affects Version/s: 0.23.6 2.0.3-alpha 3.0.0 > provide unit-test coverage of class > org.apache.hadoop.fs.DelegationTokenRenewer.RenewAction > -- > > Key: HADOOP-9046 > URL: https://issues.apache.org/jira/browse/HADOOP-9046 > Project: Hadoop Common > Issue Type: Test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 >Reporter: Ivan A. Veselovsky >Assignee: Ivan A. Veselovsky >Priority: Minor > Attachments: HADOOP-9046-branch-0.23--c.patch, > HADOOP-9046-branch-0.23-over-9049.patch, HADOOP-9046-branch-0.23.patch, > HADOOP-9046--c.patch, HADOOP-9046-over-9049.patch, HADOOP-9046.patch > > > The class org.apache.hadoop.fs.DelegationTokenRenewer.RenewAction has zero > coverage in entire cumulative test run. Provide test(s) to cover this class. > Note: the request submitted to HDFS project because the class likely to be > tested by tests in that project. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9046) provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewAction
[ https://issues.apache.org/jira/browse/HADOOP-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HADOOP-9046: --- Attachment: HADOOP-9046-branch-0.23--c.patch Patch "HADOOP-9046-branch-0.23--c.patch" ports the changes to "branch-0.23". The trunk patch "HADOOP-9046--c.patch" is applicable to "branch-2" as well. > provide unit-test coverage of class > org.apache.hadoop.fs.DelegationTokenRenewer.RenewAction > -- > > Key: HADOOP-9046 > URL: https://issues.apache.org/jira/browse/HADOOP-9046 > Project: Hadoop Common > Issue Type: Test >Reporter: Ivan A. Veselovsky >Assignee: Ivan A. Veselovsky >Priority: Minor > Attachments: HADOOP-9046-branch-0.23--c.patch, > HADOOP-9046-branch-0.23-over-9049.patch, HADOOP-9046-branch-0.23.patch, > HADOOP-9046--c.patch, HADOOP-9046-over-9049.patch, HADOOP-9046.patch > > > The class org.apache.hadoop.fs.DelegationTokenRenewer.RenewAction has zero > coverage in entire cumulative test run. Provide test(s) to cover this class. > Note: the request submitted to HDFS project because the class likely to be > tested by tests in that project. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9044) add FindClass main class to provide classpath checking of installations
[ https://issues.apache.org/jira/browse/HADOOP-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-9044: --- Affects Version/s: 2.0.3-alpha Status: Patch Available (was: In Progress) > add FindClass main class to provide classpath checking of installations > --- > > Key: HADOOP-9044 > URL: https://issues.apache.org/jira/browse/HADOOP-9044 > Project: Hadoop Common > Issue Type: New Feature > Components: util >Affects Versions: 1.1.0, 2.0.3-alpha >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-9044.patch, HADOOP-9044.patch > > > It's useful in postflight checking of a hadoop installation to verify that > classes load, especially codes with external JARs and native codecs. > An entry point designed to load a named class and create an instance of that > class can do this -and be invoked from any shell script or tool that does the > installation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9044) add FindClass main class to provide classpath checking of installations
[ https://issues.apache.org/jira/browse/HADOOP-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-9044: --- Attachment: HADOOP-9044.patch patch against trunk > add FindClass main class to provide classpath checking of installations > --- > > Key: HADOOP-9044 > URL: https://issues.apache.org/jira/browse/HADOOP-9044 > Project: Hadoop Common > Issue Type: New Feature > Components: util >Affects Versions: 1.1.0, 2.0.3-alpha >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-9044.patch, HADOOP-9044.patch > > > It's useful in postflight checking of a hadoop installation to verify that > classes load, especially codes with external JARs and native codecs. > An entry point designed to load a named class and create an instance of that > class can do this -and be invoked from any shell script or tool that does the > installation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9046) provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewAction
[
https://issues.apache.org/jira/browse/HADOOP-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505561#comment-13505561
]
Hadoop QA commented on HADOOP-9046:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12555168/HADOOP-9046--c.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1834//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1834//console
This message is automatically generated.
> provide unit-test coverage of class
> org.apache.hadoop.fs.DelegationTokenRenewer.RenewAction
> --
>
> Key: HADOOP-9046
> URL: https://issues.apache.org/jira/browse/HADOOP-9046
> Project: Hadoop Common
> Issue Type: Test
>Reporter: Ivan A. Veselovsky
>Assignee: Ivan A. Veselovsky
>Priority: Minor
> Attachments: HADOOP-9046-branch-0.23-over-9049.patch,
> HADOOP-9046-branch-0.23.patch, HADOOP-9046--c.patch,
> HADOOP-9046-over-9049.patch, HADOOP-9046.patch
>
>
> The class org.apache.hadoop.fs.DelegationTokenRenewer.RenewAction has zero
> coverage in entire cumulative test run. Provide test(s) to cover this class.
> Note: the request submitted to HDFS project because the class likely to be
> tested by tests in that project.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9046) provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewAction
[ https://issues.apache.org/jira/browse/HADOOP-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HADOOP-9046: --- Attachment: HADOOP-9046--c.patch In the new version of patch "HADOOP-9046--c.patch" the following is done: 1) fixed the sync problem found by Robert: now the method org.apache.hadoop.fs.DelegationTokenRenewer#removeRenewAction(T) will not block for long time. 2) in the test all the time periods made scalable relative one constant. 3) the test made fater: now it takes ~7 seconds. > provide unit-test coverage of class > org.apache.hadoop.fs.DelegationTokenRenewer.RenewAction > -- > > Key: HADOOP-9046 > URL: https://issues.apache.org/jira/browse/HADOOP-9046 > Project: Hadoop Common > Issue Type: Test >Reporter: Ivan A. Veselovsky >Assignee: Ivan A. Veselovsky >Priority: Minor > Attachments: HADOOP-9046-branch-0.23-over-9049.patch, > HADOOP-9046-branch-0.23.patch, HADOOP-9046--c.patch, > HADOOP-9046-over-9049.patch, HADOOP-9046.patch > > > The class org.apache.hadoop.fs.DelegationTokenRenewer.RenewAction has zero > coverage in entire cumulative test run. Provide test(s) to cover this class. > Note: the request submitted to HDFS project because the class likely to be > tested by tests in that project. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[ https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505509#comment-13505509 ] Daryn Sharp commented on HADOOP-8989: - Sorry, this fell off my radar. I've been swamped. bq. Has anyone studied the impact on the NN yet? W/o having reviewed the patch, I initially don't think this could be any worse than ls -R. > hadoop dfs -find feature > > > Key: HADOOP-8989 > URL: https://issues.apache.org/jira/browse/HADOOP-8989 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Marco Nicosia >Assignee: Jonathan Allen > Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch > > > Both sysadmins and users make frequent use of the unix 'find' command, but > Hadoop has no correlate. Without this, users are writing scripts which make > heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs > -lsr is somewhat taxing on the NameNode, and a really slow experience on the > client side. Possibly an in-NameNode find operation would be only a bit more > taxing on the NameNode, but significantly faster from the client's point of > view? > The minimum set of options I can think of which would make a Hadoop find > command generally useful is (in priority order): > * -type (file or directory, for now) > * -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments) > * -print0 (for piping to xargs -0) > * -depth > * -owner/-group (and -nouser/-nogroup) > * -name (allowing for shell pattern, or even regex?) > * -perm > * -size > One possible special case, but could possibly be really cool if it ran from > within the NameNode: > * -delete > The "hadoop dfs -lsr | hadoop dfs -rm" cycle is really, really slow. > Lower priority, some people do use operators, mostly to execute -or searches > such as: > * find / \(-nouser -or -nogroup\) > Finally, I thought I'd include a link to the [Posix spec for > find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7868) Hadoop native fails to compile when default linker option is -Wl,--as-needed
[ https://issues.apache.org/jira/browse/HADOOP-7868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505422#comment-13505422 ] Hudson commented on HADOOP-7868: Integrated in Hadoop-Hdfs-0.23-Build #449 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/449/]) HADOOP-7868. Hadoop native fails to compile when default linker option is -Wl,--as-needed. (Trevor Robinson via tgraves) (Revision 1414370) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414370 Files : * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/acinclude.m4 * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/configure.ac > Hadoop native fails to compile when default linker option is -Wl,--as-needed > > > Key: HADOOP-7868 > URL: https://issues.apache.org/jira/browse/HADOOP-7868 > Project: Hadoop Common > Issue Type: Bug > Components: native >Affects Versions: 1.0.0, 2.0.0-alpha > Environment: Ubuntu Precise, Ubuntu Oneiric, Debian Unstable >Reporter: James Page >Assignee: Trevor Robinson > Fix For: 1.2.0, 2.0.2-alpha, 0.23.6 > > Attachments: hadoop-7868-b1.txt, HADOOP-7868.patch, > HADOOP-7868-portable.patch > > > Recent releases of Ubuntu and Debian have switched to using --as-needed as > default when linking binaries. > As a result the AC_COMPUTE_NEEDED_DSO fails to find the required DSO names > during execution of configure resulting in a build failure. > Explicitly using "-Wl,--no-as-needed" in this macro when required resolves > this issue. > See http://wiki.debian.org/ToolChain/DSOLinking for a few more details -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9084) TotalOrderPartitioner fails on hadoop running on top of gpfs (or any parallel or distributed filesystem)
[ https://issues.apache.org/jira/browse/HADOOP-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505360#comment-13505360 ] Hudson commented on HADOOP-9084: Integrated in Hadoop-Yarn-trunk #50 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/50/]) HADOOP-9064. Ameding CHANGES.txt with correct JIRA ID, wrongfully used HADOOP-9084 before (tucu) (Revision 1414347) HADOOP-9084. Augment DelegationTokenRenewer API to cancel the tokens on calls to removeRenewAction. (kkambatl via tucu) (Revision 1414326) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414347 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414326 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegationTokenRenewer.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDelegationTokenRenewer.java > TotalOrderPartitioner fails on hadoop running on top of gpfs (or any parallel > or distributed filesystem) > > > Key: HADOOP-9084 > URL: https://issues.apache.org/jira/browse/HADOOP-9084 > Project: Hadoop Common > Issue Type: Bug > Components: filecache, fs, native >Affects Versions: 1.0.3, 1.0.4 >Reporter: giovanni delussu >Assignee: giovanni delussu >Priority: Critical > Fix For: 1.0.3, 1.0.4 > > Attachments: PATCH-HADOOP-9084-origin-branch-1.0.patch > > > When running a job who uses TotalOrderPartitioner (like TeraSort or > BulkImport of HBase) on hadoop running on top of gpfs (instead of hdfs) the > program fails to find the file _partition.lst because is looking for it in > the wrong directory. The confusion is between local fs meaning not hdfs and > local meaning distributed fs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9064) Augment DelegationTokenRenewer API to cancel the tokens on calls to removeRenewAction
[
https://issues.apache.org/jira/browse/HADOOP-9064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505361#comment-13505361
]
Hudson commented on HADOOP-9064:
Integrated in Hadoop-Yarn-trunk #50 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/50/])
HADOOP-9064. Ameding CHANGES.txt with correct JIRA ID, wrongfully used
HADOOP-9084 before (tucu) (Revision 1414347)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414347
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
> Augment DelegationTokenRenewer API to cancel the tokens on calls to
> removeRenewAction
> -
>
> Key: HADOOP-9064
> URL: https://issues.apache.org/jira/browse/HADOOP-9064
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.0.2-alpha
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Fix For: 2.0.3-alpha
>
> Attachments: hadoop-9064.patch, hadoop-9064.patch, hadoop-9064.patch,
> hadoop-9064.patch
>
>
> Post HADOOP-9049, FileSystems register with DelegationTokenRenewer (a
> singleton), to renew tokens.
> To avoid a bunch of defunct tokens clog the NN, we should augment the API to
> {{#removeRenewAction(boolean cancel)}} and cancel the token appropriately.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
