[jira] [Resolved] (HADOOP-3470) Bad coding style: The member fields in org.apache.hadoop.ipc.metrics.RpcMetrics are public

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-3470.
--

Resolution: Won't Fix

Closing this as won't fix given the metrics system has been replaced.

> Bad coding style: The member fields in 
> org.apache.hadoop.ipc.metrics.RpcMetrics are public
> --
>
> Key: HADOOP-3470
> URL: https://issues.apache.org/jira/browse/HADOOP-3470
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: metrics
>Reporter: Tsz Wo Nicholas Sze
>
> In org.apache.hadoop.ipc.metrics.RpcMetrics,
> {code}
> //the following are member fields
>   public MetricsTimeVaryingRate rpcQueueTime = new 
> MetricsTimeVaryingRate("RpcQueueTime");
>   public MetricsTimeVaryingRate rpcProcessingTime = new 
> MetricsTimeVaryingRate("RpcProcessingTime");
>   public Map  metricsList = 
> Collections.synchronizedMap(new HashMap());
> {code}
> Then, the fields are accessed directly in other classes.  For example, 
> org.apache.hadoop.ipc.RPC.Server.call(...)
> {code}
> ...
>   MetricsTimeVaryingRate m = 
> rpcMetrics.metricsList.get(call.getMethodName());
>   if (m != null) {
>   m.inc(processingTime);
>   }
>   else {
>   rpcMetrics.metricsList.put(call.getMethodName(), new 
> MetricsTimeVaryingRate(call.getMethodName()));
>   m = rpcMetrics.metricsList.get(call.getMethodName());
>   m.inc(processingTime);
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-3437) mapred.job.tracker default value/docs appear out of sync with code

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-3437.
--

Resolution: Incomplete

I'm going to close this as stale.

> mapred.job.tracker default value/docs appear out of sync with code
> --
>
> Key: HADOOP-3437
> URL: https://issues.apache.org/jira/browse/HADOOP-3437
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: conf
>Affects Versions: 0.18.0
>Reporter: Steve Loughran
>Priority: Minor
>
> This is the settings for mapred.job.tracker in SVN_HEAD. If you dont override 
> these, you get a RuntimeException
> 
>   mapred.job.tracker
>   local
>   The host and port that the MapReduce job tracker runs
>   at.  If "local", then jobs are run in-process as a single map
>   and reduce task.
>   
> 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-2681) NullPointerException in TaskRunner.java when system property "hadoop.log.dir" is not set

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-2681.
--

Resolution: Duplicate

Closing this as a duplicate of HADOOP-3438.

> NullPointerException in TaskRunner.java when system property "hadoop.log.dir" 
> is not set
> 
>
> Key: HADOOP-2681
> URL: https://issues.apache.org/jira/browse/HADOOP-2681
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: conf
>Affects Versions: 0.15.2
>Reporter: Xu Zhang
>Priority: Minor
>  Labels: newbie
>
> Currently, NullPointerException exception is thrown on line 321 in 
> TaskRunner.java when system property "hadoop.log.dir" is not set.  Instead of 
> a NullPointerException exception, I expected a default value for 
> "hadoop.log.dir" to be used, or to see a more meaningful error message that 
> could have helped me figure out what was wrong (like, telling me that I 
> needed to set "hadoop.log.dir" and how to do so).
> Here is one instance of such exceptions:
> WARN mapred.TaskRunner: task_200801181719_0001_m_00_0 Child Error
> java.lang.NullPointerException
>   at java.io.File.(File.java:222)
>   at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:321)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-3249) Browsing DFS behind gateway

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-3249.
--

Resolution: Won't Fix

Closing this as won't fix. Proxy solutions are totally realistic.  Plus Apache 
Knox provides some of this functionality.

> Browsing DFS behind gateway
> ---
>
> Key: HADOOP-3249
> URL: https://issues.apache.org/jira/browse/HADOOP-3249
> Project: Hadoop Common
>  Issue Type: Wish
> Environment: Red-Hat cluster
>Reporter: Martin Boeker
>Priority: Minor
>   Original Estimate: 5h
>  Remaining Estimate: 5h
>
> Dear Hadoop guys,
> I'm urgently trying to make a way for users to be able to see the contents of 
> a Hadoop DFS that is behind a gateway. I'm using port forwarding on the 
> gateway itself to point to the DFS web interface, something like this:
> [gateway_external_IP]:50070 >> [node_internal_IP]:50070
> This works fine, if I go to http://gateway_external_ip:50070/ I can view the 
> DFS cluster html page from the outside world. The problem is that if I click 
> on any of the slave node links, it forwards to http://node_hostname/.., which 
> obviously doesn't work. I really need to get this going, a couple of projects 
> require this to be implemented.
> I'm willing to do this any way possible, I don't really need to use the 50070 
> web interface, even a simple directory structure would do, but I'm not sure 
> how to implement that either, because I don't know of a way to make an httpd 
> or ftpd use "bin/hadoop dfs -lsr /" as the root directory. I'd also be 
> willing to make people use a proxy server if that would fix my issue somehow..
> If anyone can help, I would greatly appreciate it, like I said it's kind of 
> urgent and I'm running out of ideas to try..
> Thanks a lot in advance,
> -Martin



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-3228) Replace Text.hashcode with a better hash function for non-ascii strings

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-3228.
--

Resolution: Fixed

I'm going to close this as fixed.

> Replace Text.hashcode with a better hash function for non-ascii strings
> ---
>
> Key: HADOOP-3228
> URL: https://issues.apache.org/jira/browse/HADOOP-3228
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Owen O'Malley
>
> Should we change the hash function for Text to something that handles 
> non-ascii characters better?
> http://bailey.svn.sourceforge.net/viewvc/bailey/trunk/src/java/org/apache/bailey/util/Hash.java?view=markup



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-3231) move Stringifier to org.apache.hadoop.io.serialization package and rename DefaultStringifier to Stringifier

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-3231.
--

Resolution: Incomplete

closing this as stale.

> move Stringifier to org.apache.hadoop.io.serialization package and rename 
> DefaultStringifier to Stringifier
> ---
>
> Key: HADOOP-3231
> URL: https://issues.apache.org/jira/browse/HADOOP-3231
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: io
>Reporter: Owen O'Malley
>
> The Stringinfier already uses the Serializer under the hood, and thus doesn't 
> really need to be parameterized further. I propose that we move the 
> DefaultStringifier to org.apache.hadoop.io.serializer.Stringifier and remove 
> the Stringifier interface.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-353) Run datanode (or other hadoop servers) inside tomcat

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-353.
-

Resolution: Won't Fix

I'm closing this out as Won't Fix.  At this point, I think there is zero 
interesting in making Hadoop run inside another app server.

> Run datanode (or other hadoop servers) inside tomcat
> 
>
> Key: HADOOP-353
> URL: https://issues.apache.org/jira/browse/HADOOP-353
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: eric baldeschwieler
>
> Barry Kaplan is running hadoop data nodes inside tomcat and encountering some 
> issues.
> http://issues.apache.org/jira/browse/HADOOP-211#action_12419360
> I'm filing this bug to capture discussion about the pros and cons of such an 
> approach.  I'd be curious to know what others on the list (who know more 
> about java/tomcat) think about this proposal.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-145) io.skip.checksum.errors property clashes with LocalFileSystem#reportChecksumFailure

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-145.
-

Resolution: Fixed

Long ago.

> io.skip.checksum.errors property clashes with 
> LocalFileSystem#reportChecksumFailure
> ---
>
> Key: HADOOP-145
> URL: https://issues.apache.org/jira/browse/HADOOP-145
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: io
>Reporter: stack
>Assignee: Owen O'Malley
>
> Below is from email to the dev list on Tue, 11 Apr 2006 14:46:09 -0700.
> Checksum errors seem to be a fact of life given the hardware we use.  They'll 
> often cause my jobs to fail so I have been trying to figure how to just skip 
> the bad records and files.  At the end is a note where Stefan pointed me at 
> 'io.skip.checksum.errors'.   This property, when set, triggers special 
> handling of checksum errors inside SequenceFile$Reader: If a checksum, try to 
> skip to next record.  Only, this behavior can conflict with another checksum 
> handler that moves aside the problematic file whenever a checksum error is 
> found.  Below is from a recent log.
> 060411 202203 task_r_22esh3  Moving bad file 
> /2/hadoop/tmp/task_r_22esh3/task_m_e3chga.out to 
> /2/bad_files/task_m_e3chga.out.1707416716
> 060411 202203 task_r_22esh3  Bad checksum at 3578152. Skipping entries.
> 060411 202203 task_r_22esh3  Error running child
> 060411 202203 task_r_22esh3 java.nio.channels.ClosedChannelException
> 060411 202203 task_r_22esh3 at 
> sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:89)
> 060411 202203 task_r_22esh3 at 
> sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:276)
> 060411 202203 task_r_22esh3 at 
> org.apache.hadoop.fs.LocalFileSystem$LocalFSFileInputStream.seek(LocalFileSystem.java:79)
> 060411 202203 task_r_22esh3 at 
> org.apache.hadoop.fs.FSDataInputStream$Checker.seek(FSDataInputStream.java:67)
> 060411 202203 task_r_22esh3 at 
> org.apache.hadoop.fs.FSDataInputStream$PositionCache.seek(FSDataInputStream.java:164)
> 060411 202203 task_r_22esh3 at 
> org.apache.hadoop.fs.FSDataInputStream$Buffer.seek(FSDataInputStream.java:193)
> 060411 202203 task_r_22esh3 at 
> org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:243)
> 060411 202203 task_r_22esh3 at 
> org.apache.hadoop.io.SequenceFile$Reader.seek(SequenceFile.java:420)
> 060411 202203 task_r_22esh3 at 
> org.apache.hadoop.io.SequenceFile$Reader.sync(SequenceFile.java:431)
> 060411 202203 task_r_22esh3 at 
> org.apache.hadoop.io.SequenceFile$Reader.handleChecksumException(SequenceFile.java:412)
> 060411 202203 task_r_22esh3 at 
> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:389)
> 060411 202203 task_r_22esh3 at 
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:209)
> 060411 202203 task_r_22esh3 at 
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:709)
> (Ignore line numbers.  My code is a little different from main because I've 
> other debugging code inside in SequenceFile.  Otherwise I'm running w/ head 
> of hadoop).
> The SequenceFile$Reader#handleChecksumException is trying to skip to next 
> record but the file has been closed by the move-aside.
> On the list there is some discussion on merit of moving aside file when bad 
> checksum found.  I've trying to test what happens if we leave the file in 
> place but haven't had a checksum error in a while.  
> Opening this issue so place to fill in experience as we go.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Why Hadoop-trunk-commit always fails?

[Ted Yu]

Here is the warning from enforcer:

[WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence
failed with message:
Failed while enforcing releasability the error(s) are [
Dependency convergence error for
org.apache.hadoop:hadoop-auth:3.0.0-20140718.043141-4847 paths to
dependency are:
+-org.apache.hadoop:hadoop-kms:3.0.0-SNAPSHOT
  +-org.apache.hadoop:hadoop-auth:3.0.0-20140718.043141-4847
and
+-org.apache.hadoop:hadoop-kms:3.0.0-SNAPSHOT
  +-org.apache.hadoop:hadoop-common:3.0.0-20140718.043201-4831
+-org.apache.hadoop:hadoop-auth:3.0.0-SNAPSHOT
and
+-org.apache.hadoop:hadoop-kms:3.0.0-SNAPSHOT
  +-org.apache.hadoop:hadoop-common:3.0.0-20140718.043201-4831
+-org.apache.hadoop:hadoop-auth:3.0.0-SNAPSHOT
]

FYI


On Thu, Jul 17, 2014 at 9:38 PM, Vinayakumar B 
wrote:

> Hi,
> Hadoop-trunk-commit build always fails with message similar to below.
> Anybody knows about this?
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-enforcer-plugin:1.3.1:enforce
> (depcheck) on project hadoop-yarn-server-tests: Some Enforcer rules
> have failed. Look above for specific messages explaining why the rule
> failed. -> [Help 1]
>
>
>
> Regards,
> Vinay
>

Why Hadoop-trunk-commit always fails?

[Vinayakumar B]

Hi,
Hadoop-trunk-commit build always fails with message similar to below.
Anybody knows about this?

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-enforcer-plugin:1.3.1:enforce
(depcheck) on project hadoop-yarn-server-tests: Some Enforcer rules
have failed. Look above for specific messages explaining why the rule
failed. -> [Help 1]



Regards,
Vinay

[jira] [Created] (HADOOP-10857) Native Libraries Guide doen't mention a dependency on openssl-development

[Tsuyoshi OZAWA (JIRA)]

Tsuyoshi OZAWA created HADOOP-10857:
---

 Summary: Native Libraries Guide doen't mention a dependency on 
openssl-development
 Key: HADOOP-10857
 URL: https://issues.apache.org/jira/browse/HADOOP-10857
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA


{{maven compile -Pnative}} fails without installing openssl-development 
package(e.g. libssl-dev). We should describe it in Native Libraries Guide.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-3262) make Hadoop compile under Apache Harmony

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-3262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-3262.
--

Resolution: Won't Fix

RIP Harmony.





> make Hadoop compile under Apache Harmony
> 
>
> Key: HADOOP-3262
> URL: https://issues.apache.org/jira/browse/HADOOP-3262
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: build
>Reporter: Doug Cutting
> Attachments: HADOOP-3262.patch
>
>
> Some small changes are required to get Hadoop Core to compile with Apache 
> Harmony.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-3148) build-contrib.xml should inherit hadoop version parameter from root build.xml

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-3148.
--

Resolution: Fixed

Stale issue.

> build-contrib.xml should inherit hadoop version parameter from root build.xml
> -
>
> Key: HADOOP-3148
> URL: https://issues.apache.org/jira/browse/HADOOP-3148
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: build
>Reporter: Vinod Kumar Vavilapalli
>Priority: Minor
>
> This is needed in HOD (and may be useful in other contrib projects), which, 
> in some cases, may be compiled and built separately. After HADOOP-3137, HOD 
> will obtain its version from build parameter ${version}, and this will fail 
> to give proper version when built independently(at src/contrib/hod level).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-3126) org.apache.hadoop.examples.RandomTextWriter$Counters fluctuate when RandonTextWriter job is running

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-3126.
--

Resolution: Fixed

I'm going to close this as fixed since it probably was.

> org.apache.hadoop.examples.RandomTextWriter$Counters fluctuate when 
> RandonTextWriter job is running
> ---
>
> Key: HADOOP-3126
> URL: https://issues.apache.org/jira/browse/HADOOP-3126
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Runping Qi
>
> On the web GUI page, the value for RECORDS_WRITTEN and BYTES_WRITTEN do not 
> increase monotonically.
> Rather, their values go up and down.
> I suspect something wrong with how the counters are updated.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-3122) test-patch target should check @SuppressWarnings(...)

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-3122.
--

Resolution: Incomplete

Closing this as stale.

> test-patch target should check @SuppressWarnings(...)
> -
>
> Key: HADOOP-3122
> URL: https://issues.apache.org/jira/browse/HADOOP-3122
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Reporter: Tsz Wo Nicholas Sze
>Priority: Minor
>
> The Java annotation @SuppressWarnings(...) tag can be used to get rid of 
> compiler warnings. In our patch process, QA should check 
> @SuppressWarnings(...) tag to prevent abusing this tag.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-3120) Large #of tasks failing at one time can effectively hang the jobtracker

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-3120.
--

Resolution: Incomplete

I'm going to close this as stale. 

> Large #of tasks failing at one time can effectively hang the jobtracker 
> 
>
> Key: HADOOP-3120
> URL: https://issues.apache.org/jira/browse/HADOOP-3120
> Project: Hadoop Common
>  Issue Type: Bug
> Environment: Linux/Hadoop-15.3
>Reporter: Pete Wyckoff
>Priority: Minor
>
> We think that JobTracker.removeMarkedTaks does so much logging when this 
> happens (ie logging thousands of failed taks per cycle) that nothing else can 
> go on (since it's called from a synchronized method) and thus by the next 
> cycle, the next waves of jobs have failed and we again have 10s of thousands 
> of failures to log and on and on.
> At least, the above is what we observed - just a continual printing of those 
> failures and nothing else happening on and on. Of course the original jobs 
> may have ultimately failed but new jobs come in to perpetuate the problem.
> This has happened to us a number of times and since we commented out the 
> log.info in that method we haven't had any problems. Although thousands and 
> thousands of task failures are hopefully not that common.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-3037) Hudson needs to add src/test for checking javac warnings

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-3037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-3037.
--

Resolution: Not a Problem

Warning checks have been there for a while.

> Hudson needs to add src/test for checking javac warnings
> 
>
> Key: HADOOP-3037
> URL: https://issues.apache.org/jira/browse/HADOOP-3037
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Affects Versions: hudson
>Reporter: Amareshwari Sriramadasu
> Fix For: hudson
>
>
> I think src/test is not added in javac warnings checker.  HADOOP-3031 looks 
> at warnings introduced.
> Hudson needs to add src/test for checking javac warnings



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-3023) [HOD] build file requires exact python version

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-3023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-3023.
--

Resolution: Not a Problem

HOD has since been banished from the source tree.

> [HOD] build file requires exact python version
> --
>
> Key: HADOOP-3023
> URL: https://issues.apache.org/jira/browse/HADOOP-3023
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Reporter: Nigel Daley
>Assignee: Vinod Kumar Vavilapalli
> Attachments: HADOOP-3023
>
>
> To run the HOD pyunit tests, the build file checks that an exact python 
> version is present:
> +  
> +
> +
> +  
> This is too restrictive (we have 2.5.2 on our Solaris build machine).  I 
> suggest that you don't check the python version and instead issue a warning 
> message if the pyunit test fail stating that python 2.5.1 or great is 
> required (and optionally showing them what version they tried to use).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-2980) slow reduce copies - map output locations not being fetched even when map complete

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-2980.
--

Resolution: Incomplete

I'm going to close this as stale.  If people are still seeing this as an issue, 
they should file a new jira with new data!

> slow reduce copies - map output locations not being fetched even when map 
> complete
> --
>
> Key: HADOOP-2980
> URL: https://issues.apache.org/jira/browse/HADOOP-2980
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 0.15.3
>Reporter: Joydeep Sen Sarma
>
> maps are long finished. reduces are stuck looking for map locations. they 
> make progress - but slowly. it almost seems like they get new map locations 
> every minute or so:
> 2008-03-07 18:50:52,737 INFO org.apache.hadoop.mapred.ReduceTask: 
> task_200803041231_3586_r_21_0 done copying 
> task_200803041231_3586_m_004620_0 output from hadoop082.sf2p.facebook.com..
> 2008-03-07 18:50:53,733 INFO org.apache.hadoop.mapred.ReduceTask: 
> task_200803041231_3586_r_21_0: Got 0 new map-outputs & 0 obsolete 
> map-outputs from tasktracker and 0 map-outputs from previous failures
> 2008-03-07 18:50:53,733 INFO org.apache.hadoop.mapred.ReduceTask: 
> task_200803041231_3586_r_21_0 Got 0 known map output location(s); 
> scheduling...
> ...
> 2008-03-07 18:51:49,767 INFO org.apache.hadoop.mapred.ReduceTask: 
> task_200803041231_3586_r_21_0 Got 50 known map output location(s); 
> scheduling...
> 2008-03-07 18:51:49,767 INFO org.apache.hadoop.mapred.ReduceTask: 
> task_200803041231_3586_r_21_0 Scheduled 41 of 50 known outputs (0 slow 
> hosts and 9 dup hosts)
> they get about 50 locations at a time and this 1 minute delay pattern is 
> surprisingly common ..



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-2975) IPC server should not allocate a buffer for each request

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-2975.
--

Resolution: Incomplete

A lot has changed here. Closing as stale.

> IPC server should not allocate a buffer for each request
> 
>
> Key: HADOOP-2975
> URL: https://issues.apache.org/jira/browse/HADOOP-2975
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: ipc
>Affects Versions: 0.16.0
>Reporter: Hairong Kuang
>Assignee: Ankur
> Attachments: Hadoop-2975-v1.patch, Hadoop-2975-v2.patch, 
> Hadoop-2975-v3.patch
>
>
> Currently the IPC server allocates a buffer for each incoming request. The 
> buffer is thrown away after the request is serialized. This leads to very 
> inefficient heap utilization. It would be nicer if all requests from one 
> connection could share a same common buffer since the ipc server has only one 
> request is being read from a socket at a time.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-2960) A mapper should use some heuristics to decide whether to run the combiner during spills

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-2960.
--

Resolution: Won't Fix

Closing at won't fix, given the -1.

> A mapper should use some heuristics to decide whether to run the combiner 
> during spills
> ---
>
> Key: HADOOP-2960
> URL: https://issues.apache.org/jira/browse/HADOOP-2960
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Runping Qi
>
> Right now, the combiner, if set, will be called for each spill, no mapper 
> whether the combiner can actually reduce the values.
> The mapper should use some heuristics to decide whether to run the combiner 
> during spills.
> One of such heuristics is to check the the ratio of  the nymber of keys to 
> the number of unique keys in the spill.
> The combiner will be called only if that ration exceeds certain threshold 
> (say 2).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-2922) sequencefiles without keys

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-2922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-2922.
--

Resolution: Won't Fix

Stale

> sequencefiles without keys
> --
>
> Key: HADOOP-2922
> URL: https://issues.apache.org/jira/browse/HADOOP-2922
> Project: Hadoop Common
>  Issue Type: New Feature
>Affects Versions: 0.16.0
>Reporter: Joydeep Sen Sarma
>
> sequencefiles are invaluable for storing compressed/binary data. but when we 
> use them to store serialized records - we don't use the key part at all (just 
> put something dummy there to satisfy the api). i have heard of other projects 
> using the same tactics (jaql/cascading).
> so this is a request to have a modified version of sequencefiles that don't 
> incur the space and compute overhead of processing/storing these dummy keys.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-2921) align map splits on sorted files with key boundaries

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-2921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-2921.
--

Resolution: Fixed

Stale

> align map splits on sorted files with key boundaries
> 
>
> Key: HADOOP-2921
> URL: https://issues.apache.org/jira/browse/HADOOP-2921
> Project: Hadoop Common
>  Issue Type: New Feature
>Affects Versions: 0.16.0
>Reporter: Joydeep Sen Sarma
>
> (this is something that we have implemented in the application layer - may be 
> useful to have in hadoop itself).
> long term log storage systems often keep data sorted (by some sort-key). 
> future computations on such files can often benefit from this sort order. if 
> the job requires grouping by the sort-key - then it should be possible to do 
> reduction in the map stage itself.
> this is not natively supported by hadoop (except in the degenerate case of 1 
> map file per task) since splits can span the sort-key. however aligning the 
> data read by the map task  to sort key boundaries is straightforward - and 
> this would be a useful capability to have in hadoop.
> the definition of the sort key should be left up to the application (it's not 
> necessarily the key field in a Sequencefile) through a generic interface - 
> but otherwise - the sequencefile and text file readers can use the extracted 
> sort key to align map task data with key boundaries.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-2892) providing temp space management for applications

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-2892.
--

Resolution: Duplicate

In one sense, this has been resolved by the usage of /tmp.  But in reality, 
this request has been reborn in HDFS-6382.

> providing temp space management for applications
> 
>
> Key: HADOOP-2892
> URL: https://issues.apache.org/jira/browse/HADOOP-2892
> Project: Hadoop Common
>  Issue Type: New Feature
>Reporter: Olga Natkovich
>
> It would be greate if hadoop can provide temp space for applications to use. 
> This would be useful for any applications that chain M-R jobs, perform 
> checkpoint and need to store some application specific temp results. 
> DeleteOnExit for files and directories would be ideal.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-2882) HOD should put line breaks in to hadoop-site.xml

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-2882.
--

Resolution: Won't Fix

HOD is just a legend. Did it really exist? No one knows.

> HOD should put line breaks in to hadoop-site.xml
> 
>
> Key: HADOOP-2882
> URL: https://issues.apache.org/jira/browse/HADOOP-2882
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Owen O'Malley
>
> It would help a lot if the hadoop-site files generated by HOD were readable. 
> Newlines would be a good start.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-2864) Improve the Scalability and Robustness of IPC

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-2864.
--

Resolution: Fixed

This has changed so much since this JIRA was filed that I'm just going to close 
this as stale.

> Improve the Scalability and Robustness of IPC
> -
>
> Key: HADOOP-2864
> URL: https://issues.apache.org/jira/browse/HADOOP-2864
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: ipc
>Affects Versions: 0.16.0
>Reporter: Hairong Kuang
>Assignee: Hairong Kuang
> Attachments: RPCScalabilityDesignWeb.pdf
>
>
> This jira is intended to enhance IPC's scalability and robustness. 
> Currently an IPC server can easily hung due to a disk failure or garbage 
> collection, during which it cannot respond to the clients promptly. This has 
> caused a lot of dropped calls and delayed responses thus many running 
> applications fail on timeout. On the other side if busy clients send a lot of 
> requests to the server in a short period of time or too many clients 
> communicate with the server simultaneously, the server may be swarmed by 
> requests and cannot work responsively. 
> The proposed changes aim to 
> # provide a better client/server coordination
> #* Server should be able to throttle client during burst of requests.
> #* A slow client should not affect server from serving other clients.
> #* A temporary hanging server should not cause catastrophic failures to 
> clients.
> # Client/server should detect remote side failures. Examples of failures 
> include: (1) the remote host is crashed; (2) the remote host is crashed and 
> then rebooted; (3) the remote process is crashed or shut down by an operator;
> # Fairness. Each client should be able to make progress.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-2860) "ant tar" should not copy the modified configs into the tarball

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-2860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-2860.
--

Resolution: Won't Fix

We no longer use ant.

(insert pink panther theme here, using the words "dead ant" to represent the 
horn section)

> "ant tar" should not copy the modified configs into the tarball
> ---
>
> Key: HADOOP-2860
> URL: https://issues.apache.org/jira/browse/HADOOP-2860
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: build
>Reporter: Owen O'Malley
>
> When generating releases, it is counter-intuitive that the tarball contains 
> the configuration files from the developer's test environment.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-2846) Large input data-sets throw "java.net.SocketTimeoutException: timed out waiting for rpc response" exception

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-2846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-2846.
--

Resolution: Cannot Reproduce

Closing this as stale, esp since it is likely long since fixed.

> Large input data-sets throw "java.net.SocketTimeoutException: timed out 
> waiting for rpc response" exception
> ---
>
> Key: HADOOP-2846
> URL: https://issues.apache.org/jira/browse/HADOOP-2846
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 0.15.3
>Reporter: Amir Youssefi
>
> Pig scripts can run over a data set of 1 day. Using the same script and same 
> number of nodes on a larger data set (of 30 days) fails and throws following 
> exception after 1+ hour of running. 
> java.net.SocketTimeoutException: timed out waiting for rpc response
> at org.apache.hadoop.ipc.Client.call(Client.java:484)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:184)
> at $Proxy1.getJobStatus(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy1.getJobStatus(Unknown Source)
> at 
> org.apache.hadoop.mapred.JobClient$NetworkedJob.ensureFreshStatus(JobClient.java:182)
> at 
> org.apache.hadoop.mapred.JobClient$NetworkedJob.isComplete(JobClient.java:237)
> at 
> org.apache.pig.impl.mapreduceExec.MapReduceLauncher.launchPig(MapReduceLauncher.java:189)
> at 
> org.apache.pig.impl.physicalLayer.POMapreduce.open(POMapreduce.java:136)
> at 
> org.apache.pig.impl.physicalLayer.POMapreduce.open(POMapreduce.java:129)
> at 
> org.apache.pig.impl.physicalLayer.POMapreduce.open(POMapreduce.java:129)
> at 
> org.apache.pig.impl.physicalLayer.PhysicalPlan.exec(PhysicalPlan.java:39)
> at 
> org.apache.pig.impl.physicalLayer.IntermedResult.exec(IntermedResult.java:122)
> at org.apache.pig.PigServer.store(PigServer.java:445)
> at org.apache.pig.PigServer.store(PigServer.java:413)
> at 
> org.apache.pig.tools.grunt.GruntParser.processStore(GruntParser.java:135)
> at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:327)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:54)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:54)
> at org.apache.pig.Main.main(Main.java:258)
> timed out waiting for rpc response
> Re-runing always hits the same at %3 progress.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-2835) "hadoop fs -help ..." should not require a NameNode to show help messages

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-2835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-2835.
--

Resolution: Fixed

Long ago fixed.

> "hadoop fs -help ..." should not require a NameNode to show help messages
> -
>
> Key: HADOOP-2835
> URL: https://issues.apache.org/jira/browse/HADOOP-2835
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Reporter: Tsz Wo Nicholas Sze
>Priority: Minor
>
> For example, if we do "hadoop fs -help get" before started a NameNode, we 
> will get 
> {code}
> bash-3.2$ ./bin/hadoop fs -help get
> 08/02/14 15:59:52 INFO ipc.Client: Retrying connect to server: 
> some-host:some-port. Already tried 1 time(s).
> 08/02/14 15:59:54 INFO ipc.Client: Retrying connect to server: 
> some-host:some-port. Already tried 2 time(s).
> 08/02/14 15:59:56 INFO ipc.Client: Retrying connect to server: 
> some-host:some-port. Already tried 3 time(s).
> 08/02/14 15:59:58 INFO ipc.Client: Retrying connect to server: 
> some-host:some-port. Already tried 4 time(s).
> 08/02/14 16:00:00 INFO ipc.Client: Retrying connect to server: 
> some-host:some-port. Already tried 5 time(s).
> 08/02/14 16:00:02 INFO ipc.Client: Retrying connect to server: 
> some-host:some-port. Already tried 6 time(s).
> 08/02/14 16:00:04 INFO ipc.Client: Retrying connect to server: 
> some-host:some-port. Already tried 7 time(s).
> 08/02/14 16:00:06 INFO ipc.Client: Retrying connect to server: 
> some-host:some-port. Already tried 8 time(s).
> 08/02/14 16:00:08 INFO ipc.Client: Retrying connect to server: 
> some-host:some-port. Already tried 9 time(s).
> 08/02/14 16:00:10 INFO ipc.Client: Retrying connect to server: 
> some-host:some-port. Already tried 10 time(s).
> Bad connection to FS. command aborted.
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-2776) Web interface uses internal hostnames on EC2

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-2776.
--

Resolution: Won't Fix

I'm going to close this as won't fix. I don't think this is anything that we 
actually can fix here other than providing a complicated hostname mapping 
system for web interfaces.

Part of the frustration I'm sure stems from a misunderstanding of what is 
actually happening:

bq. The slaves file has the public names listed.

The slaves file is only used by the shell code to run ssh connections.  It has 
absolutely zero impact on the core of Hadoop.

bq.  Resolving a public name inside EC2 returns the private IP (which would 
reverse to the internal DNS name).

Hadoop makes the perfectly valid assumption that the hostname the system tells 
us is a valid, network-connectable hostname.  It is, from the inside of EC2.  
We have no way to know that you are attempting to connect from a completely 
different address that is being forwarded from some external entity.

Proxying connections into a private network space is a perfectly valid 
solution.  

> Web interface uses internal hostnames on EC2
> 
>
> Key: HADOOP-2776
> URL: https://issues.apache.org/jira/browse/HADOOP-2776
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: contrib/cloud
>Affects Versions: 0.15.1
> Environment: EC2 ami-a324c1ca
>Reporter: David Phillips
>
> The web interface, for example http://$MASTER_HOST:50030/machines.jsp, uses 
> internal hostnames when running on EC2.  This makes it impossible to access 
> from outside EC2.
> The slaves file has the public names listed.  Resolving a public name inside 
> EC2 returns the private IP (which would reverse to the internal DNS name).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-2608) Reading sequence file consumes 100% cpu with maximum throughput being about 5MB/sec per process

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-2608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-2608.
--

Resolution: Fixed

I'm going to close this out as stale.

> Reading sequence file consumes 100% cpu with maximum throughput being about 
> 5MB/sec per process
> ---
>
> Key: HADOOP-2608
> URL: https://issues.apache.org/jira/browse/HADOOP-2608
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: io
>Reporter: Runping Qi
>
> I did some tests on the throughput of scanning block-compressed sequence 
> files.
> The sustained throughput was bounded at 5MB/sec per process, with the cpu of 
> each process maxed at 100%.
> It seems to me that the cpu consumption is too high and the throughput is too 
> low for just scanning files.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-2560) Processing multiple input splits per mapper task

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-2560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-2560.
--

Resolution: Duplicate

This appears to predate MFIF/CFIF, as introduced by HADOOP-4565 which appears 
to fix the issue.  I'm going to close this out as resolved as a result.

> Processing multiple input splits per mapper task
> 
>
> Key: HADOOP-2560
> URL: https://issues.apache.org/jira/browse/HADOOP-2560
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Runping Qi
>Assignee: dhruba borthakur
> Attachments: multipleSplitsPerMapper.patch
>
>
> Currently, an input split contains a consecutive chunk of input file, which 
> by default, corresponding to a DFS block.
> This may lead to a large number of mapper tasks if the input data is large. 
> This leads to the following problems:
> 1. Shuffling cost: since the framework has to move M * R map output segments 
> to the nodes running reducers, 
> larger M means larger shuffling cost.
> 2. High JVM initialization overhead
> 3. Disk fragmentation: larger number of map output files means lower read 
> throughput for accessing them.
> Ideally, you want to keep the number of mappers to no more than 16 times the 
> number of  nodes in the cluster.
> To achive that, we can increase the input split size. However, if a split 
> span over more than one dfs block,
> you lose the data locality scheduling benefits.
> One way to address this problem is to combine multiple input blocks with the 
> same rack into one split.
> If in average we combine B blocks into one split, then we will reduce the 
> number of mappers by a factor of B.
> Since all the blocks for one mapper share a rack, thus we can benefit from 
> rack-aware scheduling.
> Thoughts?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-2462) MiniMRCluster does not utilize multiple local directories in "mapred.local.dir"

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-2462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-2462.
--

Resolution: Incomplete

Stale.

> MiniMRCluster does not utilize multiple local directories in 
> "mapred.local.dir"
> ---
>
> Key: HADOOP-2462
> URL: https://issues.apache.org/jira/browse/HADOOP-2462
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.15.0
>Reporter: Konstantin Shvachko
>
> My hadoop-site.xml specifies 4 local directories
> {code}
> 
>   mapred.local.dir
>   ${hadoop.tmp.dir}/mapred/local1, ${hadoop.tmp.dir}/mapred/local2, 
>  ${hadoop.tmp.dir}/mapred/local3, 
> ${hadoop.tmp.dir}/mapred/local4
> 
> {code}
> and I am looking at MiniMRCluster.TaskTrackerRunner
> There are several things here:
> # localDirBase value is set to
> {code}
> "/tmp/h/mapred/local1, /tmp/h/mapred/local2, /tmp/h/mapred/local3, 
> /tmp/h/mapred/local4"
> {code}
> and I get a hierarchy of directories with commas and spaces in the names. 
> I think this was not designed to work with multiple dirs.
> # Further down, all new directories are generated with the same name
> {code}
> File ttDir = new File(localDirBase, 
>   Integer.toString(trackerId) + "_" + 0);
> {code}
> So in fact only one directory is created. I think the intension was to have i 
> instead of 0
> {code}
> File ttDir = new File(localDirBase, 
>   Integer.toString(trackerId) + "_" + i);
> {code}
> # On windows MiniMRCluster.TaskTrackerRunner in this case throws an 
> IOException, 
> which is silently ignored by all but the TestMiniMRMapRedDebugScript   MiniMR 
> tests.
> {code}
> java.io.IOException: Mkdirs failed to create 
> /tmp/h/mapred/local1, /tmp/h/mapred/local2, /tmp/h/mapred/local3, 
> /tmp/h/mapred/local4/0_0
>   at 
> org.apache.hadoop.mapred.MiniMRCluster$TaskTrackerRunner.(MiniMRCluster.java:124)
>   at org.apache.hadoop.mapred.MiniMRCluster.(MiniMRCluster.java:293)
>   at org.apache.hadoop.mapred.MiniMRCluster.(MiniMRCluster.java:244)
>   at 
> org.apache.hadoop.mapred.TestMiniMRClasspath.testClassPath(TestMiniMRClasspath.java:163)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:585)
>   at junit.framework.TestCase.runTest(TestCase.java:154)
>   at junit.framework.TestCase.runBare(TestCase.java:127)
>   at junit.framework.TestResult$1.protect(TestResult.java:106)
>   at junit.framework.TestResult.runProtected(TestResult.java:124)
>   at junit.framework.TestResult.run(TestResult.java:109)
>   at junit.framework.TestCase.run(TestCase.java:118)
>   at junit.framework.TestSuite.runTest(TestSuite.java:208)
>   at junit.framework.TestSuite.run(TestSuite.java:203)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:478)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:344)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)
> {code}
> I am marking it as "Major" because we actually do not test multiple local 
> directories.
> Looks like it was introduced rather recently by HADOOP-1819.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HADOOP-10856) HarFileSystem and HarFs support for HDFS encryption

[Andrew Wang (JIRA)]

Andrew Wang created HADOOP-10856:


 Summary: HarFileSystem and HarFs support for HDFS encryption
 Key: HADOOP-10856
 URL: https://issues.apache.org/jira/browse/HADOOP-10856
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134)
Reporter: Andrew Wang
Assignee: Andrew Wang


We need to examine support for Har with HDFS encryption.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-2082) randomwriter should complain if there are too many arguments

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-2082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-2082.
--

Resolution: Fixed

way old and likely fixed by now.

> randomwriter should complain if there are too many arguments
> 
>
> Key: HADOOP-2082
> URL: https://issues.apache.org/jira/browse/HADOOP-2082
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Owen O'Malley
>
> A user was moving from 0.13 to 0.14 and was invoking randomwriter with a 
> config on the command line like:
> bin/hadoop jar hadoop-*-examples.jar randomwriter output conf.xml
> which worked in 0.13, but in 0.14 it ignores the conf.xml without 
> complaining. The equivalent is 
> bin/hadoop jar hadoop-*-examples.jar randomwriter -conf conf.xml output 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-1754) A testimonial page for hadoop?

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-1754.
--

Resolution: Fixed

We have entire conferences now.

Closing.

> A testimonial page for hadoop?
> --
>
> Key: HADOOP-1754
> URL: https://issues.apache.org/jira/browse/HADOOP-1754
> Project: Hadoop Common
>  Issue Type: Wish
>  Components: documentation
>Reporter: Konstantin Shvachko
>Priority: Minor
>
> Should we create a testimonial page on hadoop wiki with a link from Hadoop 
> home page so that people 
> could share their experience of using Hadoop?
> I see some satisfied users out there. :)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-1688) TestCrcCorruption hangs on windows

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-1688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-1688.
--

Resolution: Fixed

Closing this as stale.

> TestCrcCorruption hangs on windows
> --
>
> Key: HADOOP-1688
> URL: https://issues.apache.org/jira/browse/HADOOP-1688
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.14.0
> Environment: Windows
>Reporter: Konstantin Shvachko
>
> TestCrcCorruption times out on windows saying just that it timed out.
> No other useful information in the log.
> Some kind of timing issue, because if I run it with output=yes then it 
> succeeds.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-1496) Test coverage target in build files using emma

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-1496.
--

Resolution: Won't Fix

I'm going to close this now with won't fix given the clover coverage.

> Test coverage target in build files using emma
> --
>
> Key: HADOOP-1496
> URL: https://issues.apache.org/jira/browse/HADOOP-1496
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
> Environment: all
>Reporter: woyg
>Priority: Minor
> Attachments: emma.tgz, hadoop_clover.patch, patch.emma.txt, 
> patch.emma.txt.2
>
>
> Test coverage targets for Hadoop using emma. 
> Test coverage will help in identifying the components which are not poperly 
> covered in tests and write test cases for it.
> Emma (http://emma.sourceforge.net/) is a good tool for coverage.
> If you have something else in mind u can suggest.
> I have a patch ready with emma.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-1464) IPC server should not log thread stacks at the info level

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-1464.
--

Resolution: Fixed

I'm going to close this out as stale.  I suspect this is no longer an issue.

> IPC server should not log thread stacks at the info level
> -
>
> Key: HADOOP-1464
> URL: https://issues.apache.org/jira/browse/HADOOP-1464
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: ipc
>Affects Versions: 0.12.3
>Reporter: Hairong Kuang
>
> Currently when IPC server get a call which becomes too old, i.e. the call has 
> not been served for too long time, it dumps all thread stacks to logs at the 
> info level. Because the size of all thread stacks size might be very big, it 
> would be better to log them at the debug level.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-1024) Add stable version line to the website front page

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-1024.
--

Resolution: Fixed

This was done forever ago.

> Add stable version line to the website front page
> -
>
> Key: HADOOP-1024
> URL: https://issues.apache.org/jira/browse/HADOOP-1024
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Owen O'Malley
>
> I think it would be worthwhile to add two lines to the top of the welcome 
> website page:
> Stable version: 0.10.1
> Latest version: 0.11.1 
> With the number linking off to the respective release like so: 
> http://www.apache.org/dyn/closer.cgi/lucene/hadoop/hadoop-0.10.1.tar.gz
> We can promote versions from Latest to Stable when they have proven 
> themselves.
> Thoughts?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HADOOP-10855) Allow Text to be read with a known length

[Todd Lipcon (JIRA)]

Todd Lipcon created HADOOP-10855:


 Summary: Allow Text to be read with a known length
 Key: HADOOP-10855
 URL: https://issues.apache.org/jira/browse/HADOOP-10855
 Project: Hadoop Common
  Issue Type: Improvement
  Components: io
Affects Versions: 2.6.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Minor


For the native task work (MAPREDUCE-2841) it is useful to be able to store 
strings in a different fashion than the default (varint-prefixed) 
serialization. We should provide a "read" method in Text which takes an 
already-known length to support this use case while still providing Text 
objects back to the user.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HADOOP-10854) unit tests for the shell scripts

[Allen Wittenauer (JIRA)]

Allen Wittenauer created HADOOP-10854:
-

 Summary: unit tests for the shell scripts
 Key: HADOOP-10854
 URL: https://issues.apache.org/jira/browse/HADOOP-10854
 Project: Hadoop Common
  Issue Type: Test
Reporter: Allen Wittenauer


With HADOOP-9902 moving a lot of functionality to functions, we should build 
some unit tests for them.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-8025) change default distcp log location to be /tmp rather than cwd

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-8025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-8025.
--

Resolution: Won't Fix

> change default distcp log location to be /tmp rather than cwd
> -
>
> Key: HADOOP-8025
> URL: https://issues.apache.org/jira/browse/HADOOP-8025
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Allen Wittenauer
>Priority: Trivial
> Attachments: HADOOP-8025-branch-1.0.txt
>
>
> distcp loves to leave emtpy files around.  this puts them in /tmp so at least 
> they are easy to find and kill.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-8026) various shell script fixes

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-8026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-8026.
--

Resolution: Duplicate

This is all part of HADOOP-9902 now.

> various shell script fixes
> --
>
> Key: HADOOP-8026
> URL: https://issues.apache.org/jira/browse/HADOOP-8026
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Allen Wittenauer
> Attachments: HADOOP-8026-branch-1.0.txt
>
>
> Various shell script fixes:
> * repair naked $0s so that dir detections work
> * remove superfluous JAVA_HOME settings
> * use /usr/bin/pdsh in slaves.sh if it exists



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HADOOP-8100) share web server information for http filters

[Allen Wittenauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-8100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-8100.
--

Resolution: Won't Fix

> share web server information for http filters
> -
>
> Key: HADOOP-8100
> URL: https://issues.apache.org/jira/browse/HADOOP-8100
> Project: Hadoop Common
>  Issue Type: New Feature
>Affects Versions: 1.0.0, 0.23.2, 0.24.0
>Reporter: Allen Wittenauer
> Attachments: HADOOP-8100-branch-1.0.patch
>
>
> This is a simple fix which shares the web server bind information for 
> consumption down stream for 3rd party plugins.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Need some help with Hadoop QA report

[Nitin Pawar]

Answers in red:

1) It states "-1 core tests <...>".  Does it mean one core test failed
after the proposed patch?  If yes, where can I find some info about what
exactly failed?
If I follow "Test results" link below, I see "111 failures (+80) , 129
skipped (+23)".  Does it mean "total 111 tests failed which is 80 more
because of the patch"?
Where can I see errors only for those failed tests which failed because of
this patch?
If I expand (press "+" at Failed test column), I see errors which do not
look relevant to that patch:
Timed out waiting for Mini HDFS Cluster to start
java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/
v2/jobhistory/JobHistoryUtils
etc...

Ans: "-1" stands for the vote given by the test framework. That means the
patch submitted does provide a clean build as it has failed test cases.
What unit test cases failed due to the patch are mentioned in the jira
ticket.


2) Hadoop QA reports that "-1 findbugs. The patch appears to cause Findbugs
(version 2.0.3) to fail." but there is no other information.  How can I
figure out what's wrong?

Ans: Due to the patch, findbugs plugin in the build if failing. You can run
the same on your machine as well to see if it's causing any new issues with
and without applying your patch.

In other words, I am asking for help about how to understand from that
report what's wrong with that patch.


On Thu, Jul 17, 2014 at 5:11 PM, Dmitry Sivachenko 
wrote:

> Hello!
>
> I am trying to figure out what's wrong with Hadoop QA report in
> https://issues.apache.org/jira/browse/HADOOP-10783
>
> 1) It states "-1 core tests <...>".  Does it mean one core test failed
> after the proposed patch?  If yes, where can I find some info about what
> exactly failed?
> If I follow "Test results" link below, I see "111 failures (+80) , 129
> skipped (+23)".  Does it mean "total 111 tests failed which is 80 more
> because of the patch"?
> Where can I see errors only for those failed tests which failed because of
> this patch?
> If I expand (press "+" at Failed test column), I see errors which do not
> look relevant to that patch:
> Timed out waiting for Mini HDFS Cluster to start
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils
> etc...
>
>
> 2) Hadoop QA reports that "-1 findbugs. The patch appears to cause
> Findbugs (version 2.0.3) to fail." but there is no other information.  How
> can I figure out what's wrong?
>
> In other words, I am asking for help about how to understand from that
> report what's wrong with that patch.
>
> Thanks in advance!
>
>
>


-- 
Nitin Pawar

Need some help with Hadoop QA report

[Dmitry Sivachenko]

Hello!

I am trying to figure out what's wrong with Hadoop QA report in 
https://issues.apache.org/jira/browse/HADOOP-10783

1) It states "-1 core tests <...>".  Does it mean one core test failed after 
the proposed patch?  If yes, where can I find some info about what exactly 
failed?
If I follow "Test results" link below, I see "111 failures (+80) , 129 skipped 
(+23)".  Does it mean "total 111 tests failed which is 80 more because of the 
patch"?
Where can I see errors only for those failed tests which failed because of this 
patch?
If I expand (press "+" at Failed test column), I see errors which do not look 
relevant to that patch:
Timed out waiting for Mini HDFS Cluster to start
java.lang.NoClassDefFoundError: 
org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils
etc...


2) Hadoop QA reports that "-1 findbugs. The patch appears to cause Findbugs 
(version 2.0.3) to fail." but there is no other information.  How can I figure 
out what's wrong?

In other words, I am asking for help about how to understand from that report 
what's wrong with that patch.

Thanks in advance!

Build failed in Jenkins: Hadoop-Common-0.23-Build #1013

[Apache Jenkins Server]

See 

--
[...truncated 8295 lines...]
Running org.apache.hadoop.fs.TestFileSystemTokens
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.492 sec
Running org.apache.hadoop.fs.TestFSMainOperationsLocalFileSystem
Tests run: 49, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.657 sec
Running org.apache.hadoop.fs.TestLocalFSFileContextSymlink
Tests run: 61, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 2.548 sec
Running org.apache.hadoop.fs.TestHarFileSystem
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.319 sec
Running org.apache.hadoop.fs.TestFcLocalFsUtil
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.579 sec
Running org.apache.hadoop.fs.TestLocalDirAllocator
Tests run: 30, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.063 sec
Running org.apache.hadoop.fs.TestLocalFileSystemPermission
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.537 sec
Running org.apache.hadoop.fs.TestFileSystemCaching
Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.883 sec
Running org.apache.hadoop.fs.TestLocalFSFileContextCreateMkdir
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.784 sec
Running org.apache.hadoop.fs.TestPath
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.793 sec
Running org.apache.hadoop.fs.TestListFiles
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.544 sec
Running org.apache.hadoop.fs.TestHarFileSystemBasics
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.262 sec
Running org.apache.hadoop.fs.TestChecksumFileSystem
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.836 sec
Running org.apache.hadoop.fs.TestGetFileBlockLocations
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.776 sec
Running org.apache.hadoop.fs.TestFsShellCopy
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.338 sec
Running org.apache.hadoop.fs.TestDU
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.164 sec
Running org.apache.hadoop.fs.TestAvroFSInput
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.554 sec
Running org.apache.hadoop.fs.shell.TestPathExceptions
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.089 sec
Running org.apache.hadoop.fs.shell.TestCommandFactory
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.178 sec
Running org.apache.hadoop.fs.shell.TestPathData
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.765 sec
Running org.apache.hadoop.fs.shell.TestCopy
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.729 sec
Running org.apache.hadoop.fs.TestHardLink
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.263 sec
Running org.apache.hadoop.fs.TestFilterFileSystem
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.582 sec
Running org.apache.hadoop.fs.TestLocalFSFileContextMainOperations
Tests run: 56, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.223 sec
Running org.apache.hadoop.fs.TestTrash
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.083 sec
Running org.apache.hadoop.fs.viewfs.TestChRootedFileSystem
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.115 sec
Running org.apache.hadoop.fs.viewfs.TestViewFileSystemDelegation
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.454 sec
Running org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs
Tests run: 54, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.652 sec
Running org.apache.hadoop.fs.viewfs.TestFcCreateMkdirLocalFs
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.049 sec
Running 
org.apache.hadoop.fs.viewfs.TestViewFileSystemWithAuthorityLocalFileSystem
Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.195 sec
Running org.apache.hadoop.fs.viewfs.TestChRootedFs
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.081 sec
Running org.apache.hadoop.fs.viewfs.TestFSMainOperationsLocalFileSystem
Tests run: 49, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.308 sec
Running org.apache.hadoop.fs.viewfs.TestViewFsTrash
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.939 sec
Running org.apache.hadoop.fs.viewfs.TestViewfsFileStatus
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.761 sec
Running org.apache.hadoop.fs.viewfs.TestViewFsLocalFs
Tests run: 42, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.774 sec
Running org.apache.hadoop.fs.viewfs.TestViewFsWithAuthorityLocalFs
Tests run: 42, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.881 sec
Running org.apache.hadoop.fs.viewfs.TestViewFsConfig
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.397 sec
Running org.apache.hadoop.fs.viewfs.TestFcPermissionsLocalFs
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0,

[jira] [Created] (HADOOP-10853) Refactor create instance of CryptoCodec and add CryptoCodecFactory

[Yi Liu (JIRA)]

Yi Liu created HADOOP-10853:
---

 Summary: Refactor create instance of CryptoCodec and add 
CryptoCodecFactory
 Key: HADOOP-10853
 URL: https://issues.apache.org/jira/browse/HADOOP-10853
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: security
Reporter: Yi Liu
Assignee: Yi Liu


We should be able to create instance of *CryptoCodec*:
* via codec class name. (Applications may have config for different crypto 
codecs)
* via algorithm/mode/padding. (For automatically decryption, we need to find 
correct crypto codec and proper implementation)
* a default crypto codec through specific config. 

This JIRA is for
* Create instance through cipher suite(algorithm/mode/padding)
* Refactor create instance of {{CryptoCodec}} into {{CryptoCodecFactory}}

We need to get all crypto codecs in system, this can be done via a Java 
ServiceLoader + hadoop.security.crypto.codecs config.




--
This message was sent by Atlassian JIRA
(v6.2#6252)