[jira] [Resolved] (HADOOP-3494) Improve S3FileSystem data integrity using MD5 checksums
[ https://issues.apache.org/jira/browse/HADOOP-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved HADOOP-3494. --- Resolution: Invalid Improve S3FileSystem data integrity using MD5 checksums --- Key: HADOOP-3494 URL: https://issues.apache.org/jira/browse/HADOOP-3494 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Reporter: Tom White Make use of S3 MD5 checksums to verify writes and reads. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-3495) Support legacy S3 buckets containing underscores
[ https://issues.apache.org/jira/browse/HADOOP-3495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved HADOOP-3495. --- Resolution: Won't Fix Resolving this since as Ken points out we should let S3 report the problem. Support legacy S3 buckets containing underscores Key: HADOOP-3495 URL: https://issues.apache.org/jira/browse/HADOOP-3495 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Reporter: Tom White Priority: Minor For bucket names containing an underscore we fail with an exception, however it should be possible to support them. See proposal in https://issues.apache.org/jira/browse/HADOOP-930?focusedCommentId=12601991#action_12601991 by Chris K Wensel. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-3518) Want NIO.2 (JSR 203) file system provider for Hadoop FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14066396#comment-14066396 ] Tom White commented on HADOOP-3518: --- I think it is, and having [java.nio.file|http://docs.oracle.com/javase/7/docs/api/java/nio/file/package-summary.html] as an interface to Hadoop filesystems would be generally useful. Want NIO.2 (JSR 203) file system provider for Hadoop FileSystem --- Key: HADOOP-3518 URL: https://issues.apache.org/jira/browse/HADOOP-3518 Project: Hadoop Common Issue Type: Wish Components: fs Reporter: Tom White JSR 203 (aka NIO.2 or more NIO) is defining a rich set of classes for interacting with files and file systems (as well as other NIO enhancements). It is scheduled to be released as a part of Java 7. This motivation behind this issue is to see if NIO.2 can be used as an interface to Hadoop's FileSystem class before NIO.2 is finalized, thus giving Hadoop developers an opportunity to influence NIO's design (if necessary). Also, learning more about NIO.2 may inform design decisions for Hadoop filesystems. The starting point for this work should be the java.nio.file.spi package (http://openjdk.java.net/projects/nio/javadoc/java/nio/file/spi/package-summary.html). There is an example of a filesystem provider (for ZIP files) linked from the OpenJDK page for NIO.2: http://openjdk.java.net/projects/nio/. This page also has other useful links, such as a JavaOne talk, javadoc and source code. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9623) Update jets3t dependency
[ https://issues.apache.org/jira/browse/HADOOP-9623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682321#comment-13682321 ] Tom White commented on HADOOP-9623: --- Overall it looks fine. Do Jets3tNativeS3FileSystemContractTest and Jets3tS3FileSystemContractTest pass with this change? They are live tests run against the real S3 service and are not run by default. You'll need to put S3 credentials into hadoop-common-project/hadoop-common/src/test/resources/core-site.xml to run them. Nit: catch blocks should be on the same line as the preceding brace. Update jets3t dependency Key: HADOOP-9623 URL: https://issues.apache.org/jira/browse/HADOOP-9623 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 3.0.0, 2.1.0-beta Reporter: Timothy St. Clair Labels: maven Attachments: HADOOP-9623.patch Current version referenced in pom is 0.6.1 (Aug 2008), updating to 0.9.0 enables mvn-rpmbuild to build against system dependencies. http://jets3t.s3.amazonaws.com/RELEASE_NOTES.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9220) Unnecessary transition to standby in ActiveStandbyElector
[
https://issues.apache.org/jira/browse/HADOOP-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656937#comment-13656937
]
Tom White commented on HADOOP-9220:
---
+1 for Todd's patch. I tested it manually and the original problem no longer
occurs with the patch.
Nit: did you mean to introduce {{new Exception}} in the debug line?
Unnecessary transition to standby in ActiveStandbyElector
-
Key: HADOOP-9220
URL: https://issues.apache.org/jira/browse/HADOOP-9220
Project: Hadoop Common
Issue Type: Bug
Components: ha
Reporter: Tom White
Assignee: Tom White
Priority: Critical
Attachments: HADOOP-9220.patch, HADOOP-9220.patch, hadoop-9220.txt
When performing a manual failover from one HA node to a second, under some
circumstances the second node will transition from standby - active -
standby - active. This is with automatic failover enabled, so there is a ZK
cluster doing leader election.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9424) The hadoop jar invocation should include the passed jar on the classpath as a whole
[
https://issues.apache.org/jira/browse/HADOOP-9424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13655973#comment-13655973
]
Tom White commented on HADOOP-9424:
---
This change would basically load X.jar in the system classloader. RunJar also
loads any classes and resources in a classes directory in X.jar, and any
embedded JARs in a lib directory, but these would not be added to the system
classpath with this fix. So the same scenario that you describe could occur for
Bar in X.jar and Foo in an embedded JAR.
I'm also wary of making an environmental/classpath change like this without
running tests with Hive, Pig etc. Perhaps Bigtop testing could help here.
What's the scenario that you are seeing this in? Is the workaround not
sufficient?
The hadoop jar invocation should include the passed jar on the classpath as
a whole
-
Key: HADOOP-9424
URL: https://issues.apache.org/jira/browse/HADOOP-9424
Project: Hadoop Common
Issue Type: Bug
Components: util
Affects Versions: 2.0.3-alpha
Reporter: Harsh J
Assignee: Harsh J
Priority: Minor
Attachments: HADOOP-9424.patch
When you have a case such as this:
{{X.jar - Classes = Main, Foo}}
{{Y.jar - Classes = Bar}}
With implementation details such as:
* Main references Bar and invokes a public, static method on it.
* Bar does a class lookup to find Foo (Class.forName(Foo)).
Then when you do a {{HADOOP_CLASSPATH=Y.jar hadoop jar X.jar Main}}, the
Bar's method fails with a ClassNotFound exception cause of the way RunJar
runs.
RunJar extracts the passed jar and includes its contents on the ClassLoader
of its current thread but the {{Class.forName(…)}} call from another class
does not check that class loader and hence cannot find the class as its not
on any classpath it is aware of.
The script of hadoop jar should ideally include the passed jar argument to
the CLASSPATH before RunJar is invoked, for this above case to pass.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9459) ActiveStandbyElector can join election even before Service HEALTHY, and results in null data at ActiveBreadCrumb
[
https://issues.apache.org/jira/browse/HADOOP-9459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656306#comment-13656306
]
Tom White commented on HADOOP-9459:
---
+1 looks good to me.
ActiveStandbyElector can join election even before Service HEALTHY, and
results in null data at ActiveBreadCrumb
Key: HADOOP-9459
URL: https://issues.apache.org/jira/browse/HADOOP-9459
Project: Hadoop Common
Issue Type: Bug
Components: ha
Affects Versions: 2.0.2-alpha
Reporter: Vinay
Assignee: Vinay
Priority: Critical
Attachments: HDFS-4463.patch, hdfs-4463.txt
ActiveStandbyElector can store null at ActiveBreadCrumb in the below race
condition. At further all failovers will fail resulting NPE.
1. ZKFC restarted.
2. due to machine busy, first zk connection is expired even before the health
monitoring returned the status.
3. On re-establishment transitionToActive will be called, at this time
appData will be null,
4. So now ActiveBreadCrumb will have null.
5. After this any failovers will fail throwing
{noformat}java.lang.NullPointerException
at
org.apache.hadoop.util.StringUtils.byteToHexString(StringUtils.java:171)
at
org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:892)
at
org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:797)
at
org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:475)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:545)
at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497){noformat}
Should not join the election before service is HEALTHY
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6103) Configuration clone constructor does not clone all the members.
[ https://issues.apache.org/jira/browse/HADOOP-6103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13641607#comment-13641607 ] Tom White commented on HADOOP-6103: --- Thanks for the patch, Amit. Can you port the unit test to the branch-1 patch too please? Configuration clone constructor does not clone all the members. --- Key: HADOOP-6103 URL: https://issues.apache.org/jira/browse/HADOOP-6103 Project: Hadoop Common Issue Type: Bug Components: conf Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Fix For: 0.21.0 Attachments: HADOOP-6103-branch-1.patch, patch-6103.txt Currently, Configuration(Configuration other) constructor does not clone all the members. It clones only resources, properties, overlay and finalParameters. It needs to clone loadDefaults, classLoader, defaultResources, quietmode. This resulted in bugs like HADOOP-4975 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9258) Add stricter tests to FileSystemContractTestBase
[ https://issues.apache.org/jira/browse/HADOOP-9258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13607902#comment-13607902 ] Tom White commented on HADOOP-9258: --- The live tests now pass for me. +1 Add stricter tests to FileSystemContractTestBase Key: HADOOP-9258 URL: https://issues.apache.org/jira/browse/HADOOP-9258 Project: Hadoop Common Issue Type: Sub-task Components: test Affects Versions: 1.1.1, 2.0.3-alpha Reporter: Steve Loughran Assignee: Steve Loughran Attachments: HADOOP-9258-8.patch, HADOOP-9528-2.patch, HADOOP-9528-3.patch, HADOOP-9528-4.patch, HADOOP-9528-5.patch, HADOOP-9528-6.patch, HADOOP-9528-7.patch, HADOOP-9528.patch The File System Contract contains implicit assumptions that aren't checked in the contract test base. Add more tests to define the contract's assumptions more rigorously for those filesystems that are tested by this (not Local, BTW) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9301) hadoop client servlet/jsp/jetty/tomcat JARs creating conflicts in Oozie HttpFS
[
https://issues.apache.org/jira/browse/HADOOP-9301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13603300#comment-13603300
]
Tom White commented on HADOOP-9301:
---
+1
hadoop client servlet/jsp/jetty/tomcat JARs creating conflicts in Oozie
HttpFS
Key: HADOOP-9301
URL: https://issues.apache.org/jira/browse/HADOOP-9301
Project: Hadoop Common
Issue Type: Bug
Components: build
Affects Versions: 2.0.3-alpha
Reporter: Roman Shaposhnik
Assignee: Alejandro Abdelnur
Priority: Blocker
Fix For: 2.0.4-alpha
Attachments: HADOOP-9301.patch
Here's how to reproduce:
{noformat}
$ cd hadoop-client
$ mvn dependency:tree | egrep 'jsp|jetty'
[INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26.cloudera.2:compile
[INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26.cloudera.2:compile
[INFO] | +- javax.servlet.jsp:jsp-api:jar:2.1:compile
{noformat}
This has a potential for completely screwing up clients like Oozie, etc –
hence a blocker.
It seems that while common excludes those JARs, they are sneaking in via
hdfs, we need to exclude them too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9258) Add stricter tests to FileSystemContractTestBase
[
https://issues.apache.org/jira/browse/HADOOP-9258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13602210#comment-13602210
]
Tom White commented on HADOOP-9258:
---
I tried running Jets3tS3FileSystemContractTest by setting the following
properties in src/test/resources/core-site.xml:
{noformat}
property
nametest.fs.s3.name/name
values3://mytestbucket/value
descriptionThe name of the s3 file system for testing./description
/property
property
namefs.s3.awsAccessKeyId/name
valuexxx/value
/property
property
namefs.s3.awsSecretAccessKey/name
valuexxx/value
/property
{noformat}
However, testLSRootDir is consistently hanging with the following stacktrace.
Steve, did you manage to run against S3 yet?
{noformat}
main prio=5 tid=10500 nid=0x100601000 runnable [1005fd000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at
com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
- locked 7bb659288 (a java.lang.Object)
at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
at
com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
- locked 7bb659338 (a com.sun.net.ssl.internal.ssl.AppInputStream)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
- locked 7bb66ad28 (a java.io.BufferedInputStream)
at
org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
at
org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
at
org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)
at
org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
at
org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
at
org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at
org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:357)
at
org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestGet(RestS3Service.java:686)
at
org.jets3t.service.impl.rest.httpclient.RestS3Service.listObjectsInternal(RestS3Service.java:1083)
at
org.jets3t.service.impl.rest.httpclient.RestS3Service.listObjectsImpl(RestS3Service.java:1046)
at org.jets3t.service.S3Service.listObjects(S3Service.java:1299)
at org.jets3t.service.S3Service.listObjects(S3Service.java:1271)
at org.jets3t.service.S3Service.listObjects(S3Service.java:1137)
at
org.apache.hadoop.fs.s3.Jets3tFileSystemStore.listSubPaths(Jets3tFileSystemStore.java:279)
at
org.apache.hadoop.fs.s3.S3FileSystem.listStatus(S3FileSystem.java:202)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1430)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1470)
at org.apache.hadoop.fs.FileSystem$4.init(FileSystem.java:1745)
at
org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1744)
at
org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1727)
at
org.apache.hadoop.fs.FileSystem$5.handleFileStat(FileSystem.java:1820)
at org.apache.hadoop.fs.FileSystem$5.hasNext(FileSystem.java:1797)
at
org.apache.hadoop.fs.FileSystemContractBaseTest.assertListFilesFinds(FileSystemContractBaseTest.java:719)
at
org.apache.hadoop.fs.FileSystemContractBaseTest.testLSRootDir(FileSystemContractBaseTest.java:704)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at
[jira] [Updated] (HADOOP-9154) SortedMapWritable#putAll() doesn't add key/value classes to the map
[
https://issues.apache.org/jira/browse/HADOOP-9154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom White updated HADOOP-9154:
--
Resolution: Fixed
Fix Version/s: 2.0.4-beta
1.2.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Karthik!
SortedMapWritable#putAll() doesn't add key/value classes to the map
---
Key: HADOOP-9154
URL: https://issues.apache.org/jira/browse/HADOOP-9154
Project: Hadoop Common
Issue Type: Bug
Components: io
Affects Versions: 1.1.1, 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Fix For: 1.2.0, 2.0.4-beta
Attachments: HADOOP-9124.patch, hadoop-9154-branch1.patch,
hadoop-9154-draft.patch, hadoop-9154-draft.patch, hadoop-9154.patch,
hadoop-9154.patch, hadoop-9154.patch, hadoop-9154.patch, hadoop-9154.patch
In the following code from {{SortedMapWritable}}, #putAll() doesn't add
key/value classes to the class-id maps.
{code}
@Override
public Writable put(WritableComparable key, Writable value) {
addToMap(key.getClass());
addToMap(value.getClass());
return instance.put(key, value);
}
@Override
public void putAll(Map? extends WritableComparable, ? extends Writable t){
for (Map.Entry? extends WritableComparable, ? extends Writable e:
t.entrySet()) {
instance.put(e.getKey(), e.getValue());
}
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9304) remove addition of avro genreated-sources dirs to build
[ https://issues.apache.org/jira/browse/HADOOP-9304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13578564#comment-13578564 ] Tom White commented on HADOOP-9304: --- +1 I tested with Eclipse. remove addition of avro genreated-sources dirs to build --- Key: HADOOP-9304 URL: https://issues.apache.org/jira/browse/HADOOP-9304 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 2.0.4-beta Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-9304.patch The avro maven plugin automatically adds those dirs to the source dirs of the module. this is just a POM clean up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9117) replace protoc ant plugin exec with a maven plugin
[ https://issues.apache.org/jira/browse/HADOOP-9117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13577937#comment-13577937 ] Tom White commented on HADOOP-9117: --- I tried the patch with Eclipse and the generated files were picked up correctly. +1 replace protoc ant plugin exec with a maven plugin -- Key: HADOOP-9117 URL: https://issues.apache.org/jira/browse/HADOOP-9117 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 2.0.2-alpha Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-9117.patch, HADOOP-9117.patch, HADOOP-9117.patch, HADOOP-9117.patch, HADOOP-9117.patch, HADOOP-9117.patch The protoc compiler is currently invoked using ant plugin exec. There is a bug in the ant plugin exec task which does not consume the STDOUT or STDERR appropriately making the build to stop sometimes (you need to press enter to continue). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9297) remove old record IO generation and tests
[
https://issues.apache.org/jira/browse/HADOOP-9297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13577943#comment-13577943
]
Tom White commented on HADOOP-9297:
---
+1 pending jenkins
remove old record IO generation and tests
-
Key: HADOOP-9297
URL: https://issues.apache.org/jira/browse/HADOOP-9297
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.0.4-beta
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Fix For: 2.0.4-beta
Attachments: HADOOP-9297.patch
Remove their processing from the common POM and delete the following files:
{code}
hadoop-common-project/hadoop-common/src/test/ddl/buffer.jr
hadoop-common-project/hadoop-common/src/test/ddl/int.jr
hadoop-common-project/hadoop-common/src/test/ddl/string.jr
hadoop-common-project/hadoop-common/src/test/ddl/test.jr
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/FromCpp.java
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/RecordBench.java
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/TestBuffer.java
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/TestRecordIO.java
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/TestRecordVersioning.java
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/ToCpp.java
hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/typedbytes/TestIO.java
{code}
All these code is used exclusively within the files being removed. It does
not affect any component in a live cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9124) SortedMapWritable violates contract of Map interface for equals() and hashCode()
[ https://issues.apache.org/jira/browse/HADOOP-9124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved HADOOP-9124. --- Resolution: Fixed Fix Version/s: 1.2.0 Committed to branch 1. SortedMapWritable violates contract of Map interface for equals() and hashCode() Key: HADOOP-9124 URL: https://issues.apache.org/jira/browse/HADOOP-9124 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.1.1, 2.0.2-alpha Reporter: Patrick Hunt Assignee: Surenkumar Nihalani Priority: Minor Fix For: 1.2.0, 2.0.3-alpha, 0.23.7 Attachments: hadoop-9124-branch1.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch This issue is similar to HADOOP-7153. It was found when using MRUnit - see MRUNIT-158, specifically https://issues.apache.org/jira/browse/MRUNIT-158?focusedCommentId=13501985page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13501985 -- o.a.h.io.SortedMapWritable implements the java.util.Map interface, however it does not define an implementation of the equals() or hashCode() methods; instead the default implementations in java.lang.Object are used. This violates the contract of the Map interface which defines different behaviour for equals() and hashCode() than Object does. More information here: http://download.oracle.com/javase/6/docs/api/java/util/Map.html#equals(java.lang.Object) The practical consequence is that SortedMapWritables containing equal entries cannot be compared properly. We were bitten by this when trying to write an MRUnit test for a Mapper that outputs MapWritables; the MRUnit driver cannot test the equality of the expected and actual MapWritable objects. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9230) TestUniformSizeInputFormat fails intermittently
[
https://issues.apache.org/jira/browse/HADOOP-9230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573423#comment-13573423
]
Tom White commented on HADOOP-9230:
---
I guess that the legacy split check was included as a sanity check. Since the
split calculation is slightly different, the occasional difference should not
be too surprising. Nor should it be a problem for correctness, since the aim of
the calculation is to produce roughly equal-sized splits for distcp. Therefore
I think we can remove the legacy check in order to ensure that the test always
passes. +1 for the patch.
bq. I would expect the math discrepancy to lead to more than 10% failure rate
though.
Although the difference between floor and ceiling will almost always produce
different target split sizes differing by 1 (whenever the number of splits
doesn't exactly divide the total size), only very rarely will this affect
whether a file is or is not included in any particular split. This is why the
failures are relatively rare.
E.g. on one failure I got the file sizes were 1482, 2012, 1860, ... (making a
total size of 31439) and for 9 maps the legacy target size works out at 3493,
while the new target size is 3494. The first legacy split only included 1482
(since 1482 + 2012 3493) while the first split for the new code includes both
1482 and 2012.
TestUniformSizeInputFormat fails intermittently
---
Key: HADOOP-9230
URL: https://issues.apache.org/jira/browse/HADOOP-9230
Project: Hadoop Common
Issue Type: Bug
Components: test
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Labels: distcp
Attachments: hadoop-9230.patch
TestUniformSizeFileInputFormat fails intermittently. I ran the test 50 times
and noticed 5 failures.
Haven't noticed any particular pattern to which runs fail.
A sample stack trace is as follows:
{noformat}
java.lang.AssertionError: expected:1944 but was:1820
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.failNotEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:126)
at org.junit.Assert.assertEquals(Assert.java:470)
at org.junit.Assert.assertEquals(Assert.java:454)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.checkAgainstLegacy(TestUniformSizeInputFormat.java:244)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.testGetSplits(TestUniformSizeInputFormat.java:126)
at
org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.testGetSplits(TestUniformSizeInputFormat.java:252)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8528) Add pure java snappy as an option
[ https://issues.apache.org/jira/browse/HADOOP-8528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572329#comment-13572329 ] Tom White commented on HADOOP-8528: --- Is the snappy-java JAR avail in maven repos? It is, see http://mvnrepository.com/artifact/org.iq80.snappy/snappy Add pure java snappy as an option - Key: HADOOP-8528 URL: https://issues.apache.org/jira/browse/HADOOP-8528 Project: Hadoop Common Issue Type: Wish Reporter: Dave Revell Priority: Minor Using native libs for snappy has some disadvantages: - Requires libhadoop and libsnappy to be installed on servers, test machines, and any server running a minicluster with a SNAPPY CF - Requires configuration of java.library.path - Complicates the POM A pure java implementation of snappy is available, ASL2 licensed: https://github.com/dain/snappy . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9124) SortedMapWritable violates contract of Map interface for equals() and hashCode()
[ https://issues.apache.org/jira/browse/HADOOP-9124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-9124: -- Resolution: Fixed Fix Version/s: 2.0.3-alpha Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just committed this. Thanks Suren for the contribution, and Karthik for the reviews. SortedMapWritable violates contract of Map interface for equals() and hashCode() Key: HADOOP-9124 URL: https://issues.apache.org/jira/browse/HADOOP-9124 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 2.0.2-alpha Reporter: Patrick Hunt Assignee: Surenkumar Nihalani Priority: Minor Fix For: 2.0.3-alpha Attachments: HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch This issue is similar to HADOOP-7153. It was found when using MRUnit - see MRUNIT-158, specifically https://issues.apache.org/jira/browse/MRUNIT-158?focusedCommentId=13501985page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13501985 -- o.a.h.io.SortedMapWritable implements the java.util.Map interface, however it does not define an implementation of the equals() or hashCode() methods; instead the default implementations in java.lang.Object are used. This violates the contract of the Map interface which defines different behaviour for equals() and hashCode() than Object does. More information here: http://download.oracle.com/javase/6/docs/api/java/util/Map.html#equals(java.lang.Object) The practical consequence is that SortedMapWritables containing equal entries cannot be compared properly. We were bitten by this when trying to write an MRUnit test for a Mapper that outputs MapWritables; the MRUnit driver cannot test the equality of the expected and actual MapWritable objects. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9265) S3 blockstore filesystem breaks part of the Filesystem contract
[
https://issues.apache.org/jira/browse/HADOOP-9265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom White updated HADOOP-9265:
--
Attachment: HADOOP-9265.patch
Thanks for the feedback. Here's an updated version which creates a root inode
on S3 in case it doesn't exist.
S3 blockstore filesystem breaks part of the Filesystem contract
---
Key: HADOOP-9265
URL: https://issues.apache.org/jira/browse/HADOOP-9265
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 1.1.1, 3.0.0
Reporter: Steve Loughran
Assignee: Tom White
Attachments: HADOOP-9265.patch, HADOOP-9265.patch
The extended tests of HADOOP-9258 show that s3 is failing things which we
always expected an FS to do
# {{getFileStatus(/)}} to return a {{FileStatus}} -currently it returns a
{{FileNotFoundException}}.
# {{rename(somedir,somedir/childdir)}} to fail. currently it returns true
after deleting all the data in {{somedir/}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9265) S3 blockstore filesystem breaks part of the Filesystem contract
[
https://issues.apache.org/jira/browse/HADOOP-9265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom White updated HADOOP-9265:
--
Attachment: HADOOP-9265.patch
Here's a fix for both these problems. With the patch and HADOOP-9258 all the
tests in Jets3tS3FileSystemContractTest and TestInMemoryS3FileSystemContract
pass.
S3 blockstore filesystem breaks part of the Filesystem contract
---
Key: HADOOP-9265
URL: https://issues.apache.org/jira/browse/HADOOP-9265
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 1.1.1, 3.0.0
Reporter: Steve Loughran
Attachments: HADOOP-9265.patch
The extended tests of HADOOP-9258 show that s3 is failing things which we
always expected an FS to do
# {{getFileStatus(/)}} to return a {{FileStatus}} -currently it returns a
{{FileNotFoundException}}.
# {{rename(somedir,somedir/childdir)}} to fail. currently it returns true
after deleting all the data in {{somedir/}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-9265) S3 blockstore filesystem breaks part of the Filesystem contract
[
https://issues.apache.org/jira/browse/HADOOP-9265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom White reassigned HADOOP-9265:
-
Assignee: Tom White
S3 blockstore filesystem breaks part of the Filesystem contract
---
Key: HADOOP-9265
URL: https://issues.apache.org/jira/browse/HADOOP-9265
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 1.1.1, 3.0.0
Reporter: Steve Loughran
Assignee: Tom White
Attachments: HADOOP-9265.patch
The extended tests of HADOOP-9258 show that s3 is failing things which we
always expected an FS to do
# {{getFileStatus(/)}} to return a {{FileStatus}} -currently it returns a
{{FileNotFoundException}}.
# {{rename(somedir,somedir/childdir)}} to fail. currently it returns true
after deleting all the data in {{somedir/}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9261) S3n filesystem can move a directory under itself -and so lose data
[
https://issues.apache.org/jira/browse/HADOOP-9261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13567638#comment-13567638
]
Tom White commented on HADOOP-9261:
---
+1 looks good to me.
S3n filesystem can move a directory under itself -and so lose data
--
Key: HADOOP-9261
URL: https://issues.apache.org/jira/browse/HADOOP-9261
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 1.1.1, 2.0.2-alpha
Environment: Testing against S3 bucket stored on US West (Read after
Write consistency; eventual for read-after-delete or write-after-write)
Reporter: Steve Loughran
Attachments: HADOOP-9261.patch
The S3N filesystem {{rename()}} doesn't make sure that the destination
directory is not a child or other descendant of the source directory. The
files are copied to the new destination, then the source directory is
recursively deleted, so losing data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9258) Add stricter tests to FileSystemContractTestBase
[ https://issues.apache.org/jira/browse/HADOOP-9258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13567644#comment-13567644 ] Tom White commented on HADOOP-9258: --- +1 once the S3 fixes have been committed. Add stricter tests to FileSystemContractTestBase Key: HADOOP-9258 URL: https://issues.apache.org/jira/browse/HADOOP-9258 Project: Hadoop Common Issue Type: Improvement Components: test Affects Versions: 1.1.1, 2.0.3-alpha Reporter: Steve Loughran Assignee: Steve Loughran Attachments: HADOOP-9528-2.patch, HADOOP-9528-3.patch, HADOOP-9528.patch The File System Contract contains implicit assumptions that aren't checked in the contract test base. Add more tests to define the contract's assumptions more rigorously for those filesystems that are tested by this (not Local, BTW) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8545) Filesystem Implementation for OpenStack Swift
[ https://issues.apache.org/jira/browse/HADOOP-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13567682#comment-13567682 ] Tom White commented on HADOOP-8545: --- I wonder whether this would be better hosted outside Hadoop Common, since it's an optional feature, and won't get tested against Swift during routine Hadoop development. We recommended this course of action for KFS when it became QFS recently: http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201210.mbox/%3ccc946699.1c97b%25thi...@quantcast.com%3E It's a non-trivial amount of code: it adds over 35 Java classes in 9 packages to core. Does it add any new dependencies? I tried compiling and it complained about not being able to find org.apache.http.HttpHeaders. It looks like the native implementation supports large files now. If so, then the block store implementation is redundant and can be removed. Filesystem Implementation for OpenStack Swift - Key: HADOOP-8545 URL: https://issues.apache.org/jira/browse/HADOOP-8545 Project: Hadoop Common Issue Type: New Feature Components: fs Affects Versions: 2.0.3-alpha, 1.1.2 Reporter: Tim Miller Assignee: Dmitry Mezhensky Attachments: HADOOP-8545-1.patch, HADOOP-8545-2.patch, HADOOP-8545-3.patch, HADOOP-8545-4.patch, HADOOP-8545-5.patch, HADOOP-8545-6.patch, HADOOP-8545-javaclouds-2.patch, HADOOP-8545.patch, HADOOP-8545.patch Add a filesystem implementation for OpenStack Swift object store, similar to the one which exists today for S3. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9124) SortedMapWritable violates contract of Map interface for equals() and hashCode()
[ https://issues.apache.org/jira/browse/HADOOP-9124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-9124: -- Attachment: HADOOP-9124.patch Thanks Suren. I've attached a patch with a modification to SortedMapWritable's equals method so that it is similar to the one in MapWritable. Does this look OK to you? SortedMapWritable violates contract of Map interface for equals() and hashCode() Key: HADOOP-9124 URL: https://issues.apache.org/jira/browse/HADOOP-9124 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 2.0.2-alpha Reporter: Patrick Hunt Priority: Minor Attachments: HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch This issue is similar to HADOOP-7153. It was found when using MRUnit - see MRUNIT-158, specifically https://issues.apache.org/jira/browse/MRUNIT-158?focusedCommentId=13501985page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13501985 -- o.a.h.io.SortedMapWritable implements the java.util.Map interface, however it does not define an implementation of the equals() or hashCode() methods; instead the default implementations in java.lang.Object are used. This violates the contract of the Map interface which defines different behaviour for equals() and hashCode() than Object does. More information here: http://download.oracle.com/javase/6/docs/api/java/util/Map.html#equals(java.lang.Object) The practical consequence is that SortedMapWritables containing equal entries cannot be compared properly. We were bitten by this when trying to write an MRUnit test for a Mapper that outputs MapWritables; the MRUnit driver cannot test the equality of the expected and actual MapWritable objects. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-9124) SortedMapWritable violates contract of Map interface for equals() and hashCode()
[ https://issues.apache.org/jira/browse/HADOOP-9124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White reassigned HADOOP-9124: - Assignee: Surenkumar Nihalani SortedMapWritable violates contract of Map interface for equals() and hashCode() Key: HADOOP-9124 URL: https://issues.apache.org/jira/browse/HADOOP-9124 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 2.0.2-alpha Reporter: Patrick Hunt Assignee: Surenkumar Nihalani Priority: Minor Attachments: HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch This issue is similar to HADOOP-7153. It was found when using MRUnit - see MRUNIT-158, specifically https://issues.apache.org/jira/browse/MRUNIT-158?focusedCommentId=13501985page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13501985 -- o.a.h.io.SortedMapWritable implements the java.util.Map interface, however it does not define an implementation of the equals() or hashCode() methods; instead the default implementations in java.lang.Object are used. This violates the contract of the Map interface which defines different behaviour for equals() and hashCode() than Object does. More information here: http://download.oracle.com/javase/6/docs/api/java/util/Map.html#equals(java.lang.Object) The practical consequence is that SortedMapWritables containing equal entries cannot be compared properly. We were bitten by this when trying to write an MRUnit test for a Mapper that outputs MapWritables; the MRUnit driver cannot test the equality of the expected and actual MapWritable objects. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9124) SortedMapWritable violates contract of Map interface for equals() and hashCode()
[ https://issues.apache.org/jira/browse/HADOOP-9124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13565369#comment-13565369 ] Tom White commented on HADOOP-9124: --- What about my comment above about mirroring the change from HADOOP-7153? SortedMapWritable violates contract of Map interface for equals() and hashCode() Key: HADOOP-9124 URL: https://issues.apache.org/jira/browse/HADOOP-9124 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 2.0.2-alpha Reporter: Patrick Hunt Priority: Minor Attachments: HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch This issue is similar to HADOOP-7153. It was found when using MRUnit - see MRUNIT-158, specifically https://issues.apache.org/jira/browse/MRUNIT-158?focusedCommentId=13501985page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13501985 -- o.a.h.io.SortedMapWritable implements the java.util.Map interface, however it does not define an implementation of the equals() or hashCode() methods; instead the default implementations in java.lang.Object are used. This violates the contract of the Map interface which defines different behaviour for equals() and hashCode() than Object does. More information here: http://download.oracle.com/javase/6/docs/api/java/util/Map.html#equals(java.lang.Object) The practical consequence is that SortedMapWritables containing equal entries cannot be compared properly. We were bitten by this when trying to write an MRUnit test for a Mapper that outputs MapWritables; the MRUnit driver cannot test the equality of the expected and actual MapWritable objects. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9258) Add stricter tests to FileSystemContractTestBase
[ https://issues.apache.org/jira/browse/HADOOP-9258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13565394#comment-13565394 ] Tom White commented on HADOOP-9258: --- Steve, good to see more tests. The S3 failures sound like bugs to me, so it should be possible to fix them without breaking applications. Do you agree? Are you planning on including the local FS too? Add stricter tests to FileSystemContractTestBase Key: HADOOP-9258 URL: https://issues.apache.org/jira/browse/HADOOP-9258 Project: Hadoop Common Issue Type: Improvement Components: test Affects Versions: 1.1.1, 2.0.3-alpha Reporter: Steve Loughran Assignee: Steve Loughran Attachments: HADOOP-9528-2.patch, HADOOP-9528.patch The File System Contract contains implicit assumptions that aren't checked in the contract test base. Add more tests to define the contract's assumptions more rigorously for those filesystems that are tested by this (not Local, BTW) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9124) SortedMapWritable violates contract of Map interface for equals() and hashCode()
[
https://issues.apache.org/jira/browse/HADOOP-9124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13564177#comment-13564177
]
Tom White commented on HADOOP-9124:
---
{quote}
So, different *MapWritables have added Writables in different order won't lead
to same class-id mappings. Also, After a mapping is removed, there is no
reference counting to remove the class-id mapping that is no longer needed.
Hence, we should not check classToIdMap idToClassMap
{quote}
I agree. According to
[http://docs.oracle.com/javase/6/docs/api/java/util/Map.html#equals(java.lang.Object)]
bq. two maps {{m1}} and {{m2}} represent the same mappings if
{{m1.entrySet().equals(m2.entrySet())}}
So only the values of the entry set should be used for the basis for testing
equality.
SortedMapWritable violates contract of Map interface for equals() and
hashCode()
Key: HADOOP-9124
URL: https://issues.apache.org/jira/browse/HADOOP-9124
Project: Hadoop Common
Issue Type: Bug
Components: io
Affects Versions: 2.0.2-alpha
Reporter: Patrick Hunt
Priority: Minor
Attachments: HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch,
HADOOP-9124.patch, HADOOP-9124.patch, HADOOP-9124.patch
This issue is similar to HADOOP-7153. It was found when using MRUnit - see
MRUNIT-158, specifically
https://issues.apache.org/jira/browse/MRUNIT-158?focusedCommentId=13501985page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13501985
--
o.a.h.io.SortedMapWritable implements the java.util.Map interface, however it
does not define an implementation of the equals() or hashCode() methods;
instead the default implementations in java.lang.Object are used.
This violates the contract of the Map interface which defines different
behaviour for equals() and hashCode() than Object does. More information
here:
http://download.oracle.com/javase/6/docs/api/java/util/Map.html#equals(java.lang.Object)
The practical consequence is that SortedMapWritables containing equal entries
cannot be compared properly. We were bitten by this when trying to write an
MRUnit test for a Mapper that outputs MapWritables; the MRUnit driver cannot
test the equality of the expected and actual MapWritable objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9154) SortedMapWritable#putAll() doesn't add key/value classes to the map
[
https://issues.apache.org/jira/browse/HADOOP-9154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13560600#comment-13560600
]
Tom White commented on HADOOP-9154:
---
Sorry to come to this late, but I feel that we should just fix the bug and not
perform such extensive refactoring here. This is stable code and we need to be
sure that such a change doesn't break compatibility.
The following fix plus a unit test to show the problem should suffice, no?
{noformat}
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SortedMapWritable.java
b/hadoop-common-p
index eee744e..b533d94 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SortedMapWritable.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SortedMapWritable.java
@@ -141,7 +141,7 @@ public void putAll(Map? extends WritableComparable, ?
extends Writable t) {
for (Map.Entry? extends WritableComparable, ? extends Writable e:
t.entrySet()) {
- instance.put(e.getKey(), e.getValue());
+ put(e.getKey(), e.getValue());
}
}
{noformat}
SortedMapWritable#putAll() doesn't add key/value classes to the map
---
Key: HADOOP-9154
URL: https://issues.apache.org/jira/browse/HADOOP-9154
Project: Hadoop Common
Issue Type: Bug
Components: io
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: HADOOP-9124.patch, hadoop-9154-draft.patch,
hadoop-9154.patch, hadoop-9154.patch
In the following code from {{SortedMapWritable}}, #putAll() doesn't add
key/value classes to the class-id maps.
{code}
@Override
public Writable put(WritableComparable key, Writable value) {
addToMap(key.getClass());
addToMap(value.getClass());
return instance.put(key, value);
}
@Override
public void putAll(Map? extends WritableComparable, ? extends Writable t){
for (Map.Entry? extends WritableComparable, ? extends Writable e:
t.entrySet()) {
instance.put(e.getKey(), e.getValue());
}
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9234) Failing over to an active service with haadmin should fail when auto-failover is enabled
Tom White created HADOOP-9234:
-
Summary: Failing over to an active service with haadmin should
fail when auto-failover is enabled
Key: HADOOP-9234
URL: https://issues.apache.org/jira/browse/HADOOP-9234
Project: Hadoop Common
Issue Type: Bug
Components: ha
Affects Versions: 2.0.2-alpha
Reporter: Tom White
Using haadmin to fail over to an already active namenode fails with
{noformat}
Failover failed: Can't failover to an active service
{noformat}
if automatic failover is not enabled, but succeeds when automatic failover is
enabled.
We should make the automatic failover case have the same error message.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9220) Unnecessary transition to standby in ActiveStandbyElector
[ https://issues.apache.org/jira/browse/HADOOP-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13559135#comment-13559135 ] Tom White commented on HADOOP-9220: --- It's true that the elector checks for a stale ZK client, but that doesn't prevent the problem here which is caused by i) having multiple watchers for the ZK client (due to the creation of a new watcher in monitorLockNodeAsync), and ii) a postponed call to recheckElectability unnecessarily forcing a new election (this call doesn't go through the watcher). Unnecessary transition to standby in ActiveStandbyElector - Key: HADOOP-9220 URL: https://issues.apache.org/jira/browse/HADOOP-9220 Project: Hadoop Common Issue Type: Bug Components: ha Reporter: Tom White Assignee: Tom White Attachments: HADOOP-9220.patch, HADOOP-9220.patch When performing a manual failover from one HA node to a second, under some circumstances the second node will transition from standby - active - standby - active. This is with automatic failover enabled, so there is a ZK cluster doing leader election. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9220) Unnecessary transition to standby in ActiveStandbyElector
[ https://issues.apache.org/jira/browse/HADOOP-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-9220: -- Attachment: HADOOP-9220.patch I've written a test which fails without the patch. Basically it checks that the number of times that the HA service transitions to active is as expected. There is another part to the fix, in addition to the previous patch. In ZKFailoverController#recheckElectability() the check may be postponed if the FC has ceded its active state and is waiting for a timeout (10s) before rejoining the election. The trouble is that the FC may have become active again in the intervening time, but recheckElectability() doesn't take account of this (and will call ActiveStandbyElector#createLockNodeAsync), and so the FC will transition to standby and then to active again. The fix I have implemented changes a postponed recheckElectability() to check if the FC is not currently active before joining the election. Unnecessary transition to standby in ActiveStandbyElector - Key: HADOOP-9220 URL: https://issues.apache.org/jira/browse/HADOOP-9220 Project: Hadoop Common Issue Type: Bug Components: ha Reporter: Tom White Assignee: Tom White Attachments: HADOOP-9220.patch, HADOOP-9220.patch When performing a manual failover from one HA node to a second, under some circumstances the second node will transition from standby - active - standby - active. This is with automatic failover enabled, so there is a ZK cluster doing leader election. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9220) Unnecessary transition to standby in ActiveStandbyElector
[ https://issues.apache.org/jira/browse/HADOOP-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-9220: -- Status: Patch Available (was: Open) Unnecessary transition to standby in ActiveStandbyElector - Key: HADOOP-9220 URL: https://issues.apache.org/jira/browse/HADOOP-9220 Project: Hadoop Common Issue Type: Bug Components: ha Reporter: Tom White Assignee: Tom White Attachments: HADOOP-9220.patch, HADOOP-9220.patch When performing a manual failover from one HA node to a second, under some circumstances the second node will transition from standby - active - standby - active. This is with automatic failover enabled, so there is a ZK cluster doing leader election. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9220) Unnecessary transition to standby in ActiveStandbyElector
Tom White created HADOOP-9220: - Summary: Unnecessary transition to standby in ActiveStandbyElector Key: HADOOP-9220 URL: https://issues.apache.org/jira/browse/HADOOP-9220 Project: Hadoop Common Issue Type: Bug Components: ha Reporter: Tom White Assignee: Tom White When performing a manual failover from one HA node to a second, under some circumstances the second node will transition from standby - active - standby - active. This is with automatic failover enabled, so there is a ZK cluster doing leader election. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9220) Unnecessary transition to standby in ActiveStandbyElector
[ https://issues.apache.org/jira/browse/HADOOP-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-9220: -- Attachment: HADOOP-9220.patch The reason for this behaviour is because there can be multiple watchers registered for a given ZK client in ActiveStandbyElector. (The monitorLockNodeAsync() method creates a new watcher object for the existing ZK client.) This can cause multiple invocations of joinElectionInternal() for a single watch event, each of which will make a call to create the lock znode. The first call will cause the a transition to active, while subsequent ones will cause a transition to standby (in the isNodeExists clause of the processResult() method). In a manual failover scenario the node will still transition to active again, since the other node has ceded from the election for 10s, but it's still an unnecessary transition that could be eliminated. I did some manual testing with the attached patch, and the extra transition was avoided. I'll see if I can write a unit test for it. Unnecessary transition to standby in ActiveStandbyElector - Key: HADOOP-9220 URL: https://issues.apache.org/jira/browse/HADOOP-9220 Project: Hadoop Common Issue Type: Bug Components: ha Reporter: Tom White Assignee: Tom White Attachments: HADOOP-9220.patch When performing a manual failover from one HA node to a second, under some circumstances the second node will transition from standby - active - standby - active. This is with automatic failover enabled, so there is a ZK cluster doing leader election. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9212) Potential deadlock in FileSystem.Cache/IPC/UGI
[ https://issues.apache.org/jira/browse/HADOOP-9212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-9212: -- Resolution: Fixed Fix Version/s: 2.0.3-alpha Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just committed this. Potential deadlock in FileSystem.Cache/IPC/UGI -- Key: HADOOP-9212 URL: https://issues.apache.org/jira/browse/HADOOP-9212 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.0.2-alpha Reporter: Tom White Assignee: Tom White Fix For: 2.0.3-alpha Attachments: 1_jcarder_result_0.png, HADOOP-9212.patch, HADOOP-9212.patch jcarder found a cycle which could lead to a potential deadlock. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9212) Potential deadlock in FileSystem.Cache/IPC/UGI
Tom White created HADOOP-9212: - Summary: Potential deadlock in FileSystem.Cache/IPC/UGI Key: HADOOP-9212 URL: https://issues.apache.org/jira/browse/HADOOP-9212 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.0.2-alpha Reporter: Tom White Assignee: Tom White jcarder found a cycle which could lead to a potential deadlock. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9212) Potential deadlock in FileSystem.Cache/IPC/UGI
[ https://issues.apache.org/jira/browse/HADOOP-9212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-9212: -- Attachment: 1_jcarder_result_0.png The scenario that leads to the potential deadlock: * FS.Cache.closeAll(), which is holding the FS.Cache lock, calls DFS's close method which calls close on the RPC proxy, which eventually calls ipc.Client.stop() and takes lock on Hashtable of connections * ipc.Client.getConnection(), which is holding lock on Hashtable of connections, takes lock on UGI's class during some UGI setup that trigger's UGI.ensureInitialized() * UGI.getCurrentUser(), which is holding UGI class lock, calls getLoginUser() which calls Credentials.readTokenStorageFile, which uses FileSystem, taking a lock on FileSystem.Cache Potential deadlock in FileSystem.Cache/IPC/UGI -- Key: HADOOP-9212 URL: https://issues.apache.org/jira/browse/HADOOP-9212 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.0.2-alpha Reporter: Tom White Assignee: Tom White Attachments: 1_jcarder_result_0.png jcarder found a cycle which could lead to a potential deadlock. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9212) Potential deadlock in FileSystem.Cache/IPC/UGI
[ https://issues.apache.org/jira/browse/HADOOP-9212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-9212: -- Status: Patch Available (was: Open) Potential deadlock in FileSystem.Cache/IPC/UGI -- Key: HADOOP-9212 URL: https://issues.apache.org/jira/browse/HADOOP-9212 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.0.2-alpha Reporter: Tom White Assignee: Tom White Attachments: 1_jcarder_result_0.png, HADOOP-9212.patch jcarder found a cycle which could lead to a potential deadlock. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9212) Potential deadlock in FileSystem.Cache/IPC/UGI
[ https://issues.apache.org/jira/browse/HADOOP-9212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-9212: -- Attachment: HADOOP-9212.patch Here's a fix to break the cycle by using the Java File API in Credentials.readTokenStorageFile (overloading the method to take File) so the FileSystem.Cache lock need not be taken. Potential deadlock in FileSystem.Cache/IPC/UGI -- Key: HADOOP-9212 URL: https://issues.apache.org/jira/browse/HADOOP-9212 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.0.2-alpha Reporter: Tom White Assignee: Tom White Attachments: 1_jcarder_result_0.png, HADOOP-9212.patch jcarder found a cycle which could lead to a potential deadlock. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9213) Create a Jenkins job to run jcarder
Tom White created HADOOP-9213: - Summary: Create a Jenkins job to run jcarder Key: HADOOP-9213 URL: https://issues.apache.org/jira/browse/HADOOP-9213 Project: Hadoop Common Issue Type: Task Components: build Reporter: Tom White Assignee: Tom White It would be useful to have a nightly job to look for deadlocks in the Hadoop source code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9212) Potential deadlock in FileSystem.Cache/IPC/UGI
[ https://issues.apache.org/jira/browse/HADOOP-9212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-9212: -- Attachment: HADOOP-9212.patch Thanks for the review Todd. Here's a new patch with your feedback addressed. Potential deadlock in FileSystem.Cache/IPC/UGI -- Key: HADOOP-9212 URL: https://issues.apache.org/jira/browse/HADOOP-9212 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.0.2-alpha Reporter: Tom White Assignee: Tom White Attachments: 1_jcarder_result_0.png, HADOOP-9212.patch, HADOOP-9212.patch jcarder found a cycle which could lead to a potential deadlock. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9097) Maven RAT plugin is not checking all source files
[
https://issues.apache.org/jira/browse/HADOOP-9097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551220#comment-13551220
]
Tom White commented on HADOOP-9097:
---
Regarding the source files with third party licenses, by my reading of
http://apache.org/legal/resolved.html#required-third-party-notices and the
licenses in hadoop-hdfs-project/hadoop-hdfs/src/main/native/util/tree.h and
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c,
it is necessary to add the licenses to the Hadoop LICENSE.txt files. I see
that lz4.c's license is in the common LICENSE.txt file, but tree.h isn't in
HDFS's. So that should be fixed.
I notice that the following files are empty and can presumably be removed:
*
hadoop-hdfs-project/hadoop-hdfs/src/test/aop/org/apache/hadoop/hdfs/server/datanode/DataXceiverAspects.aj
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolProtocolBuffers/overview.html
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/MockApp.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/MockContainer.java
Though not strictly necessary, it would be nice to add .git/** and .idea/** to
the top-level excludes.
With these changes I managed to get a clean run of {{mvn apache-rat:check}}
with the combined patch.
+1 on the combined patch. Thanks a lot for doing this work Tom.
Maven RAT plugin is not checking all source files
-
Key: HADOOP-9097
URL: https://issues.apache.org/jira/browse/HADOOP-9097
Project: Hadoop Common
Issue Type: Bug
Components: build
Affects Versions: 2.0.3-alpha, 0.23.5
Reporter: Tom White
Assignee: Thomas Graves
Priority: Critical
Fix For: 2.0.3-alpha, 0.23.6
Attachments: HADOOP-9097-branch-0.23-entire.patch,
HADOOP-9097-branch-0.23.patch, HADOOP-9097-entire.patch, HADOOP-9097.patch,
HADOOP-9097-remove-branch23.sh, HADOOP-9097-remove-branch2.sh,
HADOOP-9097-remove-entire.sh
Running 'mvn apache-rat:check' passes, but running RAT by hand (by
downloading the JAR) produces some warnings for Java files, amongst others.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9097) Maven RAT plugin is not checking all source files
[ https://issues.apache.org/jira/browse/HADOOP-9097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551307#comment-13551307 ] Tom White commented on HADOOP-9097: --- Sorry I missed the remove script. That looks good to me. Maven RAT plugin is not checking all source files - Key: HADOOP-9097 URL: https://issues.apache.org/jira/browse/HADOOP-9097 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 2.0.3-alpha, 0.23.5 Reporter: Tom White Assignee: Thomas Graves Priority: Critical Fix For: 2.0.3-alpha, 0.23.6 Attachments: HADOOP-9097-branch-0.23-entire.patch, HADOOP-9097-branch-0.23.patch, HADOOP-9097-entire.patch, HADOOP-9097.patch, HADOOP-9097-remove-branch23.sh, HADOOP-9097-remove-branch2.sh, HADOOP-9097-remove-entire.sh Running 'mvn apache-rat:check' passes, but running RAT by hand (by downloading the JAR) produces some warnings for Java files, amongst others. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9183) Potential deadlock in ActiveStandbyElector
[ https://issues.apache.org/jira/browse/HADOOP-9183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-9183: -- Resolution: Fixed Fix Version/s: 2.0.3-alpha Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just committed this. Thanks for the reviews, Todd. Potential deadlock in ActiveStandbyElector -- Key: HADOOP-9183 URL: https://issues.apache.org/jira/browse/HADOOP-9183 Project: Hadoop Common Issue Type: Bug Components: ha Affects Versions: 2.0.2-alpha Reporter: Tom White Assignee: Tom White Fix For: 2.0.3-alpha Attachments: 2_jcarder_result_1.png, 3_jcarder_result_0.png, HADOOP-9183.patch, HADOOP-9183.patch, HADOOP-9183.patch A jcarder run found a potential deadlock in the locking of ActiveStandbyElector and ActiveStandbyElector.WatcherWithClientRef. No deadlock has been seen in practice, this is just a theoretical possibility at the moment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9183) Potential deadlock in ActiveStandbyElector
[ https://issues.apache.org/jira/browse/HADOOP-9183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-9183: -- Attachment: HADOOP-9183.patch Thanks Todd. Here's a new patch to avoid the race you mention. TestActiveStandbyElectorRealZK no longer fails with the artificial 1s sleep. Potential deadlock in ActiveStandbyElector -- Key: HADOOP-9183 URL: https://issues.apache.org/jira/browse/HADOOP-9183 Project: Hadoop Common Issue Type: Bug Components: ha Affects Versions: 2.0.2-alpha Reporter: Tom White Assignee: Tom White Attachments: 2_jcarder_result_1.png, 3_jcarder_result_0.png, HADOOP-9183.patch, HADOOP-9183.patch, HADOOP-9183.patch A jcarder run found a potential deadlock in the locking of ActiveStandbyElector and ActiveStandbyElector.WatcherWithClientRef. No deadlock has been seen in practice, this is just a theoretical possibility at the moment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9183) Potential deadlock in ActiveStandbyElector
[ https://issues.apache.org/jira/browse/HADOOP-9183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546896#comment-13546896 ] Tom White commented on HADOOP-9183: --- To test this removes the deadlock, I ran jcarder on TestZKFailoverController without the changes and it found the cycles attached. When I ran with the patch no cycles were found. Potential deadlock in ActiveStandbyElector -- Key: HADOOP-9183 URL: https://issues.apache.org/jira/browse/HADOOP-9183 Project: Hadoop Common Issue Type: Bug Components: ha Affects Versions: 2.0.2-alpha Reporter: Tom White Assignee: Tom White Attachments: 2_jcarder_result_1.png, 3_jcarder_result_0.png, HADOOP-9183.patch, HADOOP-9183.patch, HADOOP-9183.patch A jcarder run found a potential deadlock in the locking of ActiveStandbyElector and ActiveStandbyElector.WatcherWithClientRef. No deadlock has been seen in practice, this is just a theoretical possibility at the moment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9183) Potential deadlock in ActiveStandbyElector
Tom White created HADOOP-9183: - Summary: Potential deadlock in ActiveStandbyElector Key: HADOOP-9183 URL: https://issues.apache.org/jira/browse/HADOOP-9183 Project: Hadoop Common Issue Type: Bug Components: ha Affects Versions: 2.0.2-alpha Reporter: Tom White Assignee: Tom White A jcarder run found a potential deadlock in the locking of ActiveStandbyElector and ActiveStandbyElector.WatcherWithClientRef. No deadlock has been seen in practice, this is just a theoretical possibility at the moment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9183) Potential deadlock in ActiveStandbyElector
[ https://issues.apache.org/jira/browse/HADOOP-9183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-9183: -- Attachment: 3_jcarder_result_0.png 2_jcarder_result_1.png Potential deadlock in ActiveStandbyElector -- Key: HADOOP-9183 URL: https://issues.apache.org/jira/browse/HADOOP-9183 Project: Hadoop Common Issue Type: Bug Components: ha Affects Versions: 2.0.2-alpha Reporter: Tom White Assignee: Tom White Attachments: 2_jcarder_result_1.png, 3_jcarder_result_0.png A jcarder run found a potential deadlock in the locking of ActiveStandbyElector and ActiveStandbyElector.WatcherWithClientRef. No deadlock has been seen in practice, this is just a theoretical possibility at the moment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9183) Potential deadlock in ActiveStandbyElector
[ https://issues.apache.org/jira/browse/HADOOP-9183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-9183: -- Attachment: HADOOP-9183.patch This patch fixes the problem by making two changes. First, the queue of events in WatcherWithClientRef is dispensed with, and instead the process method blocks until the ZK object is set back on the watcher. This should be acceptable since the set operation is a simple method call, so there is minimal overhead. Second, the locking order ActiveStandbyElector - WatcherWithClientRef is enforced, to prevent cycles. Note also that the CountDownLatch can safely have its countDown() method called outside the synchronized section (which is to protect the ZK field). Indeed it must, since getNewZooKeeper is holding the ActiveStandbyElector object lock while it waits for the ZK connection event. This means that the event cannot be processed until the lock is released (this is the current behaviour today), but we need to signal that the connect event was received. Potential deadlock in ActiveStandbyElector -- Key: HADOOP-9183 URL: https://issues.apache.org/jira/browse/HADOOP-9183 Project: Hadoop Common Issue Type: Bug Components: ha Affects Versions: 2.0.2-alpha Reporter: Tom White Assignee: Tom White Attachments: 2_jcarder_result_1.png, 3_jcarder_result_0.png, HADOOP-9183.patch A jcarder run found a potential deadlock in the locking of ActiveStandbyElector and ActiveStandbyElector.WatcherWithClientRef. No deadlock has been seen in practice, this is just a theoretical possibility at the moment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9183) Potential deadlock in ActiveStandbyElector
[ https://issues.apache.org/jira/browse/HADOOP-9183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-9183: -- Status: Patch Available (was: Open) Potential deadlock in ActiveStandbyElector -- Key: HADOOP-9183 URL: https://issues.apache.org/jira/browse/HADOOP-9183 Project: Hadoop Common Issue Type: Bug Components: ha Affects Versions: 2.0.2-alpha Reporter: Tom White Assignee: Tom White Attachments: 2_jcarder_result_1.png, 3_jcarder_result_0.png, HADOOP-9183.patch A jcarder run found a potential deadlock in the locking of ActiveStandbyElector and ActiveStandbyElector.WatcherWithClientRef. No deadlock has been seen in practice, this is just a theoretical possibility at the moment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9183) Potential deadlock in ActiveStandbyElector
[ https://issues.apache.org/jira/browse/HADOOP-9183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-9183: -- Attachment: HADOOP-9183.patch This is a simpler patch which removes the locking on WatcherWithClientRef, since it suffices to use ActiveStandbyElector's lock. Potential deadlock in ActiveStandbyElector -- Key: HADOOP-9183 URL: https://issues.apache.org/jira/browse/HADOOP-9183 Project: Hadoop Common Issue Type: Bug Components: ha Affects Versions: 2.0.2-alpha Reporter: Tom White Assignee: Tom White Attachments: 2_jcarder_result_1.png, 3_jcarder_result_0.png, HADOOP-9183.patch, HADOOP-9183.patch A jcarder run found a potential deadlock in the locking of ActiveStandbyElector and ActiveStandbyElector.WatcherWithClientRef. No deadlock has been seen in practice, this is just a theoretical possibility at the moment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9153) Support createNonRecursive in ViewFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13535015#comment-13535015 ] Tom White commented on HADOOP-9153: --- Shouldn't FilterFileSystem and ChRootedFileSystem both call fs.createNonRecursive()? Support createNonRecursive in ViewFileSystem Key: HADOOP-9153 URL: https://issues.apache.org/jira/browse/HADOOP-9153 Project: Hadoop Common Issue Type: Improvement Components: viewfs Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: HADOOP-9153.patch Implement createNonRecursive in ViewFileSystem. Currently an Unsupported... exception is thrown when it's called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8545) Filesystem Implementation for OpenStack Swift
[ https://issues.apache.org/jira/browse/HADOOP-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13534024#comment-13534024 ] Tom White commented on HADOOP-8545: --- Dmitry, can you use the same large file format as Swift, even if some of it is handled on the Hadoop side? I think we should avoid creating a new file format. Filesystem Implementation for OpenStack Swift - Key: HADOOP-8545 URL: https://issues.apache.org/jira/browse/HADOOP-8545 Project: Hadoop Common Issue Type: New Feature Components: fs Affects Versions: 2.0.3-alpha, 1.1.2 Reporter: Tim Miller Priority: Minor Attachments: HADOOP-8545-1.patch, HADOOP-8545-javaclouds-2.patch, HADOOP-8545.patch, HADOOP-8545.patch Add a filesystem implementation for OpenStack Swift object store, similar to the one which exists today for S3. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9098) Add missing license headers
[ https://issues.apache.org/jira/browse/HADOOP-9098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13529802#comment-13529802 ] Tom White commented on HADOOP-9098: --- +1. Thanks, Arpit. Add missing license headers --- Key: HADOOP-9098 URL: https://issues.apache.org/jira/browse/HADOOP-9098 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 1.1.1 Reporter: Tom White Assignee: Arpit Agarwal Priority: Blocker Fix For: 1.2.0 Attachments: HADOOP-9098.patch, missing-headers.txt There are missing license headers in some source files (e.g. TestUnderReplicatedBlocks.java is one) according to the RAT report. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8545) Filesystem Implementation for OpenStack Swift
[ https://issues.apache.org/jira/browse/HADOOP-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13529812#comment-13529812 ] Tom White commented on HADOOP-8545: --- I'm not sure whether the blockstore implementation is the right approach. The main drawback is that it's a Hadoop-specific format, which other tools can't read, and which will introduce a support burden. The S3 blockstore was introduced when there was a limit on the size of files, which has since been removed, so if I was doing it today I would probably not have a S3 blockstore implementation. I would encourage you to look at more general support for large files in Swift, e.g. http://docs.openstack.org/developer/swift/overview_large_objects.html. Filesystem Implementation for OpenStack Swift - Key: HADOOP-8545 URL: https://issues.apache.org/jira/browse/HADOOP-8545 Project: Hadoop Common Issue Type: New Feature Components: fs Affects Versions: 2.0.3-alpha, 1.1.2 Reporter: Tim Miller Priority: Minor Attachments: HADOOP-8545-1.patch, HADOOP-8545-javaclouds-2.patch, HADOOP-8545.patch, HADOOP-8545.patch Add a filesystem implementation for OpenStack Swift object store, similar to the one which exists today for S3. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9097) Maven RAT plugin is not checking all source files
Tom White created HADOOP-9097: - Summary: Maven RAT plugin is not checking all source files Key: HADOOP-9097 URL: https://issues.apache.org/jira/browse/HADOOP-9097 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 2.0.3-alpha, 0.23.5 Reporter: Tom White Priority: Blocker Fix For: 2.0.3-alpha, 0.23.6 Running 'mvn apache-rat:check' passes, but running RAT by hand (by downloading the JAR) produces some warnings for Java files, amongst others. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9097) Maven RAT plugin is not checking all source files
[
https://issues.apache.org/jira/browse/HADOOP-9097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504572#comment-13504572
]
Tom White commented on HADOOP-9097:
---
It looks like the plugin is configured to only check pom.xml in some places:
{noformat}
plugin
groupIdorg.apache.rat/groupId
artifactIdapache-rat-plugin/artifactId
configuration
includes
includepom.xml/include
/includes
/configuration
/plugin
{noformat}
We should change this to include everything by default, and list exclusions in
cases where it is not possible to add a license header (e.g. binary files).
Service loader files (under META-INF), can have headers added since # is
recognized as a comment.
Maven RAT plugin is not checking all source files
-
Key: HADOOP-9097
URL: https://issues.apache.org/jira/browse/HADOOP-9097
Project: Hadoop Common
Issue Type: Bug
Components: build
Affects Versions: 2.0.3-alpha, 0.23.5
Reporter: Tom White
Priority: Blocker
Fix For: 2.0.3-alpha, 0.23.6
Running 'mvn apache-rat:check' passes, but running RAT by hand (by
downloading the JAR) produces some warnings for Java files, amongst others.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9098) Add missing license headers
Tom White created HADOOP-9098: - Summary: Add missing license headers Key: HADOOP-9098 URL: https://issues.apache.org/jira/browse/HADOOP-9098 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 1.1.1 Reporter: Tom White Priority: Blocker Fix For: 1.2.0 There are missing license headers in some source files (e.g. TestUnderReplicatedBlocks.java is one) according to the RAT report. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9076) Fix QA bot in YARN project
[ https://issues.apache.org/jira/browse/HADOOP-9076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13502849#comment-13502849 ] Tom White commented on HADOOP-9076: --- You're right, it hasn't picked up the change. I replied to the thread. Fix QA bot in YARN project -- Key: HADOOP-9076 URL: https://issues.apache.org/jira/browse/HADOOP-9076 Project: Hadoop Common Issue Type: Bug Components: build Reporter: Radim Kolar Priority: Trivial purpose of this ticket is to run automatic tests on patches to YARN project which have not working QA bot. There seems to be zero interest in fixing QA bot to make it work for YARN subproject. If you need to get your patch for YARN tested, just subscribe and attach it to this ticket. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9049) DelegationTokenRenewer needs to be Singleton and FileSystems should register/deregister to/from.
[ https://issues.apache.org/jira/browse/HADOOP-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-9049: -- Resolution: Fixed Fix Version/s: 2.0.3-alpha Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just committed this. Thanks for the patch, Karthik, and for the review, Daryn. DelegationTokenRenewer needs to be Singleton and FileSystems should register/deregister to/from. Key: HADOOP-9049 URL: https://issues.apache.org/jira/browse/HADOOP-9049 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.0.2-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Fix For: 2.0.3-alpha Attachments: hadoop-9049.patch, hadoop-9049.patch Currently, DelegationTokenRenewer is not singleton. Each filesystem using it spawns its own DelegationTokenRenewer. Also, they don't stop the Renewer leading to other problems. A single DelegationTokenRenewer should be sufficient for all FileSystems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9076) Fix QA bot in YARN project
[ https://issues.apache.org/jira/browse/HADOOP-9076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13502073#comment-13502073 ] Tom White commented on HADOOP-9076: --- Radim - Vinod and I have been trying to get the automated commit jobs working for YARN. Gavin from Apache Infra kindly fixed it yesterday: http://mail-archives.apache.org/mod_mbox/www-builds/201211.mbox/%3Ccb6401cdc764%24421a76a0%24c64f63e0%24%4016degrees.com.au%3E Fix QA bot in YARN project -- Key: HADOOP-9076 URL: https://issues.apache.org/jira/browse/HADOOP-9076 Project: Hadoop Common Issue Type: Task Components: build Reporter: Radim Kolar Priority: Trivial Attachments: pstree-update2.txt purpose of this ticket is to run automatic tests on patches to YARN project which have not working QA bot. There seems to be zero interest in fixing QA bot to make it work for YARN subproject. If you need to get your patch for YARN tested, just subscribe and attach it to this ticket. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8860) Split MapReduce and YARN sections in documentation navigation
[ https://issues.apache.org/jira/browse/HADOOP-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved HADOOP-8860. --- Resolution: Fixed Fix Version/s: 0.23.5 I just pulled this into 0.23 too after discussing with Bobby and Tom. Split MapReduce and YARN sections in documentation navigation - Key: HADOOP-8860 URL: https://issues.apache.org/jira/browse/HADOOP-8860 Project: Hadoop Common Issue Type: Task Components: documentation Affects Versions: 2.0.1-alpha Reporter: Tom White Assignee: Tom White Fix For: 2.0.3-alpha, 0.23.5 Attachments: HADOOP-8860-23.patch, HADOOP-8860-23.sh, HADOOP-8860.patch, HADOOP-8860.patch, HADOOP-8860.sh, HADOOP-8860.sh This JIRA is to change the navigation on http://hadoop.apache.org/docs/r2.0.1-alpha/ to reflect the fact that MapReduce and YARN are separate modules/sub-projects. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9041) FileSystem initialization can go into infinite loop
[ https://issues.apache.org/jira/browse/HADOOP-9041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13500240#comment-13500240 ] Tom White commented on HADOOP-9041: --- Can you convert the Groovy test into a JUnit test so we have a regression test? FileSystem initialization can go into infinite loop --- Key: HADOOP-9041 URL: https://issues.apache.org/jira/browse/HADOOP-9041 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.0.2-alpha Reporter: Radim Kolar Assignee: Yanbo Liang Priority: Critical Attachments: fstest.groovy, HADOOP-9041.patch, HADOOP-9041.patch More information is there: https://jira.springsource.org/browse/SHDP-111 Referenced source code from example is: https://github.com/SpringSource/spring-hadoop/blob/master/src/main/java/org/springframework/data/hadoop/configuration/ConfigurationFactoryBean.java from isolating that cause it looks like if you register: org.apache.hadoop.fs.FsUrlStreamHandlerFactory before calling FileSystem.loadFileSystems() then it goes into infinite loop. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8860) Split MapReduce and YARN sections in documentation navigation
[ https://issues.apache.org/jira/browse/HADOOP-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-8860: -- Target Version/s: 2.0.3-alpha (was: 2.0.2-alpha) Fix Version/s: 2.0.3-alpha Hadoop Flags: Reviewed I just committed this to branch-2 (using the original script). Split MapReduce and YARN sections in documentation navigation - Key: HADOOP-8860 URL: https://issues.apache.org/jira/browse/HADOOP-8860 Project: Hadoop Common Issue Type: Task Components: documentation Affects Versions: 2.0.1-alpha Reporter: Tom White Assignee: Tom White Fix For: 2.0.3-alpha Attachments: HADOOP-8860.patch, HADOOP-8860.patch, HADOOP-8860.sh, HADOOP-8860.sh This JIRA is to change the navigation on http://hadoop.apache.org/docs/r2.0.1-alpha/ to reflect the fact that MapReduce and YARN are separate modules/sub-projects. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8860) Split MapReduce and YARN sections in documentation navigation
[ https://issues.apache.org/jira/browse/HADOOP-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-8860: -- Attachment: HADOOP-8860-23.patch HADOOP-8860-23.sh The changes don't apply to the 0.23 branch either, so I created equivalents for that branch. Bobby, please let me know if you'd like me to commit them there too. Split MapReduce and YARN sections in documentation navigation - Key: HADOOP-8860 URL: https://issues.apache.org/jira/browse/HADOOP-8860 Project: Hadoop Common Issue Type: Task Components: documentation Affects Versions: 2.0.1-alpha Reporter: Tom White Assignee: Tom White Fix For: 2.0.3-alpha Attachments: HADOOP-8860-23.patch, HADOOP-8860-23.sh, HADOOP-8860.patch, HADOOP-8860.patch, HADOOP-8860.sh, HADOOP-8860.sh This JIRA is to change the navigation on http://hadoop.apache.org/docs/r2.0.1-alpha/ to reflect the fact that MapReduce and YARN are separate modules/sub-projects. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8860) Split MapReduce and YARN sections in documentation navigation
[ https://issues.apache.org/jira/browse/HADOOP-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-8860: -- Attachment: HADOOP-8860.patch HADOOP-8860.sh Sorry about that. Here's an updated version. Split MapReduce and YARN sections in documentation navigation - Key: HADOOP-8860 URL: https://issues.apache.org/jira/browse/HADOOP-8860 Project: Hadoop Common Issue Type: Task Components: documentation Affects Versions: 2.0.1-alpha Reporter: Tom White Assignee: Tom White Attachments: HADOOP-8860.patch, HADOOP-8860.patch, HADOOP-8860.sh, HADOOP-8860.sh This JIRA is to change the navigation on http://hadoop.apache.org/docs/r2.0.1-alpha/ to reflect the fact that MapReduce and YARN are separate modules/sub-projects. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7115) Add a cache for getpwuid_r and getpwgid_r calls
[ https://issues.apache.org/jira/browse/HADOOP-7115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494164#comment-13494164 ] Tom White commented on HADOOP-7115: --- +1 the latest patch looks good to me. Add a cache for getpwuid_r and getpwgid_r calls --- Key: HADOOP-7115 URL: https://issues.apache.org/jira/browse/HADOOP-7115 Project: Hadoop Common Issue Type: Bug Affects Versions: 0.22.0, 2.0.2-alpha, 0.23.4 Reporter: Arun C Murthy Assignee: Alejandro Abdelnur Fix For: 0.22.1, 2.0.3-alpha Attachments: h-7115.1.patch, hadoop-7115-0.22.patch, hadoop-7115-0.22.patch, HADOOP-7115.patch, HADOOP-7115.patch, HADOOP-7115.patch, HADOOP-7115.patch As discussed in HADOOP-6978, a cache helps a lot. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8427) Convert Forrest docs to APT
[
https://issues.apache.org/jira/browse/HADOOP-8427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13492311#comment-13492311
]
Tom White commented on HADOOP-8427:
---
Andy, this looks good so far. I generated the site with {{mvn site; mvn
site:stage -DstagingDirectory=/tmp/hadoop-site}} and the converted files looked
OK. The navigation is not wired up yet though - my patch in HADOOP-8860 sets up
the nav correctly so it would help if committed that one first. Does that patch
look OK to you?
Going through the other files in Common that have not been migrated,
cluster_setup.xml and single_node_setup.xml are already in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt so can be left,
but deployment_layout.xml, index.xml, native_libraries.xml,
service_level_auth.xml, and Superusers.xml should all be migrated. They will
need to be reviewed and updated, but this can be done separately as Nicholas
suggested.
Convert Forrest docs to APT
---
Key: HADOOP-8427
URL: https://issues.apache.org/jira/browse/HADOOP-8427
Project: Hadoop Common
Issue Type: Task
Components: documentation
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
Assignee: Andy Isaacson
Labels: newbie
Attachments: hadoop8427-1.txt, hadoop8427-3.txt, hadoop8427.txt
Some of the forrest docs content in src/docs/src/documentation/content/xdocs
has not yet been converted to APT and moved to src/site/apt. Let's convert
the forrest docs that haven't been converted yet to new APT content in
hadoop-common/src/site/apt (and link the new content into
hadoop-project/src/site/apt/index.apt.vm) and remove all forrest dependencies.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8964) test-patch.sh doesn't run all tests
[ https://issues.apache.org/jira/browse/HADOOP-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13484413#comment-13484413 ] Tom White commented on HADOOP-8964: --- Vinod - that seems reasonable to me. test-patch.sh doesn't run all tests --- Key: HADOOP-8964 URL: https://issues.apache.org/jira/browse/HADOOP-8964 Project: Hadoop Common Issue Type: Bug Components: build Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli test-patch.sh tries to be smart but with bad results. When it runs tests, say on each commit, it only runs tests only from modules which have corresponding changes. This is a cause of concern for downstream modules. We ran into multiple issues in the past because of this, the most recent of it being YARN-179. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8964) test-patch.sh doesn't run all tests
[
https://issues.apache.org/jira/browse/HADOOP-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13482412#comment-13482412
]
Tom White commented on HADOOP-8964:
---
There was some discussion on this on
http://mail-archives.aurora.apache.org/mod_mbox/hadoop-common-dev/201204.mbox/%3ccaf-wd4tvkwypuuq9ibxv4uz8b2behxnpfkb5mq3d-pwvksh...@mail.gmail.com%3E
and HADOOP-8308.
I like the direction that Bobby is suggesting. We could use Maven's distinction
between unit and integration tests ({{mvn test}} vs {{mvn integration-test}}).
test-patch.sh doesn't run all tests
---
Key: HADOOP-8964
URL: https://issues.apache.org/jira/browse/HADOOP-8964
Project: Hadoop Common
Issue Type: Bug
Components: build
Reporter: Vinod Kumar Vavilapalli
Assignee: Vinod Kumar Vavilapalli
test-patch.sh tries to be smart but with bad results. When it runs tests, say
on each commit, it only runs tests only from modules which have corresponding
changes. This is a cause of concern for downstream modules. We ran into
multiple issues in the past because of this, the most recent of it being
YARN-179.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8904) Hadoop does not close output file / does not call Mapper.cleanup if exception in map
[ https://issues.apache.org/jira/browse/HADOOP-8904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479073#comment-13479073 ] Tom White commented on HADOOP-8904: --- This change makes the behaviour the same as the old API where the Mapper's close() method is called in a finally block - which is a good thing. Will it cause incompatibilities with existing code - i.e. any that assume cleanup() won't be called if the map throws an exception? We should at least mark this as an incompatible change with a note saying that you need to override the Mapper's (or Reducer's) run() method to restore the old behaviour. Hadoop does not close output file / does not call Mapper.cleanup if exception in map - Key: HADOOP-8904 URL: https://issues.apache.org/jira/browse/HADOOP-8904 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Daniel Dai Assignee: Daniel Dai Attachments: HADOOP-23-2.patch, HADOOP-8904-1.patch Find this in Pig unit test TestStore under Windows. There are dangling files because map does not close the file when exception happens in map(). In Windows, Hadoop will not remove a file if it is not closed. This happens in reduce() as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8887) Use a Maven plugin to build the native code using CMake
[ https://issues.apache.org/jira/browse/HADOOP-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13473251#comment-13473251 ] Tom White commented on HADOOP-8887: --- bq. Does it make sense to put it in Hadoop for now, and then spin it off into another project later? If we do that we should at least put it in a org.apache.hadoop package since the Hadoop PMC doesn't control the org.apache.maven namespace. Use a Maven plugin to build the native code using CMake --- Key: HADOOP-8887 URL: https://issues.apache.org/jira/browse/HADOOP-8887 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HADOOP-8887.001.patch, HADOOP-8887.002.patch, HADOOP-8887.003.patch, HADOOP-8887.004.patch Currently, we build the native code using ant-build invocations. Although this works, it has some limitations: * compiler warning messages are hidden, which can cause people to check in code with warnings unintentionally * there is no framework for running native unit tests; instead, we use ad-hoc constructs involving shell scripts * the antrun code is very platform specific * there is no way to run a specific native unit test * it's more or less impossible for scripts like test-patch.sh to separate a native test failing from the build itself failing (no files are created) or to enumerate which native tests failed. Using a native Maven plugin would overcome these limitations. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8887) Use a Maven plugin to build the native code using CMake
[ https://issues.apache.org/jira/browse/HADOOP-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472272#comment-13472272 ] Tom White commented on HADOOP-8887: --- This plugin looks general purpose and it's in the org.apache.maven package, so perhaps it should go in the Maven Plugins project? Use a Maven plugin to build the native code using CMake --- Key: HADOOP-8887 URL: https://issues.apache.org/jira/browse/HADOOP-8887 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HADOOP-8887.001.patch, HADOOP-8887.002.patch, HADOOP-8887.003.patch, HADOOP-8887.004.patch Currently, we build the native code using ant-build invocations. Although this works, it has some limitations: * compiler warning messages are hidden, which can cause people to check in code with warnings unintentionally * there is no framework for running native unit tests; instead, we use ad-hoc constructs involving shell scripts * the antrun code is very platform specific * there is no way to run a specific native unit test * it's more or less impossible for scripts like test-patch.sh to separate a native test failing from the build itself failing (no files are created) or to enumerate which native tests failed. Using a native Maven plugin would overcome these limitations. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8895) TokenRenewer should be an interface, it is currently a fully abstract class
[ https://issues.apache.org/jira/browse/HADOOP-8895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472404#comment-13472404 ] Tom White commented on HADOOP-8895: --- Abstract classes are easier to evolve than interfaces, since you can add a method with a default implementation without breaking implementors. TokenRenewer is only for use internally so it doesn't matter so much, but I would still leave it as it is since it's always possible to create a new implementation class that extends the abstract class. TokenRenewer should be an interface, it is currently a fully abstract class --- Key: HADOOP-8895 URL: https://issues.apache.org/jira/browse/HADOOP-8895 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.1-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Minor Attachments: HADOOP-8895.patch TokenRenewer is a fully abstract class. Making it an interface will allow classes extending other classes to implement the interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8776) Provide an option in test-patch that can enable / disable compiling native code
[ https://issues.apache.org/jira/browse/HADOOP-8776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465511#comment-13465511 ] Tom White commented on HADOOP-8776: --- +1 I tried the modified test-patch on a Mac with the --disable-native flag and it worked fine. Provide an option in test-patch that can enable / disable compiling native code --- Key: HADOOP-8776 URL: https://issues.apache.org/jira/browse/HADOOP-8776 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 3.0.0 Reporter: Hemanth Yamijala Assignee: Hemanth Yamijala Priority: Minor Attachments: HADOOP-8776.patch, HADOOP-8776.patch, HADOOP-8776.patch The test-patch script in Hadoop source runs a native compile with the patch. On platforms like MAC, there are issues with the native compile that make it difficult to use test-patch. This JIRA is to try and provide an option to make the native compilation optional. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8860) Split MapReduce and YARN sections in documentation navigation
Tom White created HADOOP-8860: - Summary: Split MapReduce and YARN sections in documentation navigation Key: HADOOP-8860 URL: https://issues.apache.org/jira/browse/HADOOP-8860 Project: Hadoop Common Issue Type: Task Components: documentation Affects Versions: 2.0.1-alpha Reporter: Tom White Assignee: Tom White -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8860) Split MapReduce and YARN sections in documentation navigation
[
https://issues.apache.org/jira/browse/HADOOP-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom White updated HADOOP-8860:
--
Attachment: HADOOP-8860.patch
HADOOP-8860.sh
Here's a patch to move things around. It will make it easier to do HADOOP-8427,
HDFS-3458, and MAPREDUCE-4282 since there is now a home for the APT fies in
those projects.
I've attached a script to do the 'svn mv' operations. Jenkins won't be able to
test the patch because of this, so I ran test-patch manually. The javadoc
warnings are spurious since this change doesn't touch any Java files.
{color:red}-1 overall{color}.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
-6 warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
Split MapReduce and YARN sections in documentation navigation
-
Key: HADOOP-8860
URL: https://issues.apache.org/jira/browse/HADOOP-8860
Project: Hadoop Common
Issue Type: Task
Components: documentation
Affects Versions: 2.0.1-alpha
Reporter: Tom White
Assignee: Tom White
Attachments: HADOOP-8860.patch, HADOOP-8860.sh
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8860) Split MapReduce and YARN sections in documentation navigation
[ https://issues.apache.org/jira/browse/HADOOP-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-8860: -- Description: This JIRA is to change the navigation on http://hadoop.apache.org/docs/r2.0.1-alpha/ to reflect the fact that MapReduce and YARN are separate modules/sub-projects. Target Version/s: 2.0.2-alpha (was: 2.0.0-alpha) Release Note: (was: This JIRA is to change the navigation on http://hadoop.apache.org/docs/r2.0.1-alpha/ to reflect the fact that MapReduce and YARN are separate modules/sub-projects.) Split MapReduce and YARN sections in documentation navigation - Key: HADOOP-8860 URL: https://issues.apache.org/jira/browse/HADOOP-8860 Project: Hadoop Common Issue Type: Task Components: documentation Affects Versions: 2.0.1-alpha Reporter: Tom White Assignee: Tom White Attachments: HADOOP-8860.patch, HADOOP-8860.sh This JIRA is to change the navigation on http://hadoop.apache.org/docs/r2.0.1-alpha/ to reflect the fact that MapReduce and YARN are separate modules/sub-projects. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8852) DelegationTokenRenewer thread is not stopped when its filesystem is closed
[ https://issues.apache.org/jira/browse/HADOOP-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13464652#comment-13464652 ] Tom White commented on HADOOP-8852: --- Generally looks good. * Rather than always starting a DelegationTokenRenewer thread in the HftpFileSystem/WebHdfsFileSystem constructor (which is a bad practice, and is why findbugs is complaining), you could do what WebHdfsFileSystem does already and only start the thread if security is enabled. See the addRenewAction() method on WebHdfsFileSystem. * The 3000ms join timeout could be made into a constant. Perhaps you could add a shutdown method to DelegationTokenRenewer which encapsulates the interrupt and join. DelegationTokenRenewer thread is not stopped when its filesystem is closed -- Key: HADOOP-8852 URL: https://issues.apache.org/jira/browse/HADOOP-8852 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.0-alpha Reporter: Tom White Assignee: Karthik Kambatla Attachments: hadoop-8852.patch, hadoop-8852.patch HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer thread when they are closed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8852) DelegationTokenRenewer thread is not stopped when its filesystem is closed
Tom White created HADOOP-8852: - Summary: DelegationTokenRenewer thread is not stopped when its filesystem is closed Key: HADOOP-8852 URL: https://issues.apache.org/jira/browse/HADOOP-8852 Project: Hadoop Common Issue Type: Bug Reporter: Tom White HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer thread when they are closed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8833) fs -text should make sure to call inputstream.seek(0) before using input stream
[
https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom White updated HADOOP-8833:
--
Attachment: HADOOP-8833.patch
+1 on the fix. I noticed that the test doesn't fail without the fix though.
This is because BZip2Codec.BZip2CompressionInputStream.readStreamHeader()
tolerates a missing (two-byte) header, so BZip2 files happen to work anyway.
I've modified the test slightly to test a deflate-compressed file, and this one
does fail without the seek fix.
fs -text should make sure to call inputstream.seek(0) before using input
stream
---
Key: HADOOP-8833
URL: https://issues.apache.org/jira/browse/HADOOP-8833
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.0.2-alpha
Reporter: Harsh J
Assignee: Harsh J
Attachments: HADOOP-8833.patch, HADOOP-8833.patch
From Muddy Dixon on HADOOP-8449:
Hi
We found the changes in order of switch and guard block in
{code}
private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
{code}
Because of this change, return value of
{code}
codec.createInputStream(i)
{code}
is changed if codec exists.
{code}
private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
FSDataInputStream i = srcFs.open(p);
// check codecs
CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
CompressionCodec codec = cf.getCodec(p);
if (codec != null) {
return codec.createInputStream(i);
}
switch(i.readShort()) {
// cases
}
{code}
New:
{code}
private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
FSDataInputStream i = srcFs.open(p);
switch(i.readShort()) { // === index (or pointer) processes!!
// cases
default: {
// Check the type of compression instead, depending on Codec class's
// own detection methods, based on the provided path.
CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
CompressionCodec codec = cf.getCodec(p);
if (codec != null) {
return codec.createInputStream(i);
}
break;
}
}
// File is non-compressed, or not a file container we know.
i.seek(0);
return i;
}
{code}
Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8825) Reinstate constructors in SequenceFile.BlockCompressWriter and SequenceFile.RecordCompressWriter for compatibility with Hadoop 1
[ https://issues.apache.org/jira/browse/HADOOP-8825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-8825: -- Resolution: Invalid Target Version/s: (was: 2.0.3-alpha) Status: Resolved (was: Patch Available) I managed to change AvroSequenceFile so it doesn't need to use the BlockCompressWriter or RecordCompressWriter constructors. Thanks for the help Bobby! Reinstate constructors in SequenceFile.BlockCompressWriter and SequenceFile.RecordCompressWriter for compatibility with Hadoop 1 Key: HADOOP-8825 URL: https://issues.apache.org/jira/browse/HADOOP-8825 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 2.0.1-alpha Reporter: Tom White Assignee: Tom White Attachments: HADOOP-8825.patch Two constructors were removed in Hadoop 2 which causes a problem for Avro being able to support Hadoop 1 and Hadoop 2. See AVRO-1170. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8825) Reinstate constructors in SequenceFile.BlockCompressWriter and SequenceFile.RecordCompressWriter for compatibility with Hadoop 1
Tom White created HADOOP-8825: - Summary: Reinstate constructors in SequenceFile.BlockCompressWriter and SequenceFile.RecordCompressWriter for compatibility with Hadoop 1 Key: HADOOP-8825 URL: https://issues.apache.org/jira/browse/HADOOP-8825 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 2.0.1-alpha Reporter: Tom White Assignee: Tom White Two constructors were removed in Hadoop 2 which causes a problem for Avro being able to support Hadoop 1 and Hadoop 2. See AVRO-1170. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8825) Reinstate constructors in SequenceFile.BlockCompressWriter and SequenceFile.RecordCompressWriter for compatibility with Hadoop 1
[ https://issues.apache.org/jira/browse/HADOOP-8825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-8825: -- Attachment: HADOOP-8825.patch This patch reinstates the constructors. I tested this by running Avro tests using Hadoop JARs built with the patch applied. Reinstate constructors in SequenceFile.BlockCompressWriter and SequenceFile.RecordCompressWriter for compatibility with Hadoop 1 Key: HADOOP-8825 URL: https://issues.apache.org/jira/browse/HADOOP-8825 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 2.0.1-alpha Reporter: Tom White Assignee: Tom White Attachments: HADOOP-8825.patch Two constructors were removed in Hadoop 2 which causes a problem for Avro being able to support Hadoop 1 and Hadoop 2. See AVRO-1170. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8825) Reinstate constructors in SequenceFile.BlockCompressWriter and SequenceFile.RecordCompressWriter for compatibility with Hadoop 1
[ https://issues.apache.org/jira/browse/HADOOP-8825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-8825: -- Status: Patch Available (was: Open) Reinstate constructors in SequenceFile.BlockCompressWriter and SequenceFile.RecordCompressWriter for compatibility with Hadoop 1 Key: HADOOP-8825 URL: https://issues.apache.org/jira/browse/HADOOP-8825 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 2.0.1-alpha Reporter: Tom White Assignee: Tom White Attachments: HADOOP-8825.patch Two constructors were removed in Hadoop 2 which causes a problem for Avro being able to support Hadoop 1 and Hadoop 2. See AVRO-1170. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8825) Reinstate constructors in SequenceFile.BlockCompressWriter and SequenceFile.RecordCompressWriter for compatibility with Hadoop 1
[ https://issues.apache.org/jira/browse/HADOOP-8825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13458749#comment-13458749 ] Tom White commented on HADOOP-8825: --- Avro is using the constructors via a class called SequenceFileBase in org.apache.hadoop.io to provide access. However, I agree that this is a poor way to do things, so making them public would be a lot better. Reinstate constructors in SequenceFile.BlockCompressWriter and SequenceFile.RecordCompressWriter for compatibility with Hadoop 1 Key: HADOOP-8825 URL: https://issues.apache.org/jira/browse/HADOOP-8825 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 2.0.1-alpha Reporter: Tom White Assignee: Tom White Attachments: HADOOP-8825.patch Two constructors were removed in Hadoop 2 which causes a problem for Avro being able to support Hadoop 1 and Hadoop 2. See AVRO-1170. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8825) Reinstate constructors in SequenceFile.BlockCompressWriter and SequenceFile.RecordCompressWriter for compatibility with Hadoop 1
[ https://issues.apache.org/jira/browse/HADOOP-8825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13458774#comment-13458774 ] Tom White commented on HADOOP-8825: --- It's to support a wrapper around SequenceFile for Avro types: http://avro.apache.org/docs/1.7.1/api/java/org/apache/avro/hadoop/io/AvroSequenceFile.html. But it would be preferable to use the existing constructors or createWriter methods. I'll take a look to see if that's possible. Reinstate constructors in SequenceFile.BlockCompressWriter and SequenceFile.RecordCompressWriter for compatibility with Hadoop 1 Key: HADOOP-8825 URL: https://issues.apache.org/jira/browse/HADOOP-8825 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 2.0.1-alpha Reporter: Tom White Assignee: Tom White Attachments: HADOOP-8825.patch Two constructors were removed in Hadoop 2 which causes a problem for Avro being able to support Hadoop 1 and Hadoop 2. See AVRO-1170. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8780) Update DeprecatedProperties apt file
[ https://issues.apache.org/jira/browse/HADOOP-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-8780: -- Resolution: Fixed Fix Version/s: 2.0.3-alpha Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just committed this. Thanks Ahmed! Update DeprecatedProperties apt file Key: HADOOP-8780 URL: https://issues.apache.org/jira/browse/HADOOP-8780 Project: Hadoop Common Issue Type: Bug Reporter: Ahmed Radwan Assignee: Ahmed Radwan Fix For: 2.0.3-alpha Attachments: HADOOP-8780.patch, HADOOP-8780_rev2.patch, HADOOP-8780_rev3.patch The current list of deprecated properties is not up-to-date. I'll will upload a patch momentarily. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8780) Update DeprecatedProperties apt file
[ https://issues.apache.org/jira/browse/HADOOP-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13453935#comment-13453935 ] Tom White commented on HADOOP-8780: --- The Jenkins job is timing out when running the HDFS tests - https://builds.apache.org/job/PreCommit-HADOOP-Build/1436/console. The HDFS pre-commit jobs are passing fine though - e.g. https://builds.apache.org/job/PreCommit-HDFS-Build/3180/console. I'm not sure why. Ahmed, can you run test-patch and post the results here please? Update DeprecatedProperties apt file Key: HADOOP-8780 URL: https://issues.apache.org/jira/browse/HADOOP-8780 Project: Hadoop Common Issue Type: Bug Reporter: Ahmed Radwan Assignee: Ahmed Radwan Attachments: HADOOP-8780.patch, HADOOP-8780_rev2.patch The current list of deprecated properties is not up-to-date. I'll will upload a patch momentarily. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8780) Update DeprecatedProperties apt file
[ https://issues.apache.org/jira/browse/HADOOP-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13453019#comment-13453019 ] Tom White commented on HADOOP-8780: --- I generated the documentation and it looked fine. +1 pending Jenkins. Update DeprecatedProperties apt file Key: HADOOP-8780 URL: https://issues.apache.org/jira/browse/HADOOP-8780 Project: Hadoop Common Issue Type: Bug Reporter: Ahmed Radwan Assignee: Ahmed Radwan Attachments: HADOOP-8780.patch, HADOOP-8780_rev2.patch The current list of deprecated properties is not up-to-date. I'll will upload a patch momentarily. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8662) remove separate pages for Common, HDFS MR projects
[ https://issues.apache.org/jira/browse/HADOOP-8662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13451960#comment-13451960 ] Tom White commented on HADOOP-8662: --- I tried the latest patch and it worked. +1 remove separate pages for Common, HDFS MR projects Key: HADOOP-8662 URL: https://issues.apache.org/jira/browse/HADOOP-8662 Project: Hadoop Common Issue Type: Improvement Components: documentation Affects Versions: site Reporter: Doug Cutting Assignee: Doug Cutting Priority: Minor Fix For: site Attachments: HADOOP-8662.patch, HADOOP-8662.patch The tabs on the top of http://hadoop.apache.org/ link to separate sites for Common, HDFS and MapReduce modules. These sites are identical except for the mailing lists. I propose we move the mailing list information to the TLP mailing list page and remove these sub-project websites. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8780) Update DeprecatedProperties apt file
[ https://issues.apache.org/jira/browse/HADOOP-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13452011#comment-13452011 ] Tom White commented on HADOOP-8780: --- The entries for mapred.create.symlink and mapreduce.job.cache.symlink.create are dropped in the patch, which looks wrong. Otherwise, +1. Update DeprecatedProperties apt file Key: HADOOP-8780 URL: https://issues.apache.org/jira/browse/HADOOP-8780 Project: Hadoop Common Issue Type: Bug Reporter: Ahmed Radwan Assignee: Ahmed Radwan Attachments: HADOOP-8780.patch The current list of deprecated properties is not up-to-date. I'll will upload a patch momentarily. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8780) Update DeprecatedProperties apt file
[ https://issues.apache.org/jira/browse/HADOOP-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13452388#comment-13452388 ] Tom White commented on HADOOP-8780: --- How about adding these two properties to a separate table on the same page? Alternatively a comment in the release notes. Update DeprecatedProperties apt file Key: HADOOP-8780 URL: https://issues.apache.org/jira/browse/HADOOP-8780 Project: Hadoop Common Issue Type: Bug Reporter: Ahmed Radwan Assignee: Ahmed Radwan Attachments: HADOOP-8780.patch The current list of deprecated properties is not up-to-date. I'll will upload a patch momentarily. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8662) remove separate pages for Common, HDFS MR projects
[
https://issues.apache.org/jira/browse/HADOOP-8662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449774#comment-13449774
]
Tom White commented on HADOOP-8662:
---
Migrating the website to Maven may take a while (and can be done in another
JIRA), so I think we should go ahead with this patch. All the changes look fine
to me, although there was a problem when I tried building it due to missing
release notes and changes from old releases. E.g.
{noformat}
docs/r0.20.0/releasenotes.html BROKEN:
/Users/tom/workspace/hadoop-site/main/author/src/documentation/content/xdocs/docs/r0.20.0/releasenotes.xml
(No such file or directory)
{noformat}
Not sure why this is happening - something to do with moving from the common
directory?
remove separate pages for Common, HDFS MR projects
Key: HADOOP-8662
URL: https://issues.apache.org/jira/browse/HADOOP-8662
Project: Hadoop Common
Issue Type: Improvement
Components: documentation
Affects Versions: site
Reporter: Doug Cutting
Assignee: Doug Cutting
Priority: Minor
Fix For: site
Attachments: HADOOP-8662.patch
The tabs on the top of http://hadoop.apache.org/ link to separate sites for
Common, HDFS and MapReduce modules. These sites are identical except for the
mailing lists. I propose we move the mailing list information to the TLP
mailing list page and remove these sub-project websites.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8762) Mark container-provided dependencies with 'provided' scope
[ https://issues.apache.org/jira/browse/HADOOP-8762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-8762: -- Assignee: Tom White Status: Patch Available (was: Open) Mark container-provided dependencies with 'provided' scope -- Key: HADOOP-8762 URL: https://issues.apache.org/jira/browse/HADOOP-8762 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 2.0.0-alpha Reporter: Tom White Assignee: Tom White Attachments: HADOOP-8762.patch For example the Tomcat and Jetty dependencies should be 'provided' since they are provided by the container (i.e. the Hadoop installation). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8762) Mark container-provided dependencies with 'provided' scope
Tom White created HADOOP-8762: - Summary: Mark container-provided dependencies with 'provided' scope Key: HADOOP-8762 URL: https://issues.apache.org/jira/browse/HADOOP-8762 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 2.0.0-alpha Reporter: Tom White For example the Tomcat and Jetty dependencies should be 'provided' since they are provided by the container (i.e. the Hadoop installation). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8762) Mark container-provided dependencies with 'provided' scope
[ https://issues.apache.org/jira/browse/HADOOP-8762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-8762: -- Attachment: HADOOP-8762.patch This patch changes container dependencies to 'provided' for Common and HDFS. With this change I could remove the excludes from hadoop-client without bringing in any extra dependencies (in fact hadoop-client was incorrectly pulling in Jetty via HDFS). Mark container-provided dependencies with 'provided' scope -- Key: HADOOP-8762 URL: https://issues.apache.org/jira/browse/HADOOP-8762 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 2.0.0-alpha Reporter: Tom White Attachments: HADOOP-8762.patch For example the Tomcat and Jetty dependencies should be 'provided' since they are provided by the container (i.e. the Hadoop installation). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
