Re: [DISCUSS] Hadoop 3.3.2 release?

[Zhihai Xu]

+1

On Wed, Sep 8, 2021 at 8:37 AM Chris Nauroth  wrote:

> +1
>
> Chao, thank you very much for volunteering on the release.
>
> Chris Nauroth
>
>
> On Tue, Sep 7, 2021 at 10:00 PM Igor Dvorzhak 
> wrote:
>
> > +1
> >
> > On Tue, Sep 7, 2021 at 10:06 AM Chao Sun  wrote:
> >
> >> Hi all,
> >>
> >> It has been almost 3 months since the 3.3.1 release and branch-3.3 has
> >> accumulated quite a few commits (118 atm). In particular, Spark
> community
> >> recently found an issue which prevents one from using the shaded Hadoop
> >> client together with certain compression codecs such as lz4 and snappy
> >> codec. The details are recorded in HADOOP-17891 and SPARK-36669.
> >>
> >> Therefore, I'm wondering if anyone is also interested in a 3.3.2
> release.
> >> If there is no objection, I'd like to volunteer myself for the work as
> >> well.
> >>
> >> Best Regards,
> >> Chao
> >>
> >
>

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Zhihai Xu]

Thanks Junping for creating release 2.8.0 (RC3).

+1 (non-binding)

* Downloaded and built from source
* Verified md5 checksums and signature
* Deployed a pseudo cluster
* verified basic HDFS operations and Pi job.
* Did a sanity check for RM and NM UI.

Thanks

zhihai

On Tue, Mar 21, 2017 at 12:02 PM, Steve Loughran 
wrote:

>
> I'm an =0 I'm afraid
>
> I think those extra files need to be pulled which triggers a repackage, at
> which point there's a couple of
>
> HADOOP-14205 No FileSystem for scheme: adl
> HADOOP-14204 S3A multipart commit failing,
>
> of the two, the ADL FS one is the most significant, though if there was
> going to be a re-roll, I'd like them in.
>
> I also worry about windows test coverage, as irrespective of whether
> people are planning to run it on a windows host in production, the hadoop
> libs get used client side in job submit, and more now in Spark .
> HADOOP-14201 showed up that (a) my new VM is a mess, so complicating
> testing and (b) gzip was failing. I Worry about gzip, as that could be a
> sign of native lib problems.. someone needs to look at it a bit more.
> However, I don't think it should really be a blocker, given the ASF doesn't
> officially redistribute windows binaries.
>
> so: some issues with packaging and a couple of patches which need to go
> in. Test coverage on windows is potentially longer, and I don't want it to
> hold things up.
>
> Anyhow: =0. If everyone else is happy to ship, then let's get it out the
> door, as that's the only way that we'll get all the in-the-field bugreps
> coming back in. We will need to release some new versions in the coming
> weeks to handle those
>
> -Steve
>
>
>
>
> > On 21 Mar 2017, at 18:43, Haibo Chen  wrote:
> >
> > Thanks Junping for working on the new release!
> >
> > +1 non-binding
> >
> > 1) Downloaded the source, verified the checksum
> > 2) Built natively from source, and deployed it to a pseudo-distributed
> > cluster
> > 3) Ran sleep and teragen job and checked both YARN and JHS web UI
> > 4) Played with yarn + mapreduce command lines
> >
> > Best,
> > Haibo Chen
> >
> > On Mon, Mar 20, 2017 at 11:18 AM, Junping Du 
> wrote:
> >
> >> ?Thanks for update, John. Then we should be OK with fixing this issue in
> >> 2.8.1.
> >>
> >> Mark the target version of HADOOP-14205 to 2.8.1 instead of 2.8.0 and
> bump
> >> up to blocker in case we could miss this in releasing 2.8.1. :)
> >>
> >>
> >> Thanks,
> >>
> >>
> >> Junping
> >>
> >> 
> >> From: John Zhuge 
> >> Sent: Monday, March 20, 2017 10:31 AM
> >> To: Junping Du
> >> Cc: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> >> yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org
> >> Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
> >>
> >> Yes, it only affects ADL. There is a workaround of adding these 2
> >> properties to core-site.xml:
> >>
> >>  
> >>fs.adl.impl
> >>org.apache.hadoop.fs.adl.AdlFileSystem
> >>  
> >>
> >>  
> >>fs.AbstractFileSystem.adl.impl
> >>org.apache.hadoop.fs.adl.Adl
> >>  
> >>
> >> I have the initial patch ready but hitting these live unit test
> failures:
> >>
> >> Failed tests:
> >>  TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
> testListStatus:257
> >> expected:<1> but was:<10>
> >>
> >> Tests in error:
> >>  TestAdlFileContextMainOperationsLive>FileContextMainOperationsBaseT
> est.
> >> testMkdirsFailsForSubdirectoryOfExistingFile:254 » AccessControl
> >>  TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
> >> testMkdirsFailsForSubdirectoryOfExistingFile:190 » AccessControl
> >>
> >>
> >> Stay tuned...
> >>
> >> John Zhuge
> >> Software Engineer, Cloudera
> >>
> >> On Mon, Mar 20, 2017 at 10:02 AM, Junping Du   >> j...@hortonworks.com>> wrote:
> >>
> >> Thank you for reporting the issue, John! Does this issue only affect ADL
> >> (Azure Data Lake) which is a new feature for 2.8 rather than other
> existing
> >> FS? If so, I think we can leave the fix to 2.8.1 to fix given this is
> not a
> >> regression and just a new feature get broken.?
> >>
> >>
> >> Thanks,
> >>
> >>
> >> Junping
> >>
> >> 
> >> From: John Zhuge mailto:jzh...@cloudera.com>>
> >> Sent: Monday, March 20, 2017 9:07 AM
> >> To: Junping Du
> >> Cc: common-dev@hadoop.apache.org;
> >> hdfs-...@hadoop.apache.org;
> >> yarn-...@hadoop.apache.org;
> >> mapreduce-...@hadoop.apache.org
> >> Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
> >>
> >> Discovered https://issues.apache.org/jira/browse/HADOOP-14205 "No
> >> FileSystem for scheme: adl".
> >>
> >> The issue were caused by backporting HADOOP-13037 to branch-2 and
> earlier.
> >> HADOOP-12666 should not be backported, but some changes are needed:
> >> property fs.adl.impl in core-default.xml and hadoop-tools-dist/pom.xml.
> >>

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha2 RC0

[Zhihai Xu]

Thanks Andrew for creating release Hadoop 3.0.0-alpha2 RC0
+1 ( non-binding)

--Downloaded source and built from it.
--Deployed on a pseudo-distributed cluster.
--Ran sample MR jobs and tested with basics HDFS operations.
--Did a sanity check for RM and NM UI.

Best,
zhihai

On Wed, Jan 25, 2017 at 8:07 AM, Kuhu Shukla 
wrote:

> +1 (non-binding)* Built from source* Deployed on a pseudo-distributed
> cluster (MAC)* Ran wordcount and sleep jobs.
>
>
> On Wednesday, January 25, 2017 3:21 AM, Marton Elek <
> me...@hortonworks.com> wrote:
>
>
>  Hi,
>
> I also did a quick smoketest with the provided 3.0.0-alpha2 binaries:
>
> TLDR; It works well
>
> Environment:
>  * 5 hosts, docker based hadoop cluster, every component in separated
> container (5 datanode/5 nodemanager/...)
>  * Components are:
>   * Hdfs/Yarn cluster (upgraded 2.7.3 to 3.0.0-alpha2 using the binary
> package for vote)
>   * Zeppelin 0.6.2/0.7.0-RC2
>   * Spark 2.0.2/2.1.0
>   * HBase 1.2.4 + zookeeper
>   * + additional docker containers for configuration management and
> monitoring
> * No HA, no kerberos, no wire encryption
>
>  * HDFS cluster upgraded successfully from 2.7.3 (with about 200G data)
>  * Imported 100G data to HBase successfully
>  * Started Spark jobs to process 1G json from HDFS (using
> spark-master/slave cluster). It worked even when I used the Zeppelin 0.6.2
> + Spark 2.0.2 (with old hadoop client included). Obviously the old version
> can't use the new Yarn cluster as the token file format has been changed.
>  * I upgraded my setup to use Zeppelin 0.7.0-RC2/Spark 2.1.0(distribution
> without hadoop)/hadoop 3.0.0-alpha2. It also worked well: processed the
> same json files from HDFS with spark jobs (from zeppelin) using yarn
> cluster (master: yarn deploy-mode: cluster)
>  * Started spark jobs (with spark submit, master: yarn) to count records
> from the hbase database: OK
>  * Started example Mapreduce jobs from distribution over yarn. It was OK
> but only with specific configuration (see bellow)
>
> So my overall impression that it works very well (at least with my
> 'smalldata')
>
> Some notes (none of them are blocking):
>
> 1. To run the example mapreduce jobs I defined HADOOP_MAPRED_HOME at
> command line:
> ./bin/yarn jar 
> share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-alpha2.jar
> pi -Dyarn.app.mapreduce.am.env="HADOOP_MAPRED_HOME={{HADOOP_COMMON_HOME}}"
> -Dmapreduce.admin.user.env="HADOOP_MAPRED_HOME={{HADOOP_COMMON_HOME}}" 10
> 10
>
> And in the yarn-site:
>
> yarn.nodemanager.env-whitelist: JAVA_HOME,HADOOP_COMMON_HOME,
> HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_
> DISTCACHE,HADOOP_YARN_HOME,MAPRED_HOME_DIR
>
> I don't know the exact reason for the change, but the 2.7.3 was more
> userfriendly as the example could be run without specific configuration.
>
> For the same reason I didn't start hbase mapreduce job with hbase command
> line app (There could be some option for hbase to define MAPRED_HOME_DIR as
> well, but by default I got ClassNotFoundException for one of the MR class)
>
> 2. For the records: The logging and htrace classes are excluded from the
> shaded hadoop client jar so I added it manually one by one to the spark
> (spark 2.1.0 distribution without hadoop):
>
> RUN wget `cat url` -O spark.tar.gz && tar zxf spark.tar.gz && rm
> spark.tar.gz && mv spark* spark
> RUN cp /opt/hadoop/share/hadoop/client/hadoop-client-api-3.0.0-alpha2.jar
> /opt/spark/jars
> RUN cp /opt/hadoop/share/hadoop/client/hadoop-client-runtime-3.0.0-alpha2.jar
> /opt/spark/jars
> ADD https://repo1.maven.org/maven2/org/slf4j/slf4j-
> log4j12/1.7.10/slf4j-log4j12-1.7.10.jar /opt/spark/jars
> ADD https://repo1.maven.org/maven2/org/apache/htrace/
> htrace-core4/4.1.0-incubating/htrace-core4-4.1.0-incubating.jar
> /opt/spark/jars
> ADD https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.
> 7.10/slf4j-api-1.7.10.jar /opt/spark/jars/
> ADD https://repo1.maven.org/maven2/log4j/log4j/1.2.17/log4j-1.2.17.jar
> /opt/spark/jars
>
> With this jars files spark 2.1.0 works well with the alpha2 version of
> HDFS and YARN.
>
> 3. The messages "Upgrade in progress. Not yet finalized." wasn't
> disappeared from the namenode webui but the cluster works well.
>
> Most probably I missed to do something, but it's a little bit confusing.
>
> (I checked the REST call, it is the jmx bean who reports that it was not
> yet finalized, the code of the webpage seems to be ok.)
>
> Regards
> Marton
>
> On Jan 25, 2017, at 8:38 AM, Yongjun Zhang  jzhan...@apache.org>> wrote:
>
> Thanks Andrew much for the work here!
>
> +1 (binding).
>
> - Downloaded both binary and src tarballs
> - Verified md5 checksum and signature for both
> - Built from source tarball
> - Deployed 2 pseudo clusters, one with the released tarball and the other
>  with what I built from source, and did the following on both:
> - Run basic HDFS operations, snapshots and distcp jobs
> - Run pi job
> - Examined HDFS webui, YARN webui.
>
> Best,
>
> --Yon

Re: [VOTE] Release Apache Hadoop 2.6.5 (RC1)

[Zhihai Xu]

Thanks Sangjin for creating release 2.6.5 RC1.

+1 (non-binding)

* Downloaded and built from source
* Verified md5 checksums and signature
* Deployed a pseudo cluster
* verified basic HDFS operations and Pi job.
* Did a sanity check for RM and NM UI.

Thanks
zhihai

On Fri, Oct 7, 2016 at 8:16 AM, Sangjin Lee  wrote:

> Thanks Masatake!
>
> Today's the last day for this vote, and I'd like to ask you to try out the
> RC and vote on this today. So far there has been no binding vote. Thanks
> again.
>
> Regards,
> Sangjin
>
> On Fri, Oct 7, 2016 at 6:45 AM, Masatake Iwasaki <
> iwasak...@oss.nttdata.co.jp> wrote:
>
> > +1(non-binding)
> >
> > * verified signature and md5.
> > * built with -Pnative on CentOS6 and OpenJDK7.
> > * built documentation and skimmed the contents.
> > * built rpms by bigtop and ran smoke-tests of hdfs, yarn and mapreduce on
> > 3-node cluster.
> >
> > Thanks,
> > Masatake Iwasaki
> >
> > On 10/3/16 09:12, Sangjin Lee wrote:
> >
> >> Hi folks,
> >>
> >> I have pushed a new release candidate (R1) for the Apache Hadoop 2.6.5
> >> release (the next maintenance release in the 2.6.x release line). RC1
> >> contains fixes to CHANGES.txt, and is otherwise identical to RC0.
> >>
> >> Below are the details of this release candidate:
> >>
> >> The RC is available for validation at:
> >> http://home.apache.org/~sjlee/hadoop-2.6.5-RC1/.
> >>
> >> The RC tag in git is release-2.6.5-RC1 and its git commit is
> >> e8c9fe0b4c252caf2ebf1464220599650f119997.
> >>
> >> The maven artifacts are staged via repository.apache.org at:
> >> https://repository.apache.org/content/repositories/
> orgapachehadoop-1050/.
> >>
> >> You can find my public key at
> >> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS.
> >>
> >> Please try the release and vote. The vote will run for the usual 5
> days. I
> >> would greatly appreciate your timely vote. Thanks!
> >>
> >> Regards,
> >> Sangjin
> >>
> >>
> >
>

Re: [VOTE] Release Apache Hadoop 2.7.3 RC2

[Zhihai Xu]

Thanks Vinod for creating release 2.7.3 RC2.

+1 (non-binding)

* Downloaded and built from source
* Checked LICENSE and NOTICE
* Deployed a pseudo cluster
* verified basic HDFS operations and Pi job.
* Did a sanity check for RM and NM UI.


zhihai

On Sat, Aug 20, 2016 at 7:29 AM, Jian He  wrote:

> +1
>
> Built from source code.
> Deployed single node cluster.
> Successfully ran some example jobs.
>
> Jian
>
> > On Aug 18, 2016, at 10:05 AM, Vinod Kumar Vavilapalli <
> vino...@apache.org> wrote:
> >
> > Hi all,
> >
> > I've created a new release candidate RC2 for Apache Hadoop 2.7.3.
> >
> > As discussed before, this is the next maintenance release to follow up
> 2.7.2.
> >
> > The RC is available for validation at: http://home.apache.org/~
> vinodkv/hadoop-2.7.3-RC2/  vinodkv/hadoop-2.7.3-RC2/>
> >
> > The RC tag in git is: release-2.7.3-RC2
> >
> > The maven artifacts are available via repository.apache.org <
> http://repository.apache.org/> at https://repository.apache.org/
> content/repositories/orgapachehadoop-1046  org/content/repositories/orgapachehadoop-1046>
> >
> > The release-notes are inside the tar-balls at location
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> hosted this at http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/
> releasenotes.html  releasenotes.html> for your quick perusal.
> >
> > As you may have noted,
> > - few issues with RC0 forced a RC1 [1]
> > - few more issues with RC1 forced a RC2 [2]
> > - a very long fix-cycle for the License & Notice issues (HADOOP-12893)
> caused 2.7.3 (along with every other Hadoop release) to slip by quite a
> bit. This release's related discussion thread is linked below: [3].
> >
> > Please try the release and vote; the vote will run for the usual 5 days.
> >
> > Thanks,
> > Vinod
> >
> > [1] [VOTE] Release Apache Hadoop 2.7.3 RC0:
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106
> 
> > [2] [VOTE] Release Apache Hadoop 2.7.3 RC1:
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg26336.html <
> https://www.mail-archive.com/hdfs-dev@hadoop.apache.org/msg26336.html>
> > [3] 2.7.3 release plan: https://www.mail-archive.com/
> hdfs-dev%40hadoop.apache.org/msg24439.html  6yv2fyrs4jlepmmr>
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Zhihai Xu]

Thanks Vinod.

+1 (non-binding)

* Downloaded and built from source
* Checked LICENSE and NOTICE
* Deployed a pseudo cluster
* Ran through MR and HDFS tests
* verified basic HDFS operations and Pi job.

Zhihai

On Fri, Jul 22, 2016 at 7:15 PM, Vinod Kumar Vavilapalli  wrote:

> Hi all,
>
> I've created a release candidate RC0 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up
> 2.7.2.
>
> The RC is available for validation at:
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ <
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>
> The RC tag in git is: release-2.7.3-RC0
>
> The maven artifacts are available via repository.apache.org <
> http://repository.apache.org/> at
> https://repository.apache.org/content/repositories/orgapachehadoop-1040/ <
> https://repository.apache.org/content/repositories/orgapachehadoop-1040/>
>
> The release-notes are inside the tar-balls at location
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> hosted this at
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html <
> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for
> your quick perusal.
>
> As you may have noted, a very long fix-cycle for the License & Notice
> issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release)
> to slip by quite a bit. This release's related discussion thread is linked
> below: [1].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1]: 2.7.3 release plan:
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <
> http://markmail.org/thread/6yv2fyrs4jlepmmr>

[jira] [Created] (HADOOP-13249) RetryInvocationHandler need wrap InterruptedException in IOException when call Thread.sleep

[zhihai xu (JIRA)]

zhihai xu created HADOOP-13249:
--

 Summary: RetryInvocationHandler need wrap InterruptedException in 
IOException when call Thread.sleep
 Key: HADOOP-13249
 URL: https://issues.apache.org/jira/browse/HADOOP-13249
 Project: Hadoop Common
  Issue Type: Bug
  Components: ipc
Affects Versions: 2.8.0
Reporter: zhihai xu
Assignee: zhihai xu


RetryInvocationHandler need wrap InterruptedException in IOException when call 
Thread.sleep. Otherwise InterruptedException can't be handled correctly by 
other components such as HDFS. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13247) The CACHE entry in FileSystem is not removed if exception happened in close

[zhihai xu (JIRA)]

zhihai xu created HADOOP-13247:
--

 Summary: The CACHE entry in FileSystem is not removed if exception 
happened in close
 Key: HADOOP-13247
 URL: https://issues.apache.org/jira/browse/HADOOP-13247
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 2.8.0
Reporter: zhihai xu
Assignee: zhihai xu


The CACHE entry in FileSystem is not removed if exception happened in close. It 
causes "Filesystem closed" IOException if the same filesystem is used later.
The following is stack trace for the exception coming out of close:
{code}
2016-06-07 18:21:18,201 ERROR hive.ql.exec.DDLTask: 
org.apache.hadoop.hive.ql.metadata.HiveException: 
java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:756)
at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4022)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:306)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:172)
at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1679)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1422)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1205)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1052)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1047)
at 
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:158)
at 
org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:76)
at 
org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:219)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at 
org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:231)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.UndeclaredThrowableException
at com.sun.proxy.$Proxy18.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
at 
org.apache.hadoop.fs.FileSystem.processDeleteOnExit(FileSystem.java:1383)
at org.apache.hadoop.fs.FileSystem.close(FileSystem.java:2006)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:900)
at 
org.apache.hadoop.hive.metastore.Warehouse.closeFs(Warehouse.java:122)
at org.apache.hadoop.hive.metastore.Warehouse.isDir(Warehouse.java:497)
at 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.createTempTable(SessionHiveMetaStoreClient.java:345)
at 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.create_table_with_environment_context(SessionHiveMetaStoreClient.java:93)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:664)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:652)
at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
at com.sun.proxy.$Proxy8.createTable(Unknown Source)
at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:1909)
at com.sun.proxy.$Proxy8.createTable(Unknown Source)

Re: Different JIRA permissions for HADOOP and HDFS

[Zhihai Xu]

Great, Thanks Junping! Yes, the JIRA assignment works for me now.

zhihai

On Mon, May 16, 2016 at 5:29 AM, Junping Du  wrote:

> Zhihai, I just set you with committer permissions on MAPREDUCE JIRA. Would
> you try if the JIRA assignment works now? I cannot help on Hive project. It
> is better to ask hive project community for help.
> For Arun's problem. from my check, the Edit permission on JIRA only
> authorized to Administrator only. I don't know if this setting is by
> intention but it was not like this previously.
> Can someone who make the change to clarify why we need this change or
> revert to whatever it used to be?
>
> Thanks,
>
> Junping
> 
> From: Arun Suresh 
> Sent: Monday, May 16, 2016 9:42 AM
> To: Zhihai Xu
> Cc: Zheng, Kai; Andrew Wang; common-dev@hadoop.apache.org;
> yarn-...@hadoop.apache.org
> Subject: Re: Different JIRA permissions for HADOOP and HDFS
>
> Not sure if this is related.. but It also looks like I am now no longer
> allowed to modify description and headline of JIRAs anymore..
> Would appreciate greatly if someone can help revert this !
>
> Cheers
> -Arun
>
> On Mon, May 16, 2016 at 1:21 AM, Zhihai Xu  wrote:
>
> > Currently I also have permission issue to access the JIRA. I can't assign
> > the JIRA(I created) to myself. For example,
> > https://issues.apache.org/jira/browse/MAPREDUCE-6696 and
> > https://issues.apache.org/jira/browse/HIVE-13760. I can't find the
> button
> > to assign the JIRA to myself.
> > I don't have this issue two three weeks ago. Did anything change
> recently?
> > Can anyone help me solve this issue?
> >
> > thanks
> > zhihai
> >
> >
> >
> >
> > On Mon, May 16, 2016 at 12:22 AM, Zheng, Kai 
> wrote:
> >
> > > It works for me now, thanks Andrew!
> > >
> > > Regards,
> > > Kai
> > >
> > > -Original Message-
> > > From: Andrew Wang [mailto:andrew.w...@cloudera.com]
> > > Sent: Monday, May 16, 2016 12:14 AM
> > > To: Zheng, Kai 
> > > Cc: common-dev@hadoop.apache.org
> > > Subject: Re: Different JIRA permissions for HADOOP and HDFS
> > >
> > > I just gave you committer permissions on JIRA, try now?
> > >
> > > On Mon, May 16, 2016 at 12:03 AM, Zheng, Kai 
> > wrote:
> > >
> > > > I just ran into the bad situation that I committed HDFS-8449 but
> can't
> > > > resolve the issue due to lacking the required permission to me. Am
> not
> > > > sure if it's caused by my setup or environment change (temporally
> > > > working in a new time zone). Would anyone help resolve the issue for
> > > > me to avoid bad state? Thanks!
> > > >
> > > > -Original Message-
> > > > From: Zheng, Kai [mailto:kai.zh...@intel.com]
> > > > Sent: Sunday, May 15, 2016 3:20 PM
> > > > To: Allen Wittenauer 
> > > > Cc: common-dev@hadoop.apache.org
> > > > Subject: RE: Different JIRA permissions for HADOOP and HDFS
> > > >
> > > > Thanks Allen for illustrating this in details. I understand. The left
> > > > question is, is it intended only JIRA owner (not sure about admin
> > > > users) can do the operations like updating a patch?
> > > >
> > > > Regards,
> > > > Kai
> > > >
> > > > -Original Message-
> > > > From: Allen Wittenauer [mailto:allenwittena...@yahoo.com.INVALID]
> > > > Sent: Saturday, May 14, 2016 9:38 AM
> > > > To: Zheng, Kai 
> > > > Cc: common-dev@hadoop.apache.org
> > > > Subject: Re: Different JIRA permissions for HADOOP and HDFS
> > > >
> > > >
> > > > > On May 14, 2016, at 7:07 AM, Zheng, Kai 
> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > Noticed this difference but not sure if it’s intended. YARN is
> > > > > similar
> > > > with HDFS. It’s not convenient. Any clarifying?
> > > >
> > > >
> > > > Under JIRA, different projects (e.g., HADOOP, YARN,
> MAPREDUCE,
> > > > HDFS, YETUS, HBASE, ACCUMULO, etc) may have different settings.  At
> > > > one point in time, all of the Hadoop subprojects were under one JIRA
> > > > project (HADOOP). But then a bunch of folks decided they didn’t want
> > > > to see the other sub projects issues so they split them up….

Re: Different JIRA permissions for HADOOP and HDFS

[Zhihai Xu]

Currently I also have permission issue to access the JIRA. I can't assign
the JIRA(I created) to myself. For example,
https://issues.apache.org/jira/browse/MAPREDUCE-6696 and
https://issues.apache.org/jira/browse/HIVE-13760. I can't find the button
to assign the JIRA to myself.
I don't have this issue two three weeks ago. Did anything change recently?
Can anyone help me solve this issue?

thanks
zhihai




On Mon, May 16, 2016 at 12:22 AM, Zheng, Kai  wrote:

> It works for me now, thanks Andrew!
>
> Regards,
> Kai
>
> -Original Message-
> From: Andrew Wang [mailto:andrew.w...@cloudera.com]
> Sent: Monday, May 16, 2016 12:14 AM
> To: Zheng, Kai 
> Cc: common-dev@hadoop.apache.org
> Subject: Re: Different JIRA permissions for HADOOP and HDFS
>
> I just gave you committer permissions on JIRA, try now?
>
> On Mon, May 16, 2016 at 12:03 AM, Zheng, Kai  wrote:
>
> > I just ran into the bad situation that I committed HDFS-8449 but can't
> > resolve the issue due to lacking the required permission to me. Am not
> > sure if it's caused by my setup or environment change (temporally
> > working in a new time zone). Would anyone help resolve the issue for
> > me to avoid bad state? Thanks!
> >
> > -Original Message-
> > From: Zheng, Kai [mailto:kai.zh...@intel.com]
> > Sent: Sunday, May 15, 2016 3:20 PM
> > To: Allen Wittenauer 
> > Cc: common-dev@hadoop.apache.org
> > Subject: RE: Different JIRA permissions for HADOOP and HDFS
> >
> > Thanks Allen for illustrating this in details. I understand. The left
> > question is, is it intended only JIRA owner (not sure about admin
> > users) can do the operations like updating a patch?
> >
> > Regards,
> > Kai
> >
> > -Original Message-
> > From: Allen Wittenauer [mailto:allenwittena...@yahoo.com.INVALID]
> > Sent: Saturday, May 14, 2016 9:38 AM
> > To: Zheng, Kai 
> > Cc: common-dev@hadoop.apache.org
> > Subject: Re: Different JIRA permissions for HADOOP and HDFS
> >
> >
> > > On May 14, 2016, at 7:07 AM, Zheng, Kai  wrote:
> > >
> > > Hi,
> > >
> > > Noticed this difference but not sure if it’s intended. YARN is
> > > similar
> > with HDFS. It’s not convenient. Any clarifying?
> >
> >
> > Under JIRA, different projects (e.g., HADOOP, YARN, MAPREDUCE,
> > HDFS, YETUS, HBASE, ACCUMULO, etc) may have different settings.  At
> > one point in time, all of the Hadoop subprojects were under one JIRA
> > project (HADOOP). But then a bunch of folks decided they didn’t want
> > to see the other sub projects issues so they split them up…. and thus
> > setting the stage for duplicate code and operational divergence in the
> source.
> >
> > Since people don’t realize or care that they are separate,
> > people will file INFRA tickets or whatever to change “their project”
> > and not the rest. This leads to the JIRA projects also diverging…
> > which ultimately drives those of us who actually look at the project as
> a whole bonkers.
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>

Re: The Activities of Apache Hadoop Community 2015

[Zhihai Xu]

Thanks for sharing this information Akira!

zhihai

On Sun, Jan 24, 2016 at 11:06 PM, Akira AJISAKA 
wrote:

> Hi folks,
>
> We wrote a blog post about the activities of Apache Hadoop community.
>
> http://ajisakaa.blogspot.com/2016/01/the-activities-of-apache-hadoop.html
>
> According to the post, the activities of Apache Hadoop Community
> was continued to expand also in 2015.
> We really appreciate the continuous contributions of the developers.
>
> We wrote similar blog posts a year ago and two years ago.
>
> (2014)
> http://ajisakaa.blogspot.com/2015/02/the-activities-of-apache-hadoop.html
> (2013)
> http://ajisakaa.blogspot.com/2014/02/the-activities-of-apache-hadoop.html
>
> Thanks,
> Akira
>

[jira] [Created] (HADOOP-12443) LocalDirAllocator shouldn't accept pathStr parameter with scheme or authority.

[zhihai xu (JIRA)]

zhihai xu created HADOOP-12443:
--

 Summary: LocalDirAllocator shouldn't accept pathStr parameter with 
scheme or authority.
 Key: HADOOP-12443
 URL: https://issues.apache.org/jira/browse/HADOOP-12443
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Reporter: zhihai xu
Assignee: zhihai xu


{{LocalDirAllocator}} shouldn't accept {{pathStr}} parameter with scheme or 
authority.
Currently {{LocalDirAllocator}} accepts {{pathStr}} with scheme or authority, 
When {{pathStr}} with scheme or authority is passed to 
{{getLocalPathForWrite}}, it will bypass {{localDirs}} to use {{pathStr}} 
directly , then the return Path will be independent with {{localDirs}}.
The reason is the following:
{{LocalDirAllocator}} will use {{new Path(new 
Path(localDirs[dirNumLastAccessed]), pathStr)}} as the return Path.
The constructor code for {{Path}} is
{code}
  public Path(Path parent, Path child) {
// Add a slash to parent's path so resolution is compatible with URI's
URI parentUri = parent.uri;
String parentPath = parentUri.getPath();
if (!(parentPath.equals("/") || parentPath.isEmpty())) {
  try {
parentUri = new URI(parentUri.getScheme(), parentUri.getAuthority(),
  parentUri.getPath()+"/", null, parentUri.getFragment());
  } catch (URISyntaxException e) {
throw new IllegalArgumentException(e);
  }
}
URI resolved = parentUri.resolve(child.uri);
initialize(resolved.getScheme(), resolved.getAuthority(),
   resolved.getPath(), resolved.getFragment());
  }
{code}
The above {{Path}} constructor code will call {{URI#resolve}} to merge the 
parent path with child path.
{code}
private static URI resolve(URI base, URI child) {
// check if child if opaque first so that NPE is thrown
// if child is null.
if (child.isOpaque() || base.isOpaque())
return child;

// 5.2 (2): Reference to current document (lone fragment)
if ((child.scheme == null) && (child.authority == null)
&& child.path.equals("") && (child.fragment != null)
&& (child.query == null)) {
if ((base.fragment != null)
&& child.fragment.equals(base.fragment)) {
return base;
}
URI ru = new URI();
ru.scheme = base.scheme;
ru.authority = base.authority;
ru.userInfo = base.userInfo;
ru.host = base.host;
ru.port = base.port;
ru.path = base.path;
ru.fragment = child.fragment;
ru.query = base.query;
return ru;
}

// 5.2 (3): Child is absolute
if (child.scheme != null)
return child;

URI ru = new URI(); // Resolved URI
ru.scheme = base.scheme;
ru.query = child.query;
ru.fragment = child.fragment;

// 5.2 (4): Authority
if (child.authority == null) {
ru.authority = base.authority;
ru.host = base.host;
ru.userInfo = base.userInfo;
ru.port = base.port;

String cp = (child.path == null) ? "" : child.path;
if ((cp.length() > 0) && (cp.charAt(0) == '/')) {
// 5.2 (5): Child path is absolute
ru.path = child.path;
} else {
// 5.2 (6): Resolve relative path
ru.path = resolvePath(base.path, cp, base.isAbsolute());
}
} else {
ru.authority = child.authority;
ru.host = child.host;
ru.userInfo = child.userInfo;
ru.host = child.host;
ru.port = child.port;
ru.path = child.path;
}

// 5.2 (7): Recombine (nothing to do here)
return ru;
}
{code}
You can see if the child's uri has scheme or authority, it won't use anything 
from parent's uri.
This will hide the issue for user. For example, user passed 
file:///build/test/temp as {{pathStr}} parameter to {{getLocalPathForWrite}}.
Later on user may run into very strange problem: /build/test/temp directory is 
full because return path is not from {{localDirs}}. This makes the issue very 
difficult for user to debug. So it will be better to reject {{pathStr}} 
parameter with scheme or authority.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: reviews for HADOOP-12178

[Zhihai Xu]

Yes, I just did a quick review for HADOOP-12178
.

Regards
zhihai

On Fri, Sep 25, 2015 at 11:13 AM, Steve Loughran 
wrote:

> Can I get a quick review of :
> https://issues.apache.org/jira/browse/HADOOP-12178
>
> I'm talking about hadoop & kerberos next week and I'd like to have less
> open patches related to reporting problems in the security codebase
>
> thanks
>
> -steve
>

[jira] [Created] (HADOOP-12413) AccessControlList should avoid calling getGroupNames in isUserInList with empty groups.

[zhihai xu (JIRA)]

zhihai xu created HADOOP-12413:
--

 Summary: AccessControlList should avoid calling getGroupNames in 
isUserInList with empty groups.
 Key: HADOOP-12413
 URL: https://issues.apache.org/jira/browse/HADOOP-12413
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 2.7.0
Reporter: zhihai xu
Assignee: zhihai xu


{{AccessControlList}} should avoid calling {{getGroupNames}} in 
{{isUserInList}} with empty {{groups}}. Currently {{AccessControlList}} will 
call {{ugi.getGroupNames()}} in {{isUserInList}} even if {{groups}} is empty. 
{{ugi.getGroupNames()}} is an expensive operation which call shell script {{id 
-gn  && id -Gn }} to get the list of groups. For example,
{{ServiceAuthorizationManager#authorize}} will call blocked ACL 
{{acls[1].isUserAllowed(user)}} to check the user permission. The default value 
for blocked ACL  is empty
{{code}}
String defaultBlockedAcl = conf.get(   
CommonConfigurationKeys.HADOOP_SECURITY_SERVICE_AUTHORIZATION_DEFAULT_BLOCKED_ACL,
 "");
{{code}}
So every time {{authorize}} is called, {{getGroupNames}} may be called.
It also caused the following warning message:
{code}
2015-09-08 14:55:34,236 WARN [Socket Reader #1 for port 52715] 
org.apache.hadoop.security.ShellBasedUnixGroupsMapping: got exception trying to 
get groups for user job_144171553_0005: id: job_144171553_0005: No such 
user
2015-09-08 14:55:34,236 WARN [Socket Reader #1 for port 52715] 
org.apache.hadoop.security.UserGroupInformation: No groups available for user 
job_144171553_0005
2015-09-08 14:55:34,236 INFO [Socket Reader #1 for port 52715] 
SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager:
 Authorization successful for job_144171553_0005 (auth:TOKEN) for 
protocol=interface org.apache.hadoop.mapred.TaskUmbilicalProtocol
{{code}}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12404) Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class.

[zhihai xu (JIRA)]

zhihai xu created HADOOP-12404:
--

 Summary: Disable caching for JarURLConnection to avoid sharing 
JarFile with other users when loading resource from URL in Configuration class.
 Key: HADOOP-12404
 URL: https://issues.apache.org/jira/browse/HADOOP-12404
 Project: Hadoop Common
  Issue Type: Improvement
  Components: conf
Reporter: zhihai xu
Assignee: zhihai xu
Priority: Minor


Disable caching for JarURLConnection to avoid sharing JarFile with other users 
when loading resource from URL in Configuration class.
Currently {{Configuration#parse}} will call {{url.openStream}} to get the 
InputStream for {{DocumentBuilder}} to parse. Based on the JDK source code, the 
calling sequence is 
url.openStream => 
[handler.openConnection.getInputStream|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/sun/net/www/protocol/jar/Handler.java]
 => [new 
JarURLConnection|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/sun/net/www/protocol/jar/JarURLConnection.java#JarURLConnection]
 => JarURLConnection.connect => [factory.get(getJarFileURL(), 
getUseCaches())|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/sun/net/www/protocol/jar/JarFileFactory.java]
 =>  
[URLJarFile.getInputStream|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/sun/net/www/protocol/jar/URLJarFile.java#URLJarFile.getJarFile%28java.net.URL%2Csun.net.www.protocol.jar.URLJarFile.URLJarFileCloseController%29]=>[JarFile.getInputStream|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/jar/JarFile.java#JarFile.getInputStream%28java.util.zip.ZipEntry%29]=>ZipFile.getInputStream
If {{URLConnection#getUseCaches}} is true (by default), URLJarFile will be 
shared for the same URL. If the shared URLJarFile is closed by other users, 
all the InputStream returned by URLJarFile#getInputStream will be closed based 
on the document: 
http://docs.oracle.com/javase/7/docs/api/java/util/zip/ZipFile.html#getInputStream(java.util.zip.ZipEntry)
So we saw the following exception at rare situation which cause a hive job 
failed in a heavyload system.
{code}
2014-10-21 23:44:41,856 ERROR org.apache.hadoop.hive.ql.exec.Task: Ended 
Job = job_1413909398487_3696 with exception 
'java.lang.RuntimeException(java.io.IOException: Stream closed)' 
java.lang.RuntimeException: java.io.IOException: Stream closed 
at 
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2484) 
at 
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2337) 
at 
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2254) 
at org.apache.hadoop.conf.Configuration.get(Configuration.java:861) 
at 
org.apache.hadoop.mapred.JobConf.checkAndWarnDeprecation(JobConf.java:2030) 
at org.apache.hadoop.mapred.JobConf.(JobConf.java:479) 
at org.apache.hadoop.mapred.JobConf.(JobConf.java:469) 
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:187) 
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:582) 
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:580) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:415) 
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j 
ava:1614) 
at 
org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:580) 
at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:598) 
at 
org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExe 
cHelper.java:288) 
at 
org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExe 
cHelper.java:547) 
at 
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:426) 
at 
org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:136) 
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153) 
at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85) 
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1516) 
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1283) 
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1101) 
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:924) 
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:919) 
at 
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation 
.java:145) 
at 
org.apache.hive.service.cli.operation.SQLOperation.access$000(SQLOperation. 
java:69) 
at 
org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.jav 
a:200) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:415) 
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j 
ava:1614) 
at 
org.apache.hadoop.hive.shims.HadoopShimsSecure.do

[jira] [Created] (HADOOP-12348) MetricsSystemImpl creates MetricsSourceAdapter with wrong time unit parameter.

[zhihai xu (JIRA)]

zhihai xu created HADOOP-12348:
--

 Summary: MetricsSystemImpl creates MetricsSourceAdapter with wrong 
time unit parameter.
 Key: HADOOP-12348
 URL: https://issues.apache.org/jira/browse/HADOOP-12348
 Project: Hadoop Common
  Issue Type: Bug
  Components: metrics
Reporter: zhihai xu
Assignee: zhihai xu
Priority: Minor


MetricsSystemImpl creates MetricsSourceAdapter with wrong time unit parameter. 
MetricsSourceAdapter expects time unit millisecond  for jmxCacheTTL but 
MetricsSystemImpl  passes time unit second to MetricsSourceAdapter constructor.
{code}
jmxCacheTS = Time.now() + jmxCacheTTL;
  /**
   * Current system time.  Do not use this to calculate a duration or interval
   * to sleep, because it will be broken by settimeofday.  Instead, use
   * monotonicNow.
   * @return current time in msec.
   */
  public static long now() {
return System.currentTimeMillis();
  }
{code}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12268) AbstractContractAppendTest#testRenameFileBeingAppended missed rename operation.

[zhihai xu (JIRA)]

zhihai xu created HADOOP-12268:
--

 Summary: AbstractContractAppendTest#testRenameFileBeingAppended 
missed rename operation.
 Key: HADOOP-12268
 URL: https://issues.apache.org/jira/browse/HADOOP-12268
 Project: Hadoop Common
  Issue Type: Bug
  Components: test
Reporter: zhihai xu
Assignee: zhihai xu


AbstractContractAppendTest#testRenameFileBeingAppended missed rename operation. 
Also TestHDFSContractAppend can pass the original test  after fix the issue at 
{{AbstractContractAppendTest#testRenameFileBeingAppended}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12258) Need translate java.nio.file.NoSuchFileException to FileNotFoundException in DeprecatedRawLocalFileStatus constructor

[zhihai xu (JIRA)]

zhihai xu created HADOOP-12258:
--

 Summary: Need translate java.nio.file.NoSuchFileException to 
FileNotFoundException in DeprecatedRawLocalFileStatus constructor
 Key: HADOOP-12258
 URL: https://issues.apache.org/jira/browse/HADOOP-12258
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Reporter: zhihai xu
Assignee: zhihai xu
Priority: Blocker


need translate java.nio.file.NoSuchFileException to FileNotFoundException in 
DeprecatedRawLocalFileStatus constructor. 
HADOOP-12045 adds nio to support access time, but nio will create 
java.nio.file.NoSuchFileException instead of FileNotFoundException.
many hadoop codes depend on FileNotFoundException to decide whether a file 
exists. for example {{FileContext.util().exists()}}. 
{code}
public boolean exists(final Path f) throws AccessControlException,
  UnsupportedFileSystemException, IOException {
  try {
FileStatus fs = FileContext.this.getFileStatus(f);
assert fs != null;
return true;
  } catch (FileNotFoundException e) {
return false;
  }
}
{code}
same for {{FileSystem#exists}}
{code}
  public boolean exists(Path f) throws IOException {
try {
  return getFileStatus(f) != null;
} catch (FileNotFoundException e) {
  return false;
}
  }
{code}
NoSuchFileException will break these functions.

Several test failures are caused by this issue:
https://builds.apache.org/job/PreCommit-YARN-Build/8630/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testRelativeDelete/
https://builds.apache.org/job/PreCommit-YARN-Build/8632/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testAbsDelete/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12252) LocalDirAllocator should not throw NPE with empty string configuration.

[zhihai xu (JIRA)]

zhihai xu created HADOOP-12252:
--

 Summary: LocalDirAllocator should not throw NPE with empty string 
configuration. 
 Key: HADOOP-12252
 URL: https://issues.apache.org/jira/browse/HADOOP-12252
 Project: Hadoop Common
  Issue Type: Bug
Reporter: zhihai xu
Assignee: zhihai xu


LocalDirAllocator should not throw NPE with empty string configuration. When an 
empty string is configured, {{LocalDirAllocator}} will throw NPE. It will be 
better to throw IOException instead of NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12189) CallQueueManager may drop elements from the queue sometimes when calling swapQueue

[zhihai xu (JIRA)]

zhihai xu created HADOOP-12189:
--

 Summary: CallQueueManager may drop elements from the queue 
sometimes when calling swapQueue
 Key: HADOOP-12189
 URL: https://issues.apache.org/jira/browse/HADOOP-12189
 Project: Hadoop Common
  Issue Type: Bug
  Components: ipc
Affects Versions: 2.7.1
Reporter: zhihai xu
Assignee: zhihai xu


CallQueueManager may drop elements from the queue sometimes when calling 
{{swapQueue}}. 
The following test failure shown some elements in the queue are dropped.
https://builds.apache.org/job/PreCommit-HADOOP-Build/7150/testReport/org.apache.hadoop.ipc/TestCallQueueManager/testSwapUnderContention/
{code}
java.lang.AssertionError: expected:<27241> but was:<27245>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at 
org.apache.hadoop.ipc.TestCallQueueManager.testSwapUnderContention(TestCallQueueManager.java:220)
{code}
It looked like the elements in the queue are dropped due to 
{{CallQueueManager#swapQueue}}
Looked at the implementation of {{CallQueueManager#swapQueue}}, there is a 
possibility that the elements in the queue are dropped. If the queue is full, 
the calling thread for {{CallQueueManager#put}} is blocked for long time. It 
may put the element into the old queue after queue in {{takeRef}} is changed by 
swapQueue, then this element in the old queue will be dropped.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12186) ActiveStandbyElector shouldn't call monitorLockNodeAsync before StatCallback for previous zkClient.exists is received.

[zhihai xu (JIRA)]

zhihai xu created HADOOP-12186:
--

 Summary: ActiveStandbyElector shouldn't call monitorLockNodeAsync 
before StatCallback for previous zkClient.exists is received.
 Key: HADOOP-12186
 URL: https://issues.apache.org/jira/browse/HADOOP-12186
 Project: Hadoop Common
  Issue Type: Bug
  Components: ha
Affects Versions: 2.7.1
Reporter: zhihai xu
Assignee: zhihai xu


ActiveStandbyElector shouldn't call {{monitorLockNodeAsync}} before 
StatCallback for previous {{zkClient.exists}} is received.
We saw RM shutdown because ActiveStandbyElector retrying monitorLockNodeAsync 
exceeded limit. The following is the logs.
Based on the log, it looks like multiple {{monitorLockNodeAsync}} are called at 
the same time due to back-to-back SyncConnected event received.
The current code doesn't prevent {{zkClient.exists}} from being called before 
AsyncCallback.StatCallback for previous {{zkClient.exists}} is received.
So the retry for {{monitorLockNodeAsync}} doesn't work correctly sometimes.
{code}
2015-07-01 19:24:12,806 INFO org.apache.zookeeper.ClientCnxn: Client session 
timed out, have not heard from server in 6674ms for sessionid 
0x14e47693cc20007, closing socket connection and attempting reconnect
2015-07-01 19:24:12,919 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session 
disconnected. Entering neutral mode...
2015-07-01 19:24:14,704 INFO org.apache.zookeeper.ClientCnxn: Opening socket 
connection to server node-1.internal/192.168.123.3:2181. Will not attempt to 
authenticate using SASL (unknown error)
2015-07-01 19:24:14,704 INFO org.apache.zookeeper.ClientCnxn: Socket connection 
established, initiating session, client: /192.168.123.3:43487, server: 
node-1.internal/192.168.123.3:2181
2015-07-01 19:24:14,707 INFO org.apache.zookeeper.ClientCnxn: Session 
establishment complete on server node-1.internal/192.168.123.3:2181, sessionid 
= 0x14e47693cc20007, negotiated timeout = 1
2015-07-01 19:24:14,712 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session 
connected.
2015-07-01 19:24:21,374 INFO org.apache.zookeeper.ClientCnxn: Client session 
timed out, have not heard from server in 6667ms for sessionid 
0x14e47693cc20007, closing socket connection and attempting reconnect
2015-07-01 19:24:21,477 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session 
disconnected. Entering neutral mode...
2015-07-01 19:24:22,640 INFO org.apache.zookeeper.ClientCnxn: Opening socket 
connection to server node-1.internal/192.168.123.3:2181. Will not attempt to 
authenticate using SASL (unknown error)
2015-07-01 19:24:22,640 INFO org.apache.zookeeper.ClientCnxn: Socket connection 
established, initiating session, client: /192.168.123.3:43526, server: 
node-1.internal/192.168.123.3:2181
2015-07-01 19:24:22,641 INFO org.apache.zookeeper.ClientCnxn: Session 
establishment complete on server node-1.internal/192.168.123.3:2181, sessionid 
= 0x14e47693cc20007, negotiated timeout = 1
2015-07-01 19:24:22,642 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session 
connected.
2015-07-01 19:24:29,310 INFO org.apache.zookeeper.ClientCnxn: Client session 
timed out, have not heard from server in 6669ms for sessionid 
0x14e47693cc20007, closing socket connection and attempting reconnect
2015-07-01 19:24:29,413 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session 
disconnected. Entering neutral mode...
2015-07-01 19:24:30,738 INFO org.apache.zookeeper.ClientCnxn: Opening socket 
connection to server node-1.internal/192.168.123.3:2181. Will not attempt to 
authenticate using SASL (unknown error)
2015-07-01 19:24:30,739 INFO org.apache.zookeeper.ClientCnxn: Socket connection 
established, initiating session, client: /192.168.123.3:43574, server: 
node-1.internal/192.168.123.3:2181
2015-07-01 19:24:30,739 INFO org.apache.zookeeper.ClientCnxn: Session 
establishment complete on server node-1.internal/192.168.123.3:2181, sessionid 
= 0x14e47693cc20007, negotiated timeout = 1
2015-07-01 19:24:30,740 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session 
connected.
2015-07-01 19:24:37,409 INFO org.apache.zookeeper.ClientCnxn: Client session 
timed out, have not heard from server in 6670ms for sessionid 
0x14e47693cc20007, closing socket connection and attempting reconnect
2015-07-01 19:24:37,512 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session 
disconnected. Entering neutral mode...
2015-07-01 19:24:38,979 INFO org.apache.zookeeper.ClientCnxn: Opening socket 
connection to server node-1.internal/192.168.123.3:2181. Will not attempt to 
authenticate using SASL (unknown error)
2015-07-01 19:24:38,979 INFO org.apache.zookeeper.ClientCnxn: Socket connection 
established, initiating session, client: /192.168.123.3:43598, server: 
node-1.internal/192.168.123.3:2181
2015-07-01 19:24:38,980 INFO org.apache.zookeeper.ClientCnxn: Session 
establishment complete on server node-1.in

[jira] [Created] (HADOOP-12117) Potential NPE from Configuration#loadProperty with allowNullValueProperties set.

[zhihai xu (JIRA)]

zhihai xu created HADOOP-12117:
--

 Summary: Potential NPE from Configuration#loadProperty with 
allowNullValueProperties set.
 Key: HADOOP-12117
 URL: https://issues.apache.org/jira/browse/HADOOP-12117
 Project: Hadoop Common
  Issue Type: Bug
  Components: conf
Affects Versions: 2.7.1
Reporter: zhihai xu
Assignee: zhihai xu


Potential NPE from Configuration#loadProperty with allowNullValueProperties set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12056) DiskChecker#checkDirs should check null pointer for the return value from File#listFiles to avoid NPE.

[zhihai xu (JIRA)]

zhihai xu created HADOOP-12056:
--

 Summary: DiskChecker#checkDirs should check null pointer for the 
return value from File#listFiles to avoid NPE.
 Key: HADOOP-12056
 URL: https://issues.apache.org/jira/browse/HADOOP-12056
 Project: Hadoop Common
  Issue Type: Bug
  Components: util
Affects Versions: 2.7.0
Reporter: zhihai xu
Assignee: zhihai xu


DiskChecker#checkDirs should check null pointer for the return value from 
File#listFiles. Based on the document for File#listFiles at 
https://docs.oracle.com/javase/7/docs/api/java/io/File.html#listFiles() :
{code}
Returns null if an I/O error occurs.
{code}
So it will be good to check null pointer and throw DiskErrorException if it is 
null.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Planning Hadoop 2.6.1 release

[Zhihai Xu]

Hi Akira,

Can we also include YARN-3242? YARN-3242 fixed a critical ZKRMStateStore
bug.
It will work better with YARN-2992.

thanks
zhihai


On Tue, May 12, 2015 at 10:38 PM, Akira AJISAKA 
wrote:

> Thanks all for collecting jiras for 2.6.1 release. In addition, I'd like
> to include the following:
>
> * HADOOP-11343. Overflow is not properly handled in calculating final iv
> for AES CTR
> * YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to
> execute any further apps
> * YARN-2992. ZKRMStateStore crashes due to session expiry
> * YARN-3013. AMRMClientImpl does not update AMRM token properly
> * YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to
> die
> * MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal to
> a reducer
>
> All of these are marked as blocker bug for 2.7.0 but not fixed in 2.6.0.
>
> Regards,
> Akira
>
>
> On 5/4/15 11:15, Brahma Reddy Battula wrote:
>
>> Hello Vinod,
>>
>> I am thinking,can we include HADOOP-11491 also..? wihout this jira harfs
>> will not be usable when cluster installed in HA mode and try to get
>> filecontext like below..
>>
>>
>> Path path = new
>> Path("har:///archivedLogs/application_1428917727658_0005-application_1428917727658_0008-1428927448352.har");
>> FileSystem fs = path.getFileSystem(new Configuration());
>> path = fs.makeQualified(path);
>> FileContext fc = FileContext.getFileContext(path.toUri(),new
>> Configuration());
>>
>>
>>
>> Thanks & Regards
>> Brahma Reddy Battula
>> 
>> From: Chris Nauroth [cnaur...@hortonworks.com]
>> Sent: Friday, May 01, 2015 4:32 AM
>> To: mapreduce-...@hadoop.apache.org; common-dev@hadoop.apache.org;
>> yarn-...@hadoop.apache.org; hdfs-...@hadoop.apache.org
>> Subject: Re: Planning Hadoop 2.6.1 release
>>
>> Thank you, Arpit.  In addition, I suggest we include the following:
>>
>> HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification
>> pipe is full
>> HADOOP-11604. Prevent ConcurrentModificationException while closing domain
>> sockets during shutdown of DomainSocketWatcher thread.
>> HADOOP-11648. Set DomainSocketWatcher thread name explicitly
>> HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there
>> is an I/O error during requestShortCircuitShm
>>
>> HADOOP-11604 and 11648 are not critical by themselves, but they are
>> pre-requisites to getting a clean cherry-pick of 11802, which we believe
>> finally fixes the root cause of this issue.
>>
>>
>> --Chris Nauroth
>>
>>
>>
>>
>> On 4/30/15, 3:55 PM, "Arpit Agarwal"  wrote:
>>
>>  HDFS candidates for back-porting to Hadoop 2.6.1. The first two were
>>> requested in [1].
>>>
>>> HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream
>>> should be non static
>>> HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt
>>> synchronization
>>>
>>> HDFS-7009. Active NN and standby NN have different live nodes.
>>> HDFS-7035. Make adding a new data directory to the DataNode an atomic and
>>> improve error handling
>>> HDFS-7425. NameNode block deletion logging uses incorrect appender.
>>> HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate
>>> block files are present in the same volume.
>>> HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes
>>> HDFS-7503. Namenode restart after large deletions can cause slow
>>> processReport.
>>> HDFS-7575. Upgrade should generate a unique storage ID for each volume.
>>> HDFS-7579. Improve log reporting during block report rpc failure.
>>> HDFS-7587. Edit log corruption can happen if append fails with a quota
>>> violation.
>>> HDFS-7596. NameNode should prune dead storages from storageMap.
>>> HDFS-7611. deleteSnapshot and delete of a file can leave orphaned blocks
>>> in the blocksMap on NameNode restart.
>>> HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause
>>> DataNode to register successfully with only one NameNode.
>>> HDFS-7733. NFS: readdir/readdirplus return null directory attribute on
>>> failure.
>>> HDFS-7831. Fix the starting index and end condition of the loop in
>>> FileDiffList.findEarlierSnapshotBlocks().
>>> HDFS-7885. Datanode should not trust the generation stamp provided by
>>> client.
>>> HDFS-7960. The full block report should prune zombie storages even if
>>> they're not empty.
>>> HDFS-8072. Reserved RBW space is not released if client terminates while
>>> writing block.
>>> HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to
>>> finalize upgrade.
>>>
>>>
>>> Arpit
>>>
>>> [1] Will Hadoop 2.6.1 be released soon?
>>> http://markmail.org/thread/zlsr6prejyogdyvh
>>>
>>>
>>>
>>> On 4/27/15, 11:47 AM, "Vinod Kumar Vavilapalli" 
>>> wrote:
>>>
>>>  There were several requests on the user lists [1] for a 2.6.1 release. I
 got many offline comments too.

 Planning to do a 2.6.1 release in a few weeks time. We already have a
 bunch
 of tickets committed to 2.7.1. I created 

Re: BugBash!--> Yes, It's Bashed!!!

[Zhihai Xu]

Yes, this is a great event. Thanks Allen for organizing this event! It is
great to see so many JIRAs are resolved.

On Sat, May 9, 2015 at 12:20 AM, Brahma Reddy Battula <
brahmareddy.batt...@hotmail.com> wrote:

> Thanks Allen, It's great job, well done..
> I wish we can have this more often, or even periodically..
>
> And thanks to all the commiters and contributors who worked towards
> success of BugBash Day..
> > Date: Fri, 8 May 2015 19:34:11 -0700
> > Subject: Re: REMINDER! REGISTRATIONS CLOSING 5/6!
> > From: yzh...@cloudera.com
> > To: yarn-...@hadoop.apache.org
> > CC: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> mapreduce-...@hadoop.apache.org
> >
> > Thanks Allen a lot for organizing the Haoop Bug Bash, very happy to see
> > many folks by face!
> >
> > Wish you all a nice weekend!
> >
> >
> > --Yongjun
> >
> >
> > On Tue, May 5, 2015 at 9:31 PM, Allen Wittenauer 
> wrote:
> >
> > >
> > > On May 5, 2015, at 8:10 PM, Allen Wittenauer  wrote:
> > >
> > > >   * We’ll be closing registrations to the Bug Bash on May
> > > 6th at 3PM Pacific time.  So make sure you do it son:
> > >
> https://www.eventbrite.com/e/apache-hadoop-global-bug-bash-tickets-16507188445
> > >
> > > That should be *noon* Pacific time.  So just do it already, ok?
> > >
> > > [I can’t tell time.  Someone should buy me an Apple Watch
> Edition
> > > or something.]
> > >
> > >
> > >
>

[jira] [Created] (HADOOP-11667) Improve Credentials class for thread safe to avoid corruption for shared credentials between Jobs.

[zhihai xu (JIRA)]

zhihai xu created HADOOP-11667:
--

 Summary: Improve Credentials class for thread safe to avoid 
corruption for shared credentials between Jobs.
 Key: HADOOP-11667
 URL: https://issues.apache.org/jira/browse/HADOOP-11667
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Reporter: zhihai xu
Assignee: zhihai xu


Improve Credentials class for thread safe to avoid corruption for shared 
credentials between Jobs.
The shared credentials corruption happened at cascading job client:
https://github.com/Cascading/cascading/commit/45b33bb864172486ac43782a4d13329312d01c0e



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11599) Client#getTimeout should use IPC_CLIENT_PING_DEFAULT when IPC_CLIENT_PING_KEY is not configured.

[zhihai xu (JIRA)]

zhihai xu created HADOOP-11599:
--

 Summary: Client#getTimeout should use IPC_CLIENT_PING_DEFAULT when 
IPC_CLIENT_PING_KEY is not configured.
 Key: HADOOP-11599
 URL: https://issues.apache.org/jira/browse/HADOOP-11599
 Project: Hadoop Common
  Issue Type: Improvement
  Components: ipc
Reporter: zhihai xu
Assignee: zhihai xu
Priority: Trivial


Client#getTimeout should use IPC_CLIENT_PING_DEFAULT instead of  hard-coded 
value (true) when IPC_CLIENT_PING_KEY is not configured.
{code}
if (!conf.getBoolean(CommonConfigurationKeys.IPC_CLIENT_PING_KEY, true)) {
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11156) DelegateToFileSystem should implement getFsStatus(final Path f).

[zhihai xu (JIRA)]

zhihai xu created HADOOP-11156:
--

 Summary: DelegateToFileSystem should implement getFsStatus(final 
Path f).
 Key: HADOOP-11156
 URL: https://issues.apache.org/jira/browse/HADOOP-11156
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Reporter: zhihai xu
Assignee: zhihai xu


DelegateToFileSystem only implemented getFsStatus() and didn't implement 
getFsStatus(final Path f). So if you call getFsStatus(final Path f), it will 
call  AbstractFileSystem.getFsStatus(final Path f) which will also call 
DelegateToFileSystem.getFsStatus(). It should implement getFsStatus(final Path 
f) to call fsImpl.getStatus(f) instead of calling fsImpl.getStatus() from 
getFsStatus().




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11089) change "-tokenCacheFile" option to support non-local FS URIs.

[zhihai xu (JIRA)]

zhihai xu created HADOOP-11089:
--

 Summary: change "-tokenCacheFile" option to support non-local FS 
URIs.
 Key: HADOOP-11089
 URL: https://issues.apache.org/jira/browse/HADOOP-11089
 Project: Hadoop Common
  Issue Type: Improvement
  Components: util
Reporter: zhihai xu
Assignee: zhihai xu


change "-tokenCacheFile" option to support non-local FS URIs.
The current code in GenericOptionsParser only support local FS URIs, It will be 
better to support non-local FS URIs. 
{code}
FileSystem localFs = FileSystem.getLocal(conf);
  Path p = localFs.makeQualified(new Path(fileName));
  if (!localFs.exists(p)) {
  throw new FileNotFoundException("File "+fileName+" does not exist.");
  }
{code}
We can change above code to
{code}
FileSystem localFs = FileSystem.getLocal(conf);
  Path p = localFs.makeQualified(new Path(fileName));
  if (!p.getFileSystem(conf).exists(p)) {
  throw new FileNotFoundException("File "+fileName+" does not exist.");
  }
{code}
This issue will depend on MAPREDUCE-6086.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11035) distcp on mr1(branch-1) fails with NPE using a short relative source path.

[zhihai xu (JIRA)]

zhihai xu created HADOOP-11035:
--

 Summary: distcp on mr1(branch-1) fails with NPE using a short 
relative source path.
 Key: HADOOP-11035
 URL: https://issues.apache.org/jira/browse/HADOOP-11035
 Project: Hadoop Common
  Issue Type: Bug
  Components: tools
Reporter: zhihai xu
Assignee: zhihai xu


distcp on mr1(branch-1) fails with NPE using a short relative source path. 
The failure is at DistCp.java, makeRelative return null at the following code:
The parameters passed to makeRelative are not same format:
root is relative path and child.getPath() is a full path.
{code}
final String dst = makeRelative(root, child.getPath());
{code}

The solution is 
change root to full path to match child.getPath().



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HADOOP-10876) The constructor of Path should not take an empty URL as a parameter

[zhihai xu (JIRA)]

zhihai xu created HADOOP-10876:
--

 Summary: The constructor of Path should not take an empty URL as a 
parameter
 Key: HADOOP-10876
 URL: https://issues.apache.org/jira/browse/HADOOP-10876
 Project: Hadoop Common
  Issue Type: Bug
Reporter: zhihai xu


The constructor of Path should not take an empty URL as a parameter, As 
discussed in HADOOP-10820, This JIRA is to change Path constructor at public 
Path(URI aUri) to check the empty URI and throw IllegalArgumentException for 
empty URI.



--
This message was sent by Atlassian JIRA
(v6.2#6252)